[Info-vax] FTP FYI

[geze...@rlgsc.com]

All,

I previously commented in this thread that I make it a practice to ZIP BACKUP Save Sets before transferring them over networks using FTP. This does reduce the size of the file, and make for smaller transfers, but the genesis of my habit was error control.

Today, when most connections have extraordinarily low error rates, the concern often seems quaint, until it is not.

I first discovered this danger when working with the VAX-11/780 connected to a pair of PDP-11/34 systems using DMC-11 1M/bit triaxial coax links. With the DMC-11 adapters supporting the DDCMP protocol, including CRC-16 on all messages, one would have reasonably thought that transmission errors were not a problem. That was incorrect. A bug in the DMC-11 would, on occasion, miss bytes when moving data between main memory and the device. 

This could show up in many ways, including encountering the "Recovery Groups" message from BACKUP. BACKUP was the least dangerous case, not all files are processed by programs checking for file integrity.

More generally, there are many ways for corruption to occur on a network transfer. The guarantees provided by the TCP/IP stack (specifically TCP and FTP) are far weaker than those provided by DDCMP/Ethernet/DAP. ZIPing a file provides an easy way to verify that a file is intact. As volumes have increased, and multiprocessors have become ubiquitous, I have transferred the drudgery of running ZIP to batch jobs. 

For large scale transfers, where I am concerned about OpenVMS-specific details (e.g. ACL, Protections, etc.) I use BACKUP to create a save set, then ZIP the save set (preserving the RMS attributes of the save set), then transfer the ZIP archive. On receipt, the ZIP archive is tested for integrity, then UNZIPed.

This choreography ensures data integrity. Discovering data corruption at a later point is much more difficult to remediate.

- Bob Gezelter, http://www.rlgsc.com