[Info-vax] Streaming a File on OpenVMS with Caché

[Stephen Hoffman]

On 2015-01-15 14:14:52 +0000, Mack Altman III said:

>> 
>> Do we care?  No.  What we care about is what's in the
>> file.  Is it text or binary data?
> We have both text and binary.

That's not particularly helpful.  That's a "we need everything" sort of 
answer, and is not an answer that can both isolate the problem here and 
that can reduce the complexity of the problem, and that can better lead 
to a solution.   Solving for arbitrary file transfers — via ftp or 
otherwise — is well beyond the scope of a newsgroup, and outside of the 
budget that most folks are operating with.

Now for some questions...

1: Irrespective of what a user might choose for an ftp transfer, what 
is the native format of and the nature of the data that needs to be 
uploaded here?   There was a reference upstream to some sort of 
configuration data, but configuration data can have any of a large 
number of very different potential formats.

2: Do you have any of your existing software installed on the client 
system that is most closely associated with the data that needs to be 
uploaded, or can some of your software be installed?


>> Apparently, your clients use some kind of FTP client program
>> to send the file to your VMS FTP server.
> I've said, "Our clients FTP the file to us so we cannot see what their 
> current settings are." If you're referring to the FTP client program 
> used, are you saying that the FTP client (e.g. CMD, FileZilla, FireFTP, 
> etc.) matters?

Different programs can have different behaviors and different defaults, yes.

Just to keep this interesting, different networks have different 
firewalls, and FTP is incompatible with modern network security.  FTP 
requires opening up the ephemeral port range is a fairly large security 
exposure, too.  Other network activity can use those same ephemeral 
ports, and can thus be exposed.


>> Is the file being transferred in text/ASCII or binary mode?
> I've said, "As far as we go with the FTP process is telling them to use 
> ASCII/BIN." That said, we can advise them to transmit via either.

Your server software cannot deal with arbitrary format files.


>> I'm guessing from "Record format: Variable length" that it's a 
>> text/ASCII transfer.
> If you can let me know how I'm supposed to be able to determine whether 
> the file is ASCII or BIN without asking the person who uploaded the 
> file, I could definitely let you know. Otherwise, you're assumption is 
> as good as mine.

Here are a few areas where I'd consider the existing design, and quite 
possibly rework the design, and why:

a: users aren't good at the command line, and FTP transfers even from a 
GUI can be a problem for experienced users.  Some will just mail the 
file.
b: if the user gets to choose the file formats, they don't and won't 
perform that selection consistently, and they won't follow the 
directions.
c: The servers can be open to random folks that might transfer random 
data into the ftp server, just for giggles, or to try to break the 
server, or in an attempt to store warz on the server.  Particularly if 
the credentials can't easily be changed and/or the credentials leak.
d: FTP is wildly insecure, and both the login credentials and the data 
in the transfer will be exposed to anyone with a privileged network 
position.
e: The server software is going to have to parse the low-level file 
formats of the file before it passes the data into that stream 
mechanism.  That's one part of what is being experiencing here, and 
this only gets more complex as additional file formats can be selected 
and uploaded, and must then be processed.  This is the format of the 
file contents, and is beyond the binary- and text-format FTP transfers 
performed within ftp.
f: There are files that will need to be managed at the client, and that 
can or will eventually need to be deleted.
g: There are files that will need to be managed on the server, and 
those too will eventually need to be deleted.
h: There's no easy way to change the authentication credentials.
i: users will pick random filenames, and there's no reliable means to 
associate a particular file with a particular source.  Maybe if you 
peek into the file, but then that's still under the control of the 
remote site.
j: there's no end-to-end encryption with ftp, which means the data in 
the transfer is exposed to anyone with a privileged position.

There are probably other issues lurking here, too.

One obvious alternative is to deploy your own data upload tool onto the 
client, either as part of an existing application environment or 
executable image, or as a separate tool, or that gets downloaded into a 
web browser on the client system and performs the necessary upload from 
there.  If this is part of the configuration data from some software 
package that's been installed on the box, then extending that to open 
up an SSL connection allows the client box to shovel the data to the 
server in a format that you (mostly) control.  No files.  No ftp.  No 
firewall issues.  Encryption.  Easily-available per-user or per-site or 
per-installation encryption keys that can also allow you to identify 
the associated source of the data even during the network connection 
too, if you want or need that.  As mentioned earlier in the thread, I'd 
tend to use libwww, libcurl or some other similar library and probably 
with REST access to the server via SSL / https TCP Port 443 as that 
port is very seldom blocked, and would completely control the format 
and encoding of the data being uploaded, but there are many other 
potential approaches available for automating the upload.  A Python or 
Powershell or other script would work, for instance.


> At this moment, I have a BIN file with the following DIR/FULL. In doing 
> a DUMP/REC, each record is terminated at 512 bytes. When reading the 
> file, the CR,LF also terminates the record.

If there's a CR-LF in the DUMP of a fixed-length 512-byte file on a VMS 
system, then the CR-LF is embedded in the file, and is originating in 
the source file out on the remote client that's performing the upload, 
or it's originating in the ftp client, or in the ftp server.  The 
latter is unlikely.




-- 
Pure Personal Opinion | HoffmanLabs LLC