[Info-vax] scp or sftp: file is "raw", needs to be parsed - possible to work around that?

[Richard Whalen]

On Monday, May 24, 2021 at 12:03:57 PM UTC-4, gregor... at gmail.com wrote:
> On Monday, May 24, 2021 at 3:15:11 PM UTC+3, rvwh... at gmail.com wrote: 
> > On Sunday, May 23, 2021 at 9:23:05 PM UTC-4, Arne Vajhøj wrote: 
> > > On 5/19/2021 2:00 PM, Gregory Reut wrote: 
> > > > On Wednesday, May 19, 2021 at 2:38:39 PM UTC+2, Arne Vajhøj wrote: 
> > > >> On 5/19/2021 7:29 AM, Gregory Reut wrote: 
> > > >>> Trying to scp a very large file off an Alpha VMS 8.3 system. Larger the file, longer it takes to start the transfer. This scp message in verbose mode gives a clue: 
> > > >>> SCP Source file is "raw", and it needs to be parsed 
> > > >>> 
> > > >>> Parsing a 20GB file takes longer than a standard scp timeout. and I need to scp files 10 times larger. 
> > > >>> Perhaps there is a way to skip the parsing? 
> > > >> I think there should be an option: 
> > > >> 
> > > >> SCP /JUST_COPY_THE_BYTES_PLEASE ... ... 
> > > >> 
> > > >> but in case there is not then maybe cut it into smaller chunks 
> > > >> like 2 GB at source system and combine back at target 
> > > >> system. 
> > > > Unfortunately I didn't find any way to force scp to skip the parsing yet. 
> > > > Regarding slicing the file - I'd definitely do that if I know how to 
> > > do it :) Would you have a hint for me? 
> > > There were a reference to a SPLIT.C - I have not used it myself, 
> > > but sound like exactly what you need. 
> > > 
> > > Arne 
> > Even when the file doesn't need to be read the way that the SCP/SFTP2 code base that MultiNet, TCPware, SSH for OpenVMS will take longer to start transferring a large file than a small file because the code makes a list of "work items" to copy parts of the file. The code doesn't start transferring data until it has made work list to cover the entire file. This can use a lot of memory and time. You can reduce the amount of memory required for the work list, and the time required to create the work list by increasing the transfer buffer size.
> Hi Richard, 
> Which buffer is that? Or, better, how to increase to and to what size? 
> Thanks in advance! 
> 
> @the community - thank you very much for lots of ideas and clarifications, it is a big help! 
> 
> Best regards, 
> Greg


For MultiNet/TCPware/SSH for OpenVMS

On either the SCP2 or SFTP command line

/BUFFER_SIZE=integer

Number of bytes of data to transfer in a buffer. Default is 7500.