[Info-vax] scp or sftp: file is "raw", needs to be parsed - possible to work around that?

[Richard Whalen]

On Sunday, May 23, 2021 at 9:23:05 PM UTC-4, Arne Vajhøj wrote:
> On 5/19/2021 2:00 PM, Gregory Reut wrote: 
> > On Wednesday, May 19, 2021 at 2:38:39 PM UTC+2, Arne Vajhøj wrote: 
> >> On 5/19/2021 7:29 AM, Gregory Reut wrote: 
> >>> Trying to scp a very large file off an Alpha VMS 8.3 system. Larger the file, longer it takes to start the transfer. This scp message in verbose mode gives a clue: 
> >>> SCP Source file is "raw", and it needs to be parsed 
> >>> 
> >>> Parsing a 20GB file takes longer than a standard scp timeout. and I need to scp files 10 times larger. 
> >>> Perhaps there is a way to skip the parsing? 
> >> I think there should be an option: 
> >> 
> >> SCP /JUST_COPY_THE_BYTES_PLEASE ... ... 
> >> 
> >> but in case there is not then maybe cut it into smaller chunks 
> >> like 2 GB at source system and combine back at target 
> >> system.
> > Unfortunately I didn't find any way to force scp to skip the parsing yet. 
> > Regarding slicing the file - I'd definitely do that if I know how to 
> do it :) Would you have a hint for me?
> There were a reference to a SPLIT.C - I have not used it myself, 
> but sound like exactly what you need. 
> 
> Arne


Even when the file doesn't need to be read the way that the SCP/SFTP2 code base that MultiNet, TCPware, SSH for OpenVMS will take longer to start transferring a large file than a small file because the code makes a list of "work items" to copy parts of the file. The code doesn't start transferring data until it has made work list to cover the entire file. This can use a lot of memory and time.  You can reduce the amount of memory required for the work list, and the time required to create the work list by increasing the transfer buffer size.