[Info-vax] VSI roadmap

[Arne Vajhøj]

On 8/23/2023 4:24 PM, Arne Vajhøj wrote:
> On 8/23/2023 8:47 AM, David Jones wrote:
>> On Tuesday, August 22, 2023 at 8:05:25 PM UTC-4, Arne Vajhøj wrote:
>>> Are there some details that could have been done
>>> different and better? Absolutely! Having record
>>> size limit of 2 GB (32 bit length) instead of
>>> 32 KB (16 bit length) would have been nice. If VAR
>>> files has length both as line header and line trailer,
>>> then such a file could easily be read backwards. I am
>>> sure that if Hein could travel back in time he could
>>> have optimized caching a lot. But wanting to change
>>> some implementation details 45 years later is
>>> no unique for RMS.
>>
>> I'm not sure that's true, given how slow hardware was 45 years ago. 
>> Adding
>> 6 bytes of meta-data to a VAR file with an average record length of 30
>> characters will increase the size by 18%. More space on disk and more I/O
>> operations to process the file.
> 
> I am not sure that I can follow the math.
> 
> Current VAR record overhead is header 2 bytes + average 0.5 even
> pad byte and 2.5/30 is 8.3%.
> 
> Increasing to 32 bit would mean header 4 bytes + average 0.5 even
> pad byte and 4.5/30 is 15.0%.
> 
> Increasing to 32 bit and add trailer length to support reading
> backwards would mean header 4 bytes + average 0.5 even pad byte
> + trailer 4 bytes and 8.5/30 is 28.3%.
> 
> Yes - it is some overhead. But NOS/VE is almost as old as
> VMS and used header 6 bytes and trailer 6 bytes. They could
> apparently live with it.

There were some areas where the 32 KB limit problem showed up
even back in time:
* having to split a logical record up in multiple physical
   records in index-sequential files using key suffixes or some
   other key scheme
* Fortran segmented files
* stream/stream_lf files being considered technically invalid
   if more than 32 K between EOL markers (including by
   CONV$ functions)

Arne