[Info-vax] RMS record metadata, was: Re: Re; Spiralog, RMS Journaling (was Re: FREESPADRIFT)

[Paul Sture]

On 2016-06-18, Simon Clubley <clubley at remove_me.eisner.decus.org-Earth.UFP> wrote:
> On 2016-06-18, Paul Sture <nospam at sture.ch> wrote:
>> On 2016-06-18, Stephen Hoffman <seaohveh at hoffmanlabs.invalid> wrote:
>>
>>> Do you count that metadata in the file size, or not? Arguably no, but
>>> it's inherently part of the file on OpenVMS and not at all easy to
>>> exclude — whether VFC or the indexed file structures or otherwise.
>>
>> Tricky.  If you are looking at the output of TYPE, then the record
>> attributes (e.g. Record attributes Carriage return carriage control)
>> make a difference to the output.
>>
>
> IMHO, it depends on the context. If you are doing block level reads,
> then you count the size of the on-disk metadata in the file size.
>
> If you are doing record level reads, then you do not include the
> metadata IMHO but you _do_ add in any additional terminator bytes to
> the length (which might not actually be stored on disk).

Yes for things like TYPE and the print symbiont, but typical end-user
applications don't need to interrogate those attributes, and the
output format is probably going to be vastly different from the
input format; I'm thinking of something like transactions headed out
of an ordering system into an accounts system here, with a printout
for audit purposes along the way.

> You need the former if you are doing an image copy. You need the latter
> if you want to tell a webserver at which point it should resume the
> download.

In a pure VMS context, at one time I would have been thinking of RFAs
there...

> Also, IMHO I think the default sequential record type for today's
> world should be stream (and hence the terminator is also included in
> the file data.) I do not think it should be variable length records.

Certainly for general text files which are to be presented via a web
interface, but we're quickly back to legacy debt for applications
dealing with transaction files and the like.  Can existing COBOL or
Fortran programs cope with stream format files without modification?

(There's a way to find out, but I'm pressed for time right now)

> The problem is that today's protocols are simply not designed to
> handle the case of metadata buried within the file contents
> themselves; the people who design this stuff have probably never
> even encountered that case.
>
> I do wonder how the IBM mainframe people handle this problem.

Throw up a load of Linux instances? I'm only half joking here; IBM
didn't invest in the ability to do that for fun.

-- 
There are two hard things in computer science, and they are cache invalidation,
naming, and off-by-one errors.