[Info-vax] Empty blocks in FILES-11 directory files.

[Mark Daniel]

On 25/9/2022 5:01 pm, Volker Halle wrote:
> Mark,

Thanks for the detailed response Volker.

> does your utility read until the 'End of file block' or the 'Highest block' of the directory file ?

It uses a "generic" file read function as "raw" blocks into memory.  It 
reads the exact number of bytes indicated by the file header.

    if (odsptr->XabFhc.xab$l_ebk <= 1)
       SizeInBytes = odsptr->XabFhc.xab$w_ffb;
    else
       SizeInBytes = ((odsptr->XabFhc.xab$l_ebk-1) << 9) +
                     odsptr->XabFhc.xab$w_ffb;
    /* round-up ensure we have a full complement of blocks allocated */
    odsptr->DataPtr = (char*)VmGet (((SizeInBytes/512)+1)*512);
    odsptr->DataLength = 0;

    odsptr->Rab.rab$l_ubf = odsptr->DataPtr;
    for (;;)
    {
       if (SizeInBytes > 512*127)
          odsptr->Rab.rab$w_usz = 512*127;
       else
       if (SizeInBytes > 0)
          odsptr->Rab.rab$w_usz = SizeInBytes;
       else
          break;

       status = sys$read (&odsptr->Rab, 0, 0);
       if (VMSnok (status)) break;

       odsptr->DataLength += odsptr->Rab.rab$w_rsz;
       odsptr->Rab.rab$l_ubf += odsptr->Rab.rab$w_rsz;
       SizeInBytes -= odsptr->Rab.rab$w_rsz;
    }

    sys$close (&odsptr->Fab, 0, 0);

So I guess to the highest block.

> Free blocks should only be at the end of the directory file - after the EFBLK.
> 
> The logical end of the directory would be the last 'directory record' in the 'End of file block' of the directory file, i.e. a directory record with DIR$W_SIZE=-1 marks the end of directory records in the current block and therefore the end of the directory in the 'End of file block'

Then it looks like the code reads the size of the file bytes and so 
scans past the DIR$W_SIZE=-1 in the end of file block and on to the end 
of the file itself.

Having been emptied by file name deletions mid-file and then 
(effectively) "shuffled" to the end of the file these should all contain 
DIR$W_SIZE=-1 and therefore qualify as empty blocks, as per your later 
post that included ...

> When compressing a directory file, EFBLK will be updated, but the blocks between EFBLK+1 and the highest block allocated will not be overwritten with zeroes, even if 'file high-water marking' is set.

> $ DUMP/DIR directory_file.DIR should show the last record as:  'xxx End of records (-1)'  in the EOF block.
> 
> The function to 'shuffle' a directory (extend or compress) is called  SHUFFLE_DIR, found in module [F11X]SHFDIR

:-)  Seems all engineering types go for unimaginative (but 
understandable) nomenclature.

It also indicates that any accessible directory file should not contain 
embedded empty blocks as they should all have been shuffled to the end.

> Directory files are and must be contiguous. DUMP/HEADER directory.DIR shows:

Still a design consideration then.

>      File characteristics:                 Contiguous, Directory
> 
> This allows more efficient reads/writes and prevents window turns, when already executing inside the XQP.
> 
> Volker.

> Mark, what is your definition of 'empty blocks' - all zeroes ?

No, not zeroes.

             /* if not end of records (in this block) sentinal */
             if (rlen && rlen != (uint)0xffff && rlen < 508)
             {
                /* records are always on even byte (word) boundaries */
                rzptr = rptr + rlen;
                if ((int)rzptr & 1) rzptr++;

                /* break from loop to process record */
                EmptyBlockCount = 0;
                break;
             }
             else
             if (rlen && rlen != (uint)0xffff && rlen > 508)
             {
                status = SS$_BUGCHECK;
8< snip 8<
/* sanity check report */
                break;
             }
             else
             if (EmptyBlockCount++ > 16)
             {
                /* empty blocks should only occur at end of directory */
                break;
             }

The |if (EmptyBlockCount++ > 16)| is recently revised code based on the 
strong feeling from c.o.v. posts all "empty blocks" from the "shuffle" 
should be at the trailing end of the directory file.  This used to be a 
sanity check because I was originally unsure of the role of these.

I should add that this code has been in use for some years and apart 
from a handful of (corrected) bugs has been (appearing to) perform well.

-- 
Anyone, who using social-media, forms an opinion regarding anything 
other than the relative cuteness of this or that puppy-dog, needs 
seriously to examine their critical thinking.