[Info-vax] Large mailboxes - RMS Indexed file internal design

[Hein RMS van den Heuvel]

On Sunday, December 6, 2020 at 7:07:22 PM UTC-5, Craig A. Berry wrote:
:
> Maybe I'm missing something, but moving from 16-bit IDs to 32-bit IDs in 
> the RFA would give you 4,294,967,295 records per VBN

I don't think you are missing something in this long winding discussion.
This subthread ( I tried changing the subject text ) was started to explain why (indexed) files with few active records most of the tiem may still grow large.
One possible reason listed was that a given bucket (vbn) can only hold 64K records over its live and that would indeed be completely resolved with a 32 bit id field.
Al long as we all realize that a given bucket can only hold a few, ok maybe hunderds, active records at any given time.

>> So the total number of records in a file that one could reference via RFA could be 4,294,967,295 * 4,294,967,295?

I'd rather rewrite that as saying that that many unique RFAs may have existed over the life or a file.
Even that's a stretch as it assumes a one block bucket size and no (index) overhead.
The max VBN for a file is 4G, but the max number of buckets is 4G divided by the bucket size. 
For a typical 16 blocks, that's 250M buckets. for a max bucket  size of 63 that's 68M unique buckets.

In real life I've seen problems with this work-queue files with a few thousand messages per day peaking a few times per day leading to trouble (significant performance impact) after several months or even years. That's easily resolved with a maintenance window  convert.

A few applications run at several messages per second and for those the files would explode after a few days or weeks. 
That is often only found after years of slow growth leading 'suddenly' leading to near catastrophic outages fixed by a daily cleanup cycle or re-write.
The reason it 'suddenly' creates a problem is because the reading process starts to need more and more time to find the next record to process and in that time new records start to overflow their buckets more often causing the reader to fall behind even more. - That's when the phones start to ring or email storms start up.

Cheers,
Hein