[Info-vax] Large mailboxes

[Tad Winters]

On 12/1/2020 12:02 PM, Michael Moroney via Info-vax wrote:
> The discussion of mailboxes and MQ kind of reminded me of an old problem:
>
> Consider an application which produces a fairly large number of small (100-200
> byte) messages. The messages are sent via TCP to another system, and this can't
> be changed. (which is why something like MQ wasn't used)
>
> We don't want to lose any messages.
> In order to do that, the code was changed to write the messages to an indexed
> file (1 key, the timestamp).  A second process reads records ftom the file,
> sends to the remote system and upon acknowledgement, deletes the record.
> (if no record, it sleeps and tries again)  If the link/remote system was down,
> messages would accumulate in the indexed files.  Once the link was back, the
> second process would empty it.
>
> In other words, an RMS indexed file was just a FIFO. By being a file, records
> are not lost. That worked, but despite the file being almost always empty or at
> most 1-2 records, the file grew and grew and that part of the system slowed. A
> workaround was to create a new version of the file once in a while, and delete
> the old one (once known to be empty).  A $ CONVERT/RECLAIM on an empty or
> nearly empty file would chew on it for a while and would make the file faster
> [note this was done only to understand what was going on].  The slowness was
> from accumulated RMS cruft from all the deleted records.
>
> So in theory, is it possible to implement a VMS file as a FIFO and work
> efficiently?  That is one process can write records in order, a second
> process can read and delete records in order, without RMS cruft from
> accumulating?
> _______________________________________________
> Info-vax mailing list
> Info-vax at rbnsn.com
> http://rbnsn.com/mailman/listinfo/info-vax_rbnsn.com
>

I wouldn't be inclined to use indexed files because of the overhead.  If
I had to use them, maybe I'd design the solution with two indexed files,
one for the first half of a day and the other for the second half.  If
the consumer has removed all the records from one of the indexed files,
and the current time is now after the time the other indexed file should
be in use, it renames and then deletes the current file.  It can then
look for the other indexed file from which to read.  Meanwhile, the
producer writes to the proper indexed file, according to the time of
day, creating a new file, if necessary.

I think I would prefer to use relative files, with record number 1
containing the values of the head, tail, and total number of records
used so that it would be a circular buffer.  I think with one more
value, you could make if possible for the total number of records to
increase, when the buffer becomes empty or on the next cycle.  Relative
files are much faster than indexed files, in my experience.