[Info-vax] CRTL and RMS vs SSIO

[Dave Froble]

On 10/6/2021 9:25 PM, Greg Tinkler wrote:
> What a good conversation, some feedback.
>> To be honest then I think the safest way to implement this is
>> to put lots of restrictions on when it is doable.
>>
>> Examples:
>> * No cluster support (announcement already states that!)
>> * Only FIX 512, STMLF and UDF are supported
>> * no mixing with traditional RMS calls
>
> My point is SSIO seems to be focused on just PostgreSQL, whereas an RMS solution is much much easier to program, uses well tested code, and is already cluster ready putting the team ahead of the game and not building issues for the future.

RMS is a bit too high level for what's being discussed.

But yeah, the real issue is that SSIO was aimed (it seems) at 
PostgreSQL.  In my opinion, that is poor software architecture and design.

>> I've a database product, a rather old product. At the time it was
>> implemented it was rather useful. But there was a locking issue. The
>> DLM locks resource names. The database would support I/O transfers of 1
>> to 127 disk blocks. How would one lock 127 contiguous disk blocks? The
>> blunt force method could be taking out 127 locks, not an optimum
>> solution. Having numeric range locking back in 1984 would have been
>> quite useful.
>
> Yup DLM uses resource names, but they can be hierarchical, like a B-Tree index. Also the resources need only exist when needed, removed it not.  The the lock tree size depends on the lock contention.

Well the perceived issue is what happens when taking out locks, and at 
some point there is a conflict.  Say needing 127 blocks locked, and the 
conflict is on the last block.  That means 126 locks to be released, and 
perhaps try again.

In reality, the large I/O buffer capability is rarely used, and then 
it's usually with exclusive file access, which precludes the need for 
block locks, just the file lock.  For random access, single block 
locking and I/O is good.  Larger I/O buffers are usually used for 
sequential access, both read only, and updating.

> This is why I made reference to Rdb, it uses this technique, and they are probably not the only ones.  NB each level controls a range of resources and each level can have it’s own fan out factor.  The depth and lowest level is always dependant on the applications requirements.
>
> FYI I am pretty sure RMS uses RFA to lock a record, this is an implied range of 1 record.

RMS has some interesting internals, basically below application usage.

Global buffers
Multiple buffers
Multi-block count

RMS can (I believe, it's been a long while) keep track of file usage, 
and provide data from an RMS buffer to a user's buffer.  No disk 
activity required.  Writes of course must go to disk.  But even so, the 
data can still be in the updated global buffers for use by multiple tasks.

>> No matter what the disk can do then the VMS file system is still
>> block oriented and I believe the system services take block offsets
>> not byte offsets.
> All disks are block based, even on Unix.  With some SSD’s yes you can do byte transfers, but this should be left to the driver to optimise.  Also with X86_64 it weill be virtualised so what the..

As long as storage is block oriented, then regardless of the numeric 
range of bytes, all blocks encompassing the byte range will need to be 
read, including locking, and written.  This usually will include data 
outside the byte range.

>> For this case, RMS really doesn't work at all well. Says why right
>> there in the name, too. Record management, not stream management.

Ayep.  RMS is record based.

> Well yes and no.  If you think about it most Unix text IO is record, ie LF terminated, and binary is fixed records not necessarily the same length in the file.
>
> RMS for $GET and $PUT are record based, but $READ and $WRITE are block based, missing is $READB and $WRITEB, not just for CRTL but useful for various applications.

Forget RMS, I/O would be at the QIO level.

> RMS ISAM with fixed length records is a pain, I have long argued ISAM should support variable length records, don’t care if they are VFC or STMLF, I would allow for both as VFC could allow for binary variable length records.

RMS keyed files can have variable record lengths.
RMS relative files require fixed length records. (if I remember correctly)
RMS sequential files can have variable record lengths.

> Likewise the keys on an ISAM file should be able to be variable based on a separator e.g “,” or <tab> or a combination.
>
>> The use of Oracle Rdb isn't viable as a dependency for many folks, and
>> lock granularity doesn't work at all well for arbitrary and overlapping
>> locking ranges.
> I think you will be a B-Tree style dynamic resource tree, similar to what Rdb uses, will work well.  Any ‘byte range’ implementation will need some index to find interesting locks, DLM uses hash which is as efficient as you can get.
>
>>> Sure it may be nice to have an API that does this for us, but hey we
>>> are programmers.
>
>> I don't want us each writing and debugging and maintaining
>> range-locking code for what is part of the C standard library, but you
>> do you.
> NO, quite the opposite.  I believe there is a POSIX standard for a locking API, and as VMS, sorry OpenVMS, wishes to maintain its POSIX stamp it should use these API’s using DLM underneath.  NB DLM is also already cluster based, but you know that.
>
>> People used to UNIX or Windows generally find the other VMS file types
>> baffling and confusing.

That is because, without additional apps, Unix I/O is a stream of bytes. 
  There is no concept of records, such as that provided by RMS.

Frankly, (and yes, I'm biased), I find records reasonable, and a stream 
of bytes baffling and confusing.  Guess it's what one is used to.

> I always wondered why the CRTL did not have some smarts to present a VFC records as STMLF and vise-versa, effectively hiding the internal record structures.  This could be done via open using the VMS extension “rfm=STMLF” which should be the default unless it is a binary file “rfm=unf”.  If the file is VFC then CRTL could to the translation.  Wishful thinking.

I would suggest the use of "VMS" in the above, rather than "CRTL".  That 
is unless one considers the CRTL VMS ...

> gt down under
>


-- 
David Froble                       Tel: 724-529-0450
Dave Froble Enterprises, Inc.      E-Mail: davef at tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA  15486