[Info-vax] RMS journaling - how it works (for dummies)

[Brian Schenkenberger]

On 2023-07-16 14:52:56 +0000, Hein RMS van den Heuvel said:

> On Sunday, July 16, 2023 at 6:51:52 AM UTC-4, d45 n0x8!7 wrote:
>> Hello Old$Boys!>> Can u help me to search a reference to info about of 
>> RMS journaling? May be DEC Journal or something other ?
> {...snip...}
> 
> A relatively common Journaling desire is Change Data Capture (CDC) 
> which RMS Journaling does NOT offer.
> The desire there being to have access to individual changed to records, 
> either for auditing purposes (who changed what when) or to be able to 
> forward such changes to another system or to a database.
> 
> There is a Commercial product called RMS-CDC owned by Brian 
> Schenkenberger which can do full CDC for any and all file changes  
> including before-images based on a magic RMS intercept.

The "intercept" isn't "magic", it's the only way to do this and do this 
properly! ANY and ALL RMS operations that can affect a file's contents 
must be captured. The record's data is obtained from the bowels of RMS. 
Not everything to be recorded or "captured" is passed as an argument! 
The BEFORE IMAGE is the real magic!

BEFOE IMAGE??? What's in the record BEFORE an $UPDATE? What WAS in the 
record BEFORE a $DELETE? It's all magic from deep in the bowels of RMS. 
Believe me, it's not simple and it's not as simple as performing a $GET 
before doing an $UPDATE or a $DELETE. There's a lot of unpleasant 
shhh..tuff in those bowels. Indexed files are expecially onerous 
because a $GET, which would seem an obvious way to obtain the BEFORE 
IMAGE, can and will corrupt the record pointers for the $UPDATE and or 
$DELETE. It has/had to be done without affecting the actual RMS call's 
functional operation. It's also done ***without*** performing 
additional I/O!


> VSI has a prototype CDC based on the standard SSI (System Service 
> Intercept) which required the programs to cooperate and declare they 
> want they actions to be intercepted. It has good potential but does not 
> offer Before Images (which may or might not be critical).

I can't see how that can/could work. SSI calls need to be conciously 
incorporated into an/the image that would be doing the RMS calls. 
Could/would this capture RMS from DCL, CONVERT, etc? without code 
modification? What about the BEFORE IMAGE? Good luck there without 
privileged code.

RMS-CDC works without any code modification needed. It's installed and 
loaded, and then the files that are of interest for change data capture 
are added to RMS-CDC's files-of-interest database. From that point on, 
anything that can and will modifiy a file's contents is captured and 
squirreled away in a log file for consumption for whatever purpose -- 
as Hein mentioned such as auditing or database or forwarding to other 
systems not necessarily VMS systems -- deemed necessary.

If a system is in a cluster, RMS-CDC needs to be on every node of the 
cluster because files can be accessed from every node in the cluster. 
This requires special syncronization and sequencing to keep all CDC 
data coordinated.

I know that Hein knows all of this because I'd worked closely with him 
for several years to implement RMS-CDC. This reply is for the 
enlightenment of others.

— VAXman