[Info-vax] OpenVMS - DCL - Data entry filtering

[Stephen Hoffman]

On 2015-03-30 00:08:03 +0000, JF Mezei said:

> On 15-03-29 18:03, Stephen Hoffman wrote:
> 
>> MAIL is a (good?) (bad?) example of what happens when you don't have a 
>> modern database, when you do have to work within the limitations on how 
>> fast you can scan directories (and this also from within a
> 
> But MAIL was originally designed like NOTES, with the expectation that 
> most messages would be stored in the main indexed MAIL.MAI file.

Other than that Notes doesn't use external files for any of its message 
store, AFAIK.  The internal organizations of Notes and Mail are quite 
different.  But you undoubtedly know that, and more.   Then there's 
that I wrote "when your designs can end up used at scales vastly beyond 
what was likely originally intended," further along in that paragraph 
that you had excerpted.   Which leads me to ask again, are you doing OK?

> VMS designers have to decide whether to go with the flow and just make 
> a file system very efficient for large number of files (turning the 
> file system into a database) or  provide database tools that allow apps 
> to store large number of "things" in a single file instead of one file 
> per "thing".

Which is how an operating system evolves, updating limits and updating 
features and variously replacing existing designs to meet new 
expectations and new requirements.

Where VMS was in the 1980s is increasingly unusable today, or can be 
more easily and cheaply dealt with on an Intel NUC 
<http://www.intel.com/content/www/us/en/nuc/products-overview.html>.  
So VMS changed.

As I've commented before, the file system as an application data store 
usually only looks good if you don't already have more appropriate 
database package readily available.

Unfortunately, the resulting morass of these older designs tends to 
become familiar and even normal to folks that use them, and which 
almost inevitably devolves into a gazillion different packages 
maintaining their own database schemes — Notes or Mail or SYSGEN or 
DECnet or TCP/IP or PCSI or AUTHORIZE or the rest — with some using the 
file system, some using RMS, some using the file system, and some 
embedding a database, and some using a combination.

This all works fine for some applications, and particularly for 
existing and older and smaller applications.   But programmers 
eventually end up facing the same basic problem that RMS was intended 
to solve.  Having a set of data access routines.   Preferably a common 
set of data access routines, and related tools.   Preferably 
integrated.  RMS was intended to allow programmers to solve these 
applications requirements fewer times and with less code, and 
preferably with a set of common solutions that would even allow 
inter=operations among packages.   For its time, RMS did quite well.  
RMS still works well for some cases, too.  It is after all a 
quite-reliable NoSQL database.

As applications deal with larger and larger piles of data and more 
complex relationships within and among that data, application 
programmers are left to use the file system as a database, or to use a 
database — this because applications increasingly have much more data, 
and much more complex data, and much more complex relationships between 
data and accessors and logging and the rest.   You blow past what RMS 
can do.   You might well reach the absurdity of directly rummaging 
SYSUAF.DAT, as I and others have implemented.  That rummaging is easy 
thanks to RMS, and nasty because changes to the record formats becomes 
entirely too visible to the applications.   Transactions and data 
abstractions become interesting even to some of the operating system 
components, such as to PCSI and the authorization database — you really 
want some data all written, or all not written.   This leads some to 
purchase and use Rdb — a relational database which would be nice to 
have in the base OS, but I just don't see that happening — or to the 
integration of databases such as SQLite or PostgreSQL.  Once these 
databases and packages start to breed, you get back to the complexity 
and the application-specific solutions that the introduction of RMS was 
intended to avoid, and the application developers and the operations 
folks and the support folks then get to deal with a whole zoo of 
different solutions.  Which, again, was part of what RMS was intended 
to solve.

Where does this commonality of availability all go?  It'd be nice, for 
instance, if OpenVMS had a mechanism that could coordinate with 
applications and RMS WBH caches and Rdb and the rest, so that you could 
get a consistent copy of the data when you use BACKUP or when you split 
off a shadowset member.  This via package-specific plug-ins or 
callbacks or otherwise, and already available as part of the default 
database packages — RMS, or otherwise — within the platform.





-- 
Pure Personal Opinion | HoffmanLabs LLC