[Info-vax] The best VMS features, was: Re: openvms renaming file

Tue May 29 11:10:56 EDT 2018

On Tuesday, May 29, 2018 at 2:48:34 AM UTC-7, Richard Maher wrote:
> On 28-May-18 11:45 PM, Jim Johnson wrote:
> > On Sunday, May 27, 2018 at 9:41:37 PM UTC-7, Stephen Hoffman wrote:
> >> On 2018-05-27 23:32:18 +0000, Simon Clubley said:
> >>
> >>> On 2018-05-27, Phillip Helbig (undress to reply)
> >>> <helbig at asclothestro.multivax.de> wrote:
> >>>>
> >>>> Certainly in the top-ten list of most useful VMS features.  Off the top
> >>>> of my head, others would be:
> >>>> HBVS,
> >>>
> >>> Yes
> >>
> >> Sort of.  For specific configurations and requirements, definitely.
> >> For many other common configurations, no.   Outside of multiple paths
> >> across multiple hosts, hardware RAID solved what HBVS provides long
> >> ago.  HBVS is simple and synchronous and inefficient approach toward
> >> replication.   For single-path access, file shares and hardware RAID
> >> all work fine, and various of those configurations can be replicated.
> >> Which reduces the cases and the configurations that really want and
> >> need HBVS.  Other platforms replicate data differently.  And some apps
> >> want or need asynch replication.  Or want to avoid the overhead of
> >> whole-disk replication.  Or the overhead of synchronous write-thru I/O,
> >>   whether due to geographic latencies or otherwise.
> >>
> >>>> clustering,
> >>>
> >>> Yes (big time)
> >>
> >> There's a whole lot of work necessary here.
> >>
> >> On OpenVMS, I'd suspect that the pricing killed clustering.
> >>
> >> An integrated DLM is handy, but clustering and its APIs and management
> >> and integration is far too firmly stuck in the 1990s.  No job
> >> management.  No LDAP integration past authentication.
> >>
> >> On other platforms, there are add-on packages that provide distributed
> >> coordination, file shares, replication, backups and other features.
> >> Which reduces the numbers of combinations where clustering really still
> >> shines.
> >>
> >>>> logical names,
> >>>
> >>> Yes
> >>
> >> Logical names?  Definitely not.  Logical names are a disaster, in most
> >> of the ways that they're commonly used.   There are better ways to do
> >> most or all of what logical names are used for, too.  Just not as
> >> easily on OpenVMS, unfortunately.   They're complete dreck when used
> >> for app settings, flags and related configuration data.
> >>
> >> I used to think that logical names were pretty handy.  Then I started
> >> working with property lists (YAML or otherwise) and with settings that
> >> were integrated into the image activator, and with file links.  Vastly
> >> preferable, far more capable, and lacking the out-of-band and
> >> collision-avoidance-by-convention issues of logical names.   All of
> >> which OpenVMS lacks.
> >>
> >>>> decent HELP and other documentation,
> >>>
> >>> The HELP command on VMS is utterly useless unless you already know
> >>> exactly what you are looking for and don't need to navigate between
> >>> topics.
> >>
> >> The HELP tool is utterly archaic on OpenVMS, and the documentation
> >> lacks navigation, recipes, example code, IDE integration and related
> >> APIs, and the ability to be kept cached and updated.  More than a
> >> little of what's documented is stale, too.
> >>
> >> For a third-party tool for another platform...
> >> https://kapeli.com/dash
> >> https://itunes.apple.com/us/app/dash-offline-api-docs/id1239167694?mt=8
> >>
> >> IDEs are up in this same range for documentation access, example code
> >> and related materials, too.
> >>
> >>>> fine-grained privileges,
> >>>
> >>> They are ok, but it's still DAC security instead of MAC security.
> >>>
> >>> They would also be more effective if VMS was a 4-mode operating system
> >>> instead of just being a 2-mode operating system with one of the modes
> >>> split across 3 hardware modes.
> >>
> >> I don't see particular benefits of the forest of privileges.   Other
> >> than the morass of the NOPRIV error.   Few (no?) folks are managing
> >> servers that way, either.   We're past the era of timesharing and
> >> semi-privileged users.  Privileges don't work even remotely well for
> >> installed images or for app isolation, either.  Privileges aren't
> >> particularly tied to the access modes, beyond the memory management
> >> controls being the foundation for all security.
> >>
> >>>> EDT,
> >>>
> >>> You are kidding, right ?
> >>
> >> For most folks, pico or nano would be preferable to EDT.   vim or emacs
> >> or a WYSIWYG editor for others.  Or an IDE.   DEC deprecated EDT a
> >> quarter-century ago.
> >>
> >>>> lexical functions,
> >>>
> >>> Lexical functions are a good idea but limited. It would be nice if it
> >>> were possible for a site to create a library of site specific lexicals
> >>> that could be used in site specific code.
> >>>
> >>> Maybe after VSI fixes everything else that is wrong with DCL.
> >>
> >> Lexical functions are what you get when your command output can't be
> >> processed directly as text, nor as objects.
> >>
> >>>> Rdb (almost a part of VMS),
> >>>
> >>> It would have been nice to have a proper database instead of the RMS
> >>> implemention of indexed files (which have _not_ aged well) as a
> >>> guaranteed part of VMS and which _any_ application could use.
> >>
> >> SQLite minimally, preferably also with PostgreSQL and MariaDB around.
> >> The Oracle databases available for purchase, for those folks that want
> >> or need those.
> >>
> >>>> DECnet (almost a part of VMS).
> >>>
> >>> Maybe in the 1980s. Totally unsuitable for any modern network.
> >>
> >> DECnet is dead.   For various reasons.  Not the least of which is
> >> security.  VSI will probably be integrating IP into OpenVMS akin to
> >> what's been done on other platforms, but much of that work will happen
> >> some years into the future.
> >>
> >>
> >>
> >>
> >> -- 
> >> Pure Personal Opinion | HoffmanLabs LLC
> > 
> > Setting aside the active discussions on bundled relational databases ...
> > 
> > VMS clustering has a feature that I'd argue is still state of the art - its handling of state during rolling upgrade.  The actual state handling is mostly an internal detail, and mostly uses schema patterns that are now well known (existing field definitions are invariant, expand and then shrink across versions).  And VMS has a very clear idea of a basement version, to support schema and data cleanup.  However, it is the fact that it is not purely primary-secondary at the node level that really helps - it means that rolling upgrades and downgrades happen on a millisecond by millisecond basis, which make them uniquely testable (again, in my experience).  I'm certain that helped the upgrades be as stable as they were, at least in the early days.
> > 
> > Beyond that, comparisons for clustering, in whole and in parts, suffers from the design trends that disagree with where the functionality best belongs..  Cluster features at the OS level fit when the applications, their interactions, and their nodes all matched with high fidelity.  If they don't, then splitting features, like DLM, out from the OS makes sense.  As time has gone on we've seen more and more of that happen - external coordination services rather than an OS service, external replication services rather than OS volume shadowing, TM services rather than an integral OS service.  And, freed of the node tie in, which forced support for a mixed pool of applications, there's been a tendency towards simplicity, both of those services and the application structures.
> > 
> > Weirdly, containerization may start to restore the application<->node pairing, but I don't think we'll see much deep integration again, unless there is some truly amazing benefit.
> > 
> > 
> > I'd also say that there's _a_ feature of logical names that I miss to this day - the construction of searchlists.  I agree that a lot of the other uses are better served with more recent application configuration models.  But the ability for me to decide to take a directory tree and preface it with a writable level for exactly my use at this moment is a great feature.
> > 
> > Fwiw,
> > Jim.
> > 
> 
> 
> It's a cluster Jim but not as we no it!
> 
> (Was it you that used to say "Whatever the question, the answer is eye"?)
> 
> VMS clusters won't scale! VMS clusters are geographically bound between 
> you and the corner shop! VMS clusters are dead because option B has won!
> 
> Oracle RAC and Cache fusion. Whatever that REDO, REDUX, ECHO cache 
> updating thing is works! Wake up!

Hmm.  Let me try again, as I thought I'd said some very positive things.

I'm one of the people who fits the model of remembering the old system fondly.  I do.  I can't help it.  The team I joined in the early/mid 80's was stellar, and the work we were doing was cutting edge.  That company, that team, and that experience shaped me in ways that no other part of my career has done.

There are two aspects to that environment that I want to call out here.  The first is the extent to which the comparative was current research - e.g. the DLM compares to Gray (Notes on Database Systems, IIRC), and DECdtm compares to the current 2PC research of the time (pre Gray & Reuter).  Some of it went beyond that - e.g. state handling during upgrades was something that to this day I cannot find published references that date back to that time.  The second was the unceasing drive for quality, which inevitably demands a critical eye to what was done.  You will not get quality by being easy on yourself.

Finally, the Compaqtion also trained me to be careful of letting emotion override facts.  No matter what I felt of pre and post Compaqtion, it was very clear that facts mattered more.

Now, from what I can tell from the online documentation and here, the reality is that the system indeed has not moved appreciably since the late 90's.  I'll again point out that every cycle that VSI is spending on getting the system onto x64 is exactly and precisely the right thing to do.  I have only support for what the engineering folks are doing there.  I think that Hoff has been stating very clearly what the only likely viable plan is that VSI can have.  I've not seen such a statement from VSI, and, well, I don't need to.

For most systems that I can think of, if you'd asked 20+ years on from the end of active development what they had that was still good the answer would be at or near a null set.  If you asked what it had that was still state of the art, it would either be a null set or solutions to some problems that literally no one cared about.

I do not believe that is the state of VMS, and it is quite a statement for what was built.  However, if you use a critical eye and do not let emotion cloud the facts, the list is smaller than you might want.

My comments on clusters highlighted two things.  First, there is a critical problem - state handling during a rolling upgrade - that VMS has a track record of handling extremely well.  And if I dig into how that happened, VMS has a uniquely testable form of a solution (AFAICT).  This is not a problem no one cares about.  Handling state upgrade is one of the known stressful points in running a service.  What I find personally fascinating is that this feature depends on not having taken some of the simplifications that came from decomposition into specialized services.

Digging further, the 'schema management' part of the mechanism is now known.  It seems to have been (presumably re-)discovered and documented during the 90's.  It uses a simple rule set that says all existing field definitions are invariant, that you can only add new fields, that there is a protocol for aging out use of obsoleted fields, and their eventual removal.

But, that is silent about how you decide it is obsolete (VMS clusters have basement versions that cover that), and it is silent about how you test upgrades and downgrades. VMS, because it does not follow a primary-secondary pattern, in essence requires that both the upgrade path (new version gets data from the old version) and the downgrade path (old version gets data from the new version) are in continuous use during an upgrade itself.  Simply stopping an upgrade partway is a good start for a test.  Is it perfect?  No.  Does it cover all parts of the system?  Not always.  But it is both simple and surprisingly good.

Now, the second part of my cluster comments reflected that there's always a debate about where technology belongs.  There are cycles and trends, and it can take some technologies a long time to settle.  And a lot of what was embedded in VMS clusters represents technology that the trends have pushed out of the core OS.  You can still compare them, but you have to be clear about why they were pushed out and account for that. And yes, the trends may reverse someday.

VMS clusters bundle enough things together - from membership, security, system management, coordination, health and lifecycle management - that it is great if the applications match the nodes, if the node population doesn't radically change very often, and if it is ok that the release times of the various features are synchronized.  For a lot of applications the trend has been away from these assumptions, so for that class of users the feature that all these are bundled with the OS becomes a negative.

OTOH there is a feature I do get with the bundling, but it is something that is hard to explain simply to people looking at the assembly of special purpose services approach.  That feature is that membership and health management permeate enough resource access such that it is not possible to modify a resource (e.g. a file) and not be a healthy member of the cluster when doing so.  And there are several services (not all, by a long shot) that cannot fail and you retain membership.  The failure of this core set uses a single channel, rather than two.

Conversely, every approach built from assembling specific services has to deal with two-channel failure pattern for any combination of services.  Now solutions are known that require application work.  Furthermore, the single-channel model in VMS clusters is/was not exhaustive, so you can't always avoid the two-channel problem on VMS.  At best it offsets, sometimes, the negatives from the bundling.

Note that what I'm trying to get across here is that an analysis like this is complicated, and you're not going to get an 'a is always better than b' answer.  And that is actually a strong statement about the work that was done - that after so much time which is better is still complicated.  And, if you truly want the VMS answer to always be better, then, as has been said elsewhere, there is a lot of work to be done.

Fwiw,
Jim.

(Please note that I've not talked about security here, or any number of other features that are also bundled that I doubt seriously are still state of the practice, never mind state of the art)