[Info-vax] Compatibility and 64-bit

[David Froble]

Stephen Hoffman wrote:
> On 2015-03-14 20:52:59 +0000, johnwallace4 at yahoo.co.uk said:
> 
>> You're aware of the differences in the code and the data structures 
>> between a 32bit world and a 64bit world, fair enough. Obviously JR is 
>> too.
>>
>> But the rest of us may still get some enlightenment from JR's and your 
>> explanations.
> 
> This is going to be slightly circuitous...
> 
> Here are some 64-bit discussions and examples 
> <http://labs.hoffmanlabs.com/node/571> 
> <http://labs.hoffmanlabs.com/node/787>, and there's a guide that's 
> subsequently been integrated into other manuals 
> <http://h71000.www7.hp.com/doc/72final/6467/6467pro.html> that 
> specifically covers the topic.  Those and particularly the manual will 
> provide a pretty good introduction.
> 
> In the specific case of descriptors, those data structures are 
> upward-compatible, meaning that the same API can be coded to properly 
> process either the older 32-bit descriptors or the 64-bit descriptors.
> 
> The programmers using BASIC, Fortran, COBOL and other descriptor-based 
> languages — yes, I'm going to use that phrase again — can largely ignore 
> the existence of descriptors used at the APIs and the subroutine calls, 
> particularly within the language environment and — to a degree — to the 
> LIBRTL calls.   The languages generally don't call attention to the 
> details of descriptors, save for cases where the programmer might need 
> to explicitly specify the argument-passing mechanism, of course.
> 
> In these cases, a future-facing approach would have the default compiler 
> behavior switch to 64-bit for these, and force the existing folks to 
> specify 32-bit descriptors or otherwise override the default.
> 
> Where descriptors tend to fall over include C, C++, Macro32 and similar 
> pointer-based languages.  In these cases, programmers — myself included 
> — have had a very nasty habit of accessing the descriptors directly, 
> which means that older APIs do not support the longer descriptors, which 
> is a problem for any arriving 64-bit descriptors.  C never saw much in 
> the way of support for descriptors beyond some macros and constants and 
> data structures, and particularly very little in the way of abstractions 
> and tools, and not all that much C code uses the lib$analyze_sdesc call, 
> and not all that much C code uses dynamic descriptors 
> <http://labs.hoffmanlabs.com/node/273>, for instance.
> 
> Now descriptors aren't all that central to updating the C compiler to 
> deal better with 64-bit native code, such as that MAYLOSEDATA3 case.  
> Though it would be nice to have better abstractions for descriptors and 
> such, and to have a path and a plan that would allow VSI and OpenVMS and 
> the partners and customers that are actively changing and migrating 
> their applications to get out of at least some of the current messes.  
> Migrate to these macros and these APIs, and your code will survive when 
> we rip out and replace the problem APIs, for instance.  (How long has it 
> been that there's no wildcard sys$getuai, and where usernames are 
> insanely still limited to a character set of up to 12 
> alphanumerics-n-underscores, as Brian Schenkenberg[er], Jean-Fran[ç]oism 
> and folks with a
> "St." or an "O'" in their names, and others have long endured, and 
> that's before considering 반기문 and other folks.  But I digress.)
> 
> As for the sorts of compromises that arise with designs within OpenVMS, 
> the use of P2 and S2 space was both useful and expedient, and that also 
> added some ugly.  Virtual address space is not flat in OpenVMS, there 
> are two 30-bit P0 and P1 process ranges and a 64-bit (technically 61?) 
> P2 process range, and two 30-bit S0 and S1 system ranges and a 64-bit 
> (61?) S2 system range.   This ties back to being able to continue to run 
> 32-bit and 64-bit code, and not forcing changes on the 32-bit code, 
> while forcing gymnastics and changes and hackery into the 64-bit code.  
> This worked around the process and address space limits of VAX, but at 
> the expense of — for instance — not having honking big images, and 
> having to recode applications to move stuff from S0 or S1 space out into 
> S2 space rather than — as happened in some other cases — recompiling the 
> code, and dealing with the cases where — as is now happening with the C 
> code that was triggering the  MAYLOSEDATA3 case — you can't stuff 
> 64-bits into an existing 32-bit data structure or existing 32-bit API, 
> or you might just say hello to my little friend 
> <http://labs.hoffmanlabs.com/node/800>.
> 
> Assuming the size of a long or an int is a longstanding portability 
> issue in C code, too.   Assuming that a long is always a 32-bit value 
> for instance, technically isn't correct C code.  More recent programs 
> would likely use uint32_t or int32_t for this per C99, for instance.  
> <http://en.wikipedia.org/wiki/C_data_types#stdint.h>  (Note the irony of 
> using a sixteen year old C language standard and "recent".)  Even BASIC, 
> Fortran and COBOL can see some of this mess, where the code assumes the 
> size of some hypothetical API context argument is 32-bit, and now the 
> 64-bit API needs to store a 64-bit.  (Hence those pesky sys$mumble64 
> calls, et al.)
> 
> Now for moving forward, I'd tend to look to C to be native 64 bit and 
> preferably with most of C11, and to increasingly relegate parts of the 
> OpenVMS environment and APIs to, well, legacy code, though those should 
> be 64-bit APIs — those APIs are very likely stuck with the sys$mumble64 
> calls for reasons of compatibility, but dropping back to 32-bit calls 
> and norms just doesn't make sense for new C or C++ work, nor for 
> substantial updates for existing work.  Go big, or go home.
> 
> As I've been mentioning for a while, there are always costs to 
> compatibility, and there are always compromises in any operating 
> system.  The team working on the operating system can optimize for 
> short-term or long-term, for compatibility with old code or new, and for 
> keeping APIs around even though they're known to be trouble.  There are 
> many trade-offs here.   For the 64-bit work, there was a short-term and 
> very effective trade-off made to the advantage of the existing source 
> code, but this means that current and future programmers now have to 
> special-case code that might need a whole lot such as migrate to the 
> various sys$mumble64 calls, rather than just having 64-bit addresses and 
> data natively, as is now common on other platforms.  There's a cost to 
> this compatibility, both in terms of what can be changed, and around how 
> much effort might be needed to implement wrappers or jackets that allow 
> the old code to continue to work.  Then there are the cases such as the 
> SYSUAF morass, where code that directly accesses that will break should 
> VSI decide to wade in and start fixing security weaknesses.  Then the 
> question here then becomes, if the team working on the operating system 
> is going to break code, do they break it minimally to get to their 
> immediate goals — a cryptographically more secure hash — or does the 
> team go big, and break more things and — for instance — integrate LDAP 
> and the rest, and move everybody forward much more substantially?
> 
> My concerns are with the longstanding practice of making the new code 
> the default path for adding the compatibility hackery and not adding 
> that into the old path — mortgaging the future for better compatibility 
> with the past — and with the cases where fixes and security updates and 
> application improvements can be deferred because they can or will break 
> source or run-time compatibility.
> 
> That a VAX-11/VMS V1.0 application executable would run unmodified on 
> OpenVMS VAX V7.3 was a testament to the software compatibility 
> engineering.  But is that the right trade-off in this era?   Make VMS 
> into a premier operating system, and not something that's a steaming 
> pile of hackery when trying something modern.  Folks with existing 
> source code assuredly won't like that in the short-term, but they'll 
> like having to do a port to another platform even less, and they'll — if 
> the right stuff is chucked and replaced — come around to the improvements.
> 
> Is any of this going to be fun?  Easy?  Quick?  Source-compatible?  Not 
> a chance.
> 
> But then VSI also has to pay the bills (and I and most others here in 
> the comp.os.vms newsgroup are not going to buy a ton of OpenVMS 
> licenses), and that might just mean piling on the classic source 
> compatibility, and, well, continuing to make a dog's breakfast of any 
> new code.
> 
> 

I find the arguments compelling ....

However, I have to ask, what are the probabilities as to where VMS is going?

If VMS is at the threshold of a new life with many new adopters who will 
use some of the suggested ideas, then by all means make it so they will 
choose VMS.

Now let's look at another possibility.  VMS will NOT get many new 
adopters, and the paying customers for VSI's services will mostly be 
existing users, with existing applications.  In this scenario, would not 
the arguments for compatibility be the better choice?

Yeah, yeah, I know, as Ricky Nelson sang in "Garden Party", "if memories 
are all I sing, I'd rather drive a truck".  Some may feel that way, and 
not be interested in VMS unless they get what they desire.

I have no clue what the future will bring, nor do I have any decent idea 
which path(s) should be chosen.  If there is anything I'd say, it would 
be "whatever's done should not be too hard for either type of user to 
adopt".

As someone who write a bunch of stuff using Basic, but who also plays 
around with data in string descriptors, I can see some nasty surprises 
coming if backward compatibility is abandoned.