[Info-vax] Unix tools on VMS, was: Re: September 6, 2016 - new Roadmap and State of the Port updates now on VSI website

[Stephen Hoffman]

On 2016-09-11 15:47:15 +0000, David Froble said:

> Stephen Hoffman wrote:
>> On 2016-09-11 12:18:10 +0000, David Froble said:
>> 
>>> Simon Clubley wrote:
>>>> 
>>>> Really simple example: GNU diff is vastly superior to the VMS version.
>>> 
>>> Never having used it, I'll have to take your word for this ....
>> 
>> GNU and BSD diff tools are easily better, yes.
>> 
>> bash has some severe ugly, it's cryptic, and it can be wildly 
>> inconsistent, but it and the capabilities of the Unix command tools are 
>> generally superior to what are available on OpenVMS via DCL and the DCL 
>> utilities.
>> 
>> Not that "''DCL'" syntax &is always 'entirely' clear to new users, 
>> "either."   DCL is definitely simpler, it's more typing, but it's 
>> unfortunately also certainly not gotten around to dealing with regular 
>> expressions or pipes or functions at all well, nor limitations around 
>> dealing with (no metadata, untagged, always-assumed ASCII or maybe MCS) 
>> text.
>> 
>> bash has limits here too, but there are bash ports that deal with UTF-8.
> 
> So nothing is perfect.  Differing strengths and weaknesses.

Ayup.     Difference is that I'd rather go for bash than DCL, if I'm 
using a command line.   If I'm going to the effort of using the command 
line, bash and related pieces are better tools.   For the era when most 
end-users used the command line, DCL is (was) a better choice.   Users 
that are new to OpenVMS are less likely to want to learn DCL, too.   
This same dichotomy between the installed base and the biggest 
potential market shows up with the TCP/IP Services command interface; 
where there's the Unix-native configuration and troubleshooting tools, 
and a DCL-ish interface.    The former is far more familiar to most 
networking folks, while the latter... I'm not at all sure that many 
folks even use that DCL interface anymore.

>>>> With GNU diff, as well as having a much more readable unified diff 
>>>> output format, you can also compare differences between two directory 
>>>> trees.
>>> 
>>> Now, what was I writing?  Oh, yes, implement on VMS.  I've got a simple 
>>> tool I call COMPARE that does just that, on VMS, compare differences in 
>>> two directories.  Most likely not the same, and I'll admit the tool 
>>> could be greatly improved.  Works for me.
>> 
>> Yes, we all have those tools.    Which work for us.   Which we all 
>> implement.   
>> http://www.digiater.nl/openvms/freeware/v80/hoffman_examples/diff_directories.com 
>> Which should tell you a little something about the adequacy of the base 
>> operating system for its appointed tasks, too.
> 
> Ok, correct me if I'm wrong, but many of those *ix tools began as 
> someone's individual work.  Where have I seen this before?  Perhaps 
> DECUS?
> 
> Now, why is such on *ix good, and the same thing on VMS bad?  This is 
> part of the lack of understanding I have.  I tend to see a double 
> standard.
> 
> Or perhaps it might be the disinclination of the OS owner to adopt such 
> independently developed tools and subsequently support them?

OpenVMS traditionally hasn't wanted to add and support new bits, and 
deliberately chose not to incorporate tools such as zip and unzip.   
How VSI goes with this?

> 
>>>> Second example: bash offers all kinds of UI features not present in DCL 
>>>> or which have been implemented in a clumsy manner in DCL (such as 
>>>> searching for previous commands).
>>> 
>>> Ok, taking your word for this also.
>>> 
>>> But, how does any of what you've written justify blaming VMS for not 
>>> being able to just compile, link, and run the *ix tools?  That's my 
>>> issue.  And my question.
>>> 
>>> If you want the tools on VMS, then feel free to implement them there.  
>>> That's how they got on *ix, right?  Somebody did some work.
>> 
>> With OpenVMS, various C APIs either don't work (e.g. SSIO), or are 
>> missing or down-revision (q.v. JR's "whiteboard"), or are less than 
>> integrated (q.v. TLS, PKE).
> 
> And so, not being familiar with *ix, you lead me to believe that in 
> that environment it's not the OS that provides services, it's the C 
> language?

C is the native environment on Unix, and a fair chunk of what is in the 
C standard library is there because it's the system interface to the 
underlying Unix system.   So a C implementation on other systems 
necessarily implements or emulates a subset of the Unix system 
interface.   There's also a far larger set of standard libraries than 
exist in OpenVMS, too.

> If so, well, that just isn't, or at least should not be, the VMS way. 
> Capabilities should be available to all languages on VMS.  Ok, now I 
> guess we are getting to my concerns.

SSIO is not something that BASIC has any concept of.   There are more 
than a few areas that are lacking integration with OpenVMS, including 
encryption, authentication and TLS.  That stuff did make it into the 
distros, but not much uses it (yet), and some of the integration of 
these and other add-ons is lacking.

> Personally, I'd think byte range locking, or whatever you want to call 
> it, could be a useful tool.  If implemented on VMS, then it should not 
> be limited to only one language, but available as an OS service.  Gets 
> me remembering HP's statement that OpenSSL can only be used from C.

Apple did this rather better, with their Secure Transport interface.    
OpenSSL would need jackets to be easily callable from languages other 
than C, Macro32 or Bliss; akin to what we've posed.

The longer-term question is whether the existing APIs and design is 
maintained, or if VSI decides to go a different route akin to what 
Microsoft .NET or NeXTSTEP (what's underneath OS X / macOS) or other 
such have used.

>> So we all go off and implement the pieces and parts, or hack around 
>> them.    We then all have different ports or different versions, and 
>> with different issues and vulnerabilities, different maintenance 
>> scheduled and updates.   We end up with bugs and misfeatures and such, 
>> too.    With the associated costs.
> 
> And that doesn't also happen on *ix?

It most certainly does.   There's far more in the common libraries, and 
that's (usually) more current.   The base configuration is vastly more 
capable.   OpenVMS used to have most everything (SMG, DTK, STR, LIB, 
OTS, language RTLs), and has used add-ons in the past couple of decade 
to provide additional features and tools.   That add-on model also 
exists on many other systems, though more of what's optional or add-on 
in OpenVMS — libraries, services, tools — is increasingly integrated 
into the base distros with other systems.   With macOS or Linux, folks 
are not adding the various common-but-missing C calls, nor messing 
around with an add-on and effectively unintegrated Apache web server, 
nor with getting LDAP working, nor adding libxml2 or Python or any 
number of other tools.  Are those tools sometimes stale, on some 
distros?  Sure.  Apple has had (and still has) issues there with macOS, 
as do other providers and other distros.

The results of the piecemeal port-it or write-it yourself is having to 
waste everybody's time over and over again, porting or adding or 
integrating.  Or upgrading.   That is centrally because I'm working 
across platforms and sites and projects, so I see a lot of these cases, 
and I see how much this adds up both in terms of wasted programming 
effort for end-users, and latent security problems due to stale code, 
simple ignorance about what is available — and ignorance around why 
folks might or will want to or have ported from OpenVMS to Linux — and 
various other factors.

Are there ways around this?   Sure.

OpenVMS is a box of parts, and I have to spend more than a little time 
assembling everything, and shaking out the inevitable weirdnesses or 
bugs.    As an example of this, why the hell does TCP/IP Services not 
just add all its users, all its directories, and the template files, 
and just control whether the service starts or not?   Things get weird 
when the service-related directories or files or the rest are not in 
the expected state.   This extends to services that are unintegrated 
add-ons on OpenVMS — the Apache web server is a salient example, as is 
the LDAP server — that should be in the base distro.   Then there's the 
inexplicable complexity around configuring the IP address and related, 
which should be a fundamental part of the base configuration and not 
some adjunct — the number of OpenVMS servers that don't use IPv4 or 
IPv6 or both is minuscule — so why is this configuration and management 
mess even remotely as complex and arcane as it is?

Why am I discussing Apache and LDAP in a topic on C and missing APIs?  
Because for more than a few applications, Apache or LDAP or these other 
services ARE part of the base APIs; the dependencies.  Then there's the 
reams of conditional code and testing that's needed to detect when some 
API or some service isn't present or isn't current or isn't secure...  
A missing or down-revision dependency.   This also ties into my various 
comments around the patch and update processing, as well as the crash 
handling.   Because if you're going to maintain APIs and add-ons, the 
existing copy-files-around model used to maintain OpenVMS is utterly 
insane; there's no analog to any of the packaging tools, or the Sparkle 
framework https://sparkle-project.org , or otherwise.   All of which is 
common on other platforms.   (Noticing a pattern here?  VSI has more 
than a little work ahead, whether they engage the community with some 
of that, and whether there's enough interest or capabilities in the 
community?)

> 
>> This isn't just C, either.   The same general problem holds for 
>> application code written to newer other and newer compilers, too.    
>> BASIC needs more than a little work to update it and overhaul it, and 
>> to haul it past the 1980s.
> 
> It's not going to get that, if it is denied access to new capabilities.
> 
>> Or to replace it with a different language that works as well or 
>> better.    We have the limits of the older non-OO or non-RT APIs, too.  
>>   But that's all fodder for another time.

Longer term, I'd like to see the OpenVMS APIs go OO, but haven't 
pondered how that would work in BASIC.    Secure Transport uses the OO 
interfaces effectively, though that will look rather different from 
itemlists and descriptors.    PKE and TLS are not integrated and the 
certificate handling is just hilarious, though some related work seems 
to be planned or in progress at VSI.


-- 
Pure Personal Opinion | HoffmanLabs LLC