[Info-vax] Beyond Open Source

[Stephen Hoffman]

On 2015-05-10 04:32:03 +0000, IanD said:

Nice> to> see> that> the> Google> Groups> interface> is> still> 
screwing> up> quoting>.>>

> On Friday, May 8, 2015 at 9:37:12 PM UTC+10, Stephen Hoffman wrote:
>> On 2015-05-08 05:38:37 +0000, IanD said:
>> 
>>> Opensource gains favour when one rapid development of a new idea but as 
>>>  the article alludes to, becomes more and more difficult as a project  
>>> grows and code contributions come in from an increasing number of  
>>> sources
>> 
>> It also becomes more difficult as the scale of the problems increases. 
>> Even for the side projects, I'm working on vastly bigger problems than 
>> I once was.   Better tools and better frameworks help somewhat here -- 
>> this is part of why I whinge so often about tools and frameworks, and 
>> why working on other platforms can produce results more quickly -- and 
>> part of the increases in scale of the work just involves working with 
>> ever-larger groups of people (or taking longer and longer to get 
>> anywhere particularly useful).
>> 
> 
> Frameworks are akin to a standard, yes, they allow faster development. 
> For example, who, on large projects develops code outside of a 
> framework now?

On OpenVMS?  It's unfortunately still pretty common.  Various of the 
existing OpenVMS sites have their own frameworks.

> Even things as simple as a standard log format makes OpenVMS painful. 
> No central registry of events, no common log file format from which to 
> easily mine information from, very painful

Then there's puttering around with the emulation of the card reader 
interface that's common with RMS.  That design and that approach 
comparatively slow, and makes for baroque logic in the source code.  It 
can scale, but then when I'm at scale, I'm increasingly going directly 
to the data store — maybe RMS for some cases yes, but increasingly 
that's SQL or some other specialized not-record-oriented database — and 
am not looking to use record-oriented operations.

> I'm often getting requests for statistics like 'How many of xyz 
> occurred over the past 3 months' in this application. To mine that 
> simple information is painful. Log files dumping data in blurb fashion 
> with little or no structure have to have scripts specifically written 
> to deal with that particular log file - drives me crazy. Linux folk 
> whip up quick scripts and often have the data ready in less time. Maybe 
> my DCL sucks but I'm not that bad at it!
> When 500 linux systems can spit out consolidated statistics versus 1 
> VMS system that seems to require a different script for each log file, 
> it makes VMS look pretty shabby by comparison

Ayup.  For cases such as this, OpenVMS doesn't scale.   But then I've 
become quite fond of rolling up events across multiple servers.

> Data events are the lifeblood for compliance checking and in this 
> regards VMS is behind IMO

Anomaly detection, as well.

> I'm going through a process mining course at the moment and that field 
> is growing in importance over just data mining, process mining on VMS 
> is painful because the different tools store different events in 
> different formats making it a lot more difficult to roll up events 
> across the system to get an end to end view

Wait 'til you try rolling up across a cluster, or across a bevy of 
standalone servers.  There's just no good way to do that with OpenVMS, 
short of acquiring or implementing your own collectors and logging.  HP 
has been selling products for larger-scale management of events, 
errors, patches, distributed management and the rest, as a value-add 
for folks that buy HP hardware.  Where VSI goes with this, we shall see.

> I'd love to see an event logger like windows has on OpenVMS :-)

Microsoft very much understands the enterprise.

As for the event logger, it's usually better to aim (much) higher than 
an existing design, as providing what another platform already does is 
a whole lot of catch-up work and for not very much of a competitive 
advantage.

> 
>>> I mentioned open source for VMS especially if a new stack was going to  
>>> be developed but the jury is out on that one
>> 
>> A new stack?  Are we referring to OpenVMS or to IP or to something 
>> else?   Bootstrapping takes vendor and community commitment, and a 
>> bunch of people, a whole lot of time, and -- again -- better tools and 
>> frameworks.   The developer and user expectations are also moving 
>> forward rather more quickly, particularly in comparison to the pace of 
>> OpenVMS in recent years.
>> 
> 
> A 'new' stack was mentioned as per the roadmap

It was simply not clear (to me) which stack you were referring to, as 
there are various "stacks" in complex software environments.

> 'New' to me means NEW, not rehashed, not reworked, not already existing 
> and given a new face, but NEW

Alas, I do not know what is new to you, what is rehashed, nor what is reworked.

Given the timeframe and the staffing involved, there's simply no way to 
ship a wholly new IP stack in a year or two.  If even within three or 
four years and more than a little field test.  Not without a huge 
investment and a whole lot of risk of incompatibilities on deployment.

> In the HP/VSI OpenVMS breakfast a question about the new stack was 
> asked and someone asked if it was Progress software under a license. No 
> specific answer was given, in fact, the answer that was given was that 
> details could not be disclosed

So?  If it wasn't the Process IP stack, there'd be denials from somebody.

>>> Lots of examples of open source being used to fork projects when people 
>>>  see a different end need for the software in question - I don't think  
>>> VMS has that luxury at this stage, the base is too small at this stage
> 
>> Are you thinking of forking OpenVMS itself?  If so, I'm skeptical that 
>> a non-commercial or non-commercially-funded entity would get anywhere 
>> with such a project, beyond basic maintenance.
>> 
>> There just aren't enough folks with the free time necessary and 
>> particularly the interest necessary to pull something with ~30 million 
>> lines of code forward (fast enough to matter) to bring more folks onto 
>> the platform.  Even within the OpenVMS community, there are and will be 
>> schisms; some see OpenVMS as a desktop platform and some as an embedded 
>> server platform.  Some look askance at newer tools and approaches, and 
>> others want or need those.  Some folks are absolutely dependent on 
>> formal vendor support and/or Oracle databases or other features, and 
>> others not so much.
> 
> In time who knows what's possible in the future

What's financially viable is far more interesting.

> IF IF IF is the pre wording I would use here
> 
> Linux builds are a dime a dozen which allows people to create 
> distributions that target a specific situation. Scientific linux, 
> pclinux, centos, puppy, kali, tails etc are all specific versions 
> targeting specific needs. One size does not fit all
> 
> That's the advantage linux has in terms of being able to be rolled to 
> meet a specific need when one arises. The one's I listed above 
> specifically exist to target a particular need and to excel in that need
> 
> VMS in contrast is mostly of the old school, trying to be monolithic, 
> an all things to all type of approach

The term "monolithic" has different connotations when referencing an 
operating system, FWIW.  OpenVMS has a monolithic kernel with loadable 
components, as does Linux.  Some other systems do not have that design. 
 Whether a single monolithic kernel is the best approach for future 
hardware?  For some systems, it will be.  For other non-cache-coherent, 
non-big-box server designs, differing kernel designs may well used.

As for your (intended) reference to a single distribution and to 
tailored and customized distributions, that can save hassles or can 
increase hassles, as it reduces or increases the number of permutations 
that vendors have to build for.   With Linux, it's why more than a few 
folks target RHEL or other specific distributions, and not "everything".

This is also the problem that Microsoft aimed .NET at; at reducing the 
number of platform permutations that they and their associated vendors 
had to deal with, when shipping products for Windows.

Akin to the Microsoft approach, OS X ships as back-built application 
"bundles" with embedded frameworks.

OpenVMS has no such packaging mechanisms, though it is (informally, 
unofficially, undocumented) possible to back-build applications on 
OpenVMS.

None of these cases address differing architectures, however.  That's 
still either messy, or involves something like Java, Lua or Python.

As for having many different things for many folks and different 
targets, Linux has vastly different staffing and support models than 
does OpenVMS, too.  With OpenVMS, there's some expectation the 
configurations will be tested and supported, and will work.   With 
Linux, different configurations may or may not work, and you might well 
be the first person to ever test a particular combination.   If you've 
got the time and the skills and the source code 
<http://www.pagerduty.com/blog/the-discovery-of-apache-zookeepers-poison-packet/>, 
you can drill down through the stack and solve the problem.  Maybe.  If 
you don't hit a vendor blob, either in Linux or in the particular 
hardware involved.

With OpenVMS and a supported hardware configuration, you call VSI and 
they get to deal with the problem.  Same usually happens with the 
vendor-supported versions of Linux, of course.

In short, software support is a formal agreement providing you with 
somebody else to blame.

> The point of open source to me is to get as many people interested in a 
> project as possible (as mentioned before, some programmers will not 
> touch closed source systems due to philosophical beliefs, so why limit 
> the number of potential people possibly working on a system).

Much like text editors, there'll be debates over software licenses.   
As getting folks interested in a project, the license is a factor, but 
the tool has to be or have a path to being useful.  Outside of its 
installed applications, there's very little reason for folks to select 
and use OpenVMS right now.   Open source will not change that.  Only a 
whole lot of work by VSI, and substantial work from the commercial 
providers in the community, might change that.  Might.

Open-sourcing does not magically produce piles of skilled programmers 
ready and willing and able to work on the platform.

> I believe even MS are going to open up .NET source code going forward, 
> they see the need to embrace certain philosophies around coding.

Microsoft knows how to compete, and they've historically been quite 
good at it.   Here, they're using open source as a cudgel.  Go read 
that open-source link I posted a while back.  As for your use of terms, 
"embrace" can have some specific connotations around Microsoft, and 
whether you intended that is not clear (to me).  "Embrace, Extend, 
Extinguish" was a longstanding shorthand for the Microsoft business 
strategy.  There are folks that wonder whether this release of .NET is 
another iteration of that, beyond the obvious beneficial nature of that 
in support of better Azure integration for current and potential Azure 
users.

> From a business support aspect, closed source can be dangerous 
> especially when the creator/owner decides to do something different in 
> life or the company closes shop! At the breakfast, there was someone in 
> anguish over the recent announcement by IBM to close down one of it's 
> messaging systems on OpenVMS - leaving this persons business vulnerable 
> going forward. Closed source is fine if you have a way of obtaining the 
> source or a pathway out should the software owner get hit by a bus but 
> it's risky for a business now and it shows up on business contingency 
> plans now as a risk

WebSphere MQ, based on discussions here.  Formerly MQ Series.   There 
are open source options and alternatives, alas porting is No Fun.

Open source isn't a panacea, as now you have to figure out how to 
maintain the code yourself, and you can end up owning the whole problem 
the package was intended to resolve.   Which means you can end up 
needing to port to some other package or tool.   IBM and HP tend to 
provide roadmaps of their enterprise products, so most companies have 
five or maybe ten years migrate.  For other vendors, watching their 
release cycles and investments can provide some indication of whether 
they're making money with the package and thus it is likely to continue 
to be available.  In short, strap the blinders on at your own risk.

As for a problem differing from your product cancellation or product 
retirement case, it's also possible for a vendor of a critical 
dependency to decide to raise their prices to expensive or even 
extractive levels.   Accordingly, there are organizations that work 
diligently to control their supply chains.  Other organizations might 
not have that foresight, or might not have the capabilities or the 
budget for that.

> I look after a system that see's certain folk trying to get rid of it 
> continually. The source code is with a company that has been sold to 
> another and they don't want to support the application anymore and the 
> people who did develop it years ago have mostly moved on now - it's a 
> huge business risk now. The eye is on transferring the functionality to 
> some Oracle system and turning off the VMS box altogether. If the code 
> was open sourced at least, we would have picked it up and reduced our 
> risk exposure but now were stuffed in terms of staying on OpenVMS :-(

If you can't license or can't acquire the package or get them to 
open-source it, then porting off of OpenVMS is probably the right 
thing, then.  There are alternative platforms.

> 
>>> As long as there are enough core developers to keep enhancements and 
>>> bug fixes coming in a reasonable time frame
>> 
>> Out of curiosity, how many folks do you think is "enough"?  VSI is 
>> aimed at a ~hundred full-time paid folks, and only a fraction of those 
>> will likely be working on OpenVMS itself (some will be working on 
>> infrastructure, customer support, and in administration and management, 
>> etc), and they probably really want more folks than the ~hundred.   It 
>> wouldn't surprise me to be able to utilize four or five hundred 
>> full-time paid folks without too much effort, for the kernel and for 
>> all those parts and pieces and layered products that are now typically 
>> part of the kernel, and for layered products and tools.  Again, the 
>> scale of the problems has increased, as have the expectations, as has 
>> the surprisingly difficult problem of increased simplicity.  Xcode is 
>> rather larger than LSEDIT, the newer language standards tend to be 
>> larger and more complex, and the typical web-facing tools and libraries 
>> and frameworks are rather larger than libwww and Mozilla Seahorse and 
>> Apache HTTPD, for instance.  Put another way, an operating system 
>> project is the proverbial bottomless pit of software developers -- up 
>> until "tactical incompetence" becomes "strategic incompetence", you can 
>> want and need and use as many developers as you have access to.
>> 
> 
> Number of programmers depends on different factors
> 
> I seem to remember a MS study that put the number at 20 as optimal. 
> Scrum I think suggests 7 +- 2 (so between 5 and 9).
> Above 20 and error rates start to creep in
> 
> I don't really know, I've worked in teams with around 15 programmers 
> working on a single application but on different subsections of it and 
> that number seems to allow for contingency (illness, holidays etc) as 
> well and increasing/decreasing workloads - I really don't know to be 
> honest

You really need to work through and think about and learn about these 
areas, such as what might differ with active projects with ~30 million 
lines of code, or more.   That's the scale VSI is at.  If not larger, 
as they have most or all of the major layered products, too.   Even 
outside of an operating system, several hundred programmers isn't a 
particularly large effort these days, either.   For operating systems?  
 Linux has a whole lot more folks involved than that.  Within a larger 
project, yes, you will have smaller teams and folks specializing in 
specific components.

> Release cycles were monthly. I guess most folk would be happy with 
> OpenVMS functionally being released on say utilities on a 3 month cycle 
> and core OpenVMS on say a 9 month/yearly update cycle?

Some parts of the OpenVMS crowd would not want continuous releases or 
even a nine-month cycle.   In the old days, eighteen months to two 
years was pretty fast for some folks.  Unfortunately, I don't believe 
it's really possible to successfully do an eighteen month big-upgrade 
release cycle any more — waiting a year to push out some changes is 
just stupid, and pushing out other changes every eighteen months will 
probably be too disruptive for some sites.  Unfortunately, there's no 
right answer here.  Which is where you get the long-term support LTS 
release model from, and rather more continuous updates for other folks.

> An OS over time seems to be going towards being a giant framework 
> provider under which different entities will be able to exist and run 
> in their own world as they see fit

In years past, this was why operating systems were sometimes known as 
"a bag of drivers".

> Docker for example will almost decouple the underlining OS from the application

Docker is increasingly popular, but then Microsoft is aimed right at 
that market with their Azure Stack (recall that there were other 
software "stacks" around?) 
<http://arstechnica.com/information-technology/2015/05/your-own-personal-azure-microsofts-new-azure-stack-for-private-clouds/>. 
  Also OpenStack (another "stack"), though hopefully that's a little 
easier to deal with via HP Helion.

> Will an OS eventually fulfill the enterprise directory paradigm I wonder?

No software will ever fulfill everything, and it'll all suck.  Some of 
it just sucks less than others.   But IT will somehow get most of it to 
work.




-- 
Pure Personal Opinion | HoffmanLabs LLC