[Info-vax] OpenVMS servers and clusters as a cloud service

[DaveFroble]

IanD wrote:
> Den 2018-01-08 kl. 09:25, skrev IanD: 
> 
> This is just silly... 
> 
>> What good is it if I run a batch job for 36 hours on VMS if the node 
>> crashes and I have to start again. I've wasted 36 hours! 
> 
>> Than you have an very badly desiged batch job. Either split it up into 
> multiple jobs or make sure in some other way that the job is restartable. 
> Such a job should of course be able to pick up where it crashed. 
> 
>> Anyway, I think that is a very big "IF" there. Show me a real life 
> example of one such job instead... 
> 
> 
> Have you worked with large data sets when crunching things like genomic data? The data sets start in the TB range and go up from there
> 
> Have a look at the machine learning competitions on the likes of kaggle and some of the cut down data sets and you'll get an idea of some of the large data volumes in use out there
> 
> Why do you think The likes of Hadoop exists?
> 
> It's to handle such volumes without the need to specifically code a dedicated workload job and handle all the complexity of machine failure
> 
> It's not just about job restartability. It's about parallel processing with redundancy natively without the need to specifically code it to the individual workload
> 
> Hadoop has within it's abilities to do more than what you could do without writing a shit load of handler code in VMS to do distributed workload management
> 
> Try writing a distributed data analyser to crunch through 5 PB of data and collate it all and to cope with machine outages/failures of an unknown type along the way on VMS and see how long it takes
> 
> Try encoding 100,000 videos to transform then from one format to another and keep track of the conversations in real time and again handle any failure along the way and see how much wrapper code it will require on the VMS platform. Google does this type of task quite a lot. So does Netflix etc. Training providers also do mass video conversation
> 
> None of these can be done within reasonable timeframes on any VMS cluster natively while taking into account any type of failure along the way
> 
> You'd effectively be trying to rewrite major chunks of what Hadoop already does and has developed over many years now
> 
> Hadoop is the modern cluster as far as data crunching is concerned. It handles natively and in real time machine failure without operator intervention. Job failure handling is built into Hadoop
> 
> Watch the linked video for at least the first few mins and then tell me how long it would take to develop the equivalent on a VMS cluster to handle everything Hadoop can already do today and to scale up.
> 
> The video makes reference to Hadoop scaling in near linear fashion up to 4500 nodes. I think that limit has been raised since
> 
> https://youtu.be/OoEpfb6yga8
> 
> You will of course have noted that Hadoop also automatically handles data redistribution when a worker thread dies or failures to respond in a given timeframe
> 
> The only thing 'silly' imo is trying to say VMS clusters can handle such large batch compute jobs natively. 
> 
> VMS clusters have been superceded in this respect by orders of magnitude. 
> The cluster world has moved on in the last 10-15 years that VMS had been asleep

Well, all very interesting.  I don't know enough to say how such jobs might be 
done on VMS.  I "think" I know enough to guess that such jobs, regardless of how 
  well VMS might handle them, would be a minority of jobs done on VMS, or any 
other system.

Some of my own expertize is in wholesale distribution.  We count widgets and 
money, we process customer orders, and other such mundane tasks.  I'm guessing 
there is a whole lot more mundane commercial activity than there is for a 
Hadoop.  I could be mistaken.

So, maybe VMS isn't so good at everything.  There is still a lot of things it's 
rather good at.

-- 
David Froble                       Tel: 724-529-0450
Dave Froble Enterprises, Inc.      E-Mail: davef at tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA  15486