[Info-vax] Large mailboxes
Dave Froble
davef at tsoft-inc.com
Sat Nov 28 15:41:26 EST 2020
On 11/28/2020 1:40 PM, Stephen Hoffman wrote:
> On 2020-11-28 17:37:37 +0000, Marc Van Dyck said:
This sounds like fun. Yes, some people have a weird concept of "fun".
The first suggestion I'd make is keep the design/requirements separate,
and completed first. Don't throw in "I can use this" at the early stage.
>> Collecting the data we need is not difficult, either from a DCL loop,
>> or from a program issuing $GETRMI calls.
I'd go for some performance at this step. A decent language, such as Basic.
:-)
>> To display them, we're going
>> to use an in-house developped web service that can store and display any
>> metric that one cares to throw at it.
The display could be real time, historical, or both. Good arguments for
both.
>>The interface to send the data
>> is cURL, which under VMS has a tendency to get stuck when used
>> intensively.
I'd consider sockets. (At one time I'd consider DECnet, but not now.)
TCP/IP on VMS will get better, or, quite possibly there will be no VMS.
So perhaps a fairly safe gamble.
>> So we'll decouple the collector and the sender, and put a
>> large mailbox between the two. If the mailbox is large enough, we can
>> tolerate a stall of the sender for several minutes, and it will catch
>> up with the mailbox contents very quickly once restarted.
>>
>> This kind of data does not need to survive a reboot (there will always
>> be perfdat to look at what happened anyway) and a mailbox is a cheap
>> and easy to use buffer for this kind of purpose....
But if storing the data happens at an early stage, then less might be lost.
> A couple of different approaches...
>
> Have the collector send a UDP multicast, and have the dashboard queue
> requests for and receive and process that data. That entirely decouples
> the sender and receiver. And the use of the multicast inherently
> discards old data.
>
> A queue whether through a mailbox or otherwise is problematic approach
> here both as it preserves previously-sent data, and as it preserves old
> data in preference to new.
I'd agree, VMS mailboxes would not be my choice for this task.
> I'd use DTLS here for sensitive data, though the available TCP/IP,
> UDP/IP, TLS, and DTLS APIs on OpenVMS are less than friendly.
Screw the APIs. There are sockets at the bottom of any of them. Just
forget the top heavy junk and use sockets.
> An alternative approach is to have the web dashboard query the collector
> either as needed or periodically, which also avoids the unnecessary and
> problematic queuing. That could be an HTTP or HTTPS GET to the collector
> (REST, et al), with approaches ranging from a blob of data or JSON or
> XML issued directly by the collector to using Tomcat or Python with
> Twisted or Tornado. Python also has extensions allowing system service
> access, including $getrmi.
> https://wiki.vmssoftware.com/VMS_specific_Python_modules
>
> HPE went heavily toward Redfish / DMTF for these and related tasks
> (though that work largely post-dates HPE interest in OpenVMS
> enhancements), so there are parallels to this approach:
> https://www.hpe.com/us/en/servers/restful-api.html — which can allow
> extending the display to other collectors. https://github.com/DMTF and
> https://dmtf.github.io/python-redfish-utility/#overview etc — and it
> means there are potentially other tools and dashboard apps that can be
> integrated.
>
> An approach using message queuing (MQ, Mosquitto, AMQP, etc) and DEC RTR
> tools and frameworks is the follow-ons to the frameworks that various of
> us were developing years ago, but again this particular case is one
> where queuing the traffic is a problem. Where dropping (old) messages
> here is appropriate. And message queues don't (typically) do that.
> OpenVMS mailboxes definitely don't. From the oldest of the dashboards we
> were doing many years ago, that dashboard was listening for
> periodically-sent Ethernet multicast frames, and more recently that
> dashboard approach would be UDP/DTLS or ilk, or an approach based on
> REST polling or ilk.
>
> DCL stinks at handling IP, more recently also stinks at handling TLS,
> and DTLS, and knows zilch about REST and JSON and other data structures.
> Which has all been a longstanding issue. Python and some other tools are
> vastly better at that and related tasks.
>
> TL;DR: I'd not use a mailbox nor queue here, but rather a datagram or
> maybe polling. That also gets curl out of the path. And can be extensible.
>
Pretty much agree.
#1 concept. Keep it modular. That way there can be multiple collection
methods, multiple data storage methods, multiple display methods. Much
easier to replace or upgrade specific modules than the whole system.
Do you envision most of the application residing and executing on the
running VMS node(s), or gathering the data and getting it to alternate
node(s) as quickly as possible? Running the application will itself be
a load on the running system. I see you're looking to use browsers for
display. Will the server be on the running system? Lots of options.
Are you looking to keep your project closely held, or possibly
collaborate with those who might already have some stuff that can be
used, or modified to use?
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: davef at tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
More information about the Info-vax
mailing list