[Info-vax] Large mailboxes

[Stephen Hoffman]

On 2020-11-28 17:37:37 +0000, Marc Van Dyck said:

> Collecting the data we need is not difficult, either from a DCL loop, 
> or from a program issuing $GETRMI calls. To display them, we're going 
> to use an in-house developped web service that can store and display any
> metric that one cares to throw at it. The interface to send the data is 
> cURL, which under VMS has a tendency to get stuck when used 
> intensively. So we'll decouple the collector and the sender, and put a 
> large mailbox between the two. If the mailbox is large enough, we can 
> tolerate a stall of the sender for several minutes, and it will catch 
> up with the mailbox contents very quickly once restarted.
> 
> This kind of data does not need to survive a reboot (there will always 
> be perfdat to look at what happened anyway) and a mailbox is a cheap 
> and easy to use buffer for this kind of purpose....

A couple of different approaches...

Have the collector send a UDP multicast, and have the dashboard queue 
requests for and receive and process that data. That entirely decouples 
the sender and receiver. And the use of the multicast inherently 
discards old data.

A queue whether through a mailbox or otherwise is problematic approach 
here both as it preserves previously-sent data, and as it preserves old 
data in preference to new.

I'd use DTLS here for sensitive data, though the available TCP/IP, 
UDP/IP, TLS, and DTLS APIs on OpenVMS are less than friendly.

An alternative approach is to have the web dashboard query the 
collector either as needed or periodically, which also avoids the 
unnecessary and problematic queuing. That could be an HTTP or HTTPS GET 
to the collector (REST, et al), with approaches ranging from a blob of 
data or JSON or XML issued directly by the collector to using Tomcat or 
Python with Twisted or Tornado. Python also has extensions allowing 
system service access, including $getrmi. 
https://wiki.vmssoftware.com/VMS_specific_Python_modules

HPE went heavily toward Redfish / DMTF for these and related tasks 
(though that work largely post-dates HPE interest in OpenVMS 
enhancements), so there are parallels to this approach: 
https://www.hpe.com/us/en/servers/restful-api.html — which can allow 
extending the display to other collectors. https://github.com/DMTF and 
https://dmtf.github.io/python-redfish-utility/#overview etc — and it 
means there are potentially other tools and dashboard apps that can be 
integrated.

An approach using message queuing (MQ, Mosquitto, AMQP, etc) and DEC 
RTR tools and frameworks is the follow-ons to the frameworks that 
various of us were developing years ago, but again this particular case 
is one where queuing the traffic is a problem. Where dropping (old) 
messages here is appropriate. And message queues don't (typically) do 
that. OpenVMS mailboxes definitely don't. From the oldest of the 
dashboards we were doing many years ago, that dashboard was listening 
for periodically-sent Ethernet multicast frames, and more recently that 
dashboard approach would be UDP/DTLS or ilk, or an approach based on 
REST polling or ilk.

DCL stinks at handling IP, more recently also stinks at handling TLS, 
and DTLS, and knows zilch about REST and JSON and other data 
structures. Which has all been a longstanding issue. Python and some 
other tools are vastly better at that and related tasks.

TL;DR: I'd not use a mailbox nor queue here, but rather a datagram or 
maybe polling. That also gets curl out of the path. And can be 
extensible.

-- 
Pure Personal Opinion | HoffmanLabs LLC