[Info-vax] "bad select 38" (OpenSSL on VMS)

[Neil Rieck]

Gents,

I finally found a few spare hours to solve the "bad select 38" problem
as seen after executing DCL command "openssl s_client -connect". I
have a fix but it is not yet "production quality" and will never never
be as cool as solutions previously produced by the OpenVMS engineers.

It is important to note that "bad select 38" is never seen on the HP-
sanctioned versions of OpenSSL, just the ports done by third party
programmers (a.k.a. we, the unwashed public).

   ###

First off, here's a few basics about "UNIX I/O" and "select()".

I'm sure most of you already know that the standard UNIX devices
(stdin, stdout, and stderr) correspond to FD (file-descriptors)
numbers: 0, 1, and 2. As you open new i/o channels (files or sockets),
you use up more file-descriptors. In the VMS world these UNIX devices
map directly to SYS$INPUT, SYS$OUTPUT, and SYS$ERROR.

In the UNIX world, if you want to stall (block) your program while
waiting for I/O to complete, you indicate channels of interest by
setting associated bits in one of two longs (one for read, one for
write), then pass them on to select(). So if you wanted to only wait
for "read readiness from stdin" you would set bit-0 in the read bit-
field then pass it on to select(). If you wanted to wait for any one
of several devices, you would just set more bits.

But here is where "our VMS problem" develops. In the UNIX world "you
can use select() to test all i/o channels" while in the VMS world "you
may only use select() to test network channels (sockets)". In fact,
error #38 is defined in file "UCX$INETDEF.H" with the label "ENOTSOCK"
which is mapped to VMS code SS$_NOTNETDEV (Socket operation on
nonsocket)

As is typical of my luck, I was working backwards by starting with the
code, but had I checked the UNIX portability document here:

UNIX, Linux to OpenVMS Porting Guidelines
http://h71000.www7.hp.com/portability/portingguidelines.html
http://h71000.www7.hp.com/portability/portingguidelines.html#_Toc262045735

...then I could have saved myself much time.

So what to do? There are two ways out of this debacle:
1) test for keyboard input without using select()
    (this is easy to do but you might miss i/o from other channels if
not careful)
2) follow the advice of the portability guide then write some code to
employ a socket to connect to SYS$INPUT (or SYS$COMMAND).

If you want to see an example of how one programmer ("I Think" it
might have been a Compaq employee but have no proof because the
program changes were not documented) solved this problem, then click
here:
http://www3.sympatico.ca/n.rieck/demo_vms_html/openvms_demo_index.html#openssl

Neil Rieck
Kitchener / Waterloo / Cambridge,
Ontario, Canada.
http://www3.sympatico.ca/n.rieck/docs/openvms_notes_ssl.html#bad-select-38