[Info-vax] Apache + mod_php performance

[Dan Cross]

In article <vdvoeq$1jerg$1 at dont-email.me>,
Dave Froble  <davef at tsoft-inc.com> wrote:
>On 10/6/2024 11:12 AM, Michael S wrote:
>> On Fri, 4 Oct 2024 17:43:02 -0000 (UTC)
>> cross at spitfire.i.gajendra.net (Dan Cross) wrote:
>>
>>> In article <vdp8kn$a67s$1 at dont-email.me>,
>>> Dave Froble  <davef at tsoft-inc.com> wrote:
>>>> On 10/3/2024 7:00 PM, Chris Townley wrote:
>>>>> [snip]
>>>>> I don't remember George, but we have certainly woken up Dave! ;)
>>>>>
>>>>> and I am sure the troll is happy...
>>>>
>>>> I'm not sure whether I've been insulted?
>>>
>>> I suspect the "troll" reference is to Lawrence.  Sadly, Arne can
>>> not help himself when it comes to resisting arguing with that
>>> clown.
>>>
>>> 	- Dan C.
>>>
>>
>> Troll or not, but the question about ability to pass open TCP socket to
>> child process (or, may be, to unrelated process) under VMS is a good
>> question.
>> As a lurker, I am waiting for the expert answer with interest.
>
>Well, some of the issue is in the text of the question.  What does one mean be 
>"pass socket"?

Perhaps an example will be illuminating.  See the program at the
bottom of this post; it is self-contained and, I claim, fairly
representative of the technique.  I wrote and tested it on
OpenBSD 7.5, on a RISC-V devbox.

A short explanation: when invoked, this program creates a
connected, unnamed Unix domain socket pair using the
`socketpair` system call.  It then forks off three other
(child) processes; these are "workers" that are invoked with one
end of the socket pair as an argument; the child closes the
other end and blocks waiting in the `recvmsg` system call,
waiting to be passed a connected socket.

Back in the parent, after creating the worker processes, the
program closes the end of the unnamed socket pair that it passed
to its children: the effect here is that the parent is holding
onto one end of the pair, while the children are (correctively)
holding onto the other.  Since there are outstanding references
on the socket pair, it's still alive.  Note that each child is
blocked waiting to receive a message on the Unix domain socket.

The parent process then enters a "listener" loop: here, it
creates a TCP socket on a given port (I used `8200`, but that's
just a detail) and loops, accepting incoming connections bound
for that port on any interface.

This is where it gets interesting, and arguably this is the
important point: when a client connects, after the `accept` the
state in the parent is that the process has a fully established
TCP connection.  It then uses the Unix file descriptor passing
API to transfer that socket to one of the child processes; one
(that is not otherwise busy; more on that in just a moment) is
blocked in `recvmsg`, as above and will win a race to be the
next to read from the childrens' end of the Unix domain socket.
This has the effect of duplicating the socket descriptor for the
just-established connection in the child; the child can now
execute "normal" IO operations on the socket, communicating with
the original client, as if it had been the one to accept the
connection in the first place.  Note that the workers never had
access to the listening socket created by the parent; they were
forked off before that was created.  Of course, the same
applies to any sockets created when the parent accepted an
incoming connection.

The parent then closes the socket descriptor that had been
created when it `accept`ed the incoming connection, and loops.
In the client, the program receives the socket descriptor from
the parent and enters an `echo` loop, doing the obvious thing
of writing whatever the client sends it back to the client,
until the client closes the connection.

This raises an obvious question: given that the child workers
are is synchronous with respect to receiving sockets from their
parent, and the `echo` loop makes no attempt at parallelism,
what happens if all of the workers are busy when a new
connection is established?  The answer is that, modulo some
buffering in the kernel, things block: the listener will shovel
new connections into the kernel for as long as the kernel will
buffer the associated single-byte writes (and other control
metadata attached to the message copying the IO descriptor to
one of the children), but then at some point, `sendmsg` will
block and the kernel will buffer a few incoming connctions that
haven't been `accept`ed yet, and then it'll start erroring on
connection establishment.  In the meanwhile, IO on the
established connections that are waiting for a worker to pick
them up will appear to block.  Note that this is basically the
behavior that Arne saw originally.

>When creating a socket, one can specify it to be shared.  What I was doing was 
>passing the information to a worker process, then letting the worker process 
>open the existing socket.
>
>So, would that be considered "passing an open socket"?

It doesn't really sound like it, but it's hard to say without
seeing some code.  What's missing in the above is any
description of _how_ the shared socket is shared with the worker
process, and when.

The salient characteristic of the Unix model is that a program
can pass a file descriptor representing any resource to another
process and that prcoess can interact with that resource as if
it had opened the descriptor itself (with the privileges of the
parent).  For a socket, for example, the socket in question need
not exist before the child is created.

>I can post some of the development code is anyone is interested.  I was working 
>on the inter-process communications when I dropped the project.  I believe I did 
>open the shared socket in the worker process.

That might be useful to see.

	- Dan C.

// Demonstration of a listener process that accepts
// incoming TCP connections and passes them to a
// worker process over Unix domain sockets.
//
// Dan Cross <cross at gajendra.net>

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/uio.h>
#include <netinet/in.h>

#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

// Sends a file descripter, `fd`, over a Unix domain socket `sd`
// via the 4.4BSD access rights passing mechanism.
//
// Returns 1 on success, -1 on failure.
int
sendfd(int sd, int fd)
{
	struct msghdr mh;
	struct iovec iv;
	struct cmsghdr *ptr;
	size_t len;
	int ret;
	char dummy;

	// Construct the control message header.  Note this is
	// malloc'ed to ensure proper alignment.
	len = CMSG_SPACE(sizeof(int));
	ptr = malloc(len);
	if (ptr == NULL)
		return -1;
	memset(ptr, 0, len);
	ptr->cmsg_len = len;
	ptr->cmsg_level = SOL_SOCKET;
	ptr->cmsg_type = SCM_RIGHTS;
	memcpy(CMSG_DATA(ptr), &fd, sizeof(int));

	// We send a single byte of dummy data in case the
	// implementation does not pass control data with an
	// otherwise empty data transfer.
	dummy = 0;
	memset(&iv, 0, sizeof(iv));
	iv.iov_base = &dummy;
	iv.iov_len = 1;

	// Construct the message header.  Points to the dummy
	// data and the control message header.
	memset(&mh, 0, sizeof(mh));
	mh.msg_name = NULL;
	mh.msg_namelen = 0;
	mh.msg_iov = &iv;
	mh.msg_iovlen = 1;
	mh.msg_control = (caddr_t)ptr;
	mh.msg_controllen = len;
	mh.msg_flags = 0;

	// Loop in case there's no room in the kernel buffer
	// to send.  Cf.Stevens et al.
	do {
		ret = sendmsg(sd, &mh, 0);
	} while (ret == 0);
	free(ptr);

	return ret;
}

// Receives a file descriptor over the Unix domain socket `sd`
// and store it into `*fdp` on success.
//
// Returns 1 on success, 0 on EOF, -1 on error.
int
recvfd(int sd, int *fdp)
{
	struct msghdr mh;
	struct iovec iv;
	struct cmsghdr *ptr;
	size_t len;
	int ret;
	char dummy;

	if (fdp == NULL)
		return -1;

	// Allocate space for the control message.
	len = CMSG_SPACE(sizeof(int));
	ptr = malloc(len);
	if (ptr == NULL)
		return -1;

	// Fill in an iovec to receive one byte of dummy data.
	// Required on some systems that do not pass control
	// messages on empty data transfers.
	memset(&iv, 0, sizeof(iv));
	iv.iov_base = &dummy;
	iv.iov_len = 1;

	// Fill in the msghdr structure.  `recvmsg(2)` will
	// update it.
	memset(&mh, 0, sizeof(mh));
	mh.msg_name = NULL;
	mh.msg_namelen = 0;
	mh.msg_iov = &iv;
	mh.msg_iovlen = 1;
	mh.msg_control = ptr;
	mh.msg_controllen = len;
	mh.msg_flags = 0;

	ret = recvmsg(sd, &mh, 0);
	if (ret <= 0) {
		free(ptr);
		return ret;
	}
	if (mh.msg_flags != 0) {
		free(ptr);
		return -1;
	}
	memcpy(fdp, CMSG_DATA(ptr), sizeof(int));
	free(ptr);

	return 1;
}

void
dispatcher(int sdworker, int port)
{
	int sd, nsd;
	struct sockaddr_in sa;
	struct sockaddr_in client;
	socklen_t clientlen;

	sd = socket(AF_INET, SOCK_STREAM, 0);
	if (sd < 0) {
		perror("socket");
		close(sdworker);
		exit(EXIT_FAILURE);
	}
	memset(&sa, 0, sizeof sa);
	sa.sin_family = AF_INET;
	sa.sin_port = htons((unsigned short)port);
	sa.sin_addr.s_addr = htonl(INADDR_ANY);
	if (bind(sd, (struct sockaddr *)&sa, sizeof sa) < 0) {
		perror("bind");
		close(sdworker);
		close(sd);
		exit(EXIT_FAILURE);
	}
	if (listen(sd, 255) < 0) {
		perror("listen");
		close(sdworker);
		close(sd);
		exit(EXIT_FAILURE);
	}

	memset(&client, 0, sizeof client);
	clientlen = sizeof client;
	while ((nsd = accept(sd, (struct sockaddr *)&client, &clientlen)) >= 0) {
		if (sendfd(sdworker, nsd) < 0) {
			perror("sendfd");
			close(sdworker);
			close(nsd);
			close(sd);
			exit(EXIT_FAILURE);
		}
		close(nsd);
	}
}

static void
echo(int sd)
{
	char *p;
	char buf[1024];
	ssize_t nb, wb;

	for (;;) {
		nb = read(sd, buf, sizeof buf);
		if (nb < 0)
			perror("read");
		if (nb <= 0)
			return;
		p = buf;
		while (nb > 0) {
			wb = write(sd, p, nb);
			if (wb < 0)
				perror("write");
			if (wb <= 0)
				return;
			nb -= wb;
			p += wb;
		}
	}
}

void
worker(int sddispatcher)
{
	int sd;

	while (recvfd(sddispatcher, &sd) > 0) {
		echo(sd);
		close(sd);
	}
}

int
main(void)
{
	int sds[2];
	pid_t pid;

	if (socketpair(AF_UNIX, SOCK_STREAM, 0, sds) > 0) {
		perror("socketpair");
		exit(EXIT_FAILURE);
	}

	for (int k = 0; k < 3; k++) {
		pid = fork();
		if (pid < 0) {
			perror("fork");
			exit(EXIT_FAILURE);
		}
		if (pid == 0) {	// Child
			close(sds[0]);
			worker(sds[1]);
			exit(EXIT_SUCCESS);
		}
	}
	close(sds[1]);
	dispatcher(sds[0], 8200);

	return EXIT_SUCCESS;
}