[Info-vax] OT: TCP: trading latency for bandwidth

[johnson.eric at gmail.com]

On Thursday, May 17, 2012 4:15:46 PM UTC-4, JF Mezei wrote:

> The end result is that the peformance was sigificantly slower on VMS
> than on OS-X.

I've also noticed the lousy performance of FTP and TCP and the stacks in general on VMS especially when compared to other OSes.  In my case, I was comparing to Linux.

Yes, I know about TCP_NODELAY and the other ACK related issues.

When I was researching the topic, I decided to switch to UDP because it has a simpler model with fewer knobs to twist or choices the stack can make.

To look at the performance, I wrote a simple program which would issue a small UDP packet.  It did this in a tight loop.  The basic question I was trying to answer was - "How much time does it take to go from user-mode, into the stack, to send a packet, and come all the way back up".  I recall sending something like a 64 byte UDP packet.

On a 1 Gbps card, that's a measly 84 bytes (64 + 20 from IP's header).  If my math is correct, transmission time for that should be under a microsecond.   

I no longer have access to a VMS box, but as I recall, it took VMS several orders of magnitude more time to send a single UDP packet than it did on something like Linux.  My memory is fuzzy, but I recall the Linux box being able to blast out nearly a million UDP packets in a second, but the VMS box struggled to hit 20,000 in a second.

The situation became more curious if you pointed that UDP packet blaster at a VMS node!  Some of the stacks on VMS are configured to send an ICMP message back to the sender IF there was no receiver for that UDP packet on that host.  This of course travels back to the senders box and further interrupts the stack and causes the sender program to take even more time to send a single packet.

The impact of the ICMP message coming back into the stack likely resembles the impact of an ACK in the case of TCP.

Lastly, if you run that UDP blaster program from a Linux box and point that at a VMS box, you can watch the VMS node became quite busy trying to service those UDP packets.  While saturating any stack on any operating with a 100k+ packets a second is a mean thing to do, the pain on VMS seems quite high by comparison.

Again, going by memory, if you run something like sys$examples:spl.com while this is happening, you will see locks, such as IOLOCK8 and whatever locks your stack uses, operating under heavy contention.  I recall Multinet's stack experiencing more pain than the HP/UCX flavor.  But both compared quite poorly to Linux.

If there's interest, I can write a fresh version of the UDP blaster program so you can try it out for yourselves.  Might be interesting to compare results.  Unfortunately, I no longer have access to a VMS box so I can only produce linux based results.  

EJ