[Info-vax] Harden TCPIP Srv OVMS again SYN FLOOD attacks

[Stephen Hoffman]

On 2018-08-13 16:42:23 +0000, Rod Regier said:

> The following cookbook will harden the IP stack for TCPIP Services for 
> OpenVMS V5.3 and above against SYN FLOOD attacks.

For folks interested in this topic, see RFC 4987 for the common 
mitigations for a TCP SYN flood attack.

https://tools.ietf.org/html/rfc4987

And some introductory fodder on this particular mess:

https://www.cisco.com/c/en/us/about/press/internet-protocol-journal/back-issues/table-contents-34/syn-flooding-attacks.html 


> These updates override the cautious HPE distributed defaults.  The 
> default was based on "small memory"
>         somaxconn=65535
>         sominconn=65535
> ...

The above increases the number of available sockets.

Which means the server will now have more sockets available to flood.

If that increase in the number of sockets has mitigated the effects of 
the SYN flood in this case, then that mitigation is likely due to some 
other factor.  Maybe insufficient bandwidth in the host(s) or app(s) or 
network attempting the SYN flood, for instance.

I'd wonder if the Hyenae SYN flood packet generator or its host was 
running out of capacity, or running into a preconfigured limit.

I don't know if TCP/IP Services is coded to detect and drop SYN floods 
from a fixed source address.  I'd tend to doubt it, though.  That'll 
mitigate a simple SYN flood.  Firewalls can provide this, if TCP/IP 
Services does not.

Either in addition to or in place of increasing the sockets — though 
pragmatically, there probably aren't all that many good reasons not to 
run with 64K sockets configured in this era — I'd suggest having a look 
at the tcp_keepinit setting, and specifically at reducing that value.  
That's one of the mitigations listed in the RFC, and it's something 
that means that flooding gets cleaned up more quickly.  Downside is 
that the DDoS only gets bigger with a shorter timeout or with more 
sockets or both, and the server still gets flooded.   This as discussed 
in the RFC.

If the services are targets for spoofed SYN floods or other DDoS 
shenanigans, then the configuration is probably headed for CloudFlare 
or other services, as I'd expect that increasing the number of sockets 
available on OpenVMS and shortening the tcp_keepinit will just consume 
more resources on the server and not markedly harden the server against 
a TCP SYN flood.

Receiving a hundred gigabytes of DDoS traffic — TCP SYN or otherwise — 
was possible five years ago, and more recently there've been various 
folks hit with terabyte-scale DDoS.   Largest DDoS I'm aware of was 1.7 
terabytes per second.  I'm not aware of any OpenVMS server that has 
sufficient network controllers and capacity to even support that volume 
of traffic, all other factors and considerations aside.


> I did a before/after test probing my test server running the OSU web 
> server using a SYN FLOOD attack generated from Hyenae.  The difference 
> was like night and day.

Haven't tried Hyenae; will have a look at that one.  That tool seems to 
have been forked to JHyenae, as well.  hping is another common tool for 
that sort of load testing.

The following part of the quoted posting was reordered...

> This can be useful if you are still exposing an OpenVMS web server to 
> the outside world.

A directly-exposed OpenVMS server is often not a particularly 
auspicious configuration.  I've long preferred OpeNVMS configured 
behind a firewall and preferably one with VPN and quite possibly one 
with appropriate filtering, and with additional capabilities for 
detecting the common sorts of IP shenanigans.  If not additionally 
configured behind a DDoS prevention service, for that matter.


-- 
Pure Personal Opinion | HoffmanLabs LLC