The Perils of TCP

I've recently been working in an environment supporting legacy webservices. What has surprised me here and elsewhere is how much or a black art TCP is. Seasoned system admins don't always know the weaknesses in the protocol, and often developers don't realise.

Send delays on small packets
First if you are getting poor performance sending data over tcp a good place to start is Nagel's Algorithm. John Nagel proposed some measures for early internet congestion control, one measure was to surpress the sending of small packets of data, this is now commonly called Nagel's Algorithm. If your application is sending a lot of small messages this can cause slow sending, delaying for upto 500ms (if your stack uses standard settings). It is turned off by the setting the TCP NO_DELAY option. What is interesting is Nagel himself suggests that it was a workaround (and admits that it doesn't interact well with other TCP control measures). He suggests that a better mechanism would be for the stack to determine how well the communication is working and adapt appropriately.

Loosing packets on shutdown
Another problem that can occur in TCP, and you can see it a lot in legacy servers is loosing packets on shutdown. This can often be prevented using the linger option, although this is buggy on some stacks. One workaround involves shutting down the write before closing which causes the FIN to be send and allows draining to happen.

Keep connected
Some when programming at the transport layer misunderstand why it is better to stay connected. Connections are subject to a ramp up in speed, stacks start sending slow and ramp up to a maximum value set in the stack. Staying connected has implications for firewalls and other enterprise environments. It it common for firewalls to drop network connections that have been idle for a configured amount of time--- this time can be less than the default Keep-Alive time for the stack. For this reason some protocols implement their own keep-alive message managed by the application.

Security
When TCP was designed and implemented it wasn't thought that people would attempt to exploit it: security was not built into the design. Over the years a number of vulnerabilities have been found with the state-machine and sequencing that allows attackers access or to bring down a service.

To be continued...