On a Monday in October, the first day of a retailer's promotion, our label service began failing about one request in five from mid morning and kept doing it until the afternoon peak passed. CPU on its two virtual machines stayed under thirty percent. Every error in its log read: [Errno 99] Cannot assign requested address.
The service is a Python application that asks our rendering service for a label PDF for each parcel, through one internal load balancer address. The call was a bare requests.post inside a helper function. Called that way, requests creates a session for the single request and closes it afterwards, so every label meant a new TCP connection and a new TLS handshake, closed by our side as soon as the response arrived.
The side that closes a TCP connection keeps it in TIME_WAIT, and on Linux that lasts sixty seconds. Until it ends, the local port it used cannot carry another connection to the same address and port. Linux takes local ports from ip_local_port_range, which by default runs from 32768 to 60999, a little over twenty eight thousand ports. Divided by sixty seconds, that lets one machine open about four hundred and seventy new connections a second to one destination before it runs out. A normal day needs about two hundred and sixty per machine. That Monday needed more than five hundred. Once the range was used up, connect failed immediately, which is why nothing looked busy.
Our load tests had never found it because the test environment spread the same traffic across four machines.
The fix was one object. The helper now uses a single requests Session shared across the process, with a connection pool sized for the number of workers, so connections stay open and are reused. TIME_WAIT sockets on each machine fell from about twenty seven thousand to under a hundred. We did not widen the port range or enable reuse of TIME_WAIT sockets, because both would have moved the limit without removing the reason for it.
node_exporter's sockstat collector reports the TIME_WAIT count, and we now alert when any machine passes ten thousand. Our load tests run at production's rate per machine rather than its total.
A connection per request is paid for in ports as well as handshakes, and each port is held for a minute after you have finished with it.
– Sergey Shinder
