Our order graph went flat at peak because the kernel was dropping StatsD packets

Our storefront has sent its business metrics to StatsD for years. Every order placed increments orders.placed: the client sends a small UDP packet to the StatsD daemon on a metrics host, and the daemon adds the packets up and flushes the totals to Graphite every ten seconds. UDP was the point. The shop never waits for monitoring, and if the metrics host is down nothing in checkout notices.

On Black Friday the orders graph on the wall display rose all morning and then went flat at about two thousand a minute from half past eleven until two. The trading team asked whether the site had hit its capacity. It had not. The orders table for the same period showed peaks above three thousand a minute.

The daemon is a single Node.js process. During the peak, page views, basket events and timers from forty storefront pods arrived faster than it could parse them, and each packet waits in the socket's kernel receive buffer until the process reads it. Ours was the distribution default, a couple of hundred kilobytes, which fills in a fraction of a second. When that buffer is full the kernel drops the next datagram and tells nobody but itself: the counter is RcvbufErrors in /proc/net/snmp, which netstat -su shows as receive buffer errors. It had climbed by forty million that day. It had also climbed on every busy Saturday that year, by smaller amounts, and nothing collected it.

The loss did not fall evenly. It happened only when traffic was highest, so the graph was most wrong at exactly the moments people looked at it, and wrong in the direction that suggested a ceiling.

We raised net.core.rmem_max and the daemon's socket buffer, which bought headroom and fixed nothing for good. The storefront clients now aggregate counters for a second before sending, which cut the packet rate by about thirty times. The receive buffer errors counter on every metrics host is scraped, and an alert fires when it moves. The orders panel now has a second line, counted from the database, so a disagreement between the two is on the screen rather than in a post mortem.

We chose UDP so that monitoring could never slow the shop down. We had not followed that through to its other half: under load, monitoring is the part that gives way.

– Sergey Shinder