Our supplier catalogue import is a Python worker running in Kubernetes. It reads each supplier's product file, validates every row and reports progress with print, one line per thousand rows, naming the supplier and the row. Run it on a laptop and you can watch it count through the file.

In October it was OOMKilled every night on one supplier's file. kubectl logs --previous ended at row 41,000, so we spent a morning reading rows 41,000 to 42,000 for anything that could blow up memory. They were ordinary. Running the same file locally showed memory climbing only after row 63,000, where the supplier had started pasting whole HTML pages into the description column, and the kill came about eight thousand rows after that.

The logs were not wrong. They were late. When Python's standard output is a terminal it is line buffered, and each print appears at once. In a container, standard output is a pipe to the runtime, and Python buffers it in blocks of a few kilobytes, writing only when the block fills or the program exits normally. Our progress lines were short, so one block held thirty of them. A process killed for memory receives SIGKILL, which nothing can catch, and whatever sat in the buffer is never written. We had been reading the last line that happened to be flushed, not the last line the worker printed.

Nothing had ever made us suspicious, because our tracebacks always arrived. Since Python 3.9, standard error is line buffered even when it is not a terminal, so errors came through promptly and progress did not.

Every Python base image we build now sets PYTHONUNBUFFERED=1. The worker reports progress through the logging module, whose stream handler flushes after every record, and it writes one JSON object per line with the supplier, the row and its own memory use. The import also streams the description column and rejects any field over a size limit, which was the actual bug.

In a terminal, print means now. In a container it means eventually, and a process killed for memory never gets to eventually.

– Sergey Shinder