Our website's catalogue comes from our trading system. Every night a report exports each product we sell online, with its description, price and stock, and our web agency's import collects the file. The report was written in 2012, and it produces an Excel 97 to 2003 file, because that is what the agency asked for at the time.

In June 2024 a roofer told one of our sales representatives that we had stopped selling wood screws online. We had not. The agency checked the file. The screws were missing from it, and so were about two thousand three hundred other products, every one of them alphabetically later than a pack of wall plugs.

A worksheet in that old Excel format holds at most 65,536 rows. The report sorted products by description, and when the catalogue passed that size, towards the end of 2022, the export wrote the first 65,536 rows and stopped without any error. The agency's import treated a product missing from the file as withdrawn and hid its page. From then on, each product we added anywhere in the alphabet pushed the last one in the list off the website, and the cut crept slowly up from the letter Z. Wall ties, waterproofing, wood preservative, workwear and zinc flashing had all quietly disappeared. The same limit was behind the loss of nearly sixteen thousand positive Covid test results in England in October 2020.

Nothing had alerted anyone, because every part did what it was built to do. The report finished, the file was valid, and the import followed its rule. The products had their own pages and search entries, so their online sales did not fall to zero in a day. They faded over eighteen months, which looked like an ordinary decline in a few product ranges.

The export now writes a CSV file, which has no row limit, and its last line states how many products it contains and the total of their prices. The agency's import rejects any file whose contents do not match that line, and will not withdraw more than one percent of the catalogue in a single night without someone confirming it. Each month we compare the number of products marked for sale online in our trading system with the number live on the site, and the two figures now agree.

When data passes between two systems, count it at both ends. A file can be complete in format and incomplete in content, and neither system will tell you.

– Serguey Shinder