I misread my profiler, my rewrite made it slower, and two lines cut memory by 60%

My laptop said the program peaked at 71 MB. The scoring machine said 175 MiB.

Same binary. Same data. Same Docker image. Neither number was wrong.

That was the first of many times during Speed Demon 2026, the 72-hour performance championship run by Hackathon Raptors, when a measurement I trusted turned out to be correct and useless at the same time. Over 106 logged steps I found that almost every mistake I made had the same shape: I read a true number that was answering a different question than the one I was asking. The profiler answered "where did the samples land", not "what would disappear if I removed this". The instruction counter answered "how much work", not "what is the core waiting for". My laptop answered "how fast is this on my laptop".

This post is the list of those questions, what each wrong reading cost me, and the measurement that finally answered the right one.

The task, and a machine you never touch

I'll stay on one of the four tracks, Track B. Answer 50,000 search queries over a 254 MB corpus of 250,000 documents. 62% are BOOL queries (AND, OR, NOT, applied strictly left to right), 38% are exact phrases, and every output line has to match the reference byte for byte. The reference Python takes 56.35 s and 2.2 GB.

Each submission is scored as one number:

K = (T_ref/T)^0.5 × (M_ref/M)^0.2 × (S_ref/S)^0.15 × (D_ref/D)^0.15 points = 50 × log10(K) 

T is wall time, M the container's peak memory, S the line count after a formatter, D the dependency count. So halving the time multiplies K by 1.41, halving memory by 1.15, and doubling the code costs 10%. Keep those weights in mind: twice in this story they decided which "obvious" improvement was actually a loss.

The scoring machine is an Intel i5-13500 with turbo off. You push a commit, wait in a queue, and two numbers come back. Here is what that machine is, next to what I was developing on:

Almost every lesson below is a row of this table biting me.

Belief 1: "Peak memory is what I allocate"

It isn't. Peak memory on the board is the cgroup's memory.peak, which charges every resident page. What becomes resident depends on the kernel, not just on you.

My scan wrote its output into many blocks per thread, each filled a little at a time. At any moment most of them were half written. On my laptop the WSL2 kernel ran transparent huge pages in madvise mode: a process gets 2 MB pages only where it asks. The scoring machine evidently ran always, so the first byte written into a fresh region could pull in a whole 2 MB page. Many half-written blocks per thread, times twelve threads, is roughly the 100 MiB I couldn't see.

The fix was two lines of C++:

#include <sys/prctl.h> prctl(PR_SET_THP_DISABLE, 1, 0, 0, 0); // a block written in part stays small pages 

Sixty percent less memory and 4.3 points, for two lines.

Then I made the same mistake again. When I rewrote the program in Rust I left the line out, and memory crept back to 69.1 MiB on a version that should have been under 50. This time I didn't guess. I switched my laptop VM to THP always, reran both builds, and watched the difference appear locally (about 5 MB), then vanish with PR_SET_THP_DISABLE. On the box the same line took that version from 69.1 to 48.8 MiB (run-174007cb966d).

The question I should have asked: not "how much do I allocate?" but "what will this kernel make resident when I touch it?" If your laptop and your target disagree about memory, look at the boring knobs first: transparent huge pages, the allocator's mmap threshold, how freed memory goes back.

Belief 2: "Shorter code will score higher"

On day two the C++ was 968 counted lines against a reference of 91, and lines are in the formula. I worked out that a version at 60% of the size and the same speed would score about 7% more, and I rewrote it from scratch in Rust: 612 lines, correct on every dataset on its first build.

The arithmetic explains it exactly. Fewer lines are worth (968/612)^0.15 = +7.1%. But 9% more time costs (0.232/0.253)^0.5 = -4.2%, and 25% more memory costs (52.7/65.6)^0.2 = -4.3%. Net: -1.8% of K, which is the 0.40 points the board took away.

A rewrite throws away dozens of small decisions you made without writing them down (the THP line was one of them). I put the C++ back on the board and kept improving the Rust until it actually won, two steps later (0.241 s, 58.0 MiB, 68.86 points). The rewrite was right; shipping it on an estimate was not.

Hold on to that arithmetic, because near the end of the contest it pointed the other way.

Belief 3: "The profiler shows me where the time goes"

A sampled profile on the laptop said building the posting lists took 25% of all cycles. I spent an evening on that phase. Then I built a variant with its stores switched off, and the run got under 2% faster.

Sampling inside that VM attributed cycles to the wrong threads. The profile was answering "where did the timer interrupts land", and in a VM with twelve busy threads that is a different question from "what does this code cost".

From then on I used one method for cost: ablation. Build a variant with exactly one part disabled, count what disappears (user cycles, instructions, wall time), put it back. It is slower than reading a flame graph, and I have never been misled by it.

Belief 4: "Fewer instructions means faster"

This is the one I'm most glad I got wrong, because the way it broke taught me how the box actually works.

perf stat said the scan retired 136 instructions per token over 36.3M tokens. The disassembly showed why: the per-document loop kept 16 values live and reloaded most of them from the stack on every token, and the hash took 25 instructions. That is a deeply satisfying thing to find. I split the loop into short ones, made the hash one 128-bit multiply, and cut 10% of the whole program's instructions (6,516M to 5,881M).

The box's answer, run-152f40943619: 0.1407 s, against 0.1389 s before. Ten percent less work, zero time saved.

So I asked a different question: what is the core waiting for? Cache-miss counters, plus ablation, gave two answers I should have looked for first:

• Phrase checks cost 144M instructions but 7.6M of the scan's 26 to 29M last-level cache misses. Each check read four scattered arrays (anchor starts, anchor records, query spans, query terms) to reject one window. Cheap in instructions, expensive in misses.
• A 107 KB per-thread array existed to drop a term's repeats within a document. It was read and written at a random place for every list-term token. I counted what it actually did: 10.94M of 11.02M list-term occurrences were already new to their document. It removed 0.7% of the postings and cost a likely cache miss on most tokens.

The fixes were plain. Each phrase anchor got a 16-byte record holding everything needed to reject a window, prefetched for every anchor in a document before the other loops run. The dedupe array went away entirely; the few repeats are dropped later, when the lists are built from sequential data.

Later I tried the instruction route once more, on purpose: vectorizing the per-document loops cut 7% of instructions and bought 1% of time (run-807c1ffd2504). Same answer.

Two things to take from this. First, with two hyperthreads per core and a lot of memory traffic, the core's instruction throughput is not the limit; its wait for memory is, and the other hyperthread happily soaks up whatever instructions you remove. Second, and more general: count what a structure actually does before you optimize it. I had spent effort making that dedupe array fast. The right number to know was "0.7%".

Belief 5: "I can't measure the machine I can't touch"

By now I trusted the box over my laptop, but the box only tells you two numbers. So I made the program tell me more through those two numbers.

The trick: time a phase inside the program, then encode that duration into the outputs the board reports. For memory, touch one 4 KB page per 4 µs of the phase and keep the pages until exit, so peak memory rises by an amount you can decode. For wall time, sleep for ten times another phase before exiting, so the noise of a single run matters ten times less.

// Phase A's duration, written into peak memory: one touched page per 4 µs. let pages = phase_a.as_micros() as usize / 4; let p = mmap(null_mut(), pages * 4096 + 4096, 3, 0x22, -1, 0); (0..pages).for_each(|k| write_volatile(p.add(k * 4096), 1)); // Phase B's duration, written into wall time, ten times over. std::thread::sleep(phase_b * 10); 

One such run (run-d59c352f1a2a) came back 8.4 MiB heavier and 64 ms slower than the normal run. Decoded: 8.6 ms parsing the queries, 6.4 ms of serial setup. Fifteen milliseconds of a 124 ms run on one thread while eleven threads waited. A later run of the same kind found the answer phase at 29 ms on the box, against 16 ms on my laptop: almost exactly the ratio of the two clocks.

The parse was the embarrassing part. I was splitting a 1.8 MB query file byte by byte, then word by word, about 28 instructions per byte, on one thread, while the corpus itself already went through a 64-byte AVX2 class mask. Reading the queries with the same mask halved the parse's instructions on the laptop (36.9M to 18.4M). On the box it was worth about 10 ms: 0.1243 s to 0.1144 s (run-fa63626812c0), an 8% step from code that was nowhere near the hot loop.

These measurement runs scored terribly on purpose (71.04 and 66.70 points). They were the best runs I made.

The general version: if a black box only reports scalars, make your program speak through them. Any benchmark harness that reports time and memory can be turned into a crude profiler of itself.

Belief 6: "If it's faster on my laptop, it's faster"

Not every disagreement went my way:

• A smaller term table (1 MB instead of 2 MB) cut about 5M last-level misses on the laptop. On the box: 0.1243 s to 0.1335 s. Longer probe chains cost more than the misses saved.
• Turning lists into bitmaps sooner made my laptop's answer phase about 10% faster. On the box: 0.1144 s to 0.1489 s (run-1aa9c7fc4435), a third slower. More 31 KB bitmaps were zeroed, filled and counted on a machine where two threads share 1.25 MB of L2.

My laptop had more cache per thread, twice the clock, AVX-512, and page faults that cost 9 µs because of the VM. Of course it disagreed. I stopped treating laptop results as measurements and started treating them as hypotheses that the box would accept or reject.

Belief 7: "The slow phase needs to be faster"

The biggest late gain came from asking a different kind of question: not "how do I make the answer phase faster", but "why does anyone have to wait for it?"

The old program scanned the whole corpus, built one index, then answered. The new design reads the corpus in phases of about 50 MB. Each phase gets its own small 16-bit postings and bitmaps, every query is answered against that phase into a running tally (count, id sum, ten smallest ids), and the next phase reuses the memory. Two things fall out of that. No full index ever exists, so memory drops. And the answers for phase k run on whichever threads have already finished scanning phase k+1, so the answer phase overlaps the scan instead of waiting for all of it, filling time when those threads would otherwise sit idle.

Here is the arithmetic from belief 2, pointing the other way. 364 more lines cost (739/1103)^0.15 = -5.8%. But 18% less time is worth (0.1102/0.0907)^0.5 = +10.2%, and 13% less memory (47.8/41.4)^0.2 = +2.9%. Net +6.8% of K, about 1.4 points, which is what the board gave.

Same formula, same kind of trade, opposite verdict. Neither "shorter is better" nor "faster is better" is a rule. The weights are.

I then tried slicing the phases differently: 1 MiB slices gave 0.0973 s, 128 KiB slices 0.0953 s. The 256 KiB slices stayed.

Belief 8: "All my datasets pass, so it's correct"

While vectorizing a loop that reads eight ids at a time, I found a buffer that grew by 2 × n entries whenever fewer than 32 were left. A 64-byte window can add 32 tokens. Growing by 2n only guarantees that room once n reaches 24. So the first document each thread read could write past its buffer, if its first 64-byte windows held more than 16 one- or two-letter tokens.

All six datasets passed, including the hand-built edge cases. The bug had been on the board for ten steps. I generated 5,000 documents made only of single letters and digits, took the reference's answer, and ran mine: heap corruption, abort. The fix was one expression, 2 * n + 32, and that dense dataset now runs in every check before a push.

The spec's edge cases test the spec. They cannot test your invariants, because the spec doesn't know about your buffers. Whenever you write a bound, write the input that sits exactly on it.

What is still wrong

• Track B ends at 0.0907 s, 41.4 MiB, 78.70 points, about 620 times faster than the reference. When I last checked, the leader was at 82.41 points and 0.064 s. I don't know how they get there, and that gap is real.
• Query terms are matched by 47 bits of their hash with no byte comparison. The chance that some non-query token in a run collides with a query term is about 3 in 10,000. It has never happened on any dataset, but "never happened" is not "can't happen", and a second tag checked only on a hit would close it cheaply.
• My laptop is still a bad model of the box. I've learned which of its answers to distrust, not how to make it agree.
• The other tracks ended at 86.16 (log aggregation), 81.93 (CSV join) and 71.92 (shortest paths). Track D in particular never got the kind of breakthrough described here.

The rules I'd give myself on day one

1. Before you trust a number, say which question it answers. Most of my wasted days came from a correct number answering the wrong question.
2. Measure cost by removal. "What disappears when this is gone" is very hard to misread.
3. On shared cores, count misses before instructions. My 10% instruction cut bought nothing; deleting one array that did 0.7% of useful work bought 3%.
4. Count what a structure does before you speed it up. The answer may be "almost nothing".
5. Make the target measure itself. Two scalars are enough to carry a phase timing out of a machine you can't log into.
6. Treat the environment as code. Two lines about huge pages beat a day of tuning, twice.
7. Do the scoring arithmetic every time. The same trade of lines for speed lost me 0.4 points once and won me 1.4 later.

Check it yourself

The repo includes the organisers' checker, which builds a track offline and runs it in the same sandbox as the board. With Docker and Python, and the sample data generated as the repo's notes describe:

git clone https://github.com/manusingh090/speed-demon-2026-entry cd speed-demon-2026-entry python check.py b --data data/b-sample --expected eeeec1e9319da98a6f3c664c78f93e7078fd603c25c2bb3027174e81e3638a8f 

Every row of track-b/PERF.md names the board run that measured it, including the failures and the deliberately terrible measurement runs, so any number in this post can be checked against the public board.

References

• Speed Demon 2026, by Hackathon Raptors: rules, scoring and the live board
• Transparent Hugepage Support, Linux kernel documentation on the always and madvise modes
• prctl(2), for PR_SET_THP_DISABLE
• The full Track B log: 106 steps, each with its board run

Thanks to Hackathon Raptors for a contest where the scoring machine had the last word on every claim, especially the ones I was sure about.