Cold Attention: What Happens When You Run Softmax at Absolute Zero

Standard attention has a temperature problem. Not in the thermal sense, but in the mathematical one: the softmax function at the heart of every transformer uses a temperature parameter that, in practice, is never zero. That non-zero temperature is what makes attention "soft" — it spreads probability mass across many candidates rather than committing to one. For language modeling, that softness is a feature. For algorithmic reasoning, it turns out to be a fundamental liability.

A new paper from Mostafa 'Neo' Mohsenvand, Attention at Absolute Zero (arXiv:2611.11937, November 2026), proposes a higher-order attention layer that operates at the zero-temperature limit of softmax. The result is a model that learns graph algorithms from small examples and executes them exactly on inputs a thousand times larger — without any intermediate supervision.

Why Soft Attention Fails at Algorithms

To understand what cold attention fixes, it helps to understand what breaks. Mohsenvand identifies three distinct failure modes that emerge as input sizes grow beyond training distribution:

Dispersion. At any positive temperature, softmax distributes weight across all candidates. As the input set grows, that weight spreads thinner. The signal from the correct "witness" — the item that proves a relationship between two others — gets diluted by noise from irrelevant candidates.

Resolution. Learned score functions produce real-valued outputs. As inputs scale, the gap between the correct witness's score and near-misses shrinks. Eventually, fixed-precision arithmetic cannot distinguish them, and the model makes errors even when it has learned the right scoring function.

Carried error. Looped or recurrent architectures pass a continuous hidden state from one iteration to the next. Small errors accumulate. By the final loop, the state has drifted far enough from the correct trajectory that the output is wrong.

These are not implementation bugs — they are structural properties of soft attention. No amount of training on larger inputs fully eliminates them.

The Cold Attention Mechanism

Cold attention addresses all three failure modes with a single architectural choice: push the temperature to zero.

At $T \to 0$, the softmax operation collapses. Instead of a probability distribution over witnesses, the layer computes exactly two quantities: the maximum score among all candidates, and the count of candidates that achieve that maximum. These two numbers are the only information passed forward — no continuous probability mass, no soft weighting.

The layer maintains state on pairs of items $(a, b)$ rather than individual items. For each pair, every third item $k$ is evaluated as a candidate witness through its two "legs": the relationship $(a, k)$ and $(k, b)$. This higher-order structure allows the model to reason about transitive relationships — the kind that appear in graph algorithms like shortest paths and reachability.

To prevent carried error, the pair state is rebuilt from scratch at every loop using the raw input and the comparison bits from the previous iteration. Nothing continuous is carried forward.

In the "full cold" setting, numerical values never enter the state directly. They are kept in registers, and the internal state receives only comparison bits — yes/no answers to questions like "is witness $k$ the best seen so far?" This makes the reasoning invariant to the specific numerical representation, as long as ordering is preserved.

What It Achieves on CLRS-30

The CLRS-30 benchmark is the standard evaluation for neural algorithmic reasoning. It covers 30 classical algorithms — BFS, DFS, Dijkstra, Floyd-Warshall, quicksort, dynamic programming, and more — and tests whether a model trained on small inputs (up to 16 nodes) can generalize to larger ones (64 nodes at test time).

Previous state-of-the-art models like FloydNet achieved 96.64% average accuracy on CLRS-30, but they relied on hints: intermediate supervision signals that tell the model what the algorithm's internal state should look like at each step. Hints are a significant crutch — they require knowing the algorithm's internals at training time, which limits applicability to settings where you only have input-output pairs.

Cold attention achieves 97.04% average accuracy without any hints, trained on a single block per task. A shared block trained simultaneously on all 30 tasks reaches 97.20%. The "full cold" configuration — where numbers enter only through comparisons — reaches 97.98% on the tasks it covers.

The generalization results are more striking. For maximum retrieval (finding the largest element in a set), the model maintains 99.95% accuracy on sets of 16,384 items despite being trained on sets of at most 16 items — a thousand-fold extrapolation. A comparable model using standard soft counts drops to 42% accuracy at 4,096 items.

The model also solves 2-SAT and two-coloring problems exactly at 500 variables. When paired with a frozen 7B language model (Qwen2.5-7B), it enables exact family-relation chaining on 1,000-statement contexts — a task where the language model alone performs near chance.

Connecting Cold Attention to Broader Trends

The paper sits at the intersection of neural algorithmic reasoning and the search for attention mechanisms that scale more predictably.

Neural algorithmic reasoning — teaching neural networks to execute classical algorithms — has been an active area since DeepMind's Neural Execution Engine work. The challenge has always been generalization: models that learn to sort 16 numbers often fail on 100. Cold attention's theoretical grounding (it can prove its own correctness on acyclic structures) offers a path toward models that generalize by construction rather than by luck.

The zero-temperature framing also connects to work on spiking neural networks, where researchers have shown that spiking attention can be modeled as the zero-temperature limit of independent two-level systems. Cold attention arrives at a similar regime from a different direction — through algorithmic correctness rather than biological plausibility.

For practitioners, the most immediately useful result may be the hybrid setup: a frozen LLM augmented with a cold attention block for specific reasoning subtasks. LLMs handle language understanding; cold attention handles the discrete, structured reasoning that LLMs currently struggle with.

Practical Implications

The code, weights, and documentation are publicly available on GitHub under the Apache 2.0 license, with pre-trained models on Hugging Face. The Python package witness_clrs provides a JAX-based implementation with training scripts for all 30 CLRS tasks.

The current limitations are worth noting. The model is designed for structured, discrete reasoning tasks — it is not a drop-in replacement for standard attention in language models. The pair-based state representation scales quadratically with input size, which constrains practical input lengths. And the "full cold" configuration requires knowing which tasks benefit from comparison-only reasoning, which is not always obvious in advance.

Conclusion

Cold attention is a precise solution to a precise problem. Soft attention fails at algorithmic reasoning because dispersion, resolution errors, and state drift conspire against exact computation. By operating at absolute zero, rebuilding state at every loop, and passing only comparison bits rather than continuous values, the cold attention layer sidesteps all three failure modes at once.

The CLRS-30 results — state-of-the-art without hints, and generalization to inputs a thousand times larger than training — suggest this is not a theoretical curiosity. For tasks where correctness matters more than expressiveness, cold attention offers a principled alternative to the soft mechanisms that dominate current architectures.