Why Your Local LLM Gives Different Answers at Temperature 0.

You set temperature to 0, fix the seed, send the same prompt twice, and get two different answers. This post explains where the difference comes from, how to see it on your own machine, and what each engine offers if you need answers that repeat.

The symptom

One measured case: I ran lm-evaluation-harness on gsm8k (100 problems) against llama.cpp's server (build b11643, Qwen2.5-7B-Instruct Q8_0, -np 8), once with one request at a time and once with eight concurrent requests. Same model file, same settings, greedy decoding. The generated text differed on 41 of the 100 problems. On problem 0 the two runs agreed for 213 characters; then one wrote "so she makes" and the other "so the total amount she makes", and the rest of the solution went its own way. The strict-match score moved from 0.83 to 0.82.

It is not just one engine:

• Thinking Machines Lab sent the same prompt 1,000 times at temperature 0 and got 80 different completions (Defeating Nondeterminism in LLM Inference, 2025).
• Yuan et al. measured up to 9% accuracy variation for DeepSeek-R1-Distill-Qwen-7B in BF16 when only the number of GPUs, the GPU type or the evaluation batch size changed (arXiv 2506.09501).
• The issue trackers of llama.cpp, Ollama, LM Studio and vLLM all have reports of temperature-0 output that changes with the prompt cache, speculative decoding, the number of slots or the GPU.

What temperature 0 and the seed actually fix

Temperature and the seed control one step: picking the next token from the probabilities the model computed. At temperature 0 the engine takes the most probable token, and the seed stops mattering.

Neither touches the probabilities themselves. If the logits come out slightly different, greedy decoding faithfully picks from slightly different numbers. So the real question is why the same model computes slightly different logits for the same input.

Cause 1: floating-point addition depends on order

A computer rounds after every floating-point addition, so (a + b) + c is not always equal to a + (b + c). Each logit is the end of millions of multiply-adds, and the order in which a GPU kernel adds them up decides the last bits.

A few lines of NumPy show it:

import numpy as np rng = np.random.default_rng(0) x = rng.standard_normal(4096).astype(np.float32) def fsum(v): # add in float32, front to back  s = np.float32(0) for t in v: s = np.float32(s + t) return s print(fsum(x) == fsum(x[::-1])) # False: a different order changes the last digits  n = np.rint(x * 2**13).astype(np.int64) # the same values on an integer grid print(n.sum() == n[::-1].sum()) # True: integer sums don't depend on order 

Keep the last two lines in mind. They come back at the end.

Cause 2: the engine changes the order for you

GPU kernels are written for speed, and the fastest way to add numbers depends on the shape of the work. Your request stays the same, but the order changes whenever the engine changes how it computes:

• Batching and concurrent requests. The server computes your request together with whatever else it is doing. A matrix multiplication for one row and for eight rows can use different kernels with different reduction orders, so your row comes out slightly different depending on who else was in the batch. Thinking Machines showed that this lack of batch invariance is the main cause in inference servers.
• The prompt cache and chunk size. A prompt computed in one piece, in chunks, or restored from a cache goes through different kernel paths. llama.cpp's server documentation says so directly: logits are not guaranteed to be bit-for-bit identical for different batch sizes, so the prompt cache (on by default) can cause nondeterministic results.
• Speculative decoding. The verify step scores several draft tokens in one batch, which is a different computation from generating them one by one.
• KV cache quantization, the GPU, the driver and library versions. Each can change which kernels run.
• Kernels that are not deterministic from run to run, such as reductions built on atomic additions. They are rarer in inference, but they exist.

Why a tiny difference changes the whole answer

Usually the top token wins by a wide margin, and a difference in the last bits changes nothing. But a long answer passes through steps where two tokens are nearly tied. There, a one-bit difference can change which token wins. From then on the model is continuing a different text, so everything after that point is regenerated: different wording, sometimes a different line of reasoning, sometimes a different final answer.

The flips go both ways, which is why scores move far less than answers. In one of my measurements (600 gsm8k problems, DeepSeek-R1-Distill-Qwen-7B Q8_0, greedy), switching between two engines that run the same weights with different numerics changed the text of most answers and flipped correctness on 53 problems: 25 became wrong and 28 became right, a net change of 3 problems. So when you compare two models or two quantizations, compare the outputs item by item, not just the average score.

See it on your own machine

Start llama.cpp's server with several slots (use your own model path):

llama-server -m qwen2.5-7b-instruct-q8_0.gguf -ngl 99 -np 8 -c 16384 

Then run this script. It sends one prompt alone, then again while seven other prompts run at the same time, and compares the two answers. It works with any OpenAI-compatible server; for Ollama or LM Studio, pass the URL and a model name as arguments.

# Send one prompt alone, then again while 7 other prompts run at the same time, # and compare the two answers. Standard library only. import json import sys import urllib.request from concurrent.futures import ThreadPoolExecutor URL = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8080/v1/chat/completions" MODEL = sys.argv[2] if len(sys.argv) > 2 else None # Ollama and LM Studio need a model name TARGET = "Explain why the sky is blue in three short paragraphs." OTHERS = [ "Write a short story about a lighthouse keeper.", "List ten facts about octopuses.", "Explain how a hash map works.", "Summarize the plot of Hamlet.", "Describe how vaccines train the immune system.", "Write four haiku about autumn.", "Explain the difference between TCP and UDP.", ] def ask(prompt): body = { "messages": [{"role": "user", "content": prompt}], "temperature": 0, "seed": 42, "max_tokens": 400, } if MODEL: body["model"] = MODEL req = urllib.request.Request(URL, data=json.dumps(body).encode(), headers={"Content-Type": "application/json"}) with urllib.request.urlopen(req, timeout=600) as r: return json.load(r)["choices"][0]["message"]["content"] alone = ask(TARGET) with ThreadPoolExecutor(max_workers=len(OTHERS) + 1) as pool: futures = [pool.submit(ask, TARGET)] + [pool.submit(ask, p) for p in OTHERS] together = futures[0].result() if alone == together: print("identical") else: n = min(len(alone), len(together)) i = next((k for k in range(n) if alone[k] != together[k]), n) print(f"different: first difference at character {i}") print("alone: ...", alone[max(0, i - 40):i + 60].replace("\n", " ")) print("together: ...", together[max(0, i - 40):i + 60].replace("\n", " ")) 

On a GPU build you will often see different and the character where the answers part. How often depends on the model, the GPU and the build, so try a few prompts. Sending the same prompt twice in a row can differ as well: the second request may reuse the cached prompt and take another path. llama.cpp lets you turn that off per request with "cache_prompt": false.

What you can do today

What to set depends on the engine, and each option costs something.

Two things none of these give you: identical results on a different GPU, let alone between NVIDIA and AMD, and a way for someone else to check your numbers without your hardware.

Making the order irrelevant

The NumPy example holds the other half of the answer: integer sums don't depend on order. That suggests a different approach. Instead of forcing every kernel to add in one fixed order, make the order not matter.

That is what I built GeulPU to do (disclosure: I'm its author). It is a local engine for GGUF models (Qwen2.5, Qwen3, Llama 3, Mistral) on NVIDIA (CUDA) and other GPUs (Vulkan). It turns activations into blocks of 32 integers with a power-of-two scale, sums the products exactly on int8 tensor cores, and fixes every remaining rounding point in a written precision contract. The GPU output matches a NumPy CPU reference bit for bit, so batching, the prompt cache and speculative decoding stop mattering:

The deterministic modes of vLLM and SGLang above cost speed. Against llama.cpp, GeulPU's default mode doesn't. On an RTX 4070 Ti with Qwen2.5-7B Q4_K_M, version 0.11.0 took 0.76–0.97× llama.cpp's time at every prompt length (512 to 65,536 tokens) and session depth (8k to 64k) measured. The llama.cpp build was b11827, using whichever of its f16 and q8_0 KV caches was faster. A recorded 54-request coding-agent session took 0.59× the time of llama.cpp's fastest setting, a 0.5B draft model. Exact mode, which also makes NVIDIA and AMD agree, is slower: 1.19–1.40× llama.cpp's time on prompts up to 8k tokens. GeulPU is Windows-first (Linux builds from source), runs dense models only for now, and is a solo project: https://github.com/wwoosshh/GeulPUv2. The study behind the "flips go both ways" numbers, with its raw data, is at https://github.com/wwoosshh/GeulPU.

Checklist

• Set the seed anyway. Not every engine or API treats temperature 0 as strictly greedy, and it costs nothing.
• Temperature 0 and a seed fix the sampling step, not the numbers it samples from.
• If answers change, look first at concurrency, the prompt cache and speculative decoding, then at the GPU, driver and library versions.
• For repeatable runs with a standard engine: one request at a time, no prompt cache, no speculation, the same hardware and versions; or the deterministic modes of vLLM and SGLang.
• Compare outputs item by item, not just the final score. Most flips cancel out in the average.
• If you need identical results with concurrency and caching on, or across GPUs, you need an engine built for it.

This article was drafted with an AI assistant and checked by the author.