Recently I released SonicBoom v0.0.6. SonicBoom is a PyTorch alternative, and I chose my favorite GNU Guile as its first-class language. I feel comfortable that I can finally do AI things with Scheme. There is no Python anywhere at runtime. The Python ecosystem caused me a lot of pain in product deployment, and now it's gone. The SBOM is reduced to a minimum, and we know what we deploy.
These days most people prefer not to type code by themselves (I still write the compiler and key infra myself), so the meaning of a programming language is the safety and optimization that come from the nature of its features. I'll have another post about programming languages in the AI era.
My idea is a bit old school hacker
I only have an RTX 3050. I once considered buying a 4090 or 5090 for my AI research, that's the mainstream. But I suddenly remembered John Carmack once said in 2013:
"There's so much you can still do on the previous console generation."
Besides, the mainstream autonomous driving chip has a similar config, say, NVIDIA DRIVE Orin-X / Jetson AGX Orin. And on AWS, A10G has the same architecture as 3050 (sm_86) with bigger VRAM. It's good bang for the buck.
So that's it, 3050 is good enough for a start.
However, I'm not going to optimize specifically for the 3050. The optimizations must be general, so you get higher performance on a better GPU.
So what did we get in v0.0.6?
Speculative decoding is on by default, and it is exact. The tokens you get with speculation on are the tokens you get with it off, bit for bit, on every run.
Qwen2.5-7B edits code at up to 523 tok/s on an RTX 3050 (8GB).
Of course you may not want to use Qwen2.5-7B for code editing, it's too weak. But it shows the power of SonicBoom. Next, we're working on MoE, e.g. Qwen3.6-35B-A3B, with flexible memory management to run big MoE models on 8GB VRAM. It seems popular today to do so.
The machine is a desktop RTX 3050 8 GB (sm_86, 20 SMs, roughly 200 GB/s of memory bandwidth) with a Ryzen 5 5500. The model is Qwen2.5-7B-Instruct Q4_K_M.
The workload is code editing: ten requests, each pasting a 100–300-line source file and asking for an edit. Decoding is greedy, and the generation length is the file's length.
The result is 471–523 tok/s for code editing. But when writing brand-new code, the speed drops back to about 41 tok/s. It's reasonable since code editing spends most of the time on copying rather than generation. But I would say code editing is one of the most common needs, so it's worth optimizing for it. The full benchmark is in the README.
But speed is not the point; quality is. Speculative decoding is a double-edged sword: it raises the speed, but in most engines it also changes the output, because the drafted tokens are checked with different kernels than plain decoding uses.
Why speculation usually changes the output
Speculative decoding drafts several tokens cheaply, then checks them in one forward pass of the real model. The pass scores k+1 positions at once, and a drafted token is kept when it matches what the model would have produced. With greedy decoding this is lossless in exact arithmetic: you can only ever accept tokens the model would have picked anyway.
GPUs don't do exact arithmetic. A verify pass is a batch of k+1 rows, while a plain decode step is a batch of one, and engines are built to pick kernels by batch shape. A matrix-vector kernel serves one row, a tiled matrix-matrix kernel serves sixteen, and the attention kernel splits its keys differently for different query counts. Each choice changes the order in which floating-point numbers are added up. That changes the last bits of the logits, and whenever two candidate tokens are close, the last bits decide which one wins. After one flipped token, the rest of the text goes its own way.
That is why, in many engines, turning speculation on can change the output. It is also why a speedup claimed with speculation on is hard to judge. You are no longer comparing the same text.
How SonicBoom keeps every row exact
However, this is just a simple case. In actual engineering, there are more things that make it complex:
kernel selection by batch shape
split-K and stream-K
fused vs. unfused epilogues
split-KV attention partitions
online softmax rescaling
KV cache write paths
rollback of rejected drafts
sliding-window ring reuse
stopping inside a verify batch
timing-based autotuning
......
BTW, each of them has its own test in SonicBoom's test suite.
Making verification cheap enough to matter
Exactness only pays if verification is cheap. On this card, a plain Qwen2.5-7B step is about 24.6 ms, essentially the time to stream the weights once. A 16-row verify costs about 1.25× that. A 64-row verify costs about 2.9×, and when a 63-token draft is accepted, that one step emits 64 tokens.
SonicBoom verifies up to 64 rows per step. Every row bucket (2 to 16, 24, 32, 48 and 64) is built, warmed and captured as a CUDA Graph when the model loads, so a verify step costs one graph replay.
Reproducible, with stated limits
SonicBoom's claim is: same version, same GPU model and same warm-up table give the same output. That includes seeded sampling: draws are keyed by seed, position and token id, so they don't depend on how the tokens were batched.
The warm-up table is measured on your own GPU when a model loads. With the table imported, a second machine with the same GPU model skips measuring entirely and should produce the same tokens. We have checked this on one machine: export, import into an empty cache, zero re-measurements, identical hashes on all ten turns. We would like to see it across machines; see the benchmark issue at the end.
What is not covered yet is batched serving: several requests sharing one batch.
Why reproducibility matters
SonicBoom is an official project of HardenedLinux, an old-school hacker community. We care about security and individual sovereignty. Our philosophy is "for fun and for profit". SonicBoom is one of our choices to help individuals get affordable AI power, and we also use it for serious products that require security. Reproducibility is critical in security and auditing.
When the same input gives the same output, you can replay any answer, explain it, and prove it hasn't changed. That is trustworthy AI.
Run it yourself
Repository: https://github.com/hardenedlinux/SonicBoom
And please help us benchmark v0.0.6 publicly:
