Backend
Twig: Parallel and Memory-Efficient Tree Speculative Decoding for Hybrid-Attention LLMs
arXiv cs.DC0 views
arXiv:2608.01651v2 Announce Type: replace Abstract: Emerging LLMs increasingly interleave full attention with recurrent linear attention to reduce long-context serving costs. Tree speculative decoding could further accelerate these hybrid models, but recurrent linear-attention layers carry parent-dependent states. Existing engines therefore verify proposal trees sequentially and snapshot the full state per node, causing latency and memory to grow with tree and batch sizes. We present Twig, a kernel--runtime co-design for tree speculative decoding of hybrid LLMs. Twig derives a tree-structured factorization of the linear-attention recurrence and realizes it with a resource-efficient GPU kernel that verifies all nodes in parallel, accelerating linear-attention tree verification by 3.4--7.7$\times$. It losslessly represents speculative state updates as token-level factors and reconstructs only the accepted state, reducing transient state memory by 82--99$\times$ and freeing GPU memory for KV caches. We integrate Twig into SGLang, a widely deployed production LLM serving engine, and further optimize its runtime through batch-wide verification budgeting. Across four models, two GPU platforms, and diverse datasets, Twig achieves up to $4.72\times$ the offline decode throughput of autoregressive decoding and up to $2.03\times$ that of the strongest tree-speculative baseline. Under online agent workloads, Twig reduces TTFT and TPOT by up to $67.6\%$ and $49.9\%$, respectively, over the strongest tree-speculative baseline.
Source:https://arxiv.org/abs/2608.01651PreviousWave-Based Dispatch for Circuit Cutting in Hybrid HPC-Quantum SystemsNextLantern: Finding Committable Transactions via Back-Propagation on DAGs
Related posts
Executable but Unfaithful: Mock Object, Stub, and Assertion Generation by LLMs and Coding Agents
0arXiv cs.SE
SaltBench: A Referee-Gated Protocol for Measuring Method Effects in Machine-Checked Software Work
0arXiv cs.SE
QDiff: Differential Testing of Quantum Software Stacks
0arXiv cs.SE
Don't Prompt Your Agents. Equip Them: Skillware as a Modular Standard for Sovereign, Auditable, and Deterministic Capability Execution for Logical Systems
0arXiv cs.SE
Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks
0arXiv cs.SE
Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations
0arXiv cs.SE