AI & ML
Placing LLM Lookup Tables by Per-Layer Prefetch Deadlines
arXiv cs.DC0 views
arXiv:2611.13413v1 Announce Type: new Abstract: DeepSeek-V4.1's 203 GB of Engram n-gram tables sit in host DRAM today, taking 190 of the 223 GiB a 4-GPU serving process uses. Row indices depend only on input tokens, so reads can start with each step and overlap earlier layers. V4.1's two Engram layers have windows that differ by 30x (0.3 vs. 9.3 ms). We show that this deadline, together with session reuse and contention on a shared device, decides which tier each table can use and which layers need cache, and we give a model that predicts the resulting slowdown from measured read times. Agent and chat traffic reuse rows within sessions, and static hot sets transfer poorly. We build SlackTier, a table tier for SGLang that issues all lookups at step start. It reads the short-window layer first and splits a small GPU row cache across layers. The path runs inside the decode CUDA graph and is bit-exact with the original gather. Layer 14's table can go to NVMe uncached with no measurable decode cost. With both tables on four NVMe disks, mean time per output token on agent and chat replays rises at most 1.4% over unmodified SGLang, and host memory falls from 212 to 34 GiB. Both tables need no cache when read over RDMA (remote direct memory access) from another node's memory; one copy serves two instances. On a KV-constrained agent workload, using the freed memory as host KV cache at the same memory budget raises median throughput by 36% and cuts median p90 time to first token from 38 to 24 s.
Source:https://arxiv.org/abs/2611.13413PreviousBursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent ServingNextForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding
Related posts
From Bias to Positionality: A critical grammar for Large Language Models
0arXiv cs.HC
Variance reduction with probing and Multilevel Monte Carlo in Lattice QCD
0arXiv cs.NA
GAIA: Geometry-Adaptive Operator Learning for Forward and Inverse Problems
0arXiv cs.NA
Agentic Symbolic Search: Approximating Solutions of PDEs Beyond Hand-crafted Expressions, Meshes, and Neural Networks
0arXiv cs.NA
MC$^2$: Monte Carlo Correction for Fast Elliptic PDE Solving
0arXiv cs.NA
Low-rank cross approximation of function-valued tensors for reduced-order modeling of parametric PDEs
0arXiv cs.NA