AI & ML
HYPIC: Position-Independent Caching for Hybrid-Attention LLM Serving
arXiv cs.DC0 views
arXiv:2607.01299v3 Announce Type: replace Abstract: In retrieval-augmented generation (RAG) and agentic LLM serving, prompts are assembled from multiple independent segments into long contexts, making prefill dominate the computation cost. To reduce this cost, position-independent caching (PIC) admits KV reuse for non-contiguous segments across requests, while hybrid-attention models reduce computational complexity using linear attention. However, existing PIC methods break down on hybrid-attention models because their per-token KV-cache reuse primitives do not transfer to per-request recurrent states. In this work, we present HYPIC, the first system to accelerate hybrid-attention LLM serving with position-independent caching. For linear-attention layers, we identify the segment-cumulative transition operator as the missing algebraic primitive and cache it for each segment, enabling near-exact and constant-time state composition. For the remaining full-attention layers, we construct a small seam window to propagate hidden states through the linear-attention layers, so that cross-segment attention can be repaired by recomputation. Finally, HYPIC introduces segment parallelism, which parallelizes cache-miss prefill across instances, turning long cold requests into an accelerable workload. Evaluated across four hybrid-attention models and five workloads, HYPIC reduces time-to-first-token (TTFT) by $3.25\times$ on average and improves QPS by $1.66\times$ over Prefix Cache, while preserving task quality with a $1.71$-point gap from Full Recompute.
Source:https://arxiv.org/abs/2607.01299PreviousEkka: Automated Diagnosis of Silent Errors in LLM InferenceNextBreaking Fault Lines: Unifying TEE-Assisted BFT Consensus in Partially Trusted Worlds
Related posts
A Graph-Based Conceptual Modeling Framework to Design for Human-Robot Coordination
0arXiv cs.RO
Learning Soft Robotic Dynamics with Active Exploration
0arXiv cs.RO
MARLIN: Multi-Agent Reinforcement Learning Guided by Language-Based Inter-Robot Negotiation
0arXiv cs.RO
Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement
0arXiv cs.RO
WayPOP: A Panoramic Open-Set Panoptic Tracking Benchmark
0arXiv cs.RO
Beyond Accuracy: Information-Theoretic Decoder Evaluation Under Latent Intent
0arXiv cs.RO