Backend
ReKV: Overlapping Quantized KV Transfer and Suffix Recomputation
arXiv cs.DC0 views
arXiv:2611.08770v1 Announce Type: new Abstract: Prefill--decode disaggregation moves a prompt's key--value (KV) cache between workers before the first token can be generated. Reducing this handoff to either cache compression or recomputation misses a useful choice: which layers to transmit, at what precision, and which layers to execute again on the decode worker. ReKV transfers a quantized prefix KV cache and one BF16 boundary state, then reconstructs a contiguous suffix while the prefix is in flight. In a model-shaped synthetic benchmark on cross-node H100s, the Q8 configuration meeting separately evaluated quality criteria reduces 8K handoff latency by $2.36\times$, $2.84\times$, and $3.65\times$ over BF16 transfer for Qwen2.5-7B, Qwen2.5-3B, and Qwen3-8B. It is also faster than the best measured transfer-only or recompute-only action at those points. A separate real-weight BF16 recomputation experiment reduces prefill-to-token-return time by $1.16$--$1.63\times$. The preferred action changes with prompt length and link bandwidth.
Source:https://arxiv.org/abs/2611.08770PreviousThe GPU Needs an Operating SystemNextAdaptive Scheduling Pipeline for Multi-Instance Asynchronous Reinforcement Learning
Related posts

Dans osFoundry, chaque agent IA a un département et un rôle, et vous n'avez plus qu'à vérifier et approuver
0dev.to AI

What Cursor Live Apps Handles and Where a Local Setup Still Wins
0dev.to AI

I Ran a Real Internal Tool Through Cursor Live Apps. Here Is the Part That Actually Helped
0dev.to AI

An agent that stops paying for data it already learned was useless
0dev.to AI
Karaoke for songs nobody has sung before: what broke, and what we learned
0dev.to AI
I priced transactional email at 10k, 50k and 200k a month. The recommended plan is often the wrong one
0dev.to AWS