AI & ML
JigSLO: Solving the Resource Puzzle for Efficient Multi-LLM Serving
arXiv cs.DC0 views
arXiv:2509.06261v3 Announce Type: replace Abstract: GPU spatial sharing co-locates instances with complementary demands, such as compute-bound prefill and memory-bound decode, promising cost-efficient multi-LLM serving. Prior systems partition compute and coordinate shared memory capacity, but do not model bandwidth contention among co-located instances of heterogeneous LLMs. This contention makes each instance's latency depend on the memory-access patterns of its co-runners, even when its own resource allocation remains unchanged. Drawing on extensive measurements, we find that the latency impact of bandwidth contention can be captured by a time-shared memory-access model with contended and uncontended periods, so the overall latency can be analytically estimated as a time-weighted average of the latency estimates for these periods. We propose JigSLO, a contention-aware multi-LLM serving system that manages memory resources and schedules each model's prefill and decode phases as separate instances, treating their co-location as a placement decision. Its contention-aware predictor estimates per-instance latency under any co-location from standalone profiles alone. Guided by these predictions, its placement algorithm jointly selects each instance's tensor parallelism, compute share, and batch configuration, pairing complementary memory-access patterns to maximize throughput within each model's latency targets. JigSLO's shared resource manager complements this on the capacity side, sharing weights, pooling KV cache, and transferring it between prefill and decode instances. Across three datasets covering chat, code generation, and long-context summarization, JigSLO outperforms state-of-the-art baselines by $2.3\times$ in SLO attainment and $1.2\times$ in throughput on average, matching their best attainment with half the GPUs on long-context workloads.
Source:https://arxiv.org/abs/2509.06261PreviousSpecSearch: Towards Hardware-Aware Efficient Speculative DecodingNextAngelfish: Leader, DAG, or Anywhere in Between
Related posts
The AI Token Casino: How the Regenerate Button Drains Your Budget
0dev.to AI
The Teacher in the Age of AI: Will the Machine Take Their Place, or Give Them Back Their Time?
0dev.to AI

Skillware v0.6.2: Section-Chunked Tax PDFs, RPC Health Gates & Strict Core Typing
0dev.to AI

GPT-6.2 Astra vs GPT-6 Astra: What Changes in Cost per Accepted Task?
0dev.to AI
Jobs That Fade, Jobs That Emerge: What the Labour Market Looks Like Up to 2030
0dev.to AI
Can an AI agent read your pricing page and submit your demo form? A dev checklist for agent-ready sites
0dev.to WebDev