Open Source
BatchGen: An Architecture for Scalable and Efficient Batch Inference
arXiv cs.DC0 views
arXiv:2606.21712v2 Announce Type: replace Abstract: Batch inference has become a central mode of AI computation, yet existing inference engines still rely on execution models designed for interactive serving. When scaled to millions of sequences, batch workloads reveal two fundamental requirements: the ability to handle extreme inter- and intra-sequence load variation that emerges only at runtime, and the ability to sustain high utilization across large fleets of GPUs. Existing systems fail to meet these requirements, losing substantial fractions of achievable throughput. We introduce a new architectural foundation for batch inference: the sequence coroutine compute model, which represents each sequence as a fine-grained, event-driven coroutine. This model exposes expressive primitives that allow the runtime to reorganize work dynamically, enabling larger expert-level batches, mitigating stragglers, reallocating work across devices, and maintaining utilization even on cost-effective or memory-constrained GPUs. Building on this abstraction, we implement BatchGen, a production-ready system that uses the coroutine model at cluster scale. On a 128-GPU cluster, BatchGen reduces batch completion time by up to $2.3\times$, and on memory-constrained accelerators it outperforms the strongest offloading baseline by up to $9.6\times$. We will open-source BatchGen at https://github.com/batchgen-project/batchgen
Source:https://arxiv.org/abs/2606.21712PreviousDemystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU CommunicationNextProgrammable Probabilistic Computer with 1,000,000 p-bits
Related posts
Signature-Kernel Based Evaluation Metrics for Robust Probabilistic and Tail-Event Forecasting
0arXiv stat.ML
Towards Pose-Robust Few-Shot Vision-Language-Action Models
0arXiv cs.RO
High-Precision Docking of Differential-Drive Mobile Robots Through LiDAR and Vision Collaboration
0arXiv cs.RO
Building Your Own: A Practical Guide to Datasets, Pipelines and Evaluation for an AI Intrusion Detector
0dev.to AI
How 5 Companies Use AI Coding Agents and Assistants Across the SDLC
0dev.to DevOps
Show HN: Pip – Your Git history puts you on a Performance Improvement Plan
0Hacker News Show