Open Source
BatchGen: An Architecture for Scalable and Efficient Batch Inference
arXiv cs.DC0 views
arXiv:2606.21712v2 Announce Type: replace Abstract: Batch inference has become a central mode of AI computation, yet existing inference engines still rely on execution models designed for interactive serving. When scaled to millions of sequences, batch workloads reveal two fundamental requirements: the ability to handle extreme inter- and intra-sequence load variation that emerges only at runtime, and the ability to sustain high utilization across large fleets of GPUs. Existing systems fail to meet these requirements, losing substantial fractions of achievable throughput. We introduce a new architectural foundation for batch inference: the sequence coroutine compute model, which represents each sequence as a fine-grained, event-driven coroutine. This model exposes expressive primitives that allow the runtime to reorganize work dynamically, enabling larger expert-level batches, mitigating stragglers, reallocating work across devices, and maintaining utilization even on cost-effective or memory-constrained GPUs. Building on this abstraction, we implement BatchGen, a production-ready system that uses the coroutine model at cluster scale. On a 128-GPU cluster, BatchGen reduces batch completion time by up to $2.3\times$, and on memory-constrained accelerators it outperforms the strongest offloading baseline by up to $9.6\times$. We will open-source BatchGen at https://github.com/batchgen-project/batchgen
Source:https://arxiv.org/abs/2606.21712PreviousDemystifying NVSHMEM: A System-Level Analysis on Symmetric Memory and Device-Initiated Operations in GPU CommunicationNextProgrammable Probabilistic Computer with 1,000,000 p-bits
Related posts
Receita Federal publica código da Calculadora de Tributos no GitHub
0dev.to GitHub
Lifting 3D Gaussians: A Wavelet Multiresolution Representation for Scalable Gaussian Splatting Streaming
0arXiv eess.IV
Binarized Spectral Compressive Imaging
0arXiv eess.IV
HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
0arXiv eess.IV
GramStatTexNet: Efficient, Interpretable, and Neuro-Inspired Texture Analysis-by-Synthesis
0arXiv cs.GR
Webhook Rate-Limiting Queue Design for US/EU Delayed Jobs, Retry, and DLQ
0dev.to Go