Backend
Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training
arXiv cs.DC0 views
arXiv:2610.19423v1 Announce Type: cross Abstract: Configuring distributed LLM training across hundreds of GPUs requires jointly optimizing tensor, pipeline, expert, context, and data parallelism alongside micro-batch size, recomputation, sharding, and overlap, a combinatorial space where memory legality can hinge on a single byte of activation accounting. Today, teams navigate it through expensive multi-node trial and error, burning GPU-hours on crashes and suboptimal recipes. We present Primus Projection, which closes this loop with two coupled components. First, a hybrid projection engine that measures what it can and simulates what it cannot: GPU microbenchmarks on as few as a single device capture compute kernels and any in-scope communication; a sub-node downscale-measure-upscale workflow analytically restores out-of-scope collectives, FSDP overlap, and the full pipeline schedule at the multi-node target. A pure simulation mode runs entirely on CPU, enabling pre-silicon planning without any GPU. Second, the Primus Tuning Agent treats the projection engine as a fast scoring oracle (seconds per candidate instead of tens of minutes) and uses it to drive an intelligent search over the configuration space. A deterministic seed planner sweeps single axes; an LLM investigation loop proposes cross-axis combinations the planner cannot reach, with an analytic memory pre-filter that rejects infeasible candidates before any GPU time is spent. Validated on Llama 3.1 (dense) and Mixtral 8x22B (MoE) across two GPU generations, all projections stay within ~10% of measured multi-node throughput, and fidelity holds at production scale: 4.7% error on DeepSeek V3 671B at 1,024 GPUs and 1.9% on Qwen3-235B-A22B at 256 GPUs. In a Mixtral 8x22B case study, the Primus Tuning Agent discovers a configuration delivering +27% throughput over the published 4-node BF16 reference at the same scale, in under 30 minutes of single-node exploration.
Source:https://arxiv.org/abs/2610.19423PreviousFederated Unlearning of Quantum Data via Classical ShadowsNextHow the Dirty Set Moves: Characterizing Schedule Reuse in Incremental Sparse Matrix Multiplication
Related posts
Three dates that catch a hard-coded time-zone bug in a session clock
0dev.to
Laravel N+1 Query Problem: How to Find and Fix It
0dev.to
LongCat-2.5 at $0.30/$1.20: Thinking Locks Temperature, 1M Context, and Vendor Benches Between Opus 4.8 and 5
0dev.to
Shipped: two packages, a registry listing, and the first seals
0dev.to
AI Agent Memory Needs Garbage Collection, Not Just Storage
0dev.to

Securing the Analytics Pipeline and Pivoting for Demo Day
0dev.to