Backend
Zero-SM NCCL: A Systematic Guide to Designing and Deploying SM-Free GPU Communication
arXiv cs.DC0 views
arXiv:2611.18075v1 Announce Type: new Abstract: GPU communication libraries such as NCCL use Streaming Multiprocessors (SMs) for data movement and synchronization, slowing overlapped computation at a cost that grows with interconnect bandwidth. We present Zero-SM NCCL, which communicates within and across scale-up domains without launching any SM kernel. It synchronizes through CUDA stream memory operations, moves data asynchronously with copy engines within a scale-up domain, and drives network RDMA from a stream-ordered CPU proxy. It supports data-movement collectives such as AllGather and AlltoAll, plus one-sided ncclPutSignal and ncclWaitSignal APIs. On GB200 NVL72 and B200 NVL8 systems, Zero-SM collectives reach up to 1.36x higher AllGather bandwidth than NCCL's legacy SM-based kernels and cut the median slowdown of overlapped GEMMs from 33% to 4%, at a higher latency per operation. In Megatron-LM training, they accelerate FSDP by up to 3.5%, and one-sided puts accelerate pipeline parallelism by up to 15%. Zero-SM communication is not a universal win, however. Through an end-to-end study across models and parallelism strategies, we show that it pays off for large transfers in GPU-bound workloads and for deep pipelines with large bubbles, but loses on small messages and host-bound workloads. We distill these findings into guidelines for enabling zero-SM communication in NCCL and building it into other libraries.
Source:https://arxiv.org/abs/2611.18075PreviousThe Exact Stabilization Time of Self-Stabilizing Coloring: Mean, Variance, and DistributionNextMulti-GPU Tridiagonal Partition Algorithm
Related posts
The $\alpha$-Limit Problem: Convergence of a Linear Degenerate Interface Transmission Problem
0arXiv cs.NA
A Spectral Proximal DC Algorithm with Nonmonotone Line Search and Extrapolation
0arXiv cs.NA
Watch Your Step: Adam's Margin Maximization Bias Depends on the Learning Rate Schedule
0arXiv stat.ML
Minimizing classical resources in variational measurement-based quantum computation for generative modeling
0arXiv stat.ML
Deep Multivariate Models with Parametric Conditionals
0arXiv stat.ML
CoordX Types: A Trustless, Swappable Type System for Shared Records
0arXiv cs.CR