PyTorch 2.15 rewrites NCCL backend to improve GPU fault recovery

PyTorch 2.15 includes a rewritten backend for Nvidia's Collective Communications Library (NCCL), designed to improve fault diagnosis and recovery when GPUs work together. The release also adds other features and comes as the software's creators continue to move it from a researcher tool toward production AI use. The update consisted of 3,117 commits from 583 contributors.

NCCL backend rewrite

The previous PyTorch backend for NCCL, called ProcessGroupNCCL, could spawn dozens of communicators per process group, leading to intermittent deadlocks and unpredictable performance. PyTorch core developer Tristan Rice, a Meta software engineer specializing in distributed computing, said during a live YouTube livestream on Thursday that the interface to the networking library needed a rewrite because it was drowning in technical debt. The new backend uses a strict rule: one process group gets only one communicator during initialization. By default, this eliminates crosstalk between multiple communicators and related complexity; programs needing parallelism must dispatch more process groups.

The updated library also better handles stragglers by allowing easier reconfiguration. If a GPU crashes or runs slowly, a third-party orchestrator can exclude the laggard and assign a replacement worker. A flight recorder captures all operations in snapshots that can be inspected after a failure, replacing the generic message users previously received when a GPU cluster stalled from a timeout or deadlock. Rice noted the cleaned-up NCCL backend, also called ProcessGroupNCCL, will give developers a blank slate, making it easier to add new features.

Why NCCL matters

NCCL, pronounced 'Nickel,' provides PyTorch with inter-GPU communication primitives for single-node and multi-node clusters. This is vital for running inference on large neural networks and for building and maintaining large language models. Workers must constantly exchange tensors, gradients, activations, and weights, and NCCL uses optimized communication paths to move data between GPUs while reducing CPU overhead compared with standard TCP-based networking.

Nvidia initially released NCCL at SC15 as a general-purpose accelerator. It was adopted by the Facebook team when they first built PyTorch in 2016, with the goal of building a Python-based version of the Lua-based Torch machine learning library. Will Stewart, co-founder of AI software provider Northflank, said in a blog post, "Every major ML lab (OpenAI, DeepMind, Meta, Nvidia) uses PyTorch. It powers everything from diffusion models and LLMs to recommender systems and robotics. This widespread adoption isn't accidental."

Other 2.15 features and rollout

Other new features include a Python transport API for asynchronous one-way data movement, suited to training jobs where weights are synchronized between trainers and generators, plus parallel processing improvements for Apple gear and support for Nvidia's next-generation Blackwell GPUs. The legacy ProcessGroupNCCL remains in the 2.15 package as a rollback option. If the new NCCL backend works as expected, the older version will be removed in the 2.16 release, planned for early next year.

Sources: The Register