Meta uses idle spare GPUs and checkpointing to keep Prometheus AI training running

Spares absorb failures

Meta said more than 11 percent of the GPUs at its Prometheus supercluster in New Albany, Ohio, are dead or sitting idle waiting to replace a failed unit. The reserve pool is part of the design for a system that Meta says will eventually reach about a gigawatt of capacity and around one million accelerators, using Nvidia Grace Blackwell rack systems and, later, Vera Rubin systems.

The current cluster uses GB200 and GB300 systems with 72 GPUs in each NVLink domain, but only 64 are used at any moment. The other eight act as hot spares. Meta said this matters because one failed GPU could otherwise take down an entire rack and interrupt a training run. With a spare, the replacement can slot in, though the failure still causes a shorter interruption.

When Prometheus began its first phase, Meta said it saw roughly 1,000 interruptions per day for every 8,000 GPUs. An interruption means something crashed, failed or timed out and the training run rolled back to the last known good checkpoint. Meta said that after about a year it reduced interruptions by 99 percent, to about 11 per day for every 8,000 GPUs.

Checkpointing limits rollback

Meta also described how it contains faults inside large training jobs that use data parallelism, where multiple copies of a model are resident in the cluster. If a fault can be isolated to one data parallel model replica, that replica goes offline while the rest of the cluster repeats the step and keeps training. The compromised slice then restarts on spare hosts and resynchronizes, turning what could be a full halt into a temporary reduction in aggregate throughput.

If a fault goes beyond the data parallel domain, Meta relies on checkpointing. The company said its distributed checkpointing architecture asynchronously saves model state, including gradients and optimizer data, to multiple storage tiers. That lets a run resume from a saved state rather than starting over. Meta framed the work as necessary to keep models such as Muse Spark and the more recent Muse Wonder on track.

Links and power shape scale

One challenge was making NVLink fabrics behave like one large accelerator. Meta said earlier H100 and H200 systems had eight GPUs per NVLink domain, while Nvidia Blackwell expanded that by nine times. With more than 600,000 NVLink connections, even marginal signal degradation triggered frequent recovery cycles. Meta said it used continuous telemetry and L11 diagnostic burns to isolate weak links before systems entered production.

Power was another issue. Meta said ramping tens of thousands of GPUs up and down for a training run can create large power transients that risk overwhelming power supply equipment. The company said that by prioritizing cluster stability and reliability it reached more than 85 percent goodput for pretraining and more than 90 percent for reinforcement learning on Muse Wonder. Looking ahead, Meta wants to cut the interruption rate to two per day for every 8,000 GPUs, or about 100,000 GPU hours of mean time between interrupts.

Sources: The Register