Meta Turns Idle AI Inference Capacity Into Better Ads Recommendations

The Inference Capacity Paradox

Meta has built a system that enhances ads recommendations by leveraging idle inference capacity in its fleet. Inference fleets face a capacity paradox: they can be globally constrained while meaningful pockets of capacity remain locally idle. Common causes include peak-provisioning and demand-variation mismatch, model-development and capacity-delivery timing mismatch, regional resource-ratio mismatch, and accelerator-type mismatch. At Meta's scale, these temporary pockets can collectively represent substantial capacity and create better ads recommendations when used safely.

Two Elasticity Levers

The system uses two complementary levers. Rank More Ads (RMA) adjusts the number of candidates evaluated by the ranking stages when opportunistic capacity is available, increasing the chance of finding ads relevant to the user. RMA requires both CPU and accelerator surplus. Serve Better Models (SBM) adjusts the share of requests routed to a more powerful model as accelerator capacity changes, keeping the number of ads entering ranking stages constant so upstream CPU demand remains stable. SBM can therefore use accelerator-only surplus that RMA cannot consume. A model-router component uses request-value signals to prioritize requests expected to benefit most from a more powerful model.

Protecting Baseline Traffic

Physical and logical isolation protects capacity reserved for baseline serving. SBM uses physical isolation: baseline and more powerful models use separate routing targets, with dedicated replicas for baseline and opportunistic replicas for the more powerful model. RMA uses logical isolation: the system calculates how much additional candidate-ranking work opportunistic replicas can support and sets a global budget. RMA uses a closed feedback loop with a central PID controller and client-side enforcement based on observed tail latency. SBM uses a corresponding loop where a central controller sets target coverage and a client-side router applies value-aware selection with an explore-and-exploit process. Multi-layer value-aware load shedding at both servers and clients reacts within milliseconds when capacity is suddenly reclaimed.

What Comes Next

Meta plans to increase workload elasticity by extending the operating range of RMA and SBM and developing new levers such as precomputation elasticity, which performs work ahead of request time. The company is also expanding model portability and cross-region inference to widen the usable capacity pool, and building an optimization-based system to automatically adjust capacity allocation across elasticity levers based on benefits to users and advertisers.

Sources: Meta Engineering