NVIDIA Brings TensorRT Multi-Device Inference Into Dynamo-Triton

NVIDIA has folded its TensorRT multi-device inference capability into Dynamo-Triton, so an application no longer has to coordinate GPU ranks itself.

What changed

TensorRT multi-device inference lets a single TensorRT network execute across several GPUs using NCCL-backed distributed collectives while retaining TensorRT's inference optimizations. Dynamo-Triton, formerly Triton Inference Server, turns the capability on in release 26.07, and one Triton KIND_MODEL instance can now own multiple GPUs: it creates per-rank execution contexts, CUDA streams and NCCL communicators, then launches the ranks together for each request from a single gRPC call.

How the serving path works

NVIDIA demonstrated the integration with Cosmos 3 Nano video generation, where Diffusers keeps orchestrating the surrounding pipeline while Dynamo-Triton serves the 36-layer denoising transformer. TensorRT multi-device inference uses Ulysses context parallelism to distribute 44,160 video tokens across as many as eight GPUs.

The distributed graph is compiled into each TensorRT plan before deployment, so Triton activates a context-parallel plan rather than converting a single-device engine. The transformer accounts for 93.4% of single-GPU generation time, and since each of the 35 denoising steps needs two predictions for classifier-free guidance, the Diffusers proxy issues 70 transformer RPCs per generation.

Latency results

All four variants ran on the same eight-GPU system at 1280x720, 189 frames at 24 FPS, with five measured generations. End-to-end latency fell from 156.595 seconds on one GPU to 87.999 on two, 53.093 on four and 34.183 seconds on eight, a 78.17% reduction. Transformer RPC time dropped 83.59%, its share of the total fell from 93.4% to 70.2%, and time outside that path stayed between 10.2 and 10.5 seconds in every configuration.

Validation compared each context-parallel output with the single-device result at the same seed: CP2 and CP4 measured a mean absolute error of 12.759 and PSNR 21.111 dB, CP8 MAE 16.316 and PSNR 19.400 dB, all inside the configured thresholds. NVIDIA does not claim the outputs are pixel-identical.

Trade-offs

Teams still have to weigh extra GPU resources against shorter latency. The benchmark does not measure concurrent request throughput, cost per generated video or total cost of ownership, which teams should check against internal service-level objectives. Dynamo-Triton 26.07 is available from NGC.