NVIDIA Introduces NodeWright, a Kubernetes-Native Package Manager for Node Fleets

NVIDIA has introduced NodeWright, an open-source, Kubernetes-native package manager that configures and updates the host operating systems of GPU clusters without disrupting running workloads. It already runs in production at NVIDIA as Skyhook and is part of DSX OS.

From scripts to declarative node management

Teams usually manage nodes with Ansible playbooks and manual runbooks. That breaks down when a kernel upgrade disables RDMA or a CVE must be fixed across hundreds of machines. GPU hardware is scarce and long training jobs cannot be rescheduled, so the real task is changing an entire fleet without killing the training run.

How NodeWright works

Packages are Kubernetes Custom Resources, so they deploy through kubectl, Helm or Argo CD. On each target node the operator runs a fixed sequence: cordon, wait for non-interruptible pods, drain the rest, apply and configure the package, then uncordon — restarting or rebooting only when needed.

Packages can tune sysctl and GRUB parameters, install security agents and remediate CVEs. NodeWright tracks every package's version per node and orders execution by declared dependencies; a CVE package can detect that a vulnerable kernel module is still loaded and mark itself failed.

Safe rollouts at scale

The DeploymentPolicy resource defines compartments — labeled groups of nodes with their own disruption budgets and rollout strategies: fixed batch sizes, linear growth (1, 2, 3…) and exponential growth (1, 2, 4, 8…). A batch threshold sets the success rate needed to advance, and a failure threshold can stop a compartment after repeated failures.

When something goes wrong the update halts instead of cascading: errors appear in the status, failed jobs are marked, and affected nodes get labels for triage through the Kubernetes API. New nodes can be required to join with a taint removed only after all packages and checks pass.

Ecosystem and availability

NodeWright integrates with NVIDIA AI Cluster Runtime (AICR), which publishes version-locked recipes of known-good configurations for Helm, Argo CD or Flux, and works alongside NVCRE and NVSentinel. Licensed under Apache 2.0, it installs with Helm from an OCI artifact; tuning packages cover Hopper and Blackwell GPUs, Google Kubernetes Engine nodes, and Amazon EKS clusters with EFA drivers.