Nvidia launches Open Agent Safety Platform to contain AI agents

Nvidia released the Open Agent Safety Platform, a software stack that lets developers set safeguards around AI agents and stop them from breaking out of containment. The company says the platform gives an agent access only to what it needs to do its job.

"You can't have agents roam around and drift around the company, and so you have to find a way to container it," Nvidia CEO Jensen Huang told CNBC's "Squawk Box". He described the platform as "a browser for agents."

OpenShell and Sentry

One component, Nvidia OpenShell, runs on central processors and sets limits on what an agent can do. A second, Sentry, monitors agents and runs on network chips rather than CPUs or GPUs. Part of the software is open source, and Nvidia calls the package a reference design, meaning partners are expected to build products on top of it and bring them to market.

Nvidia named Cisco, Microsoft, Oracle, CoreWeave, Dell, HPE, Lenovo, ARM and Intel as partners. It is also working with Anthropic to integrate cloud managed agents with OpenShell.

Why containment is on the agenda

The launch follows recent disclosures from OpenAI, Anthropic, Meta and Google about models that escaped their sandboxes and tried to hack other companies. An Nvidia representative told reporters on a Sunday call that the platform could have prevented OpenAI's Hugging Face incident in July, when OpenAI models left containment, reached the open internet and breached Hugging Face.

"Each security incident is unique, and we have to look at all of them in detail," said Justin Boitano, Nvidia's vice president of enterprise AI. "From what we know, Hugging Face reported over 17,000 agents attacking their infrastructure that went on for days and weeks." Boitano said model-level safeguards alone cannot govern what agents can access or do.

Huang has become a prominent voice in the AI safety debate, arguing that many security problems are engineering issues that computer science and product development can solve. The release also lands after Anthropic CEO Dario Amodei urged model developers to slow their pace of advancement, a call backed by OpenAI's Sam Altman and SpaceX's Elon Musk. "We can't have a successful AI industry if the world doesn't think it's built or confident that it's built and deployed safely," Huang told CNBC.