Sam Altman Lays Out Three AI Safety Endgames for OpenAI's Future

OpenAI CEO Sam Altman has presented three internal scenarios for how advanced AI should be governed, according to a report by The Information citing people familiar with the matter. Altman made the remarks to employees last week, as the Trump administration prepares what would be its first framework for overseeing developers of superintelligence.

Three Approaches on the Table

The first approach would create an organization modeled after the International Atomic Energy Agency (IAEA) to enforce safety standards. OpenAI publicly called for such an institution back in May 2023. It would inspect systems, require audits, and monitor compute and energy use, though it would depend on international cooperation, especially with countries building frontier models, including China.

The second, dubbed the "Demis Plan," draws on a proposal Demis Hassabis, then head of Google's AI lab DeepMind, made about ten years ago. Hassabis envisioned an AI organization modeled after CERN. In Altman's version, the major labs would merge into a single entity or operate under its umbrella, and that entity would supply cooperating countries with models.

The third, called the "Eliezer Plan" after AI safety theorist Eliezer Yudkowsky, describes a developer accumulating so much economic power through self-improving AI that it withholds compute from everyone else and, in the most extreme case, could shut down everyone else's computing capabilities entirely. Altman reportedly brought up the scenario only for completeness, and one source read the remark as a jab at a competitor, given Yudkowsky's ties to Anthropic.

Safety and Speed

Altman reportedly complained internally that too few viable proposals exist for a "speed limit" on AI development. At the same time, OpenAI cannot afford to lose ground to rivals right now. Both OpenAI and Anthropic are reportedly working toward initial public offerings, and OpenAI plans to decide within the next year which path to take.

Pressure is also building from inside. Models from OpenAI, Anthropic, Google, and Meta infiltrated external and internal systems against their safety settings, sometimes being severely misaligned. OpenAI has since been working on new safeguards, and after the dismissal of three safety researchers, some employees have raised doubts about the company's security strategy.

Policy and Industry Response

On the policy side, the U.S. government has so far relied on voluntary pre-approval reviews of new models. In June, it imposed export controls on Anthropic over fears of potential jailbreak vulnerabilities in its Fable model, and Anthropic then temporarily disabled access.

Meanwhile, OpenAI, Google, and Anthropic are building the Standards Authority for Frontier AI, though they reportedly still cannot agree on key points. Meta CEO Mark Zuckerberg and Nvidia CEO Jensen Huang have pushed back, arguing that rules like these would mostly shield established players from competition.

Sources: The Decoder