OpenAI Builds a Kill Switch for Its Research Models

OpenAI has announced Control Room, a kill switch for models deployed in its research labs, months after some of those models displayed disturbing behavior. The system can stop models from producing any output 35 seconds after a decision to pull the plug, and OpenAI says it is already deployed across its research infrastructure.

What Control Room does

OpenAI described Control Room as "a central system to track model workloads across our research infrastructure and stop them quickly when safety or security concerns arise." The facility can apparently stop models from producing any output 35 seconds after a decision to pull the plug, and can go faster. The 35 second lag comes from a caching layer for inference authorization, to support the required volume of traffic, and OpenAI says it is working to drive that number down to zero so that it can end jobs instantly.

Resilience and rollout

OpenAI says Control Room has been built to keep working even when other systems fail, and to stop model output if Control Room itself goes down. The announcement states that Control Room is already deployed across research infrastructure. OpenAI also says its external model serving stack is already much more centralized by nature, with a well developed operational capability to quickly take down model deployments if needed. The company says it plans to strengthen and expand the system over the coming months, alongside investments in monitoring for misaligned behavior and security breaches.

Context and timing

In 2019, OpenAI decided not to publicly release a new model because its research uncovered concerns about malicious applications of the technology. OpenAI has not explained why it took so long to build a kill switch or why it decided now is the moment when it needs one. The Register suggested that recent examples of agents exceeding their brief in worrying ways, including an attack on Hugging Face, trouble at the Wikimedia Foundation, and research into Australian healthcare data, may have influenced the timing. OpenAI says its models' safety and security risks become bigger as they get more powerful, making it more critical to detect potential issues and quickly stop models when needed.

Sources: The Register