CloudOpsBench finds reasoning is the biggest bottleneck for AI in cloud operations

CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud. Version 1.0 includes 45 tasks on AWS and was built by experienced DevOps engineers. Even the best models fail at many of the tasks, with Claude Fable 5.1 solving 60 percent and GPT-6 Astra managing 53 percent.

What the benchmark tests

CloudOpsBench covers tasks such as fixing services, removing resources, restricting access rights, rolling out updates, and decommissioning projects. All tasks run identically in the real AWS cloud and in an emulator. Extensions for GCP, Azure, and OCI are planned.

Why models fail

According to the team led by Ang Chen at the University of Michigan, the main problem is not a lack of knowledge but reasoning. About 80 percent of failures happen because agents misjudge the scope of a task, do not check all the relevant services, or stop too early. Roughly 20 percent of errors are wrong actions, some with irreversible consequences for the infrastructure. No model completes every task.

Availability

The project on GitHub is licensed under Apache 2.0, and a detailed failure analysis is also available.

Sources: The Decoder