CloudOpsBench is a new open-source benchmark designed to measure how well AI agents carry out common DevOps work in the cloud. Its 1.0 release contains 45 tasks on AWS, covering jobs such as repairing services, removing resources, limiting access, deploying updates and decommissioning projects. The benchmark was developed by experienced DevOps engineers.
Even the strongest models tested struggled with many of the tasks. Claude Fable 5.1 completed 60 percent, while GPT-6 Astra completed 53 percent. The team led by Ang Chen at the University of Michigan attributes most failures to reasoning problems rather than gaps in knowledge: about 80 percent involved agents misjudging a task’s scope, failing to inspect all relevant services or stopping before the work was complete.
Around 20 percent of errors came from incorrect actions, some of which could have irreversible effects on infrastructure. Tasks run identically in a live AWS environment and an emulator. Support for GCP, Azure and OCI is planned.
The project is available on GitHub under the Apache 2.0 license, alongside a detailed analysis of failure patterns.
Sources: The Decoder.