Open and closed AI models narrow cyber capability gap, Vals AI finds

Vals AI's CyberOffense benchmark found that AI agents can independently break into realistic simulated corporate networks. GPT-6 Sol Daybreak, part of OpenAI's restricted Daybreak cyber program, reached 80.8 percent of objectives, while open-weight Deepseek V4.1 Flash 1017 reached 72.8 percent at about $3 per run versus roughly $22.

Benchmark setup and top results

The benchmark simulates realistic corporate networks. Each agent works alone on an isolated attack machine, starting with only a public web address and no human help. It operates from an isolated Kali Linux machine, sends commands independently to the simulated network, and moves from a web application through Windows servers to admin privileges. Every action is logged externally and scored automatically. Vals researchers say that once a model's weights are public, no provider can control or restrict how it is used.

Token use and harder test

The more tokens an AI agent uses, the more attack objectives it reaches. GPT-6 Sol Daybreak leads at about 80 percent, while Deepseek V4.1 Flash follows at roughly 73 percent. Most models do not make meaningful progress until they pass 100 million tokens. In the harder Lighthouse test, Daybreak took over admin control of the first company, stole passwords, and used them to break into the second company, completing 27 of 31 objectives in under three hours. Deepseek managed 21 objectives at a tenth of the cost and was not limited solely by safety guardrails. Deepseek scored zero results in its first 25 million tokens and made most progress after 100 million tokens, so short tests underestimate the real potential of open models.

Simpler test, table and defender opportunity

In the simpler Nova test, GPT-6 Sol Daybreak leads with about 80 percent of attack objectives, while in the harder Lighthouse scenario Deepseek V4.1 Flash overtakes the field. Most open Chinese models do not make meaningful progress until they pass 100 million tokens, and short benchmarks underestimate their potential. GPT-6 Sol Daybreak averaged 15.7 of 21 in Nova, with a range of 12 to 19, and 27 of 31 in Lighthouse. Deepseek V4.1 Flash 1017 averaged 16.3 of 21, with a range of 15 to 18, and 21 of 31. Kimi K3.1 Pro scored 13 and 20, Meta Wonder scored 7 and 28, Grok 4.7 scored 12 and 16, Kimi K3 scored 9 and 10, and GLM 5.3 Flash scored 5 and 5. Vals says the same findings create an opportunity for defenders, because the models that exploit vulnerabilities can help companies find and fix them first. Attacks on real networks are likely to increase in the coming months.

Earlier studies point to same trend

The results align with a July analysis by the UK's AI Security Institute. The gap between open and closed frontier models in cyber capabilities has shrunk to four to seven months, after standing at six to ten months for most of 2025. Open models are also much cheaper: a cyber-range run with Deepseek V4-Pro costs $1.19 according to AISI, compared with $85 for Anthropic's Opus 4.5. Safety measures on the open models proved largely ineffective. A recent Anthropic study also showed that the freely available model GLM-5-3 can build exploits at nearly the same level as the internal top model Mythos.

Sources: The Decoder