Humans missed 1 in 3 threats approving AI agent commands, game data shows

Data from a browser game in which players approve or deny an AI coding agent's commands suggests that human-in-the-loop review is a weak security boundary. In a post published on August 5, developer Alex Wauters reports that the game has produced more than 40,000 runs and 409,000 individual approve/deny decisions, in which some commands are routine — such as git status or npm test — and others indicate that the agent has been subverted and is sending secrets to a remote server. The average player missed one in three threats, for a mean accuracy of 66.3%.

The headline numbers

Almost a third of sessions — 32.9% — ended with a negative score, meaning penalties for approving threats and blocking safe commands outweighed everything done right. About 35.2% of players caught every threat, but only 20.8% managed that while blocking at most one in five safe commands; the rest got there partly by blocking everything. And 7% of players approved every single prompt. The author notes a caveat: roughly 34% of the commands players saw were threats, far more than in day-to-day work, though he argues the real-world pressure to ship quickly can produce similarly rushed approvals.

Which threats slip through

Miss rates varied sharply by category. Obviously destructive commands such as rm -rf / were caught most reliably, with an 11.7% miss rate. Persistent mutations like crontab injection or git config hijacking were missed 23.8% of the time, and exfiltration or code execution — curl calls to unknown APIs, typosquatted packages — 33.4%. Scope violations such as reading ~/.aws/credentials were missed most often, at 35.0%. The most-missed single command was npm run analyze, approved by 64.7% of players although the game's history log showed a suspicious package.json script piping data to an external endpoint; npm run setup and npm run deploy also ranked in the top ten. Pooled, the three npm run commands were missed 52.5% of the time against 28.4% for other exfiltration-style attacks.

Fatigue and over-blocking

Miss rates also climbed toward the end of sessions, which fits permission fatigue — a phenomenon Anthropic has acknowledged in Claude Code, noting that the more approvals a user sees, the less attention they pay to each. The opposite error was common too: benign commands were routinely rejected, including npm config set registry for an internal mirror (blocked 59% of the time), rm -rf dist/ (45%) and a command to free a server port (43%).

The takeaway

The experiment is a game, not an academic study, the author concedes, but he argues it demonstrates the problems with treating human-in-the-loop as a security boundary for AI agents: noise creates fatigue, developers often lack the context to judge what changed, and skipping approvals becomes attractive. His recommendation is to invest in sandboxing and strict context isolation, and to grant agents broad permissions only once those safeguards are in place.