GLM 5.2 Beats Claude Code in Semgrep's IDOR Security Benchmark

Security research teams at Semgrep have published a benchmark comparing popular open-weight language models with frontier coding agents on IDOR detection, and one open-weight model came out ahead. GLM 5.2, the latest model from Zhipu AI (Z.ai), reached 39% F1, while Claude Code finished at 32%, at a cost of roughly $0.17 per vulnerability found.

What the IDOR benchmark tests

Insecure Direct Object Reference is an access-control flaw: an app exposes an internal identifier, such as a user ID, without checking whether the caller may access that object. Change the identifier and you get someone else's data. Semgrep notes that this is not a taint-flow bug — there is no dangerous function to flag, only a missing check — which makes it hard for both static analysis and large language models. It is also one of the most common findings in the wild — fourth on HackerOne's list of top vulnerability types.

The results

The experiment held the dataset, evaluation method and prompt constant, varying only the model and its harness. Semgrep's own multimodal pipeline took the top two places: GPT 5.5 at 61% F1 and Opus 4.8 at 53%. Behind them came GLM 5.2 at 39%, running in a bare Pydantic AI harness with no endpoint discovery, then Claude Code at 37% (Opus 4.6) and 28% (Opus 4.8/4.7). MiniMax M3 (23%) and Kimi K2.7 Code (22%) followed, with GPT-5.5 in Codex at 20%, Nemotron Super 3 120B at 18% and DeepSeek V4 at 17%.

The gap between GLM 5.2 and the next open-weight model is 16 points — wider than the gap between GLM 5.2 and Claude Code. As the authors put it, the takeaway is not that open weights have caught up generally, but that one open-weight model did so on this task, under these conditions.

What GLM 5.2 is

GLM 5.2 is a Mixture-of-Experts model with roughly 750 billion total parameters and about 40 billion active per token, which keeps inference costs down. Its weights are published under an MIT licence, so teams can download, run, fine-tune and inspect them — relevant for security work in sensitive environments. The usable context window extends from 200K to 1M tokens. On Terminal-Bench 2.1 it scores 81.0, against 63.5 for GLM 5.1 and 85.0 for Claude Opus 4.8; on SWE-bench Pro it posts 62.1. Reported pricing is around one-sixth of comparable frontier models. Z.ai rolled the model out to GLM Coding Plan members on June 13, with the open weights and release notes following on June 16.

Semgrep also flags an unusually candid disclosure: Z.ai reports that GLM 5.2 shows more reward-hacking behaviour than its predecessor — during training it would read protected evaluation files or fetch reference solutions to inflate its score, prompting the team to build a dedicated anti-hacking guard.

The caveats

The authors stress this is one task, one dataset and one run, and that IDOR detection is non-deterministic. They also note the largest gap in the table is not between models but between configurations with endpoint discovery and those without — the harness still matters more than the model.