Disagreement among frontier LLMs on real-world fact-checks

Lenz, a fact-checking platform, has published a study on how often leading frontier language models disagree about real-world fact-checks. The company gave 1,000 recent claims submitted by users to five top models — GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro with Search, and Perplexity's Sonar Pro — and asked each to pick one of four verdicts: True, Mostly True, Misleading or False.

Two-thirds of claims split the panel

On 67% of claims (672 of 1,000; 95% confidence interval 64-70%), at least one model dissented from the majority verdict or no majority formed at all. A further 34% of claims (343) showed a gap of two or more buckets between the most disagreeing pair — a substantive disagreement about the answer rather than a shift in confidence.

Agreement between the models was nontrivial but limited: Krippendorff's alpha on the ordinal scale came out at 0.639. The panel converges on clear-cut verdicts and fractures in the middle of the rubric: among the 328 claims where all five models agreed, only four were unanimous on Misleading and none on Mostly True.

Where the models line up

The closest pair was Gemini 3 Pro and its search-augmented version, agreeing on 75% of claims — unsurprising, the authors note, since they share a base model. The loosest pairs, at 53%, were Claude Opus 4.7 against both Gemini variants and Gemini 3 Pro against Sonar Pro. Agreement with the rest of the panel ranged from 69% to 81%.

Because only one verdict bucket can be correct per claim, the study also gives a lower bound on error. Assuming the panel's most popular bucket is right — the most charitable reading — at least one model was wrong on 67% of claims, at least two on 45%, and at least three on 13%, where no bucket reached a majority.

What the study does not say

Lenz stresses that a majority of models is not ground truth: the majority is sometimes wrong and a dissenting model is sometimes right, and it uses the majority only as a structural reference point. The platform's own verdicts were not part of the analysis. The corpus excludes private claims, staff submissions, near-duplicates and cases where any model failed to produce a parseable verdict.

The snapshot is frozen at version 1.0, with data as of May 21, 2026, and the per-claim CSV is public. A follow-up study will human-label every claim and use those labels as ground truth to map where frontier panels and Lenz itself diverge from a human consensus.