Getting the Source Right, Not Just the Fact: Source-Aware Verification for MCP Agents

Tool-using LLM agents no longer read from a single passage: through the Model Context Protocol (MCP) they call several tools and weave their outputs into one answer.

The Multiverse Computing team published ProvenanceGuard on 29 September.

Supported somewhere is not supported by the right source

Existing checkers, from RAGAS faithfulness to MiniCheck, AlignScore and SummaC, ask whether a claim is supported by the evidence once it has been pooled together.

The authors call this failure mode cross-source conflation: a claim that is true somewhere in the evidence but attributed to the wrong source. A support answer may credit a 30-day refund window to the account record when the policy document is what supports it.

What ProvenanceGuard does

ProvenanceGuard is a post-generation verification layer on top of a black-box MCP agent. Without retraining it, it reads the captured MCP trace with its tool outputs and source IDs. It then does five things in sequence: break the answer into claims, route each claim to the most relevant source, check support, compare it with the source the answer names, and decide whether to allow or block.

For the experiments the team used local models: MiniLM finds the relevant source, a DeBERTa-based NLI verifier checks support, and a local language model splits the answer into claims. A number or date absent from the source cannot pass.

Results

The system was tested on 281 real traces from a medical agent that used patient records, research articles and other tools. Experts checked 361 claims from 40 held-out answers: they said 139 should not pass, and ProvenanceGuard caught 138. It also held 67 supported claims for review, and picked the right source about 86% of the time.

It scored the highest reject/block F1, 0.802, ahead of MiniCheck (0.783), RAGAS Faithfulness (0.758), AlignScore (0.662) and SummaC-ZS (0.436).

In a harder test with similar sources it reached 0.846 F1 on blocking but named the exact source in only 50.3% of claims. A controlled test changed the named source in 50 cases while leaving the evidence intact; all 50 swaps were caught. A RARR-style repair loop resolved all 173 blocked answers, 144 of them with fallback text. The same approach also appears in NVIDIA's NVFlow finance-agent pipeline.