AI
When Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to Guess
Publication: cs.AI updates on arXiv.org
Author: Salma Roshdy Aly, Hussein Assaf, Ziad Kobti
Source published: 28/09/2026, 04:00:00 UTC
Briefing created: 28/09/2026, 04:15:08 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 28/09/2026, 04:03:24 UTC.
Briefings
A new framework named MARCH measures code judging accuracy without using ground truth labels. By analyzing internal pipeline logs, researchers found that MARCH often declares two code solutions equally good even when one is clearly incorrect, achieving 4.4% accuracy on benchmarks where direct model evaluation reaches 43.7%. This behavior stems from the judge failing to distinguish between candidates when evidence is insufficient, effectively 'giving up' on comparisons it cannot verify. The study highlights a critical gap in multi-agent verification systems that rely on retrieved documents, as these systems fail to meet the requirement of having independent evidence for each judgment.
Key points
- The MARCH framework measures code judging accuracy by analyzing internal pipeline logs rather than using ground truth labels, revealing it often declares two code solutions equally good even when one is clearly incorrect.
- MARCH achieves 4.4% accuracy on benchmarks where direct model evaluation reaches 43.7%, demonstrating that the judge fails to distinguish between candidates when evidence is insufficient.
Context and limitations
The study relies on internal pipeline logs rather than external ground truth labels, limiting the ability to verify the accuracy of the judge's verdicts. The research is based on a single preprint version (arXiv:2609.30328v1) and covers only the MARCH framework's performance on two code judging benchmarks.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 28/09/2026, 04:15:08 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeidr3vtf6tfx62pazbwtm26o7zbowbicztoq65nc4e7q6y5idlcdvq