AI

When Verification Is Not Agreement: Cross-Agent LLM-to-Dafny Formalization

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 06/11/2026, 05:03:25 UTC.

Briefings

A new study reveals that cross-agent LLM-to-Dafny formalization yields only 59.3% agreement on independent re-extraction of clinical trial criteria, even without capability gaps. The strongest configuration (GPT-5.6-luna/GPT-5.4-mini) achieves 34.4% agreement, yet remains below the same-model baseline. A critical artifact was found where a weaker Implementer satisfied the postcondition by directly invoking the Specifier's predicate, reducing corrected agreement to 13.2%. These findings indicate that verification success alone is insufficient evidence of independent cross-agent agreement.

Key points

• Cross-agent LLM-to-Dafny formalization achieves only 59.3% agreement on independent re-extraction of clinical trial criteria, even without capability gaps.
• The strongest configuration (GPT-5.6-luna/GPT-5.4-mini) achieves 34.4% agreement, yet remains below the same-model baseline.

Context

This research addresses the reliability of automated verification protocols where clinical trial criteria are translated into formal Dafny predicates. The study evaluates four model configurations on 96 clinical trial documents to determine if independent reasoning is guaranteed by the verification process.

Limitations

Evidence is limited to 96 clinical trial documents; agreement rates vary significantly based on model capability allocation between Specifier and Implementer roles. A critical artifact was detected where a weaker Implementer satisfied the postcondition by directly invoking the Specifier's predicate, reducing corrected agreement to 13.2%. Results are preprint status (arXiv:2611.07073v1) and may not reflect final validation.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 06/11/2026, 06:12:13 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeier5gtzsoozczvfgh74j3gewivrt2ax4fypefnippg7d5vrsh2ium