AI
CARAT: Do Materials LLMs Reason or Recite?
Publication: cs.AI updates on arXiv.org
Author: Jiajun Wu, Jian Yang, Zixiang Ni, Zhenzhu Li, Bin Chong
Source published: 01/10/2026, 04:00:00 UTC
Briefing created: 01/10/2026, 04:15:36 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 01/10/2026, 04:03:24 UTC.
Briefings
The CARAT benchmark investigates whether materials Large Language Models reason about crystal structures or simply recite answers from their input. Using a fixed question and gold answer across eight views, the study reveals that GraphSpace outperforms plain periodic graphs by 19.3 points, driven by 1.96 points from rendering and 46.7 points from omitting irrelevant fields. The authors also retrained their own benchmark to address a flaw where models quote structural links but fail to use them, achieving 99.8% accuracy after matching supervision. This work highlights that accuracy alone cannot distinguish between reasoning and rote memorization.
Key points
- GraphSpace outperforms a plain periodic graph by 19.3 points on the hardest benchmark families, with 46.7 points attributed to omitting irrelevant fields from the input.
- A specific rule causing the benchmark flaw skips structural links and reads lists directly, achieving 99.8% accuracy after matching supervision.
Context and limitations
The benchmark's hardest families are limited to eight matched views, and the study relies on a single preprint with no experimental validation or replication.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 01/10/2026, 04:15:36 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeib7nxitu6wffcyzsq2keqs2qubsntt53vsera3t2eyeosjkhidusq