AI

When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 07/10/2026, 00:18:24 UTC.

Briefings

A pre-registered study compared LLM-extracted facts against raw conversational turns using LoCoMo and LongMemEval datasets. Results show raw turns selected by a typed decision model (Jev) are non-inferior to extraction-based systems, with raw turns costing 3,061 times less to write. At generous budgets, extraction systems outperform raw turns in accuracy, suggesting why published results disagree.

Key points

• Raw conversational turns selected by a typed decision model (Jev) are non-inferior to LLM-extracted facts, with a one-sided 95% confidence bound of -3.0 points against a -5-point margin.
• At generous budgets, extraction-based systems outperform raw turns in accuracy, adding 1.5 points on LoCoMo and 1.1 points on LongMemEval when three of 30 candidates are kept.

Context

This study resolves conflicting findings in conversational memory research by testing whether raw turns or LLM-extracted facts are sufficient for agent memory. The findings matter because they clarify the trade-off between accuracy and efficiency in agent evaluation, with plans, code, and graded answers released.

Limitations

Evidence is limited to held-out LoCoMo and LongMemEval conversations; budget effects show reranking gains shrink as budgets grow. Results are pre-registered and subject to future re-analysis. No additional background context is supplied beyond the study's scope.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 24/10/2026, 15:14:35 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeibtgyl4zmdfrntxd7nt5lzyd7nxyuaozcgiqkxl65a24eap6scs2a