AI

Seq@K: Evaluating Language Models Under Sequential Feedback

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 30/10/2026, 06:48:24 UTC.

Briefings

This article introduces Seq@K, a new evaluation protocol for language models that measures their ability to solve problems within K sequential attempts using feedback after each failure. Evaluated on 7 benchmarks and 16 models, the study reveals that Seq@K produces a distinct model-selection landscape compared to standard metrics like Pass@K, often favoring different configurations and altering the Pareto frontier. The research concludes that Seq@K provides a more faithful metric for iterative agentic systems and highlights the critical need for practitioners to carefully design feedback loops when benchmarking such models.

Key points

• Seq@K evaluates models by measuring their success in solving a problem within K sequential attempts, with feedback provided only after each failure.
• The study found that Seq@K reveals a materially different model-selection landscape than Pass@K, showing that optimal configurations can differ significantly between the two metrics.

Context

The research addresses a gap in evaluating language models that operate in iterative agentic workflows, where models refine answers based on feedback rather than static performance. By isolating the core capability of refinement from feedback, Seq@K offers a more accurate assessment of how models handle dynamic, multi-turn interactions.

Limitations

The evaluation covers only 7 benchmarks and 16 models, limiting generalizability. Results are based on preprint data (arXiv:2610.37859v1) and may not reflect final published findings. Uncertainty exists regarding the exact magnitude of performance differences between metrics.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/10/2026, 09:53:54 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeid27ilicflhase6lutrw6bxnsckjleigziraaxzljfmtq3rvdqfny