AI

Unconstrained CTI-REALM with Open-Weights

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 14/10/2026, 12:19:26 UTC.

Briefings

A study on CTI-REALM benchmark evaluates security agents using open-weight versus closed frontier models. The main finding is that open-weight models can match the performance of the strongest closed models at less than 10% of their cost, provided enough turns are given. The benchmark's fixed message limit confounds results, as leading models fail at this limit, but relax it reveals open-weight models outperform closed ones. Stage-level metrics show models differ in specific workflow stages rather than overall score.

Key points

• CTI-REALM benchmark tests security agents converting threat reports into working detections via live telemetry queries.
• Twenty-six models were evaluated under a common harness, scoring on a panel of LLM judges.

Context

Organizations face a choice between open-weight models, which are cheaper and auditable, and expensive closed frontier models. This study investigates that choice on CTI-REALM, a benchmark for security agents.

Limitations

The benchmark's published limit on agent messages causes leading models to score zero, partly measuring which models finish in time. Results at this limit partly measure which models finish in time. With the limit relaxed, the models it was truncating recover, and the highest rewards belong to open-weight models. The benchmark's checkpoint scores largely record what an agent did or mentioned, not just its output quality.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 25/10/2026, 20:22:38 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeib6lxnmc7scxzlwryjzokqq2eaketgmeepzl2z6luohzihjg5g7zi