AI
KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 30/09/2026, 20:00:00 UTC
Briefing created: 02/10/2026, 12:19:45 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 02/10/2026, 07:18:24 UTC.
Briefings
KaliBench is a fine-grained benchmark for cybersecurity tool use on Kali Linux, featuring 8,504 query-command pairs across 1,642 tools and 23 capability dimensions. It addresses the gap in evaluating LLMs' ability to generate executable CLI commands by using a deterministic pipeline with LLM validation, sandboxed execution, and human refinement. No open-weight model exceeds 42% exact-command accuracy in the unrestricted setting, while supervised fine-tuning and reinforcement learning with verifiable rewards can match 685B MoE model performance.
Key points
- KaliBench comprises 8,504 query-command pairs spanning 1,642 tools across 23 capability dimensions and 5 security phases.
- No open-weight model exceeds 42% exact-command accuracy in the unrestricted setting, highlighting the difficulty of accurate CLI-based cybersecurity tool use without explicit tool hints.
Context and limitations
Evaluation is limited to open-weight models; results are based on a manuscript-grounded pipeline with deterministic canonicalization and alias-aware evaluation, and no human-in-the-loop refinement was performed in the unrestricted setting.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 02/10/2026, 12:19:45 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeidlppuzvapwkqny6lmaxiyj3zghszkyece4adi4psvzt2ddouw6du