AI

SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 05/11/2026, 05:03:24 UTC.

Briefings

SlackBench is a new benchmark evaluating AI agents on collaborative research workflows using real GitHub and Google Docs data. It features 140 queries across 33 models, where agents must navigate multi-party conversations and evolving code states. The best system achieved 80% accuracy, while most agents scored below 66%, indicating significant room for improvement in handling complex, distributed work environments.

Key points

• SlackBench introduces a benchmark for AI agents using real GitHub and Google Docs data to evaluate performance on collaborative research workflows.
• The benchmark includes 140 queries across 33 models, with the best system scoring 80% accuracy while most agents scored below 66%.

Context

Existing benchmarks often use synthetic data or isolated tasks, failing to capture complex multi-party dynamics of real collaborative work. SlackBench addresses this by using privacy-preserving reenactments of real academic research projects, providing a framework for evaluating agents in realistic environments.

Limitations

The dataset is limited to 140 queries across 33 models, and results are based on preprint evidence from an arXiv submission. Specific model names, exact dates of project completion, and the full scope of privacy-preserving reenactments are not fully detailed in the available evidence.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 05/11/2026, 05:11:49 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiemnzxjdedhubnqx7fmj7x5o3yjezkof4kgg3ezzyt44pczikh73u