AI

Not another tok/s post: local quants vs Claude on 9 real bugs

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 02/11/2026, 15:47:26 UTC.

Briefings

A researcher tested nine real-world regression bugs using local LLMs and Claude models on a custom C++/CUDA engine. Results showed local models scored higher on bug fixes (0.74 vs 0.58) but performed worse on tokens per task (1.60 vs 1.03). Quantization (3-bit vs NVFP4) improved speed but increased token generation time. The study highlights that local debugging often yields better code fixes but slower generation compared to API models.

Key points

• Local LLMs scored 0.74 tokens per task against Claude Opus 5.5 API on 9 real regression bugs.
• Local models generated 1.60 tokens per task while Claude Opus 5.5 API generated 1.03 tokens per task.

Context

The study compares local LLMs and Claude models on 9 real regression bugs from a custom C++/CUDA engine. Results show local models fix bugs better but generate more tokens per task, suggesting a trade-off between code correctness and generation efficiency.

Limitations

Data limited to 9 tasks with 3 runs each; results reflect specific local engine and harness conditions rather than general performance. Quantization effects masked by generation speed differences. No full edit-build-test loop included, only code review and patching.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 03/11/2026, 07:38:54 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeib4rsphiyjd3ltawqxsvfni3lyo4rkyvjjfokkklgqddxcglc34mu