AI
Running decision model locally on an RTX 4090 to find out which one is the fastest
Publication: top scoring links : multi
Author: /u/Fun-Meaning-6474
Source published: 08/10/2026, 17:04:32 UTC
Briefing created: 17/11/2026, 08:48:25 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 09/10/2026, 13:47:26 UTC.
Briefings
A local decision model benchmark tested four open-source models (Laya, D1, Clef-Flash, Lev) on an RTX 4090 for reading 9,534 centipede names. Laya emerged as the fastest at 3.9 ms per word, while Lev was 13x slower at 68 ms despite higher accuracy. The study highlights Laya's suitability for fine-tuning and notes that Lev's performance is CPU-sensitive due to its separate PyTorch server.
Key points
- Laya achieved the fastest inference time at 3.9 ms per word on an RTX 4090, significantly outperforming D1 (6.0 ms) and Clef-Flash (24.4 ms).
- Lev demonstrated superior accuracy at 98.9% but suffered 13x slower latency (68 ms) compared to Laya, indicating high CPU sensitivity.
Context
This benchmark compared four open decision models (Laya, D1, Clef-Flash, Lev) on a single RTX 4090 for a specific task: reading 9,534 centipede names and flagging names. The results show Laya is the fastest model overall, making it a recommended choice for setup, while Lev offers better accuracy at the cost of significantly higher latency.
Limitations
The study covers only four specific models on a single GPU for one specific task. Results are limited by the specific hardware (RTX 4090) and task complexity (9,534 words). Lev's performance is CPU-sensitive and may vary on other hardware. The benchmark was conducted in 2026, and the source data includes a link to a Reddit thread where the test was submitted.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 17/11/2026, 08:48:25 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeihbztlytqkqwv2ppv4xxfgtoccw5tusz3e3it757gyqd7e5cks4ve