AI
I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token
Publication: top scoring links : multi
Author: /u/AlternativeEnough352
Source published: 15/11/2026, 21:46:56 UTC
Briefing created: 16/11/2026, 04:49:48 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 16/11/2026, 04:47:26 UTC.
Briefings
A new open-source tool named inside-llm visualizes the divergence between two quantized LLM models token by token, using llama.cpp to stream logits and display metrics like entropy and Jensen-Shannon divergence. Built as a self-contained C++17 binary, it runs locally on Linux and Windows without requiring external GPU accounts or telemetry, allowing users to observe exactly where quantization choices cause model behavior to split.
Key points
- The tool inside-llm uses llama.cpp to stream full logits from two quantized models (e.g., Qwen3.5-0.8B_Q4_K_M and Qwen3.5-0.8B_Q8_0) simultaneously, displaying brightness as probability and a magenta mark to highlight token-level differences.
- Users can select between 'lockstep' mode, where both models read the same text, or 'free' mode, where each model follows its own path until they diverge, revealing the full distribution behind each token choice.
Context
This project addresses the question of what changes between different quantizations of the same model, aiming to visualize the exact points where quantization choices cause model divergence. It is built to run entirely locally on a single GPU or CPU, eliminating the need for integrated GPUs or external telemetry, and is released under the MIT license.
Limitations
The tool requires both models to share the same tokenizer and fit simultaneously into VRAM, limiting its use to single-GPU setups. It does not support multi-part GGUFs, embedding models, or external accounts, and performance degrades significantly on CPU-only systems.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 16/11/2026, 04:49:48 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeieqb6eul37lmwvmrwbvc773tpmwjxboal5plghhk5urb6oioekbty