AI
FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 27/09/2026, 20:00:00 UTC
Briefing created: 30/09/2026, 02:51:48 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 29/09/2026, 14:18:24 UTC.
Briefings
FactorEngram improves language model performance by replacing monolithic n-gram embeddings with factorized basis-level gating. This method retrieves sparsity-regularized coefficients from a shared dictionary of basis vectors, allowing related patterns to reuse common components and enabling context to modulate individual memory components. Experiments on 340M and 1B-parameter Transformers show performance gains, with ablation studies confirming that inserting the memory branch before the attention sublayer in middle layers is effective.
Key points
- FactorEngram improves language modeling and downstream task performance on 340M- and 1B-parameter Transformer backbones by replacing monolithic n-gram embeddings with factorized basis-level gating.
- The method retrieves sparsity-regularized coefficients from a shared dictionary of basis vectors, allowing related patterns to reuse common components and enabling context to modulate individual memory components.
Context and limitations
The study covers only 340M- and 1B-parameter Transformer backbones; specific results for other architectures are not reported.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/09/2026, 02:51:48 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeihtl6fx4zi2rujy5ryi4wpsizfjfynkj3yulkszwnjy3ohm2n67di