AI

A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 20/10/2026, 04:03:25 UTC.

Briefings

A new theoretical result establishes that language models forget their prompts via a power-law decay of hidden state distance, with the decay rate determined by prompt length and depth. The study identifies a 'conditional no-hair theorem' where prompt influence vanishes as generation tokens accumulate, meaning surviving prompt properties must be encoded in the generated text itself. Key findings include specific decay rates for trained (0.55-0.7) versus untrained models (0.2-0.5), a mathematical bound of O(P(log t)^(L-1)/t), and a geometric mechanism involving 'relay' tokens. Uncertainty remains regarding the exact solvability of the underlying linear relay model and the precise conditions for the no-hair property.

Key points

• Trained autoregressive transformers exhibit a power-law decay of hidden state distance from prompts, with decay rates ranging from 0.55 to 0.7 for trained models and 0.2 to 0.5 for untrained models.
• The theoretical bound for prompt influence decay is O(P(log t)^(L-1)/t), where P is prompt length, L is depth, and t is the number of generated tokens, proving the decay cannot be improved.

Context

This research addresses how language models retain or forget their initial prompts during generation, revealing that prompt influence decays geometrically as context length increases. Understanding this mechanism is crucial for improving prompt retention in large language models and analyzing their memory limitations.

Limitations

The study relies on preprint data with limited computational resources and does not account for all transformer architectures. The exact solvability of the linear relay model remains unproven, and the precise conditions for the no-hair property are not fully established.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 20/10/2026, 04:33:50 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeifhmatt6ejp3jwkpsw53bzlvarhb5jkaxeg7gnhcy7ldqpbuth2d4