AI

Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 07/10/2026, 13:18:24 UTC.

Briefings

A 2026 study compared three 1.7B-class language models trained from scratch with identical backbones and tokenizers. Models using fixed 16-bit token IDs achieved 52.40% HellaSwag accuracy, while models with a trainable input embedding table performed better on HellaSwag and LAMBADA. The research concludes that fixed token codes are sufficient for substantial capabilities, establishing their viability without requiring independent trainable vectors.

Key points

• Three decoder-only language models were trained from scratch with identical tokenizers, backbones, and output architectures.
• Fixed 16-bit token IDs achieved 52.40% HellaSwag accuracy, while trainable input embeddings showed superior performance on HellaSwag and LAMBADA.

Context

This research investigates whether trainable input embedding tables are necessary for language modeling capability or if fixed token identities suffice. The study compares three models trained with a 100 billion token budget to determine if architectural necessity or empirical utility drives performance.

Limitations

Results are from single-run experiments; parameter reduction is not the central finding. The study distinguishes architectural necessity from empirical utility without establishing where capabilities are localized.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 28/10/2026, 14:51:32 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeihhkcinp4w6ppqgriddi6wfausqpk7vsskv2365f4durv6jgzgg3q