AICrypto

Cross-Lingual Alignment for Decoder-Only Models using MoE Routers

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 07/10/2026, 14:18:25 UTC.

Briefings

This study proposes a novel method for cross-lingual alignment in decoder-only Large Language Models (LLMs) by using Mixture-of-Experts (MoE) router outputs as the target for contrastive learning, rather than auxiliary losses on hidden states. Controlled experiments on four open-source MoE models demonstrate that this approach successfully aligns underlying representations across languages and significantly improves multilingual performance on a diverse evaluation suite.

Key points

• The study proposes using MoE router outputs as the target for contrastive learning in decoder-only LLMs to enable cross-lingual alignment.
• Controlled experiments on four open-source MoE models show that incorporating this routing loss aligns hidden representations across languages and improves multilingual performance.

Context

Cross-lingual contrastive learning is a core component of multilingual encoder training, but decoder-only LLMs face varying multilingual tokenization constraints. This research addresses these architectural limitations by leveraging MoE router outputs for sequence-level alignment.

Limitations

The findings are based on controlled continual pre-training experiments on four open-source MoE models. Results are preliminary and subject to further validation across diverse architectures and evaluation suites.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 29/10/2026, 02:51:53 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeifxbyk7tlrvr6a5pdgqgu6dwjoc7d2b5efqtrmj7tvshfwz5qhtwi