AI

Representation-Space MMD for Diffusion Language Models

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 06/10/2026, 07:18:24 UTC.

Briefings

A new post-training method for diffusion language models (DLMs) minimizes Maximum Mean Discrepancy (MMD) between generated and reference distributions in feature space. The approach retains contextual features at individual token positions to estimate MMD efficiently, enabling optimization via policy gradients for discrete models or direct differentiation for continuous models. Experiments on OpenWebText and GSM8K show lower perplexity with comparable entropy, while 16B DMax-LLaDA2.0 models with hybrid masked-uniform diffusion achieve higher decoding parallelism on math and code benchmarks.

Key points

• The method minimizes Maximum Mean Discrepancy (MMD) between generated and reference distributions in feature space.
• Contextual features are retained at individual token positions to estimate MMD efficiently without full sampling trajectories.

Context

This research introduces a post-training technique for diffusion language models that leverages feature space minimization to improve generation quality and efficiency.

Limitations

MMD estimation relies on retaining contextual features at individual token positions, which may limit generalization across different token positions. Results are limited to OpenWebText and GSM8K benchmarks, and 16B DMax-LLaDA2.0 models with hybrid masked-uniform diffusion were tested on math and code benchmarks.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 19/10/2026, 00:54:41 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiak5t3avstzvrdl44nigiggnjrtolx7inv37dhqenivcsn7ukfw6e