AI

ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 09/10/2026, 05:18:24 UTC.

Briefings

ReSPO addresses gradient starvation in off-policy RLVR by replacing clipping with a two-branch sequence-level kernel. This method preserves positive gradients for under-generated responses while suppressing over-generated negative ones. Experiments on dense and MoE Qwen3 models show ReSPO accelerates early optimization, improves final training scores, and achieves higher held-out benchmark performance under rollout reuse.

Key points

• ReSPO replaces clipping with a two-branch sequence-level kernel derived from an alpha-divergence variational objective and an exponential variance-control tilt.
• ReSPO accelerates early optimization and improves final training scores on dense and MoE Qwen3 models under rollout reuse.

Context

RLVR frequently reuses rollouts across multiple policy updates, causing a mismatch between the current policy and the data-generating policy. This mismatch leads to a sign-dependent gradient starvation problem in clipped policy optimization, where under-generated positive responses are suppressed and over-generated negative responses dominate.

Limitations

Evidence is limited to findings on dense and MoE Qwen3 models under rollout reuse. The study does not report results on other architectures, hardware configurations, or specific benchmark datasets beyond the mentioned models.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 14/11/2026, 13:33:06 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeigy2guktnristu4ccyaqu3tjlwdymu63l5rv77cdbn43l5imv5jau