AI
EasyPPO: Stabilizing the Critic Is Key
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 28/09/2026, 20:00:00 UTC
Briefing created: 30/09/2026, 03:23:11 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 30/09/2026, 03:18:25 UTC.
Briefings
Research reveals that the learned critic in Proximal Policy Optimization (PPO) for large language models is a primary source of instability, specifically through truncated rollout filtering and heterogeneous return noise. The proposed EasyPPO method addresses these failures by normalizing critic regression weights with inverse standard deviations of sampled returns and using smaller critic mini-batches. Experiments on FrontierCS, AIME24, and Search-R1 demonstrate that EasyPPO maintains stability across the full training horizon while outperforming vanilla PPO, VAPO, and HL-Gauss PPO by 14.89%, 2.28%, and 9.47% respectively.
Key points
- The critic in PPO for LLMs is a major source of instability due to two failure modes: filtering truncated rollouts shifts the policy objective to reward completion, and heterogeneous return noise causes high-variance prompts to dominate updates in finite batches.
- EasyPPO mitigates these issues by normalizing critic regression weights with the inverse standard deviation of sampled returns and using moderately smaller critic mini-batches to confine outlier influence during gradient clipping.
Context and limitations
Evidence covers continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24, and multi-turn search on Search-R1. Results are from a preprint with uncertainty regarding generalizability to other reward coding schemes or hardware configurations.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/09/2026, 03:23:11 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeicgjhgizsc25jiorrezspvxcduac5dbyuhuhu3wyroimmdl2kvqxe