AI
Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 28/09/2026, 20:00:00 UTC
Briefing created: 30/09/2026, 03:56:02 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 30/09/2026, 02:18:24 UTC.
Briefings
RLVR methods like GRPO fail when rollouts exhaust budgets, leaving models without reward signals. GRAFT addresses this by using off-policy-aware cross-model trajectory exchange to replace failed groups with informative peers. The framework transfers successful and unsuccessful peer responses while controlling mismatch via sequence-level compatibility weighting and token-level importance clipping. Experiments on three heterogeneous model pairs and five mathematical benchmarks show GRAFT consistently outperforms GRPO, gaining 2.1 points on average and up to 4.5 points in model-level average performance. Stored peer trajectories preserve these gains, improving over GRPO by 1.8 points without simultaneous co-training.
Key points
- RLVR methods like GRPO fail when rollouts exhaust budgets, leaving models without reward signals.
- GRAFT transfers both successful and unsuccessful peer responses with peer-computed advantages.
Context and limitations
Results are limited to three heterogeneous model pairs and five mathematical reasoning benchmarks. The study uses preprint data with no author attribution or verification of peer review status.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 30/09/2026, 03:56:02 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeigh4mv72ebbq5ztkjm7wktgtk3xl55iqpo5vfoajxz6lqlnqpqr4y