AI

Q-Learning with Scalar Adjoint Matching

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 08/10/2026, 13:18:24 UTC.

Briefings

This article presents Q-learning with Scalar Adjoint Matching (SQAM), a method that improves off-policy reinforcement learning by using a scalar adjoint to update flow policies. SQAM combines a scalar adjoint with a value penalty at policy-generated actions, achieving significant gains on four hardest OGBench domains compared to the strongest baseline. The method was tested on a real bimanual robot, where SQAM outperformed supervised fine-tuning on all three tasks.

Key points

• SQAM combines a scalar adjoint with a value penalty at policy-generated actions to improve off-policy reinforcement learning.
• SQAM achieves gains exceeding 18 to 35 percentage points on four hardest OGBench domains compared to the strongest baseline.

Context

Flow policies capture diverse action distributions, but fine-tuning them with off-policy RL is non-trivial due to the computational cost of per-step vector-Jacobian products. Adjoint matching offers a principled update method but requires significant computation. This study derives a closed-form scalar adjoint that eliminates per-step vector-Jacobian products, enabling efficient value gradient propagation from the final action back to each flow step.

Limitations

The study's findings are limited to four hardest OGBench domains and a real bimanual robot. The method's performance on other domains or with different policy sizes remains untested. The gains are specific to the scalar adjoint formulation and may not generalize to all policy architectures or environments.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 08/11/2026, 13:51:17 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeibxw56m67drogoqoaknrmk3aqpd64hffye4uvjuvdownxbw4k5gji