AI

RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning

Read full article at source ↗

Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 08/10/2026, 01:18:24 UTC.

Briefings

RLHND is a video foundation model that jointly estimates hand pose and tactile information from monocular egocentric videos. It replaces the Cosmos 3 diffusion backbone with a deterministic clip-level feature extractor using clean-latent conditioning. The model predicts anatomically plausible joint angles and maintains consistent hand shapes across videos via a shape parameter. A separate tactile expert stream predicts dense contact and force, achieving state-of-the-art performance on pose, contact, and force estimation benchmarks. The system enables retargeting and real-world robot experiments, with code available at seungjun-moon.github.io/rlhnd.

Key points

• RLHND uses the Cosmos 3 video diffusion backbone as a pre-trained feature extractor via clean-latent conditioning.
• The model achieves state-of-the-art performance in pose estimation, contact estimation, and force estimation across benchmark datasets.

Context

Video foundation models are increasingly used for robot policy training, but existing hand trackers lack physical cues like contact and force, limiting their accuracy. RLHND addresses this by combining pose and tactile estimation from monocular egocentric videos.

Limitations

The code is publicly available but the specific experimental setup details, such as the exact robot hardware used in real-world experiments, are not fully specified in the evidence.

Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 03/11/2026, 23:44:55 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.

View archived briefing on IPFS
bafybeiglu2f2yslri4dx2xuychqjgt66vayhy555r4ybvukkxxxyqnk2ea