H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos

We present H2R-Grounder, a paired-data-free paradigm that translates third-person human-object interaction videos into frame-aligned robot manipulation videos. H2R-Grounder decomposes interaction videos into a shared representation H2Rep of (i) an abstract pose sequence capturing hand/gripper motion and (ii) a background video encoding scene and object dynamics. Trained solely on unpaired robot videos, our in-context video generation model produces physically grounded robot arms that follow human intent while preserving background consistency and realistic contact dynamics. Extensive experiments on DexYCB and diverse Internet HOI videos demonstrate state-of-the-art motion fidelity and cross-domain generalization without requiring any human-robot paired supervision.