Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning
Latent action methods differ in where a pretrained prior enters the pipeline. O²LaF initializes both the inverse and the forward dynamics from the same pretrained video DiT, and adapts each to its own role.
No pretrained prior. Both dynamics are trained from scratch.
The prior defines the feature space. The IDM is trained on top of frozen features.
A video prior on forward dynamics. The IDM is still trained from scratch.
A video prior as temporal supervision: it aligns z from outside, while both dynamics are trained from scratch.
Both sides start from the same pretrained Video DiT (adapted with two LoRAs).
Noise works as a mask. On the IDM it hides appearance. On the FDM it hides the answer, so the prediction has to come from the action. Drag both sliders: each pair below is a separately trained model, scored by a frozen linear probe.
Clean frames. Every appearance detail is visible.
Near pure noise. The FDM has to trust the action.
Pixel-space noise shown for illustration; models add noise to VAE latents.
Same reference video, same target first frame, same generation settings. Only the latent actions differ.
Press a key to play one latent action. Actions read from first-person, third-person, robot and game video all live in one latent space.
Each action query qi reads the video tokens of two adjacent frames, xi and xi+1, in a single softmax; its final state hi is read out as the latent action zi. Below are two raw attention heads per clip: no averaging over heads or blocks, no debiasing. The heat stays on what changes between the two frames, such as the character, the hands or the robot arm, and the background stays dark.
Our framework is prior-agnostic on the inverse side as well. Swapping the IDM to V-JEPA 2 probes slightly better, yet its actions leak source color and light into the target scene—showing that a strong probe alone is not enough.
Without action labels during pretraining, the learned dynamics and latent space generalize directly to downstream perception and policy evaluation.
The self-supervised IDM learns latent actions from unlabeled video. Can it provide a better foundation for fine-grained motion understanding? Train a small MLP head on them with CameraBench labels, and it tags camera motion far better than a 72B vision-language model.
Frontier AI models are quickly becoming capable robot agents, increasingly built and tested in simulators. But a simulator only approximates the real scene, and real-robot trials are slow and costly. Our FDM offers a cheap preview across this gap: starting from a real photo, it imagines what a candidate action sequence will do in the real scene, before real-world deployment.
Scripted grasps on a GR-1, run open loop in MuJoCo. Simulator outcome: the run shown, plus 20 replays with perturbed friction, mass and object position.
Read down a column: one action replayed in four different worlds. Read across a row: four actions, all starting from the same first frame.
No. Both dynamics are trained on raw video. Labels appear only in evaluation and applications.
Yes. We test ST-Transformer, VideoChat3, V-JEPA 2 and Wan2.2 as the IDM. V-JEPA 2 probes best (88.62% vs 87.95%), but carries source color and light into new scenes, whereas Wan2.2 keeps the target scene. Our guess: starting both sides from the same prior gives them aligned internal representations, so the latent actions fit the FDM more naturally and carry less appearance.
Adapting the prior matters more than size. A frozen Video DiT with only a trained head does much worse than LoRA, and rank-16 LoRA already comes close to rank 128. Keeping a 1.3B IDM fixed and growing only the FDM from 1.3B to 14B still improves decodability, so stronger predictive supervision helps on its own.
At mid noise the noisy target still shows most of the answer, so the FDM can predict the future without using the action. Near pure noise it has to rely on z. In our sweep the best point sits around tFDM = 0.98; pure noise is slightly worse.
Autoregression sees clean past frames, and bidirectional attention sees noisy future frames. Both are shortcuts around the action. Causal joint prediction removes both and probes best: 87.95% vs 83.71% (bidirectional) and 70.13% (autoregressive) on first-person MIND.
@article{jiang2026o2laf,
title = {One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning},
author = {Jiang, Yuxin and Sun, Wenqiang and Li, Jiaqi and Gu, Yuchao and Zhang, Lvmin and Tsang, Ivor and Shou, Mike Zheng},
journal = {arXiv preprint arXiv:2611.26617},
year = {2026}
}