O2LaF

One Prior,
Two Dynamics

Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning

1Show Lab, National University of Singapore2A*STAR, Singapore3Tencent4Stanford University
Top: source video, read by the IDM Bottom: zero-shot transfer, generated by the FDM
The idea

Both sides start from the video prior.

Latent action methods differ in where a pretrained prior enters the pipeline. O²LaF initializes both the inverse and the forward dynamics from the same pretrained video DiT, and adapts each to its own role.

(a) From scratch
IDM
FDM

No pretrained prior. Both dynamics are trained from scratch.

(b) Frozen visual features
IDM
FDM
❄ DINOv3 / V-JEPAfeatures

The prior defines the feature space. The IDM is trained on top of frozen features.

(c) Pretrained FDM only
IDM
FDMvideo DiT

A video prior on forward dynamics. The IDM is still trained from scratch.

(d) External video prior
❄ V-JEPA · alignment loss
IDM
FDM

A video prior as temporal supervision: it aligns z from outside, while both dynamics are trained from scratch.

Video priors as IDM & FDM
O²LaF · ours
O²LaF-DiT
IDMVideo DiT
FDMVideo DiT
Other IDM priors, studied in Priors
IDMV-JEPA 2
IDMVideo MLLM

Both sides start from the same pretrained Video DiT (adapted with two LoRAs).

Same Video DiT prior xᵢ (clean context) xᵢ₊₁ (target frame) ∇ℒ backprop xᵢ xᵢ₊₁ IDM · LoRA_IDM causal video action token qᵢ reads {xᵢ, xᵢ₊₁} masked (v↛q) ∇ LoRA_IDM, qᵢ z ∈ ℝ³² 32-D vector FDM · LoRA_FDM causal mask + z conditioning denoise · t=0.98 ∇ LoRA_FDM x̂ᵢ₊₁ t=0.98 ℒ ‖x̂ᵢ₊₁ − xᵢ₊₁‖ flow loss

Then control what each side can see.

Noise works as a mask. On the IDM it hides appearance. On the FDM it hides the answer, so the prediction has to come from the action. Drag both sliders: each pair below is a separately trained model, scored by a frozen linear probe.

IDM inputtIDM = 0
xi
xi+1
00.30.50.70.9

Clean frames. Every appearance detail is visible.

↓ tIDMtFDM →
82.6%Action probing ↑
FDM inputtFDM = 0.98
x0 · clean context
x1 · noisy target
0.50.90.980.9951

Near pure noise. The FDM has to trust the action.

Pixel-space noise shown for illustration; models add noise to VAE latents.

Comparison

Against previous methods.

Same reference video, same target first frame, same generation settings. Only the latent actions differ.

—
—
—
—
Action library

One space for every action.

Press a key to play one latent action. Actions read from first-person, third-person, robot and game video all live in one latent space.

W
ForwardNavigation · 1st person
Interpretability · Action token

What action tokens look at.

Each action query qi reads the video tokens of two adjacent frames, xi and xi+1, in a single softmax; its final state hi is read out as the latent action zi. Below are two raw attention heads per clip: no averaging over heads or blocks, no debiasing. The heat stays on what changes between the two frames, such as the character, the hands or the robot arm, and the background stays dark.

frame i frame i+1
Attention mass per transition frame iframe i+1 —
Extension · IDM prior

How other video priors behave.

Our framework is prior-agnostic on the inverse side as well. Swapping the IDM to V-JEPA 2 probes slightly better, yet its actions leak source color and light into the target scene—showing that a strong probe alone is not enough.

Applications

Two dynamics, two tools.

Without action labels during pretraining, the learned dynamics and latent space generalize directly to downstream perception and policy evaluation.

01 · IDM · Motion annotation

IDM as a motion annotator.

The self-supervised IDM learns latent actions from unlabeled video. Can it provide a better foundation for fine-grained motion understanding? Train a small MLP head on them with CameraBench labels, and it tags camera motion far better than a 72B vision-language model.

Unlabeled video→IDM self-supervised→z / h→MLP head supervised→Dolly in · Pan left · Static …
02 · FDM · Policy verification

FDM as a policy verifier.

Frontier AI models are quickly becoming capable robot agents, increasingly built and tested in simulators. But a simulator only approximates the real scene, and real-robot trials are slow and costly. Our FDM offers a cheap preview across this gap: starting from a real photo, it imagines what a candidate action sequence will do in the real scene, before real-world deployment.

Policy actions open loop→Robot-only render movable objects removed→IDM→z→FDM from the real photo→Success?

Scripted grasps on a GR-1, run open loop in MuJoCo. Simulator outcome: the run shown, plus 20 replays with perturbed friction, mass and object position.

Gallery · Game worlds

One action, many worlds.Many actions, one world.

Read down a column: one action replayed in four different worlds. Read across a row: four actions, all starting from the same first frame.

Source action, read by the IDM Generated by the FDM
FAQ

Questions.

Does O²LaF use any action labels?

No. Both dynamics are trained on raw video. Labels appear only in evaluation and applications.

Can O²LaF use other video priors?

Yes. We test ST-Transformer, VideoChat3, V-JEPA 2 and Wan2.2 as the IDM. V-JEPA 2 probes best (88.62% vs 87.95%), but carries source color and light into new scenes, whereas Wan2.2 keeps the target scene. Our guess: starting both sides from the same prior gives them aligned internal representations, so the latent actions fit the FDM more naturally and carry less appearance.

Is the gain just a bigger model?

Adapting the prior matters more than size. A frozen Video DiT with only a trained head does much worse than LoRA, and rank-16 LoRA already comes close to rank 128. Keeping a 1.3B IDM fixed and growing only the FDM from 1.3B to 14B still improves decodability, so stronger predictive supervision helps on its own.

Why train the FDM at high noise?

At mid noise the noisy target still shows most of the answer, so the FDM can predict the future without using the action. Near pure noise it has to rely on z. In our sweep the best point sits around tFDM = 0.98; pure noise is slightly worse.

Why causal joint prediction instead of autoregression?

Autoregression sees clean past frames, and bidirectional attention sees noisy future frames. Both are shortcuts around the action. Causal joint prediction removes both and probes best: 87.95% vs 83.71% (bidirectional) and 70.13% (autoregressive) on first-person MIND.

Citation
@article{jiang2026o2laf,
  title   = {One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning},
  author  = {Jiang, Yuxin and Sun, Wenqiang and Li, Jiaqi and Gu, Yuchao and Zhang, Lvmin and Tsang, Ivor and Shou, Mike Zheng},
  journal = {arXiv preprint arXiv:2611.26617},
  year    = {2026}
}