← bryceag11.github.io

What world action models do with the futures they imagine

Bryce GrantOctober 20, 20269 min read

A world action model (WAM) is a robot policy that learns to predict the future, usually as video, along with its actions. The usual explanation for their gains is that predicting the future teaches the policy how the world works. We tested that explanation on released checkpoints in simulation, where every object's pose is known, with two questions. Does the action use the future the model imagines at test time? And do the policy's features hold the future beyond what the present state already determines?

Joint video-action denoising. Cosmos Policy denoises its predicted frames and action tokens together. The rollouts come from LingBot-VA on RoboTwin 2.0, whose action also attends to its predicted frames. From the same initial state, the unmasked run puts the box in the basket and the run with the predicted frames masked from the action fails (52 of 60 unmasked episodes succeed, 0 of 60 masked). Intermediate frames show the decoded frame under noise scaled to each call's σ.

In seven of the eight WAMs whose action can read a predicted future, masking that future cuts success far below a current-frame mask, on LIBERO-10, RoboTwin 2.0 or CALVIN. Swapping that future changes which object the robot grabs first, on LIBERO-10 and on RoboTwin 2.0. On LIBERO-10, once we control for the present state and the robot's own motion, the features of six policies hold at most 0.003 more R2 for a target's future displacement.

Where the future meets the action

Zhang et al.1 sort WAMs into four paradigms, and we add latent foresight for two of our models.

FamilyHow the action sees the futureModels we tested
Joint video-action generationAction tokens attend to the frames the same network is generating Cosmos Policy2, DreamZero3, Motus4 (diffusion) and LingBot-VA5 (autoregressive)
Latent foresightAn expert predicts a future latent or feature state, and the action expert attends to it InternVLA-A16, LaWAM7
Video features to a policy headOne denoising pass of a video model, read by a separate policy VPP8
Auxiliary prediction onlyFuture prediction shapes training, and the deployed action never sees a predicted future Fast-WAM9, VLA-JEPA10

The wiring decides which tests apply. We intervened in three of the four families, on LIBERO11, RoboTwin 2.012 and CALVIN13, and scored gaze heads on BEHAVIOR-1K and DROID as well.

Reading attention misleads twice

In manipulation the object that moves next is the one in or beside the gripper, so the gaze-head score of Gandikota and Bau14 selects heads that follow the hand: distance to the gripper alone finds the next object at AUC .979 to 1.000 in all ten LIBERO policies we scored15,16,17, and the score correlates with a held-object score at Spearman .90 to 1.00. We call the heads it picks carry heads. RIFT18 masks the whole predicted-frame cache of four WAMs and finds that success drops. A whole-cache mask names no layer or head and says nothing about what the read carries.

Four interventions on the read

We call the action tokens' attention to the predicted frames the action's read of the future. A controlled gaze score counts a head as anticipatory only on frames where the next object is far from the gripper, against a distance-matched null. Size-matched blinding masks the read in every layer and head and, as the control, the same number of current-frame tokens. Restricted masks take one band of layers or k heads at a time. A key/value swap runs a second pass under a counterfactual instruction and copies only its predicted frames' keys and values into the action pass. Figure 1 shows the read in three models, with five Cosmos Policy tracker heads overlaid, the carry heads whose attention inside the predicted frames best follows the gripper.

Camera, start of chunkCurrent camera frame
Imagined by the policyImagined future frame
Camera, start of next chunkCamera frame at the next chunk
Figure 1. The imagined frame (middle) shows the arm where the next camera frame (right) later finds it. For Cosmos Policy, the heatmap marks where five tracker heads of the action look.

Standard gaze heads are carry heads, and a second population looks ahead

With the gripper confound controlled, every simulated policy we scored has heads that prefer the next object before the hand reaches it: 2 to 248 heads per LIBERO policy, 38 to 162 in each of six RoboTwin policies (distance-matched top AUC .91 to .97), all 48 perceiver heads of VPP on CALVIN (.73 to .82), and 120 of 144 π0.519 heads on BEHAVIOR-1K20 (top .890). On recorded DROID21 frames, 3 of 144 π0.5-DROID heads pass the distance-matched null (top .639). Outside π0.5 these heads are entangled with instruction mentions, so the interventions below use carry and tracker heads.

The action needs its predicted frames

On LIBERO-10 (50 episodes per arm), masking the read drops success from 48 to 10 in Cosmos Policy, 49 to 0 in LingBot-VA and 35 to 0 in DreamZero, and the size-matched current-frame masks leave 34, 49 and 24. On RoboTwin 2.0 (60 per arm), LingBot-VA drops from 52 to 0 against 50 for the current mask, InternVLA-A1 from 44 to 11 against 39, LaWAM from 55 to 36 against 57 and Motus from 45 to 0 against 27 (Table 1). On CALVIN, VPP falls from 4.16 to 0.01 tasks per chain, against 4.12 for the smaller current-frame mask (Figure 2).

Figure 2. On every benchmark, masking the predicted future (purple) drops success far more than a current-frame mask (gold). The other views locate the read in one band of layers and its tracker heads.

Masking a quarter of the layers at a time locates a read band in each model, and inside it the top 100 tracker heads carry most of the effect where 100 random heads do little (Figure 2, One band and Tracker heads). On RoboTwin 2.0, masking LingBot-VA's layers 7 to 14 alone gives 0 of 20, as masking every layer does, and the other bands leave 18 to 20. In DreamZero three heads are sufficient to block the read: on held-out initial states, masking their future reads leaves 1 of 50 successes, against 31 of 50 for their current-frame reads (Fisher p = 2×10−11).

ModelBenchmarkUnmaskedFuture maskedCurrent masked
LingBot-VARoboTwin 2.052/600/6050/60
InternVLA-A1RoboTwin 2.044/6011/6039/60
LaWAMRoboTwin 2.055/6036/6057/60
MotusRoboTwin 2.045/600/6027/60
VPPCALVIN ABC-D4.160.014.12

Table 1. Blinding outside LIBERO.

The read carries the plan

Key/value swap. LingBot-VA on RoboTwin 2.0, task "position the red stapler to the right of the phone". The counterfactual pass predicts the wrist camera over the phone and the action pass predicts it over the stapler. With the counterfactual keys and values in the action pass, the arm picks up the phone first and reaches the stapler only at step 210 (39 of 60 swapped episodes go to the counterfactual object first, 5 of 60 with the current-frame swap).

On two LIBERO-10 basket tasks, LingBot-VA moves the counterfactual object first in 39 of 40 episodes and DreamZero in 40 of 40. The same swap on current-frame keys and values moves it in 0 of 40 (McNemar p = 4×10−12 and 2×10−12), and the swapped runs fail the original task in all 40. Re-encoding LingBot-VA's counterfactual frames under the original instruction still redirects 39 of 40, so the imagined video carries the plan. On LIBERO-Object the swap redirects 179 of 200 Cosmos Policy episodes and 199 of 200 DreamZero. On RoboTwin 2.0, LingBot-VA grabs it first in 39 of 60 swapped episodes, against 5 of 60 with the current-frame swap and 40 of 60 under the counterfactual instruction (McNemar p = 5×10−9). Cosmos Policy is the weakest redirector, 10 of 40 on LIBERO-10. In one RoboTwin 2.0 episode, LingBot-VA's unmodified run reaches the gray mouse first and the swapped run reaches the blue bell.

UnmodifiedUnmodified rollout frame
Imagined frames swappedSteered rollout frame
Figure 3. Gripper distance to each named object in one RoboTwin 2.0 episode, unmodified (solid) and with the predicted-frame keys and values swapped (dashed). Restricted swaps counts counterfactual-first episodes when only a band of layers or a set of heads is swapped.

On LIBERO-10, the read band alone redirects 21 of 40 DreamZero and, on fresh initial states, 19 of 40 LingBot-VA episodes, against 40 for all layers; LingBot-VA's top 100 tracker heads redirect 30. No band or head set reached the 35 of 40 we set in advance as sufficient. On RoboTwin 2.0 the band alone carries the redirect: in 80 fresh episodes, LingBot-VA's layers 8 to 14 redirect 57, against 59 for all layers and 5 for layers 15 to 21 (Restricted swaps). ForeWAM22 builds a model around this key-value channel.

Stronger reads do not fix wrong plans

Without imagination, object choice is a distributed code: in VLA-JEPA, patching the outputs of its top-k carry heads from a counterfactual pass switches the first object with a 50 percent point at 32.5 heads [31.1, 34.0] over 288 held-out LIBERO-10 episodes. In a WAM the lever is the plan. Temporal Ratio23 scales language and plan guidance by how strongly the action attends to future frames (TR-AG). On Cosmos Policy, with settings frozen in-distribution, it lowers success on the compositional LIBERO Spatial-OOD suite24 from 171 to 139 of 600 paired episodes (Holm p = .002), and our tracker-head variants stay at or below base (157 and 169). Told to put the bowl on the cookie box, it imagines the bowl on the plate and the action follows.

A better plan through the same channel does help. Frames imagined during a successful rollout of the same or nearest scene, re-noised to the level Cosmos's action reads (σ ≥ 4), raise success from 171 to 182 of 240 (12 gained, 1 lost, Holm p = .007). Inserted clean, the same plans drop it to 124. Those plans were chosen with outcome knowledge. Without it, the value frame, a VLM judge and our gaze score fail to raise success, and an outcome-trained critic ranks held-out plans at Spearman +.34 [.31, .37] yet lifts best-of-8 success only from 168 to 172 of 240 (Holm p = .36): the sampler-seed candidates barely differ.

What the features hold beyond the state

The Fast-WAM9 authors suggest the main value of video prediction “may lie in improving world representations during training”. Horizon tomography tests that credit: a probe predicts a target object's displacement δ steps ahead from the activations, and we report its cross-fitted gain over a nested control ck, a conditional probe25:

$$\Delta R^2_k(\delta) \;=\; R^2\!\left(y_\delta \mid c_k,\, h\right) \;-\; R^2\!\left(y_\delta \mid c_k\right)$$

The control ladder. As the nuisance control grows from the scene to the full state to the robot's own motion, the probe's gain (purple) shrinks to within 0.003.

Here c1 models the present scene, c2 the full simulator state, and c3 adds the robot's own upcoming motion. The test has a blind spot by construction: under a fixed policy in a deterministic simulator, anything an internal dynamics model predicts is a function of the state, so c2 absorbs it.

beyond
Figure 4. Beyond the present scene the features predict a target's displacement well, and beyond the full state the bars collapse toward zero.

At δ = 16 on LIBERO-10, six policies score 0.46 to 0.51 beyond the present scene, 0.001 to 0.021 beyond the full state, and within 0.003 of zero once their own motion enters the control. π0.5, trained with no world objective, carries as much beyond the state as the pixel-objective Fast-WAM, 0.021 [0.015, 0.028] against 0.019 [0.013, 0.026]. RoboTwin 2.0 repeats the pattern for Fast-WAM and π0.5 at δ = 16 and for LingBot-VA at δ = 32, with LingBot-VA at δ = 16 undetermined (−0.036 to +0.018 across five fold seeds). Beyond the commanded motion the residual is at most 0.009 on LIBERO-Spatial and LIBERO-Goal (500 episodes). In matched LeWorldModel26 retrains on Push-T, a world objective added to behavior cloning raises the residual beyond state and motion by 0.0077 [0.0054, 0.0102], and no retrain reads hidden mass or friction.

What this means

For the models we tested, the imagined future's keys and values are a test-time handle on object choice, and plan selection is the lever: the action follows whatever plan reaches it, and a better plan raised success where a stronger read lowered it. On LIBERO-10 the ladder finds no extra world knowledge behind the gains from video training, which fits Yu's27 proof that world-action imitation and behavior cloning share their population action target, and ARPS's28 finding that future supervision helps the action under shift. Richens et al.29 prove that agents generalizing across many goals contain a world model, a premise ten fine-tuning tasks do not meet. Probes and steering have reached VLAs30,31,32,33 (our six-VLA study: paper, Action Atlas), and Molinari et al.34 find a world model in OpenVLA against a raw-embedding baseline without our controls.

Every intervention ran in simulation. The blocking heads and most restricted swaps come from LIBERO-10, where 96 to 99 percent of target motion happens under contact, and the ladder's RoboTwin replication covers three policies. Next: interventions on a physical arm.

More rollouts

  1. X. Zhang, X. Zeng, W. Zhang. From World Models to World Action Models: A Concise Tutorial for Robotics. arXiv:2607.00836, 2026.
  2. M. J. Kim, Y. Gao, T.-Y. Lin, et al. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. arXiv:2601.16163, 2026.
  3. S. Ye, Y. Ge, K. Zheng, et al. World Action Models are Zero-shot Policies (DreamZero). arXiv:2602.15922, 2026.
  4. H. Bi, H. Tan, S. Xie, et al. Motus: A Unified Latent Action World Model. CVPR, 2026.
  5. L. Li, Q. Zhang, Y. Luo, et al. Causal World Modeling for Robot Control (LingBot-VA). arXiv:2601.21998, 2026.
  6. InternVLA-A1 Team. InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation. arXiv:2601.02456, 2026.
  7. J. Chen, K. Wang, K. Chen, et al. LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies. arXiv:2606.15768, 2026.
  8. Y. Hu, Y. Guo, P. Wang, et al. Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations. ICML, 2025.
  9. T. Yuan, Z. Dong, Y. Liu, H. Zhao. Fast-WAM: Do World Action Models Need Test-time Future Imagination? arXiv:2603.16666, 2026.
  10. J. Sun, W. Zhang, Z. Qi, et al. VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model. arXiv:2602.10098, 2026.
  11. B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, P. Stone. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. NeurIPS Datasets and Benchmarks, 2023.
  12. T. Chen, Z. Chen, B. Chen, et al. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. arXiv:2506.18088, 2025.
  13. O. Mees, L. Hermann, E. Rosete-Beas, W. Burgard. CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks. IEEE RA-L, 2022.
  14. R. Gandikota, D. Bau. Gaze Heads: How VLMs Look at What They Describe. arXiv:2606.14703, 2026.
  15. NVIDIA. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv:2503.14734, 2025. We used the GR00T N1.7 checkpoint.
  16. M. J. Kim, C. Finn, P. Liang. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (OpenVLA-OFT). arXiv:2502.19645, 2025.
  17. M. Shukor, D. Aubakirova, F. Capuano, et al. SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics. arXiv:2506.01844, 2025.
  18. C. Zhang, J. Tong, X. Li, Y. Wang, H. Li. Keep the Future, Drop the Rollout: RIFT for World Action Models. arXiv:2608.11521, 2026.
  19. Physical Intelligence, K. Black, N. Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv:2504.16054, 2025.
  20. C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, et al. BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation. CoRL, 2022.
  21. A. Khazatsky, K. Pertsch, S. Nair, et al. DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. RSS, 2024.
  22. J. Huang, Z. Wu, Z. Zhang, Z. Wang, S. You, T. Huang. Foresight Without Seeing: Latent Futures for World Action Models. arXiv:2608.11605, 2026.
  23. U. A. Mishra, Y. Chen, D. Xu, Y. Liu, X. Chen, J. Mao. Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio. arXiv:2607.08127, 2026.
  24. Q. Li. VLAs are Confined yet Capable of Generalizing to Novel Instructions. arXiv:2505.03500, 2025.
  25. J. Hewitt, K. Ethayarajh, P. Liang, C. D. Manning. Conditional probing: measuring usable information beyond a baseline. EMNLP, 2021.
  26. L. Maes, Q. Le Lidec, D. Scieur, Y. LeCun, R. Balestriero. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels. arXiv:2603.19312, 2026.
  27. Y. Yu. On the Capability Separation Between World-Model Policy Learning and Imitated World-Action Models. arXiv:2608.22197, 2026.
  28. Q. Gu, J. Li, B. Gao, et al. The Right Future for Action: Learning Action-Relevant Predictive States in World Action Models. arXiv:2609.23369, 2026.
  29. J. Richens, T. Everitt, D. Abel. General agents need world models. ICML, 2025.
  30. B. Häon, K. C. Stocking, I. Chuang, C. Tomlin. Mechanistic Interpretability for Steering Vision-Language-Action Models. CoRL, 2025.
  31. H. Buurmeijer, C. Amo Alonso, A. Swann, M. Pavone. Observing and Controlling Features in Vision-Language-Action Models. arXiv:2603.05487, 2026.
  32. A. Swann, L. McGranahan, H. Buurmeijer, M. Kennedy III, M. Schwager. Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Models. arXiv:2603.19183, 2026.
  33. B. Grant, X. Zhao, P. Wang. Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models. NeurIPS 2026.
  34. M. Molinari, L. Nevali, S. Navani, O. Younis. Emergent World Representations in OpenVLA. arXiv:2509.24559, 2025.