World models for robots and cars: simulator, evaluator, and policy

You have 5 checkpoints of a cup-pushing policy and want to know which is best. Fifty real trials each makes 250 trials, and someone has to reset the cup and make sure it does not crash off the table. Could a world model "act it out" first? In robotics and autonomous driving, world models do three different jobs: generate training data, score policies, and act as the policy. Each job needs different validation; as an evaluator, a high correlation with real success rates still does not guarantee you pick the best policy.

One model, three jobs

The Cosmos paper (NVIDIA, January 2025) writes a world foundation model $\mathcal W$ as predicting the next state from past observations and a current "perturbation" (Fig. 3).

$$ \hat x_{t+1}=\mathcal W(x_{0:t},\,c_t) $$

$x_{0:t}$ are the observations up to time $t$, usually video frames; $c_t$ is the current perturbation, which can be a robot action, an ego-vehicle trajectory or a piece of text. The same predictor used in different ways does different jobs:

WorldArena (Shang et al., February 2026 paper) tested 14 embodied world models in exactly these three uses, data engine, policy evaluator and planner, and concluded in the abstract that "high visual quality does not necessarily translate into strong embodied task capability"; the authors call it a perception–functionality gap. World-in-World (October 2025, ICLR 2026 Oral) also reports that closed-loop task success depends more on controllability than on visual quality.

Job one: generating data

The abstract of DreamGen (NVIDIA GEAR et al., May 2025 paper) describes the pipeline: adapt an image-to-video model to the target robot, then generate videos of familiar or new tasks in varied scenes; the videos have no action labels, so pseudo-actions are inferred with a latent action model or an inverse dynamics model (IDM). This is the same idea as Genie's latent actions in the interactive video note. The paper calls this synthetic data neural trajectories, and reports that a humanoid learned 22 new behaviours from teleoperation data for a single pick-and-place task in a single environment.

The cost counts too. The DreamGen paper reports that its 240k-sample RoboCasa set took 54 hours on 1,500 L40 GPUs, which is $1500\times54=81{,}000$ GPU-hours, or about $81{,}000/240{,}000\approx0.34$ GPU-hours per sample, roughly 20 minutes of GPU time per sample. Whether a real demonstration is expensive has to be judged against that figure.

NVIDIA's Cosmos is the most complete open series on this line. The table lists only what we could confirm in public sources; all dates are NVIDIA's release dates:

NVIDIA's own two statements of Cosmos 3's size disagree: the developer blog says Nano 16B and Super 64B, while the Hugging Face collection lists Super as 65B and Edge as 4B. NVIDIA's 2026-05-31 press release claims first place among open models on several leaderboards including Physics-IQ and PAI-Bench, which is vendor-reported; on the community Physics-IQ Verified leaderboard, read on 2026-10-15, Cosmos3-Super (multiframe) scores 50.8 ± 2.2.

Another cost data point: in the V-JEPA 2 paper (June 2025), Meta placed Cosmos as the world model inside the same CEM planner, and it needed 4 minutes per action with 0% pick-and-place success, against 16 seconds per action for V-JEPA 2-AC. This is Meta's measurement and only shows that the Cosmos used in that experiment did not suit the core of such a planner; it says nothing against Cosmos as a data engine. Details are in the JEPA note.

Job two: evaluating policies

Estimating a new policy's performance without deploying it is called off-policy evaluation in RL; see RL from a fixed dataset. World models offer another route: let the policy act inside the model and count successes. Below are several works that use a world model as an evaluator; where numbers exist, each was measured by its authors on policies and tasks they chose:

We read the Veo evaluator paper itself. It fine-tunes the Veo 2 text-to-video model on robotics data and evaluates policies on the ALOHA 2 bimanual platform (4 cameras, §2, §3.1); the abstract reports more than 1,600 real-world evaluations covering 8 checkpoints and 5 tasks. The Pearson 0.86 and MMRV 0.06 refer to the single strongest policy, comparing predicted and real success rates across the OOD generalisation axes of background, small distractor, large distractor and object (§4.1); the text does not state the number of points, so count them in Fig. 8. Limitations the paper states: predicted absolute success rates are lower than real ones (§3.2, §4.1); contact-rich interactions, especially with small objects, are hard to simulate (§7); each rollout is 8 seconds, and minute-scale multi-view consistent generation remains a goal (§7); success is currently judged by humans (§7).

The Veo evaluator paper defines MMRV in §3.2 as follows (the metric comes from Li et al. 2024):

$$ \text{MMRV}=\frac1n\sum_{i=1}^n\max_j\;|R_i-R_j|\cdot\mathbb 1\big[(\hat R_i<\hat R_j)\neq(R_i<R_j)\big] $$

$R_i$ is policy $i$'s real success rate and $\hat R_i$ the world model's predicted rate. Each policy records only "the opponent it is misordered with that has the largest real gap", then the result is averaged over policies. Zero means the orderings agree completely.

Compute it once: correlation 0.98, and the top pick is still wrong (a teaching example)

Five cup-pushing policies, with invented numbers:

• Pearson: $\bar R=0.46$, $\bar{\hat R}=0.32$; the cross term $\sum(R-\bar R)(\hat R-\bar{\hat R})=0.239$ and the two sums of squares are 0.347 and 0.173, so $r=0.239/\sqrt{0.347\times0.173}\approx0.98$.
• Spearman: only D and E swap ranks, the sum of squared rank differences is $\sum d^2=2$, and $\rho=1-6\sum d^2/(n(n^2-1))=1-12/120=0.9$.
• MMRV: only the pair (D, E) is misordered, with a real gap of 0.10; D and E each record 0.10 and the rest record 0, so $\text{MMRV}=0.2/5=0.04$.

All three metrics look good, yet the best policy by the model is D, while on the real robot E is 10 percentage points higher. The correlation is held up mainly by the large gap between weak policies like A and B and the good ones. Look only at the three you actually choose among, C, D and E, and Spearman becomes $1-6\times2/(3\times8)=0.5$.

Two more points:

• Absolute values. Pearson is unchanged under $\hat R=aR+b$ with $a>0$: a model that reports every success rate at half its true value has $r=1$. The model in the table also underestimates across the board, just as the Veo evaluator paper says of itself. If you need the evaluator to decide whether a policy clears a 70% launch bar, correlation does not help.
• The real side is noisy too. With only 50 trials per policy, the standard error of the difference between D and E is about $\sqrt{0.7\times0.3/50+0.8\times0.2/50}\approx0.086$. A gap of 0.10 is only about 1.2 standard errors, so even the real ranking is shaky. For comparing two policies, see what a score increase actually tells you; for what agreement, correlation and calibration each measure, see the judge calibration note.

The policy under evaluation will exploit the evaluator

The model-based RL note showed that the harder you plan, the more likely you are to find the model's flaws. Evaluation is the same. Pick the checkpoint with the highest world-model score out of 20, and the winner tends to be the one the model overestimates most; go further and use the world-model score as a training reward, and the policy will actively seek actions that the model judges successful but that fail on the real robot. On 2026-01-12, 1X itself wrote that its generated videos are over-optimistic about task completion. When validating an evaluator, keep at least one set of policies that never took part in selection, and spot-check on real robots regularly.

Job three: acting as the policy (world action models)

The idea is to first "imagine" inside the model what the next few seconds should look like, then read actions off the imagined frames; or to have the model generate video and actions together.

Route A (1X NEO): current view + task → 14B video model generates future video → inverse dynamics model reads off actions
Route B (DreamZero, Cosmos Policy): current view + task → one model jointly generates future frames and actions

• DreamZero (NVIDIA et al., 2026-02-17 paper): a 14B autoregressive video-diffusion "World Action Model" running closed-loop control at 7 Hz. The abstract reports more than 2× the generalisation of state-of-the-art VLAs to new tasks and environments on real robots; 10–20 minutes of video-only demonstrations from other robots or humans improve unseen-task performance by over 42% (relative); adapting to a new embodiment takes 30 minutes of play data. On 2026-03-16 NVIDIA said GR00T N2 builds on DreamZero research (vendor-reported).
• Cosmos Policy (NVIDIA and Stanford, 2026-01-22 paper): post-trains Cosmos-Predict2-2B into a policy, encoding actions, future states and values as latent frames. In simulation it averages 98.5% on LIBERO and 67.1% on RoboCasa; on a real ALOHA its average score is 93.6, above π0.5's 88.6; out of distribution, π0.5 is better (92.5 vs 89.3).
• 1X world model as NEO's policy (1X, 2026-01-12 blog): a 14B video backbone mid-trained on 900 hours of egocentric human video and fine-tuned on 70 hours of NEO data; generating 5 seconds of video takes 11 seconds, plus about 1 second for the IDM. Pull-tissue success is 30% with one generation and 45% with 8 parallel generations. Limitations 1X lists itself: generated videos are over-optimistic about task completion, 3D grounding is weak, and evaluation was not blind.
• Runway Praxis-1 (2026-09-30 announcement): announced as an open-weight world action model; as of 2026-10-15 we could check only the announcement.

Now the timing: 7 Hz means about $1000/7\approx143$ ms per control step. 1X needs 11 seconds to generate 5 seconds of future, $11/5=2.2$ times slower than real time, and about 12 seconds including the IDM before a stretch of actions is ready. Our inference: this suits slow tasks that can wait, not scenes that need an immediate reaction. The Cosmos Policy comparison is a reminder that "video models as policies generalise better" depends on conditions: it wins in distribution and loses out of distribution. Separately, Physical Intelligence mentioned on 2026-04-16, introducing π0.7, that it uses a "lightweight world model" to generate visual subgoals, without details.

Driving: GAIA and the Waymo World Model

In driving, first separate open loop from closed loop. Open loop replays recorded sensor data and looks at what the policy would do at each frame; once the policy drives differently from the recording, the following frames no longer match. Closed loop lets the policy's actions drive the world model's next sensor frames, and the policy decides again from the new view; only then can you see what happens after a deviation.

As of 2026-10-15, we found no published correlation with real outcomes for GAIA-3/4, the Waymo World Model or the 1X evaluator. Tesla described its own "neural world simulator" for closed-loop evaluation in talks at ICCV 2025 and CVPR 2026 (talk video; content taken from the video's auto-generated captions), but has published no paper and no comparison with real outcomes; see the end-to-end note in the driving series.

Risk checklist

What to watch on this line next

1. Do evaluators hold across robots, tasks and policy families? The Veo evaluator (Google DeepMind, December 2025) was validated on one platform (ALOHA 2) with 8 checkpoints; WorldEval (May 2025) and dWorldEval (April 2026) each on settings chosen by their authors. A result that would change the picture: correlation, MMRV and "was the top policy picked correctly" reported together across several embodiments and held-out policy families.
2. Will driving world models publish fidelity and closed-loop validation? Neither Waymo (February 2026) nor Wayve's GAIA-4 (August 2026) published a correlation with real outcomes. A result that would change the picture: public comparisons of crash or takeover rates in simulation with real-road data.
3. When do world action models beat VLAs, and when not? Cosmos Policy (January 2026) wins in distribution and loses to π0.5 out of distribution; DreamZero (February 2026) reports more than 2× generalisation; Runway announced open weights for Praxis-1 on 2026-09-30. A result that would change the picture: third-party comparisons on the same real-robot tasks with the same amount of data.
4. Can latency and cost drop to closed-loop use? DreamZero self-reports 7 Hz; 1X self-reports generation 2.2 times slower than real time. A result that would change the picture: real-time operation on stated hardware without a drop in success rate.

Next, read do they actually understand the world?; to revisit how world models are used for planning, go back to model-based RL.