Model-based RL covered how planners use a model and how its errors compound. When you build one, the first question is often plainer: what should the model output? The full next state, the next camera frame, or only enough to get rewards and values right? What a model should predict depends on what you plan to do with it; whatever it leaves out may be impossible to recover when the task changes.
A small box-pushing task
Every choice below uses the same made-up environment. The numbers can be computed by hand and do not come from any real experiment.
A track has 8 cells, numbered 0–7. A hand sits at $h$ and a box at $b$, with the hand to the left of the box. Each step the hand moves one cell left or right ($a=-1$ or $+1$); at the left wall it stays put. When the hand moves right into the box's cell, the box is pushed one cell to the right. The step on which the box enters cell 6 earns 1 point and ends the episode.
A camera sees 24 pixels: one row of 8 pixels lights up the hand's position, one row of 8 lights up the box's position, and 8 "clouds outside the window" pixels each turn on or off with probability one half at every step. The clouds have nothing to do with the task.
The true state is just two integers $(h,b)$; the agent sees 24 pixels. The same environment supports 4 different things for a model to learn.
Predict the next state: two numbers, with a contact rule
The most direct model takes $(h,b,a)$ and outputs $(h',b')$. The true rule: $h'$ is $h+a$ (not past the left wall); $b'=b+1$ only when $a=+1$ and $h+1=b$, otherwise $b'=b$.
The model outputs only two numbers. With it you can enumerate action sequences for MPC, generate extra transitions the way Dyna does, or rescore plans directly when the reward changes. The hard part is contact: the box moves only when the hand is right next to it. That rule is discontinuous. If most training data has the hand far from the box, a network may learn "push a little when close" instead of "push only on contact" and predict positions halfway between cells.
Predict the next frame: clouds included
Without access to $(h,b)$, a model can predict all 24 pixels of the next frame from the current one. Oh et al. (2015) predicted action-conditioned future frames in Atari and reported that the predictions were useful for control.
The problem is how the loss is spent. Each cloud pixel is 0 or 1 with probability one half, so the squared-error-optimal prediction is 0.5, with expected error $0.25$. Over 8 cloud pixels:
$$ 8\times0.25=2 . $$
That error is present in every frame and cannot be removed. Now misplace the box by one cell: two pixels in the box row are wrong, for an error of $1^2+1^2=2$. Under pixel squared error, "box in the wrong cell" and "clouds unpredicted" cost the same, yet only the first one breaks planning. The grey 0.5 cloud image is also an "average frame" that never occurs in the real world, the same issue the Model-based note describes as averaging two paths into a wall.
In real images, irrelevant pixels usually far outnumber these, and the object that decides success may cover only a few dozen pixels. An observation model needs no hand-defined state, and its predictions are easy to inspect; the price is capacity and computation spent on details irrelevant to decisions, plus slower rollouts.
Predict a latent state: compress, then roll out in a small space
A middle path encodes the 24 pixels into a small vector $z$, learns $z'=g(z,a)$ and a reward predictor, and decodes back to images only when needed. PlaNet (Hafner et al., 2018) learns latent dynamics from images and plans in latent space; World Models (Ha & Schmidhuber, 2018) showed that an agent can be trained entirely inside a "dream" generated by its model and then transferred back to the real environment. The later Dreamer (Hafner et al., 2019) imagines trajectories only in latent space and learns behavior by sending gradients of learned values back through those imagined trajectories; see the Dreamer note in the world models series.
Rollouts in latent space are fast, but what $z$ keeps is decided by the training objective. If $z$ must reconstruct the current frame, it has to remember the current 8 cloud pixels, even though the next clouds are unpredictable. If $z$ only has to predict reward, it may drop information that a later task needs. The planning note already warns that a good-looking reconstruction does not guarantee that control-relevant information survived.
Predict only rewards, values, and policies: MuZero-style value equivalence
One step further: the model need not predict states or images at all. MuZero (Schrittwieser et al., 2019) has 3 parts: a representation function encodes past observations into a hidden state, a dynamics function maps a hidden state and an action to the next hidden state and an immediate reward, and a prediction function outputs a policy and a value from a hidden state. The paper states explicitly that the hidden state is not required to reconstruct observations, "nor is there any requirement for the hidden state to match the unknown, true state …" of the environment (§1).
What does this mean for box pushing? The box does not move until pushed, and once pushed it always sits just right of the hand, so the box enters cell 6 exactly when the hand first reaches cell 5. As long as the box is right of the hand and short of the goal, for any action sequence the reward time depends only on $h$, not on $b$. Take $(h,b)=(1,3)$ and $(1,4)$: moving right every step, both earn 1 point on step 4; stepping back once and then moving right, both earn it on step 6.
A model trained only for this reward may therefore map these two states to the same hidden state and ignore every cloud pixel. Nothing in the training signal asks it to tell them apart. Grimm et al. (2020) call this idea value equivalence: two models are value equivalent with respect to a set of policies and functions if they produce the same Bellman updates; the more policies and functions you include, the fewer models qualify, until only models that describe the environment completely remain.
Now change the task: each push costs 0.1 points, everything else the same (no discounting). Moving right every step, $(1,3)$ needs 3 pushes and returns $1-0.3=0.7$; $(1,4)$ needs only 2 and returns $0.8$. The state model's transitions are unchanged, so you swap in the new scoring function and replan. The value-equivalent model's reward and value heads were trained for the old task, and its hidden state may contain no $b$ at all; it needs new-task data and retraining.
How to represent transitions: MLPs, graph networks, locally linear models
Once you know what to predict, you still choose the function that represents the transition.
| Representation | How it looks for box pushing | Strength | Limitation |
|---|---|---|---|
| MLP | Concatenate $(h,b,a)$ into a vector, output $(\Delta h,\Delta b)$ | General, needs no prior structure | Must learn the contact rule from data; two boxes means redesigning the input |
| Graph network | Hand and box are nodes, relative position sits on the edge | One edge function serves every pair of objects, so a second box needs no new structure | The scene must first be split into objects |
| Locally linear | Without contact $h'=h+a,\ b'=b$; with contact, a different linear map | Plugs into solvers such as LQR and iLQR | Valid only near a stretch of trajectory where the contact mode does not change |
The graph-network example follows Sanchez-Gonzalez et al. (2018): rigid bodies become nodes, joints become edges, and the learned node and edge functions are shared across parts. The paper reports zero-shot generalization to systems with a different number of links.
A representative use of locally linear models is Levine & Abbeel (2014): fit time-varying linear dynamics near trajectories instead of learning a global model, then use those local solutions to guide a neural-network policy. This avoids global modeling, but scenes like box pushing, where the rule changes on contact, are exactly where local linearity breaks most easily.
It need not be parametric: replay, transition graphs, Gaussian processes, symbolic rules
A model does not have to be a neural network.
Replay or episodic memory. Store every observed $(s,a,r,s')$ and answer queries by lookup. In box pushing, there are only 15 states with the hand left of the box and the box left of the goal; with 2 actions that is 30 combinations. A few hundred random interactions cover them all, and in this deterministic environment the lookup model is then exact. For combinations it has not seen, it has no answer. van Hasselt et al. (2019) argue that experience replay shares key traits with model-based methods and hypothesize that when a model only generates transitions from observed states, replay-based algorithms should do at least as well; on Atari they report a replay-based algorithm exceeding the data efficiency of earlier results that used parametric models.
Transition graphs. Treat observed states as nodes and observed transitions as edges, and planning becomes a shortest-path problem: from $(1,3)$, reaching "box in cell 6" takes at least 4 edges. Search on the Replay Buffer (Eysenbach et al., 2019) uses observations in the replay buffer as nodes, estimates edge distances with a goal-conditioned value function, and runs graph search to produce a sequence of subgoals. A wrong shortcut edge is exactly what a shortest-path search will prefer.
Gaussian processes. With little data they give both a predictive mean and a variance, and the variance grows away from the data. PILCO (Deisenroth & Rasmussen, 2011) uses a Gaussian process to predict state changes, carries model uncertainty into long-horizon predictions, and computes policy gradients analytically; the paper reports learning swing-up and balancing on a real cart-pole with about 17.5 seconds of total interaction. The cost is that exact inference grows with the cube of the number of data points (Rasmussen & Williams, chapter 2), and common smooth kernels turn the box-pushing contact jump into a ramp.
Symbolic rules. Write an operator in the style of STRIPS (Fikes & Nilsson, 1971): "push right" requires the hand at $x$, the box at $x+1$, and $x+2$ empty; its effect moves the hand to $x+1$ and the box to $x+2$. If the rule is correct, it is exact for any position and any number of boxes, and needs almost no data. The difficulties: someone has to supply the symbols, or they must be learned separately, and noise such as a slipping box or continuous quantities such as friction are hard to write down.
Deterministic or stochastic
The box-pushing transitions are deterministic; the clouds are random. If the box slipped and stayed put 20% of the time, $b'$ would become a distribution, and a point prediction would land between the two outcomes. How to predict distributions and how to separate aleatoric from epistemic uncertainty is covered in the Model-based note and not repeated here. Worth noting: the dynamics function in the original MuZero paper is deterministic, and the paper leaves stochastic transitions to future work.
Side by side
The table records qualitative trade-offs from experience, not measurements; actual results depend on the task, model size, and amount of data.
| Model | Data efficiency | Uncertainty | Scale | Planning it suits | Typical failure |
|---|---|---|---|---|---|
| State MLP (ensemble optional) | Medium; needs data near contact | None from one network; ensembles approximate it | Needs an existing low-dimensional state | MPC / CEM, Dyna, gradients when differentiable | Smooths discontinuities; planner exploits extrapolation errors |
| Graph network | Higher when the structure fits | Same as above | Grows with the number of objects | Same as above, with a variable number of objects | Wrong object split, or relations not covered |
| Locally linear | Very high near a trajectory | Usually paired with a noise covariance | Covers only a stretch of trajectory | LQR / iLQR family | Fails when the contact mode changes |
| Observation model | Low; must learn every pixel | Must be modeled explicitly | Slower as pixels grow | Sampling-based planning in pixel space | Loss spent on irrelevant detail; blurry average frames |
| Latent state (with reconstruction) | Medium | Can use stochastic latent variables | Fast rollouts | CEM in latent space, training in imagination | Drops control-relevant information, or keeps too much irrelevant information |
| Value-equivalent (MuZero-style) | Depends on reward density | Deterministic in the original paper | Pairs with large-scale search | Tree search | Retraining when the task changes; hidden state hard to inspect |
| Replay / episodic memory | Covers only what was seen | None | Memory grows linearly with data | Dyna-style replay | No answer for unseen state-action pairs |
| Transition graph | Covers only what was seen | Edge distances may be wrong | Search grows costly with many nodes | Graph search, subgoals | Wrong shortcut edges are used first |
| Gaussian process | High with little data | Posterior variance | Exact inference grows cubically with data | Policy search with analytic gradients | Smooth kernels blur jumps; too slow with much data |
| Symbolic rules | Almost no data when the rules are correct | Usually none | Handles many objects | Classical symbolic planning | Where symbols come from; noise and continuous quantities are hard to express |
Choosing a starting point
First ask what the model must do at deployment. If you only want more training data, replay plus a state model with short rollouts may be enough. If you search at every step, rollout speed and error compounding matter. If the reward changes often, keep a state model that can rescore plans rather than learning only values. Reach for an observation or latent-state model only when you have pixels and the state is hard to define by hand.
As of October 2026, how far learned world models trained on large-scale video or interaction data can replace hand-built or per-task models is still changing quickly, and which latent representation suits control best is not settled. The site tracks that in the world models series; this note covers the trade-offs that are already fairly stable.
Next: how planning and learning feed each other. Return to the Model-based overview, or read planning and LQR.