PART 18 / Research

World models & prediction

When does predicting the environment improve action selection?

Guided illustrated lesson

What you will learn

Train a small predictive model and identify where rollout errors invalidate its plans.

Before you begin: Reinforcement learning + SNNs; Memory: biological & artificial

START WITH A QUESTION

Picture this.

Before moving a chair through a doorway, you imagine whether it will fit. A world model serves a related computational purpose: predict consequences before committing to an action. The prediction can be useful without being a complete replica of reality.

Rollout
A sequence of predicted or simulated future states.
Latent state
An internal representation used instead of directly observed variables.
Model bias
A systematic mismatch between a model's predictions and the process it models.

Read for the idea first. Work through the numbers next. Try answering the questions before opening the explanations.

Predict a task-relevant future

A model may predict position, an image, a reward or a compact learned representation called a latent state. The right target depends on the decision. Our scalar example predicts x_next from current x and action u using a*x + b*u. The coefficients are chosen, not learned from data. This is enough to demonstrate planning: try candidate actions inside the model, score the predicted outcomes and choose a proposal. Keep the internal prediction distinct from what the environment actually does.

Use a model to choose, then observe

Let x = 1, a = 0.8 and b = 0.5. Candidate actions −0.2, 0 and +0.2 predict states 0.7, 0.8 and 0.9. If the target is 1, the last prediction has the smallest squared distance, so the planner proposes +0.2. If the observed next state is 1.1 instead, prediction error is 0.2. We now have two measurements: how well the model predicted and how close the actual outcome was to the target. They answer different questions.

A little error can become a large story

Rollouts feed predictions back as future inputs. A biased model can drift over many imagined steps, and a planner can choose actions that exploit the model's mistakes. Replanning after fresh observations can limit how long the system relies on an uncorrected forecast. Evaluate the horizon relevant to your decision, including actions the planner actually selects. A low average one-step error is encouraging, but it is not proof of reliable planning in unusual situations.

Connect research without collapsing distinctions

Learned world-model research, including Dreamer, studies how predictions can support behavior learning. Predictive coding and active inference are related topics in theoretical neuroscience, but their assumptions and claims are not interchangeable with every engineering predictor. For this book's learning path, start with the concrete loop: observe, predict candidate consequences, choose, then compare prediction with reality. A fast reflex may run alongside a slower planner when waiting for a rollout would miss the task deadline.

SEE THE RELATIONSHIPS

From structure to behavior.

Hybrid: a labelled conceptual diagram
Figure 18.1 · Hybrid. Original educational diagram; synthetic, not a measured result.
The reasoning pathObserve state then Predict consequences then Choose action then Compare with reality1. Observe state2. Predict consequences3. Choose action4. Compare with reality
A learning flow for this part. Follow the assumptions through to an observable result.
A model ranks three actions, then its chosen prediction is compared with a separate observed outcome.
Planning uses predictions; evaluation uses consequences. Keeping those values separate exposes model error instead of accidentally assuming it away.

MAKE THE IDEA PRECISE

The mathematics.

x_hat_next = a x + b u; squared_error = (x_observed - x_hat_next)²
Symbols and units
SymbolMeaning
xCurrent scalar state
uAction input
a, bChosen model coefficients
x_hat_nextPredicted next state
x_observedMeasured next state on the same scale

Build the equation, step by step

  1. Use the current state and action to produce a next-state prediction. A scalar linear model assumes fixed coefficients and a simple state representation.
  2. Subtract prediction from the observed next state. Squaring removes the sign and emphasizes larger errors.
  3. For a position state, squared error has units of distance squared. Mixing position, angle and velocity errors in one loss requires explicit scaling. Good one-step error does not guarantee good long-horizon planning.

WORK THROUGH THE NUMBERS

Worked example

With x = 1, u = 0.2, a = 0.8 and b = 0.5, prediction is 0.9. An observed value of 1.1 produces error 0.2 and squared error 0.04. The coefficients are illustrative, not fitted data.

A small Python experiment.

Python 3 · Standard library only · Synthetic educational example. Run this in your own notebook or a local file.

x, target = 1.0, 1.0
actions = [-0.2, 0.0, 0.2]
predictions = [0.8*x + 0.5*u for u in actions]
best = min(range(len(actions)), key=lambda i: (predictions[i]-target)**2)
best_action = actions[best]
prediction = predictions[best]
observed = 1.1  # Separate synthetic observation, not the model output
print(f"action: {best_action:.1f}; predicted state: {prediction:.2f}")
print(f"prediction error squared: {(observed-prediction)**2:.2f}")

Read the code as a story

The code evaluates three actions using the same model and target. min selects the smallest predicted squared target error. It then compares the chosen prediction with a separate illustrative observation. That observation is not generated by the model, so the mismatch is visible. This is one-step planning, not a trained world model.

Expected output

action: 0.2; predicted state: 0.90
prediction error squared: 0.04

Change one quantity, predict the result, and then run the code again. Explain any difference before adding complexity.

Test your understanding.

Try it: Roll the model forward for ten steps without corrections. Explain how an incorrect coefficient can accumulate error.

Self-check: Why can a good predictor fail during planning?

The planner may choose actions outside the training distribution or exploit systematic prediction errors. Small errors can compound over many steps. Evaluate rollouts under the policy that will actually use the model, including uncertainty and constraint violations.

Where this model stops: Small average prediction error does not guarantee safe decisions on rare events.

A research question: When should a fast reactive controller override a model-based plan?

Record evidence in your mastery worksheet →

PREDICT / TRY / EXPLAIN

Separate prediction quality from task quality

Use paper, a calculator or the Python example. Write your prediction before checking the result.

  1. Calculate the three candidate predictions and squared distances to target 1.
  2. For the chosen action, compare prediction 0.9 with observation 1.1. Calculate both prediction error squared and actual target error squared.
  3. Change b in the model while keeping the observation rule separate. Explain why editing both together would hide model error.
Compare with a worked solution

Predicted target errors are 0.09, 0.04 and 0.01, so +0.2 is selected. Prediction error squared is (1.1−0.9)² = 0.04; actual target error squared is (1.1−1)² = 0.01. A model can be inaccurate on a particular transition while the action happens to end near the target. Repeated matched trials are needed to assess usefulness.

CLOSE YOUR NOTES

Can you explain it back?

These questions test the reasoning, not just a remembered phrase. Say your answer out loud or write a sentence first.

01Why is a generated future not automatically a reliable plan?

Reveal the reasoning for question 1

A plausible-looking future can be wrong about the details that determine success. Evaluate consequences and constraints under actions selected by the planner.

02If x is measured in meters, what unit does squared prediction error have?

Reveal the reasoning for question 2

Square meters. Combining that loss with angle or velocity losses requires explicit scaling rather than adding unlike quantities blindly.

TAKE THE IDEA WITH YOU

Three things to remember.

  • A world model predicts selected aspects of the future.
  • Planning error and prediction error are related but distinct.
  • Fresh observations and horizon-specific evaluation expose rollout drift.

Next we move from one agent predicting its future to several agents coordinating through limited information.

Continue to Part 19 →

THE NEXT LAYER

Chapter depth map.

This introduction provides a first model. The following chapters are outlines for deeper future editions; their full treatments have not yet been published.

Representing possible futures

18.1 State-space and latent models · Outline
  1. 18.1.1 Observed and hidden state
  2. 18.1.2 Transition and observation models
  3. 18.1.3 Latent representations and uncertainty
  4. 18.1.4 Multi-step prediction and distribution shift
  5. 18.1.5 Calibration and identifiability
18.2 Model-based reinforcement learning · Outline
  1. 18.2.1 Learned rollouts and imagined trajectories
  2. 18.2.2 Planning in latent state
  3. 18.2.3 Model bias and compounding error
  4. 18.2.4 Dreamer as a reading case study
  5. 18.2.5 Compare with model-free and classical baselines

Biological theories and engineering claims

18.3 Prediction in neuroscience · Outline
  1. 18.3.1 Internal models and sensorimotor prediction
  2. 18.3.2 Predictive processing and predictive coding
  3. 18.3.3 Active inference: assumptions and objectives
  4. 18.3.4 Distinguish theory, mechanism and experiment
  5. 18.3.5 Avoid treating a debated unifying theory as settled fact
18.4 World model meets reflex · Outline
  1. 18.4.1 Slow forecasts and fast reactive loops
  2. 18.4.2 Confidence thresholds for intervention
  3. 18.4.3 Memory and model updates
  4. 18.4.4 Counterfactual evaluation and ablations
  5. 18.4.5 Test benefit under unexpected obstacles

Read with the original sources.

These readings support the concepts and deeper investigation. The worked numbers and illustrations above are teaching examples.

Complete source register →

Where next?