What you will learn
Compare learned policies across matched architectures, seeds, and training budgets.
Before you begin: Foundations: matter, mathematics & computation; Spiking neural networks; Robotics: estimation, planning & control
START WITH A QUESTION
Picture this.
A student can learn from an answer key. A robot exploring a room often gets something weaker: it reached a destination, wasted time, or bumped into something. Reinforcement learning asks how actions can improve from consequences that may arrive much later.
- Policy
- A rule or distribution for choosing actions from available information.
- Return
- An accumulated, possibly discounted sequence of rewards.
- Value
- Expected return under a specified policy and environment.
Read for the idea first. Work through the numbers next. Try answering the questions before opening the explanations.
Walk through one interaction
At a decision step, the agent receives an observation, uses a policy to choose an action, and receives a reward and new observation after the environment responds. A policy is an action-selection rule; it can be a table, formula or neural network. State describes the variables needed to predict the environment's next-step distribution under an action. An observation may reveal only part of that state. If the same image could mean approaching or retreating, recent history can help disambiguate it.
Immediate reward is not the whole story
Suppose one route gives a small reward now and another reaches a larger reward later. Return combines rewards across a horizon. In our convention, the first reward r₀ follows the first action and has weight one; later rewards have weights γ, γ² and so on. Gamma discounts future rewards. For a finite sequence, γ = 1 gives the ordinary sum; γ = 0 keeps only the first reward. A reward is a designed signal, not an objective definition of worthwhile behavior.
Separate a sample from an expectation
A realized return belongs to one trajectory. A value function estimates expected return under a specified policy and environment. Expected means an average over possible outcomes, weighted by their probabilities. Exploration tries actions to gather information; exploitation uses current knowledge to choose promising actions. Actor-critic methods separate the policy, or actor, from a value estimator, or critic. Advanced algorithms such as PPO build on this distinction. You do not need their full objectives to understand the interaction loop.
What a spiking policy changes
An SNN can transform observations into stateful activity and decode that activity into actions. It changes the policy representation; it does not remove the need for a reward, training protocol or evaluation. Compare against a conventional recurrent policy with matched access to observations and environment interactions. Evaluate intended task success separately from training return. An agent rewarded only for moving quickly might learn to move quickly in the wrong direction. A separate held-out task metric helps reveal that mismatch.
SEE THE RELATIONSHIPS
From structure to behavior.
MAKE THE IDEA PRECISE
The mathematics.
| Symbol | Meaning |
|---|---|
| G_0 | Discounted return beginning at time zero |
| r_t | Reward at step t |
| gamma | Discount factor between zero and one |
| t | Discrete decision step |
Build the equation, step by step
- Choose a reward convention and horizon. The reward at the first transition receives weight one, the next receives γ, and later rewards receive successive powers.
- Multiply each observed reward by its discount factor and add them to obtain a realized return.
- Expected value averages returns under a policy and environment distribution. One observed return is a sample, not the value function or evidence that a policy has learned.
WORK THROUGH THE NUMBERS
Worked example
Rewards [1, 0, 2] with gamma = 0.9 produce 1 + 0 + 1.62 = 2.62. Return depends on the chosen discount and reward definition, not on a universal meaning of success.
A small Python experiment.
Python 3 · Standard library only · Synthetic educational example. Run this in your own notebook or a local file.
rewards, gamma = [1, 0, 2], 0.9
value = sum((gamma**t)*r for t,r in enumerate(rewards))
print(f"return = {value:.2f}")Read the code as a story
enumerate gives each reward a time index beginning at zero. gamma**t raises the discount factor to that index. The sum is a realized return for the supplied list; no policy is trained by this script. Keeping this arithmetic separate from learning makes it easier to debug a later training loop.
Expected output
return = 2.62
Change one quantity, predict the result, and then run the code again. Explain any difference before adding complexity.
Test your understanding.
Try it: Compare gamma = 0 and gamma = 1. Which behavior would each objective favor?
Self-check: If an agent gets a larger training return, is it better at the intended task?
Only if the reward captures the intended task and the comparison generalizes to held-out conditions. Reward exploits, changed termination rules and different budgets can make returns misleading. Keep a task-level evaluation separate from training feedback.
Where this model stops: Optimizing a reward can exploit loopholes or fail outside training conditions.
A research question: Does a structural prior improve sample efficiency after matching recurrence and compute?
Record evidence in your mastery worksheet →PREDICT / TRY / EXPLAIN
Choose between now and later
Use paper, a calculator or the Python example. Write your prediction before checking the result.
- Compute the original [1, 0, 2] return at γ = 0, 0.9 and 1.
- Compare route A with rewards [2, 0, 0] and route B with [0, 0, 3] at γ = 0.5 and 0.9.
- Describe a task where reward for speed alone could be misleading. Add a separate evaluation metric rather than assuming a larger training return solves the problem.
Compare with a worked solution
The original returns are 1, 2.62 and 3. Route A gives 2 at either discount. Route B gives 0.75 at γ = 0.5 and 2.43 at γ = 0.9, so the preferred route changes. A navigation evaluation could track completed deliveries and collisions in addition to the training reward.
CLOSE YOUR NOTES
Can you explain it back?
These questions test the reasoning, not just a remembered phrase. Say your answer out loud or write a sentence first.
01One rollout returns 2.62. Is the policy's value necessarily 2.62?
Reveal the reasoning for question 1
No. That is one sample. Value is an expectation over the relevant possible trajectories, so it needs a model or estimates from experience.
02Why might a recurrent policy help when observations omit velocity?
Reveal the reasoning for question 2
A sequence of observations can reveal motion that a single observation cannot. Recurrence provides a way to retain that history, although it still needs to learn or implement a useful estimate.
TAKE THE IDEA WITH YOU
Three things to remember.
- Actions change the data an agent will encounter next.
- Return accumulates outcomes; value averages over possible outcomes.
- Check intended task performance independently of the reward used for learning.
When the current observation is incomplete, an agent needs something from the past. Next we distinguish the different jobs hidden inside the word memory.
Continue to Part 17 →THE NEXT LAYER
Chapter depth map.
This introduction provides a first model. The following chapters are outlines for deeper future editions; their full treatments have not yet been published.
Learning from consequences
16.1 MDPs and value · Outline
- 16.1.1 States, actions, transitions and rewards
- 16.1.2 Markov assumptions and partial observability
- 16.1.3 Policies, returns and discounting
- 16.1.4 Value functions and Bellman equations
- 16.1.5 Q-learning and exploration
16.2 Policy optimization · Outline
- 16.2.1 Policy gradients and credit assignment
- 16.2.2 Actor-critic methods and variance reduction
- 16.2.3 PPO objective and clipping intuition
- 16.2.4 Reward design and unintended incentives
- 16.2.5 Evaluation seeds separate from training
Spiking and structural priors
16.3 Learning in a spiking policy · Outline
- 16.3.1 Temporal state and action decoding
- 16.3.2 Surrogate gradients versus local rules
- 16.3.3 Reward-modulated STDP and eligibility traces
- 16.3.4 Evolution strategies and neuroevolution
- 16.3.5 Cost of simulation and sample collection
16.4 Does connectome structure help? · Outline
- 16.4.1 Connectivity as an inductive-bias hypothesis
- 16.4.2 Degree-matched and randomly rewired controls
- 16.4.3 Fixed topology versus learned topology
- 16.4.4 Sample efficiency, robustness and compute
- 16.4.5 Reject the hypothesis if matched baselines win
Read with the original sources.
These readings support the concepts and deeper investigation. The worked numbers and illustrations above are teaching examples.
Primary paper / preprint
Schulman et al. (2017): Proximal Policy Optimization Algorithms ↗
Read after policy gradients. Explain the clipped objective and assess results under matched environment and training budgets.
Research tutorial / author manuscript
Neftci, Mostafa & Zenke (2019): surrogate gradient learning ↗
A deeper reading after the learning primer in Part 7. Compare the spiking forward rule with the approximate derivative used during training; the book’s scalar gradient example is an original smooth-model exercise.