PART 16 / Advanced

Reinforcement learning + SNNs

Can biological structure be a useful inductive bias when behavior must be learned?

Guided illustrated lesson

What you will learn

Compare learned policies across matched architectures, seeds, and training budgets.

Before you begin: Foundations: matter, mathematics & computation; Spiking neural networks; Robotics: estimation, planning & control

START WITH A QUESTION

Picture this.

A student can learn from an answer key. A robot exploring a room often gets something weaker: it reached a destination, wasted time, or bumped into something. Reinforcement learning asks how actions can improve from consequences that may arrive much later.

Policy
A rule or distribution for choosing actions from available information.
Return
An accumulated, possibly discounted sequence of rewards.
Value
Expected return under a specified policy and environment.

Read for the idea first. Work through the numbers next. Try answering the questions before opening the explanations.

Walk through one interaction

At a decision step, the agent receives an observation, uses a policy to choose an action, and receives a reward and new observation after the environment responds. A policy is an action-selection rule; it can be a table, formula or neural network. State describes the variables needed to predict the environment's next-step distribution under an action. An observation may reveal only part of that state. If the same image could mean approaching or retreating, recent history can help disambiguate it.

Immediate reward is not the whole story

Suppose one route gives a small reward now and another reaches a larger reward later. Return combines rewards across a horizon. In our convention, the first reward r₀ follows the first action and has weight one; later rewards have weights γ, γ² and so on. Gamma discounts future rewards. For a finite sequence, γ = 1 gives the ordinary sum; γ = 0 keeps only the first reward. A reward is a designed signal, not an objective definition of worthwhile behavior.

Separate a sample from an expectation

A realized return belongs to one trajectory. A value function estimates expected return under a specified policy and environment. Expected means an average over possible outcomes, weighted by their probabilities. Exploration tries actions to gather information; exploitation uses current knowledge to choose promising actions. Actor-critic methods separate the policy, or actor, from a value estimator, or critic. Advanced algorithms such as PPO build on this distinction. You do not need their full objectives to understand the interaction loop.

What a spiking policy changes

An SNN can transform observations into stateful activity and decode that activity into actions. It changes the policy representation; it does not remove the need for a reward, training protocol or evaluation. Compare against a conventional recurrent policy with matched access to observations and environment interactions. Evaluate intended task success separately from training return. An agent rewarded only for moving quickly might learn to move quickly in the wrong direction. A separate held-out task metric helps reveal that mismatch.

SEE THE RELATIONSHIPS

From structure to behavior.

Network: a labelled conceptual diagram
Figure 16.1 · Network. Original educational diagram; synthetic, not a measured result.
The reasoning pathObservation & state then Policy chooses action then Environment changes then Reward & learning1. Observation & state2. Policy chooses action3. Environment changes4. Reward & learning
A learning flow for this part. Follow the assumptions through to an observable result.

MAKE THE IDEA PRECISE

The mathematics.

G_0 = r_0 + gamma r_1 + gamma² r_2 + ...
Symbols and units
SymbolMeaning
G_0Discounted return beginning at time zero
r_tReward at step t
gammaDiscount factor between zero and one
tDiscrete decision step

Build the equation, step by step

  1. Choose a reward convention and horizon. The reward at the first transition receives weight one, the next receives γ, and later rewards receive successive powers.
  2. Multiply each observed reward by its discount factor and add them to obtain a realized return.
  3. Expected value averages returns under a policy and environment distribution. One observed return is a sample, not the value function or evidence that a policy has learned.

WORK THROUGH THE NUMBERS

Worked example

Rewards [1, 0, 2] with gamma = 0.9 produce 1 + 0 + 1.62 = 2.62. Return depends on the chosen discount and reward definition, not on a universal meaning of success.

A small Python experiment.

Python 3 · Standard library only · Synthetic educational example. Run this in your own notebook or a local file.

rewards, gamma = [1, 0, 2], 0.9
value = sum((gamma**t)*r for t,r in enumerate(rewards))
print(f"return = {value:.2f}")

Read the code as a story

enumerate gives each reward a time index beginning at zero. gamma**t raises the discount factor to that index. The sum is a realized return for the supplied list; no policy is trained by this script. Keeping this arithmetic separate from learning makes it easier to debug a later training loop.

Expected output

return = 2.62

Change one quantity, predict the result, and then run the code again. Explain any difference before adding complexity.

Test your understanding.

Try it: Compare gamma = 0 and gamma = 1. Which behavior would each objective favor?

Self-check: If an agent gets a larger training return, is it better at the intended task?

Only if the reward captures the intended task and the comparison generalizes to held-out conditions. Reward exploits, changed termination rules and different budgets can make returns misleading. Keep a task-level evaluation separate from training feedback.

Where this model stops: Optimizing a reward can exploit loopholes or fail outside training conditions.

A research question: Does a structural prior improve sample efficiency after matching recurrence and compute?

Record evidence in your mastery worksheet →

PREDICT / TRY / EXPLAIN

Choose between now and later

Use paper, a calculator or the Python example. Write your prediction before checking the result.

  1. Compute the original [1, 0, 2] return at γ = 0, 0.9 and 1.
  2. Compare route A with rewards [2, 0, 0] and route B with [0, 0, 3] at γ = 0.5 and 0.9.
  3. Describe a task where reward for speed alone could be misleading. Add a separate evaluation metric rather than assuming a larger training return solves the problem.
Compare with a worked solution

The original returns are 1, 2.62 and 3. Route A gives 2 at either discount. Route B gives 0.75 at γ = 0.5 and 2.43 at γ = 0.9, so the preferred route changes. A navigation evaluation could track completed deliveries and collisions in addition to the training reward.

CLOSE YOUR NOTES

Can you explain it back?

These questions test the reasoning, not just a remembered phrase. Say your answer out loud or write a sentence first.

01One rollout returns 2.62. Is the policy's value necessarily 2.62?

Reveal the reasoning for question 1

No. That is one sample. Value is an expectation over the relevant possible trajectories, so it needs a model or estimates from experience.

02Why might a recurrent policy help when observations omit velocity?

Reveal the reasoning for question 2

A sequence of observations can reveal motion that a single observation cannot. Recurrence provides a way to retain that history, although it still needs to learn or implement a useful estimate.

TAKE THE IDEA WITH YOU

Three things to remember.

  • Actions change the data an agent will encounter next.
  • Return accumulates outcomes; value averages over possible outcomes.
  • Check intended task performance independently of the reward used for learning.

When the current observation is incomplete, an agent needs something from the past. Next we distinguish the different jobs hidden inside the word memory.

Continue to Part 17 →

THE NEXT LAYER

Chapter depth map.

This introduction provides a first model. The following chapters are outlines for deeper future editions; their full treatments have not yet been published.

Learning from consequences

16.1 MDPs and value · Outline
  1. 16.1.1 States, actions, transitions and rewards
  2. 16.1.2 Markov assumptions and partial observability
  3. 16.1.3 Policies, returns and discounting
  4. 16.1.4 Value functions and Bellman equations
  5. 16.1.5 Q-learning and exploration
16.2 Policy optimization · Outline
  1. 16.2.1 Policy gradients and credit assignment
  2. 16.2.2 Actor-critic methods and variance reduction
  3. 16.2.3 PPO objective and clipping intuition
  4. 16.2.4 Reward design and unintended incentives
  5. 16.2.5 Evaluation seeds separate from training

Spiking and structural priors

16.3 Learning in a spiking policy · Outline
  1. 16.3.1 Temporal state and action decoding
  2. 16.3.2 Surrogate gradients versus local rules
  3. 16.3.3 Reward-modulated STDP and eligibility traces
  4. 16.3.4 Evolution strategies and neuroevolution
  5. 16.3.5 Cost of simulation and sample collection
16.4 Does connectome structure help? · Outline
  1. 16.4.1 Connectivity as an inductive-bias hypothesis
  2. 16.4.2 Degree-matched and randomly rewired controls
  3. 16.4.3 Fixed topology versus learned topology
  4. 16.4.4 Sample efficiency, robustness and compute
  5. 16.4.5 Reject the hypothesis if matched baselines win

Read with the original sources.

These readings support the concepts and deeper investigation. The worked numbers and illustrations above are teaching examples.

Research tutorial / author manuscript

Neftci, Mostafa & Zenke (2019): surrogate gradient learning ↗

A deeper reading after the learning primer in Part 7. Compare the spiking forward rule with the approximate derivative used during training; the book’s scalar gradient example is an original smooth-model exercise.

Complete source register →

Where next?