What you will learn
Turn an attractive idea into a preregistered comparison with realistic failure outcomes.
Before you begin: AGI connections: evidence before promises; Connectome programming
START WITH A QUESTION
Picture this.
Suppose a new recipe tastes better, but you changed the ingredients, oven temperature and cooking time at once. Which change helped? Architecture research faces the same problem when topology, learning and decoding all change together.
- Paired comparison
- A comparison using matched conditions for candidate and baseline within each experimental pair.
- Standard deviation
- A measure of spread among observed values.
- Standard error
- An estimate of the uncertainty scale of an estimator, under stated assumptions.
Read for the idea first. Work through the numbers next. Try answering the questions before opening the explanations.
Choose the contrast before the experiment
A useful question names what changes and what stays fixed. If you want to test topology, hold the encoder, decoder, training protocol and evaluation conditions constant as far as the question permits. A degree-matched rewiring control preserves a different set of properties from a fully dense baseline. Neither is universally fair; each answers a particular contrast. Write why the comparison is informative before seeing which method wins.
Use ablations as questions
An ablation changes or removes a component. Removing recurrence asks whether temporal state matters under the remaining setup. Replacing biological wiring with a matched random graph asks a more specific question about structure. A component may interact with another, so one-at-a-time removal is not always enough. A small factorial comparison can test combinations. Keep the experiment feasible and decide in advance which outcome is primary, rather than selecting the most flattering metric afterward.
Pairs can reduce irrelevant variation
If candidate and baseline face matched independent scenario sets, compute their difference within each pair. Shared difficult scenarios then affect both members. Averaging the differences estimates the mean contrast. The standard error describes uncertainty in that estimated mean under assumptions about independence and comparability. It is not the spread of individual outcomes and is not automatically a confidence interval. Repeated frames from one run are correlated observations, not hundreds of independent experimental runs.
Look at the whole result
The teaching differences [0.04, 0.02, −0.01, 0.03] average 0.02, and their estimated standard error is about 0.011. One run favors the baseline. Keep that fact visible. Four pairs demonstrate the calculation, not a universal sample-size recommendation. Separate statistical uncertainty from practical importance: a tiny reliable gain may not justify large integration costs. Reserve evaluation cases before tuning and record negative results with the same care as positive ones.
SEE THE RELATIONSHIPS
From structure to behavior.
MAKE THE IDEA PRECISE
The mathematics.
| Symbol | Meaning |
|---|---|
| delta_i | Paired candidate-minus-baseline score for independent run i |
| n | Number of paired runs |
| SE | Estimated standard error of the mean under independence assumptions |
Build the equation, step by step
- Pair candidate and baseline measurements under matched conditions and compute their differences.
- Average those differences for the estimated effect. Estimate sample standard deviation using n − 1 in the variance denominator, requiring at least two pairs.
- For independent comparable pairs, estimated standard error of the mean is sample standard deviation divided by the square root of n. Correlated trials, selection after looking at results and dataset leakage can invalidate a naive interpretation.
WORK THROUGH THE NUMBERS
Worked example
Illustrative paired differences [0.04, 0.02, -0.01, 0.03] have mean 0.020 and estimated standard error about 0.011. Four runs are a small example, not an adequate universal experimental sample size.
A small Python experiment.
Python 3 · Standard library only · Synthetic educational example. Run this in your own notebook or a local file.
from statistics import mean, stdev
from math import sqrt
differences = [0.04, 0.02, -0.01, 0.03]
print(f"mean = {mean(differences):.3f}; SE = {stdev(differences)/sqrt(len(differences)):.3f}")Read the code as a story
mean averages the paired differences. stdev uses the sample standard deviation with n−1 in the variance denominator, so at least two observations are needed. Dividing by sqrt(n) gives the standard error formula used here. None of these functions checks whether the runs were independent; that is an experimental-design responsibility.
Expected output
mean = 0.020; SE = 0.011
Change one quantity, predict the result, and then run the code again. Explain any difference before adding complexity.
Test your understanding.
Try it: Compare independent evaluation environments with repeated measurements from the same trajectory.
Self-check: Is one standard error a 95% confidence interval?
No. It is an estimated scale of uncertainty for the mean under assumptions. A confidence interval needs an appropriate multiplier and distributional method; small samples commonly require different treatment from a large-sample normal approximation. Neither interval alone establishes practical importance.
Where this model stops: A standard error is not a confidence interval or a proof of causality.
A research question: Which conclusions remain stable under rewiring, parameter sweeps and changed task distributions?
Record evidence in your mastery worksheet →PREDICT / TRY / EXPLAIN
Design an ablation table
Use paper, a calculator or the Python example. Write your prediction before checking the result.
- Make rows for the original graph, a degree-matched rewired graph and a simple recurrent baseline. Specify the same input and scoring protocol.
- Use the four teaching differences to compute the mean. Mark which pair favors the baseline.
- Explain why copying each row of data ten times would not create ten times as much independent evidence.
Compare with a worked solution
The mean is 0.02, or 2 percentage points if the scores are success fractions. The −0.01 pair favors the baseline. Duplicating observations changes a naive sample count but adds no new information; treating duplicates as independent would artificially shrink estimated uncertainty.
CLOSE YOUR NOTES
Can you explain it back?
These questions test the reasoning, not just a remembered phrase. Say your answer out loud or write a sentence first.
01Is one standard error a 95% confidence interval?
Reveal the reasoning for question 1
No. A confidence interval requires an appropriate method and multiplier under its assumptions. Small sample sizes need particular care; the standard error alone is not an interval.
02If removing a module hurts performance, have we proved the module is optimal?
Reveal the reasoning for question 2
No. It supports a contribution under that comparison. A simpler replacement or a jointly retuned alternative may still perform as well or better.
TAKE THE IDEA WITH YOU
Three things to remember.
- Make the intended contrast explicit before collecting results.
- Use independent experimental units and justify pairing.
- Inspect individual outcomes, practical effects and uncertainty together.
Next we step outside the experiment and ask when the whole approach is worth using for a practical task.
Continue to Part 23 →THE NEXT LAYER
Chapter depth map.
This introduction provides a first model. The following chapters are outlines for deeper future editions; their full treatments have not yet been published.
Architecture and learning
22.1 Structure as an inductive bias · Outline
- 22.1.1 Connectome-inspired architectures and matched controls
- 22.1.2 Circuit motifs versus generic sparsity
- 22.1.3 Training data requirements and sample efficiency
- 22.1.4 Specialized circuits emerging through learning
- 22.1.5 Transformers combined with structural priors
22.2 Timing and adaptation · Outline
- 22.2.1 Local biological learning rules at scale
- 22.2.2 Neuromodulators and missing receptor information
- 22.2.3 Architecture versus learned dynamics
- 22.2.4 Robustness to unknown delays and synaptic strengths
- 22.2.5 Compare rate and temporal representations
Embodied and efficient intelligence
22.3 Physical hypotheses · Outline
- 22.3.1 World models communicating with reflex SNNs
- 22.3.2 Always-on neuromorphic robots
- 22.3.3 Body/environment contributions to behavior
- 22.3.4 Whole-system energy versus kernel efficiency
- 22.3.5 Cases where simpler controllers are sufficient
22.4 A falsifiable proposal · Outline
- 22.4.1 Primary outcome and effect-size target
- 22.4.2 Baseline and ablation matrix
- 22.4.3 Compute and data budget
- 22.4.4 Replication and sensitivity analysis
- 22.4.5 Negative results, stopping criteria and open questions
Read with the original sources.
These readings support the concepts and deeper investigation. The worked numbers and illustrations above are teaching examples.
Primary paper
Shiu et al. (2024): a Drosophila computational brain model ↗
A connectome-based computational model of sensorimotor processing. Separate experimentally tested predictions from the model assumptions and untested behaviors.
Primary paper
Lin et al. (2024): network statistics of the fly connectome ↗
Study motif and network analysis with attention to filtering and null-model choices. Graph centrality is not automatically causal importance.
Official hardware research resource
Intel — neuromorphic computing research ↗
Study event processing, memory and mapping constraints. Do not assume public hardware access or transfer platform-specific energy claims to a new task.