PART 11 / Advanced

Connectome programming

How does a versioned wiring dataset become a reproducible simulation?

Guided illustrated lesson

What you will learn

Build a small audited subgraph, preserve metadata, and test its conversion into a sparse simulator.

Before you begin: Connectomics: wiring as data; Computational neuroscience; Spiking neural networks

START WITH A QUESTION

Picture this.

A neuron named 9001 does not need to live in the 9002nd slot of an array. Names identify things; positions organize computation. Keeping those roles separate prevents a surprisingly large class of modeling errors.

Manifest
A record of inputs, versions and transformations used in an experiment.
Index
A local array position, distinct from an external identifier.
Boundary condition
A rule describing how a selected model interacts with what lies outside it.

Read for the idea first. Work through the numbers next. Try answering the questions before opening the explanations.

Treat the dataset like an ingredient with a label

Before converting a graph, record the dataset release, specimen, query, selected cells and edge filters. A manifest is a small record of these decisions. It lets you distinguish a scientific change from an accidental data change. Official connectome services provide versioned resources, but their access procedures and schemas may evolve. Our example uses invented IDs and counts, so you can inspect every record without downloading a large dataset or supplying credentials.

Build a reversible address book

Map external IDs [101, 9001, 42] to local positions [0, 1, 2]. A Python dictionary gives the forward lookup; the original list gives the reverse lookup. Then translate both endpoints of every edge. If you later sort the IDs, you must rebuild the mapping and align all associated state arrays. Otherwise voltage that belonged to one cell can quietly be assigned to another. A program can run without errors while describing the wrong network.

Make boundary decisions explicit

What if an edge points to a cell outside your selection? You could reject the input, drop the edge while recording it, or expand the selection. These are different modeling choices. Do not silently map missing cells to index zero or pretend the missing input is known. Also decide how to aggregate duplicate records and whether counts must be positive integers. These checks are not paperwork: they define which graph your simulator will actually use.

Correctness before scale

A dense N-by-N matrix stores N² entries, even if most connections are absent. An edge list stores M records, plus fields and container overhead. Sparse storage often saves memory, but irregular access can still be slow. Start with the three-cell conversion, check endpoints and orientation, then choose a format based on measured needs. A count becomes a neural weight only after you declare its units, scaling, sign assumptions and dynamics. Preserve the original counts so those choices can be revised.

SEE THE RELATIONSHIPS

From structure to behavior.

Connectome: a labelled conceptual diagram
Figure 11.1 · Connectome. Original educational diagram; synthetic, not a measured result.
The reasoning pathPin dataset/query then Map IDs & edges then Assign model assumptions then Validate small graph1. Pin dataset/query2. Map IDs & edges3. Assign model assumptions4. Validate small graph
A learning flow for this part. Follow the assumptions through to an observable result.

MAKE THE IDEA PRECISE

The mathematics.

dense entries = N²; edge-list records = M
Symbols and units
SymbolMeaning
NNumber of selected neurons
MNumber of retained directed edges
N²All possible source-target pairs, including absent connections

Build the equation, step by step

  1. For N indexed nodes, a dense directed matrix has N rows and N columns, hence N squared entries including possible self-connections.
  2. An edge list stores M records. Each record needs source, target and whatever fields the model requires; its byte cost is not merely the weight size.
  3. Map persistent external identifiers to contiguous local indices with a reversible lookup. Sorting or filtering must not silently change the association between edges and state vectors.

WORK THROUGH THE NUMBERS

Worked example

Three external IDs map to positions 0, 1 and 2. Two retained edges are translated through that mapping. The demonstration keeps a count field unchanged and makes no physiological weighting claim.

A small Python experiment.

Python 3 · Standard library only · Synthetic educational example. Run this in your own notebook or a local file.

ids = [101, 9001, 42]
index = {neuron:i for i,neuron in enumerate(ids)}
edges = [(101, 42, 3), (42, 9001, 2)]
compact = [(index[a],index[b],count) for a,b,count in edges]
print(compact)

Read the code as a story

enumerate pairs each external ID with its local position. The dictionary maps the external name to that position. The list comprehension translates source and target but leaves the count unchanged. It will raise an error for an unknown endpoint, making the missing-data decision visible rather than silently inventing a connection.

Expected output

[(0, 2, 3), (2, 1, 2)]

Change one quantity, predict the result, and then run the code again. Explain any difference before adding complexity.

Test your understanding.

Try it: Add an edge to an ID outside the selection. Decide whether to reject it, drop it with a log, or expand the selection.

Self-check: Why should an identifier such as 9001 not become array position 9001?

Identifiers may be sparse, very large, noncontiguous or version-specific. Direct indexing wastes space and can mix datasets. Keep a mapping and its inverse, validate endpoints, and record the dataset version that gives those IDs meaning.

Where this model stops: Synthetic IDs here are not real biological annotations. Large-data access and full simulation remain separate experiments.

A research question: How does the chosen subgraph boundary alter spontaneous and driven activity?

Record evidence in your mastery worksheet →

PREDICT / TRY / EXPLAIN

Round-trip the mapping

Use paper, a calculator or the Python example. Write your prediction before checking the result.

  1. Translate the two synthetic edges to local indices.
  2. Use ids[source_index] and ids[target_index] to translate them back. Compare every field with the original records.
  3. Add (101, 777, 1). Choose and document an unknown-endpoint policy, then describe a test for that policy.
Compare with a worked solution

The compact edges are [(0, 2, 3), (2, 1, 2)], and reverse lookup recovers the exact originals. ID 777 is absent. Under a reject policy, expect a clear error. Under a drop-and-log policy, expect one rejected record and an unchanged retained graph. The policy and its count belong in the manifest.

CLOSE YOUR NOTES

Can you explain it back?

These questions test the reasoning, not just a remembered phrase. Say your answer out loud or write a sentence first.

01Why retain the reverse mapping?

Reveal the reasoning for question 1

It connects a simulated result back to the biological records that gave the local indices meaning. Without it, a plot labeled 'cell 2' can become uninterpretable.

02For N = 1000 and M = 10,000, how many dense entries are stored?

Reveal the reasoning for question 2

One million. An edge list has 10,000 records, but comparing byte costs also requires the size of each record and container overhead.

TAKE THE IDEA WITH YOU

Three things to remember.

  • Separate biological identity from computational position.
  • Validate endpoints and preserve reversible mappings.
  • A data transformation is part of the model and should be reproducible.

Once the graph is correct, we can ask where to run it. Next we measure the complete computation rather than assuming sparse activity automatically saves energy.

Continue to Part 12 →

THE NEXT LAYER

Chapter depth map.

This introduction provides a first model. The following chapters are outlines for deeper future editions; their full treatments have not yet been published.

From query to graph

11.1 Data access and identity · Outline
  1. 11.1.1 neuPrint and documented query interfaces
  2. 11.1.2 FlyWire tools and MaleCNS exports
  3. 11.1.3 Neuron IDs, cell types and region filters
  4. 11.1.4 Synapse queries, pagination and access conditions
  5. 11.1.5 Pin release, query, license and downloaded-file hash
11.2 Conversion without silent assumptions · Outline
  1. 11.2.1 Typed edge lists and duplicate handling
  2. 11.2.2 NetworkX graphs for exploration
  3. 11.2.3 Sparse adjacency matrices with verified orientation
  4. 11.2.4 PyTorch tensors and index remapping
  5. 11.2.5 Count-to-weight choices and uncertainty

From graph to experiment

11.3 Assigning dynamics · Outline
  1. 11.3.1 Neuron-model parameters are extra assumptions
  2. 11.3.2 Excitatory/inhibitory interpretation needs receptor context
  3. 11.3.3 Input mapping, delays and initial conditions
  4. 11.3.4 Subgraph boundary treatment and missing inputs
  5. 11.3.5 Validate small examples before large exports
11.4 Large sparse data · Outline
  1. 11.4.1 Streaming reads and chunked queries
  2. 11.4.2 Sparse matrix formats and memory budgets
  3. 11.4.3 Vectorization, event updates and profiling
  4. 11.4.4 Reproducible graph-to-simulation manifests
  5. 11.4.5 Activity analysis and sensitivity experiments

Read with the original sources.

These readings support the concepts and deeper investigation. The worked numbers and illustrations above are teaching examples.

Official dataset resource

MaleCNS — official release and downloads ↗

The official download page exposes versioned data, including v1.0 resources. Keep specimen, release, filtering and annotation provenance with every derived graph.

Complete source register →

Where next?