What you will learn
Build a small audited subgraph, preserve metadata, and test its conversion into a sparse simulator.
Before you begin: Connectomics: wiring as data; Computational neuroscience; Spiking neural networks
START WITH A QUESTION
Picture this.
A neuron named 9001 does not need to live in the 9002nd slot of an array. Names identify things; positions organize computation. Keeping those roles separate prevents a surprisingly large class of modeling errors.
- Manifest
- A record of inputs, versions and transformations used in an experiment.
- Index
- A local array position, distinct from an external identifier.
- Boundary condition
- A rule describing how a selected model interacts with what lies outside it.
Read for the idea first. Work through the numbers next. Try answering the questions before opening the explanations.
Treat the dataset like an ingredient with a label
Before converting a graph, record the dataset release, specimen, query, selected cells and edge filters. A manifest is a small record of these decisions. It lets you distinguish a scientific change from an accidental data change. Official connectome services provide versioned resources, but their access procedures and schemas may evolve. Our example uses invented IDs and counts, so you can inspect every record without downloading a large dataset or supplying credentials.
Build a reversible address book
Map external IDs [101, 9001, 42] to local positions [0, 1, 2]. A Python dictionary gives the forward lookup; the original list gives the reverse lookup. Then translate both endpoints of every edge. If you later sort the IDs, you must rebuild the mapping and align all associated state arrays. Otherwise voltage that belonged to one cell can quietly be assigned to another. A program can run without errors while describing the wrong network.
Make boundary decisions explicit
What if an edge points to a cell outside your selection? You could reject the input, drop the edge while recording it, or expand the selection. These are different modeling choices. Do not silently map missing cells to index zero or pretend the missing input is known. Also decide how to aggregate duplicate records and whether counts must be positive integers. These checks are not paperwork: they define which graph your simulator will actually use.
Correctness before scale
A dense N-by-N matrix stores N² entries, even if most connections are absent. An edge list stores M records, plus fields and container overhead. Sparse storage often saves memory, but irregular access can still be slow. Start with the three-cell conversion, check endpoints and orientation, then choose a format based on measured needs. A count becomes a neural weight only after you declare its units, scaling, sign assumptions and dynamics. Preserve the original counts so those choices can be revised.
SEE THE RELATIONSHIPS
From structure to behavior.
MAKE THE IDEA PRECISE
The mathematics.
| Symbol | Meaning |
|---|---|
| N | Number of selected neurons |
| M | Number of retained directed edges |
| N² | All possible source-target pairs, including absent connections |
Build the equation, step by step
- For N indexed nodes, a dense directed matrix has N rows and N columns, hence N squared entries including possible self-connections.
- An edge list stores M records. Each record needs source, target and whatever fields the model requires; its byte cost is not merely the weight size.
- Map persistent external identifiers to contiguous local indices with a reversible lookup. Sorting or filtering must not silently change the association between edges and state vectors.
WORK THROUGH THE NUMBERS
Worked example
Three external IDs map to positions 0, 1 and 2. Two retained edges are translated through that mapping. The demonstration keeps a count field unchanged and makes no physiological weighting claim.
A small Python experiment.
Python 3 · Standard library only · Synthetic educational example. Run this in your own notebook or a local file.
ids = [101, 9001, 42]
index = {neuron:i for i,neuron in enumerate(ids)}
edges = [(101, 42, 3), (42, 9001, 2)]
compact = [(index[a],index[b],count) for a,b,count in edges]
print(compact)Read the code as a story
enumerate pairs each external ID with its local position. The dictionary maps the external name to that position. The list comprehension translates source and target but leaves the count unchanged. It will raise an error for an unknown endpoint, making the missing-data decision visible rather than silently inventing a connection.
Expected output
[(0, 2, 3), (2, 1, 2)]
Change one quantity, predict the result, and then run the code again. Explain any difference before adding complexity.
Test your understanding.
Try it: Add an edge to an ID outside the selection. Decide whether to reject it, drop it with a log, or expand the selection.
Self-check: Why should an identifier such as 9001 not become array position 9001?
Identifiers may be sparse, very large, noncontiguous or version-specific. Direct indexing wastes space and can mix datasets. Keep a mapping and its inverse, validate endpoints, and record the dataset version that gives those IDs meaning.
Where this model stops: Synthetic IDs here are not real biological annotations. Large-data access and full simulation remain separate experiments.
A research question: How does the chosen subgraph boundary alter spontaneous and driven activity?
Record evidence in your mastery worksheet →PREDICT / TRY / EXPLAIN
Round-trip the mapping
Use paper, a calculator or the Python example. Write your prediction before checking the result.
- Translate the two synthetic edges to local indices.
- Use ids[source_index] and ids[target_index] to translate them back. Compare every field with the original records.
- Add (101, 777, 1). Choose and document an unknown-endpoint policy, then describe a test for that policy.
Compare with a worked solution
The compact edges are [(0, 2, 3), (2, 1, 2)], and reverse lookup recovers the exact originals. ID 777 is absent. Under a reject policy, expect a clear error. Under a drop-and-log policy, expect one rejected record and an unchanged retained graph. The policy and its count belong in the manifest.
CLOSE YOUR NOTES
Can you explain it back?
These questions test the reasoning, not just a remembered phrase. Say your answer out loud or write a sentence first.
01Why retain the reverse mapping?
Reveal the reasoning for question 1
It connects a simulated result back to the biological records that gave the local indices meaning. Without it, a plot labeled 'cell 2' can become uninterpretable.
02For N = 1000 and M = 10,000, how many dense entries are stored?
Reveal the reasoning for question 2
One million. An edge list has 10,000 records, but comparing byte costs also requires the size of each record and container overhead.
TAKE THE IDEA WITH YOU
Three things to remember.
- Separate biological identity from computational position.
- Validate endpoints and preserve reversible mappings.
- A data transformation is part of the model and should be reproducible.
Once the graph is correct, we can ask where to run it. Next we measure the complete computation rather than assuming sparse activity automatically saves energy.
Continue to Part 12 →THE NEXT LAYER
Chapter depth map.
This introduction provides a first model. The following chapters are outlines for deeper future editions; their full treatments have not yet been published.
From query to graph
11.1 Data access and identity · Outline
- 11.1.1 neuPrint and documented query interfaces
- 11.1.2 FlyWire tools and MaleCNS exports
- 11.1.3 Neuron IDs, cell types and region filters
- 11.1.4 Synapse queries, pagination and access conditions
- 11.1.5 Pin release, query, license and downloaded-file hash
11.2 Conversion without silent assumptions · Outline
- 11.2.1 Typed edge lists and duplicate handling
- 11.2.2 NetworkX graphs for exploration
- 11.2.3 Sparse adjacency matrices with verified orientation
- 11.2.4 PyTorch tensors and index remapping
- 11.2.5 Count-to-weight choices and uncertainty
From graph to experiment
11.3 Assigning dynamics · Outline
- 11.3.1 Neuron-model parameters are extra assumptions
- 11.3.2 Excitatory/inhibitory interpretation needs receptor context
- 11.3.3 Input mapping, delays and initial conditions
- 11.3.4 Subgraph boundary treatment and missing inputs
- 11.3.5 Validate small examples before large exports
11.4 Large sparse data · Outline
- 11.4.1 Streaming reads and chunked queries
- 11.4.2 Sparse matrix formats and memory budgets
- 11.4.3 Vectorization, event updates and profiling
- 11.4.4 Reproducible graph-to-simulation manifests
- 11.4.5 Activity analysis and sensitivity experiments
Read with the original sources.
These readings support the concepts and deeper investigation. The worked numbers and illustrations above are teaching examples.
Official dataset tool
neuPrint — official connectome explorer ↗
Inspect dataset availability and access requirements before writing queries. Do not silently combine IDs from different datasets.
Official dataset resource
FlyWire — official project and dataset resources ↗
An adult female fly brain resource. Record the exact release and annotation version; a brain dataset should not be described as a complete brain-and-body model.
Official dataset resource
MaleCNS — official release and downloads ↗
The official download page exposes versioned data, including v1.0 resources. Keep specimen, release, filtering and annotation provenance with every derived graph.