PART 15 / Advanced

Hybrid Physical AI

How can fast reflexes and slower reasoning cooperate without conflicting commands?

Guided illustrated lesson

What you will learn

Specify interfaces, timing, and arbitration; revisit learned planning after the RL and world-model parts.

Before you begin: Robotics: estimation, planning & control; Drones: simulation before flight; Spiking neural networks

START WITH A QUESTION

Picture this.

Imagine a delivery robot checking a route while a person steps into its path. The route planner may be useful, but the immediate obstacle cannot wait for a long deliberation. A hybrid system must decide who acts on which timescale.

Latency
Elapsed time for an individual input to produce its usable result.
Throughput
The rate at which items are processed.
Arbitration
A rule for resolving competing proposals or priorities.

Read for the idea first. Work through the numbers next. Try answering the questions before opening the explanations.

Different jobs deserve different clocks

A reflex responds to immediate evidence. Perception estimates what is around the robot. Memory retains relevant context. A planner chooses a sequence toward a goal, and a controller turns requests into motion. These functions can share information without running at the same rate. A language model or a spiking network is a candidate implementation for a function, not a reason to add that function. Start with the task and ask what information each stage must produce.

Make every message understandable

A useful interface contains a value, units, frame, observation time and any uncertainty needed by the receiver. 'Turn left' is incomplete without a command convention and authority rule. A planner's recent processing timestamp can hide an old camera observation, so distinguish acquisition time from output time. Define who can override a request and what happens to late results. These contracts allow you to test modules independently and then test disagreements between them.

Latency and throughput are different

Latency is how long one observation takes to reach a decision. Throughput is how many observations the system handles per second. A pipeline can process many items concurrently while each item still waits a long time. For serial stages, add their delays. For parallel branches, follow dependencies: two independent branches taking 5 ms and 8 ms contribute 8 ms to a join that waits for both, before communication overhead. Queueing can dominate either arrangement.

An average can hide the failure

The small example totals 14 ms against a 20 ms deadline. That leaves 6 ms of slack under the stated deterministic assumptions. A 12 ms queue pushes the total to 26 ms. Measure deadline misses and tail behavior, not only mean latency. Add one module at a time and compare task outcomes under matched conditions. If a simple perception-controller pair does as well, the larger architecture needs a stronger justification than the number of impressive boxes in its diagram.

SEE THE RELATIONSHIPS

From structure to behavior.

Hybrid: a labelled conceptual diagram
Figure 15.1 · Hybrid. Original educational diagram; synthetic, not a measured result.
The reasoning pathFast reflex + perception then Memory + world model then Planning & arbitration then Bounded control1. Fast reflex + perception2. Memory + world model3. Planning & arbitration4. Bounded control
A learning flow for this part. Follow the assumptions through to an observable result.

MAKE THE IDEA PRECISE

The mathematics.

L_total = L_sense + L_encode + L_model + L_decode; slack = D - L_total
Symbols and units
SymbolMeaning
LStage or total latency, milliseconds
DTask deadline, milliseconds
slackRemaining time budget; negative means the deadline is missed

Build the equation, step by step

  1. If stages execute serially with no overlap, add their latencies to estimate end-to-end processing delay.
  2. Subtract the total from deadline D to get slack. Negative slack means that sequence cannot meet the deadline even before considering unmodelled overhead.
  3. Pipelines and parallel branches need a dependency-based critical-path calculation. Average latency also hides jitter, queueing and rare long delays that matter to control.

WORK THROUGH THE NUMBERS

Worked example

Chosen stage costs of 3, 5, 2 and 4 ms total 14 ms. Against a 20 ms deadline, deterministic slack is 6 ms. Real tail latency needs measurement.

A small Python experiment.

Python 3 · Standard library only · Synthetic educational example. Run this in your own notebook or a local file.

stages_ms = [3, 5, 2, 4]
deadline_ms = 20
total = sum(stages_ms)
print(f"latency = {total} ms; slack = {deadline_ms-total} ms")

Read the code as a story

sum adds four serial stage durations. Subtracting from the deadline gives slack: positive means time remains in this simplified budget, negative means a miss. This calculation assumes no overlap and omits queueing unless you add it. A real measurement should begin at the relevant observation time and end when the result is usable.

Expected output

latency = 14 ms; slack = 6 ms

Change one quantity, predict the result, and then run the code again. Explain any difference before adding complexity.

Test your understanding.

Try it: Add a 12 ms queueing delay. Which design change would you test first, and why?

Self-check: Does positive average slack guarantee a deadline?

No. A deadline guarantee concerns the relevant worst case or a specified probabilistic bound, including communication and scheduling. A positive mean can coexist with frequent overruns. Record the latency distribution and define what the system does when a result arrives late.

Where this model stops: A layered diagram does not demonstrate intelligence or reliability. Interface tests and system evaluation are still required.

A research question: Can a simpler perception/controller pair match the full architecture under the same deadline?

Record evidence in your mastery worksheet →

PREDICT / TRY / EXPLAIN

Trace a late observation

Use paper, a calculator or the Python example. Write your prediction before checking the result.

  1. Draw the four serial stages with durations 3, 5, 2 and 4 ms.
  2. Insert a 12 ms queue before the model. Calculate total latency and slack.
  3. Propose one measurable change—such as dropping stale queued observations—and specify a task-level downside you would also measure.
Compare with a worked solution

The total becomes 26 ms and slack becomes −6 ms. Dropping stale inputs may reduce latency but discard useful evidence. A test should measure both deadline misses and task quality. Calling the model faster without measuring the full path would miss the queue's contribution.

CLOSE YOUR NOTES

Can you explain it back?

These questions test the reasoning, not just a remembered phrase. Say your answer out loud or write a sentence first.

01Two independent branches take 5 ms and 8 ms, and a join waits for both. Is their ideal combined delay 13 ms?

Reveal the reasoning for question 1

No. With simultaneous starts and no overhead, the join waits 8 ms. Add delays along dependencies, not across truly parallel work.

02Can a positive mean slack guarantee every deadline?

Reveal the reasoning for question 2

No. Some observations can still be late. State the required worst-case or probabilistic criterion and measure the relevant distribution.

TAKE THE IDEA WITH YOU

Three things to remember.

  • Assign functions and timescales before selecting model families.
  • Interfaces must preserve the meaning and age of information.
  • Whole-path latency and task quality determine whether a module helps.

The next part changes how actions are chosen: instead of specifying every rule, we let outcomes provide a learning signal.

Continue to Part 16 →

THE NEXT LAYER

Chapter depth map.

This introduction provides a first model. The following chapters are outlines for deeper future editions; their full treatments have not yet been published.

Five interacting levels

15.1 Reflex and perception · Outline
  1. 15.1.1 Event camera and SNN immediate-response layer
  2. 15.1.2 CNN/YOLO, depth, tracking and SLAM alternatives
  3. 15.1.3 Timestamped observations and confidence
  4. 15.1.4 Resource sharing and backpressure
  5. 15.1.5 Compare reflex response with conventional detectors
15.2 Cognition, planning and control · Outline
  1. 15.2.1 VLM/LLM interfaces and grounded observations
  2. 15.2.2 World-model and memory concepts preview
  3. 15.2.3 Task planning, reinforcement learning and reasoning
  4. 15.2.4 Robot/drone controller executes bounded commands
  5. 15.2.5 Missing capabilities must remain explicit

System contracts

15.3 Timing and arbitration · Outline
  1. 15.3.1 Fast and slow loops with explicit budgets
  2. 15.3.2 Message schemas and unit conventions
  3. 15.3.3 Priorities, conflicts and stale plans
  4. 15.3.4 Runtime monitors and safe fallback controllers
  5. 15.3.5 Avoid direct unconstrained language-to-actuator control
15.4 Evaluate the whole system · Outline
  1. 15.4.1 End-to-end tasks and baseline architecture
  2. 15.4.2 Ablate each layer independently
  3. 15.4.3 Failure propagation and recovery
  4. 15.4.4 Revisit after Parts XVI, XVIII and XX
  5. 15.4.5 Measure whether added complexity earns its cost

Read with the original sources.

These readings support the concepts and deeper investigation. The worked numbers and illustrations above are teaching examples.

Official engineering documentation

PX4 — official simulation guide ↗

Begin with simulation and versioned interfaces. Simulation success alone does not establish hardware flight readiness.

Complete source register →

Where next?