Multi-Process Architecture

HORUS topics work transparently across process boundaries. Two nodes in separate processes communicate the same way as two nodes in the same process — through shared memory. No broker, no serialization layer, no configuration.

To orchestrate multiple processes with session discovery, control routing, and e-stop propagation, see Launch System.

// simplified
// Process 1: sensor.rs
let topic: Topic<Imu> = Topic::new("imu")?;
topic.send(imu_reading);

// Process 2: controller.rs
let topic: Topic<Imu> = Topic::new("imu")?;  // same name = same topic
if let Some(reading) = topic.recv() {
    // Got it — zero-config, sub-microsecond
}

How It Works

When you call Topic::new("imu"), HORUS creates (or opens) a shared memory region. Any process on the same machine that calls Topic::new("imu") with the same type connects to the same underlying ring buffer. The shared memory backend is managed by horus_sys — you never configure paths manually.

Every HORUS topic is shared-memory-backed from creation, so a subscriber in the same thread, a different thread, or a different process all connect to the same ring buffer. What actually moves latency is core locality — not the process boundary:

The jump to ~220ns comes from crossing cores, not from crossing the process boundary — two nodes on separate threads inside one process pay the same ~220ns as two processes. No backend or layout choice avoids it; it is cache-coherence physics. The fastest realistic route is to co-locate a producer and consumer on one executor thread.

You don't configure any of this. HORUS picks one of four shared-memory backends automatically from the topology — SpscShm (1 producer : 1 consumer), MpscShm (N producers : 1 consumer), PodShm (1 producer : N consumers, POD broadcast), or FanoutShm (N : M / non-POD broadcast) — as participants join or leave.

Running Multiple Processes

Option 1: horus run with Multiple Files

# Builds and runs both files as separate processes
horus run sensor.rs controller.rs

# Mixed languages work too
horus run sensor.py controller.rs

# With release optimizations
horus run -r sensor.rs controller.rs

horus run compiles each file, then launches all processes and manages their lifecycle (SIGTERM on Ctrl+C, etc.).

Option 2: Separate Terminals

Run each node in its own terminal:

# Terminal 1
horus run sensor.rs

# Terminal 2
horus run controller.rs

Topics auto-discover via shared memory. No coordination needed.

Option 3: horus launch (YAML)

For production, declare your multi-process layout in a launch file:

# launch.yaml
nodes:
  - name: sensor
    cmd: horus run sensor.rs

  - name: controller
    cmd: horus run controller.rs

  - name: monitor
    cmd: horus run monitor.py

horus launch launch.yaml

Example: Two-Process Sensor Pipeline

Process 1 — sensor.rs:

// simplified
use horus::prelude::*;

message! {
    WheelEncoder {
        left_ticks: i64,
        right_ticks: i64,
        timestamp_ns: u64,
    }
}

struct EncoderNode {
    publisher: Topic<WheelEncoder>,
    ticks: i64,
}

impl EncoderNode {
    fn new() -> Result<Self> {
        Ok(Self {
            publisher: Topic::new("wheel.encoders")?,
            ticks: 0,
        })
    }
}

impl Node for EncoderNode {
    fn name(&self) -> &str { "Encoder" }

    fn tick(&mut self) {
        self.ticks += 10;
        self.publisher.send(WheelEncoder {
            left_ticks: self.ticks,
            right_ticks: self.ticks + 2,
            timestamp_ns: horus::now_ns(),
        });
    }
}

fn main() -> Result<()> {
    let mut sched = Scheduler::new().tick_rate(100_u64.hz());
    sched.add(EncoderNode::new()?).order(0).build()?;
    sched.run()?;
    Ok(())
}

Process 2 — odometry.rs:

// simplified
use horus::prelude::*;
use horus_robotics::prelude::*;

message! {
    WheelEncoder {
        left_ticks: i64,
        right_ticks: i64,
        timestamp_ns: u64,
    }
}

struct OdometryNode {
    encoder_sub: Topic<WheelEncoder>,
    odom_pub: Topic<Odometry>,
    last_left: i64,
    last_right: i64,
}

impl OdometryNode {
    fn new() -> Result<Self> {
        Ok(Self {
            encoder_sub: Topic::new("wheel.encoders")?,
            odom_pub: Topic::new("odom")?,
            last_left: 0,
            last_right: 0,
        })
    }
}

impl Node for OdometryNode {
    fn name(&self) -> &str { "Odometry" }

    fn tick(&mut self) {
        if let Some(enc) = self.encoder_sub.recv() {
            let dl = enc.left_ticks - self.last_left;
            let dr = enc.right_ticks - self.last_right;
            self.last_left = enc.left_ticks;
            self.last_right = enc.right_ticks;

            println!("[Odom] delta L={} R={}", dl, dr);
        }
    }
}

fn main() -> Result<()> {
    let mut sched = Scheduler::new().tick_rate(100_u64.hz());
    sched.add(OdometryNode::new()?).order(0).build()?;
    sched.run()?;
    Ok(())
}

Run them:

# Terminal 1
horus run sensor.rs

# Terminal 2
horus run odometry.rs

The WheelEncoder messages flow through shared memory at ~220ns one-way latency (the two processes run on different cores), with zero configuration.

When to Use Multi-Process

Use single-process when:

• All nodes are the same language
• You need deterministic ordering between nodes (e.g., sensor → controller → actuator)
• Latency matters at the nanosecond level
• Simpler deployment is preferred

Use multi-process when:

• Mixing Rust and Python (e.g., Rust motor control + Python ML inference)
• Process isolation is needed (safety-critical separation)
• Independent restart required (update one node without stopping others)
• Different update rates or lifecycle requirements

Introspection

HORUS CLI tools work across processes automatically:

# See all topics (from any process)
horus topic list

# Monitor a topic published by another process
horus topic echo wheel.encoders

# See all running nodes across processes
horus node list

# Check bandwidth across processes
horus topic bw wheel.encoders

Cleaning Up

Shared memory files persist after processes exit. Clean them with:

horus clean --shm    # Remove stale shared memory regions

In practice, you rarely need this — HORUS automatically cleans stale SHM on every horus CLI command and every Scheduler::new() call. The manual command is an escape hatch for debugging.

What Happens When a Process Crashes

When a process dies (even via SIGKILL or power loss):

1. SHM files persist — the kernel closes the file descriptor and releases flock locks, but the mmap'd file stays on disk
2. Other processes continue — subscribers see dropped_count() increase if the publisher was mid-write, but they don't crash
3. Backend auto-migrates — when the crashed process restarts and reconnects, the topic detects the new participant and migrates the backend (e.g., from 1:1 to 1:N) within ~10μs
4. Automatic cleanup — the next horus CLI command or Scheduler::new() call auto-cleans stale namespaces (<1ms). No manual intervention needed.

# Process 1 crashes
# Process 2 keeps running, reading stale data from the ring buffer
# Process 1 restarts
# Process 2 sees fresh data again — no reconfiguration needed

Type mismatches: If a restarted process changes its message type (e.g., from CmdVel to Twist), the join fails with an error. Both processes must use the same message type for the same topic name.

Mixed-Language Multi-Process

The most common multi-process pattern: Rust for control loops, Python for ML inference.

// simplified
use horus::prelude::*;

struct CameraNode {
    pub_img: Topic<Image>,
}

impl CameraNode {
    fn new() -> Result<Self> {
        Ok(Self { pub_img: Topic::new("camera.rgb")? })
    }
}

impl Node for CameraNode {
    fn name(&self) -> &str { "camera" }
    fn tick(&mut self) {
        let mut img = Image::new(640, 480, "rgb8");
        // ... capture from hardware ...
        self.pub_img.send(img);
    }
}

fn main() -> Result<()> {
    let mut sched = Scheduler::new().tick_rate(30_u64.hz());
    sched.add(CameraNode::new()?).order(0).build()?;
    sched.run()
}

Run both:

# Terminal 1 (Rust, 30 FPS camera)
horus run sensor.rs

# Terminal 2 (Python, ML inference)
horus run detector.py

The Image flows through shared memory pool-backed transport — the Python node gets a zero-copy view of the pixels the Rust node wrote. No serialization, no copying.

Debugging Multi-Process Systems

Identify which process owns what

horus topic list --verbose
# Shows publisher/subscriber PIDs per topic

horus node list
# Shows all running nodes across all processes with PID, rate, CPU, memory

Watch cross-process data flow

# Monitor messages from the Rust sensor in the Python process's terminal
horus topic echo camera.rgb

# Measure the actual publishing rate
horus topic hz camera.rgb

# Measure bandwidth
horus topic bw camera.rgb

Debug one process at a time

# Start the sensor normally
horus run sensor.rs

# Start the detector with verbose logging
RUST_LOG=debug horus run detector.py

Use the monitor for a system-wide view

horus monitor
# Web UI at http://localhost:3000 shows ALL nodes from ALL processes
# Topic graph view shows cross-process message flow

Common debugging workflow

1. horus topic list — verify both processes see the same topics
2. horus topic hz <topic> — verify the publisher is sending at expected rate
3. horus topic echo <topic> — verify message content is correct
4. horus node list — verify both nodes are Running (not Error or Crashed)
5. horus bb --anomalies — check for deadline misses or errors

Common Errors

Design Decisions

Why Auto-Discovery via Shared Memory Names

When you call Topic::new("imu") in two separate processes, both connect to the same shared memory region because the topic name deterministically maps to a shared memory path (managed by horus_sys). There is no registration step, no discovery protocol, and no configuration file listing topic endpoints. This works because shared memory is a kernel-level namespace — any process on the same machine that opens the same named region gets the same memory. Auto-discovery eliminates an entire class of misconfiguration bugs ("I forgot to register my topic") and means processes can start and stop in any order.

Why No Broker Process

Message brokers (like DDS in ROS2 or MQTT) add a routing hop between every publisher and subscriber. Even with optimizations, this hop adds latency and creates a single point of failure — if the broker crashes, all communication stops. HORUS uses direct shared memory: publishers write to a ring buffer, subscribers read from it, and no intermediary process routes messages. This gives sub-microsecond latency and means there is no central process that can fail. The cost is that HORUS topics only work on a single machine (cross-machine communication requires an explicit bridge).

Why the Same Transport Works Same-Process and Cross-Process

Every HORUS topic is shared-memory-backed from creation — there is no separate in-process transport that has to be swapped for a cross-process one when a topic spans processes. A publisher and subscriber in the same process talk through the same ring buffer they would use across processes; the only thing that changes with placement is core locality (~12-19ns one-way when co-located on one executor thread, ~220ns one-way once producer and consumer sit on different cores — whether that is two threads or two processes). Users write the same Topic::new("name") call regardless. This means code that works in a single-process prototype deploys to a multi-process production system with zero changes: a cross-process consumer joins an existing topic with no migration. Splitting a monolith into separate processes does not require code changes.

Trade-offs

See Also

• Shared Memory — SHM architecture, ring buffers, platform differences
• Topics (Full Reference) — topic API and backend details
• Message Performance — POD types and zero-copy transport
• Multi-Language — Rust + Python interop patterns
• CLI Reference — horus launch, horus topic list, horus clean