{"title":"Launch System","description":"Multi-process orchestration with full observability — session discovery, control topics, e-stop propagation, and coordinated shutdown","slug":"concepts/launch-system","content":"# Launch System\n\nThe Scheduler runs multiple nodes inside a single process. The launch system runs multiple *processes* on the same machine — each process can contain its own Scheduler with its own nodes.\n\n```yaml\n# formation.yaml — users only write this\nsession: formation\nnodes:\n  - name: controller\n    package: control-pkg\n    rate_hz: 100\n  - name: perception\n    package: perception-pkg\n    rate_hz: 30\n    depends_on: [controller]\n```\n\n```bash\nhorus launch formation.yaml\n```\n\n---\n\n## When to Use Scheduler vs Launch\n\n| Situation | Use |\n|-----------|-----|\n| All nodes in one language, one binary | **Scheduler only** |\n| Mixed Rust + Python nodes | **Launch** (separate processes) |\n| Crash isolation (camera crash shouldn't kill motor) | **Launch** (process boundaries) |\n| Multiple robots on one machine | **Launch** (namespaced sessions) |\n| Simulation testing, deterministic replay | **Scheduler** (`tick_once()`, `deterministic(true)`) |\n| Field deployment on embedded hardware | **Launch** (swap nodes by editing YAML, no recompile) |\n| Maximum performance (sub-microsecond IPC) | **Scheduler** (in-process backends: 3-36ns) |\n\n**Rule of thumb:** Start with Scheduler. Move to launch when you need process isolation, mixed languages, or runtime node composition without recompiling.\n\n---\n\n## How Launch Integrates with HORUS\n\nLaunch is not a separate system — it is wired into the same observability infrastructure as the Scheduler. When you run `horus launch`, this is what happens:\n\n### 1. Process Spawning\n\nLaunch parses the YAML, resolves dependencies via topological sort, and spawns each node as a separate OS process. Each process receives environment variables:\n\n- `HORUS_NODE_NAME` — the Scheduler reads this as its default name\n- `HORUS_NAMESPACE` — shared memory namespace isolation\n- `HORUS_PARAM_*` — parameters from the launch YAML, consumed by `RuntimeParams::new()`\n\n### 2. Session Registry (Auto-Discovery)\n\nAfter spawning, launch writes a session manifest to shared memory:\n\n```\n/dev/shm/horus_{namespace}/launch/{session}.json\n```\n\nIt then polls node presence files to discover which Schedulers appeared. Once a process creates a Scheduler, launch auto-detects it and records the Scheduler name, control topic, and node list — all without any configuration from the user.\n\n```bash\n# See active sessions\nhorus launch --status\n```\n\n```\nACTIVE LAUNCH SESSIONS\n\n  ● formation (3 processes, uptime: 5m 32s)\n    File: formation.yaml\n\n    NAME                 PID      STATUS     SCHEDULER            RESTARTS\n    controller           12346    running    controller           0\n    perception           12347    running    perception           0\n    drivers              12348    running    drivers              0\n```\n\n### 3. Control Topics\n\nLaunch creates a session-level control topic:\n\n```\nhorus.launch.ctl.{session}\n```\n\nCommands sent to this topic are routed to the correct per-Scheduler control topic. This means `StopNode(\"controller\")` goes to `horus.ctl.controller` automatically — CLI tools don't need to know which Scheduler owns which node.\n\n### 4. E-Stop Propagation\n\nLaunch subscribes to the `_horus.estop` topic. When any node triggers an emergency stop:\n\n1. Launch sends `GracefulShutdown` to every discovered Scheduler control topic\n2. Schedulers run their full shutdown sequence (node `shutdown()` callbacks, blackbox flush, presence cleanup)\n3. If a process doesn't exit within 2 seconds, launch sends SIGTERM\n4. If still alive after 3 more seconds, SIGKILL\n\n### 5. Coordinated Shutdown\n\nPressing Ctrl+C triggers the same three-phase shutdown:\n\n```\nPhase A: GracefulShutdown → all Scheduler control topics (2s grace)\nPhase B: SIGTERM → any survivors (3s grace)\nPhase C: SIGKILL → any still alive\n```\n\nThis ensures nodes get their `shutdown()` callback called, hardware is released, files are flushed, and presence files are cleaned up — instead of raw SIGKILL losing everything.\n\n### 6. Blackbox Events\n\nLaunch records lifecycle events to a JSONL log:\n\n```\n/dev/shm/horus_{namespace}/launch/{session}.events.jsonl\n```\n\nEvents: `SessionStart`, `NodeSpawned`, `NodeCrashed`, `NodeRestarted`, `SessionStop`. Each includes nanosecond timestamps and process details. View with:\n\n```bash\ncat /dev/shm/horus_*/launch/*.events.jsonl | jq .\n```\n\n### 7. CLI Integration\n\nNodes spawned by launch appear in standard CLI tools with session grouping:\n\n```bash\nhorus node list          # groups nodes under \"Launch: session (N nodes)\"\nhorus topic echo <topic> # live data from any launched process\nhorus topic list         # shows all topics across all launched processes\nhorus param get <key>    # reads params from launched processes\nhorus param set <key> <v># sets params at runtime on launched processes\nhorus launch --status    # shows all active sessions with per-process table\nhorus launch --stop <s>  # stops a session from any terminal\nhorus launch --list <f>  # dry-run: shows nodes and dependencies\n```\n\nExample output of `horus node list` with a running launch session:\n\n```\nRunning Nodes:\n\n  Launch: formation (3 nodes)\n  NAME                        STATUS   PRIORITY     RATE      TICKS   SOURCE\n  ---------------------------------------------------------------------------\n  pid_loop                   Running          0    100 Hz       5032   12346\n  planner                    Running          0     50 Hz       2516   12346\n  detector                   Running        100     30 Hz       1510   12347\n\n  Total: 3 node(s)\n```\n\n### 8. Node Kill Routing\n\n`horus node kill <name>` automatically detects if a node belongs to a launch session. If it does, the command routes through the launch control topic (`horus.launch.ctl.{session}`) so the launch monitor can track the stop, update the session manifest, and potentially restart the process. Non-launched nodes use the direct Scheduler control path as before.\n\n### 9. Network Replication\n\nLaunched processes can enable `horus_net` for LAN topic replication:\n\n```yaml\nsession: fleet\nenv:\n  HORUS_NET_ENABLED: \"true\"\nnodes:\n  - name: robot_a\n    command: ./target/release/controller\n  - name: robot_b\n    command: ./target/release/controller\n```\n\nNetwork replication is opt-in — build with `--features net`. When enabled, it is on by default at runtime; use `.network(false)` to disable. The launch system discovers networked Schedulers the same way as local ones — via SHM presence files and the scheduler directory.\n\n---\n\n## Launch File Reference\n\n```yaml\n# Session name (used for SHM manifest, control topic, event log)\nsession: my_robot\n\n# Global namespace (all topics prefixed)\nnamespace: robot_1\n\n# Global environment variables (applied to all nodes)\nenv:\n  HORUS_LOG_LEVEL: info\n\nnodes:\n  - name: controller          # Required: unique name\n    package: control-pkg      # Run via `horus run control-pkg`\n    # OR\n    command: \"python ctrl.py\" # Run a custom command\n\n    rate_hz: 100              # Target tick rate (hint — Scheduler controls actual rate)\n    priority: 0               # Launch order (lower = earlier)\n    namespace: /arm           # Per-node namespace prefix\n\n    params:                   # Injected as HORUS_PARAM_* env vars\n      max_speed: 1.5\n      robot_id: 1\n\n    env:                      # Extra environment variables\n      CUDA_VISIBLE_DEVICES: \"0\"\n\n    depends_on: [sensor]      # Wait for these nodes to start first\n    start_delay: 0.5          # Seconds to wait before launching\n    restart: on-failure       # \"never\" (default), \"always\", \"on-failure\"\n    args: [--verbose]         # Extra command-line arguments\n```\n\n---\n\n## Parameters: Launch YAML to Node Code\n\nParameters in the launch YAML reach your node code automatically:\n\n```yaml\n# launch.yaml\nnodes:\n  - name: controller\n    package: control-pkg\n    params:\n      max_speed: 1.5\n      robot_id: 1\n```\n\n```rust\n// In your node's init()\nlet params = RuntimeParams::new()?;\nlet speed: f64 = params.get(\"max_speed\").unwrap_or(1.0);\nlet id: i64 = params.get(\"robot_id\").unwrap_or(0);\n```\n\n```python\n# In your Python node\nparams = horus.Params()\nspeed = params.get(\"max_speed\", default=1.0)\n```\n\nThe launch system sets `HORUS_PARAM_MAX_SPEED=1.5` and `HORUS_PARAM_ROBOT_ID=1` as environment variables. `RuntimeParams::new()` reads all `HORUS_PARAM_*` variables automatically, parsing them as the appropriate type (number, boolean, or string).\n\n---\n\n## Restart Policies\n\n| Policy | Behavior |\n|--------|----------|\n| `never` (default) | Process exits, launch records the event, moves on |\n| `on-failure` | Restart only if exit code is non-zero. Exponential backoff (100ms to 10s). Max 10 restarts. |\n| `always` | Restart on any exit. Same backoff and max restarts. |\n\n---\n\n## Compared to ROS 2\n\n| Feature | ROS 2 Launch | HORUS Launch |\n|---------|-------------|--------------|\n| Config format | Python scripts or XML | YAML only |\n| Process management | roslaunch / ros2 launch | `horus launch` |\n| Parameter injection | Launch parameters → node params | `HORUS_PARAM_*` → `RuntimeParams` |\n| Session discovery | N/A | Auto-discovered from SHM |\n| Control routing | N/A | `horus.launch.ctl.{session}` topic |\n| E-stop propagation | Custom | Built-in, wired to `_horus.estop` |\n| Coordinated shutdown | SIGINT only | Control topic → SIGTERM → SIGKILL |\n| Event logging | rosout | JSONL blackbox per session |\n| Observability | `ros2 node list` (separate) | `horus node list` shows session grouping |\n\nThe key difference: ROS 2 launch is a process spawner that delegates everything to DDS. HORUS launch is wired into the SHM observability stack — it discovers Schedulers, routes control commands, propagates safety events, and records lifecycle data.\n\n---\n\n## Mixed Language Example\n\nThe primary reason launch exists — running Rust and Python nodes together:\n\n```yaml\n# mixed_robot.yaml\nsession: mixed_robot\nnodes:\n  - name: controller\n    command: ./target/release/motor_controller\n    params:\n      max_speed: 1.5\n      pid_kp: 2.0\n\n  - name: perception\n    command: python3 ml_detector.py\n    depends_on: [controller]\n    params:\n      model: yolov8n\n      confidence: 0.7\n```\n\nBoth processes communicate via SHM topics — the Rust controller publishes `cmd_vel`, the Python ML node subscribes to `camera.image` and publishes `detections`. All visible in `horus topic list`, all controllable via `horus launch --stop`.\n\n---\n\n## CLI Quick Reference\n\n```bash\n# Launch\nhorus launch robot.yaml              # start all nodes\nhorus launch robot.yaml --dry-run    # show plan without starting\nhorus launch robot.yaml --list       # show nodes and dependencies\nhorus launch robot.yaml --namespace r1  # override namespace\n\n# Monitor\nhorus launch --status                # show all active sessions\nhorus node list                      # show nodes grouped by session\nhorus topic list                     # show topics from all processes\nhorus topic echo <topic>             # stream live data\n\n# Control\nhorus launch --stop <session>        # stop a session from any terminal\nhorus node kill <name>               # stop a node (routes via launch if applicable)\n\n# Debug\nhorus param get <key>                # read runtime params\nhorus param set <key> <value>        # set params at runtime\ncat /dev/shm/horus_*/launch/*.events.jsonl | jq .  # view lifecycle events\n```","headings":["Launch System","formation.yaml — users only write this","When to Use Scheduler vs Launch","How Launch Integrates with HORUS","1. Process Spawning","2. Session Registry (Auto-Discovery)","See active sessions","3. Control Topics","4. E-Stop Propagation","5. Coordinated Shutdown","6. Blackbox Events","7. CLI Integration","8. Node Kill Routing","9. Network Replication","Launch File Reference","Session name (used for SHM manifest, control topic, event log)","Global namespace (all topics prefixed)","Global environment variables (applied to all nodes)","Parameters: Launch YAML to Node Code","launch.yaml","In your Python node","Restart Policies","Compared to ROS 2","Mixed Language Example","mixedrobot.yaml","CLI Quick Reference","Launch","Monitor","Control","Debug"],"wordCount":1538}