Production Deployment

Guide to deploying HORUS applications on production robots with real-time constraints, safety monitoring, and optimal performance.

Deployment Checklist

Four steps to go from development to production.

1. Enable Safety Monitoring

The .watchdog(Duration) method registers every RT node with the safety monitor's frozen-node detector. (The safety monitor itself is created on every run() whether or not you call .watchdog(), so emergency stop always works -- but no node is watched for freezes until you set a timeout.) Budget enforcement is implicit when nodes have .rate() set. Set the watchdog timeout on the scheduler builder:

use horus::prelude::*;

let mut scheduler = Scheduler::new()
    .watchdog(500_u64.ms())
    .tick_rate(100_u64.hz());

This is the single most important setting for production. The watchdog detects hung nodes and triggers graduated degradation automatically.

2. Configure Real-Time (Optional)

HORUS provides two RT modes that enable OS-level SCHED_FIFO scheduling and mlockall() memory locking:

// Graceful: try RT, warn and continue if unavailable
let mut scheduler = Scheduler::new()
    .prefer_rt()
    .watchdog(500_u64.ms())
    .tick_rate(100_u64.hz());

// Strict: panic if the system lacks RT capabilities
let mut scheduler = Scheduler::new()
    .require_rt()
    .watchdog(500_u64.ms())
    .tick_rate(1000_u64.hz());

Use .prefer_rt() in most cases. It tries to enable RT features and records any that failed as degradations rather than errors.

Use .require_rt() only for hard real-time deployments, where you need a guarantee that RT is actually active. It panics the moment you call it if the system offers neither SCHED_FIFO nor mlockall, and run() returns an error if either of those two features later fails to apply -- so a safety-critical system never runs in degraded mode silently.

Even without .prefer_rt(), the scheduler auto-enables mlockall when RT nodes are present and the system permits it. This prevents 10-100 ms page fault spikes under memory pressure.

3. Set Node Budgets

Assign tick budgets and deadlines to safety-critical nodes. Setting either .budget() or .deadline() automatically promotes the node to the RT execution class -- but only when no execution class was set explicitly. Combining them with .compute(), .async_io() or .on() is a validation error and .build() fails:

scheduler.add(motor_controller)
    .rate(1000_u64.hz())
    .budget(200_u64.us())       // Must finish within 200 us
    .deadline(800_u64.us())     // Absolute latest: 800 us
    .on_miss(Miss::SafeMode)    // Enter safe state on miss
    .build()?;

scheduler.add(lidar_processor)
    .rate(20_u64.hz())
    .budget(10_u64.ms())
    .on_miss(Miss::Skip)        // Drop this tick, continue next
    .build()?;

The Miss policy controls what happens on deadline miss:

4. Enable Flight Recorder

The blackbox records scheduler events to an in-memory ring for post-mortem analysis:

let mut scheduler = Scheduler::new()
    .watchdog(500_u64.ms())
    .blackbox(64)             // bounds the in-memory ring only — see below
    .tick_rate(100_u64.hz());

Data is written to .horus/blackbox/ in the working directory. See the Blackbox page for analysis tools.

Binary Size and Flash Budget

By default every HORUS executable statically links the whole runtime, with nothing to install next to it. That is what makes a deployed binary a single file you can scp onto a robot, and it is also why the debug build is enormous. (C++ projects can opt into a shared runtime instead — see below.) Plan flash against the release number, and never ship a debug build to a target.

Measured on x86-64 Linux (GCC 15, glibc) with the one-node hello-world horus new scaffolds — a single node, one publisher, one tick():

The release C++ figure is the result of three link settings the generated CMakeLists.txt applies for you: -ffunction-sections -fdata-sections, -Wl,--gc-sections, and -s in Release only. Without them the same binary is 17.6 MB; with --gc-sections but no -s, 6.7 MB. Nothing is asked of you — horus build --release sets all three.

If you need to go smaller than the release figure:

• Strip the Rust binary. The C++ release build is stripped for you by that -s, so strip gains nothing there. The Rust one is not: strip -s takes it from 2.9 MB to 2.1 MB, which is the same place C++ lands. Keep the symbols if you want a readable backtrace out of the field; that is what the 0.8 MB buys.
• Drop capabilities you do not run. horus run --enable net links in the networking stack; every other --enable name is forwarded to cargo as a feature of your own crate, so it links in only what you declared under [rust.features]. Build the deployment target with only what the robot uses.
• Share the runtime across processes. C++ projects can link the runtime dynamically instead of statically: set link = "shared" under [cpp] in horus.toml, or export HORUS_CPP_LINK=shared for a single build ("dynamic" is accepted as a synonym; an unrecognised value is an error, not a silent fall back to static). That takes each node binary from about 2 MB to tens of kilobytes, plus one shared libhorus_cpp.so the whole robot loads — so it pays from the second executable onwards. The cost is that the library has to travel with the binaries: it is built into the HORUS source tree's target/, not your project directory, so horus deploy's rsync does not pick it up. Copy it to the robot yourself and point LD_LIBRARY_PATH at its directory. Static stays the default: it needs no loader path, no extra deployment step, and no version skew between a binary and a library beside it.

RT Kernel Setup (Linux)

For hard real-time guarantees, you need a PREEMPT_RT kernel.

Installing the RT Kernel

The package name differs by distribution:

# Ubuntu
sudo apt install linux-image-realtime

# Debian
sudo apt install linux-image-rt-amd64      # linux-image-rt-arm64 on 64-bit ARM

sudo reboot

Or let horus setup-rt choose for you — it detects the distribution (Ubuntu, Debian, Fedora and Arch) and asks before installing. It asks on a terminal only: a provisioning script that runs it without a TTY gets a decline and no kernel, unless it sets HORUS_ASSUME_YES=1.

Verify after reboot:

uname -a   # Should show "PREEMPT_RT" in the output

Setting RT Permissions

On the robot, horus setup-rt writes a user-scoped /etc/security/limits.d/99-horus-rt.conf for whoever invokes it (--check first to see what is missing, --undo to remove that file). The realtime-group recipe below is the alternative, and the one to use if several users need RT on the same machine: add your robot user to the realtime group and raise the memlock limit in /etc/security/limits.conf:

@realtime   -   rtprio     99
@realtime   -   memlock    unlimited
@realtime   -   nice       -20

Then add the user:

sudo groupadd -f realtime
sudo usermod -aG realtime robot_user

Log out and back in for the changes to take effect.

Verifying RT Capabilities

HORUS prints its RT capability detection at startup. Look for these lines:

[SCHEDULER] Memory locked (mlockall)
[SCHEDULER] RT scheduling enabled (SCHED_FIFO, priority 50)
[SCHEDULER] CPU affinity set to cores [2, 3]

If any feature is unavailable with .prefer_rt(), it appears as a degradation warning instead of an error.

Example: Production Robot Configuration

A production configuration combining every step of the checklist above:

use horus::prelude::*;

fn main() -> Result<()> {
    let mut scheduler = Scheduler::new()
        .prefer_rt()
        .watchdog(500_u64.ms())
        .blackbox(64)
        .max_deadline_misses(5)
        .tick_rate(1000_u64.hz());

    // Safety-critical: motor control at 1 kHz
    scheduler.add(MotorController::new())
        .order(0)
        .rate(1000_u64.hz())
        .budget(200_u64.us())
        .deadline(800_u64.us())
        .on_miss(Miss::SafeMode)
        .build()?;

    // Important: state estimation at 200 Hz
    scheduler.add(StateEstimator::new())
        .order(10)
        .rate(200_u64.hz())
        .on_miss(Miss::Warn)
        .build()?;

    // Best-effort: telemetry at 10 Hz
    scheduler.add(TelemetryPublisher::new())
        .order(200)
        .rate(10_u64.hz())
        .async_io()
        .build()?;

    scheduler.run()?;
    Ok(())
}

.on_miss() applies only to RT nodes. A node that sets .rate(), .budget() or .deadline() without an explicit execution class -- like the motor and estimator nodes above -- is promoted to RT automatically, and whichever of budget and deadline you leave unset is derived from the rate (80% and 95% of the period), so its .on_miss() policy is live. On .compute()/.async_io()/.on() nodes there is no deadline to miss, so the telemetry node above omits it.

Graceful Degradation

When .prefer_rt() is used and a feature cannot be applied, it is recorded as a degradation, not an error. The RT features that can degrade independently are:

• RT Priority -- SCHED_FIFO scheduling (also covers a missing PREEMPT_RT kernel and a clamped priority)
• Memory Locking -- mlockall() to prevent page faults
• CPU Affinity -- pinning to specific cores (set via .cores(&[..]) or HORUS_RT_CORES)

The scheduler continues running with whichever features succeeded. This lets you develop on a laptop (no RT kernel) and deploy on a production system (full RT) without changing code.

Monitoring in Production

Safety Statistics

Query the safety monitor for budget overruns and deadline misses:

if let Some(stats) = scheduler.safety_stats() {
    println!("State: {:?}", stats.state());
    println!("Budget overruns: {}", stats.budget_overruns());
    println!("Deadline misses: {}", stats.deadline_misses());
    println!("Watchdog expirations: {}", stats.watchdog_expirations());
}

The safety monitor applies graduated degradation automatically: warn, then reduce rate, then isolate the node and latch an emergency stop. It also queues enter_safe_state() on the node — but that runs on the node's own executor thread, so a node hung inside tick() may never execute it. The emergency stop does not depend on it. See Watchdogs.

Node Metrics

Get per-node performance data:

for m in scheduler.metrics() {
    println!("{}: avg={:.3} ms", m.name(), m.avg_tick_duration_ms());
}

Blackbox for Post-Mortem

After an incident, the blackbox contains a timestamped log of scheduler events including scheduler start/stop, budget violations, deadline misses, node errors, and emergency stops (a watchdog escalation shows up as an EmergencyStop event, not as its own event type). See Blackbox for details.

See Also

• Safety Monitor -- watchdog, budget enforcement, and deadline miss policies
• Scheduler Configuration -- full builder API reference
• BlackBox Flight Recorder -- flight recorder and post-mortem analysis