Safety Monitor

The Safety Monitor enforces the timing contracts you declare on your nodes: it tracks node liveness, checks tick budgets and deadlines, and triggers an emergency stop once a violation threshold is crossed. It is fault-detection machinery, not a safety qualification — see Certification Status.

Overview

The Safety Monitor includes:

• Watchdogs: Monitor node liveness — trigger emergency stop if a critical node hangs
• WCET Enforcement: Worst-Case Execution Time budgets — halt if a node takes too long
• Deadline Tracking: Count deadline misses and trigger emergency stop at threshold
• Emergency Stop: Immediate system halt on critical failures

The Scheduler manages the safety monitor internally — you enable it via builder methods and the scheduler automatically feeds watchdogs, checks WCET, and triggers emergency stops.

Enabling Safety Monitoring

The .watchdog() builder method on Scheduler::new() is what arms the watchdogs — at startup the scheduler registers one for every RT node and treats those nodes as critical. Budget and deadline checks need no flag: they are always active for nodes that have .rate() set.

use horus::prelude::*;
use std::time::Duration;

// Zero tolerance for deadline misses — the first miss trips the e-stop
let mut scheduler = Scheduler::new()
    .watchdog(Duration::from_millis(100))
    .max_deadline_misses(1)
    .require_rt();

// Tolerates a few deadline misses — 10 before the e-stop trips
let mut scheduler = Scheduler::new()
    .watchdog(Duration::from_millis(10))
    .max_deadline_misses(10)
    .prefer_rt();

Safety Builder Methods

Watchdog resolution is 1 ms — the scheduler config stores whole milliseconds. Duration::ZERO disables the watchdog entirely. A non-zero timeout below 1 ms is raised to 1 ms and logged as a warning, because truncating it to 0 would silently turn the watchdog off. Anything else truncates toward zero with no diagnostic, so .watchdog(1500_u64.us()) runs a 1 ms watchdog and e-stops at 3 ms rather than the 4.5 ms you asked for. Sub-millisecond watchdogs are not supported.

Configuring Critical Nodes

After configuring the scheduler, register critical nodes with timing constraints using the node builder:

use horus::prelude::*;
use std::time::Duration;

let mut scheduler = Scheduler::new()
    .watchdog(Duration::from_millis(100));

// RT node with WCET budget, deadline and its own watchdog
scheduler.add(motor_controller)
    .order(0)
    .rate(1000_u64.hz())      // 1 kHz → RT class
    .budget(500_u64.us())     // 500μs max execution time
    .deadline(1000_u64.us())  // 1ms deadline
    .watchdog(50_u64.ms())    // registers this node as critical
    .build()?;

scheduler.add(sensor_fusion)
    .order(1)
    .rate(200_u64.hz())       // 200 Hz → RT class
    .budget(2000_u64.us())
    .deadline(5000_u64.us())
    .watchdog(100_u64.ms())   // registers this node as critical
    .build()?;

scheduler.run()?;

Watchdogs

Watchdogs monitor node liveness. The response to a missed heartbeat is graduated — a single expiry only warns; an emergency stop is triggered for a critical node once it is 3× past its watchdog timeout.

Every node is fed the same way, wherever it runs: after its tick, and only if that tick returned Ok. A panic or a hang therefore lets the watchdog expire, which is the behaviour you want.

Any node, executor-owned or main-loop:
  Node tick → Ok → watchdog fed → timer reset
              ↓
            Err / panic / hang → not fed → timer runs out

Failure scenario:
  1× timeout → Warning   (logged, node keeps ticking)
  2× timeout → Unhealthy
  3× timeout → Critical  (node isolated; EMERGENCY STOP latched
                          for critical nodes)

Timeout Guidelines

Watchdog timeout should be:
  - Longer than expected execution time
  - At most one third of the safety-critical response time
    (emergency stop fires at 3× the configured timeout)

Example:
  Expected tick period:  10ms
  Safety deadline:      100ms
  Watchdog timeout:      30ms  (3× period; e-stop at ~90ms)

WCET Enforcement

Worst-Case Execution Time (WCET) budgets ensure nodes complete within time limits. Set budgets via the node builder:

scheduler.add(motor_controller)
    .order(0)
    .rate(1000_u64.hz())      // 1 kHz → RT class
    .budget(500_u64.us())     // 500μs max execution time
    .deadline(1000_u64.us())  // 1ms deadline
    .build()?;

WCET violations are recorded (console, BlackBox, RtStats, the safety monitor's counters) but do not by themselves stop the robot, and being a critical node does not change that. Escalation is per-node and explicit via .budget_policy(..): BudgetPolicy::Warn (the default) logs only, BudgetPolicy::Enforce shuts down the offending node after a 2x overrun, and BudgetPolicy::EmergencyStop triggers the emergency stop. Every violation is also recorded in the BlackBox.

Emergency Stop

Emergency stop is triggered automatically by:

• Critical node watchdog reaching 3× its timeout
• A budget violation on a node configured with .budget_policy(BudgetPolicy::EmergencyStop)
• A critical node that panics in enter_safe_state() or shutdown() and so cannot reach a safe state
• Exceeding the max_deadline_misses threshold
• A deadline miss on a node configured with .on_miss(Miss::Stop) — this fires on the first miss, on both the main loop and the RT executor, and does not wait for max_deadline_misses

When emergency stop triggers:

1. All node execution is halted
2. An emergency stop event is recorded in the BlackBox
3. The scheduler transitions to emergency state

Everything above is local to one machine and needs no configuration. If you run a fleet with the net capability enabled, an e-stop is also broadcast to peers over _horus.estop — and that channel is authenticated. A node that has no HORUS_ESTOP_KEY provisioned rejects incoming remote e-stops rather than acting on them, so a fleet-wide halt will not reach it. See Authenticating the networked e-stop.

Inspecting After Emergency Stop

use horus::prelude::*;

let mut scheduler = Scheduler::new()
    .watchdog(100_u64.ms())
    .blackbox(64);   // 64 MB flight recorder — without it there is nothing to inspect

// ... application runs and hits emergency stop ...

// Inspect what happened via BlackBox.
// `get_blackbox()` is a `#[doc(hidden)]` internal and returns `None`
// unless `.blackbox(size_mb)` was set on the builder above.
if let Some(bb) = scheduler.get_blackbox() {
    let anomalies = bb.lock().expect("blackbox mutex poisoned").anomalies();
    println!("=== SAFETY EVENTS ({}) ===", anomalies.len());
    for record in &anomalies {
        println!("[tick {}] {:?}", record.tick, record.event);
    }
}

Best Practices

1. Start with Conservative Budgets

Set generous WCET budgets initially, then tighten after profiling:

// Start: 3× expected execution time
scheduler.add(motor).budget(1500_u64.us()).build()?;

// After profiling: 2× measured worst case
scheduler.add(motor).budget(1000_u64.us()).build()?;

2. Layer Safety Checks

Use watchdogs (liveness) and WCET budgets (timing) together:

// Both work together to catch different failure modes
scheduler.add(motor)
    .rate(1000_u64.hz())
    .watchdog(50_u64.ms())   // Liveness — also registers `motor` as critical
    .budget(500_u64.us())    // Timing
    .build()?;

3. Choose the Right Configuration

4. Test Your Safety Configuration

Verify that your critical nodes register with the timing constraints you expect:

#[test]
fn test_safety_critical_setup() {
    let mut scheduler = Scheduler::new()
        .watchdog(100_u64.ms());

    scheduler.add(test_node)
        .order(0)
        .rate(1000_u64.hz())
        .budget(100_u64.us())
        .watchdog(10_u64.ms())   // registers this node as critical
        .build()
        .expect("should register critical node");
}

See Also

• Scheduling - Scheduler overview and node ordering
• BlackBox Flight Recorder - Event recording for post-mortem analysis
• Circuit Breaker - Fault tolerance patterns
• Scheduler Configuration - Presets and node configuration
• Network Configuration - Fleet e-stop authentication and link-loss behaviour