Safety Monitor

The Safety Monitor provides real-time safety monitoring for safety-critical robotics applications. It enforces timing constraints, monitors node health, and triggers emergency stops when safety violations occur.

Overview

The Safety Monitor includes:

• Watchdogs: Monitor node liveness — trigger emergency stop if a critical node hangs
• WCET Enforcement: Worst-Case Execution Time budgets — halt if a node takes too long
• Deadline Tracking: Count deadline misses and trigger emergency stop at threshold
• Emergency Stop: Immediate system halt on critical failures

The Scheduler manages the safety monitor internally — you enable it via builder methods and the scheduler automatically feeds watchdogs, checks WCET, and triggers emergency stops.

Enabling Safety Monitoring

The .watchdog() builder method on Scheduler::new() is what arms the watchdogs — at startup the scheduler registers one for every RT node and treats those nodes as critical. Budget and deadline checks need no flag: they are always active for nodes that have .rate() set.

use horus::prelude::*;
use std::time::Duration;

// Zero tolerance for deadline misses (surgical / medical)
let mut scheduler = Scheduler::new()
    .watchdog(Duration::from_millis(100))
    .max_deadline_misses(1)
    .require_rt();

// Tolerates a few deadline misses (industrial / CNC)
let mut scheduler = Scheduler::new()
    .watchdog(Duration::from_millis(10))
    .max_deadline_misses(10)
    .prefer_rt();

Safety Builder Methods

Configuring Critical Nodes

After configuring the scheduler, register critical nodes with timing constraints using the node builder:

use horus::prelude::*;
use std::time::Duration;

let mut scheduler = Scheduler::new()
    .watchdog(Duration::from_millis(100));

// RT node with WCET budget, deadline and its own watchdog
scheduler.add(motor_controller)
    .order(0)
    .rate(1000_u64.hz())      // 1 kHz → RT class
    .budget(500_u64.us())     // 500μs max execution time
    .deadline(1000_u64.us())  // 1ms deadline
    .watchdog(50_u64.ms())    // registers this node as critical
    .build()?;

scheduler.add(sensor_fusion)
    .order(1)
    .rate(200_u64.hz())       // 200 Hz → RT class
    .budget(2000_u64.us())
    .deadline(5000_u64.us())
    .watchdog(100_u64.ms())   // registers this node as critical
    .build()?;

scheduler.run()?;

Watchdogs

Watchdogs monitor node liveness. The response to a missed heartbeat is graduated — a single expiry only warns; an emergency stop is triggered for a critical node once it is 3× past its watchdog timeout.

Where the node runs decides what the watchdog actually detects. A node on an executor (RT, compute, event, async I/O) is fed after a successful tick, so a panic or a hang lets its watchdog expire — which is the behaviour you want. A node on the main loop — a BestEffort node given its own .watchdog(), and every node under .deterministic(true) — is fed before its tick runs. Its watchdog therefore detects a hang, but not repeated tick failures: the feed has already happened by the time the tick fails.

Executor-owned node:
  Node tick → success → watchdog fed → timer reset

Main-loop node:
  watchdog fed → Node tick (success or failure) → timer already reset

Failure scenario:
  1× timeout → Warning   (logged, node keeps ticking)
  2× timeout → Unhealthy (node skipped in the tick loop)
  3× timeout → Critical  (node isolated; EMERGENCY STOP latched
                          for critical nodes)

Timeout Guidelines

Watchdog timeout should be:
  - Longer than expected execution time
  - At most one third of the safety-critical response time
    (emergency stop fires at 3× the configured timeout)

Example:
  Expected tick period:  10ms
  Safety deadline:      100ms
  Watchdog timeout:      30ms  (3× period; e-stop at ~90ms)

WCET Enforcement

Worst-Case Execution Time (WCET) budgets ensure nodes complete within time limits. Set budgets via the node builder:

scheduler.add(motor_controller)
    .order(0)
    .rate(1000_u64.hz())      // 1 kHz → RT class
    .budget(500_u64.us())     // 500μs max execution time
    .deadline(1000_u64.us())  // 1ms deadline
    .build()?;

For critical nodes, WCET violations trigger an emergency stop. For non-critical nodes, violations are logged and recorded in the BlackBox.

Emergency Stop

Emergency stop is triggered automatically by:

• Critical node watchdog reaching 3× its timeout
• Critical node WCET violation
• Critical node deadline miss
• Exceeding the max_deadline_misses threshold

When emergency stop triggers:

1. All node execution is halted
2. An emergency stop event is recorded in the BlackBox
3. The scheduler transitions to emergency state

Everything above is local to one machine and needs no configuration. If you run a fleet with the net capability enabled, an e-stop is also broadcast to peers over _horus.estop — and that channel is authenticated. A node that has no HORUS_ESTOP_KEY provisioned rejects incoming remote e-stops rather than acting on them, so a fleet-wide halt will not reach it. See Authenticating the networked e-stop.

Inspecting After Emergency Stop

use horus::prelude::*;

let mut scheduler = Scheduler::new()
    .watchdog(100_u64.ms())
    .blackbox(64);   // 64 MB flight recorder — without it there is nothing to inspect

// ... application runs and hits emergency stop ...

// Inspect what happened via BlackBox.
// `get_blackbox()` is a `#[doc(hidden)]` internal and returns `None`
// unless `.blackbox(size_mb)` was set on the builder above.
if let Some(bb) = scheduler.get_blackbox() {
    let anomalies = bb.lock().expect("blackbox mutex poisoned").anomalies();
    println!("=== SAFETY EVENTS ({}) ===", anomalies.len());
    for record in &anomalies {
        println!("[tick {}] {:?}", record.tick, record.event);
    }
}

Best Practices

1. Start with Conservative Budgets

Set generous WCET budgets initially, then tighten after profiling:

// Start: 3× expected execution time
scheduler.add(motor).budget(1500_u64.us()).build()?;

// After profiling: 2× measured worst case
scheduler.add(motor).budget(1000_u64.us()).build()?;

2. Layer Safety Checks

Use watchdogs (liveness) and WCET budgets (timing) together:

// Both work together to catch different failure modes
scheduler.add(motor)
    .rate(1000_u64.hz())
    .watchdog(50_u64.ms())   // Liveness — also registers `motor` as critical
    .budget(500_u64.us())    // Timing
    .build()?;

3. Choose the Right Configuration

4. Test Your Safety Configuration

Verify that your critical nodes register with the timing constraints you expect:

#[test]
fn test_safety_critical_setup() {
    let mut scheduler = Scheduler::new()
        .watchdog(100_u64.ms());

    scheduler.add(test_node)
        .order(0)
        .rate(1000_u64.hz())
        .budget(100_u64.us())
        .watchdog(10_u64.ms())   // registers this node as critical
        .build()
        .expect("should register critical node");
}

See Also

• Scheduling - Scheduler overview and node ordering
• BlackBox Flight Recorder - Event recording for post-mortem analysis
• Circuit Breaker - Fault tolerance patterns
• Scheduler Configuration - Presets and node configuration
• Network Configuration - Fleet e-stop authentication and link-loss behaviour