Real-Time Nodes

HORUS provides industrial-grade real-time support for time-critical robotics applications through per-node timing constraints and safety monitoring infrastructure.

Overview

Real-time nodes give a node an explicit timing contract — a period, a tick budget and a deadline — plus a declared response when it is missed. Reach for them when:

A loop has to run at a fixed rate, and a late tick is a fault rather than a delay
A hung node has to be detected rather than waited on
A deadline miss needs a defined response (warn, skip, safe state, stop) instead of silent drift

The contract is enforced, not guaranteed: HORUS measures overruns and reacts to them, but nothing in the repository bounds worst-case latency. See Certification Status.

Key Features

Real-Time Registration

There is no separate real-time trait. Nodes implement the standard Node trait, and a node becomes real-time when it is registered with timing constraints:

use horus::prelude::*; // Provides {Scheduler, Node, Miss, RtStats, DurationExt, FailurePolicy, RtConfig, RtScheduler}
use std::time::Duration;

let mut scheduler = Scheduler::new();

scheduler.add(YourControlNode::new())
    .order(0)                            // No effect here: .rate() moves this node
                                         // off the main tick loop to the RT executor
    .rate(1000_u64.hz())                 // 1kHz control loop
    .budget(Duration::from_micros(100))  // 100μs worst-case execution time
    .deadline(Duration::from_millis(1))  // 1ms deadline for 1kHz control
    .on_miss(Miss::SafeMode)             // Enter the node's safe state on a miss
    .build()?;

Priority Levels

Two independent knobs control priority:

.order(u32) — position in the main tick loop (lower runs earlier). A node given .rate(), .budget() or .deadline() leaves that loop for the RT executor — by default its own thread — where .order() sequences nothing and it runs concurrently with the rest.
.priority(i32) — OS thread priority for the node's RT thread (SCHED_FIFO 1-99, higher = more priority). Requires CAP_SYS_NICE or root, and degrades gracefully when unavailable.

Suggested .order() ranges:

Range	Use Case
0-9	Critical real-time (motor control, safety)
10-49	High priority (sensors, fast control loops)
50-99	Normal priority (processing, planning)
100-199	Low priority (logging, diagnostics)
200+	Background (telemetry, non-essential)

Deadline Miss Policies

Deadline-miss behaviour is chosen per node with .on_miss(Miss):

Policy	Behavior
`Miss::Warn`	Log a warning and continue normally (default)
`Miss::Skip`	Skip the next cycle — the tick that missed has already completed — then resume
`Miss::SafeMode`	Call `enter_safe_state()` on the node, then keep ticking
`Miss::Stop`	Stop the entire scheduler (last resort)

Safety Monitor

The safety monitor provides comprehensive runtime protection:

WCET Monitoring: measures each tick against its budget and reports overruns. Under the default BudgetPolicy::Warn an overrun is only logged — nothing stops the tick. Use .budget_policy(BudgetPolicy::Enforce) to have the node stopped once a tick exceeds twice its budget (a single 1x overrun is logged but not acted on)
Deadline Monitoring: Tracks and responds to deadline misses
Watchdog Timers: Detects hung or crashed nodes
Emergency Stop: Immediate shutdown for critical failures

Budget and deadline monitoring are active for any node registered with .rate(), .budget() or .deadline() — no flag needed. Watchdog timers additionally require Scheduler::watchdog(Duration), shown under Watchdog and Safe-Mode Configuration.

Basic Usage

Adding RT Nodes

use horus::prelude::*;
use std::time::Duration;

let mut scheduler = Scheduler::new();

// Add an RT node with explicit constraints using fluent API
scheduler.add(MotorControlNode::new())
    .order(0)                            // No effect here: .rate() moves this node
                                         // off the main tick loop to the RT executor
    .rate(1000_u64.hz())                 // 1kHz control loop
    .budget(Duration::from_micros(100))  // 100μs tick budget
    .deadline(Duration::from_millis(1))  // 1ms deadline
    .build()?;

// Note: .budget() and .deadline() implicitly enable RT scheduling even without
// .rate(). .rate() on its own auto-derives budget (80% of the period) and
// deadline (95% of the period); explicit .budget()/.deadline() override those
// defaults.

// Mix with regular nodes
scheduler.add(SensorNode::new()).order(10).build()?;

scheduler.run()?;

Watchdog and Safe-Mode Configuration

For a fixed-rate loop that must fail into a safe state rather than drift:

use horus::prelude::*;
use std::time::Duration;

// 1 kHz tick, RT where the OS allows it, watchdog armed
let mut scheduler = Scheduler::new()
    .tick_rate(1000_u64.hz())
    .prefer_rt()        // mlockall + SCHED_FIFO where available
    .watchdog(Duration::from_millis(100));

scheduler.add(ControlLoopNode::new())
    .order(0)
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(50))
    .deadline(Duration::from_micros(500))
    .on_miss(Miss::SafeMode)
    .build()?;

ℹ️`.watchdog()` does not override `.on_miss()`

Scheduler::watchdog(..) registers every RT node as a critical node, but that no longer short-circuits the per-node Miss policy. A critical node's tick-budget overrun is recorded and reported without escalating on its own — the per-node BudgetPolicy decides, and the default BudgetPolicy::Warn only logs. A deadline miss is counted and fed to the graduated degradation ladder, then dispatched through Miss; the safety monitor stops the run only when one node misses its deadline max_deadline_misses times consecutively (default 100). The count is per node, and any tick that meets its deadline resets it, so a node that misses intermittently and recovers never reaches the ceiling. A process-wide lifetime total still exists, but it only feeds the timing report.

So the configuration above does what Miss::SafeMode reads like: the node is safed once on the way into safe mode and keeps ticking.

The watchdog latches a system-wide emergency stop when a critical node stops ticking for more than 3x its watchdog timeout — a hung node, not a late one. The other opt-in escalations are BudgetPolicy::EmergencyStop (e-stop on any budget violation) and Miss::Stop.

High-Performance Configuration

For racing robots or competition systems:

use horus::prelude::*;
use std::time::Duration;

// Configure for maximum performance (10kHz tick rate)
let mut scheduler = Scheduler::new()
    .tick_rate(10_000_u64.hz())
    .prefer_rt()        // mlockall + SCHED_FIFO where available
    .cores(&[2, 3]);

scheduler.add(TractionControl::new())
    .order(0)
    .rate(10_000_u64.hz())
    .budget(Duration::from_micros(20))
    .deadline(Duration::from_micros(100))  // 10kHz
    .build()?;

Advanced Features

Safe-State Hooks

The Node trait has two safety hooks. Both are ordinary runtime methods with default implementations, so override only the ones you need:

impl Node for SafetyCriticalNode {
    fn tick(&mut self) { /* ... */ }

    // Report whether the node is currently in a safe state
    fn is_safe_state(&self) -> bool {
        self.motors_stopped
    }

    // Drive the node into its safe state
    fn enter_safe_state(&mut self) {
        self.pwm = 0.0;
    }
}

Pair enter_safe_state() with .on_miss(Miss::SafeMode) — that policy is what calls it on a deadline miss. It fires on the transition into safe mode, not on every miss, so a node under sustained overload is safed once rather than every cycle. The latch clears as soon as the node meets its deadline again, so a later degradation is caught too and a flapping node produces one entry per episode. The node is not isolated and keeps ticking, and the scheduler does not poll is_safe_state() for automatic recovery. This is the behaviour whether or not a scheduler watchdog is configured; a watchdog adds a system-wide emergency stop only when a critical node stops ticking for 3x its watchdog timeout (see above).

Custom Deadline Policies

Configure response to deadline misses per node with .on_miss(Miss):

use horus::prelude::*;

// Motor controller: enter safe mode on deadline miss
scheduler.add(motor).rate(1000_u64.hz()).on_miss(Miss::SafeMode).build()?;

// Video encoder: drop the frame, keep streaming
scheduler.add(encoder).rate(30_u64.hz()).on_miss(Miss::Skip).build()?;

// Last resort: stop the whole scheduler
scheduler.add(estop_monitor).rate(100_u64.hz()).on_miss(Miss::Stop).build()?;

Failure Handling and Degradation

HORUS has no automatic fallback-node substitution — nothing swaps in a backup implementation for you. What it does have is three mechanisms: a failure policy for nodes that error out, Miss::SafeMode for nodes that overrun their timing, and a graduated degradation ladder that runs whether or not you configure either.

⚠️A node that keeps missing its deadline is degraded automatically

The safety monitor is created unconditionally, and it tracks consecutive deadline misses for every node that has a .rate(). The ladder is not opt-in and Miss::Warn — the default — does not disable it:

Consecutive misses	Action
3	Warning logged
5	Node's rate halved
10	Node isolated — marked `Isolated` and `enter_safe_state()` is called. Only main-loop nodes are actually skipped; an RT node has no health gate in its executor and keeps ticking
20	Node killed — `shutdown()` called and it is removed from execution permanently

A node that recovers is restored to its original rate after 100 successful ticks at the reduced one. A node that reaches 20 does not come back for the life of the process.

If a node of yours has "stopped running" after a period of overload, this ladder is the first thing to check — the misses have to be consecutive, so an intermittently slow node will sit at the warn stage rather than climbing.

use horus::prelude::*;
use std::time::Duration;

// Restart a recoverable node up to 3 times with backoff
scheduler.add(PerceptionNode::new())
    .failure_policy(FailurePolicy::restart(3, Duration::from_millis(50)))
    .build()?;

// Tolerate failures in a non-critical node with a cooldown
scheduler.add(TelemetryLogger::new())
    .failure_policy(FailurePolicy::skip(10, Duration::from_secs(60)))
    .build()?;

// Drop into the node's own safe state when it misses a deadline
scheduler.add(MotorControl::new())
    .rate(1000_u64.hz())
    .on_miss(Miss::SafeMode)
    .build()?;

Starting Points by Robot Type

HORUS has no preset constructors — each robot type is just a different scheduler tick rate plus the per-node constraints described above:

Standard Industrial Robot

let mut scheduler = Scheduler::new();
// Default 100Hz tick; add .watchdog() and per-node .failure_policy() as needed

1 kHz Control Loop

let mut scheduler = Scheduler::new().tick_rate(1000_u64.hz());
// 1kHz control, add per-node .rate()/.budget()/.deadline() for budget enforcement

Racing/Competition Robot

let mut scheduler = Scheduler::new().tick_rate(10_000_u64.hz());
// 10kHz control, add per-node .rate()/.budget()/.deadline() for timing constraints

Performance Characteristics

The measured figures HORUS publishes cover the IPC/topic paths, not scheduler tick latency or monitoring overhead. Every live topic backend is shared-memory and cross-process, and the backend is selected automatically — there is no intra-process or same-thread path, and no backend name for you to choose. Measured with robotics_messages_benchmark on an Intel Core i7-10750H @ 2.60 GHz (6C/12T), powersave governor, 50,000 iterations per message type:

Message	Size	Median	p99
`CmdVel`	16 B	75 ns	135 ns
`Imu`	304 B	121 ns	235 ns
`JointCommand`	928 B	135 ns	226 ns
`LaserScan`	1480 B	210 ns	283 ns

Timing uses RDTSC with calibrated overhead subtraction; see Benchmarks for the full methodology. At a 1kHz control rate a 75 ns hop leaves essentially the whole 1ms period for the node's own work, so message transport is rarely what breaks a budget.

One scheduler-side resolution limit is worth knowing:

Setting	Resolution	Notes
`Scheduler::watchdog()`	1ms	Sub-millisecond timeouts are rounded up to the 1ms floor the config can store

Mixed RT and Normal Nodes

HORUS supports mixed criticality systems by ordering nodes in the tick sequence and attaching timing constraints only to the ones that need them:

use horus::prelude::*;
use std::time::Duration;

let mut scheduler = Scheduler::new();

// Critical RT nodes (order 0-9)
scheduler.add(FlightControl::new())
    .order(0)
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(100))
    .deadline(Duration::from_millis(1))
    .build()?;

// Important processing (order 10-49)
scheduler.add(PathPlanning::new())
    .order(20)
    .build()?;

// Normal tasks (order 50-99)
scheduler.add(TelemetryLogger::new())
    .order(60)
    .build()?;

// Background tasks (order 200+)
scheduler.add(DataUploader::new())
    .order(200)
    .build()?;

Best Practices

WCET Budget Setting

Set budgets 20-30% higher than typical execution:

// If typical execution is 75μs
scheduler.add(MotorControl::new())
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(100)) // Add 25% margin
    .build()?;

.budget() overrides the budget .rate() derives on its own — see below.

Deadline Configuration

.rate(f) derives budget = 80% of the period and deadline = 95% of the period. Setting .budget() alone makes deadline = budget, so a budget overrun is also a deadline miss. Override either explicitly:

scheduler.add(MotorControl::new())
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(500))   // override default 800μs
    .deadline(Duration::from_micros(900)) // override default 950μs
    .build()?;

Priority Assignment

Reserve the .order() ranges listed under Priority Levels, and keep .priority() (the OS thread priority) for nodes that genuinely need to preempt others on the CPU.

Testing Under Load

Always verify timing constraints. Observed timing is read back from the scheduler, not from the node:

#[test]
fn test_wcet_compliance() -> Result<()> {
    let mut scheduler = Scheduler::new().tick_rate(1000_u64.hz());

    scheduler.add(MotorControl::new())
        .order(0)
        .rate(1000_u64.hz())
        .budget(Duration::from_micros(100))
        .build()?;

    scheduler.run_for(Duration::from_millis(500))?;

    // The lookup key is the node's Node::name(), which defaults to the type
    // name — "MotorControl" here, unless the node overrides name()
    let stats = scheduler.rt_stats("MotorControl").unwrap();
    assert_eq!(stats.budget_violations(), 0);
    Ok(())
}

RtStats also exposes worst_execution(), avg_execution_us(), jitter_us() and deadline_misses(). Two caveats: Scheduler::rt_stats() is #[doc(hidden)] — public but unstable, so treat it as a debugging aid rather than a supported API — and it returns None unless the node was registered as RT via .rate(), .budget() or .deadline().

Migration Guide

From Standard Nodes

Your node keeps implementing Node — there is no separate real-time trait. A node becomes real-time through its registration.

Add RT constraints to scheduler.add():

// Before (regular node)
scheduler.add(node).order(0).build()?;

// After (RT node with constraints)
scheduler.add(node)
    .order(0)
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(100))
    .deadline(Duration::from_millis(1))
    .on_miss(Miss::SafeMode)
    .build()?;

Configure scheduler for RT:

let mut scheduler = Scheduler::new().tick_rate(1000_u64.hz()).prefer_rt();

Test under load and check rt_stats() for budget violations — see Testing Under Load

Troubleshooting

WCET Violations

Symptom: "WCET violation" warnings in logs

Solutions:

Increase WCET budget
Optimize node computation
Add warm-up period
Check for blocking operations

Deadline Misses

Symptom: "Deadline miss" errors

Solutions:

Check for priority inversions
Reduce system load
Increase deadline
Relax the miss policy with .on_miss(Miss::Skip) or .on_miss(Miss::Warn)

Emergency Stops

Symptom: Scheduler terminates unexpectedly

Solutions:

Review safety monitor logs
Check critical node health
Raise the emergency-stop threshold with Scheduler::max_deadline_misses(n) (default 100)
Verify WCET budgets

Watchdog Timeouts

Symptom: "Watchdog expired" errors

Solutions:

Ensure nodes complete quickly
Remove blocking I/O
Increase watchdog timeout
Check for infinite loops

Complete Example

use horus::prelude::*;
use std::time::Duration;

struct MotorControlNode {
    pwm_value: f32,
}

impl MotorControlNode {
    fn compute_control(&self) -> f32 {
        // Replace with the application's allocation-free control law.
        0.0
    }

    fn send_pwm_signal(&self) {
        // Replace with a non-blocking hardware write.
    }
}

impl Node for MotorControlNode {
    fn name(&self) -> &str {
        "motor_control"
    }

    fn init(&mut self) -> Result<()> {
        hlog!(info, "Motor control initialized");
        Ok(())
    }

    fn tick(&mut self) {
        // Read sensors, compute control, send PWM
        self.pwm_value = self.compute_control();
        self.send_pwm_signal();
    }

    fn shutdown(&mut self) -> Result<()> {
        self.pwm_value = 0.0;  // Safe shutdown
        hlog!(info, "Motor control shutdown");
        Ok(())
    }
}

fn main() -> Result<()> {
    // 1 kHz tick rate; per-node budget and deadline are enforced below
    let mut scheduler = Scheduler::new().tick_rate(1000_u64.hz());

    // Add RT control node with fluent API
    scheduler.add(MotorControlNode { pwm_value: 0.0 })
        .order(0)
        .rate(1000_u64.hz())
        .budget(Duration::from_micros(100))
        .deadline(Duration::from_millis(1))
        .on_miss(Miss::SafeMode)
        .build()?;

    // Run the system
    scheduler.run()
}

System-Level RT Configuration

The node builder provides node-level timing constraints. For system-level configuration (memory locking, CPU affinity, RT scheduling), see Real-Time Configuration (RtConfig).

use horus::prelude::*;
use std::time::Duration;

// System-level: Configure kernel RT features
RtConfig::builder()
    .memory_locked(true)
    .scheduler(RtScheduler::Fifo)
    .priority(80)
    .cpu_affinity(&[2, 3])
    .build()
    .apply()?;

// Node-level: Add nodes with timing constraints using fluent API
scheduler.add(ControlNode::new())
    .order(0)
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(100))  // Tick budget
    .deadline(Duration::from_millis(1))  // Deadline
    .build()?;

Next Steps

Configure Real-Time Settings (RtConfig) for system-level RT
Explore Safety Monitor for detailed monitoring
Review Performance Optimization for timing analysis
Read Architecture for system design details
Check Testing Guide for RT test strategies