Real-Time Nodes

HORUS provides industrial-grade real-time support for time-critical robotics applications through per-node timing constraints and safety monitoring infrastructure.

Overview

Real-time nodes enable deterministic execution for:

Surgical robots requiring guaranteed response times
Industrial robots with hard real-time control loops
Aerospace and defense systems with strict deadlines
Safety-critical autonomous vehicles

Key Features

Real-Time Registration

There is no separate real-time trait. Nodes implement the standard Node trait, and a node becomes real-time when it is registered with timing constraints:

use horus::prelude::*; // Provides {Scheduler, Node, Miss, RtStats, DurationExt, FailurePolicy, RtConfig, RtScheduler}
use std::time::Duration;

let mut scheduler = Scheduler::new();

scheduler.add(YourControlNode::new())
    .order(0)                            // Runs first in the tick sequence
    .rate(1000_u64.hz())                 // 1kHz control loop
    .budget(Duration::from_micros(100))  // 100μs worst-case execution time
    .deadline(Duration::from_millis(1))  // 1ms deadline for 1kHz control
    .on_miss(Miss::SafeMode)             // Enter the node's safe state on a miss
    .build()?;

Priority Levels

Two independent knobs control priority:

.order(u32) — position in the main tick loop (lower runs earlier). A node given .rate(), .budget() or .deadline() leaves that loop for the RT executor — by default its own thread — where .order() sequences nothing and it runs concurrently with the rest.
.priority(i32) — OS thread priority for the node's RT thread (SCHED_FIFO 1-99, higher = more priority). Requires CAP_SYS_NICE or root, and degrades gracefully when unavailable.

Suggested .order() ranges:

Range	Use Case
0-9	Critical real-time (motor control, safety)
10-49	High priority (sensors, fast control loops)
50-99	Normal priority (processing, planning)
100-199	Low priority (logging, diagnostics)
200+	Background (telemetry, non-essential)

Deadline Miss Policies

Deadline-miss behaviour is chosen per node with .on_miss(Miss):

Policy	Behavior
`Miss::Warn`	Log a warning and continue normally (default)
`Miss::Skip`	Skip the next cycle — the tick that missed has already completed — then resume
`Miss::SafeMode`	Call `enter_safe_state()` on the node, then keep ticking
`Miss::Stop`	Stop the entire scheduler (last resort)

Safety Monitor

The safety monitor provides comprehensive runtime protection:

WCET Monitoring: measures each tick against its budget and reports overruns. Under the default BudgetPolicy::Warn an overrun is only logged — nothing stops the tick. Use .budget_policy(BudgetPolicy::Enforce) to have the node stopped after an overrun
Deadline Monitoring: Tracks and responds to deadline misses
Watchdog Timers: Detects hung or crashed nodes
Emergency Stop: Immediate shutdown for critical failures

Budget and deadline monitoring are active for any node registered with .rate(), .budget() or .deadline() — no flag needed. Watchdog timers additionally require Scheduler::watchdog(Duration), shown under Safety-Critical Configuration.

Basic Usage

Adding RT Nodes

use horus::prelude::*;
use std::time::Duration;

let mut scheduler = Scheduler::new();

// Add an RT node with explicit constraints using fluent API
scheduler.add(MotorControlNode::new())
    .order(0)                            // Order 0 (runs first in the tick sequence)
    .rate(1000_u64.hz())                 // 1kHz control loop
    .budget(Duration::from_micros(100))  // 100μs tick budget
    .deadline(Duration::from_millis(1))  // 1ms deadline
    .build()?;

// Note: .budget() and .deadline() implicitly enable RT scheduling even without
// .rate(). .rate() on its own auto-derives budget (80% of the period) and
// deadline (95% of the period); explicit .budget()/.deadline() override those
// defaults.

// Mix with regular nodes
scheduler.add(SensorNode::new()).order(10).build()?;

scheduler.run()?;

Safety-Critical Configuration

For medical or aerospace applications:

use horus::prelude::*;
use std::time::Duration;

// Configure for safety-critical operation (1kHz tick rate)
let mut scheduler = Scheduler::new()
    .tick_rate(1000_u64.hz())
    .prefer_rt()        // mlockall + SCHED_FIFO where available
    .watchdog(Duration::from_millis(100));

scheduler.add(SurgicalRobotControl::new())
    .order(0)
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(50))
    .deadline(Duration::from_micros(500))
    .on_miss(Miss::SafeMode)
    .build()?;

⚠️`.watchdog()` overrides `.on_miss()` for RT nodes

Scheduler::watchdog(..) registers every RT node as a critical node. For a critical node, the first tick-budget overrun or deadline miss latches a system-wide emergency stop and ends the run — before the per-node Miss policy is consulted, and regardless of max_deadline_misses.

So the configuration above does not safe the node and keep ticking, which is what Miss::SafeMode reads like. It stops the whole scheduler on the first overrun:

 EMERGENCY STOP: Critical node ... exceeded tick budget: 5.06ms > 100µs
 Emergency stop activated - shutting down scheduler

That is often the right behaviour for a surgical robot — but decide it deliberately. Drop .watchdog(..) if you want the graduated Miss policy instead; the safety monitor still runs, and Miss::SafeMode then safes the node once and keeps ticking as documented.

High-Performance Configuration

For racing robots or competition systems:

use horus::prelude::*;
use std::time::Duration;

// Configure for maximum performance (10kHz tick rate)
let mut scheduler = Scheduler::new()
    .tick_rate(10_000_u64.hz())
    .prefer_rt()        // mlockall + SCHED_FIFO where available
    .cores(&[2, 3]);

scheduler.add(TractionControl::new())
    .order(0)
    .rate(10_000_u64.hz())
    .budget(Duration::from_micros(20))
    .deadline(Duration::from_micros(100))  // 10kHz
    .build()?;

Advanced Features

Safe-State Hooks

The Node trait has two safety hooks. Both are ordinary runtime methods with default implementations, so override only the ones you need:

impl Node for SafetyCriticalNode {
    fn tick(&mut self) { /* ... */ }

    // Report whether the node is currently in a safe state
    fn is_safe_state(&self) -> bool {
        self.motors_stopped
    }

    // Drive the node into its safe state
    fn enter_safe_state(&mut self) {
        self.pwm = 0.0;
    }
}

Pair enter_safe_state() with .on_miss(Miss::SafeMode) — that policy is what calls it on a deadline miss. It fires on the transition into safe mode, not on every miss, so a node under sustained overload is safed once rather than every cycle. The latch clears as soon as the node meets its deadline again, so a later degradation is caught too and a flapping node produces one entry per episode. The node is not isolated and keeps ticking, and the scheduler does not poll is_safe_state() for automatic recovery. This is the behaviour when no scheduler watchdog is configured — with .watchdog(..), an RT node's first miss latches an emergency stop instead (see above).

Custom Deadline Policies

Configure response to deadline misses per node with .on_miss(Miss):

use horus::prelude::*;

// Motor controller: enter safe mode on deadline miss
scheduler.add(motor).rate(1000_u64.hz()).on_miss(Miss::SafeMode).build()?;

// Video encoder: drop the frame, keep streaming
scheduler.add(encoder).rate(30_u64.hz()).on_miss(Miss::Skip).build()?;

// Last resort: stop the whole scheduler
scheduler.add(estop_monitor).rate(100_u64.hz()).on_miss(Miss::Stop).build()?;

Failure Handling and Degradation

HORUS has no automatic fallback-node substitution — nothing swaps in a backup implementation for you. What it does have is three mechanisms: a failure policy for nodes that error out, Miss::SafeMode for nodes that overrun their timing, and a graduated degradation ladder that runs whether or not you configure either.

⚠️A node that keeps missing its deadline is degraded automatically

The safety monitor is created unconditionally, and it tracks consecutive deadline misses for every node that has a .rate(). The ladder is not opt-in and Miss::Warn — the default — does not disable it:

Consecutive misses	Action
3	Warning logged
5	Node's rate halved
10	Node isolated — marked `Isolated` and `enter_safe_state()` is called. Only main-loop nodes are actually skipped; an RT node has no health gate in its executor and keeps ticking
20	Node killed — `shutdown()` called and it is removed from execution permanently

A node that recovers is restored to its original rate after 100 successful ticks at the reduced one. A node that reaches 20 does not come back for the life of the process.

If a node of yours has "stopped running" after a period of overload, this ladder is the first thing to check — the misses have to be consecutive, so an intermittently slow node will sit at the warn stage rather than climbing.

use horus::prelude::*;
use std::time::Duration;

// Restart a recoverable node up to 3 times with backoff
scheduler.add(PerceptionNode::new())
    .failure_policy(FailurePolicy::restart(3, Duration::from_millis(50)))
    .build()?;

// Tolerate failures in a non-critical node with a cooldown
scheduler.add(TelemetryLogger::new())
    .failure_policy(FailurePolicy::skip(10, Duration::from_secs(60)))
    .build()?;

// Drop into the node's own safe state when it misses a deadline
scheduler.add(MotorControl::new())
    .rate(1000_u64.hz())
    .on_miss(Miss::SafeMode)
    .build()?;

Starting Points by Robot Type

HORUS has no preset constructors — each robot type is just a different scheduler tick rate plus the per-node constraints described above:

Standard Industrial Robot

let mut scheduler = Scheduler::new();
// Default 100Hz tick; add .watchdog() and per-node .failure_policy() as needed

Medical/Surgical Robot

let mut scheduler = Scheduler::new().tick_rate(1000_u64.hz());
// 1kHz control, add per-node .rate()/.budget()/.deadline() for budget enforcement

Racing/Competition Robot

let mut scheduler = Scheduler::new().tick_rate(10_000_u64.hz());
// 10kHz control, add per-node .rate()/.budget()/.deadline() for timing constraints

Performance Characteristics

The measured figures HORUS publishes cover the IPC/topic paths, not scheduler tick latency or monitoring overhead. Every live topic backend is shared-memory and cross-process, and the backend is selected automatically — there is no intra-process or same-thread path, and no backend name for you to choose. Measured with robotics_messages_benchmark on an Intel Core i7-10750H @ 2.60 GHz (6C/12T), powersave governor, 50,000 iterations per message type:

Message	Size	Median	p99
`CmdVel`	16 B	75 ns	135 ns
`Imu`	304 B	121 ns	235 ns
`JointCommand`	928 B	135 ns	226 ns
`LaserScan`	1480 B	210 ns	283 ns

Timing uses RDTSC with calibrated overhead subtraction; see Benchmarks for the full methodology. At a 1kHz control rate a 75 ns hop leaves essentially the whole 1ms period for the node's own work, so message transport is rarely what breaks a budget.

One scheduler-side resolution limit is worth knowing:

Setting	Resolution	Notes
`Scheduler::watchdog()`	1ms	Sub-millisecond timeouts are rounded up to the 1ms floor the config can store

Mixed RT and Normal Nodes

HORUS supports mixed criticality systems by ordering nodes in the tick sequence and attaching timing constraints only to the ones that need them:

use horus::prelude::*;
use std::time::Duration;

let mut scheduler = Scheduler::new();

// Critical RT nodes (order 0-9)
scheduler.add(FlightControl::new())
    .order(0)
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(100))
    .deadline(Duration::from_millis(1))
    .build()?;

// Important processing (order 10-49)
scheduler.add(PathPlanning::new())
    .order(20)
    .build()?;

// Normal tasks (order 50-99)
scheduler.add(TelemetryLogger::new())
    .order(60)
    .build()?;

// Background tasks (order 200+)
scheduler.add(DataUploader::new())
    .order(200)
    .build()?;

Best Practices

WCET Budget Setting

Set budgets 20-30% higher than typical execution:

// If typical execution is 75μs
scheduler.add(MotorControl::new())
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(100)) // Add 25% margin
    .build()?;

.budget() overrides the budget .rate() derives on its own — see below.

Deadline Configuration

.rate(f) derives budget = 80% of the period and deadline = 95% of the period. Setting .budget() alone makes deadline = budget, so a budget overrun is also a deadline miss. Override either explicitly:

scheduler.add(MotorControl::new())
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(500))   // override default 800μs
    .deadline(Duration::from_micros(900)) // override default 950μs
    .build()?;

Priority Assignment

Reserve the .order() ranges listed under Priority Levels, and keep .priority() (the OS thread priority) for nodes that genuinely need to preempt others on the CPU.

Testing Under Load

Always verify timing constraints. Observed timing is read back from the scheduler, not from the node:

#[test]
fn test_wcet_compliance() -> Result<()> {
    let mut scheduler = Scheduler::new().tick_rate(1000_u64.hz());

    scheduler.add(MotorControl::new())
        .order(0)
        .rate(1000_u64.hz())
        .budget(Duration::from_micros(100))
        .build()?;

    scheduler.run_for(Duration::from_millis(500))?;

    // The lookup key is the node's Node::name(), which defaults to the type
    // name — "MotorControl" here, unless the node overrides name()
    let stats = scheduler.rt_stats("MotorControl").unwrap();
    assert_eq!(stats.budget_violations(), 0);
    Ok(())
}

RtStats also exposes worst_execution(), avg_execution_us(), jitter_us() and deadline_misses(). Two caveats: Scheduler::rt_stats() is #[doc(hidden)] — public but unstable, so treat it as a debugging aid rather than a supported API — and it returns None unless the node was registered as RT via .rate(), .budget() or .deadline().

Migration Guide

From Standard Nodes

Your node keeps implementing Node — there is no separate real-time trait. A node becomes real-time through its registration.

Add RT constraints to scheduler.add():

// Before (regular node)
scheduler.add(node).order(0).build()?;

// After (RT node with constraints)
scheduler.add(node)
    .order(0)
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(100))
    .deadline(Duration::from_millis(1))
    .on_miss(Miss::SafeMode)
    .build()?;

Configure scheduler for RT:

let mut scheduler = Scheduler::new().tick_rate(1000_u64.hz()).prefer_rt();

Test under load and check rt_stats() for budget violations — see Testing Under Load

Troubleshooting

WCET Violations

Symptom: "WCET violation" warnings in logs

Solutions:

Increase WCET budget
Optimize node computation
Add warm-up period
Check for blocking operations

Deadline Misses

Symptom: "Deadline miss" errors

Solutions:

Check for priority inversions
Reduce system load
Increase deadline
Relax the miss policy with .on_miss(Miss::Skip) or .on_miss(Miss::Warn)

Emergency Stops

Symptom: Scheduler terminates unexpectedly

Solutions:

Review safety monitor logs
Check critical node health
Raise the emergency-stop threshold with Scheduler::max_deadline_misses(n) (default 100)
Verify WCET budgets

Watchdog Timeouts

Symptom: "Watchdog expired" errors

Solutions:

Ensure nodes complete quickly
Remove blocking I/O
Increase watchdog timeout
Check for infinite loops

Complete Example

use horus::prelude::*;
use std::time::Duration;

struct MotorControlNode {
    pwm_value: f32,
}

impl MotorControlNode {
    fn compute_control(&self) -> f32 {
        // Replace with the application's allocation-free control law.
        0.0
    }

    fn send_pwm_signal(&self) {
        // Replace with a non-blocking hardware write.
    }
}

impl Node for MotorControlNode {
    fn name(&self) -> &str {
        "motor_control"
    }

    fn init(&mut self) -> Result<()> {
        hlog!(info, "Motor control initialized");
        Ok(())
    }

    fn tick(&mut self) {
        // Read sensors, compute control, send PWM
        self.pwm_value = self.compute_control();
        self.send_pwm_signal();
    }

    fn shutdown(&mut self) -> Result<()> {
        self.pwm_value = 0.0;  // Safe shutdown
        hlog!(info, "Motor control shutdown");
        Ok(())
    }
}

fn main() -> Result<()> {
    // Configure for safety-critical operation (1kHz tick rate)
    let mut scheduler = Scheduler::new().tick_rate(1000_u64.hz());

    // Add RT control node with fluent API
    scheduler.add(MotorControlNode { pwm_value: 0.0 })
        .order(0)
        .rate(1000_u64.hz())
        .budget(Duration::from_micros(100))
        .deadline(Duration::from_millis(1))
        .on_miss(Miss::SafeMode)
        .build()?;

    // Run the system
    scheduler.run()
}

System-Level RT Configuration

The node builder provides node-level timing constraints. For system-level configuration (memory locking, CPU affinity, RT scheduling), see Real-Time Configuration (RtConfig).

use horus::prelude::*;
use std::time::Duration;

// System-level: Configure kernel RT features
RtConfig::builder()
    .memory_locked(true)
    .scheduler(RtScheduler::Fifo)
    .priority(80)
    .cpu_affinity(&[2, 3])
    .build()
    .apply()?;

// Node-level: Add nodes with timing constraints using fluent API
scheduler.add(ControlNode::new())
    .order(0)
    .rate(1000_u64.hz())
    .budget(Duration::from_micros(100))  // Tick budget
    .deadline(Duration::from_millis(1))  // Deadline
    .build()?;

Next Steps

Configure Real-Time Settings (RtConfig) for system-level RT
Explore Safety Monitor for detailed monitoring
Review Performance Optimization for timing analysis
Read Architecture for system design details
Check Testing Guide for RT test strategies