Deterministic Execution

HORUS supports deterministic execution where the same inputs produce identical outputs every run. This is useful for debugging, regression testing, and systems where a reproducible run order matters.

Why Determinism Matters

Debugging: Non-deterministic bugs are hard to reproduce. Deterministic mode lets you replay exact sequences.

Testing: Repeatable unit tests and regression testing require the same outputs for the same inputs.

Safety Standards: Standards like ISO 26262 and IEC 61508 require predictable execution timing and reproducible behaviour. Determinism is one input to such an argument, never the argument itself — see Certification Status for what HORUS does and does not provide.

Determinism Levels

HORUS offers increasing levels of execution control. Only the last of them gives run-to-run reproducibility:

Level 1: Basic Ordering (Default)

Scheduler::new() on its own is not guaranteed to be order-deterministic. By default the scheduler groups nodes by execution class and dispatches them to dedicated executor threads, so any node configured with .compute(), .on(), .async_io(), or a .rate() leaves the main loop. A scheduler with only plain main-loop nodes looks like this:

use horus::prelude::*;

fn main() -> Result<()> {
    let mut scheduler = Scheduler::new();
    scheduler.add(my_node).order(0).done()?;
    scheduler.run()?;
    Ok(())
}

Guarantees:

• Nodes that stay on the main loop are driven by the ready-dispatch executor: it honours publisher → subscriber dependencies registered during init(), and falls back to .order() tiers when there is none — which is the usual case, since a topic first used inside tick() registers too late to enter the graph
• Within one step — nodes sharing an .order() value with no dependency between them — execution is parallel, so their interleaving is not fixed
• No learning phase (no runtime profiling)
• Reproducible ordering across runs requires .deterministic(true) — see Level 4

Level 2: Explicit Order + Execution Class

Use .order() to fix the tick sequence, and pick each node's execution class explicitly:

use horus::prelude::*;

let mut scheduler = Scheduler::new();

scheduler.add(pid_controller).order(0).rate(1000_u64.hz()).done()?;  // real-time, 1 kHz
scheduler.add(sensor_reader).order(10).rate(200_u64.hz()).done()?;   // real-time, 200 Hz
scheduler.add(data_logger).order(100).done()?;                       // main-loop, best effort

scheduler.run()?;

Order guidelines:

Execution class is separate from order: .rate() makes a node real-time, .compute() sends it to the parallel thread pool, .on(topic) makes it data-triggered, .async_io() puts it on the async pool. A node with none of these stays on the main loop.

A node becomes real-time by builder configuration, not by implementing a different trait — .rate(), .budget() or .deadline() each make a node RT. There is a single node trait, Node, whose tick method is fn tick(&mut self).

Order is explicit — no runtime profiling needed. Note that reproducible ordering across runs still requires .deterministic(true) (see Level 4), since .rate(), .compute(), .on() and .async_io() each hand the node to an executor thread.

Failure handling is not derived from order or execution class — there is no default policy. Opt in per node with .failure_policy(...): FailurePolicy::Fatal, FailurePolicy::restart(3, 50_u64.ms()), FailurePolicy::skip(10, 60_u64.secs()), or FailurePolicy::Ignore.

Level 3: Safety Monitoring

Use builder methods to enable safety monitoring:

use horus::prelude::*;

let mut scheduler = Scheduler::new()
    .watchdog(500_u64.ms())
    .prefer_rt();

Adds:

• Watchdog timers (500ms timeout for frozen nodes), which also mark every RT node critical
• Graduated degradation on sustained deadline misses — warn at 3 consecutive, half rate at 5, isolate and safe at 10, shut down at 20. This ladder is driven by deadline misses, not by the watchdog, and runs for any .rate() node whether or not a watchdog is configured
• Memory locking (mlockall) and the real-time scheduling class (SCHED_FIFO) — these come from .prefer_rt(), which warns and degrades gracefully if the system lacks RT capabilities. Use .require_rt() instead to panic loudly.

The safety monitor and its emergency-stop latch exist on every scheduler regardless of configuration. .watchdog(...) sets the default timeout, arms a watchdog for every RT node, and marks those nodes critical — which governs liveness only: a node that has not ticked for 3× its watchdog timeout is isolated and latches an emergency stop. A critical node's budget overruns and deadline misses still follow its own .budget_policy() / .on_miss() and the graduated ladder, exactly as they would without a watchdog. Budget and deadline monitoring are active for any node that .rate() made real-time; they are not enforced for .compute(), .on() or .async_io() nodes.

Level 4: Full Deterministic Mode

Use the .deterministic(true) builder method to force sequential main-thread execution with a virtual SimClock:

use horus::prelude::*;

let mut scheduler = Scheduler::new()
    .deterministic(true)
    .blackbox(100);

Adds:

• Virtual time (a SimClock replaces the wall clock)
• Execution tracing — opt in separately with .with_recording()
• BlackBox flight recorder (100MB)
• Static execution order — all nodes stay on the main thread, no executor threads are spawned

The per-tick RNG is not one of the additions. The scheduler installs a fresh generator before every tick on every execution path — the main thread, the parallel dispatch workers and the RT threads alike — seeded from the tick number and the node name (tick_number * 0x517cc1b727220a95 + hash(node_name)), so horus::time::rng(|r| ...) already produces the same sequence for the same tick on the same node under the default parallel scheduler. There is no user-supplied seed. Only calls made outside tick() fall back to an entropy-seeded generator. .deterministic(true) changes the clock and the execution order, not the random stream.

The node-name hash comes from the standard library's DefaultHasher, whose algorithm Rust does not promise to keep stable across releases. Runs of the same binary are therefore reproducible, and that is what replay and regression testing need. If you must reproduce a run months later — an experiment attached to a paper, say — archive the binary rather than the source, because rebuilding on a newer toolchain can change the hash and with it the RNG stream.

Tick Budget Monitoring

HORUS tracks per-node tick execution time. When a node's tick exceeds its budget, a BudgetViolation is recorded and surfaced via RtStats::budget_violations(). Budgets are auto-derived as 80% of the period from .rate() (and deadlines as 95% of the period), or set explicitly:

use horus::prelude::*;
use horus::scheduling::BudgetPolicy;

let mut scheduler = Scheduler::new();

scheduler.add(motor_controller)
    .order(0)
    .rate(1000_u64.hz())
    .budget(500_u64.us())      // 500μs max execution time
    .deadline(900_u64.us())    // 900μs deadline
    .budget_policy(BudgetPolicy::EmergencyStop)
    .done()?;

scheduler.run()?;

A node does not implement a violation callback — overruns are handled by the scheduler according to the node's .budget_policy():

BudgetPolicy is not in horus::prelude — import it with use horus::scheduling::BudgetPolicy;.

Deterministic vs Non-Deterministic

.deterministic(true) (sequential, main thread only):

Run 1: Node A → Node B → Node C
Run 2: Node A → Node B → Node C  (identical)
Run 3: Node A → Node B → Node C  (identical)

Default (nodes dispatched to executor threads by class):

Run 1: Node A → Node B → Node C
Run 2: Node B → Node A → Node C  (thread scheduling varies)

Certification Status

How to Read the Status Columns

The comparison tables on this page and on Scheduler Configuration use three status values, each meaning something narrow:

Best Practices

Use Deterministic Seeds

For randomness you manage yourself — HORUS's own per-tick horus::time::rng is tick-seeded in every mode, deterministic mode or not:

impl Node for RandomNode {
    fn init(&mut self) -> Result<()> {
        self.rng = StdRng::seed_from_u64(42);
        Ok(())
    }

    fn tick(&mut self) {
        // Generates same sequence every run
        let value = self.rng.gen::<f64>();
    }
}

Avoid System Time

// BAD: Non-deterministic
fn tick(&mut self) {
    let now = std::time::Instant::now();  // Different every run
}

// GOOD: Use internal state counter
fn tick(&mut self) {
    self.tick_count += 1;  // Same sequence every run
}

When you do need a timestamp, use horus::time::now() — it reads the scheduler's clock, which is the virtual SimClock under .deterministic(true), so it replays identically.

Control External Inputs

// For testing, mock external sensors
#[cfg(test)]
impl Node for SensorNode {
    fn tick(&mut self) {
        self.output = self.test_data[self.tick_count % self.test_data.len()];
    }
}

Summary

Next Steps

• Execution Modes - Sequential vs parallel execution
• BlackBox Flight Recorder - Event recording for post-mortem analysis
• Circuit Breaker - Fault tolerance patterns