Deterministic Execution

HORUS supports deterministic execution where the same inputs produce identical outputs every run. This is useful for debugging, regression testing, and safety-critical systems.

Why Determinism Matters

Debugging: Non-deterministic bugs are hard to reproduce. Deterministic mode lets you replay exact sequences.

Testing: Repeatable unit tests and regression testing require the same outputs for the same inputs.

Safety Certification: Standards like ISO 26262 and IEC 61508 require predictable execution timing and reproducible behavior.

Determinism Levels

HORUS offers increasing levels of execution control. Only the last of them gives run-to-run reproducibility:

Level 1: Basic Ordering (Default)

Scheduler::new() on its own is not guaranteed to be order-deterministic. By default the scheduler groups nodes by execution class and dispatches them to dedicated executor threads, so any node configured with .compute(), .on(), .async_io(), or a .rate() leaves the main loop. A scheduler with only plain main-loop nodes looks like this:

use horus::prelude::*;

fn main() -> Result<()> {
    let mut scheduler = Scheduler::new();
    scheduler.add(my_node).order(0).done()?;
    scheduler.run()?;
    Ok(())
}

Guarantees:

• Nodes that stay on the main loop are driven by the ready-dispatch executor: it honours publisher → subscriber dependencies, and falls back to .order() tiers when there is no topic metadata
• Within one step — nodes sharing an .order() value with no dependency between them — execution is parallel, so their interleaving is not fixed
• No learning phase (no runtime profiling)
• Reproducible ordering across runs requires .deterministic(true) — see Level 4

Level 2: Explicit Order + Execution Class

Use .order() to fix the tick sequence, and pick each node's execution class explicitly:

use horus::prelude::*;

let mut scheduler = Scheduler::new();

scheduler.add(pid_controller).order(0).rate(1000_u64.hz()).done()?;  // real-time, 1 kHz
scheduler.add(sensor_reader).order(10).rate(200_u64.hz()).done()?;   // real-time, 200 Hz
scheduler.add(data_logger).order(100).done()?;                       // main-loop, best effort

scheduler.run()?;

Order guidelines:

Execution class is separate from order: .rate() makes a node real-time, .compute() sends it to the parallel thread pool, .on(topic) makes it data-triggered, .async_io() puts it on the async pool. A node with none of these stays on the main loop.

A node becomes real-time by builder configuration, not by implementing a different trait — .rate(), .budget() or .deadline() each make a node RT. There is a single node trait, Node, whose tick method is fn tick(&mut self).

Order is explicit — no runtime profiling needed. Note that reproducible ordering across runs still requires .deterministic(true) (see Level 4), since .rate(), .compute(), .on() and .async_io() each hand the node to an executor thread.

Failure handling is not derived from order or execution class — there is no default policy. Opt in per node with .failure_policy(...): FailurePolicy::Fatal, FailurePolicy::restart(3, 50_u64.ms()), FailurePolicy::skip(10, 60_u64.secs()), or FailurePolicy::Ignore.

Level 3: Safety Monitoring

Use builder methods to enable safety monitoring:

use horus::prelude::*;

let mut scheduler = Scheduler::new()
    .watchdog(500_u64.ms())
    .prefer_rt();

Adds:

• Watchdog timers (500ms timeout for frozen nodes), which also mark every RT node critical
• Graduated degradation on sustained deadline misses — warn at 3 consecutive, half rate at 5, isolate and safe at 10, shut down at 20. This ladder is driven by deadline misses, not by the watchdog, and runs for any .rate() node whether or not a watchdog is configured
• Memory locking (mlockall) and the real-time scheduling class (SCHED_FIFO) — these come from .prefer_rt(), which warns and degrades gracefully if the system lacks RT capabilities. Use .require_rt() instead to panic loudly.

The safety monitor and its emergency-stop latch exist on every scheduler regardless of configuration. .watchdog(...) sets the default timeout, arms a watchdog for every RT node, and marks those nodes critical — after which their first budget overrun or deadline miss latches an emergency stop rather than following the graduated ladder. Budget and deadline monitoring are active for any node that .rate() made real-time; they are not enforced for .compute(), .on() or .async_io() nodes.

Level 4: Full Deterministic Mode

Use the .deterministic(true) builder method to force sequential main-thread execution with a virtual SimClock:

use horus::prelude::*;

let mut scheduler = Scheduler::new()
    .deterministic(true)
    .blackbox(100);

Adds:

• Virtual time (a SimClock replaces the wall clock)

• Deterministic per-tick RNG via horus::time::rng(|r| ...) — the seed is derived from the tick number plus the node name (tick_number * 0x517cc1b727220a95 + hash(node_name)), so the same tick on the same node always produces the same sequence. There is no user-supplied seed.

  The node-name hash comes from the standard library's DefaultHasher, whose algorithm Rust does not promise to keep stable across releases. Runs of the same binary are therefore reproducible, and that is what replay and regression testing need. If you must reproduce a run months later — an experiment attached to a paper, say — archive the binary rather than the source, because rebuilding on a newer toolchain can change the hash and with it the RNG stream.

• Execution tracing — opt in separately with .with_recording()

• BlackBox flight recorder (100MB)

• Static execution order — all nodes stay on the main thread, no executor threads are spawned

Tick Budget Monitoring

HORUS tracks per-node tick execution time. When a node's tick exceeds its budget, a BudgetViolation is recorded and surfaced via RtStats::budget_violations(). Budgets are auto-derived as 80% of the period from .rate() (and deadlines as 95% of the period), or set explicitly:

use horus::prelude::*;
use horus::scheduling::BudgetPolicy;

let mut scheduler = Scheduler::new();

scheduler.add(motor_controller)
    .order(0)
    .rate(1000_u64.hz())
    .budget(500_u64.us())      // 500μs max execution time
    .deadline(900_u64.us())    // 900μs deadline
    .budget_policy(BudgetPolicy::EmergencyStop)
    .done()?;

scheduler.run()?;

A node does not implement a violation callback — overruns are handled by the scheduler according to the node's .budget_policy():

BudgetPolicy is not in horus::prelude — import it with use horus::scheduling::BudgetPolicy;.

Deterministic vs Non-Deterministic

.deterministic(true) (sequential, main thread only):

Run 1: Node A → Node B → Node C
Run 2: Node A → Node B → Node C  (identical)
Run 3: Node A → Node B → Node C  (identical)

Default (nodes dispatched to executor threads by class):

Run 1: Node A → Node B → Node C
Run 2: Node B → Node A → Node C  (thread scheduling varies)

Best Practices

Use Deterministic Seeds

For randomness you manage yourself — HORUS's own per-tick horus::time::rng is already deterministic, see Level 4:

impl Node for RandomNode {
    fn init(&mut self) -> Result<()> {
        self.rng = StdRng::seed_from_u64(42);
        Ok(())
    }

    fn tick(&mut self) {
        // Generates same sequence every run
        let value = self.rng.gen::<f64>();
    }
}

Avoid System Time

// BAD: Non-deterministic
fn tick(&mut self) {
    let now = std::time::Instant::now();  // Different every run
}

// GOOD: Use internal state counter
fn tick(&mut self) {
    self.tick_count += 1;  // Same sequence every run
}

When you do need a timestamp, use horus::time::now() — it reads the scheduler's clock, which is the virtual SimClock under .deterministic(true), so it replays identically.

Control External Inputs

// For testing, mock external sensors
#[cfg(test)]
impl Node for SensorNode {
    fn tick(&mut self) {
        self.output = self.test_data[self.tick_count % self.test_data.len()];
    }
}

Summary

Next Steps

• Execution Modes - Sequential vs parallel execution
• BlackBox Flight Recorder - Event recording for post-mortem analysis
• Circuit Breaker - Fault tolerance patterns