Circuit Breaker

The circuit breaker pattern prevents cascading failures by temporarily disabling failing nodes. When a node fails repeatedly, the scheduler stops ticking it — skipping it for a cooldown, or restarting it with backoff — giving it time to recover.

Overview

The pattern protects against:

• Cascading failures from one failing node
• Resource exhaustion from repeated retry attempts
• System-wide slowdowns from blocked calls

Horus implements this with per-node failure policies, and failure handling is opt-in: a node with no .failure_policy(...) logs the failure and keeps running. Set a policy explicitly on each node that needs one, and the scheduler then handles the cooldowns, backoffs, and recovery for it.

States

A node only has a failure handler if a policy was set for it. The state the handler reports depends on that policy:

Unlike the classic circuit-breaker pattern, there is no half-open trial state and no success threshold.

How It Works

Skip: active → suppressed

Each failed tick increments failure_count. When it reaches max_failures, the node is suppressed for cooldown and the counter resets:

FailurePolicy::skip(3, 30_u64.secs()):
  failure_count = 1 → active (keep ticking)
  failure_count = 2 → active (keep ticking)
  failure_count = 3 → suppressed for 30s (node is skipped)

Skip: suppressed → active

Once the cooldown expires the node is scheduled normally again — there is no probationary single-call phase. FailureHandler::should_allow() returns true as soon as Instant::now() >= suppressed_until.

Skip: recovery

One successful tick clears the failure state. FailureHandler::record_success() resets failure_count to 0 and clears the suppression window.

Restart: healthy → backoff

A failed tick increments restart_count, re-runs the node's init(), and holds the node in backoff for initial_backoff * 2^(restart_count - 1). A successful tick clears the backoff. Once restart_count exceeds max_restarts the policy escalates to a fatal stop: the faulted node is put into its safe state and the scheduler stops.

FailurePolicy::restart(3, 100_u64.ms()):
  failure 1 → restart, backoff 100ms
  failure 2 → restart, backoff 200ms
  failure 3 → restart, backoff 400ms
  failure 4 → restart_count > max_restarts → fatal stop

Per-Node Failure Policies

There is no scheduler-level switch and no default policy. Scheduler::new() creates no failure handlers — a node registered without .failure_policy(...) gets none, and a panicking tick is logged while the scheduler keeps running. Set a policy on each node that needs one, using the node builder:

use horus::prelude::*;

let mut scheduler = Scheduler::new();

// Critical node: stop immediately on failure
scheduler.add(motor_controller)
    .order(0)
    .failure_policy(FailurePolicy::Fatal)
    .done()?;

// Sensor node: restart up to 5 times, 100ms initial backoff
scheduler.add(sensor_reader)
    .order(1)
    .failure_policy(FailurePolicy::restart(5, 100_u64.ms()))
    .done()?;

// Logging node: ignore failures entirely
scheduler.add(data_logger)
    .order(5)
    .failure_policy(FailurePolicy::Ignore)
    .done()?;

Failure policies:

.order() tiers affect execution order only, never failure handling.

Monitoring

Enable the BlackBox flight recorder with Scheduler::new().blackbox(64) (size in MB) — it is off by default. Node failures surface as BlackBoxEvent::NodeError { name, error, severity }; there is no dedicated failure-policy state-change event. After a failure, inspect the recorded anomalies:

if let Some(bb) = scheduler.get_blackbox() {
    let bb = bb.lock().unwrap();
    for record in bb.anomalies() {
        println!("[tick {}] {:?}", record.tick, record.event);
    }
}

Best Practices

1. Set a failure policy explicitly: nodes have none by default. A node registered without .failure_policy(...) logs its failures and keeps running.

2. Set failure policies per node: Critical nodes should use FailurePolicy::Fatal, non-critical nodes can use Restart or Ignore.

3. Test failure scenarios: Verify your system behaves correctly when nodes are temporarily disabled by a Skip cooldown or a Restart backoff.

See Also

• BlackBox Flight Recorder - Event logging including node errors
• Safety Monitor - Real-time safety monitoring
• Scheduler Configuration - SchedulerConfig and node configuration