BlackBox Flight Recorder

The BlackBox flight recorder provides continuous event logging for post-mortem analysis. Like an aircraft flight recorder, it captures all significant events leading up to failures in a circular buffer.

Overview

The BlackBox:

Records scheduler, node, and safety events automatically
Uses a fixed-size circular buffer (oldest events discarded when full)
Captures anomalies (errors, deadline misses, emergency stops)
Requires no manual instrumentation — the Scheduler records events automatically

Enabling BlackBox

Use the .blackbox(size_mb) builder method to enable the BlackBox:

use horus::prelude::*;

// 16MB black box for general production
let mut scheduler = Scheduler::new()
    .blackbox(16);

// 1GB black box for safety-critical systems with watchdog
let mut scheduler = Scheduler::new()
    .watchdog(500_u64.ms())
    .blackbox(1024);

// 100MB black box for hard real-time systems
let mut scheduler = Scheduler::new()
    .blackbox(100);

What Gets Recorded

The BlackBox automatically captures events during scheduler execution:

Event	Description
Scheduler start/stop	When the scheduler begins and ends
Node execution	Each node tick with duration and success/failure
Node errors	Failed node executions
Deadline misses	Nodes that missed their timing deadline
Budget violations	Nodes that exceeded their execution time budget
Failure policy events	Failure policy state transitions
Emergency stops	Safety system activations
Custom events	User-defined markers

Post-Mortem Debugging

After a failure, the BlackBox contains the sequence of events leading up to it. Use the Scheduler's blackbox access to inspect:

use horus::prelude::*;

let mut scheduler = Scheduler::new()
    .blackbox(16);

// ... application runs ...

// After a failure, inspect the blackbox
if let Some(bb) = scheduler.get_blackbox() {
    let bb = bb.lock().unwrap();

    // Get all anomalies (errors, deadline misses, e-stops)
    let anomalies = bb.anomalies();
    println!("=== ANOMALIES ({}) ===", anomalies.len());
    for record in &anomalies {
        println!("[tick {}] {:?}", record.tick, record.event);
    }

    // Get all events (full history)
    let all_events = bb.events();
    println!("\n=== LAST 20 EVENTS ===");
    for record in all_events.iter().rev().take(20) {
        println!("[tick {}] {:?}", record.tick, record.event);
    }
}

Circular Buffer Behavior

The BlackBox uses a fixed-size circular buffer. When full, the oldest events are discarded:

Buffer capacity: 50,000 records (10MB)

Event 1 → [1, _, _, _, _]     New events fill the buffer
Event 2 → [1, 2, _, _, _]
...
Event N → [1, 2, ..., N-1, N]  Buffer full
Event N+1 → [2, 3, ..., N, N+1]  Oldest dropped

This ensures bounded memory usage while keeping the most recent events for debugging.

Recommended Buffer Sizes

Use Case	Configuration	Buffer Size
Development	`.blackbox(16)`	16 MB
Long-running production	`.blackbox(100)`	100 MB
Safety-critical	`.blackbox(1024)`	1 GB