BlackBox Flight Recorder

The BlackBox flight recorder provides continuous event logging for post-mortem analysis. Like an aircraft flight recorder, it captures all significant events leading up to failures in a circular buffer.

Overview

The BlackBox:

Records scheduler, node, and safety events automatically
Uses a fixed-size circular buffer (oldest events discarded when full)
Captures anomalies (errors, deadline misses, emergency stops)
Requires no manual instrumentation — the Scheduler records events automatically

Enabling BlackBox

Use the .blackbox(size_mb) builder method to enable the BlackBox:

use horus::prelude::*;

// 16MB black box for development and small deployments
let mut scheduler = Scheduler::new()
    .blackbox(16);

// 1GB black box for safety-critical systems
let mut scheduler = Scheduler::new()
    .blackbox(1024)
    .watchdog(100_u64.ms()); // separate feature: frozen-node detection

// 100MB black box for long-running production or deterministic mode
let mut scheduler = Scheduler::new()
    .blackbox(100);

What Gets Recorded

The BlackBox automatically captures events during scheduler execution:

Event	Description
Scheduler start/stop	When the scheduler begins and ends
Node errors	Failed node executions
Deadline misses	Nodes that missed their timing deadline
WCET violations	Nodes that exceeded their execution time budget
Emergency stops	Safety system activations
Custom events	Free-form category/message markers, e.g. the scheduler's own "N nodes stopped" safety note

Post-Mortem Debugging

After a failure, the BlackBox contains the sequence of events leading up to it. Use the Scheduler's get_blackbox() accessor to inspect it — it is marked #[doc(hidden)], so it is internal-facing and does not appear in the generated API docs. It hands back the shared handle, so lock it before reading:

use horus::prelude::*;

let mut scheduler = Scheduler::new()
    .blackbox(16);

// ... application runs ...

// After a failure, inspect the blackbox
if let Some(bb) = scheduler.get_blackbox() {
    let bb = bb.lock().unwrap();

    // Get all anomalies (errors, deadline misses, e-stops)
    let anomalies = bb.anomalies();
    println!("=== ANOMALIES ({}) ===", anomalies.len());
    for record in &anomalies {
        println!("[tick {}] {:?}", record.tick, record.event);
    }

    // Get all events (full history)
    let all_events = bb.events();
    println!("\n=== LAST 20 EVENTS ===");
    for record in all_events.iter().rev().take(20) {
        println!("[tick {}] {:?}", record.tick, record.event);
    }
}

That accessor only works while the process is still alive. The recorder also persists to .horus/blackbox/ in the working directory (a write-ahead log, plus a JSON snapshot on clean shutdown), so after a crash you read the same events back with the CLI:

horus blackbox --anomalies          # errors, deadline misses, budget violations, e-stops
horus blackbox --last 20            # the last 20 events
horus blackbox --event DeadlineMiss # filter by event type

The reader walks up from wherever you are to find that .horus/blackbox/, so a subdirectory of the project works. Run it outside a project and there is nothing to find, and it reads the machine-global store instead — which holds whatever ran last on this machine, from any project. The first line of output names the directory it used; during incident analysis, read it. See horus blackbox for the exact search order and the per-platform fallback path.

Circular Buffer Behavior

The BlackBox uses a fixed-size circular buffer. When full, the oldest events are discarded:

Buffer capacity: ~80,000 records (16MB at ~200 bytes/record)

Event 1 → [1, _, _, _, _]     New events fill the buffer
Event 2 → [1, 2, _, _, _]
...
Event N → [1, 2, ..., N-1, N]  Buffer full
Event N+1 → [2, 3, ..., N, N+1]  Oldest dropped

This ensures bounded memory usage while keeping the most recent events for debugging.

Recommended Buffer Sizes

Use Case	Configuration	Buffer Size
Development	`.blackbox(16)`	16 MB
Long-running production / deterministic mode	`.blackbox(100)`	100 MB
Safety-critical	`.blackbox(1024)`	1 GB