Tutorial 9: Record & Replay (C++)

Record your nodes' topic data while your robot runs, then replay it offline for debugging. This is the robotics equivalent of a flight recorder.

What You'll Learn

• Using horus run --record to capture topic data, and horus record list/info/clean to manage the sessions
• Using horus record replay to play back a recording, and horus record inject to run live nodes against it
• Using BlackBox for crash analysis
• Designing nodes that work with both live and recorded data

Recording Live Data

Recording is enabled per run with the --record flag — pass a session name and every topic your nodes publish or subscribe to is snapshotted once per tick for the lifetime of the run:

# Record this run's topic data into a named session
horus run --record session_001

# Then manage captured sessions with the `horus record` subcommands:
horus record list                 # list captured sessions
horus record info session_001     # inspect one session
horus record clean --older-than 7 # delete sessions older than 7 days

Each snapshot is stamped with a microsecond wall-clock timestamp and the tick number. Because it is a per-tick sample rather than a per-message log, a topic published twice inside one tick contributes only its latest value.

Replaying Recorded Data

horus record replay re-runs the recording itself: it builds a fresh scheduler containing only replay nodes, which republish the recorded outputs onto the same SHM topics. None of your compiled node code is loaded — this is for inspecting what happened, at whatever speed you like:

# Replay a session at original speed (pass the session name)
horus record replay session_001

# Replay at 2x speed (for faster analysis)
horus record replay session_001 --speed 2.0

# Time-travel: replay a specific tick window
horus record replay session_001 --start-tick 350 --stop-tick 500

To run your own recompiled nodes against the recorded sensor data, use horus record inject instead. It replays only the nodes you name and launches your source file alongside them with horus run, so live code and recorded data share the same topics:

# Replay the recorded lidar node; run controller.cpp live against it
horus record inject session_001 --nodes lidar_node --script controller.cpp

# Same, at half speed, over one tick window
horus record inject session_001 --nodes lidar_node --script controller.cpp \
    --speed 0.5 --start-tick 350 --stop-tick 500

Designing for Replay

The key pattern: separate your processing nodes from your hardware drivers. Processing nodes subscribe to topics — they don't care if data comes from live hardware or a recording.

// This controller works identically with live OR recorded data
class Controller : public horus::Node {
public:
    Controller() : Node("controller") {
        scan_sub_ = subscribe<horus::msg::LaserScan>("lidar.scan");
        cmd_pub_  = advertise<horus::msg::CmdVel>("cmd_vel");
    }

    void tick() override {
        auto scan = scan_sub_->recv();
        if (!scan) return;

        // This code runs the same whether lidar.scan comes from
        // a real LiDAR or from horus record inject
        float min_range = 999.0f;
        for (int i = 0; i < 360; i++) {
            float r = scan->get()->ranges[i];
            if (r > 0.01f && r < min_range) min_range = r;
        }

        horus::msg::CmdVel cmd{};
        cmd.linear = min_range > 0.5f ? 0.3f : 0.0f;
        cmd_pub_->send(cmd);
    }

private:
    horus::Subscriber<horus::msg::LaserScan>* scan_sub_;
    horus::Publisher<horus::msg::CmdVel>* cmd_pub_;
};

BlackBox for Crash Analysis

horus::blackbox::record() writes each event into the shared-memory log ring buffer, which lives outside your process — so the events are still there after a crash:

// Record important events during operation
horus::blackbox::record("controller", "Obstacle detected at 0.3m");
horus::blackbox::record("safety", "E-stop triggered");
horus::blackbox::record("motor", "Current spike: 15A");

// After a crash, inspect with CLI. Entries land at Warning level with the
// category as the node name and the message prefixed "[blackbox] ":
// horus log safety            # filter by the category you passed
// horus log --level warn -n 100
// horus log --since 5m

horus blackbox --last 100 reads a separate store: it shows the events the scheduler itself writes to the flight recorder (deadline misses, budget violations, node errors, emergency stops), not the ones you record from C++.

Workflow

1. Run robot        → horus run --record test
2. Bug happens      → stop the run (Ctrl+C)
3. Fix controller   → edit code, recompile
4. Test with data   → horus record inject test --nodes lidar_node --script controller.cpp
5. Verify fix       → no bug? ship it

No need to set up hardware again. No need to reproduce the exact scenario. The recording has everything.

Key Takeaways

• horus run --record <session> captures topic data per tick; horus record list/info/clean manage the captured sessions
• horus record replay re-runs the recording itself; horus record inject feeds recorded data to your live nodes
• Design nodes as pure topic processors — they work with live AND recorded data
• horus::blackbox::record() events land in the shared-memory log ring buffer, which outlives the crashed process — read them back with horus log
• Replay at different speeds for fast iteration