{"title":"Tutorial: Debug with Record & Replay","description":"Use built-in recording to capture a timing bug, replay it deterministically, and test a fix with mixed replay","slug":"tutorials/record-replay-debugging","content":"# Tutorial: Debug with Record & Replay\n\n\nEvery robotics engineer has hit the bug that only happens in the field. The robot drifts, the arm overshoots, the planner freezes — but only under specific sensor conditions you cannot reproduce at your desk. In ROS, you would reach for `rosbag` — an external tool that records DDS messages to a file. HORUS takes a different approach: recording is **built into the scheduler**, capturing every node's inputs and outputs at tick granularity with zero-copy overhead. This tutorial walks you through a real debugging workflow: record a timing bug, replay it deterministically, isolate the cause, and verify your fix — all without touching the robot again.\n\n## Prerequisites\n\n- [Quick Start](/getting-started/quick-start) completed\n- Familiarity with [Nodes and Topics](/concepts/core-concepts-topic)\n- Understanding of [Record & Replay](/advanced/record-replay) concepts\n\n## What You'll Build\n\nA 3-node robot system with a deliberate timing bug, then:\n1. Record the buggy execution\n2. Replay the recording to reproduce the bug deterministically\n3. Use `horus record diff` to compare runs\n4. Fix the bug and verify the fix with mixed replay\n\n**Time estimate**: ~20 minutes\n\n## Step 1: Create the Project\n\n```bash\nhorus new replay-debug -r\ncd replay-debug\n```\n\n## Step 2: Build a Buggy Robot\n\nReplace `src/main.rs` with a 3-node system: a simulated sensor, a controller with a timing bug, and a monitor. The controller accumulates drift when the sensor value crosses a threshold — a realistic bug that only manifests under certain input patterns.\n\n```rust\nuse horus::prelude::*;\n\n// ── Messages ──────────────────────────────────────────────────\n// The `message!` macro is the canonical way to declare messages. The\n// `#[fixed]` variant produces a `#[repr(C)]` POD struct and auto-derives\n// Clone, Copy, Default, Debug, Serialize, Deserialize, and LogSummary.\n// (The bare `prelude` does not re-export serde, so `#[derive(Serialize)]`\n// on a plain struct would fail to compile.)\n\nmessage! {\n    #[fixed]\n    SensorReading {\n        value: f32,\n        timestamp_ns: u64,\n    }\n}\n\nmessage! {\n    #[fixed]\n    ControlOutput {\n        command: f32,\n        error: f32,\n        integral: f32,\n    }\n}\n\n// ── Sensor Node ───────────────────────────────────────────────\n\nstruct Sensor {\n    pub_reading: Topic<SensorReading>,\n    tick_count: u64,\n}\n\nimpl Sensor {\n    fn new() -> Result<Self> {\n        Ok(Self {\n            pub_reading: Topic::new(\"sensor.reading\")?,\n            tick_count: 0,\n        })\n    }\n}\n\nimpl Node for Sensor {\n    fn name(&self) -> &str { \"sensor\" }\n\n    fn tick(&mut self) {\n        let t = self.tick_count as f32 * 0.01;\n        // Sine wave with occasional spikes (the trigger condition)\n        let spike = if self.tick_count % 73 == 0 { 5.0 } else { 0.0 };\n        let value = (t * 2.0).sin() * 10.0 + spike;\n\n        let _ = self.pub_reading.send(SensorReading {\n            value,\n            timestamp_ns: horus::time::now().as_nanos() as u64,\n        });\n        self.tick_count += 1;\n    }\n}\n\n// ── Controller Node (with bug) ────────────────────────────────\n\nstruct Controller {\n    sub_reading: Topic<SensorReading>,\n    pub_output: Topic<ControlOutput>,\n    integral: f32,\n    last_error: f32,\n    setpoint: f32,\n}\n\nimpl Controller {\n    fn new() -> Result<Self> {\n        Ok(Self {\n            sub_reading: Topic::new(\"sensor.reading\")?,\n            pub_output: Topic::new(\"ctrl.output\")?,\n            integral: 0.0,\n            last_error: 0.0,\n            setpoint: 0.0,\n        })\n    }\n}\n\nimpl Node for Controller {\n    fn name(&self) -> &str { \"controller\" }\n\n    fn tick(&mut self) {\n        let reading = match self.sub_reading.recv() {\n            Some(r) => r,\n            None => return,\n        };\n\n        let error = self.setpoint - reading.value;\n        let dt = horus::time::dt().as_secs_f32();\n\n        // BUG: No anti-windup on integral term.\n        // When sensor spikes, the integral accumulates without bound,\n        // causing the controller to drift after the spike passes.\n        self.integral += error * dt;\n\n        let derivative = if dt > 0.0 { (error - self.last_error) / dt } else { 0.0 };\n        self.last_error = error;\n\n        let command = 0.5 * error + 0.1 * self.integral + 0.01 * derivative;\n\n        let _ = self.pub_output.send(ControlOutput {\n            command,\n            error,\n            integral: self.integral,\n        });\n    }\n}\n\n// ── Monitor Node ──────────────────────────────────────────────\n\nstruct Monitor {\n    sub_output: Topic<ControlOutput>,\n    tick_count: u64,\n}\n\nimpl Monitor {\n    fn new() -> Result<Self> {\n        Ok(Self {\n            sub_output: Topic::new(\"ctrl.output\")?,\n            tick_count: 0,\n        })\n    }\n}\n\nimpl Node for Monitor {\n    fn name(&self) -> &str { \"monitor\" }\n\n    fn tick(&mut self) {\n        if let Some(output) = self.sub_output.recv() {\n            // Print every 100 ticks\n            if self.tick_count % 100 == 0 {\n                hlog!(info, \"cmd={:.2}  err={:.2}  integral={:.2}\",\n                    output.command, output.error, output.integral);\n            }\n        }\n        self.tick_count += 1;\n    }\n}\n\n// ── Main ──────────────────────────────────────────────────────\n\nfn main() -> Result<()> {\n    let mut scheduler = Scheduler::new()\n        .tick_rate(100_u64.hz());\n\n    scheduler.add(Sensor::new()?).order(0).build()?;\n    scheduler.add(Controller::new()?).order(1).build()?;\n    scheduler.add(Monitor::new()?).order(2).build()?;\n\n    scheduler.run()\n}\n```\n\nRun it briefly and observe the integral drifting:\n\n```bash\nhorus run\n# Watch for a few seconds, then Ctrl+C\n# You'll see the integral climbing after spike events\n```\n\n## Step 3: Record the Bug\n\nNow run the same system with recording enabled. This captures every node's inputs and outputs at every tick:\n\n\nOr record without changing code using the CLI:\n\n```bash\nhorus run --record buggy_run\n# Run for 10 seconds, then Ctrl+C\n```\n\nCheck the recording:\n\n```bash\nhorus record list --long\n# Output:\n#   buggy_run    3 nodes    1000 ticks    245 KB    2026-03-28 14:32\n```\n\nInspect what was captured:\n\n```bash\nhorus record info buggy_run\n# Shows per-node tick ranges, topics recorded, file sizes\n```\n\n## Step 4: Replay and Reproduce\n\nReplay the recording. The bug reproduces identically every time — same inputs, same execution order, same outputs:\n\n```bash\nhorus record replay buggy_run\n```\n\nSlow it down to watch the spike events:\n\n```bash\nhorus record replay buggy_run --speed 0.25\n```\n\nJump directly to the problematic region (if you noticed the drift starting around tick 400):\n\n```bash\nhorus record replay buggy_run --start-tick 350 --stop-tick 500\n```\n\n\n## Step 5: Export and Analyze\n\nExport the recording for offline analysis:\n\n```bash\nhorus record export buggy_run --output buggy.json --format json\nhorus record export buggy_run --output buggy.csv --format csv\n```\n\nThe JSON export lets you script analysis — grep for the integral crossing a threshold, plot the spike correlation, etc.\n\n## Step 6: Fix the Bug\n\nThe fix is simple — add integral anti-windup clamping. Update the controller:\n\n```rust\n// simplified\nimpl Node for Controller {\n    fn name(&self) -> &str { \"controller\" }\n\n    fn tick(&mut self) {\n        let reading = match self.sub_reading.recv() {\n            Some(r) => r,\n            None => return,\n        };\n\n        let error = self.setpoint - reading.value;\n        let dt = horus::time::dt().as_secs_f32();\n\n        // FIX: Clamp integral to prevent windup\n        self.integral = (self.integral + error * dt).clamp(-10.0, 10.0);\n\n        let derivative = if dt > 0.0 { (error - self.last_error) / dt } else { 0.0 };\n        self.last_error = error;\n\n        let command = 0.5 * error + 0.1 * self.integral + 0.01 * derivative;\n\n        let _ = self.pub_output.send(ControlOutput {\n            command,\n            error,\n            integral: self.integral,\n        });\n    }\n}\n```\n\n## Step 7: Verify the Fix with Mixed Replay\n\nThis is the key step that `rosbag` cannot do. Use **mixed replay** to feed the exact same sensor data from the buggy recording into your fixed controller:\n\n```bash\n# Inject the recorded sensor node, run the fixed controller live\nhorus record inject buggy_run --nodes sensor\n```\n\nOr programmatically in Rust:\n\n```rust,ignore\nuse horus::prelude::*;\nuse std::path::PathBuf;\n\nfn main() -> Result<()> {\n    let mut scheduler = Scheduler::new()\n        .tick_rate(100_u64.hz())\n        .with_recording();  // Record the fixed run too\n\n    // Replay the recorded sensor (same spikes, same timing)\n    scheduler.add_replay(\n        PathBuf::from(\"~/.local/share/horus/recordings/buggy_run/sensor@001.horus\"),\n        0,\n    )?;\n\n    // Run the FIXED controller live against recorded sensor data\n    scheduler.add(Controller::new()?).order(1).build()?;\n    scheduler.add(Monitor::new()?).order(2).build()?;\n\n    scheduler.run()\n}\n```\n\nNow record this fixed run and compare:\n\n```bash\nhorus run --record fixed_run\n# Let it run the same duration, Ctrl+C\n\nhorus record diff buggy_run fixed_run\n# Shows tick-by-tick differences in ctrl.output\n# The integral no longer drifts past +/-10.0\n```\n\n## Step 8: Clean Up\n\n```bash\n# Keep the fixed run, delete the buggy one\nhorus record delete buggy_run\n\n# Or clean all recordings older than 7 days\nhorus record clean --older-than 7\n```\n\n## Key Takeaways\n\n- **Recording is zero-overhead**: It reads directly from shared memory slots — no extra serialization, no external process\n- **Replay is deterministic**: Same inputs at the same tick produce identical outputs every time\n- **Mixed replay is the killer feature**: Replay recorded sensors while running new code live — impossible with external bag tools\n- **`horus record diff`** lets you prove your fix works by comparing buggy vs fixed runs on identical inputs\n- **No hardware needed**: Once recorded, you debug entirely at your desk\n\n## Challenges\n\n**(a) Add a regression test**: Write a script that runs `horus record inject buggy_run --nodes sensor`, checks the integral stays within bounds, and returns exit code 0/1. Add it to your CI.\n\n**(b) Override a sensor value**: Use `--override sensor.reading ...` during replay to test what happens if the spike amplitude doubles. Does your fix still hold?\n\n**(c) Compare algorithm versions**: Record a session, then modify the PID gains. Use mixed replay + diff to find the gains that minimize overshoot on the recorded input data.\n\n## Common Errors\n\n| Symptom | Cause | Fix |\n|---------|-------|-----|\n| `horus record list` shows nothing | Recording was not enabled | Add `.with_recording()` or use `--record` flag |\n| Replay produces different output | Code changed between record and replay | Expected for mixed replay; use `replay_from` for exact reproduction |\n| `inject` node not receiving data | Topic name mismatch | Topic names are case-sensitive and must match exactly |\n| Recording files are large | Long session or high-frequency large messages | Use `horus record clean` or set `max_snapshots` to bound size |\n\n## See Also\n\n- [Record & Replay Reference](/advanced/record-replay) — Full API documentation\n- [BlackBox Flight Recorder](/advanced/blackbox) — Lightweight crash forensics (always-on, bounded storage)\n- [Deterministic Mode](/advanced/deterministic-mode) — Required for bit-identical replay\n- [PID Controller Recipe](/recipes/pid-controller) — Production PID with anti-windup","headings":["Tutorial: Debug with Record & Replay","Prerequisites","What You'll Build","Step 1: Create the Project","Step 2: Build a Buggy Robot","Watch for a few seconds, then Ctrl+C","You'll see the integral climbing after spike events","Step 3: Record the Bug","Run for 10 seconds, then Ctrl+C","Output:","buggyrun    3 nodes    1000 ticks    245 KB    2026-03-28 14:32","Shows per-node tick ranges, topics recorded, file sizes","Step 4: Replay and Reproduce","Step 5: Export and Analyze","Step 6: Fix the Bug","Step 7: Verify the Fix with Mixed Replay","Inject the recorded sensor node, run the fixed controller live","Let it run the same duration, Ctrl+C","Shows tick-by-tick differences in ctrl.output","The integral no longer drifts past +/-10.0","Step 8: Clean Up","Keep the fixed run, delete the buggy one","Or clean all recordings older than 7 days","Key Takeaways","Challenges","Common Errors","See Also"],"wordCount":1544}