Troubleshooting

HORUS includes utility scripts to help you update, recover from broken installations, and verify your setup.

Quick Reference

Quick Diagnostic Steps

When your HORUS application isn't working:

1. Check the Monitor: Run horus monitor to see active nodes, topics, and message flow
2. Examine Logs: Look for error messages in your terminal output
3. Verify Topics: Ensure publisher and subscriber use exact same topic names
4. Check Shared Memory: Look in /dev/shm/horus/ for stale HORUS memory regions
5. Test Individually: Run nodes one at a time to isolate the problem

Updating HORUS

To update to the latest version:

cd /path/to/horus
git pull
./install.sh

To preview changes before updating:

git fetch
git log HEAD..@{u}  # See what's new
git pull
./install.sh

If you have uncommitted changes:

git stash
git pull
./install.sh
git stash pop  # Restore your changes

Manual Recovery

Use when: Build errors, corrupted cache, installation broken

Quick Steps

# Navigate to HORUS source directory
cd /path/to/horus

# 1. Clean build artifacts
cargo clean

# 2. Remove cached libraries
rm -rf ~/.horus/cache

# 3. Fresh install
./install.sh

When to Use Recovery

Symptoms requiring recovery:

1. Build fails:

   error: could not compile `horus_core`
   

2. Corrupted cache:

   error: failed to load source for dependency `horus_core`
   

3. Binary doesn't work:

   Copy

   $ horus --help
   Segmentation fault
   

4. Version mismatches:

   error: the package `horus` depends on `horus_core 0.1.0`,
   but `horus_core 0.1.3` is installed
   

5. Broken after system updates:

   • Rust updated
   • System libraries changed
   • GCC/Clang updated

What Gets Removed

By cargo clean:

• target/ directory (build artifacts)

By rm -rf ~/.horus/cache:

• Installed libraries
• Cached dependencies

Never removed (safe):

• ~/.horus/config (user settings)
• ~/.horus/credentials (registry auth)
• Project-local .horus/ directories
• Your source code

Full Reset (Nuclear Option)

If the quick steps don't work, do a complete reset:

# Remove everything HORUS-related
cargo clean
rm -rf ~/.horus
rm -f ~/.cargo/bin/horus

# Fresh install
./install.sh

Installation Issues

Problem: "Rust not installed"

$ ./install.sh
 Error: Rust is not installed

Solution:

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Then try again
./install.sh

Problem: "C compiler not found"

Solution:

# Ubuntu/Debian/Raspberry Pi OS - Install ALL required packages
sudo apt update
sudo apt install -y build-essential pkg-config \
  libssl-dev libudev-dev libasound2-dev \
  libx11-dev libxrandr-dev libxi-dev libxcursor-dev libxinerama-dev \
  libwayland-dev wayland-protocols libxkbcommon-dev \
  libvulkan-dev libfontconfig-dev libfreetype-dev \
  libv4l-dev

# Fedora/RHEL
sudo dnf groupinstall "Development Tools"
sudo dnf install -y pkg-config openssl-devel systemd-devel alsa-lib-devel \
  libX11-devel libXrandr-devel libXi-devel libXcursor-devel libXinerama-devel \
  wayland-devel wayland-protocols-devel libxkbcommon-devel \
  vulkan-devel fontconfig-devel freetype-devel \
  libv4l-devel

Problem: Build fails with linker errors

error: linking with `cc` failed: exit status: 1
error: could not find native static library `X11`, perhaps an -L flag is missing?

Solution:

# Install ALL missing system libraries (most common cause)
# Ubuntu/Debian/Raspberry Pi OS
sudo apt update
sudo apt install -y build-essential pkg-config \
  libssl-dev libudev-dev libasound2-dev \
  libx11-dev libxrandr-dev libxi-dev libxcursor-dev libxinerama-dev \
  libwayland-dev wayland-protocols libxkbcommon-dev \
  libvulkan-dev libfontconfig-dev libfreetype-dev \
  libv4l-dev

# Or run manual recovery (see Manual Recovery section)
cargo clean && rm -rf ~/.horus/cache && ./install.sh

Update Issues

Problem: "Build failed" during update

Solution:

# Try manual recovery
cargo clean && rm -rf ~/.horus/cache && ./install.sh

Problem: "Already up to date" but binary broken

Solution:

# Force rebuild
./install.sh

Runtime Issues

"horus: command not found"

Solution:

# Add to PATH (add to ~/.bashrc or ~/.zshrc)
export PATH="$HOME/.cargo/bin:$PATH"

# Then reload shell
source ~/.bashrc  # or restart terminal

# Verify
which horus
horus --help

Binary exists but doesn't run

$ horus --help
Segmentation fault

Solution:

# Full recovery
cargo clean && rm -rf ~/.horus/cache && ./install.sh

Version mismatch errors

error: the package `horus` depends on `horus_core 0.1.0`,
but `horus_core 0.1.3` is installed

Why this happens:

• You updated the horus CLI to a new version
• Your project's .horus/ directory still has cached dependencies from the old version
• The cached Cargo.lock references incompatible library versions

Solution (Recommended - Fast & Easy):

# Clean cached build artifacts and dependencies
horus run --clean

# This removes .horus/target/ and forces a fresh build
# with the new version

Alternative Solutions:

Option 2: Manual cleanup

# Remove the entire .horus directory
rm -rf .horus/

# Next run will rebuild from scratch
horus run

Option 3: Manual recovery (for persistent issues)

# Only needed if --clean doesn't work
# This reinstalls HORUS libraries globally
cd /path/to/horus
cargo clean && rm -rf ~/.horus/cache && ./install.sh

For multiple projects:

# Clean all projects in your workspace
find ~/your-projects -type d -name ".horus" -exec rm -rf {}/target/ \;

"HORUS source directory not found" (Rust projects)

Error: HORUS source directory not found. Please set HORUS_SOURCE environment variable.

Solution:

# Option 1: Set HORUS_SOURCE (recommended for non-standard installations)
export HORUS_SOURCE=/path/to/horus
echo 'export HORUS_SOURCE=/path/to/horus' >> ~/.bashrc

# Option 2: Install HORUS to a standard location
# The CLI checks these paths automatically:
#   - ~/softmata/horus
#   - /horus
#   - /opt/horus
#   - /usr/local/horus

# Verify HORUS source is found
horus build

Why this happens:

• horus run needs to find HORUS core libraries for Rust compilation
• It auto-detects standard installation paths
• For custom installations, set $HORUS_SOURCE

Topic Creation Errors

Symptom: Application crashes on startup with:

Error: Failed to create `Topic<MyMessage>`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value'

Common Causes:

1. Stale Shared Memory from Previous Run

   • HORUS uses /dev/shm/horus/ directory for communication
   • If your app crashes, these files persist

   Fix: Clean shared memory:

   Copy

   # Remove all HORUS shared memory
   rm -rf /dev/shm/horus/
   

2. Insufficient Permissions on /dev/shm

   Fix: Check permissions:

   Copy

   ls -la /dev/shm/horus/topics/
   # Should show your user as owner
   
   # If not, remove with sudo
   sudo rm -rf /dev/shm/horus/
   
   # Fix permissions (if needed)
   sudo chmod 1777 /dev/shm
   

3. Disk Space Full on /dev/shm

   Fix: Check available space:

   Copy

   df -h /dev/shm
   

4. Conflicting Topic Names

   • Two Topics with same name but different types

   Fix: Use unique topic names:

   Copy

   // BAD: Same name, different types
   let topic1: Topic<f32> = Topic::new("data")?;
   let topic2: Topic<String> = Topic::new("data")?;  // CONFLICT!
   
   // GOOD: Different names
   let topic1: Topic<f32> = Topic::new("sensor_data")?;
   let topic2: Topic<String> = Topic::new("status_data")?;
   

General Code Fix:

// Topic names become file paths on the underlying shared memory system.
// Use simple, descriptive names with dots (not slashes):
let topic = Topic::new("sensor_data")?;
let topic = Topic::new("camera.front.raw")?;

"No such file or directory" when creating Topic

Symptom: Application crashes with:

thread 'main' panicked at 'Failed to create publisher 'camera': No such file or directory'

Cause: You're using slashes (/) in your topic name. While slashes work on Linux (parent directories are created automatically), they fail on macOS where shared memory uses shm_open() which doesn't support embedded slashes.

HORUS topic names map to files under /dev/shm/horus/topics/ on Linux:

Topic: "sensors.camera"  →  /dev/shm/horus/topics/horus_sensors.camera  (cross-platform)
Topic: "sensors/camera"  →  Works on Linux only, fails on macOS

Fix: Use dots instead of slashes for cross-platform compatibility:

// NOT RECOMMENDED - fails on macOS
let topic: Topic<f32> = Topic::new("sensors/camera")?;
let topic: Topic<Twist> = Topic::new("robot/cmd_vel")?;

// RECOMMENDED - works on all platforms
let topic: Topic<f32> = Topic::new("sensors.camera")?;
let topic: Topic<Twist> = Topic::new("robot.cmd_vel")?;

Coming from ROS? ROS uses slashes (/sensor/lidar) because it uses network-based naming. HORUS uses dots because topic names map directly to shared memory file names. See Topic Naming for details.

Topic Not Found / No Messages Received

Symptom: Subscriber node never receives messages even though publisher is sending.

// recv() always returns None
if let Some(data) = self.data_sub.recv() {
    println!("Got data");  // Never prints
}

Common Causes:

1. Topic Name Mismatch (Typo)

   • This is the #1 cause

   Fix: Verify exact topic names:

   Copy

   // Publisher
   let pub_topic: Topic<f32> = Topic::new("sensor_data")?;  // Note: sensor_data
   
   // Subscriber (TYPO! Missing underscore)
   let sub_topic: Topic<f32> = Topic::new("sensordata")?;
   
   // CORRECT:
   let sub_topic: Topic<f32> = Topic::new("sensor_data")?;  // Exact match
   

   Debug with Monitor:

   Copy

   horus monitor
   

   Check the "Topics" section to see active topic names.

2. Type Mismatch

   • Publisher and subscriber use different message types

   Fix: Ensure both use same type:

   Copy

   // Publisher
   let pub_topic: Topic<f32> = Topic::new("data")?;
   pub_topic.send(3.14);
   
   // Subscriber (WRONG TYPE)
   let sub_topic: Topic<f64> = Topic::new("data")?;  // f64 != f32
   
   // CORRECT:
   let sub_topic: Topic<f32> = Topic::new("data")?;  // Same type
   

3. Publisher Hasn't Sent Yet

   • Subscriber starts before publisher sends first message
   • This is normal! First recv() will return None

   Fix: Check multiple ticks:

   Copy

   impl Node for SubscriberNode {
       fn tick(&mut self) {
           if let Some(msg) = self.topic.recv() {
               // Process message
           } else {
               // No message yet - this is OK on first few ticks
           }
       }
   }
   

4. Wrong Priority Order

   • Subscriber runs before publisher in same tick

   Fix: Set priorities correctly:

   Copy

   // Publisher should run first (lower order number)
   scheduler.add(PublisherNode::new()?).order(0).build()?;
   
   // Subscriber runs after (higher order number)
   scheduler.add(SubscriberNode::new()?).order(1).build()?;
   

Application Hangs / Deadlock

Symptom: Your app starts but freezes with no error messages.

Starting application...
[Nodes initialized]
[Application freezes - no output]

Common Causes:

1. Infinite Loop in tick()

   Copy

   // BAD: Never returns!
   fn tick(&mut self) {
       loop {
           // Process data
       }
   }
   
   // GOOD: Tick returns after work
   fn tick(&mut self) {
       self.process_data();
       // Return naturally — scheduler calls tick() again next frame
   }
   

2. Blocking Operations in tick()

   Copy

   // BAD: Blocks scheduler
   fn tick(&mut self) {
       std::thread::sleep(Duration::from_secs(10));  // Blocks everything!
   }
   
   // GOOD: Use tick counter for delays
   fn tick(&mut self) {
       self.tick_count += 1;
   
       // Execute every 10 ticks (~167ms at 60 FPS)
       if self.tick_count % 10 == 0 {
           self.slow_operation();
       }
   }
   

3. Waiting Forever for Messages

   Copy

   // BAD: Blocking wait
   fn tick(&mut self) {
       while self.data_sub.recv().is_none() {
           // Infinite loop if no messages!
       }
   }
   
   // GOOD: Non-blocking receive
   fn tick(&mut self) {
       if let Some(data) = self.data_sub.recv() {
           // Process data
       }
       // Continue even if no message
   }
   

4. Circular Priority Dependencies

   • Node A waits for Node B, Node B waits for Node A

   Fix: Ensure data flows one direction:

   Copy

   // BAD: Circular dependency
   // Node A (priority 0) subscribes to "data_b"
   // Node B (priority 1) subscribes to "data_a"
   // Both wait for each other!
   
   // GOOD: Unidirectional flow
   // Node A (priority 0) publishes to "data_a"
   // Node B (priority 1) subscribes to "data_a", publishes to "data_b"
   // Node C (priority 2) subscribes to "data_b"
   

5. Debug with Logging

   Copy

   fn tick(&mut self) {
       hlog!(debug, "Tick started");
       // Your code here
       hlog!(debug, "Tick completed");
   }
   

   If you see "Tick started" but never "Tick completed", the hang is in your code.

Messages Silently Dropped

Symptom: Publisher sends messages but subscriber never receives them, and no error is reported.

Cause: For non-POD (serialized) messages, the serialized data exceeds the slot size (default 8KB). The send() method is lossy — it retries briefly then drops the message, incrementing an internal failure counter.

Fix:

1. Check Message Size:

use std::mem::size_of;

// POD messages always fit (slot = size_of::<T>())
// Non-POD messages must serialize within the slot size (default 8KB)
println!("Message size: {} bytes", size_of::<MyMessage>());

2. Keep Messages Reasonably Sized:

// BAD: Variable size — may exceed limit
#[derive(Clone, Serialize, Deserialize)]
pub struct LargeMessage {
    pub data: Vec<u8>,
}

// GOOD: Fixed size
#[derive(Clone, Serialize, Deserialize)]
pub struct LargeMessage {
    pub data: [u8; 4096],  // Fixed 4KB
}

// BETTER: Split into multiple messages
#[derive(Clone, Serialize, Deserialize)]
pub struct MessageChunk {
    pub chunk_id: u32,
    pub total_chunks: u32,
    pub data: [u8; 1024],
}

3. Use Monitor to Check:

# The monitor shows send failure counts per topic
horus monitor

Build and Compilation Issues

"unresolved import" or "cannot find type in this scope"

Symptom: Code won't compile, missing types or functions.

Fix: Ensure HORUS is in your Cargo.toml dependencies:

[dependencies]
horus = { path = "..." }
horus_library = { path = "..." }  # For standard messages (CmdVel, Twist, etc.)

Import the prelude:

use horus::prelude::*;  // Provides Twist, LaserScan, CmdVel, etc.

"trait bound ... is not satisfied"

Symptom: Compiler says your message doesn't implement required traits.

Fix: Add required derives:

// Add these three derives to all messages
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MyMessage {
    pub field: f32,
}

Performance Issues

Problem: Slow builds

Solution:

# Use release mode (optimized)
horus run --release

Problem: Large disk usage

Solution:

# Clean old cargo cache
cargo clean

# Remove unused dependencies
cargo install cargo-cache
cargo cache --autoclean

Problem: Large .horus/target/ directory (Rust projects)

Why this happens:

• Cargo stores build artifacts in .horus/target/
• Debug builds are unoptimized and larger
• Incremental compilation caches intermediate files

Solution:

# Clean build artifacts in current project
rm -rf .horus/target/

# Or use horus clean flag (next build will be slower)
horus run --clean

# Regular cleanup (if working on multiple projects)
find . -type d -name ".horus" -exec rm -rf {}/target/ \;

# Add to .gitignore (already included in horus new templates)
echo ".horus/target/" >> .gitignore

Disk usage typical sizes:

• .horus/target/debug/: ~10-100 MB (incremental builds)
• .horus/target/release/: ~5-50 MB (optimized, no debug symbols)

Best practices:

• .horus/ is in .gitignore by default
• Clean periodically if disk space is limited

Using the Monitor to Debug

The monitor is your best debugging tool for runtime issues.

Starting the Monitor:

# Terminal 1: Run your application
horus run

# Terminal 2: Start monitor
horus monitor

Monitor Features:

1. Nodes Tab:

• Shows all running nodes
• Displays node state (Running, Error, Stopped)
• Shows tick count and timing
• Highlights nodes that aren't ticking (stuck)

2. Topics Tab:

• Lists all active topics
• Shows message types
• Displays publisher/subscriber counts
• 0 publishers = no one is sending
• 0 subscribers = no one is listening

3. Metrics Tab:

• IPC Latency: Communication time (should be <1µs)
• Tick Duration: How long each node takes
• Message Counts: Total sent/received
  • If sent > 0 but received = 0, subscriber issue
  • If sent = 0, publisher issue

4. Graph Tab:

• Visual node graph
• Shows message flow between nodes
• Disconnected nodes = topic mismatch

Debug Workflow:

1. Check Nodes tab
   -> All nodes Running? (If Error, check logs)

2. Check Topics tab
   -> Topics exist? (If no, topic name typo)
   -> Publishers > 0? (If no, publisher not working)
   -> Subscribers > 0? (If no, subscriber not created)

3. Check Metrics tab
   -> Messages sent > 0? (If no, publisher not sending)
   -> Messages received > 0? (If no, subscriber not receiving)
   -> IPC latency sane? (If >1ms, system issue)

4. Check Graph tab
   -> Nodes connected? (If no, topic name mismatch)

Example Debug Session:

# Problem: Subscriber not receiving messages

# Monitor shows:
# Nodes: SensorNode (Running), DisplayNode (Running)
# Topics: "sensor_data" (1 pub, 0 sub)  <-- AHA!

# Issue: No subscribers!
# Fix: Check DisplayNode - likely wrong topic name

Reading Log Output

Log Levels

HORUS nodes can log at different severity levels:

fn tick(&mut self) {
    hlog!(debug, "Detailed info for debugging");
    hlog!(info, "Normal informational message");
    hlog!(warn, "Something unusual happened");
    hlog!(error, "Something went wrong!");
}

Log Format

Console output uses ANSI-colored formatting:

[INFO] [SensorNode] Sensor initialized
│      │            │
│      │            └─ Message
│      └─ Node name
└─ Log level (INFO, WARN, ERROR, DEBUG)

Timestamps are included in the shared memory log buffer (visible in the monitor), formatted as HH:MM:SS.mmm.

Common Patterns and Anti-Patterns

[OK] DO: Check recv() for None

fn tick(&mut self) {
    if let Some(msg) = self.topic.recv() {
        // Process message
    }
    // No message? That's OK, just continue
}

[FAIL] DON'T: Unwrap recv()

fn tick(&mut self) {
    let msg = self.topic.recv().unwrap();  // PANIC if no message!
}

[OK] DO: Use Result for errors

impl Node for MyNode {
    fn init(&mut self) -> Result<()> {
        if self.sensor.is_broken() {
            return Err(Error::node("MyNode", "Sensor initialization failed"));
        }
        Ok(())
    }
}

[FAIL] DON'T: panic!() in nodes

fn init(&mut self) -> Result<()> {
    if self.sensor.is_broken() {
        panic!("Sensor broken");  // DON'T DO THIS
    }
    Ok(())
}

[OK] DO: Keep tick() fast

fn tick(&mut self) {
    // Quick operations only
    let data = self.sensor.read_cached();
    self.topic.send(data);
}

[FAIL] DON'T: Block in tick()

fn tick(&mut self) {
    thread::sleep(Duration::from_millis(100));  // Blocks everything!
    let data = self.network.fetch();  // Network I/O blocks!
}

Best Practices

Regular Maintenance

Weekly (active development):

git pull && ./install.sh  # Pulls latest and rebuilds

After system updates:

# If Rust/GCC updated, run manual recovery
cargo clean && rm -rf ~/.horus/cache && ./install.sh

CI/CD Integration

# In CI pipeline
./install.sh || (cargo clean && rm -rf ~/.horus/cache && ./install.sh)

Debugging Workflow

1. First: Check horus works

   Copy

   horus --help
   

2. If issues: Update

   Copy

   git pull && ./install.sh
   

3. If errors: Manual recovery

   Copy

   cargo clean && rm -rf ~/.horus/cache && ./install.sh
   

Getting Help

If you're still having issues:

1. Try manual recovery:

   Copy

   cargo clean && rm -rf ~/.horus/cache && ./install.sh
   

2. Add Debug Logging:

   Copy

   // Add hlog!(debug, ...) in your nodes to trace execution
   hlog!(debug, "Node state: {:?}", self.state);
   

3. Test with Minimal Example:

   • Strip down to simplest possible code
   • Add complexity back one piece at a time
   • Identify what causes the error

4. Check System Resources:

   Copy

   # Check available shared memory
   df -h /dev/shm
   
   # Check HORUS files
   ls -lh /dev/shm/horus/topics/
   
   # Clean if needed
   rm -rf /dev/shm/horus/
   

5. Report the issue:

   • GitHub: https://github.com/softmata/horus/issues
   • Include: full error message, minimal code example, OS and platform

Next Steps

• Installation - First-time installation guide
• CLI Reference - All horus commands
• Examples - Working code examples
• Performance - Optimization tips
• Testing - Test your nodes to prevent runtime errors