TensorPool API

HORUS provides efficient tensor memory management through shared memory pools, enabling zero-copy data sharing between processes.

Overview

The TensorPool system consists of:

• TensorPool - tensor allocation via shared memory (mmap-backed)
• TensorHandle - RAII wrapper for automatic memory management
• Tensor - 168-byte Pod tensor descriptor (metadata only)
• Device - Pod-safe device location (Device::cpu(), Device::cuda(N))

For most users, the high-level domain types (Image, PointCloud, DepthImage) are the recommended API — they handle pool management automatically. TensorPool and TensorHandle are internal types for advanced use cases.

Recommended: Use Domain Types

Most users should use Image, PointCloud, or DepthImage instead of working with TensorPool/TensorHandle directly. These types handle pool management automatically:

use horus::prelude::*;

// Create an image — pool allocation is automatic
let mut img = Image::new(640, 480, ImageEncoding::Rgb8)?;
img.fill(&[255, 0, 0]); // Red

// Send via topic — zero-copy, only the 224-byte ImageDescriptor travels
let topic: Topic<Image> = Topic::new("camera/rgb")?;
topic.send(&img);

// Receive
if let Some(img) = topic.recv() {
    let pixel = img.pixel(0, 0); // Direct pixel access
}

// Point clouds work the same way
let pc = PointCloud::new(10000, 3, TensorDtype::F32)?;
let depth = DepthImage::new(640, 480, TensorDtype::F32)?;

See Basic Examples — Camera Image Pipeline for complete send/recv examples.

Auto-Managed Pools (Advanced)

If you need raw tensors rather than a domain type, the simplest path is Topic<Tensor>, which auto-manages a pool per topic:

use horus::prelude::*;
use horus::types::Tensor;

let topic: Topic<Tensor> = Topic::new("camera.rgb")?;

// Allocate from the topic's auto-managed pool
let handle = topic.alloc_tensor(&[1080, 1920, 3], TensorDtype::U8, Device::cpu())?;

// Write pixel data
let pixels = handle.data_slice_mut()?;
// ... fill pixels ...

// Send — only the 168-byte descriptor goes through the ring buffer
topic.send_handle(&handle);

// Receiver side
let topic: Topic<Tensor> = Topic::new("camera.rgb")?;

if let Some(handle) = topic.recv_handle() {
    let data = handle.data_slice()?;  // Zero-copy access
    println!("Shape: {:?}, Dtype: {:?}", handle.shape(), handle.dtype());
}
// Refcount decremented automatically on drop

The pool is created lazily on first use and shared across all Topic<Tensor> instances with the same name — even across processes. Pool IDs are derived deterministically from the topic name using FNV-1a hashing.

Manual CPU TensorPool

For advanced use cases requiring direct pool control.

Creating a Pool

use horus::prelude::*;
use horus::memory::{PoolAllocator, TensorPool, TensorPoolConfig};
use std::sync::Arc;

// Create with default config (1024 slots, 1GB pool)
let pool = Arc::new(TensorPool::new(1, TensorPoolConfig::default())?);

// Or customize
let config = TensorPoolConfig {
    pool_size: 2 * 1024 * 1024 * 1024, // 2GB
    max_slots: 2048,
    slot_alignment: 64,                 // Cache-line aligned
    allocator: PoolAllocator::Mmap,     // the only backend today
};
let pool = Arc::new(TensorPool::new(1, config)?);

TensorPool::new is create-or-open: a consumer process that calls it with the same pool_id attaches to the pool that already exists in shared memory rather than making a second one.

Allocating Tensors

use horus::prelude::*;
use horus::memory::TensorHandle;
use horus::types::Tensor;

// Allocate a 1080p RGB image
let handle = TensorHandle::alloc(
    pool.clone(),
    &[1080, 1920, 3],
    TensorDtype::U8,
    Device::cpu(),
)?;

// Access data
let data: &mut [u8] = handle.data_slice_mut()?;

// Get tensor descriptor (for sending through Topic)
let tensor: &Tensor = handle.tensor();

// Clone increases refcount automatically
let handle2 = handle.clone();
// Refcount decremented on drop

Supported Data Types

TensorPoolConfig

PoolAllocator has a single variant, Mmap. There are no preset constructors — build the struct or start from TensorPoolConfig::default().

Pool Statistics

let stats = pool.stats();
println!("Pool: {}/{} slots used, {}/{} bytes",
    stats.allocated_slots, stats.max_slots,
    stats.used_bytes, stats.pool_size);

GPU Support (CUDA)

Device can describe a GPU location — Device::cuda(0), Device::parse("cuda:1") — and a Tensor descriptor can carry it:

use horus::prelude::*;
use horus::types::Tensor;

let mut tensor = Tensor::default();
let dev = Device::cuda(0);
tensor.device_type = dev.device_type;
tensor.device_id = dev.device_id;
assert!(tensor.device().is_cuda());

No pool can back that device today. PoolAllocator has one variant, Mmap, so allocating from a pool with a CUDA device fails:

Memory error: Allocation failed: device mismatch: requested cuda:0 but pool
backend targets cpu (backend: mmap). Use a pool created with the matching
backend, or pass Device::cpu() to alloc().

There is no cuda_available() or cuda_device_count() function in the crate, and no CudaTensorPool. Keep tensors on the CPU as the transport, and do device placement inside your inference framework (torch, ONNX Runtime, TensorRT) after reading the pool memory.

Tensor Descriptor

Tensor is a 168-byte Pod descriptor that acts as a lightweight handle to tensor data in shared memory. Only the descriptor travels through the ring buffer — the actual data stays in-place.

// Key fields and methods on Tensor:
let shape = tensor.shape();         // &[u64] - dimensions
let dtype = tensor.dtype;           // TensorDtype (pub field)
let dev = tensor.device();          // Device (cpu or cuda)
let size = tensor.size;             // Total size in bytes (pub field)
tensor.device_type = Device::cuda(0).device_type; // pub field
tensor.device_id   = Device::cuda(0).device_id;   // pub field

Device::cpu(), Device::cuda(0), Device::cuda(1), etc. describe where the data lives; device_id is a u32, so any GPU index fits in the descriptor. As the GPU section above notes, allocating from a pool still only succeeds with Device::cpu() — a CUDA device on a descriptor is metadata for your own code to act on.

Python API

The same pools are reachable from Python as horus.TensorPool and horus.Tensor. A Python process and a Rust process sharing a pool_id share the memory.

import numpy as np
import horus

pool = horus.TensorPool(pool_id=1, size_mb=1024, max_slots=1024)

# Allocate from the pool, or copy an existing array in
handle = pool.alloc((1080, 1920, 3), dtype="float32")
frame = horus.Tensor.from_numpy(np.zeros((480, 640, 3), dtype=np.uint8))

view = frame.numpy()          # zero-copy ndarray over the pool slot
print(frame.shape, frame.dtype, frame.device, pool.stats()["allocated_slots"])

There is no horus.cuda_is_available() and no horus.ai module; model loading and inference are your own code. See Python Tensors for ML for the complete Python surface.

Performance

CPU TensorPool

These come from the repo's own benchmarks/benches/tensor_pool.rs and are hardware-dependent — they are not part of the published figures in benchmarks/README.md, which covers topic/IPC latency only. Allocation cost scales with tensor size because return_slot() zeroes the data region before the slot returns to the free list.

Regardless of tensor size, only the 168-byte descriptor travels through the ring buffer — the data stays in the mmap and is read in place by the receiver. There are no GPU pool figures to quote: there is no GPU allocator in the crate.

Best Practices

1. Use auto-managed pools for raw tensors: Prefer Topic<Tensor> with alloc_tensor()/send_handle()/recv_handle() — the framework handles pool lifecycle
2. Wrap in Arc: When using manual pools, TensorPool doesn't implement Clone — use Arc<TensorPool> for sharing
3. Reuse pools: Create pools once at startup, not per-tensor
4. Use TensorHandle: Prefer TensorHandle over manual retain/release for automatic refcounting
5. Match dtypes: Ensure sender and receiver use same dtype
6. Keep tensors on the CPU: The pool cannot allocate on Device::cuda(N); move batches to the GPU inside your inference framework
7. Watch both pool limits: stats() reports allocated_slots and used_bytes — which one a leaking pipeline exhausts first depends on tensor size. With the 1GB/1024-slot default, leaked 1080p RGB frames run the data region out after ~170 allocations (173 slots used); small tensors hit max_slots first

See Also

• Tensor Messages - Tensor, TensorDtype, Device types and domain wrappers
• Topic & Shared Memory - How HORUS achieves zero-copy IPC