Hi HN, I'm Lakshay. Raveen and I built Cerulion, an open-source Rust robotics runtime that deterministically replays recorded multi-process runs against new code and runs existing ROS 2 nodes unmodified. Repo: https://github.com/cerulion-inc/cerulion

During my MIT PhD in off-road autonomy, a robot with two lidars and six cameras had too much latency between ROS processes. ROS 2's loaned messages need POD data, which rules out the standard sensor types like Image and PointCloud2. We ended up using a single process, which is very brittle.

And when it broke, I couldn't reproduce the heisenbug reliably. ROS bags record messages, not which inputs callbacks consumed or in what order, so replaying them doesn't reproduce execution.

Cerulion records messages, per-input reads and scheduler traces in standard MCAP. `bag play --resim all --verify` re-executes your native nodes on the recorded clock and compares outputs byte-for-byte or within tolerances. Exit 0 matches, 1 names the first divergent topic/frame, 6 means schedule divergence, and `--report` writes the verdict for CI. The trace can show LiDAR frame 47 read instead of 46. Replay uses an isolated shared-memory namespace, safe beside live graphs, and eligible ROS 2 processes replay individually.

Everything else feeds that loop, in the order data moves:

1. Run. Dependency-ordered scheduling with period/data/sync triggers, deadlines, backpressure, parallel same-level nodes and profile-based partitioning. Processes free-run without barriers and still replay deterministically. Live runs pause/resume, and UMWAIT/WFE parks idle consumers. Rust nodes (Python via source build) use a compiler-stable ABI, and macros reject wall-clock reads and thread spawns at compile time.

ROS 2: prebuilt Jazzy/Lyrical `rmw_cerulion` runs nodes unmodified in the same graph (11.12 µs median at 1 MiB loaned, MoveIt 2 unchanged). `LD_PRELOAD` gives them shared-memory loans through `std::vector`/`std::string`. `ros2 attach` bridges live DDS robots, and `ros2 migrate` rewrites C++ publish sites for loaning.

2. Move. iceoryx2 shared memory with no serialization, no payload copy and no payload-sized allocation, variable-length messages included. A 1 MiB `sensor_msgs/Image` round trip between two processes takes 4.08 µs median / 7.38 µs p99 (under 2x) with recording on, versus 10.95 ms for stock ROS 2 Jazzy. p50 stays flat from 64 B to 16 MiB.

3. Capture. At that round-trip latency we can afford always-on or event-based bagging. Flashback keeps ~30 seconds in memory and saves it automatically on crashes, topic stalls or declared incidents.

4. Reach. Account-paired robots and desks find each other without addresses. Pub/sub is authenticated and on demand, with signed grants the robot verifies offline. Iroh QUIC hole-punches or relays, and opening the LAN up routes through mDNS and QUIC/TLS 1.3. Remote topics mirror into local shared memory.

5. Diagnose. `cerulion doctor` finds host problems and names the fix. Hindsight (early access) gives coding agents recording analysis over MCP.

Quickstart: run obstacle avoidance, record, change a controller constant, verify the divergence. The CLI needs a free account, used for automatic robot pairing. Everything runs on Linux (x86_64/ARM64), macOS and Android (JNI), with more platforms coming.

We'd love to hear your feedback! If you run ROS 2 on a real robot, point this at your stack and tell us what breaks.

Docs: https://docs.cerulion.com

Site: https://cerulion.com

Comments URL: https://news.ycombinator.com/item?id=50474106

Points: 1

# Comments: 0