Bugs That Produce Plausible Output
ros2_pi turns one webcam on a Raspberry Pi 5 into a live triangle mesh of a room. The Pi captures and streams over Wi-Fi; a GPU machine detects ORB keypoints, estimates 6-DoF camera pose with PnP, runs Depth Anything V2 through ONNX Runtime on CUDA, fuses depth into a TSDF volume, extracts a surface with marching cubes, and streams the mesh to a hand-written WebGL dashboard. The code started on 2026-09-08. On 2026-09-19 the whole chain ran end to end for the first time. It is the C++ successor to an earlier Python version,
piros2, and it is a learning project, not production.
The rule from the start: every performance claim is printed by a script that asserts on it. Each pipeline stage has a gate under tools/gates/ that runs the real system on a recorded clip and fails on a number. These are the bugs those gates found, and the thing they have in common.
1. Six days of correct-looking 3D from one inverted rotation
On the day the PnP odometry landed, its gate failed on a simple check: pan the camera right, and the published camera pose turns left.
Everything else had looked fine for six days. The frame moved when the camera moved. The residual check passed, because it measures alignment error and is blind to sign. The rotation-fit unit test checked everything about the estimate except the direction of its inverse. And the reconstruction looked like a room, because a TSDF built from consistently mirrored poses still produces a surface. Mirrored poses are self-consistent. Every frame agrees with every other frame about the wrong answer.
The root cause was one line. The pose update composed the point motion M where it needed the camera motion M^-1:
// in camera_step(): the camera moves opposite to the points it observes
return point_motion.inv();The fix was a transpose. The measured effect, as the gap between the fused surface and a reference built from the same frames: 1.344 m before, 0.446 m after. A 3x improvement in the thing the project exists to produce, from a bug that no visual inspection had flagged.
A CameraStep test suite now closes that loop, so the direction of the inverse is asserted rather than assumed.
2. CUDA was installed, working, and not being used
Depth inference ran at 517 ms a frame inside the pipeline. A standalone probe, same libraries, same flags, same model, ran at 51 ms on the GPU. No error. One log line, easy to miss, named a CUDA library that could not be loaded, and ONNX Runtime fell back to the CPU provider without complaint.
The difference was who owned the executable. In ROS 2, nodes are loaded as shared libraries into component_container_isolated, which is someone else’s binary. The CUDA execution provider is itself dlopened by ONNX Runtime, and a dlopened library resolves its dependencies through its own RPATH, its loader chain, and the main executable’s paths. The CUDA libraries were installed without root, in a pinned location, and found through my library’s search path.
The trap: modern linkers emit DT_RUNPATH by default, not DT_RPATH, and RUNPATH is not inherited down a dlopen chain. It applies to the library’s own direct dependencies and stops there. Measured in a standalone executable to be sure: the same build with RUNPATH ran at 236.6 ms on CPU, and with RPATH at 51.2 ms on CUDA.
Two fixes, belt and braces: link with -Wl,--disable-new-dtags so the old inheriting RPATH is emitted, and explicitly dlopen the CUDA libraries by absolute path before creating the session, so they are already resident when the provider asks for them.
The gate now asserts the provider name as well as the time. Depth must report CUDAExecutionProvider and come in under an 80 ms budget. Final: 55.1 ms mean, 17.4 Hz.
3. A zero-copy test that measured the one case that cannot fail
ROS 2 intra-process communication can hand a message between nodes in the same process without copying it. The gate checks this directly: subscribers record the address of the buffer they received, and the gate compares it with the address the publisher sent.
First run: 429 of 429 buffers matched. Then a second consumer was added for an unrelated reason, and the next run read 0 of 574.
rclcpp moves a published unique_ptr into the last subscription that takes ownership, and copies it for every other one. With one subscriber, there is no “other one”, so the test passes whatever the subscribers declare. The configuration being tested was the only configuration in which the test could not fail.
The fix was for subscribers to take ConstSharedPtr, which lets every consumer share the one buffer, and for the gate to require at least two consumers. The control run turns intra-process communication off and must match almost nothing. A passing run now reads 504 of 504 with it on, and 0 of 395 with it off.
4. The smaller ones, same shape
| Gate result | What was actually happening |
|---|---|
| Keypoint cost 0.00 ms, PASS | The gate grepped the last stats line, which now came from the depth node’s idle window. Gates select by node name and require cost > 0 |
| Keypoints just inside the 8 ms budget | The whole workspace was building at -O0. 7.90 ms became 5.99 ms at -O2 |
| An 11-point regression in match rate | The gate compared a 20 s window against the average of a 60 s clip. Not a regression. Counting 0/0 frames as zero was worth another 5 points |
| 12% of fused frames colourless | The executor runs callbacks in registration order, so depth sometimes arrived before its colour frame |
| Teardown gate PASS | timeout put ros2 launch in a process group that Ctrl-C never reached, and the gate signalled only one of the two recipes it should have checked |
Not one of these is a crash. Each one produced a number, and the number was plausible.
What a gate looks like
The depth gate is the template. Run one is the real pipeline and asserts:
- the provider is
CUDAExecutionProvider, - mean cost is under 80 ms and the rate over 10 Hz,
- every output frame is byte-identical to its source frame (no stale or reordered pairs),
- zero NaN or out-of-range depths, and at least 85% coverage of the clip.
Run two is the adversarial control: the same binary with use_cuda:=false. It must report the CPU provider, and it must fail the budget.
A budget that the thing it excludes can meet is not an assertion.
That one line is the whole idea. A threshold that a known-bad configuration also passes discriminates nothing, and the only way to know is to run the bad configuration on purpose. The zero-copy gate’s control is intra-process communication switched off, and it must match almost nothing.
The measured pipeline
Every number below is printed by a gate, on the same recorded clip, on a GTX 1660 SUPER.
| Stage | Measured |
|---|---|
| Capture (Pi 5, 720p MJPEG over Wi-Fi) | 44 to 59 Hz, kernel-stamped |
| Decode | 1.90 ms / frame, zero-copy to every consumer |
| ORB keypoints (x500) | 5.99 ms against an 8 ms budget |
| PnP odometry | 1.42 px mean reprojection over 97 inliers |
| Depth (CUDA) | 55.1 ms mean, 58.2 ms p95, against 80 ms |
| TSDF fusion (15 mm voxels) | 15.3 ms against 20 ms |
| Marching cubes | 790,668 triangles in 2.8 s, decimated to 120,000 |
| Dashboard | 10 Hz stats for 0.77% of pipeline throughput |
Takeaway
The expensive bugs in a perception pipeline are not the ones that crash. They are the ones where the output is internally consistent: mirrored poses that agree with each other, a CPU fallback that is merely slow, a zero-copy test in the one configuration that cannot copy.
So the question to ask of every check is not “does it pass” but “what does this not touch?”, and the cheapest way to answer it is to run a configuration you know is wrong and confirm the check fails. If it does not, it was never checking anything.
The next step for this pipeline is closing the loop, which is what turns visual odometry into SLAM.