Multi-Machine and Zenoh

Running one graph across two machines: what must match, multicast over Wi-Fi, static peers, the bandwidth arithmetic that decides what may cross the link, VPNs, and the Zenoh RMW.

Author

Benedict Thekkel

The Three Things That Must Match

Splitting a graph across a robot and a development machine is the normal arrangement: the robot carries sensors and controllers, the desktop runs RViz, rqt and the editor. It works when three settings agree, and fails confusingly when any one of them does not.

ROS_DOMAIN_ID - nodes only see peers with the same value. The default 0 is also everyone else’s default.
ROS_AUTOMATIC_DISCOVERY_RANGE (or the deprecated ROS_LOCALHOST_ONLY) - if discovery is limited to localhost, traffic never leaves the machine.
RMW_IMPLEMENTATION - both ends must use the same DDS vendor.

All three are covered in 00_Discovery_and_RMW.ipynb. The multi-machine point is different: keeping three variables in agreement across two machines by hand is the largest available source of wasted time. Define them once in configuration management and render them to both hosts, so the answer to “are they the same” is structural rather than something to check.

# what a remote host actually has, which is not always what you set
ssh robot -t 'bash -lc "env | grep -E \"^ROS_|^RMW_|^CYCLONEDDS\""'

Beyond the three, the machines must be on the same subnet with no router in between for multicast to work, and the clocks should agree. NTP on both is worth setting up early; a robot whose clock is minutes off produces stamps that break every tf2 lookup on the desktop, and the symptom points at tf rather than at time. See ../03_Spatial_and_Temporal/02_Conventions_and_Time.ipynb.

Multicast, and Wi-Fi

Automatic discovery needs multicast, and most Wi-Fi access points drop or rate-limit it. A robot on Wi-Fi should therefore be expected to have flaky discovery rather than treated as a surprise when it does.

Check whether multicast reaches the other machine at all:

# on one machine
ros2 multicast receive
# on the other
ros2 multicast send

If that fails while ordinary ping works, multicast is the problem and no amount of domain or RMW fiddling will fix it. The answer is static peers, which skips multicast entirely:

# on the dev box
export ROS_STATIC_PEERS=192.168.2.17
# on the robot
export ROS_STATIC_PEERS=192.168.2.109
export ROS_AUTOMATIC_DISCOVERY_RANGE=SUBNET

Try it in a shell first to confirm it is the fix, then make it permanent the same way as everything else rather than leaving an export in one .bashrc that the other machine knows nothing about. Setting ROS_AUTOMATIC_DISCOVERY_RANGE=OFF alongside explicit peers gives fully deterministic discovery, which is the configuration to reach for when you want the network to stop being a variable.

For a graph bigger than two machines, a discovery server scales better than a peer list on every host; see 00_Discovery_and_RMW.ipynb.

Bandwidth Is the Real Constraint

Once discovery works, the link becomes the bottleneck, and the arithmetic is unforgiving.

Raw 1280x720 RGB8 at 30 fps is about 83 MB/s. That is beyond what any Wi-Fi link should be asked to carry, and it does not fail cleanly: the pipeline stalls, frame rates collapse, and because DDS is doing its best the symptom looks like slow nodes rather than a saturated link.

Two rules follow.

Always stream compressed between machines. image_transport’s compressed transport turns that 83 MB/s into a few MB/s; see ../05_Perception/00_Images_and_Calibration.ipynb.

Count subscribers, because each one is another copy across the link. DDS delivers per subscriber, so five nodes on the desktop subscribing to a robot’s camera pull five streams over the same Wi-Fi. This has been measured on this site’s own hardware: with five dev-box subscribers each pulling its own copy of a camera stream from a Raspberry Pi over Wi-Fi, the readers ran at about 2 frames per second each, against 14.7 Hz for a single reader - the link collapsed rather than degraded. The fix was a relay: one subscriber on the desktop takes the stream across the link once and republishes it locally, and the other four subscribe to the local copy. The public piros2 and ros2_pi projects record the measurements and the rework.

So the pattern for anything large crossing a link:

robot:    camera -> compressed topic  ----link----> desktop: relay -> local topic -> N subscribers

Related choices that follow from the same arithmetic:

Best-effort QoS for streams over wireless. Reliable delivery on a lossy link produces retry storms that degrade everything else on it.
Process where the data is. Publishing a 5 MP image so a desktop can reduce it to a pose is the wrong split; send the pose.
Compose on the robot. Nodes in one container exchange large messages as pointers instead of over the network; ros2_pi was rewritten in C++ for exactly this. See ../01_Core_Concepts/04_Executors_Lifecycle_and_Composition.ipynb.
Check with ros2 topic bw rather than estimating, on every topic that crosses the link.

VPNs and Crossing Networks

Reaching a robot that is not on your subnet (a site network, a cloud dev machine, home to office) breaks automatic discovery, because multicast does not cross routers.

A VPN makes the machines look local again: WireGuard or Tailscale puts both ends on one virtual subnet, and discovery can then work. Three caveats, all of which have bitten people:

The VPN interface must be the one DDS binds, and a multi-homed machine will pick wrong. Pin it; see 00_Discovery_and_RMW.ipynb.
Multicast over a VPN is not guaranteed. WireGuard is point-to-point and does not carry multicast, so static peers or a discovery server is required rather than optional.
Latency and MTU change behaviour. Tunnelling reduces the MTU, so large messages fragment more, and DDS reliability over a high-latency link behaves quite differently from on a LAN.

For a wide-area link, DDS is the wrong tool and there are purpose-built options: rosbridge (JSON over WebSocket, good for browsers and weak clients), Foxglove (remote visualisation and bag inspection), and the Zenoh bridge below. Fleet tooling is 08_Testing_Deployment_Ops/03_Security_and_Fleet_Ops.ipynb.

And note the security point: DDS is unauthenticated and unencrypted by default. A VPN is a perimeter around an open graph, not authentication. SROS2 is the in-band answer and is covered in 08_Testing_Deployment_Ops/03_Security_and_Fleet_Ops.ipynb.

Zenoh

rmw_zenoh_cpp replaces DDS with Eclipse Zenoh, and it exists because DDS’s weaknesses are structural rather than implementation details: multicast discovery, n-squared discovery traffic, and behaviour on lossy or wide-area links.

sudo apt install ros-jazzy-rmw-zenoh-cpp
export RMW_IMPLEMENTATION=rmw_zenoh_cpp
ros2 run rmw_zenoh_cpp rmw_zenohd          # the router, one per machine or per network

What changes:

A router, not multicast. Nodes connect to a Zenoh router, which peers with other routers. Discovery is explicit and scales with the number of routers rather than the square of the participants.
Wide-area is a design goal. Routers peer across networks without a VPN and without a discovery-server workaround.
One configuration surface instead of per-vendor XML.
Dropping a subscriber does not cost another copy over the link in the same way, because the router fans out locally.

The costs, as of Jazzy:

The router must be running, and it is the new thing that was forgotten.
It is newer. It is a supported RMW from Jazzy onward and the intended default direction, but the ecosystem’s documentation, tuning advice and accumulated folklore are all about DDS.
Not interoperable with DDS. The whole graph moves, or none of it. zenoh-bridge-ros2dds exists to bridge a DDS graph to Zenoh, which is the incremental path and is also how a Zenoh link can be used for just the wide-area hop while each site stays on DDS.

When to reach for it: a robot on a flaky wireless link, a fleet across sites, or a graph large enough that discovery traffic is a measurable cost. When not to: a single robot on a wired LAN, where DDS works and every piece of advice you will find applies to it.