ros2probe · Non-intrusive Observability for ROS 2

The problem

Observability tools for ROS 2 are themselves observed.

Existing graph, CLI, and recording tools all join the middleware graph. They perturb the very properties engineers are trying to measure.

Problem 01

ros2 daemon

Runs as a middleware participant to cache the graph, continuously participating in discovery. Contributes to O(N²) discovery traffic and exhibits graph-state inconsistency when nodes crash or networks partition.

Problem 02

ros2 topic hz / echo / bw

Creates a new subscriber just to observe. Triggers QoS negotiation, injects receive-side load, and appears in ros2 topic list. The act of measurement deforms the system.

Problem 03

rosbag2

Registers as a full middleware participant with subscribers for every recorded topic. Observation and storage cannot be separated. Under constrained links the recorder itself starves the control plane.

Our approach

A participant-free wire tap with adaptive visibility.

ros2probe sits outside the middleware runtime. It reads wire frames at the kernel network path, filters them on a session-driven endpoint whitelist, and falls back gracefully when IP fragmentation or shared-memory transports get in the way. Today we parse the RTPS wire format for DDS; the same architecture will absorb the Zenoh wire format as a parser plug-in.

Non-intrusive Observation

Passive Observer architecture. No middleware participant is created at steady state. Logical topology integrity is preserved. Discovery, QoS negotiation, and data-plane load are untouched.

Adaptive Kernel-side Selectivity

Session-level topic interest compiles into a kernel filter. In fragmentation-avoided deployments: perfect per-packet selectivity. Otherwise: a fragment-aware fallback that preserves correctness without assuming any specific middleware configuration.

Discovery Race-free Selectivity

Data-centric, event-driven GID synchronisation: each SEDP announcement immediately updates the kernel whitelist. RTPS protocol ordering guarantees the GID is registered before any DATA can arrive, eliminating races without a time-based window.

Adaptive Visibility

The main observer stays zero-participant. When the user explicitly observes a shared-memory topic, ros2probe spawns a short-lived, namespace-isolated subprocess that joins the DDS domain only inside its mount namespace — the runtime itself never becomes a middleware participant.

How it works

Architecture, features, and the research behind it.

Switch between tabs for the layered architecture, the feature surface, live demos, and the paper.

Every other ROS 2 observability tool attaches somewhere inside the application stack: rclcpp, rcl, rmw, or the middleware itself. ros2probe is the only layer that sits below the middleware, in the kernel network path.

Attachment view · where each tool hooks

┌────────────────────────────────────────────┐
│  ROS 2 application                         │
│  (your nodes, callbacks, lifecycle)        │
├────────────────────────────────────────────┤
│  rclcpp / rclpy       <-- ros2_tracing     │
│  rcl                  <-- ros2_tracing     │
├────────────────────────────────────────────┤
│  rmw abstraction      <-- CARET            │
├────────────────────────────────────────────┤
│  Middleware runtime                        │
│  (FastDDS, CycloneDDS ...)                 │
│    participants / readers / writers        │
│                       <-- ros2 topic echo  │
│                           rosbag2          │
│                           ddsrecorder      │
│                           FastDDS Stats    │
├────────────────────────────────────────────┤
│  Wire protocol                             │
│  RTPS  (Zenoh: planned)                    │
├────────────────────────────────────────────┤
│  Kernel network stack                      │
│  UDP / IPv4 / IPv6 / NIC                   │
│                                            │
│    eBPF socket filter  <-- ros2probe       │
│    zero-copy ring buf                      │
└────────────────────────────────────────────┘

  Every other tool  =  inline attachment
  ros2probe         =  out-of-band tap

Runtime view · it COPIES, it does not STEAL

 ROS 2 normal flow          ros2probe observer
 (nothing changes)          (reads a copy)

 ┌───────────────┐          ┌───────────────┐
 │ Node A (sub)  │          │ rp CLI / gui  │
 │ receives      │          │ shows         │
 │ normally      │          │ live state    │
 └───────┬───────┘          └───────┬───────┘
         ▲                          ▲
 ┌───────┴───────┐          ┌───────┴───────┐
 │ Middleware    │          │ ros2probe     │
 │ (FastDDS...)  │          │ daemon        │
 └───────┬───────┘          └───────┬───────┘
         ▲                          ▲
 ┌───────┴───────┐          ┌───────┴───────┐
 │ Kernel socket │          │ eBPF filter   │
 │ duplicates    ├──copies─▶│ reads a copy  │
 └───────┬───────┘          └───────────────┘
         ▲
 ┌───────┴───────┐       The kernel duplicates
 │ NIC / wire    │       the sk_buff. One copy
 │ (unchanged)   │       follows the normal path
 └───────┬───────┘       down to the NIC. Another
         ▲               copy is handed to the
 ┌───────┴───────┐       eBPF filter. ros2probe
 │ Node B (pub)  │       never removes or modifies
 └───────────────┘       the original packet.
          

The others · inline

Attach inside the stack

ros2_tracing adds tracepoints in rclcpp / rcl. CARET wraps rmw with LD_PRELOAD. rosbag2, ros2 topic echo, ddsrecorder, and FastDDS Statistics all create endpoints in the middleware itself. Each becomes part of the stack being observed.

ros2probe · out of band

Tap from below the middleware

A kernel-space eBPF socket filter sees wire frames as they leave or enter the NIC. Userspace reassembles, decodes, and reconstructs the graph. The middleware never sees ros2probe.

Userspace pipeline

L1 · Capture

Zero-copy ring buffer, per-interface workers

Kernel timestamps arrive with each frame. Loopback is included by default so intra-host UDP traffic is captured too.

L2 · Wire decoder

RTPS parser, fragment reassembly

Vendor-neutral across DDS implementations. OMG RTPS is the only ABI. The Zenoh wire format will plug in at the same layer.

L3 · Graph reconstruction

Participants, endpoints, nodes, topics

ros_discovery_info binds endpoint GUIDs to ROS node namespaces. TTL sweep keeps state fresh when nodes disappear.

L4 · Applications

CLI · GUI · MCAP recorder

A single Unix socket command bus multiplexes all consumers. Sessions open and close, the kernel filter follows automatically.

∅

Zero ROS subscriptions

ros2probe never joins the middleware graph. Your ros2 topic list stays clean.

⇅

Full graph reconstruction

Participants, endpoints, node names, pub/sub relationships, all derived from wire discovery and ros_discovery_info.

◉

SHM topic visibility

SHM-only topics are identified and made observable on demand via namespace-isolated shadow subscribers.

⟳

Live topic metrics

Publish rate (hz), bandwidth (bw), end-to-end delay, and decoded echo, all with sliding-window statistics.

▣

MCAP recording

Bag files with optional zstd / lz4 compression, pause / resume, and per-topic selection.

⌘

CLI + GUI

Same data, two interfaces. rp for shells and scripts, rp gui for visual exploration.

◎

Topic classification

tf, parameter_events, rosout, debug, and SHM-only topics are automatically classified and separated in every UI.

⚡

Data-centric discovery sync

Each SEDP event immediately updates the kernel GID whitelist. RTPS protocol ordering ensures the GID is registered before DATA arrives — no time-based window needed.

⌖

Vendor-neutral

Wire-level observation only. No vendor-specific symbols, no middleware version coupling, no BTF/CO-RE fragility.

1. Start the runtime

# auto-escalates to root if needed (CAP_NET_ADMIN + CAP_NET_RAW)
rp run

2. Observe topics without touching the graph

rp topic list
rp topic info /chatter
rp topic hz    /chatter
rp topic bw    /scan
rp topic delay /imu
rp topic echo  /chatter --field data

3. Record a bag without joining the middleware graph

rp bag record /chatter /scan -o session.mcap --compression-format zstd

4. Open the desktop UI

rp gui

Not all of these tools are built for the same thing. ros2_tracing and CARET are performance-analysis tools for callback chains and executors. rosbag2 and ddsrecorder are loggers. ros2 daemon is a CLI cache. ros2probe aims at live middleware-level observation. The table below makes this explicit in the first row.

	ros2 daemon	ros2 topic	rosbag2	ros2_tracing	CARET	ros2probe
Purpose	CLI graph cache	Live topic CLI	Message logging	Callback / executor tracing	E2E callback-chain latency	Live middleware-level observation
Graph observation	△	✗	✗	✗	△	●
Message recording	✗	✗	●	✗	✗	●
Callback / executor trace	✗	✗	✗	●	●	✗
Non-intrusive	✗	✗	✗	△	△	●
Process restart not required	●	●	●	●	✗	●
Middleware-neutral	✗	✗	△	△	✗	●
Steady-state overhead	high	none (off)	none (off)	low	high	minimal
Observer Effect on graph	yes	yes	yes	limited	limited	none

● full support · △ partial / conditional · ✗ not supported

ros2_tracing and CARET solve a different problem (internal performance analysis) and are complementary to ros2probe, not competitors.

Abstract

Observability tools for ROS 2 are themselves observed: existing daemons, ros2 topic CLIs, and rosbag2 recorders all join the middleware graph and perturb discovery, QoS negotiation, and data-plane load. These are precisely the properties engineers try to measure.

We present ros2probe, a middleware-level observability architecture that taps wire frames at the kernel network path with an eBPF socket filter, never creating a middleware participant at steady state. The design contributes (i) a passive observer that preserves logical topology integrity, (ii) adaptive kernel-side semantic selectivity, (iii) discovery race-free selectivity through data-centric, event-driven GID synchronisation, and (iv) an adaptive visibility strategy for shared-memory blind spots.

On a real testbed of laptops, a Jetson Orin Nano, and a Raspberry Pi running teleoperation, IMU, and sensor-grade LiDAR and camera workloads derived from actual product specifications (Velodyne / Ouster / Hesai LiDAR tiers, 720p / 1080p RGB and depth streams), ros2probe eliminates observer-induced network overhead, contributes zero SPDP / SEDP traffic beyond ros2 daemon, lowers CPU and RSS cost relative to both ros2 daemon and the active rosbag2 / ros2 topic hz tools, and sustains a drop-rate saturation frontier at least on par with rosbag2 under a synthetic rate × size stress sweep. The architecture is not tied to any specific middleware: RTPS is supported today, Zenoh is planned.

StatusPre-submission · in preparation

TrackMiddleware / systems

ArtifactsOpen-source runtime + CLI + GUI

SupportedFastDDS, CycloneDDS · Zenoh planned

PlatformLinux 5.15+ · x86_64 · aarch64

Download preprint (PDF) Supplementary material

Four contributions

Non-intrusive Observation

Passive Observer architecture. No middleware participant at steady state. The monitoring tool does not become part of what it measures.

Adaptive Kernel-side Selectivity

Session-level topic interest compiles to a kernel filter. Fragment-aware fallback preserves correctness regardless of middleware configuration.

Discovery Race-free Selectivity

Data-centric, event-driven GID synchronisation: each SEDP announcement immediately updates the kernel whitelist. RTPS protocol ordering structurally prevents discovery-vs-data races.

Adaptive Visibility

The main observer stays zero-participant. SHM-only topics become visible on demand: ros2probe launches a short-lived subprocess inside an isolated mount namespace, and only that subprocess joins the DDS domain. The runtime itself never registers as a middleware participant.

Experiments run on a real testbed (laptops, Jetson Orin Nano, Raspberry Pi 4/5) with four workloads derived from actual sensor product specifications: teleoperation (S1 · Twist 100 Hz), IMU (S2 · 200 Hz), a four-tier LiDAR ladder (S3 · 2D → VLP-16 → VLP-32C → HDL-64E / OS1-128), and a four-tier camera ladder (S4 · JPEG 720p → Depth 640×480 → RGB 720p → RGB 1080p).

Experiment 01

Observer network impact under bandwidth constraint

Wired and Wi-Fi links, each of S1–S4 run independently. We measure Δ(TX+RX) added by the observer, plus drop rate and end-to-end latency (p50 / p95 / p99) on the observed topic.

vs rosbag2 · ros2 topic hz / echo

Experiment 02

Discovery impact

SPDP and SEDP pub/sub packet counts per minute over a static N-node ROS 2 graph (N ∈ {2, 5, 10, 20, 50}). ros2probe contributes zero at steady state.

vs ros2 daemon

Experiment 03

CPU and memory overhead

Four scenarios (idle · discovery-only observer · bag record · topic hz) × workloads {S1, S2, S3 64ch, S4 1080p} run independently × three hosts (Laptop, Orin Nano, Pi). pidstat CPU % and VmRSS.

vs ros2 daemon · rosbag2 · ros2 topic hz

Experiment 04

Real-time throughput ceiling (synthetic stress)

Synthetic /stress topic swept across rate ∈ {10…1000} Hz × size ∈ {4 KB…16 MB} on laptop localhost. Drop-rate 1% saturation frontier compared head-to-head.

vs rosbag2

Experiment 05 · optional

Auxiliary verification

Inter-vendor graph reconstruction (FastDDS + CycloneDDS mixed), SHM-only topic visibility via rp discover, and loopback-interface auto-capture for intra-host UDP traffic.

vendor / SHM / loopback

Figure · Exp.1 results — coming soon

Result figures will be added once the experiments finish.

Command	Description
rp run	Start the runtime daemon (auto-escalates to root if needed)
rp gui	Launch the desktop GUI
rp topic list	List topics with recordable / internal separation
rp topic hz <topic>	Live publish rate (mean / min / max / stddev)
rp topic bw <topic>	Live bandwidth
rp topic delay <topic>	End-to-end delay from header.stamp
rp topic echo <topic>	Stream decoded messages (rclpy-backed)
rp bag record [TOPICS…]	Record MCAP bag · zstd / lz4 / none
rp node list · info	Discovered nodes and their endpoints
rp discover	Force ros_discovery_info broadcast (SHM-aware shadow mode)

Requirement	Notes
Linux 5.15+	eBPF socket filter, AF_PACKET TPACKET_V3
ROS 2 Humble+	Iron, Jazzy, Rolling also supported
Rust ≥ 1.85	Stable + nightly (rust-src) — installed via rustup
bpf-linker	Required for the eBPF program
libssl-dev	Always required (reqwest TLS)
libgtk-3-dev	Full build only (rfd file dialog) · skip for minimal
Python 3 + rclpy	Only for rp topic echo / delay
Optional	Jumbo frames, DDS XML profile that avoids IP fragmentation

ros2probe: Observe ROS 2 without being observed.

Observability tools for ROS 2 are themselves observed.

ros2 daemon

ros2 topic hz / echo / bw

rosbag2

A participant-free wire tap with adaptive visibility.

Non-intrusive Observation

Adaptive Kernel-side Selectivity

Discovery Race-free Selectivity

Adaptive Visibility

Architecture, features, and the research behind it.

Attach inside the stack

Tap from below the middleware

Userspace pipeline

Zero-copy ring buffer, per-interface workers

RTPS parser, fragment reassembly

Participants, endpoints, nodes, topics

CLI · GUI · MCAP recorder

Zero ROS subscriptions

Full graph reconstruction

SHM topic visibility

Live topic metrics

MCAP recording

CLI + GUI

Topic classification

Data-centric discovery sync

Vendor-neutral

1. Start the runtime

2. Observe topics without touching the graph

3. Record a bag without joining the middleware graph

4. Open the desktop UI

Four contributions

Non-intrusive Observation

Adaptive Kernel-side Selectivity

Discovery Race-free Selectivity

Adaptive Visibility

Observer network impact under bandwidth constraint

Discovery impact

CPU and memory overhead

Real-time throughput ceiling (synthetic stress)

Auxiliary verification

Same data, two interfaces.

CLI at a glance

Install.

1 · System dependencies

2 · Rust toolchain

3 · Clone

4 · Build & Install (pick one)

If you use ros2probe in your research.