Ground robots
How Foxborne rebuilds a ground robot incident from the flight log, the companion journal and the ground receive log, worked through example incident INC-0143: vibration, a USB radio disconnect and a PX4 Hold 11.88 s after the last ground heartbeat.
On this page10
Ground robots drive where the ground is rough and the radio path runs close to it. Foxborne covers tracked and wheeled ground robots. When one stops mid-lane, the review board asks two questions: why did it stop, and why 11.9 s after the last heartbeat?
Example incident INC-0143 answers both from three sources. Every value on this page comes from the console's example dataset.
The example platform#
The example dataset's ground robots are T4 tracked UGVs, UGV-01 to UGV-06, built for route clearance and breaching support. A T4 weighs 312 kg, is listed at 2.2 m/s and drives on skid-steer tracks under PX4's differential rover.
| Part | In the example dataset | Recorded by |
|---|---|---|
| Flight controller | Pixhawk 6X, PX4 v1.16.0 | The flight log |
| Companion computer | Jetson AGX Orin 64 GB, Ubuntu 22.04, running perception_node, nav_bridge, mavlink-routerd and the collector | The journal and the collector's process samples |
| Mesh radio | MANET, USB Ethernet | Nothing. It keeps no log Foxborne can read. |
| GNSS receiver | u-blox F9P, dual antenna | The flight log |
| Lidar | 32 beams, Ethernet | The journal |
| Battery | 48 V Li-ion, 14S | The flight log |
The radio hangs off the companion computer's USB bus as a network interface, and INC-0143 happens on that link. Vehicle parts and their logs maps each part in detail.
Rules tuned for tracked vehicles#
The example configuration scopes several rules to the T4:
| Rule | T4 setting in the example configuration |
|---|---|
| vibration_high | Above 24 m/s² for 1 s, against 12 m/s² on the Q4 quad |
| device_disconnect | Any disconnect on an allowlisted port, applied to T4 vehicles |
| memory_pressure | 24,576 MiB, an override of the 6,144 MiB used on the Q4 |
| battery_sag | More than 2.0 V in 2 s above 70% track current. Version 0.9, off by default |
The 24 m/s² vibration threshold is a program choice for tracked vehicles, not a PX4 number. Set each platform's value from runs you know were good.
The sources#
INC-0143 happened on UGV-02, run R-0934, in Breach lane 2 on 25 Sep 2026. The run aligned three files with a widest bound of ±55 ms.
| Source | File or lane | Clock and anchor | Bound |
|---|---|---|---|
| Flight log | 15_37_12.ulg | PX4 boot clock, anchored by GPS time from sensor_gps, 498 fixes | ±40 ms |
| Companion journal | ugv02-agx, 6 units, 7,411 records | Companion wall clock, chrony synchronized, offset +0.9 ms | ±6 ms |
| Ground receive log | System ID 22, 509 records | Range control receiver clock, NTP, offset −0.3 ms | ±15 ms plus latency: median 88 ms, 95th percentile 380 ms |
What happened#
The report's sequence has eight events:
| Event | Time, UTC | Source | What it shows | Level |
|---|---|---|---|---|
| E1 | 15:41:00.912 | Flight log | Vibration above 24 m/s² for 1 s in the obstacle belt | Observed |
| E2 | 15:41:02.604 | Journal | Kernel: USB device 1-2.3 disconnected (mesh radio) | Correlated |
| E3 | 15:41:02.790 | Journal | mavlink-routerd cannot reach range control | Observed |
| E4 | 15:41:03.713 | Ground receive | Heartbeat due and not received. Nothing for 31.0 s | Observed |
| E5 | 15:41:05.286 | Flight log | No ground station heartbeat on TELEM2 | Observed |
| E6 | 15:41:14.293 | Flight log | GCS connection loss: switching to Hold | Observed |
| E7 | 15:41:32.214 | Journal | Kernel: usb0 registered again | Observed |
| E8 | 15:42:10.511 | Flight log | Mission resumed from range control | Observed |
The vibration came first. accel_vibration_metric in vehicle_imu_status read 24.31 m/s² at 15:41:00.912, held above the threshold to 26.84 m/s² a second later and peaked at 38.6 m/s² at 15:41:02.312.
The kernel dropped the radio 1.69 s after the vibration onset and 0.29 s after its peak, 227.6 m from the lane's start line. It logged usb 1-2.3: USB disconnect, device number 5, and unregistered the usb0 network interface 7 ms later.
The report files that disconnect as correlated with the vibration: 1.69 s apart against a bound of ±46 ms, so the order is supported.
mavlink-routerd logged Error sending udp packet (Network is unreachable) at 15:41:02.790. Range control's last receipt from system ID 22 had arrived at 15:41:02.713, and the next came 31.0 s later.
Why PX4 held 11.88 s after the last heartbeat#
COM_DL_LOSS_T is 10 s on UGV-02, yet the Hold came 11.88 s after the last ground station heartbeat reached PX4. PX4's timer does not start at that heartbeat. It starts at the last telemetry_status report that still carried it.
PX4's MAVLink receiver keeps heartbeat_type_gcs true for up to 2.5 s after a ground heartbeat (HEARTBEAT_TIMEOUT_US). It re-checks the flag on every heartbeat it receives, and the companion computer's own heartbeats kept arriving over the UART once a second.
Last ground heartbeat PX4 handled 15:41:02.412
Last telemetry_status still true 15:41:04.286 1.874 s later, inside the 2.5 s timeout
First telemetry_status reading false 15:41:05.286 2.874 s after the heartbeat
COM_DL_LOSS_T, 10 s from 15:41:04.286 15:41:14.286
Hold, from NAV_DLL_ACT 1 15:41:14.293 11.881 s after the heartbeatfailsafe_flags gives the cause, gcs_connection_lost, and NAV_DLL_ACT = 1 selects Hold. The report files the Hold as observed, "As configured". The UGV braked to a stop 1.0 s later, 242.9 m from the start line, and held for 56.2 s.
Range control resumed the mission at 15:42:10.511 with DO_SET_MODE from system 255. failsafe_entered walks through PX4's timing in full.
The radio that keeps no log#
The mesh radio keeps no log Foxborne can read. The kernel on the companion computer saw it leave the USB bus at 15:41:02.604 and return at 15:41:31.802, and usb0 registered again at 15:41:32.214.
For those 29.2 s, the radio's own state is not observed. Whether it rebooted, lost power or only lost its USB link, nothing in the three sources says. The vehicle drawing labels the radio Keeps no log, and State not observed while its link to the companion is down.
The kernel records that the device disconnected, not why. A loose connector, a cable fault and a radio reset look the same in this log.
The finding#
The investigator wrote the finding as a Hypothesis:
The mesh radio dropped off the companion computer's USB bus 1.69 s after vibration crossed 24 m/s² in the obstacle belt, and 0.29 s after its 38.6 m/s² peak. Range control stopped hearing UGV-02, and PX4 held the vehicle 11.9 s after the last ground heartbeat reached it:
COM_DL_LOSS_Tcounts its 10 s from the lasttelemetry_statusthat still reported that heartbeat, andNAV_DLL_ACT= 1 selects Hold. That vibration unseated the radio's USB connector is a hypothesis until the connector is inspected.
The next check is physical: inspect the USB-C connector on the UGV-02 radio for wear. Then run Lane 2 again with the connector strain-relieved, recording dmesg -w and accel_vibration_metric at 10 Hz. If the disconnect repeats with the connector secured, test the radio on the shaker table.
What the evidence shows and does not#
| The evidence shows | It does not show |
|---|---|
| Vibration crossed 24 m/s² 1.69 s before the kernel dropped the radio | That the vibration caused the disconnect |
| PX4 held exactly as configured: 10 s after its last report of a ground heartbeat | What the radio did during the 29.2 s it was off the bus |
| Range control heard nothing from system ID 22 for 31.0 s | When the last heartbeat left the vehicle to better than 380 ms, the run's 95th percentile latency |
The reviewer's open question reaches across the fleet: did UGV-01 and UGV-03 log USB resets in the belt on their lane rehearsals?
Terrain can cause the same gap#
A ground robot's radio path runs low, so terrain can mask it too. In the example dataset's INC-0137, UGV-05 loses the link while it patrols the north side of a berm. Read the map shows how Foxborne checks the line from the range control mast against the site's elevation data.
Related#
- vibration_high reads
accel_vibration_metricagainst the platform threshold. - device_disconnect turns the kernel's disconnect line into a finding.
- failsafe_entered explains how PX4 times a lost ground link and picks the action.
- Multirotor drones shows a telemetry gap with a different cause.