Skip to content
Docs
foxborne.comRequest a pilot

Ground robots

How Foxborne rebuilds a ground robot incident from the flight log, the companion journal and the ground receive log, worked through example incident INC-0143: vibration, a USB radio disconnect and a PX4 Hold 11.88 s after the last ground heartbeat.

ExplanationMarkdown
On this page10

Ground robots drive where the ground is rough and the radio path runs close to it. Foxborne covers tracked and wheeled ground robots. When one stops mid-lane, the review board asks two questions: why did it stop, and why 11.9 s after the last heartbeat?

Example incident INC-0143 answers both from three sources. Every value on this page comes from the console's example dataset.

The example platform#

The example dataset's ground robots are T4 tracked UGVs, UGV-01 to UGV-06, built for route clearance and breaching support. A T4 weighs 312 kg, is listed at 2.2 m/s and drives on skid-steer tracks under PX4's differential rover.

PartIn the example datasetRecorded by
Flight controllerPixhawk 6X, PX4 v1.16.0The flight log
Companion computerJetson AGX Orin 64 GB, Ubuntu 22.04, running perception_node, nav_bridge, mavlink-routerd and the collectorThe journal and the collector's process samples
Mesh radioMANET, USB EthernetNothing. It keeps no log Foxborne can read.
GNSS receiveru-blox F9P, dual antennaThe flight log
Lidar32 beams, EthernetThe journal
Battery48 V Li-ion, 14SThe flight log

The radio hangs off the companion computer's USB bus as a network interface, and INC-0143 happens on that link. Vehicle parts and their logs maps each part in detail.

Rules tuned for tracked vehicles#

The example configuration scopes several rules to the T4:

RuleT4 setting in the example configuration
vibration_highAbove 24 m/s² for 1 s, against 12 m/s² on the Q4 quad
device_disconnectAny disconnect on an allowlisted port, applied to T4 vehicles
memory_pressure24,576 MiB, an override of the 6,144 MiB used on the Q4
battery_sagMore than 2.0 V in 2 s above 70% track current. Version 0.9, off by default

The 24 m/s² vibration threshold is a program choice for tracked vehicles, not a PX4 number. Set each platform's value from runs you know were good.

The sources#

INC-0143 happened on UGV-02, run R-0934, in Breach lane 2 on 25 Sep 2026. The run aligned three files with a widest bound of ±55 ms.

SourceFile or laneClock and anchorBound
Flight log15_37_12.ulgPX4 boot clock, anchored by GPS time from sensor_gps, 498 fixes±40 ms
Companion journalugv02-agx, 6 units, 7,411 recordsCompanion wall clock, chrony synchronized, offset +0.9 ms±6 ms
Ground receive logSystem ID 22, 509 recordsRange control receiver clock, NTP, offset −0.3 ms±15 ms plus latency: median 88 ms, 95th percentile 380 ms

What happened#

The report's sequence has eight events:

EventTime, UTCSourceWhat it showsLevel
E115:41:00.912Flight logVibration above 24 m/s² for 1 s in the obstacle beltObserved
E215:41:02.604JournalKernel: USB device 1-2.3 disconnected (mesh radio)Correlated
E315:41:02.790Journalmavlink-routerd cannot reach range controlObserved
E415:41:03.713Ground receiveHeartbeat due and not received. Nothing for 31.0 sObserved
E515:41:05.286Flight logNo ground station heartbeat on TELEM2Observed
E615:41:14.293Flight logGCS connection loss: switching to HoldObserved
E715:41:32.214JournalKernel: usb0 registered againObserved
E815:42:10.511Flight logMission resumed from range controlObserved

The vibration came first. accel_vibration_metric in vehicle_imu_status read 24.31 m/s² at 15:41:00.912, held above the threshold to 26.84 m/s² a second later and peaked at 38.6 m/s² at 15:41:02.312.

The kernel dropped the radio 1.69 s after the vibration onset and 0.29 s after its peak, 227.6 m from the lane's start line. It logged usb 1-2.3: USB disconnect, device number 5, and unregistered the usb0 network interface 7 ms later.

The report files that disconnect as correlated with the vibration: 1.69 s apart against a bound of ±46 ms, so the order is supported.

mavlink-routerd logged Error sending udp packet (Network is unreachable) at 15:41:02.790. Range control's last receipt from system ID 22 had arrived at 15:41:02.713, and the next came 31.0 s later.

Why PX4 held 11.88 s after the last heartbeat#

COM_DL_LOSS_T is 10 s on UGV-02, yet the Hold came 11.88 s after the last ground station heartbeat reached PX4. PX4's timer does not start at that heartbeat. It starts at the last telemetry_status report that still carried it.

PX4's MAVLink receiver keeps heartbeat_type_gcs true for up to 2.5 s after a ground heartbeat (HEARTBEAT_TIMEOUT_US). It re-checks the flag on every heartbeat it receives, and the companion computer's own heartbeats kept arriving over the UART once a second.

INC-0143, from the flight logExample datasetText
Last ground heartbeat PX4 handled       15:41:02.412
Last telemetry_status still true        15:41:04.286   1.874 s later, inside the 2.5 s timeout
First telemetry_status reading false    15:41:05.286   2.874 s after the heartbeat
COM_DL_LOSS_T, 10 s from 15:41:04.286   15:41:14.286
Hold, from NAV_DLL_ACT 1                15:41:14.293   11.881 s after the heartbeat
Example data0 s2 s4 s6 s8 s10 s12 s10 stimeout 2.5 s+0.000 sLast GCS HEARTBEAT handled on TELEM2+1.874 sLast telemetry_status with heartbeat_type_gcs true+2.874 sFirst report with it false, 2.874 s after the heartbeat+11.874 sCOM_DL_LOSS_T (10 s) runs out, counted from row 2+11.881 sgcs_connection_lost; NAV_DLL_ACT 1 selects Hold0 s4 s8 s12 s10 s+0.000 sLast GCS heartbeat handled+1.874 sLast report with heartbeat_type_gcs true+2.874 sFirst report with it false+11.874 sCOM_DL_LOSS_T (10 s) runs out+11.881 sgcs_connection_lost: Hold
How PX4 reached Hold in INC-0143. Offsets run from the last ground station heartbeat PX4 handled, at 15:41:02.412. COM_DL_LOSS_T counts its 10 s from the last telemetry_status that still reported that heartbeat, 1.874 s later.

failsafe_flags gives the cause, gcs_connection_lost, and NAV_DLL_ACT = 1 selects Hold. The report files the Hold as observed, "As configured". The UGV braked to a stop 1.0 s later, 242.9 m from the start line, and held for 56.2 s.

Range control resumed the mission at 15:42:10.511 with DO_SET_MODE from system 255. failsafe_entered walks through PX4's timing in full.

The radio that keeps no log#

The mesh radio keeps no log Foxborne can read. The kernel on the companion computer saw it leave the USB bus at 15:41:02.604 and return at 15:41:31.802, and usb0 registered again at 15:41:32.214.

For those 29.2 s, the radio's own state is not observed. Whether it rebooted, lost power or only lost its USB link, nothing in the three sources says. The vehicle drawing labels the radio Keeps no log, and State not observed while its link to the companion is down.

The kernel records that the device disconnected, not why. A loose connector, a cable fault and a radio reset look the same in this log.

The finding#

The investigator wrote the finding as a Hypothesis:

The mesh radio dropped off the companion computer's USB bus 1.69 s after vibration crossed 24 m/s² in the obstacle belt, and 0.29 s after its 38.6 m/s² peak. Range control stopped hearing UGV-02, and PX4 held the vehicle 11.9 s after the last ground heartbeat reached it: COM_DL_LOSS_T counts its 10 s from the last telemetry_status that still reported that heartbeat, and NAV_DLL_ACT = 1 selects Hold. That vibration unseated the radio's USB connector is a hypothesis until the connector is inspected.

The next check is physical: inspect the USB-C connector on the UGV-02 radio for wear. Then run Lane 2 again with the connector strain-relieved, recording dmesg -w and accel_vibration_metric at 10 Hz. If the disconnect repeats with the connector secured, test the radio on the shaker table.

What the evidence shows and does not#

The evidence showsIt does not show
Vibration crossed 24 m/s² 1.69 s before the kernel dropped the radioThat the vibration caused the disconnect
PX4 held exactly as configured: 10 s after its last report of a ground heartbeatWhat the radio did during the 29.2 s it was off the bus
Range control heard nothing from system ID 22 for 31.0 sWhen the last heartbeat left the vehicle to better than 380 ms, the run's 95th percentile latency

The reviewer's open question reaches across the fleet: did UGV-01 and UGV-03 log USB resets in the belt on their lane rehearsals?

Terrain can cause the same gap#

A ground robot's radio path runs low, so terrain can mask it too. In the example dataset's INC-0137, UGV-05 loses the link while it patrols the north side of a berm. Read the map shows how Foxborne checks the line from the range control mast against the site's elevation data.