Multirotor drones

On a multirotor, the flight controller rarely works alone. A companion computer runs perception and routes MAVLink to the radio, and range control listens on the ground. When telemetry stops, the review board asks: did the radio fail, or did the companion computer stop talking?

Example incident INC-0142 answers that question from four sources. Every value on this page comes from the console's example dataset.

The example platform#

The example dataset lists the Q4 recon quad for route reconnaissance, at 6.8 kg and 12 m/s. Its parts, and the log that records each one:

The flight controller talks to the companion over a UART at 921600 baud, and the companion reaches the radio over Ethernet. Telemetry to the ground passes through mavlink-routerd on the companion, which is where INC-0142 happens. Vehicle parts and their logs maps each part in detail.

The sources#

INC-0142 happened on UAS-04, run R-0931, on the return leg of Route Iron on 24 Sep 2026. The run aligned four files with a widest bound of ±46 ms.

What happened#

The report's sequence has eight events. The level column is the one the report shows for each line.

Memory led the way. The collector's samples read 6,188, 6,402 and 6,655 MiB, and the rule confirmed the climb at 14:32:03.204. The kernel then killed process 2213, perception_node, with 6,823,516 kB of anonymous memory resident.

mavlink-router.service stopped 34 ms after the kill, and 19 ms after systemd recorded the exit of perception.service. Range control last heard system ID 4 at 14:32:03.912 and heard nothing more until 14:32:10.874.

The router came back with the service. systemd started it at 14:32:09.203, and mavlink-routerd opened the flight controller UART at 14:32:10.702 and the mesh link at 14:32:10.718. Range control heard the vehicle again 156 ms later.

The flight log saw the same gap from the other side. At 14:32:06.510, telemetry_status on TELEM2 reported no ground station heartbeat for 2.5 s, and PX4 logged GCS connection regained at 14:32:12.944.

The finding#

The investigator wrote the finding as a Hypothesis:

Telemetry stopped reaching range control within one second of the kernel killing perception_node for running out of memory. mavlink-router.service stopped 34 ms after the kill, 19 ms after systemd recorded the exit of perception.service, and came back with it, which suggests the router unit is bound to the perception unit. The binding is a hypothesis until the unit files are checked.

The report names the check that settles it. On the bench unit, run systemctl show mavlink-router.service -p BindsTo -p PartOf -p Requires. If the binding is there, kill perception_node with SIGKILL, confirm the gap reproduces and record /proc/pressure/memory at 10 Hz.

Where the clocks decide the order#

Two pairs in this sequence show how bounds work between sources. Alignment and event order covers the method.

The stall and the kill sit inside their combined bound, so Foxborne claims no order between them. The heartbeat was due well after the router stopped, even allowing for the receiver's latency.

What the evidence shows#

• perception_node held more than 6,144 MiB for three samples, and the kernel killed it for running out of memory.
• The router stopped within 34 ms of the kill. The next heartbeat was due at range control 0.76 s later, and none arrived until the router reopened the mesh link.
• PX4 recorded no data link failsafe. The 6.96 s gap is shorter than COM_DL_LOSS_T (10 s), so none was expected.

What it does not show#

• Which came first, the stall or the kill. They are 39 ms apart, inside their 46 ms combined bound.
• Why the router stopped. A unit binding would explain it, but the unit files have not been checked.
• What the router did during the gap. mavlink-routerd wrote nothing between 14:32:04.152 and 14:32:10.702.
• When the last heartbeat left the vehicle. With 412 ms of latency, it left between 14:32:03.485 and 14:32:03.927.
• What the radio did. It keeps no log, so its state is never observed.

Memory on a computer that shares it with the GPU#

The Q4 carries the Orin NX 16 GB, where GPU buffers share memory with processes. At the last sample before the kill, nvmap held 8,212 MiB. That is why the memory_pressure rule holds Q4 vehicles to 6,144 MiB for 3 samples, while T4 vehicles use 24,576 MiB.

The journal also held an early warning. At 14:13:53.118, 18 minutes before the kill, perception_node logged map tile cache: cache_max_mb not set; eviction disabled. The report keeps "Why did the tile cache grow without a limit on this build?" as an open question.

What to send#

All four sources were present in INC-0142, and each one carried events the others could not.

Without the journal, the kill and the router stop would not be on the timeline at all. Platforms gives the full list for PX4 vehicles.

Related#

• Alignment and event order explains the order-unknown pair in this incident.
• memory_pressure and telemetry_gap describe the two rules behind this incident's alerts.
• process_exit, topic_stall and service_restart produced the other findings.
• Ground robots shows a telemetry gap with a different cause.