# Multirotor drones

URL: /platforms/drones

How Foxborne rebuilds a multirotor incident from the flight log, the companion journal, process samples and the ground receive log, worked through example incident INC-0142, where telemetry stopped within a second of a companion service being killed.



On a multirotor, the flight controller rarely works alone. A companion computer runs perception and routes MAVLink to the radio, and range control listens on the ground. When telemetry stops, the review board asks: did the radio fail, or did the companion computer stop talking?

Example incident INC-0142 answers that question from four sources. Every value on this page comes from the console's example dataset.

## The example platform [#the-example-platform]

The example dataset lists the Q4 recon quad for route reconnaissance, at 6.8 kg and 12 m/s. Its parts, and the log that records each one:

| Part               | In the example dataset                                                                             | Recorded by                                     |
| ------------------ | -------------------------------------------------------------------------------------------------- | ----------------------------------------------- |
| Flight controller  | Pixhawk 6C, PX4 v1.15.4                                                                            | The flight log                                  |
| Companion computer | Jetson Orin NX 16 GB, Ubuntu 22.04, running `perception_node`, `mavlink-routerd` and the collector | The journal and the collector's process samples |
| Mesh radio         | MANET, Ethernet                                                                                    | Nothing. It keeps no log Foxborne can read.     |
| GNSS receiver      | u-blox M10                                                                                         | The flight log                                  |
| Depth camera       | Stereo depth, 10 Hz                                                                                | The flight log                                  |
| Battery            | 6S Li-ion, 22 Ah                                                                                   | The flight log                                  |

The flight controller talks to the companion over a UART at 921600 baud, and the companion reaches the radio over Ethernet. Telemetry to the ground passes through `mavlink-routerd` on the companion, which is where INC-0142 happens. [Vehicle parts and their logs](/platforms/vehicle-parts) maps each part in detail.

## The sources [#the-sources]

INC-0142 happened on UAS-04, run R-0931, on the return leg of Route Iron on 24 Sep 2026. The run aligned four files with a widest bound of ±46 ms.

| Source             | File                       | Clock and anchor                                                       | Bound                                                     |
| ------------------ | -------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------- |
| Flight log         | `14_13_19.ulg`             | PX4 boot clock, anchored by GPS time from `sensor_gps`, 1,081 fixes    | ±40 ms                                                    |
| Companion journal  | `uas04-orin-journal.json`  | Companion wall clock, chrony synchronized, offset −1.8 ms              | ±6 ms                                                     |
| Process samples    | `uas04-orin-samples.jsonl` | Companion monotonic clock, sharing boot ID `9b2e4c1d` with the journal | ±6 ms                                                     |
| Ground receive log | `rc1-receive.jsonl`        | Range control receiver clock, NTP, offset +0.4 ms                      | ±15 ms plus latency: median 96 ms, 95th percentile 412 ms |

## What happened [#what-happened]

The report's sequence has eight events. The level column is the one the report shows for each line.

| Event | Time, UTC    | Source          | What it shows                                           | Level                           |
| ----- | ------------ | --------------- | ------------------------------------------------------- | ------------------------------- |
| E1    | 14:32:01.204 | Process samples | `perception_node` memory above 6,144 MiB for 3 samples  | <Evidence level="observed" />   |
| E2    | 14:32:04.079 | Flight log      | Last `obstacle_distance` sample before a 7.5 s stall    | Order against E3 unknown        |
| E3    | 14:32:04.118 | Journal         | Kernel killed `perception_node` (out of memory)         | Order against E2 unknown        |
| E4    | 14:32:04.133 | Journal         | `perception.service` main process exited, status 9/KILL | <Evidence level="observed" />   |
| E5    | 14:32:04.152 | Journal         | `mavlink-router.service` stopped                        | <Evidence level="observed" />   |
| E6    | 14:32:04.912 | Ground receive  | Heartbeat due and not received. Nothing for 6.96 s      | <Evidence level="correlated" /> |
| E7    | 14:32:09.214 | Journal         | `perception.service` started                            | <Evidence level="observed" />   |
| E8    | 14:32:10.874 | Ground receive  | Telemetry received again                                | <Evidence level="observed" />   |

Memory led the way. The collector's samples read 6,188, 6,402 and 6,655 MiB, and the rule confirmed the climb at 14:32:03.204. The kernel then killed process 2213, `perception_node`, with 6,823,516 kB of anonymous memory resident.

`mavlink-router.service` stopped 34 ms after the kill, and 19 ms after systemd recorded the exit of `perception.service`. Range control last heard system ID 4 at 14:32:03.912 and heard nothing more until 14:32:10.874.

The router came back with the service. systemd started it at 14:32:09.203, and `mavlink-routerd` opened the flight controller UART at 14:32:10.702 and the mesh link at 14:32:10.718. Range control heard the vehicle again 156 ms later.

The flight log saw the same gap from the other side. At 14:32:06.510, `telemetry_status` on TELEM2 reported no ground station heartbeat for 2.5 s, and PX4 logged **GCS connection regained** at 14:32:12.944.

## The finding [#the-finding]

The investigator wrote the finding as a <Evidence level="hypothesis" />:

> Telemetry stopped reaching range control within one second of the kernel killing perception\_node for running out of memory. mavlink-router.service stopped 34 ms after the kill, 19 ms after systemd recorded the exit of perception.service, and came back with it, which suggests the router unit is bound to the perception unit. The binding is a hypothesis until the unit files are checked.

The report names the check that settles it. On the bench unit, run `systemctl show mavlink-router.service -p BindsTo -p PartOf -p Requires`. If the binding is there, kill `perception_node` with `SIGKILL`, confirm the gap reproduces and record `/proc/pressure/memory` at 10 Hz.

## Where the clocks decide the order [#where-the-clocks-decide-the-order]

Two pairs in this sequence show how bounds work between sources. [Alignment and event order](/concepts/alignment) covers the method.

| Pair                                                          | Apart  | Combined bound | Result                       |
| ------------------------------------------------------------- | ------ | -------------- | ---------------------------- |
| E2, flight log ±40 ms, and E3, journal ±6 ms                  | 39 ms  | ±46 ms         | Order unknown                |
| E5, journal ±6 ms, and E6, receipt ±15 ms plus 412 ms latency | 0.76 s | ±0.43 s        | Ordered, filed as correlated |

The stall and the kill sit inside their combined bound, so Foxborne claims no order between them. The heartbeat was due well after the router stopped, even allowing for the receiver's latency.

## What the evidence shows [#what-the-evidence-shows]

* `perception_node` held more than 6,144 MiB for three samples, and the kernel killed it for running out of memory.
* The router stopped within 34 ms of the kill. The next heartbeat was due at range control 0.76 s later, and none arrived until the router reopened the mesh link.
* PX4 recorded no data link failsafe. The 6.96 s gap is shorter than `COM_DL_LOSS_T` (10 s), so none was expected.

## What it does not show [#what-it-does-not-show]

* **Which came first, the stall or the kill.** They are 39 ms apart, inside their 46 ms combined bound.
* **Why the router stopped.** A unit binding would explain it, but the unit files have not been checked.
* **What the router did during the gap.** `mavlink-routerd` wrote nothing between 14:32:04.152 and 14:32:10.702.
* **When the last heartbeat left the vehicle.** With 412 ms of latency, it left between 14:32:03.485 and 14:32:03.927.
* **What the radio did.** It keeps no log, so its state is never observed.

## Memory on a computer that shares it with the GPU [#memory-on-a-computer-that-shares-it-with-the-gpu]

The Q4 carries the Orin NX 16 GB, where GPU buffers share memory with processes. At the last sample before the kill, `nvmap` held 8,212 MiB. That is why the [memory\_pressure](/rules/memory-pressure) rule holds Q4 vehicles to 6,144 MiB for 3 samples, while T4 vehicles use 24,576 MiB.

The journal also held an early warning. At 14:13:53.118, 18 minutes before the kill, `perception_node` logged `map tile cache: cache_max_mb not set; eviction disabled`. The report keeps "Why did the tile cache grow without a limit on this build?" as an open question.

## What to send [#what-to-send]

All four sources were present in INC-0142, and each one carried events the others could not.

| Source                                  | Need        | In INC-0142                                                                                           |
| --------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------- |
| PX4 flight log, `.ulg`                  | Required    | `14_13_19.ulg`, from the fleet log archive: the stall, the vehicle's own link view and the GPS anchor |
| Companion journal, `journalctl -o json` | Recommended | `uas04-orin-journal.json`, 18,442 records: the kill, the exit and the router stop                     |
| Ground receive log, `.jsonl`            | Recommended | `rc1-receive.jsonl`, 1,359 records: the telemetry gap                                                 |
| Collector samples, `.jsonl`             | Optional    | `uas04-orin-samples.jsonl`, 2,840 records: the memory climb                                           |

Without the journal, the kill and the router stop would not be on the timeline at all. [Platforms](/platforms#what-we-need-from-your-vehicle) gives the full list for PX4 vehicles.

## Related [#related]

* [Alignment and event order](/concepts/alignment) explains the order-unknown pair in this incident.
* [memory\_pressure](/rules/memory-pressure) and [telemetry\_gap](/rules/telemetry-gap) describe the two rules behind this incident's alerts.
* [process\_exit](/rules/process-exit), [topic\_stall](/rules/topic-stall) and [service\_restart](/rules/service-restart) produced the other findings.
* [Ground robots](/platforms/ground-robots) shows a telemetry gap with a different cause.