Skip to content
Docs
foxborne.comRequest a pilot

Memory pressure

Flags a watched process whose resident memory stays at or above its threshold for three samples in a row. It observes memory use on the companion computer, not a leak and not what followed.

Rule referenceMarkdown
On this page7
memory_pressurev1.1On by default
Detects
A watched process stays above its memory threshold.
Threshold
RSS at or above 6,144 MiB for 3 samples
Reads
Process samples
Report wording
Observed: memory rose above threshold

What it reads#

The rule reads process samples from the Foxborne collector: one record per watched process per sample, with its resident memory in rss_kib. The collector policy names the processes and the rate. In the example dataset it samples perception_node and mavlink-routerd once per second.

uas04-orin-samples.jsonlExample dataset, INC-0142, the first of rows 2,811 to 2,813, as the evidence inspector shows itJSON
{
  "t_utc": "2026-09-24T14:32:01.204Z",
  "mono_ns": 1149990000000,
  "boot_id": "9b2e4c1d",
  "proc": "perception_node",
  "pid": 2213,
  "rss_kib": 6336512,
  "cpu_pct": 187.4,
  "restarts": 0
}

rss_kib is in KiB, so 6,336,512 KiB is 6,188 MiB. The samples carry the companion's monotonic time and boot ID, so they share the journal's clock and its error bound.

When it fires#

A finding is produced when a watched process holds 6,144 MiB or more for 3 samples in a row. At one sample per second, the three samples span 2 s. The event sits on the first of them, and the third confirms it.

ValueExample from INC-0142
Samples6,188, 6,402 and 6,655 MiB
First sample at or above14:32:01.204
Confirmed at14:32:03.204
Threshold6,144 MiB for 3 samples
LocationRows 2,811 to 2,813 of uas04-orin-samples.jsonl

The finding's title names the process and the line: perception_node memory above 6,144 MiB for 3 samples.

Wording and evidence level#

Every finding reads Observed: memory rose above threshold. The samples show the value directly, so the level is observed.

What the finding does not mean#

  • It does not say the process leaked. In INC-0142 perception_node logged at startup that its tile cache had no size limit. The report still asks, as an open question, why the cache grew without a limit on that build.
  • It does not say the memory caused what followed. The kernel killed perception_node 0.914 s after the confirming sample. The kill is a separate observation in the journal, and linking the two is the engineer's work.
  • A sample is one reading at one moment. The kill line reports anon-rss:6823516kB and file-rss:10240kB, 6,673.6 MiB resident in all. That is 18.6 MiB above the last sample, taken 0.914 s earlier, and the kernel's kB are KiB.
  • Resident memory leaves out GPU buffers. On the example's Jetson modules, the CPU and the GPU share one memory. The event page for the INC-0142 kill lists 8,212 MiB of GPU buffers (nvmap) under Not in these counters.
  • Below the line is silent. For the 44 samples before the crossing, from 14:31:17.204 to 14:32:00.204, perception_node sat between 6,117 and 6,143 MiB. None of them produced a finding, so read the Process samples lane as well as the flags.

Threshold and overrides#

In the example dataset the threshold reads RSS at or above 6,144 MiB for 3 samples, applied to Q4 vehicles. The rule also carries the dataset's one override, T4 vehicles: 24,576 MiB.

Under the threshold, the Threshold column shows both as Q4 vehicles. T4 vehicles: 24,576 MiB.

An override covers vehicles whose hardware differs. In the example, the T4's companion computer has 64 GB of memory against 16 GB on the Q4, both shared by CPU and GPU.

Change either value under Rules with Edit. The rule saves as memory_pressure 1.2, and findings already produced keep version 1.1 and its threshold. The collector's Sample rate also changes what 3 samples span: at 5 per second they cover 0.4 s, not 2 s.

Example finding#

In INC-0142, UAS-04 flew the return leg of Route Iron. perception_node read 6,122 MiB at 14:32:00.204, then 6,188, 6,402 and 6,655 MiB over the next three samples. The finding was confirmed at 14:32:03.204, and the kernel killed the process at 14:32:04.118.

The Process samples lane draws the line as Threshold 6,144 MiB. The Findings row labels the finding Memory above threshold, and it was the incident's first finding.

On the event page, Memory around this event plots 8 s before to 4 s after the first sample. Filled points are at or above the dashed threshold, and a table lists the samples within 2.1 s of the event.

The event page's Across the fleet panel counts 6 occurrences of the same signature on 4 vehicles in the last 30 days. A single finding can be the start of a pattern, and that panel is where the pattern shows.