Service exit

What it reads#

The rule reads the companion journal, whether the collector uploaded it or you imported a journalctl JSON export. It looks only at what systemd writes about the units on its allowlist. systemd writes those entries itself as PID 1, and each one names the unit in its UNIT field.

{
  "__REALTIME_TIMESTAMP": "1790260324133000",
  "__MONOTONIC_TIMESTAMP": "1152919000",
  "_BOOT_ID": "9b2e4c1d0f8a4e57b3c2a1d9e8f7c6b5",
  "_HOSTNAME": "uas04-orin",
  "PRIORITY": "5",
  "SYSLOG_IDENTIFIER": "systemd",
  "_PID": "1",
  "UNIT": "perception.service",
  "MESSAGE": "perception.service: Main process exited, code=killed, status=9/KILL"
}

The example dataset holds three shapes of line. Each one ends a unit's active state in a different way.

Reading the exit line#

systemd reports how the main process ended in two fields, code and status. They read differently for each ending:

• code=exited. The process ended on its own, and status is its exit status. In INC-0130 the unit exited with status 1 each time, after a malformed calibration file was deployed.
• code=killed, status=9/KILL. Signal 9 (SIGKILL) ended the process, and a process cannot catch it. In INC-0142 the kernel's out-of-memory killer sent it, and its kill line comes 15 ms before this one.
• code=dumped, status=6/ABRT. Signal 6 (SIGABRT) ended the process, which dumped core on the way out. In INC-0139, perception_node logged a CUDA illegal memory access 57 ms earlier on the same clock.
• Stopped MAVLink router. No exit code, because systemd stopped the unit on purpose. It names the unit by its Description=, and a router that crashed would log Main process exited instead.

When it fires#

A finding is produced as soon as systemd writes that an allowlisted unit left the active state. There is no count and no duration to meet: one line is enough. The event sits on that line, and the finding cites the file, the line number and the file's SHA-256.

The Findings row labels this one perception.service killed. The router's stop 19 ms later produced a second finding, labeled Router stopped.

Wording and evidence level#

Every finding reads Observed: process exited, because the journal line shows directly that the unit left the active state. The wording is the same for a crash, a kill and a clean stop. Read the line itself before you write the report's finding.

What the finding does not mean#

• The unit did not necessarily crash. The rule fires whenever a unit leaves the active state, whatever the reason. In INC-0142 systemd stopped mavlink-router.service 19 ms after perception.service exited, and mavlink-routerd logged nothing itself until 14:32:10.702.
• The line says how the process ended, not why. status=9/KILL names the signal, and the kernel's kill line names the sender. Why perception_node grew until the kernel killed it is a separate question, and the example report keeps it open.
• A stop can be a consequence. A dependency such as BindsTo= makes systemd stop one unit when the unit it is bound to fails. The INC-0142 report treats that binding as a hypothesis until the unit files are checked.
• An exit on an unaligned clock has no order against the flight log. In INC-0139 the companion clock had no anchor. The crash cannot be placed before or after the obstacle data stall, and the report's finding says so.

Threshold and overrides#

The threshold is the allowlist itself. In the example dataset it reads 6 units on the allowlist, applied to All vehicles. Which units belong on it is your configuration, and the rule ignores every unit that is not listed.

For an example of a units list, the example collector policy's Journal units field reads perception.service, mavlink-router.service, chronyd.service, collector.service, kernel. The collector reads only those units and ignores the rest of the journal on the vehicle. A unit it does not read never reaches this rule through its uploads.

Change the threshold under Rules with Edit. The rule saves as process_exit 1.3, and findings already produced keep version 1.2 and its threshold. See Thresholds, scopes and overrides.

Example finding#

In INC-0142, UAS-04 flew the return leg of Route Iron. At 14:32:04.118 the kernel killed perception_node, pid 2213, for running out of memory. systemd logged the main process exit at 14:32:04.133 and the unit's failure, with result oom-kill, at 14:32:04.134.

At 14:32:04.152 systemd stopped mavlink-router.service: 34 ms after the kill line and 19 ms after the exit. Both systemd lines produced findings. Range control's next heartbeat was due 0.76 s after the router stopped, and the receive gap lasted 6.96 s.

In INC-0139 the rule caught a crash on a companion clock that was never synchronized. The exit line sits on the companion's own axis, 57 ms after the CUDA error on the same clock. The Findings row flags findings on aligned clocks only, so this one appears in the companion lanes and in Events in view.

In INC-0130, a bench soak on UAS-04, perception.service restarted four times in 50 s after a malformed calibration file was deployed. It exited with status 1 each time, and the file was reverted.

Related#

• service_restart reports systemd bringing a unit back.
• memory_pressure caught the memory climb before the INC-0142 kill.
• telemetry_gap shows what range control heard after the router stopped.
• journalctl JSON exports lists the journal fields Foxborne reads.
• Set the collector policy chooses the units the collector uploads.