Skip to content
Docs
foxborne.comRequest a pilot

Service restart

Flags each automatic restart that systemd schedules for an allowlisted unit on the companion computer. It records that systemd brought the unit back, not that the unit works again.

Rule referenceMarkdown
On this page7
service_restartv1.0On by default
Detects
systemd schedules a restart of an allowlisted unit.
Threshold
Any restart
Reads
Companion journal
Report wording
Observed: service restarted

What it reads#

The rule reads the companion journal, from the collector or from a journalctl JSON export. It looks for one line, which systemd writes as PID 1 when it schedules an automatic restart of an allowlisted unit:

uas04-orin-journal.jsonExample dataset, INC-0142, the MESSAGE field of line 18,356Text
perception.service: Scheduled restart job, restart counter is at 1.

The counter is systemd's count of automatic restarts for the unit, so 1 marks the first. Whether systemd restarts a unit at all, and how long it waits first, comes from the unit's Restart= and RestartSec= settings.

When it fires#

The threshold reads Any restart: each scheduled restart of an allowlisted unit produces a finding. In INC-0142, systemd wrote these lines about the unit and its neighbor, in this order:

uas04-orin-journal.jsonExample dataset, INC-0142, systemd lines, times in UTCText
14:32:04.133  perception.service: Main process exited, code=killed, status=9/KILL
14:32:04.134  perception.service: Failed with result 'oom-kill'.
14:32:09.141  perception.service: Scheduled restart job, restart counter is at 1.
14:32:09.203  Started MAVLink router.
14:32:09.214  Started Perception service.

The restart job came 5.007 s after the unit failed, at 14:32:09.141 against 14:32:04.134. RestartSec= defaults to 100 ms, so this unit's own settings ask for a longer wait.

Wording and evidence level#

Every finding reads Observed: service restarted. The journal line shows directly that systemd scheduled the restart, so the level is observed.

What the finding does not mean#

  • The unit did not necessarily come back healthy. "Started" means systemd started the process, not that the process works. In INC-0142, perception_node logged that it was streaming obstacle data again at 14:32:11.596, 2.382 s after systemd reported the unit started.
  • It does not say why the unit stopped. Read the exit line before it, which process_exit reports. In INC-0142 the kernel killed the main process for memory, and in INC-0130 the process exited with status 1.
  • One finding is not a loop. The counter shows which automatic restart this is. A restart loop shows as a rising counter and repeated exits close together, as in INC-0130: four restarts in 50 s.
  • The restart follows configuration. A restart is systemd applying the unit's settings, and the settings it loaded can differ from the file you expect. Ask the running system with systemctl show perception.service -p OOMPolicy -p Restart -p RestartUSec.

Threshold and overrides#

The threshold is Any restart, applied to All vehicles and to allowlisted units only. Change it under Rules with Edit. The rule saves as service_restart 1.1, and findings already produced keep version 1.0 and its threshold; see Thresholds, scopes and overrides.

When a restart is expected, for example during a scheduled bench test, dismiss the finding and give that reason. The dismissal leaves the evidence unchanged, and the audit log keeps it with the reason. See Dismissing a finding.

Example finding#

In INC-0142 the kernel killed perception_node on UAS-04 at 14:32:04.118. systemd scheduled the restart at 14:32:09.141, started mavlink-router.service at 14:32:09.203 and reported perception.service started at 14:32:09.214. The Findings row labels the finding Service restarted.

The collector's samples show the new process, pid 3398, at 412 MiB at 14:32:10.204, with a restart count of 1. On the event page, Unit history lists what systemd logged for the units involved, from the kill to the restart. It covers systemd lines and the kernel's kill line within 12 s of the event, in the columns Time, UTC, From first, Unit and Message.

A started unit is not yet a working link. mavlink-router.service started at 14:32:09.203, but mavlink-routerd opened the flight controller's UART only at 14:32:10.702 and the mesh link at 14:32:10.718. Range control heard the vehicle again at 14:32:10.874, 1.671 s after the router's unit started.

In INC-0130, a bench soak on UAS-04, perception.service exited with status 1 at 09:42:10.003. systemd scheduled a restart with the counter at 1 at 09:42:15.090, 5.087 s later, and the unit exited again at 09:42:21.040. The report records four restarts in 50 s after a malformed calibration file was deployed, and the file was reverted.