# Service restart

URL: /rules/service-restart

Flags each automatic restart that systemd schedules for an allowlisted unit on the companion computer. It records that systemd brought the unit back, not that the unit works again.



<RuleHeader id="service_restart" />

## What it reads [#what-it-reads]

The rule reads the companion journal, from the collector or from a `journalctl` JSON export. It looks for one line, which systemd writes as PID 1 when it schedules an automatic restart of an allowlisted unit:

```text title="uas04-orin-journal.json" caption="Example dataset, INC-0142, the MESSAGE field of line 18,356"
perception.service: Scheduled restart job, restart counter is at 1.
```

The counter is systemd's count of automatic restarts for the unit, so 1 marks the first. Whether systemd restarts a unit at all, and how long it waits first, comes from the unit's `Restart=` and `RestartSec=` settings.

## When it fires [#when-it-fires]

The threshold reads **Any restart**: each scheduled restart of an allowlisted unit produces a finding. In INC-0142, systemd wrote these lines about the unit and its neighbor, in this order:

```text title="uas04-orin-journal.json" caption="Example dataset, INC-0142, systemd lines, times in UTC"
14:32:04.133  perception.service: Main process exited, code=killed, status=9/KILL
14:32:04.134  perception.service: Failed with result 'oom-kill'.
14:32:09.141  perception.service: Scheduled restart job, restart counter is at 1.
14:32:09.203  Started MAVLink router.
14:32:09.214  Started Perception service.
```

The restart job came 5.007 s after the unit failed, at 14:32:09.141 against 14:32:04.134. `RestartSec=` defaults to 100 ms, so this unit's own settings ask for a longer wait.

## Wording and evidence level [#wording-and-evidence-level]

Every finding reads **Observed: service restarted**. The journal line shows directly that systemd scheduled the restart, so the level is observed.

## What the finding does not mean [#what-the-finding-does-not-mean]

* **The unit did not necessarily come back healthy.** "Started" means systemd started the process, not that the process works. In INC-0142, `perception_node` logged that it was streaming obstacle data again at 14:32:11.596, 2.382 s after systemd reported the unit started.
* **It does not say why the unit stopped.** Read the exit line before it, which [process\_exit](/rules/process-exit) reports. In INC-0142 the kernel killed the main process for memory, and in INC-0130 the process exited with status 1.
* **One finding is not a loop.** The counter shows which automatic restart this is. A restart loop shows as a rising counter and repeated exits close together, as in INC-0130: four restarts in 50 s.
* **The restart follows configuration.** A restart is systemd applying the unit's settings, and the settings it loaded can differ from the file you expect. Ask the running system with `systemctl show perception.service -p OOMPolicy -p Restart -p RestartUSec`.

## Threshold and overrides [#threshold-and-overrides]

The threshold is **Any restart**, applied to **All vehicles** and to allowlisted units only. Change it under **Rules** with **Edit**. The rule saves as `service_restart 1.1`, and findings already produced keep version 1.0 and its threshold; see [Thresholds, scopes and overrides](/rules/configure).

When a restart is expected, for example during a scheduled bench test, dismiss the finding and give that reason. The dismissal leaves the evidence unchanged, and the audit log keeps it with the reason. See [Dismissing a finding](/rules#dismissing-a-finding).

## Example finding [#example-finding]

In INC-0142 the kernel killed `perception_node` on UAS-04 at 14:32:04.118. systemd scheduled the restart at 14:32:09.141, started `mavlink-router.service` at 14:32:09.203 and reported `perception.service` started at 14:32:09.214. The **Findings** row labels the finding **Service restarted**.

The collector's samples show the new process, pid 3398, at 412 MiB at 14:32:10.204, with a restart count of 1. On the event page, **Unit history** lists what systemd logged for the units involved, from the kill to the restart. It covers systemd lines and the kernel's kill line within 12 s of the event, in the columns **Time, UTC**, **From first**, **Unit** and **Message**.

A started unit is not yet a working link. `mavlink-router.service` started at 14:32:09.203, but `mavlink-routerd` opened the flight controller's UART only at 14:32:10.702 and the mesh link at 14:32:10.718. Range control heard the vehicle again at 14:32:10.874, 1.671 s after the router's unit started.

In INC-0130, a bench soak on UAS-04, `perception.service` exited with status 1 at 09:42:10.003. systemd scheduled a restart with the counter at 1 at 09:42:15.090, 5.087 s later, and the unit exited again at 09:42:21.040. The report records four restarts in 50 s after a malformed calibration file was deployed, and the file was reverted.

## Related [#related]

* [process\_exit](/rules/process-exit) records the exit that comes before a restart.
* [memory\_pressure](/rules/memory-pressure) shows the memory climb behind the INC-0142 kill.
* [Inspect an event](/investigate/event) describes the **Unit history** section of the event page.
* [journalctl JSON exports](/collect/formats/journald) lists the journal fields Foxborne reads.
