Inspect an event

Every event in Foxborne points to one record in one original file, at a stated line, row or byte offset. The evidence inspector shows that record in a drawer over the timeline. The event page goes further. It adds the detail the source carries and what every other source logged in the 3 minutes before. It also counts the same signature across the fleet.

Before you start#

• All five roles can inspect original evidence. See Roles and permissions.
• Views are audited. Each time the inspector or the event page shows an event, the audit log records Viewed evidence with the event's title and your name.
• The examples use the kernel kill in INC-0142 from the example dataset, event 5a705b4a.

Read the evidence inspector#

1. 01

   Select the event

   Select the event on the timeline, in Events in view, on the map or on the vehicle diagram. The inspector slides in from the right edge of the screen, over the page, under the event's title. The timeline, the map and the event table keep their full width underneath. Select another event, or press ← or →, to switch the inspector in place, and close it with × or Esc. Chips give the rule and its version, and Own clock, Manual alignment or Dismissed when they apply.

2. 02

   Read when it happened

   When gives the time and bound: 14:32:04.118 ±6 ms for the kill. A receive record adds its latency, as in + 412 ms latency. Elapsed counts from log start (T+18:44.716), Source clock names the clock (Companion wall clock, chrony offset −1.8 ms) and Ingested says when the record arrived. A record that arrived late adds, in amber, how long after the event that was.

   On an unaligned companion clock, the time reads companion clock in place of a bound, with no Elapsed row.

3. 03

   Read where it happened

   Position places the event on the track, when the run has one.

   Row	Kernel kill in INC-0142
   MGRS	13R FP 7639 8764
   Latitude, longitude	The same point in degrees, to five decimals
   Ground elevation	1,148 m
   Speed	5.50 m/s, from the track over the previous 200 ms. Below 0.05 m/s it reads Stationary
   To range control	1.20 km
   From the time bound	How far the vehicle moves within the bound and latency, from 0.05 m up. The last obstacle_distance sample reads ±0.22 m along the track

   An event on an unaligned companion clock reads No position. This event is on the companion clock, which is not aligned, so it cannot be placed on the track.

4. 04

   Check the source

   Source names the original: File uas04-orin-journal.json, Location Line 18,314, cursor i=5014, SHA-256 with a copy button, Parser journal 0.6.0 and Collected via Companion collectors. Locations take the form of the source. A journal event gives its line in the export, process samples give rows such as Rows 2,811 to 2,813 and receipts give lines. A flight log event gives a message and byte offset, such as Message 1,146,099, byte 53,503,302.

   Original record shows the record with a Copy button. The fields that matter most, such as MESSAGE, recv_utc, rss_kib and nav_state, are highlighted.

   {
     "__REALTIME_TIMESTAMP": "1790260324118000",
     "__MONOTONIC_TIMESTAMP": "1152904000",
     "_BOOT_ID": "9b2e4c1d0f8a4e57b3c2a1d9e8f7c6b5",
     "_HOSTNAME": "uas04-orin",
     "PRIORITY": "3",
     "SYSLOG_IDENTIFIER": "kernel",
     "_TRANSPORT": "kernel",
     "MESSAGE": "Out of memory: Killed process 2213 (perception_node) total-vm:11873248kB, anon-rss:6823516kB, file-rss:10240kB, shmem-rss:0kB, UID:1001 pgtables:14720kB oom_score_adj:0"
   }

   Values lists what the rule or parser extracted, when there is anything. The memory finding in INC-0142 gives Samples 6,188, 6,402 and 6,655 MiB and Confirmed at 14:32:03.204.

5. 05

   Check the order against nearby events

   Order against nearby events compares the event with up to 7 others within 6 s, nearest first. For the kill:

   Event	Verdict	Lane and basis
   systemd: a process of perception.service was killed by the OOM killer	3 ms later	Companion journal. Same clock.
   mavlink-router.service stopped	34 ms later	Companion journal. Same clock.
   Last obstacle_distance sample before a 7.5 s stall	Order unknown	Flight controller. 39 ms apart, inside ±46 ms combined.

   Between anchored clocks a verdict reads, for example, 73 ms earlier with ±46 ms combined. After a manual alignment, the basis adds , manual alignment. Events that cannot be compared share one No claim row: "The companion clock has no shared anchor with the flight log." Select a row to move to that event.

6. 06

   Read the rule

   For a finding, Rule lists Rule, Detects, Threshold and Wording. The exit of perception.service reads process_exit 1.2, "An allowlisted systemd unit leaves the active state.", "6 units on the allowlist" and "Observed: process exited". Dismiss this finding sits below: see Work an incident.

7. 07

   Copy a reference

   The footer holds Open event, Add to report and Copy reference. Copy reference copies the incident and event IDs, the file, its full SHA-256 and the location, then confirms with Evidence reference to the clipboard.

   INC-0142/5a705b4a uas04-orin-journal.json sha256:a0741c955f3301a6dffc1a85c93e0eb315e53227c790f517ff7d484e98e0cdf4 Line 18,314, cursor i=5014

Open the event page#

Open event, or the expand icon Open the event page, opens /app/incidents/INC-0142/events/5a705b4a, under the breadcrumbs Incidents, INC-0142 and Event 5a705b4a. Search reaches it too: its Evidence group matches the start of an event ID or, for a query longer than 3 characters, text in the title.

Header and actions#

The header gives the incident and file, the title and a message line:

• a journal event shows its MESSAGE;
• a flight log record shows its topic and first three fields;
• a receipt shows the message, system ID, sequence number and receive time;
• a process sample shows the process, its PID, RSS and CPU.

The chips give the level, the lane, the rule and the bound, or Own clock in its place. A last chip, such as In the report, step 3, says where the event sits in the report. On the right, Across the fleet, 30 days and Vehicles count this signature.

The level follows the record. Journal priority 2 or more severe reads Critical, 3 Error and 4 Warning. A failsafe reads Critical, while a finding, a link change, a stall or a vibration event reads Warning. Anything else is Info, and the kill, at priority 3, reads Error.

The action bar steps through the incident's events, Event 6 of 26 in INC-0142, also with ← and →. Show on the timeline opens the Reconstruction tab with the event selected. Discussion, with the number of notes beside it, opens the drawer described below. Copy as offers Original record, Markdown summary and Evidence reference:

**Kernel killed perception_node (out of memory)**

- Time: 14:32:04.118 UTC, ±6 ms
- Vehicle: UAS-04, run R-0931
- Source: Companion journal, uas04-orin-journal.json sha256:a0741c955f3301a6dffc1a85c93e0eb315e53227c790f517ff7d484e98e0cdf4 Line 18,314, cursor i=5014
- Incident: INC-0142

```
Out of memory: Killed process 2213 (perception_node) total-vm:11873248kB, anon-rss:6823516kB, file-rss:10240kB, shmem-rss:0kB, UID:1001 pgtables:14720kB oom_score_adj:0
```

Across the fleet#

The panel counts the same signature over the last 30 days: Occurrences, Vehicles and Runs, with a bar per day and this event's day highlighted. Rows for Vehicle, Where, Software and Autopilot give the most common value's share. For the kill, the counts are 2, 1 and 2: this run and R-0926, both on UAS-04.

Highlights and detail#

Highlights repeats the essentials: Time, Bound, Source, File and Location, then Vehicle, Run, Where, Rule and Finding. Rule reads as ID, version and threshold, such as memory_pressure 1.1: RSS at or above 6,144 MiB for 3 samples. One detail section follows when the source carries more:

Breadcrumbs, record, tags and context#

Breadcrumbs lists what every source logged in the 3 minutes before the event, newest first, with This event first, 12 to a page. Filter it with Filter breadcrumbs or by source: All, Flight log, Journal, Processes and Receive. Each crumb gives its time relative to the event and an expandable Record. Unbroken heartbeats fold into one crumb, and so do each process's samples.

Record shows the original as a tree with Copy. Tags lists key and value pairs, grouped Vehicle, Process, Source and Place. Context holds cards for the Vehicle, Flight controller, Companion computer, Radio, Clock and Run.

In the report, seen before and discussion#

Discussion, in the action bar, opens a drawer with three sections. In the report says where the event sits in the report, such as Step 3 of 8 in the sequence. An event outside it reads Not in the sequence yet. Seen before lists the other occurrences, each with its vehicle, run, incident and time. For the kill, the one other occurrence is on R-0926 and carries No incident. Discussion holds notes on this evidence: write in Add a note on this evidence and select Post, which records Commented in the audit log and shows the note in the drawer at once.

Check the result#

• The SHA-256 matches the one in the run's Source files table for the same file.
• Copy reference gives a line you can paste into a ticket or a message.
• The audit log shows Viewed evidence with your name and the event's title.

Troubleshoot#

This event does not exist. The console adds "It may belong to a different incident." Back to INC-0142 returns to the incident.

The event shows Own clock. It sits on a companion clock that never synchronized: it has no position, and its breadcrumbs list only records on the same clock. Align the clock by hand if a matching pair exists.

Seen before reads No other occurrence in the last 30 days. The signature has not appeared anywhere else in that window.

Post does nothing. The note is empty. Write it, then select Post.

Next#