# Inspect an event

URL: /investigate/event

Trace any event back to one record in one original file. The evidence inspector gives its time, bound, position, source and order in a drawer over the timeline, and the event page adds what its source and every other source recorded around it.



Every event in Foxborne points to one record in one original file, at a stated line, row or byte offset. The evidence inspector shows that record in a drawer over the timeline. The event page goes further. It adds the detail the source carries and what every other source logged in the 3 minutes before. It also counts the same signature across the fleet.

## Before you start [#before-you-start]

* All five roles can inspect original evidence. See [Roles and permissions](/security/roles).
* Views are audited. Each time the inspector or the event page shows an event, the audit log records **Viewed evidence** with the event's title and your name.
* The examples use the kernel kill in INC-0142 from the example dataset, event `5a705b4a`.

## Read the evidence inspector [#read-the-evidence-inspector]

<Steps>
  <Step title="Select the event">
    Select the event on the timeline, in **Events in view**, on the map or on the vehicle diagram. The inspector slides in from the right edge of the screen, over the page, under the event's title. The timeline, the map and the event table keep their full width underneath. Select another event, or press <Kbd>←</Kbd> or <Kbd>→</Kbd>, to switch the inspector in place, and close it with × or <Kbd>Esc</Kbd>. Chips give the rule and its version, and **Own clock**, **Manual alignment** or **Dismissed** when they apply.
  </Step>

  <Step title="Read when it happened">
    **When*&#x2A; gives the time and bound: 14:32:04.118 ±6 ms for the kill. A receive record adds its latency, as in **+ 412 ms latency**. **Elapsed** counts from log start (T+18:44.716), **Source clock** names the clock (Companion wall clock, chrony offset −1.8 ms) and **Ingested** says when the record arrived. A record that arrived late adds, in amber, how long after the event that was.

    On an unaligned companion clock, the time reads **companion clock** in place of a bound, with no **Elapsed** row.
  </Step>

  <Step title="Read where it happened">
    **Position** places the event on the track, when the run has one.

    | Row                     | Kernel kill in INC-0142                                                                                                                            |
    | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
    | **MGRS**                | `13R FP 7639 8764`                                                                                                                                 |
    | **Latitude, longitude** | The same point in degrees, to five decimals                                                                                                        |
    | **Ground elevation**    | 1,148 m                                                                                                                                            |
    | **Speed**               | 5.50 m/s, from the track over the previous 200 ms. Below 0.05 m/s it reads **Stationary**                                                          |
    | **To range control**    | 1.20 km                                                                                                                                            |
    | **From the time bound** | How far the vehicle moves within the bound and latency, from 0.05 m up. The last `obstacle_distance&#x60; sample reads **±0.22 m along the track** |

    An event on an unaligned companion clock reads &#x2A;*No position. This event is on the companion clock, which is not aligned, so it cannot be placed on the track.**
  </Step>

  <Step title="Check the source">
    **Source** names the original: **File** `uas04-orin-journal.json`, **Location** Line 18,314, cursor i=5014, **SHA-256** with a copy button, **Parser** `journal 0.6.0` and **Collected via** Companion collectors. Locations take the form of the source. A journal event gives its line in the export, process samples give rows such as Rows 2,811 to 2,813 and receipts give lines. A flight log event gives a message and byte offset, such as Message 1,146,099, byte 53,503,302.

    **Original record** shows the record with a **Copy** button. The fields that matter most, such as `MESSAGE`, `recv_utc`, `rss_kib` and `nav_state`, are highlighted.

    ```json title="Original record, event 5a705b4a" caption="Example dataset, INC-0142"
    {
      "__REALTIME_TIMESTAMP": "1790260324118000",
      "__MONOTONIC_TIMESTAMP": "1152904000",
      "_BOOT_ID": "9b2e4c1d0f8a4e57b3c2a1d9e8f7c6b5",
      "_HOSTNAME": "uas04-orin",
      "PRIORITY": "3",
      "SYSLOG_IDENTIFIER": "kernel",
      "_TRANSPORT": "kernel",
      "MESSAGE": "Out of memory: Killed process 2213 (perception_node) total-vm:11873248kB, anon-rss:6823516kB, file-rss:10240kB, shmem-rss:0kB, UID:1001 pgtables:14720kB oom_score_adj:0"
    }
    ```

    **Values** lists what the rule or parser extracted, when there is anything. The memory finding in INC-0142 gives **Samples** 6,188, 6,402 and 6,655 MiB and **Confirmed at** 14:32:03.204.
  </Step>

  <Step title="Check the order against nearby events">
    **Order against nearby events** compares the event with up to 7 others within 6 s, nearest first. For the kill:

    | Event                                                                 | Verdict       | Lane and basis                                          |
    | --------------------------------------------------------------------- | ------------- | ------------------------------------------------------- |
    | systemd: a process of perception.service was killed by the OOM killer | 3 ms later    | Companion journal. Same clock.                          |
    | mavlink-router.service stopped                                        | 34 ms later   | Companion journal. Same clock.                          |
    | Last obstacle\_distance sample before a 7.5 s stall                   | Order unknown | Flight controller. 39 ms apart, inside ±46 ms combined. |

    Between anchored clocks a verdict reads, for example, **73 ms earlier*&#x2A; with **±46 ms combined*&#x2A;. After a manual alignment, the basis adds **, manual alignment**. Events that cannot be compared share one **No claim** row: "The companion clock has no shared anchor with the flight log." Select a row to move to that event.
  </Step>

  <Step title="Read the rule">
    For a finding, **Rule** lists **Rule**, **Detects**, **Threshold** and **Wording**. The exit of perception.service reads `process_exit 1.2`, "An allowlisted systemd unit leaves the active state.", "6 units on the allowlist" and "Observed: process exited". **Dismiss this finding** sits below: see [Work an incident](/investigate/incidents#dismiss-a-finding).
  </Step>

  <Step title="Copy a reference">
    The footer holds **Open event**, **Add to report** and **Copy reference**. **Copy reference** copies the incident and event IDs, the file, its full SHA-256 and the location, then confirms with **Evidence reference to the clipboard**.

    ```text title="Copy reference" caption="Example dataset, INC-0142"
    INC-0142/5a705b4a uas04-orin-journal.json sha256:a0741c955f3301a6dffc1a85c93e0eb315e53227c790f517ff7d484e98e0cdf4 Line 18,314, cursor i=5014
    ```
  </Step>
</Steps>

## Explain a record [#explain-a-record]

A flight log message or a journal entry is terse. **Explain**, beside **Copy** under **Original record**, opens a reading of it written by the [AI models](/concepts/ai-models) below the record. **Hide explanation** closes it. It appears for flight log messages, journal entries, receiver records, process samples and rule results.

The reading opens with what the record is. For the USB disconnect in INC-0143 it reads "A companion journal entry from kernel, priority info. The kernel lost a USB device. The numbers after usb are bus, then port on each hub." Each field follows with its value and what it means, with values converted where it helps: `__MONOTONIC_TIMESTAMP` as **262.597 s since boot**, `rss_kib` in MiB, `PRIORITY` with its name.

Where a record is known to leave something out, **What it does not say** names it. For the disconnect: "Why the device left the bus. An unplugged cable, a loose connector, lost power and a device reset all log the same line." For the two receiver records either side of a telemetry gap, it says why the MAVLink sequence number cannot count the lost messages.

The reading ends: "Written by the models on your inference node from the PX4, systemd and MAVLink references in the bundle. The record above is the evidence; this is a reading of it." It is never added to the report or the case file.

## Open the event page [#open-the-event-page]

**Open event**, or the expand icon **Open the event page**, opens `/app/incidents/INC-0142/events/5a705b4a`, under the breadcrumbs **Incidents**, **INC-0142** and **Event 5a705b4a**. Search reaches it too: its **Evidence** group matches the start of an event ID or, for a query longer than 3 characters, text in the title.

### Header and actions [#header-and-actions]

The header gives the incident and file, the title and a message line:

* a journal event shows its `MESSAGE`;
* a flight log record shows its topic and first three fields;
* a receipt shows the message, system ID, sequence number and receive time;
* a process sample shows the process, its PID, RSS and CPU.

The chips give the level, the lane, the rule and the bound, or **Own clock** in its place. A last chip, such as **In the report, step 3**, says where the event sits in the report. On the right, **Across the fleet, 30 days** and **Vehicles** count this signature.

The level follows the record. Journal priority 2 or more severe reads **Critical**, 3 **Error** and 4 **Warning**. A failsafe reads **Critical**, while a finding, a link change, a stall or a vibration event reads **Warning**. Anything else is **Info**, and the kill, at priority 3, reads **Error**.

The action bar steps through the incident's events, **Event 6 of 26 in INC-0142**, also with <Kbd>←</Kbd> and <Kbd>→</Kbd>. **Show on the timeline** opens the **Reconstruction** tab with the event selected. **Discussion**, with the number of notes beside it, opens the drawer described below. **Copy as** offers **Original record**, **Markdown summary** and **Evidence reference**:

````markdown title="Copy as, Markdown summary" caption="Example dataset, INC-0142"
**Kernel killed perception_node (out of memory)**

- Time: 14:32:04.118 UTC, ±6 ms
- Vehicle: UAS-04, run R-0931
- Source: Companion journal, uas04-orin-journal.json sha256:a0741c955f3301a6dffc1a85c93e0eb315e53227c790f517ff7d484e98e0cdf4 Line 18,314, cursor i=5014
- Incident: INC-0142

```
Out of memory: Killed process 2213 (perception_node) total-vm:11873248kB, anon-rss:6823516kB, file-rss:10240kB, shmem-rss:0kB, UID:1001 pgtables:14720kB oom_score_adj:0
```
````

### Across the fleet [#across-the-fleet]

The panel counts the same signature over the last 30 days: **Occurrences**, **Vehicles** and **Runs**, with a bar per day and this event's day highlighted. Rows for **Vehicle**, **Where**, **Software** and **Autopilot** give the most common value's share. For the kill, the counts are 2, 1 and 2: this run and R-0926, both on UAS-04.

### Highlights and detail [#highlights-and-detail]

**Highlights** repeats the essentials: **Time**, **Bound**, **Source**, **File** and **Location**, then **Vehicle**, **Run**, **Where**, **Rule** and **Finding**. **Rule** reads as ID, version and threshold, such as `memory_pressure 1.1: RSS at or above 6,144 MiB for 3 samples`. One detail section follows when the source carries more:

| Section                                                       | Appears for                   | What it shows                                                                                                                                                                                                               |
| ------------------------------------------------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Kernel report**                                             | An out-of-memory kill         | Who invoked the kill, the allocation, the constraint, the victim and its memory, the call trace, the largest tasks and the full report. For the kill: **Invoked by** tile\_decode, thread 2247 of perception\_node on CPU 5 |
| **Failsafe decision**                                         | A failsafe                    | PX4's steps from the last ground heartbeat to the action, timed on the flight log clock, with the parameters in effect                                                                                                      |
| **Unit history**                                              | A systemd event near others   | What systemd logged for the units involved within 12 s. "The journal records the order; it does not record why the router unit stopped."                                                                                    |
| **Vibration around this event**, **Memory around this event** | A vibration or memory finding | 8 s before to 4 s after, with the rule threshold dashed and nearby samples marked **Above** or **Below**                                                                                                                    |
| **Receive gap**                                               | A telemetry gap               | **Last received**, **Next received** and **Silence**: 6.962 s, about 6 heartbeats not received at 1 Hz, in INC-0142                                                                                                         |
| **USB and network**                                           | The radio's USB disconnect    | Kernel and network lines for the radio's port until the link comes back. "The radio itself keeps no log."                                                                                                                   |
| **Link state**                                                | A `telemetry_status` record   | Each heartbeat flag, true or false                                                                                                                                                                                          |

### Breadcrumbs, record, tags and context [#breadcrumbs-record-tags-and-context]

**Breadcrumbs** lists what every source logged in the 3 minutes before the event, newest first, with **This event** first, 12 to a page. Filter it with **Filter breadcrumbs** or by source: **All**, **Flight log**, **Journal**, **Processes** and **Receive**. Each crumb gives its time relative to the event and an expandable **Record**. Unbroken heartbeats fold into one crumb, and so do each process's samples.

**Record** shows the original as a tree with **Copy**. **Tags** lists key and value pairs, grouped **Vehicle**, **Process**, **Source** and **Place**. **Context** holds cards for the **Vehicle**, **Flight controller**, **Companion computer**, **Radio**, **Clock** and **Run**.

### In the report, seen before and discussion [#in-the-report-seen-before-and-discussion]

**Discussion**, in the action bar, opens a drawer with three sections. **In the report** says where the event sits in the report, such as &#x2A;*Step 3 of 8 in the sequence.** An event outside it reads &#x2A;*Not in the sequence yet.** **Seen before** lists the other occurrences, each with its vehicle, run, incident and time. For the kill, the one other occurrence is on R-0926 and carries **No incident**. **Discussion** holds notes on this evidence: write in **Add a note on this evidence** and select **Post**, which records **Commented** in the audit log and shows the note in the drawer at once.

## Check the result [#check-the-result]

* The **SHA-256** matches the one in the run's **Source files** table for the same file.
* **Copy reference** gives a line you can paste into a ticket or a message.
* The audit log shows **Viewed evidence** with your name and the event's title.

## Troubleshoot [#troubleshoot]

**This event does not exist.** The console adds "It may belong to a different incident." **Back to INC-0142** returns to the incident.

**The event shows Own clock.** It sits on a companion clock that never synchronized: it has no position, and its breadcrumbs list only records on the same clock. [Align the clock by hand](/investigate/align-a-clock) if a matching pair exists.

**Seen before reads No other occurrence in the last 30 days.** The signature has not appeared anywhere else in that window.

**Post does nothing.** The note is empty. Write it, then select **Post**.

## Next [#next]

<Cards>
  <Card title="Verify a report against the originals" href="/investigate/verify" icon="hash">
    Check every excerpt against its file.
  </Card>

  <Card title="Evidence integrity" href="/concepts/evidence-integrity" icon="shield">
    How Foxborne keeps originals and hashes.
  </Card>

  <Card title="Collector JSONL records" href="/reference/collector-jsonl" icon="code">
    The fields behind a process sample.
  </Card>

  <Card title="Audit log" href="/security/audit-log" icon="audit">
    Where each view and edit is recorded.
  </Card>
</Cards>
