# Automations
URL: /integrate/automations
Four automations act when a sortie closes out or an alert fires. They triage the sortie, pack failsafe evidence, catch a repeat hardware fault and seal an approved case, and anything that would leave the enclave waits for a named approver.
Detection rules produce findings, and alert rules decide who hears about them. Automations do the work that follows: "What Foxborne does on its own when a sortie closes out or an alert fires." The program starts with four, A1 to A4, **New automation** adds your own, and each one is switched on or off.
The guardrail holds for all four: "Anything that would move content out of the enclave waits for a named approver." Every delivery an automation makes goes into the delivery ledger.
## The Automations page [#the-automations-page]
**Automations** sits in the **Connect** group of the sidebar, at `/app/automations`. It has two tabs: **Automations** and **Run history**, at `/app/automations/runs`. The sidebar count shows how many runs are held for approval.
The **Automations** tab opens with four figures. In the example dataset, **On** reads 3 of 4 and **Runs since 23 Sep** reads 10. **Held for approval** reads 1, and **Named approvers** lists the program's two approvers.
The automations are a table with four columns: **Automation** (its ID, its name, and who turned it on and when), **Trigger**, **Last ran** (with its number of runs and any still waiting) and **On**, the switch. Select a row to open its drawer. The drawer gives **Trigger**, **Conditions**, **Actions** and **Guardrails**, then **This program**, which says how your program applies it and who turned it on. **Run history** at its foot opens the automation's runs.
## Before you start [#before-you-start]
* The destinations each automation uses have egress on. See the first step below.
* Each sortie has a test card that lists its expected sources and any planned failsafe tests. A1 and A2 read it.
* Your program has named approvers. On the **Exports** tab of **Integrations**, **Approval policy** lists them under **Named approvers**.
## Turn an automation on [#turn-an-automation-on]
| Automation | Destinations it uses in the example dataset |
| ---------- | ------------------------------------------------------------------------ |
| A1 | Jira Data Center (project REL) and Mattermost (test-conductor) |
| A2 | Email, to assign the draft case file; PagerDuty, if the program has one |
| A3 | Jira Data Center (project REL) and email for the weekly digest |
| A4 | S3 with Object Lock, the case's ticket and the SIEM over syslog with TLS |
See [Alerts](/integrate/alerts), [Ticketing](/integrate/ticketing) and [Forward logs to your SIEM](/integrate/siem).
A4 cannot run without WORM storage. Until S3 with Object Lock is configured with egress on, its row reads **Needs S3 with Object Lock**, and its drawer warns: "A4 writes each closed case to WORM storage, and none is configured." Select **Set up S3 with Object Lock** at the foot of the drawer to open that destination's settings.
Select the switch in the automation's row. The toast reads, for example, **A1 is on**, and **This program** adds who turned it on and when. The audit log records **Enabled automation** with the automation's ID and name.
Open **Run history**. Each run lists its start time, the automation and its trigger, the run and vehicle, the result and a status.
## Create an automation [#create-an-automation]
Select **New automation** at the top right of the **Automations** page. The dialog notes: "What Foxborne does on its own. Anything that would move content out of the enclave still waits for a named approver."
Give it a **Name**. Under **When**, choose **A sortie closes out**, or an alert rule firing, such as **R3 Failsafe entered fires**. **Only if** takes the conditions in your words; left empty, the automation runs every time the trigger fires.
Tick one or more actions under **Then**:
| Action | What it does |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| **Open a draft case** | Opens a draft case for the investigator on the sortie window |
| **Post an L0 summary to Mattermost** | Posts an L0 summary to the test-conductor channel |
| **Open a Jira issue** | Opens a Jira Data Center issue in project REL that holds the case link |
| **Draft an evidence pack** | Freezes the window across all sources and drafts the PDF case file, marked DRAFT, for a named approver |
The guardrails are fixed and the dialog says so: notifications carry L0 metadata only, and nothing leaves the enclave until a named approver signs off.
Select **Create automation**. It takes the next ID, such as **A5**, and starts turned on. The toast reads, for example, **A5 Jamming report to range safety created and on**, and the audit log records **Created automation**. An action reaches its destination only while that destination's egress is on.
## Sortie closeout triage (A1) [#sortie-closeout-triage-a1]
Ingest complete for a sortie: S3 ObjectCreated to SQS, drop folder or removable-media import, or an API "bundle complete" call.
Every source on the test card has arrived (autopilot log, ground-station
`.tlog`
, companion journal, ROS 2 bag), or 30 minutes have passed with nothing new.
Verify the sender's SHA-256 manifest, align clocks and run the rule pack. If any finding is High, open a case and a Jira Data Center or GitLab issue that holds the case link, and post an L0 summary to Mattermost.
A missing source becomes its own finding. Notifications carry no technical data.
"A case the automation opens stays a draft until an investigator opens it on a window." A run that finishes parsing while A1 is on closes out through A1.
In the example dataset, A1 closed out run R-0931 and drafted INC-0142:
| Step | A1 on R-0931 |
| --------------- | ----------------------------------------------------------- |
| Ingest complete | 4 of 4 sources on the test card at 15:01:20 |
| Sender manifest | SHA-256 verified for 4 files |
| Clocks | Aligned, widest bound ±46 ms |
| Rule pack | 13 rules, 6 findings in 0.9 s |
| Alerts | R4 Memory pressure, Critical; R2 Telemetry gap, Medium |
| Case | Drafted INC-0142, which the investigator opened at 15:02:11 |
| Ticket | REL-418 in Jira Data Center, with the case link |
| Mattermost | L0 summary to test-conductor |
On run R-0930 the companion journal never arrived. After 30 minutes with nothing new, A1 closed the sortie out with 2 of 3 sources and recorded the missing journal as its own finding.
## Failsafe evidence pack (A2) [#failsafe-evidence-pack-a2]
R3 fires.
Armed, and the test card does not list a planned failsafe test for that window.
Freeze a ±120 s window across all sources. Draft the PDF case file with the CUI banner, footer and designation indicator from program settings, marked DRAFT. Assign it to the test director. Send an L0 PagerDuty page if the program has one.
Nothing leaves the enclave until a named approver signs off.
In the example dataset, INC-0143 has the transition A2 waits for. UGV-02's flight log shows `vehicle_status.failsafe` turning true with `gcs_connection_lost`, and PX4 switched to Hold. The run's steps read:
| Step | A2 on R-0934 |
| ------------- | ----------------------------------------------------------------------------------------- |
| Trigger | R3, for the Hold at 15:41:14.293 |
| Armed | Yes, since 15:37:40.118 |
| Test card | Breach lane 2, rehearsal 3 lists no planned failsafe test |
| Frozen window | 15:39:14.293 to 15:43:14.293 on the flight log, companion journal and ground receive log |
| Case file | Drafted with the CUI banner, footer and designation indicator from Settings, marked DRAFT |
| Assigned | The test director, by email |
| Page | Not sent. No PagerDuty service is configured |
| Release | Held. EXP-0034 has 1 of 2 approvals |
The run shows **Held for approval**. Its drawer adds **Export request**: "EXP-0034 asks to take the pack to removable media. 1 of 2 approvals so far." Removable media leaves the enclave, so both named approvers must sign.
## Repeat hardware fault (A3) [#repeat-hardware-fault-a3]
R5 fires.
The same device (serial, port path or CAN interface) has fired in 3 or more of the vehicle's last 10 sorties.
Open a maintenance ticket (ServiceNow, Jira or GitLab) listing sorties and timestamps; mark the vehicle "inspect before next sortie"; include it in the weekly email digest.
Deduplicate per device. Auto-close after 10 clean sorties.
In the example dataset, R5 fired for the mesh radio on USB port path 1-2.3 of UGV-02. A3 read **Not met**: "The mesh radio disconnected on 1 of UGV-02's last 10 sorties, and A3 needs 3." The same port path had disconnected on three other vehicles in 30 days, but "A3 counts per vehicle, so these do not add up."
## Case closure and retention (A4) [#case-closure-and-retention-a4]
Case moves to Approved.
None.
Build a BagIt bag (originals, derived MCAP and Parquet, final PDF, sha256 and sha512 manifests). Write it to WORM storage: S3 Object Lock in compliance mode in GovCloud, or on-prem WORM. Record the tag-manifest SHA-256 in the ticket. Forward an audit event over syslog/TLS, Splunk HEC or Sentinel. Optionally write Parquet to the program data lake.
Two-person approval for any destination outside the enclave. Every delivery is written to the ledger.
A4 is off in the example dataset. Its switch opens **A4 cannot run yet** until S3 with Object Lock is configured with egress on. The dialog adds: "Two-person approval still applies to any destination outside the enclave, and every delivery is written to the ledger."
Under RFC 8493, the tag manifest lists every payload manifest, and each payload manifest lists every payload file with its checksum. The tag-manifest SHA-256 in the ticket therefore pins the whole bag.
## Run history [#run-history]
| Status | What it means | Example result |
| --------------------- | -------------------------------------------------- | ---------------------------------------------------------------------------- |
| **Completed** | The automation ran to the end | "R4 fired at Warning. No finding was High, so no case." |
| **Stopped** | The run could not be triaged | "Stopped. The flight log is encrypted, so it was rejected and no rules ran." |
| **Held for approval** | An export waits for its named approvers | "Held for approval." |
| **Waiting** | The sources are still arriving or parsing | "Waiting for the flight log to finish parsing." |
| **Not met** | The trigger fired, and the conditions did not hold | "Not met." |
Open a run for its drawer. **Result** repeats the status and the result, and **Steps** lists each step with its value.
**Export request** links the held export, and **Deliveries** opens each delivery's ledger entry. **Automation** restates the actions and guardrails.
## Turn an automation off [#turn-an-automation-off]
Select the switch again. The toast reads, for example, "A1 is off. Sorties still import, and alerts still fire."
The audit log records **Disabled automation**.
## Check the result [#check-the-result]
* **Run history** lists the run with **Completed**, and its drawer shows each step.
* The **Delivery ledger** tab lists each delivery under **Automations**, with its class and payload SHA-256.
* After A1 drafts a case, the incident's **Activity** tab shows Foxborne's entry: "sortie closeout triage (A1) drafted this case".
## Troubleshoot [#troubleshoot]
**A run reads Waiting.** Ingest is still in progress, and the result reads "Waiting for the flight log to finish parsing" with the percentage done.
**A run reads Stopped, and no rules ran.** A file failed validation, and the **Validation** step says why, such as an encrypted flight log. See [Import statuses and fixes](/collect/troubleshoot-imports).
**A sortie was not triaged.** The result reads "A1 was off when this run finished parsing, so it was not triaged." Turn A1 on before the next sortie.
**A2 froze no window for a failsafe.** The vehicle was not armed, or the test card listed a planned failsafe test for that window.
**A3 reads Not met.** The same device has to fire in 3 or more of that vehicle's last 10 sorties.
**A4 will not switch on.** Configure S3 with Object Lock and turn on its egress first.
## Next [#next]
* [Alerts](/integrate/alerts) defines R3 and R5, the triggers for A2 and A3.
* [Ticketing](/integrate/ticketing) connects the trackers A1, A3 and A4 write to.
* [Export formats and evidence bundles](/integrate/exports) covers the approvals a held export needs.