Clocks, anchors and error bounds

An incident review starts with one question: what failed first. Answering it means putting events from different logs on one axis, and each of those logs keeps time its own way. The flight controller counts from boot. The companion computer keeps wall time. The ground station records when a packet arrived, not when it left.

Foxborne never pretends those clocks agree. Each source gets a clock, an anchor that places the clock on UTC, and an error bound that says how far off that placement can be. Events keep all three, and the timeline draws the bound around every one.

Three kinds of time#

The bounds in the last column come from the example dataset. Your own bounds follow from your hardware and your time sources, and the run page shows the value for every file you import.

Boot time: the flight log#

Every topic in a ULog carries a timestamp field, a uint64 count of microseconds since the flight controller started. A sample at 1,132,793,000 µs happened 1,132.793 s after boot, and that alone says nothing about when boot happened.

The anchor comes from sensor_gps. Each GPS message reads one instant on both clocks: timestamp plus timestamp_time_relative on the boot clock, and time_utc_usec in UTC from the receiver. A fix whose time_utc_usec is 0, which the receiver reports right after a cold start, carries no UTC time and anchors nothing. The fixes that do carry it give the offset from boot to UTC, and the run page reports how many there were.

uint64 timestamp                # time since system start (microseconds)
int32 timestamp_time_relative   # timestamp + timestamp_time_relative = Time of the UTC timestamp since system start, (microseconds)
uint64 time_utc_usec            # Timestamp (microseconds, UTC), this is the timestamp which comes from the gps module. It might be unavailable right after cold start, indicated by a value of 0

In run R-0931 of the example dataset, 1,081 fixes put the flight controller's boot at 14:13:11.286 UTC, so the sample above lands at 14:32:04.079. GPS time that reaches the flight controller over a serial link carries latency and jitter, which is why the example carries ±40 ms rather than microseconds.

A run with no GPS fix, such as an indoor hover, has no anchor at all. Foxborne marks it Elapsed time only, shows times from log start, and presents nothing on that run as wall-clock time.

Wall time: the companion journal#

The systemd journal stamps every entry with __REALTIME_TIMESTAMP, the wall clock in microseconds since the epoch at the moment journald received the entry. Each entry also carries __MONOTONIC_TIMESTAMP and _BOOT_ID, which together place it on the companion's monotonic clock for that boot.

Wall time is only as good as the clock under it, and on the companion computer chrony disciplines that clock. Foxborne trusts the journal's wall time only while chrony reports a synchronized source. When chronyc tracking shows Leap status : Not synchronised, the journal has no anchor and stays on its own clock.

System time     : 0.000006523 seconds slow of NTP time
Root delay      : 0.013639022 seconds
Root dispersion : 0.001100737 seconds
Leap status     : Normal

The chrony manual bounds the clock error as the system time offset plus the root dispersion plus half the root delay. On the output above that is about 7.9 ms. In run R-0931, chrony was synchronized with an offset of −1.8 ms, and the example dataset carries ±6 ms on the journal.

Process samples from the Foxborne collector share the journal's clock. They record the companion's monotonic time and boot ID, so they inherit the journal's anchor and bound.

Arrival time: the ground receive log#

The MAVLink receiver at range control stamps each message when it arrives, on a clock kept by NTP. In the example dataset that clock is good to ±15 ms, but only for arrival.

A receipt always comes after the transmission. The departure is earlier by the link latency, which Foxborne measures per run: in R-0931 the median was 96 ms and the 95th percentile was 412 ms. Latency extends the bound on one side only, toward earlier times, because it can never move a transmission later than its receipt.

What a bound means at speed#

A clock bound is also a distance. Multiply it by the vehicle's speed and you get how far along its path an event could sit. With INC-0142's combined bound of 46 ms:

On a ground robot the bound is under 20 cm of track. On a jet it can move an event across a boundary line, which is why the evidence inspector also states the bound as meters along the track.

Where the console shows each bound#

• Run page, Source files. The Clock and anchor column names each file's clock and anchor, and Bound shows its error: ±40 ms, ±40 ms + latency for receive logs, or No anchor when there is none.
• Run page, Ordering bounds between sources. A matrix of every pair of files: Same clock, the combined bound such as ±46 ms, or No claim.
• Timeline. Every lane label carries its bound, and every event is drawn with its error bracket and, for receipts, a dashed latency extension.
• Evidence inspector. The When block gives the time, the bound and any latency, followed by the source clock.

When a source has no anchor#

A source without a trustworthy anchor keeps its own clock. Foxborne shows its events on a separate axis, gives them no position on the map, and makes no before or after claim between them and anchored sources. The clock_uncertain rule, which is always on, raises a warning finding whenever this happens.

An engineer can then align the clock by hand from a pair of events that happened at the same moment on both clocks. The alignment carries its own, wider bound and is marked as manual everywhere it is used. Collector 0.8.1 and later also record MAVLink SYSTEM_TIME pairs, which give companion evidence a shared anchor with the flight log.

The ordering bound#

Above a set bound, Foxborne makes no before or after claim between two sources, whatever the times say. The bound is 2.0 s by default. Admins change it in Settings, under Time and position, choosing 0.5 s, 1.0 s, 2.0 s or 5.0 s. Lowering it makes more pairs unknown, and a raised bound is recorded in every report produced afterwards.

Next#

• Alignment and event order explains how two bounds combine and when an order is reported as unknown.
• Align a clock by hand walks through anchoring an unsynchronized companion clock.
• PX4 ULog and journalctl JSON exports list every time field Foxborne reads.