Observability

Status: proposed — consolidated 2026-09-01 from material already designed in 010-tmux-executor.md, 030-http-api.md, 002-data-model.md, and 050-ui-dashboard.md. No new design decisions were made here, except where flagged as a new Open Question below.

Scope

Run logs on executing hosts, the central events journal as an audit/activity feed, host liveness (worker heartbeats, last_seen_at), orchestrator/worker tracing (tracing crate usage), and dashboard live views.

Run Logs

Each run keeps an on-disk log on the worker host that executed it — <platform data dir>/runs/<run_id>.log — with raw pane text appended on every poll (010-tmux-executor.md § Disk Persistence). This is deliberately the audit trail of record: run output text is never written to PostgreSQL (§ Data Retention, 002-data-model.md), only structured metadata about it (line count, byte count, timestamp) is journaled as an event.

GET /runs/{id}/logs serves this log through the orchestrator, which never reads the worker’s filesystem directly — the worker relays it over its resident uplink connection as a Log chunk frame ({type: "log_chunk", run_id, text, timestamp}, 030-http-api.md § Worker Uplink), appended incrementally as new text is captured.

Events Journal

The append-only events table is the audit/activity-feed backbone. Topic enumeration, by domain (002-data-model.md § Event Topics):

• Project management: project.created, project.updated, project.archived
• Work items: work_item.created, work_item.updated, work_item.status_changed, work_item.reparented, work_item.deleted
• Runs: run.created, run.spawning, run.started, run.finished, run.failed, run.stopped, run.signal_received, run.output_appended (metadata only, never full text), run.lease_acquired, run.lease_released, run.orphan_detected
• Workers / hosts: worker.registered, worker.offline, worker.instance_suspect, worker.instance_reinstalled

Every run.* payload carries attempt alongside run_id, so a consumer can tell which incarnation of a restarted run an event describes; run.finished and run.failed never overlap for the same exit (one rule, no overlap — see 002-data-model.md § Event Topics for the exact split). Events are retained 30 days by default ([retention] events_days in config.toml, 060-configuration.md), purged best-effort at orchestrator startup.

Host Liveness

Workers send a heartbeat frame every 30 s over their resident uplink connection; the orchestrator marks a host offline after 90 s without one (3 missed beats) and updates hosts.last_seen_at on every heartbeat regardless of derived status (030-http-api.md § Worker Uplink). This 90 s wire threshold is distinct from the worker’s own local reconciliation loop (30 s) and liveness poll (10 s), which never cross the wire (010-tmux-executor.md).

Status is derived, not stored as a free choice: registered means the operator created the host row and no worker has heartbeated yet; online/offline are heartbeat-derived; disabled is sticky — a heartbeat from a disabled host still updates last_seen_at but never flips status, and only POST /hosts/{id}/enable clears it (resolving to online or offline depending on whether last_seen_at is inside the liveness window) (030-http-api.md).

Two liveness-adjacent events (OQ-14, confirmed — see 061-security.md § Instance-suspect clearing) round out the picture: worker.instance_reinstalled (info-level, silent instance_id update) and worker.instance_suspect (host excluded from new dispatch until an operator runs pam host clear-suspect).

SSE Event Stream

GET /api/v1/events/stream is the activity-feed transport (030-http-api.md § Server-Sent Events):

• Optional topic query parameter, glob-capable (run.*)
• Payload envelope: id: <seq> / event: <topic> / data: <json>
• Reconnect via Last-Event-ID header — server resumes from seq + 1; if that seq was already purged, it sends a full snapshot then resumes live
• No server-sent heartbeats on this channel — reconnection relies on TCP keepalive

Dashboard Live Views

The SPA consumes the SSE stream directly for its global Activity Feed drawer — one entry per event, showing timestamp, topic, a clickable resource link, and a short contextual description, filterable by topic prefix (project.* / work_item.* / run.*) (050-ui-dashboard.md § Activity Feed). The connection is a plain web_sys::EventSource against /api/v1/events/stream, with the same Last-Event-ID reconnect cursor described above.

Live terminal panes are a separate transport (WebSocket, not SSE) — see 050-ui-dashboard.md § Live Pane Rendering for the snapshot/coalescing detail, cross-referenced but not duplicated here since it is UI rendering, not journal/audit observability.

Open Questions

• OQ-18: no orchestrator/worker structured-logging (tracing crate) design exists anywhere in the docs set — log levels, output format (plain vs JSON), and whether/how RUST_LOG or an equivalent is exposed are all undecided. Registered in 071-risks-and-open-questions.md.