Documentation Index

Canonical documentation for project-agent-manager (pam) — a multi-host platform (central orchestrator + per-host workers, PostgreSQL-backed; ADR-012) for hierarchical project/task management and AI task orchestration via tmux-hosted LLM CLIs.

All architecture and contract documents are written in English and carry frontmatter with an implementation: status. Never treat a proposed (or placeholder) document as shipped behavior — see 004-implementation-reference.md for the truthful snapshot of what exists in code.

Numbering Scheme

Documents are numbered 0xx and grouped by domain bands. Reading order within the table is numeric.

Reading Order

Reviews: reviews/ — dated design-review records (e.g., 2026-08-29); historical, never canon.

Canon Selection

Which document answers which question — each decision lives in exactly one canonical document:

Documentation Rules

• Every architecture/contract/UI/roadmap document carries frontmatter: type, authority (canonical|derived), implementation (proposed|partial|placeholder|not-applicable), verification, source, last_verified, owners.
• implementation: placeholder marks a domain skeleton with no designed content yet — it only reserves the number and points at scattered material; never cite it as a decision source.
• authority: canonical documents own their decisions; other documents link instead of restating.
• Overturning a working default recorded in 071-risks-and-open-questions.md must update the affected canonical doc(s) and code in the same commit.
• Architecture/contract docs are in English; top-level README may be bilingual (Korean/English).
• Diagrams: Small inline state/dependency diagrams are allowed in prose; CANONICAL diagrams must be .mermaid files under assets/diagrams/ and registered in the diagram registry (no orphans).
• Verification convention: One executable one-line directive per document (e.g., “Execute all endpoints and validate against curl”).
• Numbering: a new document takes the next free number in its domain band; a new domain claims the next free decade (e.g., 080–089). Numbers are never reused after a document is deleted.

System Overview

System Boundary

Project Agent Manager (pam) is a multi-host platform for hierarchical project management and AI task orchestration. A central orchestrator owns the API, the web dashboard, and a PostgreSQL database; worker agents on execution hosts run the LLM CLI sessions inside each host’s local tmux. This topology was accepted on 2026-08-29 and deliberately overturns the original single-machine, SQLite premise (ADR-012; the overturn is recorded in 071-risks-and-open-questions.md).

The system consists of:

• Central orchestrator (pam serve): a single long-lived process hosting the axum HTTP API, the embedded SPA (Rust/WASM — Leptos), run dispatch, the worker registry, and the PostgreSQL-backed store
• PostgreSQL: the single source of truth — projects, work items, runs, the worker/host registry, and the append-only event journal
• Workers (pam worker): one per execution host; registers with the central orchestrator over an outbound connection (workers never accept inbound connections — NAT-safe), supervises tmux sessions on the host’s dedicated socket pam, streams pane captures upstream, and relays attach traffic
• tmux server (per host): a dedicated tmux server on socket pam hosting LLM CLI processes as sessions; CLI processes are children of the host’s tmux server, NOT of the worker
• LLM CLI processes: long-lived CLI instances (claude, codex, grok, gemini) inside those sessions
• Optional gateway: liteLLM-style proxy routing CLI traffic (configured per-run via environment injection); when present, it is one centrally-shared, network-reachable instance, not a per-host local process (ADR-013)

Component Architecture

flowchart TB
    subgraph Operator["Operator"]
        Browser["Browser / Tauri Desktop"]
        CLI["pam CLI"]
    end

    subgraph Central["Central Orchestrator — pam serve"]
        API["axum HTTP API · SSE · WS"]
        SPA["Embedded Leptos SPA"]
        Dispatch["Run Dispatcher"]
        Fanout["Event Fan-out — in-process broadcast"]
    end

    PG[("PostgreSQL<br/>single source of truth")]
    Gw["Optional LLM gateway<br/>one shared instance"]

    subgraph HostA["Worker Host — macOS · pam worker"]
        WorkerA["Worker Agent<br/>outbound uplink only"]
        TmuxA["tmux Server<br/>socket: pam"]
        WrapA["pam exec run wrapper"]
        Claude["claude CLI"]
        Codex["codex CLI"]
        LogsA["Run logs · providers.toml<br/>host-local disk"]
    end

    subgraph HostB["Worker Host — Linux · pam worker"]
        WorkerB["Worker Agent<br/>outbound uplink only"]
        TmuxB["tmux Server<br/>socket: pam"]
        WrapB["pam exec run wrapper"]
        Grok["grok CLI"]
        Gemini["gemini CLI — optional"]
    end

    Browser -->|"HTTPS + bearer token"| API
    CLI -->|"HTTPS + bearer token<br/>all commands via API"| API
    API --> SPA
    API --> PG
    Dispatch --> PG
    Fanout --> API

    WorkerA -->|"outbound uplink<br/>fetch runs · post events<br/>worker token + TLS"| API
    WorkerB -->|"outbound uplink<br/>fetch runs · post events<br/>worker token + TLS"| API
    WrapA -->|"POST run signals<br/>retried · spooled on partition"| API
    WrapB -->|"POST run signals<br/>retried · spooled on partition"| API

    WorkerA --> TmuxA
    WorkerA --> LogsA
    TmuxA --> WrapA
    WrapA --> Claude
    WrapA --> Codex

    WorkerB --> TmuxB
    TmuxB --> WrapB
    WrapB --> Grok
    WrapB --> Gemini

    Claude -.->|"optional routing"| Gw
    Codex -.->|"optional routing"| Gw
    Grok -.->|"optional routing"| Gw
    Gemini -.->|"optional routing"| Gw

    classDef operatorStyle fill:#e1f5ff,stroke:#01579b,stroke-width:2px
    classDef centralStyle fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
    classDef dbStyle fill:#fffde7,stroke:#827717,stroke-width:2px
    classDef workerStyle fill:#e0f2f1,stroke:#004d40,stroke-width:2px
    classDef tmuxStyle fill:#fff3e0,stroke:#e65100,stroke-width:2px
    classDef providerStyle fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px
    classDef gatewayStyle fill:#fce4ec,stroke:#880e4f,stroke-width:2px
    classDef storageStyle fill:#f1f8e9,stroke:#33691e,stroke-width:2px

    class Browser,CLI operatorStyle
    class API,SPA,Dispatch,Fanout centralStyle
    class PG dbStyle
    class WorkerA,WorkerB,WrapA,WrapB workerStyle
    class TmuxA,TmuxB tmuxStyle
    class Claude,Codex,Grok,Gemini providerStyle
    class Gw gatewayStyle
    class LogsA storageStyle

The system is organized around three control planes sharing a common append-only event journal:

1. Project Management State Plane: Projects, work items (milestone/epic/task/subtask), and status board (todo/doing/done)
2. Run Supervision Plane: Dispatch, spawn, reconcile, restart, and completion signal handling for LLM CLI sessions, distributed across the central orchestrator (dispatch) and workers (execution)
3. Terminal Observation Plane: Pane capture streaming from workers to the central hub, and attach relay from clients through the hub to workers

Runtime Topology

Network Bindings

• Central API: configured bind address (default 0.0.0.0:7898), served with TLS in deployment — no longer localhost-only (ADR-012)
• Worker → central: outbound registration, heartbeat, event/telemetry uplink, and pane/attach relays; the worker holds no listener
• Authentication: browsers and the CLI use bearer tokens against the central API; workers authenticate with a pre-provisioned worker token (mTLS under review — OQ-10)
• Single-operator trust model: no multi-tenant isolation or RBAC

Placement

• Worker hosts: macOS (primary) and Linux (secondary) — the executor is Unix-only by design (ADR-011)
• Central orchestrator: any host with PostgreSQL reachability; running pam serve and pam worker on the same machine recovers the original single-machine deployment without a separate mode

File Locations

• Central host, platform data dir (~/Library/Application Support/pam/ on macOS, ~/.local/share/pam/ on Linux): config.toml, auth token files
• Worker host, platform data dir: worker config.toml, provider catalog, per-run on-disk logs under runs/
• PostgreSQL: all persistent state (projects, work items, runs, hosts, events)

tmux Integration

Unchanged from the original design, now executed by the worker on each host. The tmux server uses a dedicated socket named pam to isolate pam-managed sessions from system tmux instances:

• Socket creation: tmux -L pam -S /tmp/tmux-$UID/pam
• Session naming: pam-<short_run_id> where short_run_id is the first 8 lowercase hex characters of the run UUID
• CLI processes run as children of the tmux server, allowing detached operation and observation

Event Streaming

• Worker → central: lifecycle and pane-metadata events flow upstream over the worker’s registration connection
• Central fan-out: in-process tokio::sync::broadcast for connected clients (the central orchestrator is a single process; PostgreSQL LISTEN/NOTIFY is not required for fan-out)
• Server-Sent Events endpoint: /api/v1/events/stream
• Append-only event journal stored in the events table in PostgreSQL for activity feed and audit trail

Control Plane Coordination

The three control planes coordinate through the shared event stream: each plane’s state changes are journaled as events in PostgreSQL at the central hub, and each plane subscribes to the events of the other planes (e.g., terminal observation listens for run.started to begin capture-pane streaming). The authoritative enumeration of which plane emits which topic lives in 002-data-model.md §Event Topics — this document deliberately does not restate the list.

Operational Review Perspective

Reviewing the central/worker split for correctness means checking boundaries that this document describes only at the topology level, not at the decision level. A reviewer should follow these pointers rather than treat this document’s summary as the full picture:

• Worker authentication. §Network Bindings states the current default (pre-provisioned worker token over TLS), but the token lifecycle — provisioning, rotation, revocation, and the mTLS alternative — is still an open decision, tracked as OQ-10 in 071-risks-and-open-questions.md. Treat OQ-10, not this file, as the source for worker-trust review.
• Worker uplink protocol. §Network Bindings and §Event Streaming assume an outbound worker→central connection but do not commit to a transport. The transport choice (persistent WebSocket with heartbeat vs. a polling fallback) is tracked as OQ-11 in the same document and blocks Phase 2 implementation — review it before assuming either transport is final.
• Security documentation gap. 061-security.md is currently a placeholder (see docs/README.md Reading Order). Until it is written, this document’s Network Bindings section is the only committed description of the trust model. A security-focused review should record that as a gap to close, not as evidence the trust model has already been reviewed.

This section adds no new decisions of its own — it exists so an operational review of the central/worker boundary starts from the documents that actually own those decisions.

Non-Goals

The following features are explicitly out of scope for pam:

Superseded non-goals (overturned 2026-08-29, ADR-012): “remote multi-host workers excluded” and “PostgreSQL backend excluded” — both were pillars of the original single-machine design and are now core to the architecture.

Data Model

Database Schema

pam uses PostgreSQL as its single source of truth (ADR-012, overturning the original SQLite decision). Core tables follow. All IDs are UUID (gen_random_uuid() where defaulted). All timestamps are TIMESTAMPTZ (UTC). Enumeration-like columns use TEXT with CHECK constraints — the application layer (pam-core enums) remains the authority for legal values.

hosts

CREATE TABLE hosts (
    id UUID PRIMARY KEY,
    name TEXT UNIQUE NOT NULL,  -- operator-chosen worker/host name
    platform TEXT NOT NULL CHECK(platform IN ('macos', 'linux')),
    token_fingerprint TEXT,  -- fingerprint of the worker auth token (never the token itself)
    status TEXT NOT NULL CHECK(status IN ('registered', 'online', 'offline', 'disabled')),
    instance_id UUID,  -- worker-generated, persisted in the worker's local data dir; NULL until the first instance-id-aware registration
    instance_suspect BOOLEAN NOT NULL DEFAULT false,  -- pinned instance_id disagrees with a recent/concurrent presenter (OQ-14)
    instance_flagged_at TIMESTAMPTZ,  -- when instance_suspect last transitioned false -> true; NULL while not suspect
    last_seen_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL
);

Columns:

• id: Unique host identifier (UUID)
• name: Operator-chosen unique worker name (used for dispatch targeting and display)
• platform: macos or linux — worker hosts are Unix-only (ADR-011)
• token_fingerprint: Hash fingerprint identifying which worker token this host authenticates with
• status: Registry state — registered (known, not yet seen), online (heartbeat current), offline (heartbeat lost), disabled (excluded from dispatch)
• instance_id: Per-process worker instance UUID (OQ-14, 071-risks-and-open-questions.md) — generated once by the worker process and persisted in its local data dir, then sent at every registration and heartbeat. Guards against one worker token cloned onto two machines aliasing a single hosts row. Pinned, not overwritten: when an incoming instance_id disagrees with the stored value, the column is updated only when the mismatch resolves as a legitimate reinstall (see instance_suspect below) — overwriting on every mismatch would let an alternating clone erase its own trail
• instance_suspect: Set true when an incoming instance_id disagrees with the pinned value AND the gap between now and the pinned instance’s last last_seen_at is within the offline threshold (90s) — i.e., the pinned instance still looked alive when the new one showed up, so the mismatch reads as a concurrent clone rather than a reinstall. A host in this state is excluded from new dispatch (alongside disabled) but its already-running runs are left untouched; the mismatch is journaled once on the false→true transition, not on every subsequent heartbeat (debounced) while it stays suspect. Cleared by the operator (pam host clear-suspect, 040-cli.md), which also accepts the new instance_id as the pinned value
• instance_flagged_at: Timestamp of the instance_suspect false→true transition; NULL while not suspect. Reset to NULL when the operator clears the flag
• last_seen_at: Last heartbeat/registration timestamp; drives online/offline transitions (default cadence: heartbeat every 30s; a host is marked offline after 90s without one — 3 missed beats). This same 90s window is the recency threshold instance_suspect compares against — when the gap since the pinned instance’s last heartbeat already exceeds it, the mismatch is read as a legitimate reinstall instead: instance_id updates silently and an info-level event is journaled, no instance_suspect flag set
• created_at / updated_at: Record lifecycle timestamps

projects

CREATE TABLE projects (
    id UUID PRIMARY KEY,
    name TEXT NOT NULL,
    slug TEXT UNIQUE NOT NULL,  -- URL-safe identifier
    description TEXT NOT NULL DEFAULT '',
    status TEXT NOT NULL CHECK(status IN ('active', 'archived')),
    created_at TIMESTAMPTZ NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL
);

Columns:

• id: Unique project identifier (UUIDv4)
• name: Human-readable project name
• slug: URL-safe identifier for CLI/API references
• description: Optional project description
• status: Project lifecycle state (active/archived)
• created_at: Creation timestamp
• updated_at: Last modification timestamp

Archived semantics: an archived project is read-only for structural mutation — no new work items may be created in it, and no new runs may be created against its work items. Runs already in flight when the project was archived continue to completion, and their events still journal. Un-archiving (archived → active) is permitted and restores full mutability.

work_items

CREATE TABLE work_items (
    id UUID PRIMARY KEY,
    project_id UUID NOT NULL,
    parent_id UUID,  -- Self-referential FK for hierarchy
    kind TEXT NOT NULL CHECK(kind IN ('milestone', 'epic', 'task', 'subtask')),
    title TEXT NOT NULL,
    description TEXT NOT NULL DEFAULT '',
    status TEXT NOT NULL CHECK(status IN ('todo', 'doing', 'done', 'cancelled')),
    assignee TEXT,  -- Optional assignee identifier
    path TEXT NOT NULL,  -- Materialized path for subtree queries
    sort_order INTEGER NOT NULL DEFAULT 0,
    created_at TIMESTAMPTZ NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL,
    FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE,
    FOREIGN KEY (parent_id) REFERENCES work_items(id) ON DELETE CASCADE,
    CHECK (parent_id IS NULL OR parent_id <> id)  -- hierarchy guard: no self-parenting
);

Columns:

• id: Unique work item identifier (UUIDv4)
• project_id: Parent project reference
• parent_id: Parent work item for hierarchical structure (NULL for root items)
• kind: Work item type (milestone/epic/task/subtask)
• title: Work item name
• description: Optional detailed description
• status: Current workflow state (todo/doing/done/cancelled)
• assignee: Optional person/team assigned
• path: Materialized path for efficient subtree queries (see Path Semantics)
• sort_order: Ordering within siblings
• created_at: Creation timestamp
• updated_at: Last modification timestamp

Hierarchy guards: CHECK (parent_id IS NULL OR parent_id <> id) rejects self-parenting at the database level. Additionally, a parent and its children must share the same project_id — a documented constraint enforced at the application layer, since a column CHECK cannot compare values across rows. Reparenting is the mutation that must re-check this invariant: the new parent must belong to the same project_id as the moved item, and cross-project moves are rejected. Preventing deeper cycles (A → B → A beyond the trivial self-cycle) likewise remains an application-layer concern, tracked as OQ-6 in 071-risks-and-open-questions.md.

runs

CREATE TABLE runs (
    id UUID PRIMARY KEY,
    task_id UUID NOT NULL,
    host_id UUID,  -- Worker host assigned to execute this run (NULL while pending)
    provider TEXT NOT NULL,  -- claude, codex, grok, gemini
    session_name TEXT NOT NULL,  -- tmux session name: pam-xxxxxxxx (unique per host)
    status TEXT NOT NULL CHECK(status IN ('pending', 'spawning', 'running', 'exited', 'failed', 'stopped')),
    attempt INT NOT NULL DEFAULT 1,  -- incarnation epoch: incremented centrally on each restart requeue
    last_progress_seq BIGINT,  -- highest applied progress seq within the current attempt (NULL until first progress)
    failure_reason TEXT,  -- machine-readable terminal reason (host_lost, session_lost, spawn_failed, restart_exhausted, dispatch_timeout, no_eligible_host)
    pending_since TIMESTAMPTZ NOT NULL,  -- dispatch-timeout anchor; set to created_at at insert, reset to now() on every failed -> pending requeue
    cwd TEXT NOT NULL,  -- Working directory for CLI execution (on the worker host)
    exit_code INTEGER,  -- Process exit code (NULL if not exited)
    started_at TIMESTAMPTZ,
    finished_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL,
    FOREIGN KEY (task_id) REFERENCES work_items(id) ON DELETE CASCADE,
    FOREIGN KEY (host_id) REFERENCES hosts(id),
    UNIQUE (host_id, session_name)
);

Columns:

• id: Unique run identifier (UUID)
• task_id: Associated task work item; must reference a work item of kind task or subtask only. Application-enforced — a column CHECK cannot read the referenced row. The central reconciliation pass treats a run pointing at a milestone or epic as a data-integrity anomaly to journal, not to auto-repair
• host_id: Worker host executing the run; assigned at dispatch (NULL while pending)
• session_name: tmux session name (format: pam-<short_run_id>); uniqueness is scoped per host via UNIQUE (host_id, session_name)
• provider: LLM provider identifier (claude/codex/grok/gemini)
• status: Run lifecycle state (pending/spawning/running/exited/failed/stopped)
• attempt: Incarnation epoch — which execution incarnation of the run is speaking. Incremented centrally each time the restart policy requeues the run (failed → pending); workers never allocate attempts (they never write the database). The value is injected into the wrapper’s spawn environment so the wrapper can stamp it on every signal, where it is REQUIRED. A signal whose attempt is less than runs.attempt comes from a dead incarnation and is discarded — no status write, no event emission. An attempt greater than runs.attempt is impossible under central issuance and is rejected as a protocol error
• last_progress_seq: Highest applied seq among repeated, dedup-exempt signal kinds (progress) within the current attempt (NULL until the first progress signal is applied). Rule: apply a progress signal only if seq > last_progress_seq. Reset on restart requeue — the new incarnation numbers its progress signals from scratch
• failure_reason: Machine-readable terminal reason (host_lost, session_lost, spawn_failed, restart_exhausted, dispatch_timeout, no_eligible_host); NULL while the run is non-terminal or ended without a distinct reason
• pending_since: Dispatch-timeout anchor (OQ-15, 071-risks-and-open-questions.md). Set equal to created_at when the row is first inserted, and reset to now() on every failed → pending requeue. Kept distinct from created_at (which never changes after insert) precisely so a requeued run gets a fresh dispatch-timeout window instead of inheriting the original creation time — a run that failed after 55 minutes and requeued would otherwise be measured against a created_at already 55 minutes stale, and would breach the 5-minute dispatch timeout the instant it re-entered pending
• cwd: Working directory for CLI process execution (set at pam run start via --cwd flag; default: current working directory of CLI invocation)
• exit_code: Process exit code when terminated
• started_at: Process start timestamp
• finished_at: Process completion timestamp
• created_at: Record creation timestamp
• updated_at: Last modification timestamp

UNIQUE-NULL semantics on UNIQUE (host_id, session_name): PostgreSQL treats NULLs as distinct in unique constraints, so rows with host_id IS NULL — runs still in pending, not yet dispatched to a host — never collide with each other or with host-assigned rows. The constraint binds only once a host is assigned at dispatch. This is intended, not a bug: it enforces per-host session-name uniqueness exactly when a session can actually exist (post-dispatch) without blocking concurrent pending runs.

processed_signals

Dedup marker table that gives one-shot lifecycle signals exactly-once semantics under at-least-once delivery (spool + replay):

CREATE TABLE processed_signals (
    run_id UUID NOT NULL,
    attempt INT NOT NULL,
    kind TEXT NOT NULL,
    received_at TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (run_id, attempt, kind),
    FOREIGN KEY (run_id) REFERENCES runs(id) ON DELETE CASCADE
);

Columns:

• run_id / attempt / kind: The dedup key — one row per applied one-shot signal of a given kind within a given incarnation
• received_at: When the signal was first applied (forensics only; not part of matching)

Ingest semantics — the guard lives centrally, at the signal-ingest edge; the worker never queries the database, so the judgment cannot live worker-side:

• One-shot kinds (started, finished, failed) are exactly-once via this table: ingest performs INSERT ... ON CONFLICT DO NOTHING before applying; a conflict means the signal was already applied and it is dropped. These kinds carry no seq — it would be pure cost
• Repeated kinds (progress) are dedup-exempt: naturally repeatable, ordered instead via runs.last_progress_seq (apply only if seq > last_progress_seq). seq is optional on the signal payload and carried only by these repeated kinds
• The stale-epoch guard and the dedup both run strictly upstream of BOTH the run-status write AND the event journal / SSE emission — skipping the status write while still journaling the event would leak phantom events to the dashboard
• Ordering is a lexicographic comparison on (attempt, seq). The two axes are orthogonal: attempt answers “which execution incarnation is speaking”; seq answers “is this newer than what I already applied within that incarnation”. Neither subsumes the other

events

CREATE TABLE events (
    seq BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    topic TEXT NOT NULL,
    payload JSONB NOT NULL,  -- structured event data
    created_at TIMESTAMPTZ NOT NULL
);

Columns:

• seq: Monotonically increasing sequence number
• topic: Event type identifier (see Event Topics)
• payload: Structured event data as JSONB
• created_at: Event timestamp

_sqlx_migrations

Standard sqlx migration tracking table (auto-created by sqlx).

Indexes

CREATE INDEX work_items_project_id ON work_items(project_id);
CREATE INDEX work_items_parent_id ON work_items(parent_id);
CREATE INDEX work_items_path ON work_items(path text_pattern_ops);
CREATE INDEX runs_task_id ON runs(task_id);
CREATE INDEX runs_host_id ON runs(host_id);
CREATE INDEX events_topic_created_at ON events(topic, created_at);

These indexes optimize common queries:

• Project-scoped work item lookups
• Parent-child relationship queries
• Path-prefix subtree scans (board rendering, subtree listing — see Path Semantics)
• Task-associated run queries
• Host-scoped run queries (worker reconciliation)
• Topic-filtered event queries (SSE ?topic= filter, retention purge)

Path Semantics

The path column in work_items implements materialized path storage for efficient subtree queries:

Format: <short_id>/<parent_short_id>/<grandparent_short_id>/...

• Root items: path = own short ID (first 8 characters of UUID)
• Child items: path = <parent_path>/<own_short_id>
• Short IDs: First 8 lowercase hex characters of UUID

Example:

Item A (id: a1b2c3d4-...)        path: "a1b2c3d4"
  └─ Item B (id: e5f6a7b8-...)    path: "a1b2c3d4/e5f6a7b8"
      └─ Item C (id: c9d0e1f2-...) path: "a1b2c3d4/e5f6a7b8/c9d0e1f2"

Subtree Queries: two forms exist; which one is used depends on what the traversal needs.

Path-prefix (primary):

SELECT * FROM work_items
WHERE path = ? OR path LIKE ? || '/%'
ORDER BY path, sort_order;

A single index scan on work_items_path (text_pattern_ops exists for exactly this LIKE 'prefix%' shape). This is the default for read paths — board rendering, subtree listing.

Recursive CTE:

WITH RECURSIVE subtree AS (
    SELECT * FROM work_items WHERE id = ?
    UNION ALL
    SELECT w.* FROM work_items w
    INNER JOIN subtree s ON w.parent_id = s.id
)
SELECT * FROM subtree ORDER BY path, sort_order;

Used when the traversal needs live parent links rather than the materialized path — for example during a reparent, when path is mid-rewrite.

Reparent rewrite: moving a work item rewrites path for the moved item and every descendant. This is a single transaction; the subtree is located by the old path prefix before the rewrite begins. work_item.reparented is journaled once, for the moved item — not once per descendant.

Short-id collisions: path segments and tmux session names (pam-<short_run_id>) both use the first 8 lowercase hex characters of a UUID, and an 8-hex prefix is NOT guaranteed unique. The scopes differ: runs.session_name is unique per host via UNIQUE (host_id, session_name), while path short-ids are unique only within a sibling set in practice. path is a lookup accelerator, never an identity — resolution always goes through id. Where a collision would break a display, the UI disambiguates by showing more characters.

Status Transitions

Work Item Status

stateDiagram-v2
    [*] --> todo
    todo --> doing
    doing --> done
    todo --> done: complete without starting
    done --> doing: reopen
    doing --> todo
    todo --> cancelled
    doing --> cancelled
    done --> cancelled
    cancelled --> todo: reopen
    cancelled --> [*]

Legal transitions:

• todo → doing: Start work on item
• doing → done: Complete work
• todo → done: Complete without starting (work finished out-of-band); justified by the pam task done CLI verb (040-cli.md) and Kanban board drag (050-ui-dashboard.md)
• done → doing: Reopen for additional work
• doing → todo: Defer work
• any → cancelled: Cancel work
• cancelled → todo: Reopen cancelled item

Parent/child status coupling:

• Cancelling a parent cascades cancelled to all non-done descendants; done descendants keep their status — history is not rewritten
• A parent does NOT auto-transition to done when its children complete. Roll-up is a display concern (the dashboard may show a completion ratio); the stored parent status only changes through an explicit transition, because auto-roll-up would make the parent’s status unattributable to any actor, which conflicts with the transition-enforcement model in ### Enforcement

Run Status

stateDiagram-v2
    [*] --> pending
    pending --> spawning: spawn requested
    spawning --> running: process started
    running --> exited: clean exit
    running --> failed: error exit
    running --> stopped: user stopped
    spawning --> failed: spawn failed
    pending --> failed: dispatch timeout / no eligible host
    failed --> pending: automatic restart (bounded retries)
    exited --> [*]
    failed --> [*]
    stopped --> [*]

Legal transitions:

• pending → spawning: Initiate tmux session creation
• spawning → running: CLI process confirmed running
• spawning → failed: tmux or CLI process creation failed
• pending → failed: Dispatch-timeout failure — the only exit for a run no worker ever claims. Two causes: dispatch timeout — a run sits in pending beyond the dispatch timeout (default 5 minutes, configurable, measured from runs.pending_since — NOT created_at; see the runs schema above) without being claimed by any worker, and the central orchestrator writes failed with failure_reason = 'dispatch_timeout'; no eligible host — POST /runs accepted the run but no online, non-disabled, non-suspect host matched the requested target, so the run is created in pending and fails at the same timeout with failure_reason = 'no_eligible_host'. Host unassignment — a pending run whose target host is disabled, marked suspect (hosts.instance_suspect, OQ-14), or goes offline before the claim — does NOT immediately fail the run: it stays pending and may be claimed by another eligible host, failing only at the dispatch timeout. The 5-minute default is confirmed (OQ-15, 071-risks-and-open-questions.md)
• running → exited: Process terminated with exit code 0
• running → failed: Process terminated with non-zero exit code
• running → stopped: User-initiated stop
• failed → pending: The canonical restart-requeue edge. Performed centrally by the restart policy; on requeue, runs.attempt is incremented and the per-attempt columns (started_at, finished_at, exit_code, last_progress_seq, pending_since) are reset — pending_since resets to now() so the requeued attempt gets a fresh dispatch-timeout window (OQ-15) rather than inheriting the original created_at. A requeue MAY select a different host than the failed attempt used — it is the only automatic escape hatch when the original host is lost or unhealthy, and this includes runs tombstoned failure_reason = 'host_lost' (OQ-12, confirmed: requeue-eligible) — see 010-tmux-executor.md § Restart Policy for the accepted trade-off that decision carries. 010-tmux-executor.md owns restart POLICY only (trigger conditions, attempt caps, backoff)
• exited/failed/stopped → [*]: Terminal states

Ownership: This document owns the run state machine as canon — the legal edges above are the single authority for run-status transitions. 010-tmux-executor.md owns restart POLICY only (trigger conditions, attempt caps, backoff) and conforms to this machine; other renderings of the graph (provider adapter transition tables, lifecycle diagrams) are derived views and must not introduce edges absent here.

Enforcement

Value legality — which values a status column may hold — is enforced by the CHECK constraints in the schema blocks above. Transition legality — which edges between those values are permitted — is enforced at the application layer by a single pam-core state-machine module, which is the sole gate: every mutating path must pass through it, including HTTP handlers, CLI commands, and signal normalization. DB-level transition enforcement is deliberately not used: a CHECK constraint cannot cheaply see the previous state (it evaluates the incoming row in isolation), so encoding a (previous, next) pair would require triggers or two-row gymnastics that buy nothing under the single-writer architecture (see Concurrency Model — all writes flow through central). The cheap DB guards that are taken are the structural ones — the work_items hierarchy CHECK and same-project constraint above — not transition rules.

Event Topics

The append-only event journal uses these topics:

Project Management:

• project.created: New project created
• project.updated: Project metadata modified
• project.archived: Project marked as archived

Work Items:

• work_item.created: New work item created
• work_item.updated: Work item metadata modified
• work_item.status_changed: Work item status transition
• work_item.reparented: Work item moved to new parent
• work_item.deleted: Work item deleted

Runs:

• run.created: Run record created (pending status)
• run.spawning: tmux session creation started
• run.started: CLI process confirmed running
• run.finished: Wrapper process exited — any exit code; journaled on every wrapper exit, without exception
• run.failed: Reserved for the failure shapes where no wrapper exit occurred or the provider hook reported failure: spawn failure (the session never started) and a provider completion hook reporting Failed
• run.stopped: Process stopped by user
• run.signal_received: Signal sent to process
• run.output_appended: New output captured (NOTE: payload contains only metadata, not full text)
• run.lease_acquired: Interactive attach lease granted
• run.lease_released: Interactive attach lease released or expired
• run.orphan_detected: Unmapped tmux session discovered during reconciliation

Run topic rules (normative): every run.* payload carries the run’s attempt value alongside run_id — a consumer (SSE client, activity feed) must be able to tell which incarnation of a restarted run an event describes; without it, events from a dead incarnation are indistinguishable from the live one. run.finished and run.failed do not overlap: a non-zero exit journals run.finished with exit_code ≠ 0 and sets runs.status = failed — it does NOT additionally journal run.failed. One rule, no overlap.

Workers / Hosts:

• worker.registered: Worker authenticated to the orchestrator (host → online)
• worker.offline: Heartbeat lost past the offline threshold (host → offline)
• worker.instance_suspect: A registration or heartbeat presented an instance_id that disagrees with the pinned value while the pinned instance still looked recently alive (OQ-14) — journaled once on the false→true transition, debounced while the host stays suspect
• worker.instance_reinstalled: A registration presented a different instance_id after the pinned instance had already gone quiet past the offline threshold — read as a legitimate reinstall, info-level only, hosts.instance_id updates silently

Important: Run output text is NOT journaled to events table. Only structured metadata (line count, byte count, timestamp) is included in payloads.

Storage Policy

Hybrid Persistence Model

pam uses hybrid persistence combining traditional state tables with an append-only event journal:

Current State: Stored in normalized tables (projects, work_items, runs) for efficient queries and updates.

Event Journal: Append-only events table for:

• Activity feed and audit trail
• Server-Sent Events (SSE) streaming at /api/v1/events/stream
• In-memory fan-out via tokio::sync::broadcast

Rationale: Full event sourcing was rejected to avoid complexity in replay and projection maintenance. The hybrid model provides current state query performance with event streaming capabilities.

Data Retention

Never in PostgreSQL:

• tmux pane text and scrollback (held in the central snapshot cache in-memory + per-run on-disk logs on the worker host under its data dir runs/)
• Large output payloads (events carry only metadata)

In PostgreSQL:

• Current project/work item/run state
• Worker/host registry
• Event journal with small structured payloads (JSONB)
• Schema migration history

Events retention: Events table rows are retained for 30 days by default. Retention period is configurable via [retention] events_days setting in the configuration file. Purge is best-effort and executed at orchestrator startup. No special events are emitted during purge; the HTTP event stream may report purged sequence ranges for client awareness.

Concurrency Model

Write Model:

• PostgreSQL connection pool (sqlx) owned by the central orchestrator
• Workers never connect to PostgreSQL directly — all writes flow through the central API/uplink, keeping database credentials on exactly one host
• Short transactions; run dispatch transitions use SELECT ... FOR UPDATE on the run row to serialize dispatch claims

Read Model:

• Multiple read connections from the same pool
• Read-only transactions for queries

Test Support:

• MemStore implementation for in-memory testing
• Same Store trait interface as the PostgreSQL implementation
• Enables fast unit tests without database I/O; a containerized PostgreSQL (ephemeral database per run) covers migration and dialect tests in CI

Entity Relationships

erDiagram
    hosts ||--o{ runs : "executes on"
    projects ||--o{ work_items : "contains"
    work_items ||--o{ work_items : "parent-child"
    work_items ||--o{ runs : "executes as"
    work_items ||--o{ events : "generates"
    runs ||--o{ events : "generates"
    projects ||--o{ events : "generates"
    runs ||--o{ processed_signals : "dedup ledger"

    hosts {
        uuid id PK
        text name UK "Human-readable host name"
        text platform "macos|linux"
        text token_fingerprint "Worker token hash — raw token never stored"
        text status "registered|online|offline|disabled"
        uuid instance_id "Worker-generated, per-process — NULL until first instance-id-aware registration (OQ-14)"
        boolean instance_suspect "Pinned instance_id disagrees with a recent/concurrent presenter (OQ-14)"
        timestamptz instance_flagged_at "instance_suspect false->true transition time; NULL while not suspect"
        timestamptz last_seen_at "Worker heartbeat"
        timestamptz created_at
        timestamptz updated_at
    }

    projects {
        uuid id PK
        text name "Human-readable name"
        text slug UK "URL-safe identifier"
        text description "Optional description"
        text status "active|archived"
        timestamptz created_at
        timestamptz updated_at
    }

    work_items {
        uuid id PK
        uuid project_id FK "References projects(id)"
        uuid parent_id FK "Self-reference for hierarchy"
        text kind "milestone|epic|task|subtask"
        text title "Item name"
        text description "Optional details"
        text status "todo|doing|done|cancelled"
        text assignee "Optional assignee"
        text path "Materialized path"
        integer sort_order "Sibling ordering"
        timestamptz created_at
        timestamptz updated_at
    }

    runs {
        uuid id PK
        uuid task_id FK "References work_items(id)"
        uuid host_id FK "References hosts(id) — NULL while pending"
        text provider "claude|codex|grok|gemini"
        text session_name "tmux session name — unique per host"
        text status "pending|spawning|running|exited|failed|stopped"
        integer attempt "Incarnation epoch — incremented on restart requeue"
        bigint last_progress_seq "Highest applied progress seq in this attempt"
        text failure_reason "host_lost|session_lost|spawn_failed|restart_exhausted|dispatch_timeout|no_eligible_host"
        text cwd "Working directory"
        integer exit_code "Process exit code"
        timestamptz pending_since "Dispatch-timeout anchor — created_at at insert, reset to now() on failed->pending requeue (OQ-15)"
        timestamptz started_at "Process start time"
        timestamptz finished_at "Process end time"
        timestamptz created_at
        timestamptz updated_at
    }

    processed_signals {
        uuid run_id PK,FK
        integer attempt PK
        text kind PK
        timestamptz received_at
    }

    events {
        bigint seq PK "GENERATED ALWAYS AS IDENTITY"
        text topic "Event type identifier"
        jsonb payload "Event data"
        timestamptz created_at
    }

The entity relationship diagram shows the key relationships and cardinalities between the five core tables, including the processed_signals dedup ledger (one row per applied one-shot signal kind per attempt, removed with its run via ON DELETE CASCADE). For complete schema definitions, see the SQL blocks above.

Architecture Decisions (ADR Index)

Index of accepted architecture decisions. Each decision is specified in the document listed under “Specified in” — this file never restates the rationale (single source of truth; see README.md).

Adding a new ADR

1. Append a row with the next ADR number; link the document that specifies it.
2. The decision text stays one sentence; rationale lives only in the target doc.

Implementation Reference

Truthful snapshot of what exists in code versus what the canonical documents propose. Never treat a proposed document as shipped behavior: this file is the map from design to reality.

What Exists (Phase 0: design docs + scaffold)

Compiling, gate-clean Cargo workspace. All four gates pass locally (cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo build --workspace, cargo test --workspace — 29 tests across the workspace).

ADR-012 pivot (2026-08-29): the canonical docs now describe a multi-host topology — central orchestrator (pam serve) + per-host workers (pam worker), PostgreSQL as the single source of truth. The scaffold below still reflects the pre-pivot single-machine/SQLite design and remains gate-clean; code catches up from Phase 1 (PgStore, API-backed CLI, pam worker / pam host command groups). See 003-architecture-decisions.md ADR-012.

web/dist/index.html is a placeholder (the real Leptos/WASM SPA lands in Phase 3 via trunk build; no Node toolchain needed to build today). Toolchain pinned to 1.93.1 in rust-toolchain.toml, .tool-versions, and CI — bumped together in one commit per the CLAUDE.md rule.

Scaffold-level decisions (divergences recorded for Phase 1+)

• sqlx is built with the macros feature (required for sqlx::migrate!()); no query! macros are used anywhere, so no DATABASE_URL is needed at compile time.
• SqliteStore::open is async (sqlx pool creation performs I/O).
• short_run_id/session_name_for return Option<String> (invalid input → None, never panic).
• Pool is max_connections(1) — each pooled sqlite::memory: connection is its own database; revisit pool width in Phase 1 for file-backed DBs.
• foreign_keys(true) pragma is enabled (FK constraints are inert without it).
• pam-cli currently depends only on clap + anyhow; internal path dependencies are registered in [workspace.dependencies] for Phase 1 adoption.
• Workspace [workspace.package] omits name/license (not inheritable / deliberately unset).
• Migration 0001 matches 002-data-model.md including ON DELETE CASCADE FKs and description TEXT NOT NULL DEFAULT ''.

Target Contract vs Implemented

CI (.github/workflows/ci.yml)

Three jobs, no service containers, no DATABASE_URL anywhere: test (four gates as separate steps, toolchain 1.93.1, Cargo.lock-keyed cache), docs (frontmatter + relative-link + mermaid-reference check in stdlib Python; README.md navigation files exempt; mermaid-cli parse is continue-on-error), coverage (cargo-llvm-cov → lcov artifact). From Phase 1 the test job needs an ephemeral PostgreSQL service container for PgStore tests (see 063-testing-strategy.md).

Git State

Branch main only. No remote configured yet — origin registration and first push are planned for September 2026 (see 071-risks-and-open-questions.md OQ-1).

Maintenance Rule

This file is authority: derived: it never makes decisions, it reports them. Any commit that changes what exists must update this file in the same commit.

Engineering Rules

Standard engineering conventions and tooling guidelines for the project-agent-manager (pam) repository.

Language Rules

• Architecture/contract docs under docs/: Must be written in English. Each file must carry YAML frontmatter with an implementation: status key.
• README.md: Bilingual (Korean-first) is acceptable.
• User-facing chat replies: Follow the user’s conversation language.
• Code, code comments, and commit messages: Must be in English.

Toolchain Pinning

• Rust Version Match: The following files/fields MUST all name the SAME Rust version (currently 1.93.1):
  • rust-toolchain.toml
  • .tool-versions
  • The toolchain: input in .github/workflows/ci.yml
• Version Bumping: Any version bump must change all three in ONE commit, never mixed with feature changes. Local gates are meaningless if they run a different compiler than CI.

Docs Rules

• Documentation Index: docs/README.md is the canon index. New docs must be linked from it.
• Frontmatter Requirement: Every architecture/contract/UI/roadmap document carries frontmatter:

  type: architecture | contract | rules | roadmap
  authority: canonical | derived
  implementation: proposed | partial | placeholder | shipped | not-applicable
  verification: <how-to-verify>
  source: <source-file-or-context>
  last_verified: YYYY-MM-DD
  owners:
    - owner-username
  

• Shipped Behavior Source of Truth: Never treat a doc marked proposed (or otherwise not shipped) as describing shipped behavior. Code and tests are the source of truth.

Git Policy

• Single Branch: main.
• Solo-Developer Workflow: All work lands directly on main — no feature branches, no parallel agent-team branches. A branch is created only when the user explicitly names one.
• Remotes: Remote is git.agentthread.dev. Push/remote registration is deferred until configured (planned: September 2026). Until then: no remotes, no pushes.
• Commit Gate: Commit only after all four quality gates (build, test, format, clippy) pass successfully.

Domain Rules

Project scope and domain-specific rules (including safety boundaries) for the project-agent-manager (pam) repository.

Project Scope

• pam (project-agent-manager): Hierarchical project/task management plus AI task orchestration that drives subscription LLM CLIs (claude/codex/grok/gemini) inside long-lived tmux sessions on the dedicated socket pam (one tmux server per worker host), observable from a web dashboard.
• Multi-host topology (ADR-012): A central orchestrator (pam serve — axum REST + embedded SPA, PostgreSQL as the single source of truth) dispatches runs to per-host workers (pam worker, outbound-only connections, no DB credentials). Single-machine deployment = running serve + worker on one host.
• Worker Host Targets: macOS (primary) and Linux (secondary) — Windows and mobile are out of scope (ADR-011/ADR-012 in docs/003-architecture-decisions.md).
• Tech Stack:
  • Rust 2024 edition
  • tokio (asynchronous runtime)
  • axum 0.8 (web framework)
  • sqlx 0.8 (PostgreSQL library)
  • clap 4.5 (CLI parser)
  • rust-embed (embedding asset in binary)
• Canonical Architecture Index: docs/README.md.

tmux Safety

• Dedicated Socket Constraint: Only ever talk to the dedicated socket: tmux -L pam ....
• Destructive Command Ban: NEVER run kill-server (or any other destructive commands) on the default socket.
• Session Naming: Only sessions whose names match ^pam-[0-9a-f]{8}$ may be managed.
• Protocol Independence: Pane text is display-only — never parse pane output as a protocol.

tmux Executor

The tmux executor runs inside the worker (pam worker) on each execution host. The worker receives dispatch from the central orchestrator over its outbound connection; all tmux mechanics below are local to the worker host. The central orchestrator never invokes tmux.

Session Identity & Naming

The tmux executor uses a dedicated socket pam to isolate all project-agent-manager sessions from the user’s default tmux instance. Every tmux invocation uses tmux -L pam ... to ensure complete operational separation.

Session naming: pam-<short_run_id> where <short_run_id> is the first 8 lowercase hexadecimal characters of the run UUID. This format provides blast-radius isolation while maintaining traceability.

Window titles: Set to the task slug for human-readable identification in tmux session lists.

Rationale for dedicated socket: A separate socket prevents accidental interference with the user’s personal tmux sessions. If the worker crashes or is killed, the operator can still manually inspect orphaned sessions on that host via tmux -L pam list-sessions without affecting the default tmux server.

Name validation: Before ANY tmux invocation, session names are validated against the regex ^pam-[0-9a-f]{8}$. This prevents command injection and ensures we only operate on sessions we created.

Spawn Path

Initial Session Creation

When a run transitions from pending to spawning (the central orchestrator dispatches it to a host), the worker on that host creates a new tmux session:

tmux -L pam new-session -d -s pam-<short_run_id> -c <workdir> -x 220 -y 50

Parameters:

• -d: Detach from session (the worker operates headless)
• -s: Session name with pam- prefix
• -c: Initial working directory from run configuration
• -x 220 -y 50: Initial pane dimensions (220 columns × 50 rows)

Foreground Command

The session’s foreground command is NOT the provider CLI directly. Instead, it runs our own wrapper:

pam exec run <run_id>

This wrapper is responsible for:

1. Reading run configuration: Fetch the run spec from the central orchestrator API (the worker injects PAM_API_URL and the run’s auth context into the wrapper environment; the wrapper never holds PostgreSQL credentials — only the central orchestrator connects to the database). The spec includes provider, arguments, environment variables, and working directory.

2. Provider configuration lookup: Load provider details from the host’s local TOML catalog (binary path, argv template, environment mappings, hooks config) — provider CLIs are installed per host, so the catalog lives on the worker host.

3. Environment normalization and injection:

   • Apply environment mappings from the provider catalog
   • Inject proxy/gateway base URLs (ANTHROPIC_BASE_URL, OPENAI_BASE_URL, etc.) if configured
   • Apply standard HTTP(S)_PROXY variables
   • This injection point ensures traffic stays proxied even during interactive attach

4. CLI process spawning: Execute the provider CLI with normalized arguments and environment.

5. Lifecycle signal reporting:

   • After successful spawn: POST a started signal to the central orchestrator at $PAM_API_URL/api/internal/runs/{id}/signals (the orchestrator journals the corresponding run.started event — the wrapper posts signals, never writes the journal itself)
   • After CLI exit: POST a finished signal with the exit code to the same endpoint
   • Attempt and sequence stamping: the wrapper stamps attempt on every signal and, for progress signals, additionally a monotonic seq — both at emission time. The attempt is read from the injected spawn environment (the worker injects the centrally allocated attempt alongside PAM_API_URL); seq is carried only by repeated kinds (progress) — one-shot kinds (started, finished, failed) carry no seq, being deduplicated centrally by (run_id, attempt, kind)
   • If the central orchestrator is unreachable, the wrapper retries with backoff and writes undeliverable signals to the worker-side spool on disk. Because the wrapper is short-lived (it exits after hold-open), the worker — not the wrapper — owns the spool and replays queued signals once the uplink recovers; signals are not dropped. The spool preserves each signal’s attempt/seq verbatim, and the worker MUST NOT renumber signals at replay time — renumbering at replay would make every replayed signal look newest and fabricate ordering that never happened. Ordering and dedup are enforced centrally, at the signal-ingest edge, never worker-side: the worker never queries the database (see 002-data-model.md, processed_signals)

6. Hold-open behavior: After the CLI exits, the wrapper keeps the tmux session alive for a configurable period (default 5 minutes) to allow final output capture and manual inspection. The session displays a message indicating the run has completed.

Session-to-Run Binding

The wrapper identifies its run ID from the command-line argument and uses this to:

• Read its run configuration from the dispatch payload assembled by the worker (run record fields — provider, cwd, task reference — are passed at spawn time as arguments/environment; the wrapper never talks to PostgreSQL, which lives only on the central host per ADR-012)
• Post lifecycle signals to the correct run record
• Include the run ID in all API calls for correlation

Supervision & Reconciliation

Contrast with grok-fleet-orchestrator

In grok-fleet-orchestrator, the GrokRunner supervises child processes directly:

• The fleet worker spawns grok as a direct child subprocess
• The worker owns the process lifecycle and can restart with backoff
• Process supervision uses tokio::process::Child with wait/kill semantics

In project-agent-manager:

• The tmux SERVER owns the processes, not the worker or the central orchestrator
• The worker only controls session creation/termination via tmux commands
• Process supervision is indirect via liveness queries to tmux

Liveness Monitoring

The worker polls its local tmux for liveness using format strings:

tmux -L pam display-message -p -t pam-<short_run_id> -F "#{session_dead}"
tmux -L pam display-message -p -t pam-<short_run_id> -F "#{pane_dead}"

• #{session_dead}: Returns “1” if the session no longer exists
• #{pane_dead}: Returns “1” if the pane has terminated (CLI exited)

These queries run on the liveness poll loop — a worker-local timer, default every 10 seconds. It is cheap by construction: display-message against the host’s own tmux socket, with no call to the central orchestrator. Its job is fast detection of a dead pane or a vanished session so the exited transition is not delayed by the slower reconciliation cadence.

The liveness poll loop is not the reconciliation loop. They are two separate timers with different costs and different periods:

The periods differ because the costs differ. Polling liveness only at the reconciliation period would triple worst-case detection latency for an exited CLI; running a full reconciliation at the liveness period would triple central fetch load for no added signal.

Reconciliation Loop

Trigger conditions:

• On worker startup (before registering with the central orchestrator)
• Periodic timer (default every 30 seconds — the reconciliation cadence, distinct from the 10-second liveness poll above)
• Manual reconciliation command

Reconciliation logic (the worker reconciles its host; the central orchestrator reconciles the fleet — runs whose host is offline):

1. Query desired state: Fetch this host’s runs with status running or spawning from the central orchestrator. For partition tolerance the worker maintains a local run cache persisted on disk (in its data dir, alongside the signal spool) — refreshed on every successful fetch, so desired state survives both a partition and a worker restart happening at the same time.

2. Query actual state: Execute tmux -L pam list-sessions -F "#{session_name}" and filter for names matching ^pam-[0-9a-f]{8}$.

3. Compare and reconcile (the local run cache is the comparison baseline; the worker itself never queries the database):

   • Orphan detection: tmux sessions not present in the local run cache → mark as orphaned, queue for cleanup
   • Missing sessions: Desired runs (centrally recorded, cached locally) without tmux sessions → central transitions the run to failed (if spawning) or to failed with failure_reason = session_lost (if running); where the restart policy applies, central then decides whether to requeue (failed → pending, incrementing attempt) or leave the run terminal
   • Zombie processes: Sessions with pane_dead=1 but run status still running → transition to exited based on the wrapper’s finished signal (which may arrive via spool replay) or timeout
   • Terminal-but-alive sessions: A session whose run record is terminal (exited/failed/stopped) or no longer desired, while the tmux session still lives — produced by stale signals from a previous incarnation, a stop that reached the database but never reached the worker, or manual recovery after a tmux server crash. The session’s 8-hex name prefix maps it back to its run, so this case is distinguishable from true orphans without widening the desired-state fetch. Cleanup routes through safety rule 3’s two-part verification, with the action chosen by the discrimination table below

Terminal-but-alive discrimination. “Terminal run, live session” has two distinct meanings, separated by the incarnation epoch (runs.attempt):

The split matters: killing every terminal-but-alive session indiscriminately would turn a long worker outage into an automatic slaughter of healthy CLI sessions the moment the worker returns.

Central Fleet Reconciliation

The loop above is host-local: each worker reconciles its own host against its local run cache. The central orchestrator separately reconciles the fleet — state no single worker can see. Central-side cases:

1. Host offline (partition). A host whose heartbeat is lost is marked offline; its runs are left untouched at partition time and reconcile on reconnect. The worker’s host-local reconciliation does NOT force-fail runs — any force-fail is the separate central action in case 3 below, never a worker-side behavior.

2. Host-online gating. Central-side inferences about a run are valid only while its host is online (heartbeat current). From the central seat, “no output, no signals” from a live host means the CLI hung; from an offline host it is observationally ambiguous — a partition and a hang are indistinguishable. Output-silence timeouts (the CLI-hang hard timeout, default 30 minutes) are therefore enforced centrally and fire only while the heartbeat is current; while a host is offline, the host-lost bound below governs instead.

3. Host-lost bound (T_dead). A host continuously offline for T_dead (measured from hosts.last_seen_at) is presumed lost: central force-fails that host’s non-terminal runs with failure_reason = host_lost. Confirmed default: 24 hours, configurable (OQ-12, 071-risks-and-open-questions.md). The generous bound is intentional: worker-down ≠ CLI-down — the tmux server owns the processes, not the worker, and surviving sessions are re-adopted on worker restart, so a short bound would convert routine worker downtime into run loss. These host_lost tombstones are requeue-eligible — see § Restart Policy above for the requeue-eligibility statement and its accepted trade-off.

4. Late signals after a host_lost tombstone. A replayed finished(attempt = N) arriving after the tombstone is journaled but does not resurrect the run — the tombstone wins. The operator sees the event and can act; automated resurrection is a possible later enhancement, not a v1 need.

Disabled Host — Worker-Side Behavior

Host disable is an operator exclusion from dispatch, not a shutdown. The API surface and the sticky/derived status semantics are owned by 030-http-api.md (Hosts registry — disable is drain-only); this section states only what the worker on a disabled host does.

A worker whose host is disabled behaves exactly as before, minus new work:

Drain completion is observed, not signalled: the operator watches the host’s active-run count fall to zero (GET /hosts/{id}). Nothing in the worker forces that count down — a disabled host with a long-running CLI stays busy until that CLI finishes on its own or an operator stops the run explicitly.

Restart Policy

This section owns restart policy only — trigger conditions, attempt caps, and backoff. The run status machine itself (the legal edge set, including the canonical restart-requeue edge failed → pending, which increments runs.attempt centrally) is canonical in 002-data-model.md; this document conforms to it and introduces no edges of its own.

Requeue-eligible trigger set (OQ-12, confirmed). All terminal failure_reason values are requeue-eligible, host_lost included — a run tombstoned because its host went dark for T_dead (24h default, § Central Fleet Reconciliation below) is requeued to another host exactly like any other failed run. This closes a documentation gap the restart policy previously left silent (this section named itself as owning “trigger conditions” without ever stating whether host_lost was in scope). The trade-off is accepted, not eliminated: combined with the Terminal-but-alive discrimination rule above — which deliberately does NOT auto-kill an epoch-matching live session found after a host_lost tombstone — a requeue can leave the original host still doing real (or idle) work under the old incarnation while a new incarnation runs elsewhere. That old session is invisible to any automatic signal path (its late signals are discarded as stale-epoch, and it is flagged in the UI rather than killed) until an operator notices the flag and cleans it up manually. Blocking the requeue instead would trade this operator-visible orphan risk for the opposite failure — every run on a host that goes dark stays stuck running/spawning forever with no terminal exit — which the working default already rejected as worse (see OQ-12 in 071-risks-and-open-questions.md).

Restart attempts are capped and follow exponential backoff:

• Maximum attempts: 3 (configurable per task)
• Initial delay: 10 seconds
• Backoff multiplier: 2×
• Maximum delay: 60 seconds

Graceful stop escalation: SIGTERM to pane’s process group → wait up to 10 seconds → SIGKILL to pane process (if still running) → then kill-session for session cleanup.

Kill-before-respawn: before creating a session for a requeued (respawned) run, apply the graceful-stop ladder above to any tmux session already holding the target session name, then proceed with new-session. The session name derives from the run UUID, so an existing session with that name is necessarily a previous incarnation of the same run — safety rule 3’s two-part verification passes by construction. This is mandatory, not polish: cumulative restart backoff totals roughly 130 s even in the worst case, while the previous incarnation’s session survives far longer (wrapper hold-open defaults to 5 minutes after CLI exit; unmapped sessions are held for inspection for 1 hour), so the previous session still exists at every respawn point and the name collides deterministically — without cleanup-before-respawn, every attempt inside the window fails on new-session and the attempt budget is exhausted without a single spawn.

Orphan Handling

Orphan definition: An orphan is a session on socket pam named pam-* that cannot be mapped to a desired run by reconciliation. After worker restart, the run record usually still exists centrally (and in the worker’s local cache) and is recovered; only unmappable sessions are orphans and get cleaned.

Orphaned sessions are:

1. Logged to the events table with run.orphan_detected topic
2. Held for inspection for a configurable period (default 1 hour)
3. Cleaned up via tmux -L pam kill-session -t <session>

Observation

Capture-pane Polling

Terminal output is captured using tmux capture-pane:

tmux -L pam capture-pane -p -e -t pam-<short_run_id>

Parameters:

• -p: Output to stdout (pipe-readable)
• -e: Include escape sequences for ANSI formatting
• -t: Target session

Polling Cadence and Tradeoffs

Default cadence: Once per second (1 Hz) for active sessions. Note: This is the source rate for snapshots (emitted only on content change), while 10 fps is the coalescing cap for the interactive attach WebSocket relay layer—these two limits apply to different layers.

Tradeoffs:

• Higher frequency (2-4 Hz): More responsive UI, higher CPU load, more data transfer
• Lower frequency (0.2-0.5 Hz): Better resource efficiency, laggy terminal feel
• Adaptive: Throttle when no recent activity detected (idle threshold: 30 seconds)

ANSI Handling

The -e flag preserves ANSI escape sequences for colors and formatting. These are:

1. Stored verbatim in the worker’s snapshot cache and uplinked unchanged (the central cache holds the same verbatim bytes)
2. Served by the central orchestrator to WebSocket/SSE clients with appropriate content-type
3. Rendered by the web dashboard using xterm.js or similar terminal emulator

Snapshot Cache (two tiers)

Snapshot caching exists at two points in the pipeline, with different owners:

Worker-side cache (this document’s pipeline):

• Structure: In-memory cache storing the latest pane text per run, used for change detection — each poll compares against the cached snapshot and uplinks a new snapshot only when the content changed.
• Bounds: Maximum 1000 active snapshots, LRU eviction when exceeded.
• Purpose: Keeps the 1 Hz poll off the uplink — tmux is read locally at 1 Hz regardless, but the worker→orchestrator stream carries only changed snapshots.

Central snapshot cache (see 002-data-model.md retention policy and 030-http-api.md pane endpoints):

• The orchestrator holds the latest snapshot per run (received over the uplink) and serves GET /runs/{id}/pane, the read-only /pane/ws fanout, and reconnecting clients from it. Client fanout is a central concern — workers never serve clients directly.

Full snapshot cost (~220×50 ≈ 11,000 chars) is justified by eliminating incremental update complexity.

Cache entry lifecycle (both tiers):

• Created on first capture for a run
• Updated on each poll / each uplink receipt (replace, not append)
• Evicted on LRU when cache full or run completes
• Invalidated on run deletion

Disk persistence: Each run maintains an on-disk log at <platform data dir>/runs/<run_id>.log (~/Library/Application Support/pam/runs/ on macOS, ~/.local/share/pam/runs/ on Linux) with raw pane text appended on each poll. This provides audit trails and debugging material.

Streaming to SSE/WS

Two WebSocket endpoints serve terminal data:

1. Read-only viewer: /api/v1/runs/{id}/pane/ws

   • Multiplexes from the snapshot cache
   • Multiple concurrent viewers allowed
   • Receives full snapshots with sequence numbers (sent only when pane content changes, detected via hash compare)
   • No input channel (display-only)

2. Interactive attach: /api/v1/runs/{id}/attach/ws

   • Single-writer lease enforced
   • Bidirectional byte stream (PTY relay)
   • See “Interactive Attach” section below

Critical Design Constraint

Pane text is display-only, never parsed as a completion protocol. This is a hard lesson from grok-fleet-orchestrator’s ACP (Agent Completion Protocol): never treat a wire/screen format you don’t own as a contract. Completion signals must come from provider hooks or explicit user marks, not screen scraping.

Interactive Attach

Single-writer Lease

Interactive attach uses a lease system to prevent multiple writers from racing:

Lease acquisition:

• Requested via WebSocket upgrade to /api/v1/runs/{id}/attach/ws
• Lease holder stored in-memory with TTL (default 5 minutes)
• Lease refreshed on activity (heartbeat every 30 seconds)

Lease conflicts:

• Second attach attempt is rejected after WebSocket upgrade with error frame {"type":"error","code":"lease_held","holder":<current_holder_info>}
• UI can prompt to “force release” (requires confirmation and shows current holder)

Lease release:

• Explicit release on client disconnect
• TTL expiration on inactivity
• Manual revoke via API or UI

PTY Relay

The attach flow spawns a PTY relay process:

1. The worker spawns a subprocess running tmux -L pam attach -t pam-<short_run_id>
2. This subprocess uses portable-pty to create a PTY
3. The tmux attach process stdout/stdin are bridged to the PTY
4. The PTY byte stream is proxied bidirectionally over the WebSocket

Implementation: Uses portable-pty crate for cross-platform PTY spawning, with the subprocess wrapped in a tokio::process::Child for cleanup on WebSocket close.

Input Safety

Programmatic input via tmux send-keys is restricted to an allowlist for safety:

Allowlisted keys:

• Control sequences: Ctrl+C, Ctrl+D (SIGINT/EOF)
• Enter, Backspace, Delete
• Printable ASCII (0x20-0x7E)

Restricted:

• Shell metacharacters that could execute commands: |, &, ;, $, backticks
• Escape sequences that could reconfigure terminals
• tmux command sequences (e.g., Ctrl+B prefix)

Interactive attach: When a human holds the lease via the PTY relay, input is unrestricted (full keyboard and terminal control). The allowlist only applies to programmatic send-keys operations.

Read-only Viewer Multiplexing

Multiple read-only viewers can attach simultaneously:

• All receive the same pane text stream from the snapshot cache
• No coordination needed beyond cache invalidation on updates
• Each viewer tracks its own “last seen offset” for incremental updates

Safety Rules

The 4 Safety Rules

1. Never run kill-server on the default socket

   • The user’s personal tmux must never be affected
   • Only use tmux -L pam kill-server for catastrophic cleanup
   • All normal operations use session-specific commands

2. Validate session names before ANY tmux invocation

   • Regex: ^pam-[0-9a-f]{8}$
   • Prevents command injection via user input
   • Applied to all -t arguments and session list parsing

3. Cleanup only sessions we created

   • Two-part verification: a) Session exists in our database registry (runs table) b) Session name matches our prefix (pam-)
   • Orphan cleanup still respects prefix constraint
   • Never target user sessions, even if they somehow got on our socket

4. Restrict send-keys to allowlisted set

   • Programmatic input limited to safe keys
   • Prevents remote code execution via terminal injection
   • Interactive attach (human with lease) is unrestricted

Explicit NEVER List

• NEVER run tmux kill-server without the -L pam flag
• NEVER use glob patterns in session targeting (e.g., tmux kill-session -t "pam-*")
• NEVER target sessions based on user-provided paths or workdirs
• NEVER pass unvalidated user input to any -t flag
• NEVER send arbitrary shell commands via send-keys
• NEVER parse pane output as structured completion data
• NEVER access sessions on the default socket (no -L flag)

Failure Modes

Diagrams

stateDiagram-v2
    %% Rendering rule — two planes are overlaid in this one diagram:
    %%   Run-status (DB) states: pending, spawning, running, exited, failed, stopped.
    %%   Edges between them conform to docs/002-data-model.md, which is canon for
    %%   the run status machine.
    %%   tmux-session-plane states: hold_open, orphaned. These describe the tmux
    %%   session itself, carry no runs.status value of their own, and their edges
    %%   are session actions, not DB status transitions.

    [*] --> pending: run created (central assigns host)

    pending --> spawning: worker claims run, spawns session
    spawning --> spawning: session name held by prior incarnation — graceful-stop it, retry
    spawning --> running: wrapper posts run.started event

    running --> running: capture-pane polling continues
    running --> hold_open: CLI exits, wrapper posts run.finished
    running --> failed: wrapper exits unexpectedly
    running --> stopped: user/manual stop
    running --> running: central partition — no state change, signals spool

    hold_open --> exited: hold-open timer expires
    hold_open --> exited: user closes session

    exited --> [*]: session cleanup
    failed --> [*]: session cleanup
    stopped --> [*]: kill-session

    failed --> pending: restart-policy requeue (attempt + 1 at re-dispatch)

    running --> orphaned: worker crashes, session persists
    orphaned --> running: worker restart, session re-adopted (no runs.status change)
    orphaned --> [*]: unmapped after inspection hold — journal-only cleanup

    spawning --> failed: spawn fails (CLI not found, permission error)
    running --> failed: tmux server crash on host
    running --> stopped: force stop via API

    pending --> failed: host presumed lost (T_dead=24h exceeded, host_lost)
    spawning --> failed: host presumed lost (T_dead=24h exceeded, host_lost)
    running --> failed: host presumed lost (T_dead=24h exceeded, host_lost)

    note right of pending
        PostgreSQL: run.status = 'pending'
        host_id assigned by dispatcher
        Entered by dispatch or by restart-policy requeue (from failed)
        No tmux session exists
    end note

    note right of spawning
        PostgreSQL: run.status = 'spawning'
        tmux session created but no run.started event yet
    end note

    note right of running
        PostgreSQL: run.status = 'running'
        tmux session active on worker host, capture-pane polling 1 Hz
    end note

    note right of hold_open
        tmux-session-plane state: the run row is already terminal
        PostgreSQL: run.status = 'exited' or 'failed'
        CLI exited, wrapper holds session open for inspection
        Default 5 minutes, then cleanup
    end note

    note right of orphaned
        tmux-session-plane state — no runs.status value
        Session on socket pam (pam-*) that reconciliation cannot map
        to any desired run; no run row transitions on its account
        Cleanup is journal-only: run.orphan_detected event,
        inspection hold, then kill-session
    end note

    note right of failed
        PostgreSQL: run.status = 'failed'
        Terminal unless the restart policy requeues it
        (failed -> pending while attempts remain; 002 is canon)
        Session cleaned up via tmux -L pam kill-session

        host_lost (OQ-12, confirmed 2026-09-01): a host continuously
        offline for T_dead (default 24h, measured from hosts.last_seen_at)
        is presumed lost; central force-fails its non-terminal runs.
        Requeue-eligible like any other failed run — accepted trade-off:
        the original host's live session (if any) can go undetected
        until an operator notices and cleans it up. See
        010-tmux-executor.md § Restart Policy / § Host-lost bound.
    end note

    note right of stopped
        PostgreSQL: run.status = 'stopped'
        Graceful shutdown via kill-session
    end note

sequenceDiagram
    participant C as central orchestrator<br/>(pam serve)
    participant W as worker agent<br/>(pam worker)
    participant T as tmux -L pam<br/>(worker host)
    participant V as read-only viewers<br/>/pane/ws via central
    participant A as interactive attach<br/>/attach/ws via central
    participant P as provider CLI
    participant H as provider hook<br/>completion signal

    Note over W,T: Capture-pane polling loop (1 Hz, on worker host)
    loop every 1 second
        W->>T: capture-pane -p -e -t pam-<id>
        T-->>W: ANSI pane text
        W->>C: uplink snapshot for run_id (if hash changed)
        C->>V: WS message with full snapshot (sent only on change)
    end

    Note over C,V: Multiple viewers multiplexed from latest snapshot
    V->>C: read latest snapshot
    C-->>V: cached pane text
    V->>V: render in xterm.js terminal

    Note over A,C: Interactive attach flow (relayed through central)
    A->>C: WS upgrade request to /attach/ws
    C->>C: check single-writer lease availability
    C-->>A: 101 Switching Protocols (lease granted)
    C->>W: attach request over worker uplink

    W->>W: spawn PTY relay
    Note over W,T: portable-pty + tmux attach
    W->>T: tmux -L pam attach -t pam-<id>
    T-->>W: bidirectional PTY stream

    A->>C: WS binary message (user input)
    C->>W: relay input
    W->>T: write to PTY stdin
    T->>P: forward to CLI stdin
    P-->>T: CLI stdout
    T-->>W: PTY stdout
    W-->>C: relay output
    C-->>A: WS binary message (terminal output)

    Note over A,C: Lease heartbeat every 30s
    A->>C: WS ping/pong
    C->>C: refresh lease TTL

    Note over A,C: Detach or timeout
    A->>C: WS close or lease timeout
    C->>W: stop attach relay
    W->>T: kill PTY relay
    C->>C: release lease

    Note over H,C: Completion signal flow (primary signal)
    P->>P: CLI completes work
    P->>H: native completion hook fires
    H->>C: POST /api/internal/runs/{id}/signals
    C->>C: emit run.signal_received{kind=completed}
    C->>C: work_item → done (config-gated)
    C->>V: broadcast work_item.status_changed

    Note over W,C: Fallback — wrapper posts event
    P->>P: CLI exits (no native hook)
    Note over W: pam exec run wrapper observes exit
    W->>C: POST /api/internal/runs/{id}/signals<br/>run.finished{exit_code}
    Note over W,C: On partition: spool locally, replay after reconnect
    C->>C: transition to hold-open state
    W->>C: continue snapshot uplink for inspection period

flowchart LR
    subgraph Worker["pam worker (execution host)"]
        Cache[("Worker snapshot cache<br/>change detection")]
        Poll["tmux capture-pane<br/>1 Hz poll"]
    end

    subgraph Tmux["tmux -L pam"]
        Session["pam-<short_run_id>"]
        Pane["tmux pane"]
        CLI["Provider CLI"]
        Wrapper["pam exec run"]
    end

    subgraph CentralG["central orchestrator"]
        Central["serve: API + fanout"]
        CCache[("Central snapshot cache<br/>client serving")]
        DB[(PostgreSQL)]
    end

    subgraph Clients["Web Clients"]
        Viewer["Read-only viewer<br/>/pane/ws"]
        Attach["Interactive attach<br/>/attach/ws"]
    end

    Poll -->|"capture-pane -p -e"| Pane
    Pane -->|"ANSI text"| Cache
    Cache -->|"changed snapshots<br/>uplink only on change"| CCache
    Central -->|"SSE/WS stream"| Viewer
    Central -->|"WS relay"| Attach
    CCache --- Central
    Central --- DB

    Attach -->|"attach relay<br/>through central"| Session
    Session -->|"bidirectional"| Pane
    Pane -->|"stdout/stderr"| CLI

    CLI -->|"exit code"| Wrapper
    Wrapper -->|"finished signal<br/>(spooled on partition)"| Central

    style DB fill:#f5f5f7,stroke:#999
    style Cache fill:#f5f5f7,stroke:#999
    style CCache fill:#f5f5f7,stroke:#999

Cross-References

• Provider Adapters: Completion signal ladder, provider catalog, proxy configuration
• HTTP API Contract: WebSocket protocol, event schema
• Data Model: runs and events table definitions

Provider Adapters

ProviderAdapter Capability Model

Each provider adapter declares its capabilities through a structured interface that determines how the pam exec run wrapper interacts with the provider CLI.

Capability Declaration

Binary resolution: bin field specifies the executable name. Resolved via which <bin> at runtime. If not found, provider is marked unavailable (not fatal error).

Argument template: argv_template is a list of strings with substitution placeholders:

• {args}: User-provided command-line arguments
• {prompt}: Single prompt string (for completion-style APIs)
• {api_key}: API key from environment or secure storage
• {model}: Model identifier (e.g., claude-3-5-sonnet-20241022)

Example: ["agent", "serve", "--bind", "127.0.0.1:0", "--secret", "{api_key}"]

Environment normalization: env_map defines how provider-specific environment variables are normalized. Includes proxy base URLs, API keys, and provider-specific settings.

TUI characteristics: Describes terminal behavior for capture interpretation:

• line_oriented: Output is line-buffered (e.g., many CLI tools)
• full_screen: Uses ncurses/termcap (e.g., interactive TUI)
• raw_mode: Direct character I/O (e.g., password prompts)

This affects how pane text is rendered and whether we apply special handling for escape sequences.

Completion signals: Declares which completion mechanisms the adapter supports:

• native_hook: Provider has built-in webhooks or callback mechanism
• exit_code: CLI exit code indicates success/failure
• idle_detection: No activity for threshold indicates completion (unreliable, hint only)

Hooks config generation: Template for generating provider-specific hook configuration files (e.g., .anthropic/hooks.json for Claude, .gemini/config.toml for Gemini CLI).

Capability Interface

#![allow(unused)]
fn main() {
pub struct ProviderAdapter {
    pub id: String,
    pub bin: String,
    pub argv_template: Vec<String>,
    pub env_map: HashMap<String, String>,  // pam var → provider env var
    pub tui_type: TuiType,
    pub completion_signals: Vec<CompletionSignal>,
    pub hooks_config_template: Option<String>,
}

pub enum TuiType {
    LineOriented,
    FullScreen,
    RawMode,
}

pub enum CompletionSignal {
    NativeHook,
    ExitCode,
    IdleDetection,
}
}

Completion-Signal Ladder

Completion detection uses a trust ladder with multiple rungs. Each rung provides increasing confidence that a task has truly completed.

The 4 Rungs

Signal Trust Hierarchy

1. Provider-native hooks (PRIMARY for task-level completion)

• Provider’s own completion mechanism posts to $PAM_API_URL/api/internal/runs/{id}/signals — the central orchestrator’s URL, injected by the wrapper at spawn time (the hook runs on the worker host but reports to the hub)
• Indicates the provider confirms task completion from its perspective
• Emits run.signal_received event; no run status change
• Optionally transitions work_item to done (config-gated, default off)
• Most reliable signal when available

2. Wrapper journal (authoritative for session-level exit)

• The pam exec run wrapper posts run.finished event with exit code
• Indicates the CLI process exited, but not whether it completed its work
• Exit code 0 → run transitions to exited
• Exit code non-zero → run transitions to failed

3. Manual mark (lowest trust, highest override authority)

• User explicitly marks task as done via UI or CLI
• User provides confidence level: “certain”, “likely”, “unsure”
• Transitions work_item to done (not run status)
• Always overrides other signals

4. Pane-idle heuristic (UI hint ONLY, never transitions status)

• No output in pane for threshold time (default 10 minutes)
• Used only for UI display: “appears idle, may be complete”
• Never used for status transitions (too many false positives)

Normalization into RunSignal Enum

All completion signals normalize into a single RunSignal enum:

#![allow(unused)]
fn main() {
pub enum RunSignal {
    Started { attempt: u32, timestamp: DateTime<Utc> },
    Finished { attempt: u32, exit_code: i32, timestamp: DateTime<Utc> },
    Completed { attempt: u32, result: TaskResult, timestamp: DateTime<Utc> },
    Failed { attempt: u32, error: String, timestamp: DateTime<Utc> },
    Progress { attempt: u32, seq: i64, message: String, percent: Option<u8> },
}

pub enum TaskResult {
    Success,
    PartialSuccess { reason: String },
    UserRejected,
}
}

Signal sources post to /api/internal/runs/{id}/signals endpoint (worker-token authenticated — callers live on worker hosts):

• Wrapper posts Started and Finished
• Provider hooks post Completed, Failed, or Progress

Attempt and sequence fields (wire contract: 030-http-api.md):

• Every variant carries attempt — the run’s incarnation epoch (runs.attempt). It is allocated centrally at dispatch time (incremented on each restart-policy requeue), injected into the spawn environment, and stamped by the emitting wrapper at emission time; replayed spool entries keep their original numbers — the worker never renumbers at replay.
• Progress additionally carries seq (int64, monotonic within (run_id, attempt)): it is the one repeated, dedup-exempt kind. The one-shot kinds (Started, Finished, Completed, Failed) carry no seq — their exactly-once semantics come from the central processed_signals dedup table instead.

Manual signals do not use this endpoint: user-initiated actions (run stop, mark-done) arrive over the public API with the user bearer token (e.g., POST /runs/{id}/stop); the orchestrator normalizes them internally into signals with source: "manual". Users never hold worker tokens.

Signal → Event Topic Mapping

Each accepted signal journals exactly one event topic. The topic vocabulary itself is owned by 002-data-model.md §Event Topics — this table only states which signal produces which topic.

run.finished and run.failed do not overlap (normative). run.finished means a wrapper exited — the process ran and reported its exit code, whatever that code was. run.failed is reserved for the two cases where no wrapper exit exists to report: spawn failure, and a provider hook reporting Failed. One rule, no ambiguity: a consumer counting terminal events never sees both topics for the same (run_id, attempt).

Status Transition Rules

Note (ingest guard — single source of truth): this table describes the status transitions themselves. Every signal reaches it only after passing the central ingest guard at the signal-ingest edge — epoch check, dedup, and (attempt, seq) ordering, strictly upstream of both the status write and the event journal / SSE emission. The guard’s normative specification lives in 030-http-api.md (Internal Endpoints → Provider Hook Receiver); a signal dropped there produces no state change and no event.

Note: Work item completion (transition to done) is managed separately via work_item.status_changed events and configuration-gated transitions.

Work Item Auto-Transition Configuration

When a provider hook emits Completed signal, the central orchestrator may optionally auto-transition the associated work item to done status (not the run status—run status stays unchanged with run.signal_received emitted only).

Configuration: auto_mark_done: bool (default: false)

• false: Require manual mark for work_item → done transition
• true: Auto-transition work_item to done when Completed signal received

This configuration is gated because provider hooks are task-level signals while run lifecycle is session-level.

Consequence of the default (deliberate, not a gap): with auto_mark_done: false, a run reaching a terminal state (exited, failed, stopped) does not move its work item — run status and work-item status are decoupled by default. A task typically remains in todo even after its run completes (nothing sets doing on run start either); work-item status is human-owned and moves only by manual action (pam task done, board drag, PATCH status).

Why Pane Scraping is Banned

Screen scraping the tmux pane output to detect completion is explicitly prohibited:

Lesson from grok-fleet-orchestrator ACP: Never treat a wire/screen format you don’t own as a contract. Terminal output is designed for humans, not machines. It can change format, include escape sequences, wrap unexpectedly, or include transient messages that look like completion but aren’t.

False positives: A CLI might output “Done!” as a progress indicator, not final completion. Or output might include “Error:” but then continue successfully.

Fragility: Provider CLI updates can change output format without notice, breaking scrapers.

Correct approach: Use provider-native hooks when available (the completion-signal ladder’s strongest rung). Fall back to manual confirmation, never screen scraping.

Provider Catalog

The provider catalog is a TOML configuration file that defines all available providers. This is NOT stored in the database; it’s runtime configuration that can be edited without database migrations.

Catalog Location

• Default path: <platform data dir>/providers.toml (~/Library/Application Support/pam/providers.toml on macOS, ~/.local/share/pam/providers.toml on Linux)
• Path configurable via catalog_path in the main config file (see 040-cli.md); the main config file location itself is overridable via PAM_CONFIG
• Hot-reload on file change (the worker watches for modifications on its host)

TOML Format Example

# Provider catalog for project-agent-manager
# Each provider defines how pam interacts with a specific AI provider CLI

[providers.claude]
id = "claude"
name = "Anthropic Claude"
bin = "claude"
argv_template = [
    "agent", "serve",
    "--bind", "127.0.0.1:0",
    "--secret", "{api_key}"
]
env_map = """
ANTHROPIC_API_KEY=ANTHROPIC_API_KEY
ANTHROPIC_BASE_URL=ANTHROPIC_BASE_URL
HTTPS_PROXY=HTTPS_PROXY
HTTP_PROXY=HTTP_PROXY
"""
tui_type = "full_screen"
completion_signals = ["native_hook", "exit_code"]
hooks_config_template = """
{
  "post_completion_url": "{pam_api_url}/api/internal/runs/{run_id}/signals"
}
"""
installed_detection = ["--version"]

[providers.codex]
id = "codex"
name = "OpenAI Codex"
bin = "codex"
argv_template = [
    "agent", "run",
    "--model", "{model}",
    "--api-key", "{api_key}",
    "{args}"
]
env_map = """
OPENAI_API_KEY=OPENAI_API_KEY
OPENAI_BASE_URL=OPENAI_BASE_URL
HTTPS_PROXY=HTTPS_PROXY
HTTP_PROXY=HTTP_PROXY
"""
tui_type = "line_oriented"
completion_signals = ["exit_code"]
hooks_config_template = null
installed_detection = ["--version"]

[providers.grok]
id = "grok"
name = "xAI Grok"
bin = "grok"
argv_template = [
    "agent", "serve",
    "--bind", "127.0.0.1:0",
    "--secret", "{api_key}"
]
env_map = """
GROK_API_KEY=GROK_API_KEY
OPENAI_BASE_URL=OPENAI_BASE_URL
ANTIGRAVITY_BASE_URL=ANTIGRAVITY_BASE_URL
ANTHROPIC_BASE_URL=ANTHROPIC_BASE_URL
FLEET_LLM_GATEWAY_URL=FLEET_LLM_GATEWAY_URL
HTTPS_PROXY=HTTPS_PROXY
HTTP_PROXY=HTTP_PROXY
"""
tui_type = "full_screen"
completion_signals = ["native_hook", "exit_code"]
hooks_config_template = """
{
  "completion_webhook": "{pam_api_url}/api/internal/runs/{run_id}/signals"
}
"""
installed_detection = ["--version"]

[providers.gemini]
id = "gemini"
name = "Google Gemini"
bin = "gemini-cli"
argv_template = [
    "chat",
    "--model", "{model}",
    "--api-key", "{api_key}",
    "{args}"
]
env_map = """
GEMINI_API_KEY=GEMINI_API_KEY
GEMINI_BASE_URL=GEMINI_BASE_URL
GOOGLE_API_KEY=GOOGLE_API_KEY
HTTPS_PROXY=HTTPS_PROXY
HTTP_PROXY=HTTP_PROXY
"""
tui_type = "line_oriented"
completion_signals = ["exit_code"]
hooks_config_template = null
installed_detection = ["--version"]
# Optional: canonical Phase 4 scope is claude/codex/grok; gemini included only if installed

Runtime Presence Probing

pam provider doctor command checks provider availability:

# Check all providers
pam provider doctor

# Check specific provider
pam provider doctor claude

Detection method:

1. Binary resolution: which <bin>
2. Version check: <bin> <installed_detection_args> (e.g., claude --version)
3. Exit code 0 → provider available
4. Exit code non-zero or binary not found → provider unavailable

Degradation behavior: If a provider is unavailable:

• Tasks configured for that provider fail to spawn
• UI shows “Provider not available: run pam provider doctor”
• The host continues serving other providers
• No crash, just degraded functionality for that provider

Catalog Hot-Reload

The worker watches the provider catalog file for modifications on its host:

• On change: Reload catalog, validate TOML, update in-memory provider registry
• Existing runs: Continue with old provider config (no mid-run config changes)
• New runs: Use updated provider config
• Errors: Log validation errors, keep old config (don’t break on typo)

Proxy/Gateway Configuration

Proxy and gateway configuration allows provider CLIs to route traffic through a centrally-shared gateway (e.g., a single liteLLM instance reachable over the network, not a per-host local process — ADR-013) or corporate HTTP proxies.

Per-Provider Environment Injection

The pam exec run wrapper injects proxy-related environment variables BEFORE spawning the provider CLI. This ensures:

• All provider CLI traffic goes through the configured gateway
• Proxy settings persist even during interactive attach (user manually attaching to tmux session)
• No need to configure proxy in multiple places

Injection point: Wrapper step (c) in the spawn path, after reading run config and provider catalog, before CLI spawn.

Environment Variable Mappings

Based on grok-fleet-orchestrator’s apply_llm_proxy_envs pattern, adapted for our providers:

Standard HTTP proxy variables (all providers):

• HTTPS_PROXY: HTTPS proxy URL
• HTTP_PROXY: HTTP proxy URL
• NO_PROXY: Comma-separated bypass list

Uncertain variables (verify at implementation):

• GROK_API_KEY: May use different env var name (check docs)
• ANTIGRAVITY_BASE_URL: Verify if grok CLI uses this or different var
• GOOGLE_API_KEY: Check if gemini-cli uses this or just GEMINI_API_KEY

Gateway URL Format

When a gateway is configured (e.g., a shared liteLLM instance at a network-reachable address, not localhost), the wrapper normalizes base URLs:

# For OpenAI-compatible gateways (codex, grok)
OPENAI_BASE_URL=https://gateway.internal:8080/v1

# For Anthropic (claude)
ANTHROPIC_BASE_URL=https://gateway.internal:8080/v1

# For Gemini (gemini)
GEMINI_BASE_URL=https://gateway.internal:8080

The apply_llm_proxy_envs function from grok-fleet-orchestrator shows the pattern:

#![allow(unused)]
fn main() {
let url_trimmed = url.trim_end_matches('/');
cmd.env("OPENAI_BASE_URL", format!("{url_trimmed}/v1"));
cmd.env("ANTHROPIC_BASE_URL", url_trimmed);
cmd.env("GEMINI_BASE_URL", url_trimmed);
}

Configuration Source

Gateway configuration is NOT in the provider catalog (too user-specific). Instead:

Priority order (highest to lowest):

1. User config file: <platform data dir>/config.toml (macOS: ~/Library/Application Support/pam/; Linux: ~/.local/share/pam/)

   [gateway]
   url = "https://gateway.internal:8080"
   api_key = "sk-gateway-key"
   

2. Environment variables: PAM_GATEWAY_URL, PAM_GATEWAY_API_KEY
3. Provider-specific env vars set by user: ANTHROPIC_BASE_URL, etc.

Core CLI settings (PAM_API_URL, PAM_TOKEN, etc.) now follow the same config-file-first rule — see 060-configuration.md § Precedence Order (resolves OQ-16, 071-risks-and-open-questions.md).

Gateway Operations Scope

IN SCOPE for pam:

• Injecting proxy/gateway base URLs into provider CLI environment
• Passing through HTTP(S)_PROXY variables
• Ensuring all CLI traffic respects proxy settings

OUT OF SCOPE:

• Managing the gateway lifecycle (liteLLM, custom gateway start/stop)
• Gateway health checks, load balancing, or failover
• Gateway authentication beyond passing through API keys
• Multi-gateway routing or A/B testing

pam’s role is to point CLIs at a gateway, not to operate the gateway itself.

Interactive Attach Considerations

Important: Because proxy injection happens in the pam exec run wrapper BEFORE the CLI spawn, the settings persist even when a user attaches interactively to the tmux session.

• User attaches: tmux -L pam attach -t pam-<short_run_id>
• Provider CLI is already running with proxy env vars set
• All interactive CLI commands continue using the gateway
• No need for user to configure proxy in their shell

This is a key advantage over post-attach proxy configuration.

Per-Provider Capability Matrix

Legend:

• ✅ Confirmed from grok-fleet-orchestrator code or common knowledge
• ⚠️ Plausible but uncertain (verify at implementation)
• ❌ Known limitation or missing capability

Verification needed at implementation:

• Grok’s exact base URL env var (may be different from ANTIGRAVITY_BASE_URL)
• Gemini CLI’s existence and configuration format
• Codex’s webhook/hook support (if any)

Cross-References

• tmux Executor: Spawn path, environment injection point, observation
• Data Model: providers table vs catalog file decision
• HTTP API Contract: /api/internal/runs/{id}/signals endpoint schema
• Prior art: apply_llm_proxy_envs pattern from the grok-fleet-orchestrator sibling repository (external reference, not part of this repo)

HTTP API Contract

Authentication & Transport

Bind Configuration

• Bind address: PAM_BIND env var (default: 0.0.0.0) — the central orchestrator is reachable from worker hosts and remote browsers (ADR-012)
• Port: PAM_PORT env var (default: 7898)
• TLS: served via TLS in deployment (termination strategy — reverse proxy vs native — is a deployment choice; the token model below assumes TLS protects the transport)
• CORS: Not required (embedded SPA is same-origin)

Dev mode exception: the trunk serve dev server proxies /api → 127.0.0.1:PAM_PORT for local development.

Authentication Scheme

• Method: Bearer token in Authorization header
• Token format: 32-character hex string
• Storage: platform data dir token file (~/Library/Application Support/pam/token on macOS, ~/.local/share/pam/token on Linux) on the central host
• CLI behavior: Automatically reads and injects token
• Generation: Auto-generated by the central orchestrator on first start if missing
• Worker authentication is separate: workers register with a pre-provisioned worker token (PAM_WORKER_TOKEN), stored in the worker host’s platform data dir; whether to upgrade to mutual TLS is tracked as OQ-10

Token injection flow: The orchestrator generates the token file in its platform data dir on first start; the CLI reads that file automatically; PAM_TOKEN env var takes precedence over the file.

Error Envelope

All error responses follow this structure:

{
  "error": {
    "code": "string",
    "message": "string",
    "details": {} // optional, context-specific
  }
}

Standard HTTP status codes:

• 400 Bad Request (validation errors)
• 401 Unauthorized (missing/invalid token)
• 404 Not Found (resource missing)
• 409 Conflict (state conflicts, lease acquisition)
• 500 Internal Server Error
• 503 Service Unavailable (database/executor issues)

REST API Endpoints

All endpoints prefixed with /api/v1.

Projects

Notes:

• slug auto-generated from name if not provided
• Delete performs soft-archive (status → archived)

Work Items

Notes:

• kind values: milestone, epic, task, subtask
• status values: todo, doing, done, cancelled
• parent_id references work_items.id (self-FK for hierarchy)
• Board move = PATCH status field only
• path field computed from parent chain (read-only)

Reparent semantics (PATCH parent_id):

• Nullable — passing null promotes the item to the project root
• A move whose target parent belongs to a different project is rejected 400 (the project_id invariant is re-checked on every reparent)
• A move that would place an item inside its own subtree is rejected 400 (cycle guard)
• The move rewrites path for the item and all of its descendants in one transaction, and journals work_item.reparented once, for the moved item only — descendants emit no event of their own
• Transition legality, the hierarchy guards, and the path-rewrite rule are owned by 002-data-model.md; this endpoint only exposes them

Runs

Notes:

• provider values: claude, codex, grok, gemini
• session_name format: pam-<8hex> (tmux socket pam on the executing host; uniqueness is scoped per host)
• host_id on run objects identifies the executing worker host
• status values: pending, spawning, running, exited, failed, stopped
• keys allowlist: alphanumeric, Enter, Ctrl+C (executor-side validation)
• Pane snapshots arrive over the executing worker’s uplink stream (workers accept no inbound connections — the orchestrator never pulls from a worker); the orchestrator caches the latest snapshot and serves it
• Logs endpoint streams the run’s log file, relayed by the orchestrator from the executing host (path per tmux-executor: <platform data dir>/runs/<run_id>.log on that host)
• Runs may only target work items of kind task or subtask (application-level validation)
• POST /runs always creates the run in pending and returns 202 — it never fails synchronously because no host is currently eligible. Dispatch is asynchronous; the response confirms admission, not placement
• An unclaimed run fails at the dispatch timeout (default 5 minutes, configurable, measured from runs.pending_since — not created_at, which does not reset on requeue; confirmed OQ-15 fix, 071-risks-and-open-questions.md). Both terminal outcomes are recorded at that timeout, never at claim time: failure_reason = 'no_eligible_host' when no online, non-disabled host ever matched the requested target — whether or not the request named a specific host — and failure_reason = 'dispatch_timeout' when a host matched but never claimed the run. A target host going offline or being disabled does not immediately fail the run. The pending → failed transition itself is owned by 002-data-model.md
• A run requeued by the restart policy re-enters pending with attempt incremented, and MAY be claimed by a different host than the previous attempt — host affinity is not preserved across attempts unless the run pinned a host at creation

Hosts (Worker Registry)

Notes:

• status values: registered, online, offline, disabled
• Worker registration and heartbeat are internal endpoints (/api/internal/worker/*), authenticated with the worker token — not part of the public surface
• Hosts are created by the operator (pam host add) and receive a worker token at provisioning time; a worker cannot self-register an unknown host
• disabled is sticky: a heartbeat from a disabled host updates last_seen_at but never status. Only an explicit POST /hosts/{id}/enable leaves disabled — an operator exclusion must survive the host continuing to run
• enable’s target status is derived, not fixed: it clears the disabled flag and resolves to online when last_seen_at falls inside the liveness window, otherwise offline
• disable is drain-only: runs already executing on the host continue, keep streaming pane output, and relay signals to completion. Only new dispatch is withheld. Disabling never kills a session and never triggers the graceful stop ladder
• registered means the operator created the row and no worker has heartbeated yet; online and offline are heartbeat-derived. A host never returns to registered
• The active-run count from GET /hosts/{id} is how an operator judges drain completion after a disable

Provider Catalog

Events

Event topics: See 002-data-model.md for canonical enumeration (project.*, work_item.*, run.*, worker.*).

Server-Sent Events (SSE)

Event Stream Endpoint

GET /api/v1/events/stream

Query Parameters:

• topic: Optional topic filter (supports glob patterns: run.*)
• Last-Event-ID: Resume cursor (maps to events.seq)

Payload Envelope

id: <seq>
event: <topic>
data: <json_payload>

Example:

id: 1234
event: run.started
data: {"run_id": "f8d9e2a1-4b7c-4e21-9a03-5c6d8e1f2a3b", "attempt": 1, "task_id": "3c9a7f10-2d84-4f6e-b1c5-90ab7d4e6f22", "provider": "claude", "session_name": "pam-f8d9e2a1"}

Payload identifier rules (normative):

• run_id and task_id are UUID strings, not integers — they carry runs.id and work_items.id verbatim (see 002-data-model.md)
• Every run.* event payload carries attempt alongside run_id, so a consumer can tell which incarnation of the run the event describes. Without it, a restarted run’s events are indistinguishable from the dead incarnation’s replayed ones (runs.attempt, see 002-data-model.md)
• session_name’s 8-hex suffix is the leading 8 characters of run_id; it is a display/lookup convenience, never an identifier in its own right
• The canonical topic enumeration is owned by 002-data-model.md §Event Topics

Reconnect Semantics

• Client includes Last-Event-ID: <seq> header on reconnect
• Server resumes from seq + 1 (exclusive cursor)
• If seq not found (purged), server sends full snapshot then resumes live
• No heartbeats: rely on TCP keepalive

Note: Events are kept 30 days by default, with best-effort purge at orchestrator startup (see Data Retention policy in 002-data-model.md).

WebSocket Protocols

Live Pane WebSocket (Read-Only)

WS /api/v1/runs/{id}/pane/ws

Purpose: Server → client streaming of terminal pane updates.

Protocol Design: Full snapshot on change with sequence numbers.

Server Frame Format:

{
  "type": "pane_snapshot",
  "seq": 123,
  "run_id": "abc123",
  "capture": "<terminal content>",
  "timestamp": "2026-08-29T12:34:56Z"
}

Sequence namespace note: the pane-snapshot seq above orders WebSocket snapshot frames for the terminal view. It is a different sequence from the seq carried by repeated run signals (progress) at the signal-ingest edge (see Provider Hook Receiver below) — the two occupy separate namespaces and must not be conflated.

Justification: Full snapshot chosen over delta encoding because:

• Terminal panes are relatively small (actual spawn size 220×50 ≈ 11,000 chars, ~11 KB per full snapshot)
• Avoids delta frame synchronization complexity
• Simplifies client state management (no diff application)
• Modern xterm.js handles full replacements efficiently

Throttling: Source capture polling is 1 Hz (snapshots emitted only on change); the 10 fps cap applies to the interactive attach WebSocket relay’s frame coalescing.

Attach WebSocket (Interactive)

WS /api/v1/runs/{id}/attach/ws

Purpose: Bidirectional PTY relay for interactive terminal session.

Authentication: Token passed as query parameter ?token= (browser WebSocket clients cannot set headers). Note: token appears in URLs, so avoid logging full WebSocket URLs in logs.

Lease Enforcement:

• Single-writer lease prevents concurrent attach conflicts
• First client acquires lease automatically
• Subsequent clients receive lease-acquired error

Server → Client Frames:

1. Lease Acquired (confirmation):

{
  "type": "lease_acquired",
  "run_id": "abc123",
  "attached_at": "2026-08-29T12:34:56Z"
}

2. Lease Held (error to others):

{
  "type": "error",
  "code": "lease_held",
  "message": "Session already attached by another client"
}

3. PTY Data (terminal output):

{
  "type": "data",
  "bytes": "<base64-encoded binary PTY output>"
}

4. Resize (client should sync dimensions):

{
  "type": "resize",
  "rows": 24,
  "cols": 80
}

Client → Server Frames:

1. Input (keystrokes):

{
  "type": "input",
  "bytes": "<base64-encoded input bytes>"
}

2. Resize (request PTY resize):

{
  "type": "resize",
  "rows": 24,
  "cols": 80
}

3. Lease Release (explicit detach):

{
  "type": "release_lease"
}

Single-Writer Enforcement:

• Server tracks lease holder per run
• Non-holder clients receive lease_held on any frame attempt
• Lease auto-releases on WebSocket close
• Server terminates attach WS when run exits

Note: The 409 Conflict status code in the REST status-code table applies to REST conflicts, not the WebSocket attach handshake (which uses the {"type":"error","code":"lease_held"} frame for lease rejection after upgrade).

Internal Endpoints

Worker Uplink

Transport (resolved, OQ-11): persistent bidirectional WebSocket, one resident connection per worker process. A worker opens WS /api/internal/worker/uplink immediately after a successful POST /api/internal/worker/register and keeps it open for the process lifetime, reconnecting with backoff on drop. Heartbeat, run-claim dispatch, and the snapshot/signal/log uplink all ride this single connection — periodic polling is not the primary path; it remains a documented fallback only if the persistent connection proves brittle in the field (071-risks-and-open-questions.md).

Registration stays a one-shot REST call (simple request/response, happens once per worker process start); everything recurring moves to the WS frames below.

Registration

POST /api/internal/worker/register

Authenticated with the worker token. Resolves the caller against an operator-provisioned hosts row — a worker cannot self-register an unknown host (002-data-model.md).

Request:

{
  "host_name": "mac-studio-01",
  "platform": "macos",
  "instance_id": "a1b2c3d4-5e6f-4789-9abc-def012345678"
}

instance_id (OQ-14, confirmed, 071-risks-and-open-questions.md): generated once by the worker process and persisted in its local data dir, then sent unchanged on every registration and heartbeat for the life of that persisted value. Stored on hosts.instance_id (002-data-model.md). When an incoming instance_id disagrees with the pinned value, the server compares now against the pinned instance’s last_seen_at (captured before the update) to the existing 90-second offline threshold: past the threshold reads as a legitimate reinstall (the value updates silently, an info-level worker.instance_reinstalled event is journaled); within it reads as a suspected clone (the pinned instance_id is left unchanged, hosts.instance_suspect is set, and a worker.instance_suspect anomaly event is journaled once on the false→true transition). See 002-data-model.md for the full column and transition detail.

Response:

{
  "host_id": "3c9a7f10-2d84-4f6e-b1c5-90ab7d4e6f22",
  "status": "registered" | "online"
}

Uplink WebSocket

WS /api/internal/worker/uplink

Authentication: Authorization bearer header with the worker token — unlike the browser-facing Attach WebSocket, the worker is not a browser and can set headers on the handshake, so no query-param workaround is needed.

Server → Client (orchestrator → worker) frames:

1. Heartbeat ack:

{ "type": "heartbeat_ack", "timestamp": "2026-09-01T12:00:00Z" }

2. Run dispatch (a claimed run pushed to this host):

{
  "type": "run_dispatch",
  "run_id": "f8d9e2a1-4b7c-4e21-9a03-5c6d8e1f2a3b",
  "task_id": "3c9a7f10-2d84-4f6e-b1c5-90ab7d4e6f22",
  "provider": "claude",
  "session_name": "pam-f8d9e2a1",
  "attempt": 1,
  "cwd": "/Users/op/projects/pam"
}

3. Error (protocol-level, e.g. malformed frame):

{ "type": "error", "code": "string", "message": "string" }

Client → Server (worker → orchestrator) frames:

1. Heartbeat — sent every 30 s:

{ "type": "heartbeat", "host_id": "3c9a7f10-2d84-4f6e-b1c5-90ab7d4e6f22", "instance_id": "a1b2c3d4-5e6f-4789-9abc-def012345678", "timestamp": "2026-09-01T12:00:00Z" }

2. Run dispatch ack:

{ "type": "run_dispatch_ack", "run_id": "f8d9e2a1-...", "attempt": 1, "accepted": true }

3. Snapshot — pane capture relay; the orchestrator caches the latest and serves it via GET /runs/{id}/pane and the Live Pane WebSocket:

{ "type": "snapshot", "run_id": "f8d9e2a1-...", "capture": "<terminal content>", "timestamp": "2026-09-01T12:00:00Z" }

4. Signal — lifecycle/progress relay; the frame wraps the same body documented under Provider Hook Receiver below (kind/attempt/seq/source/payload/timestamp), and is subject to the identical ingest-ordering, dedup, and epoch rules — this uplink frame and the POST /api/internal/runs/{id}/signals path are two transports for the same contract, never two contracts:

{ "type": "signal", "run_id": "f8d9e2a1-...", "kind": "progress", "attempt": 1, "seq": 42, "source": "wrapper", "payload": {}, "timestamp": "2026-09-01T12:00:00Z" }

5. Log chunk — appended run-log text relay, backing GET /runs/{id}/logs:

{ "type": "log_chunk", "run_id": "f8d9e2a1-...", "text": "...", "timestamp": "2026-09-01T12:00:00Z" }

Notes:

• Claim serialization is unchanged by this transport: the orchestrator still performs the SELECT ... FOR UPDATE-guarded claim (002-data-model.md) before pushing run_dispatch; the WS only changes how the claimed run reaches the worker — a push down the worker’s own outbound connection, never a connection the orchestrator opens toward the worker.
• 30 s heartbeat cadence; the orchestrator marks a host offline after 90 s without one (3 missed beats). This is a distinct timer from the worker’s own local reconciliation loop (30 s) and liveness poll (10 s) documented in 010-tmux-executor.md — those never cross the wire.
• worker.registered fires when a host’s derived status flips to online (first heartbeat after registration, or reconnection after an offline gap); worker.offline fires when the 90 s threshold is crossed (002-data-model.md §Event Topics).
• On reconnect after a drop, the worker replays its on-disk signal spool verbatim (attempt/seq preserved, never renumbered) before resuming live heartbeat — the same at-least-once delivery model documented under Provider Hook Receiver applies here.
• runs.attempt is allocated centrally only; the worker never assigns it. run_dispatch carries the attempt the orchestrator already incremented; the worker’s run_dispatch_ack and every subsequent signal for that run echo it verbatim.
• OQ-14 (worker instance identity, 071-risks-and-open-questions.md): confirmed 2026-09-01 — instance_id is carried on registration and every heartbeat and stored on hosts.instance_id (002-data-model.md). The mismatch-detection algorithm is the recency-based heuristic described above (§ Worker Uplink Registration): an incoming instance_id that disagrees with the pinned value is judged a reinstall or a suspected clone by comparing the pinned instance’s last-seen gap against the 90-second offline threshold. A suspect host is excluded from new dispatch (alongside disabled) but its already-running runs are left untouched; an operator clears the flag with pam host clear-suspect (040-cli.md), which also accepts the new instance_id as the pinned value.

Provider Hook Receiver

POST /api/internal/runs/{id}/signals

Purpose: Receive completion signals from provider-native hooks (claude-hooks, etc.).

Trust Model:

• Callers are wrappers and provider hooks on worker hosts, plus the workers themselves — all remote relative to the orchestrator
• Authenticated with the worker token (same credential class as the registration uplink; a dedicated signal token remains under review — OQ-8)

Request Shape:

{
  "kind": "started" | "finished" | "completed" | "failed" | "progress",
  "attempt": 1,
  "seq": 42,
  "source": "wrapper" | "provider-hook" | "manual",
  "payload": {},
  "timestamp": "2026-08-29T12:34:56Z"
}

Notes:

• kind values are lowercase, mapped 1:1 to provider-adapters’ RunSignal enum
• attempt (required, integer): the incarnation epoch of the emitting process — the run’s current attempt number (runs.attempt, see 002-data-model.md). It is allocated centrally (incremented on each restart-policy requeue) and injected into the spawn environment; the emitting wrapper (pam exec run) stamps every signal with it. Numbers are stamped at emission time and ride the worker’s on-disk spool verbatim on replay — the worker never renumbers.
• seq (optional, int64): carried ONLY by repeated kinds (progress); the one-shot kinds (started, finished, completed, failed) omit it — they are exactly-once via the central dedup table. Monotonic within (run_id, attempt); a progress signal is applied only when seq > runs.last_progress_seq.
• source values: wrapper (from pam exec run), provider-hook (from provider-native hooks), manual (normalized internally from public-API actions such as POST /runs/{id}/stop — manual sources do not call this endpoint directly; users authenticate with bearer tokens, not worker tokens)

Ingest ordering (normative) — the signal-ingest edge applies, in order:

1. Epoch / dedup / ordering guard. Compare the signal’s (attempt, seq) pair against the run row:
   • attempt < runs.attempt → the signal comes from a dead incarnation: discard — no status write, no event emission.
   • attempt > runs.attempt → impossible under central attempt allocation: protocol error (see response table).
   • attempt == runs.attempt → pass the processed_signals dedup table (defined in 002-data-model.md; PRIMARY KEY (run_id, attempt, kind), ingest does INSERT … ON CONFLICT DO NOTHING — a conflict means the signal was already applied and is dropped), then, for repeated kinds, the seq > runs.last_progress_seq ordering check.
2. Run status write. The guarded transition itself, per the status-transition rules in 020-provider-adapters.md.
3. Event journal + SSE fan-out. Journal the resulting run.* event and push it to SSE subscribers.

Step 1 sits strictly upstream of both step 2 and step 3. A guard placed after either one would let a phantom event reach the dashboard for a signal whose status write was skipped — e.g., a replayed duplicate failed emitting run.failed to SSE while the run stays running. The two axes are orthogonal: attempt says which incarnation is speaking; seq says whether this signal is newer within that incarnation.

Response semantics:

Protocol-error envelope examples:

{
  "error": {
    "code": "invalid_attempt",
    "message": "signal attempt exceeds the run's current attempt",
    "details": { "signal_attempt": 3, "current_attempt": 1 }
  }
}

{
  "error": {
    "code": "missing_attempt",
    "message": "signal payload did not include an attempt field",
    "details": { "current_attempt": 1 }
  }
}

Other errors follow the standard status-code table: 400 (malformed payload), 404 (unknown run), 500 (internal failure).

Integration: Hooks configured in provider adapters (see 020-provider-adapters.md).

Versioning

Current API version: /api/v1

Future versioning strategy:

• Major version bump (/api/v2) for breaking changes
• Backward-compatible additions within /api/v1
• Deprecation headers (Sunset) for phased removal

OpenAPI generation: Deferred to post-MVP (tooling selection TBD: utoipa vs hand-written).

Completion Signal Ladder

For run termination detection, override authority (conflict winner: manual > provider hook > wrapper journal) is distinct from signal trust (task-level PRIMARY = provider hooks; session-level = wrapper journal; manual lowest; pane-idle UI hint only). See the canonical specification in 020-provider-adapters.md.

CLI Contract

Command Structure

The pam binary uses clap 4.5 derive for command parsing.

Global Flags

• --json: Output format as JSON (applies to all list/show commands)
• --config <path>: Override config file path (highest priority — see 060-configuration.md § Precedence Order)
• --api-url <url>: Override central orchestrator URL (highest priority; default for CLI commands: http://127.0.0.1:7898)
• --token <token>: Override bearer token (highest priority)
• -h, --help: Display help
• -V, --version: Display version

Command Inventory

Core Commands

pam serve

Start the central orchestrator (API server and dashboard).

pam serve [--bind <ADDR>] [--port <PORT>] [--database-url <DSN>] [--config <PATH>]

Flags:

• --bind <ADDR>: Bind address (default: 0.0.0.0, env: PAM_BIND)
• --port <PORT>: Server port (default: 7898, env: PAM_PORT)
• --database-url <DSN>: PostgreSQL connection string (env: PAM_DATABASE_URL; required — no default)
• --config <PATH>: Config file path (default: <platform data dir>/config.toml, env: PAM_CONFIG)

Semantics: Serves the embedded SPA (Rust/WASM — Leptos), the HTTP API, and the worker registry; owns the only PostgreSQL connection. Workers connect outbound to this process. Runs until Ctrl+C.

pam migrate

Run database migrations.

pam migrate [--database-url <DSN>]

Flags:

• --database-url <DSN>: PostgreSQL connection string (env: PAM_DATABASE_URL)

Semantics: Applies pending migrations against PostgreSQL. Idempotent if already applied.

pam doctor

Health check for installation and dependencies.

pam doctor

Semantics: Checks central API reachability, PostgreSQL reachability (when run on the orchestrator host with a DSN configured), local tmux availability (relevant on worker hosts), config validity. Exits 0 if healthy, 1 if issues found.

Worker Commands

pam worker

Run the worker agent on an execution host.

pam worker --name <HOST_NAME> [--api-url <URL>] [--config <PATH>]

Flags:

• --name <HOST_NAME>: Host name as registered centrally (required; must match a pam host add record)
• --api-url <URL>: Central orchestrator base URL (env: PAM_API_URL; required — no default)
• --config <PATH>: Worker config path (default: <platform data dir>/worker.toml, env: PAM_WORKER_CONFIG)

Semantics: Registers with the central orchestrator over an outbound connection (no inbound listener), then executes dispatched runs in local tmux sessions on socket pam. Authenticates with the worker token (PAM_WORKER_TOKEN env or the platform data dir worker-token file) provisioned by pam host add. Runs until Ctrl+C.

Host Commands

pam host add

Register a new execution host.

pam host add <name> --platform <PLATFORM>

Arguments/Flags:

• <name>: Unique host name (required)
• --platform <PLATFORM>: macos or linux (required)

Semantics: Creates the host record (status: registered) and prints a worker token to install on that host. The token is shown once — store it in the worker host’s worker-token file.

pam host list

List worker hosts with status and last-seen time.

pam host disable / pam host enable

Exclude a host from dispatch / return it to the pool.

pam host rotate-token

Issue a fresh worker token for an already-registered host (OQ-10, confirmed, 071-risks-and-open-questions.md).

pam host rotate-token <name>

Arguments:

• <name>: Host name to rotate (required; must already exist)

Semantics: Mints a new worker token, prints it once (same one-time-display contract as pam host add), and records its fingerprint on hosts.token_fingerprint. The previous token keeps working until the worker actually presents the new one — there is no forced grace-period cutover — so restart the worker on <name> with the new token promptly after rotating. See 061-security.md for the full token-rotation procedure and when to use it (suspected leak, routine credential hygiene, offboarding a host).

pam host clear-suspect

Clear a host’s instance-suspect flag (OQ-14, confirmed, 071-risks-and-open-questions.md).

pam host clear-suspect <name>

Arguments:

• <name>: Host name to clear (required; must currently have hosts.instance_suspect = true)

Semantics: An operator trust-transfer action. Clears hosts.instance_suspect and hosts.instance_flagged_at, and accepts the instance_id most recently presented by that host as the new pinned value — so use it only after confirming (out of band) which of the two machines presenting the same worker token is the legitimate one. See 002-data-model.md for the detection algorithm this clears the output of.

Project Commands

pam project add

Create a new project.

pam project add <name> [--slug <SLUG>]

Arguments:

• <name>: Project name (required)

Flags:

• --slug <SLUG>: URL-friendly slug (auto-generated from name if omitted)

Semantics: Creates project record in database. Returns project ID.

pam project list

List all projects.

pam project list [--status <STATUS>]

Flags:

• --status <STATUS>: Filter by status (active, archived)

Output: Table format (id, name, slug, status, created_at) or JSON.

pam project show

Display project details.

pam project show <id>

Arguments:

• <id>: Project ID or slug

Semantics: Shows project metadata, work item count, recent activity.

pam project archive

Archive a project.

pam project archive <id>

Arguments:

• <id>: Project ID or slug

Semantics: Soft-deletes project (status → archived). Work items remain intact.

Task Commands

pam task add

Create a new task, epic, milestone, or subtask.

pam task add <title> --project <id> [--parent <id>] [--kind <KIND>]

Arguments:

• <title>: Work item title (required)

Flags:

• --project <id>: Parent project ID (required)
• --parent <id>: Parent work item ID (for epics/subtasks)
• --kind <KIND>: Item type (task, milestone, epic, subtask; default: task)

Semantics: Creates work item. If --parent specified, creates hierarchical relationship. Path computed from parent chain.

pam task list

List work items.

pam task list --project <id> [--kind <KIND>] [--status <STATUS>]

Flags:

• --project <id>: Project ID (required)
• --kind <KIND>: Filter by kind (task, milestone, epic, subtask)
• --status <STATUS>: Filter by status (todo, doing, done, cancelled)

Output: Tree view showing hierarchy, or flat list with path column.

pam task show

Display work item details.

pam task show <id>

Arguments:

• <id>: Work item ID

Semantics: Shows description, status, assignee, parent/child relationships, associated runs.

pam task move

Move work item across board columns.

pam task move <id> --status <STATUS>

Arguments:

• <id>: Work item ID

Flags:

• --status <STATUS>: New status (todo, doing, done, cancelled)

Semantics: Updates work item status. Triggers work_item.status_changed event. Legal transitions follow the work-item transition table in 002-data-model.md — including todo → done directly: a task completed without ever entering doing (finished out-of-band, or its run completed while auto_mark_done is off) does not need to pass through Doing first.

pam task board

Display Kanban board for project.

pam task board <project-id>

Arguments:

• <project-id>: Project ID

Output: Four-column layout (todo/doing/done/cancelled) with work item counts.

pam task done

Mark task as completed.

pam task done <id>

Arguments:

• <id>: Work item ID

Semantics: Shortcut for pam task move <id> --status done. Valid from both todo and doing (see the transition table in 002-data-model.md) — a task does not need to sit in doing first. Typical use: work completed out-of-band, or a run finished while auto_mark_done was off and the item was never moved.

Run Commands

pam run start

Start a new AI provider session.

pam run start --task <id> --provider <provider> [--host <NAME>] [--cwd <PATH>]

Flags:

• --task <id>: Task or Subtask ID (required)
• --provider <provider>: Provider name (claude, codex, grok, gemini; required)
• --host <NAME>: Target worker host name (optional; default: auto-select an online host)
• --cwd <PATH>: Working directory path on the target host (default: the worker’s configured working root)

Semantics: Creates run record, dispatches to the target host; the worker there spawns the tmux session pam-<8hex>. Returns run ID, host, and session name. Provider wrapper CLI launched in session.

pam run list

List runs.

pam run list [--task <id>] [--status <STATUS>]

Flags:

• --task <id>: Filter by task ID
• --status <STATUS>: Filter by status (pending, spawning, running, exited, failed, stopped)

Output: Table with run ID, task ID, provider, session name, status, started/finished timestamps.

pam run logs

View run logs.

pam run logs <id> [--follow]

Arguments:

• <id>: Run ID

Flags:

• --follow: Stream logs in real-time (like tail -f)

Semantics: Streams tmux session output via capture-pane. Follow mode polls every 1s.

pam run stop

Stop a running session.

pam run stop <id>

Arguments:

• <id>: Run ID

Semantics: Sends termination signal to tmux session. Status updates to stopped.

pam run pane

Display terminal pane snapshot.

pam run pane <id>

Arguments:

• <id>: Run ID

Semantics: Captures current terminal state via tmux capture-pane. Displays as text. Not suitable for ANSI rendering (use web UI for that).

pam run attach

Attach to tmux session directly.

pam run attach <id>

Arguments:

• <id>: Run ID

Semantics: For a run executing on the local host, execs tmux -L pam attach -t <session_name> for direct terminal takeover (detach with Ctrl+B, D). For runs on remote worker hosts, the CLI cannot reach the remote tmux socket — use the web UI’s relayed attach (WebSocket PTY relay through the orchestrator).

Note: This is the local terminal attach mode. For web-based attach (any host), see the WebSocket attach protocol in 030-http-api.md. For tmux mechanics, see 010-tmux-executor.md.

Provider Commands

pam provider list

List available AI providers.

pam provider list

Output: Table with provider ID, name, enabled status, configured status.

pam provider doctor

Test provider configuration and health.

pam provider doctor [<provider-id>]

Arguments:

• <provider-id>: Provider ID (optional; if omitted, checks all configured providers)

Semantics: Tests provider CLI availability, API key configuration, basic connectivity. Returns detailed health report.

Hidden Commands

pam exec run <run_id>

Internal foreground execution wrapper for tmux sessions.

pam exec run <run_id>

Arguments:

• <run_id>: Run ID

Semantics: Session’s foreground execution wrapper. Reads the run + provider config, normalizes/injects environment variables (including proxy base URLs), spawns the provider CLI, and POSTs run.started / run.finished{exit_code} to the internal signals endpoint, then holds the pane open for approximately 5 minutes for observation. Not exposed in help.

Note: Direct tmux attach is a separate path: pam run attach execs tmux -L pam attach -t <session_name> locally.

Environment Variables

Environment variables are the third-priority source for core CLI settings — below a CLI flag and below a config file value, above the built-in default. Full four-source order (resolves OQ-16): 060-configuration.md § Precedence Order.

Note: The platform data dir — ~/Library/Application Support/pam/ on macOS, ~/.local/share/pam/ on Linux, resolved via the directories crate (ADR-011) — holds config.toml and token on the central host, and worker.toml, worker-token, providers.toml, and runs/ on worker hosts.

Configuration File

Config file location: platform data dir (~/Library/Application Support/pam/config.toml on macOS, ~/.local/share/pam/config.toml on Linux)

Sections (orchestrator):

[server]
bind = "0.0.0.0"
port = 7898

[database]
url = "postgresql://pam@localhost:5432/pam"

[providers]
catalog_path = "<platform data dir>/providers.toml"

Worker config (worker.toml on each execution host):

[worker]
name = "dev-mac"
api_url = "https://pam.example.internal:7898"

[providers]
catalog_path = "<platform data dir>/providers.toml"

Providers catalog (separate TOML file): format is owned by 020-provider-adapters.md — see its “Provider Catalog” section for the canonical schema (id, bin, argv_template, env map, completion hooks config).

Exit Codes

Output Formats

Default (Human-Readable)

Tables with aligned columns for list commands. Structured text for show commands.

JSON Format

--json flag outputs valid JSON for machine parsing.

Example:

pam project list --json

Returns:

{
  "projects": [
    {
      "id": "abc123",
      "name": "My Project",
      "slug": "my-project",
      "status": "active",
      "created_at": "2026-08-29T12:34:56Z"
    }
  ]
}

Relationships to Other Components

• HTTP API (central orchestrator): ALL CLI commands — PM domain (project/task/board) included — call the central HTTP API using PAM_TOKEN (ADR-007, superseded 2026-08-29: direct store access would spread PostgreSQL credentials to every CLI host)
• Worker: pam worker runs on each execution host and executes dispatched runs; pam run attach and pane access flow through the orchestrator’s relay (see 030-http-api.md)
• tmux executor: workers own the tmux mechanics on their host; see 010-tmux-executor.md
• Provider adapters: pam provider doctor invokes adapter-specific health checks against the host’s catalog; see 020-provider-adapters.md

UI Dashboard - Information Architecture and Desktop Shell

Information Architecture

Global Layout

Top Navigation Bar

• Logo/brand (pam)
• Activity feed button (right side, notification badge for recent events)
• Orchestrator status indicator (green/red dot)
• Hosts summary chip (online/total worker hosts — click navigates to /hosts)
• Settings button

Main Content Area

• Single-page application with client-side routing
• Persistent activity feed drawer (collapsible, right side)

Page Hierarchy

1. Projects List Page (/projects)

Purpose: Overview and navigation to all projects.

Layout: Grid of project cards.

Per Project Card:

• Project name (click to navigate to project detail)
• Slug badge
• Status indicator (active/archived)
• Task count (total, broken down by status: todo/doing/done)
• Last activity timestamp
• Quick action button (menu: archive, settings)

Empty State: “No projects yet. Create your first project to get started.”

Sorting: Default by last activity, option to sort by name or creation date.

2. Project Detail Page (/projects/{id})

Purpose: Project work items hierarchy and Kanban board.

Layout: Split view.

Left Panel - Work Items Tree:

• Hierarchical tree of milestones → epics → tasks → subtasks
• Expand/collapse nodes
• Status icons per item
• Click to navigate to task detail
• Filter controls (by kind, by status, by assignee)

Right Panel - Kanban Board:

• Four columns: Todo, Doing, Done, Cancelled
• Cards show: title, assignee badge, run count (if any), subtask count
• Drag-and-drop between columns (PATCH status via API); legal drops follow the work-item transition table in 002-data-model.md, including a direct Todo → Done drag — a task completed without entering Doing (finished out-of-band, or its run completed while auto_mark_done is off) skips the intermediate column
• Click card to navigate to task detail
• “New Task” button (opens modal for quick creation)

Breadcrumbs: Projects → {project name}

3. Task Detail Page (/tasks/{id})

Purpose: Task information, run history, and live terminal pane.

Layout: Vertical sections.

Header:

• Task title (editable inline)
• Status badge (todo/doing/done/cancelled) - clickable to change
• Assignee field (editable)
• Kind badge (task/subtask)
• Path display (parent hierarchy links)

Description Section:

• Markdown editor/view toggle
• Empty state: “Add a description…”

Runs Section:

• List of runs (table: provider, session name, status, timestamps, actions)
• “Start Run” button (opens modal: provider selection)
• Per-run actions: logs, stop button (if running), attach button

Live Terminal Pane:

• Embedded xterm.js terminal (read-only)
• Connects to WS /api/v1/runs/{id}/pane/ws
• Shows latest pane snapshot
• Auto-updates on server push
• Empty state: “No active terminal session”

Metadata Footer:

• Created/updated timestamps
• Parent/child relationship links

Breadcrumbs: Projects → {project} → {task} or Projects → {project} → {milestone} → {epic} → {task}

4. Run View Page (/runs/{id})

Purpose: Detailed run information and interactive terminal access.

Layout: Split vertical.

Top Section - Run Info:

• Run ID (copy button)
• Task link (navigation)
• Provider badge (claude/codex/grok/gemini)
• Host badge (executing worker host name; offline hosts tinted with a warning)
• Session name (display: pam-<8hex>)
• Status badge (pending/spawning/running/exited/failed/stopped)
• Working directory path (on the executing host)
• Started/finished timestamps
• Exit code (if exited/failed)
• Actions: stop button (if running)

Middle Section - Interactive Terminal:

• xterm.js terminal (full interactive)
• “Attach” button (disabled if lease held by another client)
• “Detach” button (active when attached)
• Single-writer lease enforcement
• Connects to WS /api/v1/runs/{id}/attach/ws

Bottom Section - Run Logs:

• Scrollable log history (pane snapshots over time)
• “Download Logs” button

Breadcrumbs: Projects → {project} → {task} → Run {id}

5. Hosts Page (/hosts)

Purpose: Worker fleet overview (multi-host topology, ADR-012).

Layout: Table of worker hosts.

Per Host Row:

• Host name + platform badge (macos/linux)
• Status (registered/online/offline/disabled)
• Last-seen timestamp, active run count
• Actions: disable/enable dispatch

Empty State: “No worker hosts registered. Run pam host add <name> --platform <platform> and start pam worker on that host.”

6. Activity Feed (Global Drawer)

Purpose: Real-time event journal across all resources.

Layout: Collapsible right drawer.

Feed Items (per event from SSE stream):

• Timestamp
• Event topic (project.created, work_item.updated, run.started, etc.)
• Resource links (clickable to navigate to relevant page)
• Contextual details (e.g., “Task ‘Fix bug’ moved to Done”)

Filters: Toggle by event type (project., work_item., run.*)

Empty State: “No recent activity”

7. Command Palette (Future Ambition)

Trigger: Cmd+K (or Ctrl+K)

Purpose: Quick navigation and action execution.

Features:

• Fuzzy search across projects, tasks, runs
• Quick actions: “Create task…”, “Start run…”, “Move to Done…”
• Keyboard navigation
• Deferred to post-MVP

SPA Stack

Core Technologies

• Leptos: Rust/WASM UI framework (CSR mode; fine-grained reactivity — no virtual DOM, no separate state library)
• wasm-bindgen / web-sys: browser interop layer
• xterm.js: the ONLY JavaScript dependency — browser terminal emulation has no mature Rust/WASM replacement; wrapped as a component driven through wasm-bindgen interop

State Management

• Leptos resources/signals: server-state synchronization
  • Resources fetch projects, work items, runs; refetch on mutation
  • Fine-grained reactivity drives board and live-view updates without manual cache invalidation (replaces TanStack Query)
  • Optimistic updates for drag-and-drop board moves
• Leptos Router: client-side routing with typed params (replaces TanStack Router)

Terminal Rendering

• xterm.js (sole JS dependency, see Core Technologies): terminal emulator
  • ANSI escape sequence rendering
  • Read-only pane display (WS /api/v1/runs/{id}/pane/ws)
  • Interactive attach session (WS /api/v1/runs/{id}/attach/ws)
  • Canvas-based rendering (performance optimized)

Real-Time Features

• Server-Sent Events (SSE): activity feed event stream
  • web_sys::EventSource connection to /api/v1/events/stream
  • Topic filtering support
  • Reconnect with Last-Event-ID cursor
• WebSocket: terminal pane and attach (web_sys::WebSocket)
  • Auto-reconnect on disconnect
  • Lease state tracking for attach

Development Mode

trunk dev server:

• Runs on port 5173 (default) with /api proxied to 127.0.0.1:PAM_PORT
• Hot rebuild of the WASM bundle
• CORS not needed (same-origin after proxy)

Command: trunk serve (from the SPA workspace member under web/)

Production Build

Build Process:

1. trunk build --release compiles the WASM bundle into web/dist/
2. Rust-embed includes dist/ in the binary
3. Axum serves embedded files at /
4. API routes at /api/v1 (unaffected)

Embedded File Structure:

• web/dist/index.html → /
• web/dist/assets/* → /assets/* (WASM + the xterm.js bundle)
• Fallback: all non-API routes serve index.html (SPA routing)

Desktop Shell (Tauri v2 — Confirmed)

Shell Role: Monitoring Thin Client, Never an Execution Node

• Monitors the central orchestrator: hosts, runs, events, live terminals
• Views projects, work items, and other resources (board, detail, logs)
• User actions are forwarded as API requests ONLY — the orchestrator and its workers perform all work (spawning, supervision, provider execution stay in tmux on worker hosts)
• No local filesystem or shell capabilities required: configure Tauri capabilities with everything disabled (see 061-security.md)
• Target platforms: macOS and Linux desktops — matching the execution-node targets (ADR-011); Windows is out of scope and mobile is excluded. The shell itself is platform-independent by construction (it only loads the orchestrator URL), so no platform-specific shell code is anticipated

Comparison Record: Electron vs Tauri v2

Decision: Tauri v2 (Accepted 2026-08-29)

Decision: Ship web-first (embedded SPA in single binary) for MVP; the desktop shell is Tauri v2 — confirmed, no longer deferred. Web-first delivery still stands; only the shell implementation is scheduled for Phase 6.

Rationale:

1. Web-first suffices for single-operator local use case: Embedded SPA in binary already provides “app-like” experience without overhead
2. Size and memory: OS webview reuse (5-10 MB shell) vs ~150 MB bundled Chromium
3. Rust alignment: single-language stack; Tauri’s Rust core matches the existing architecture with no Node.js sidecar
4. Security surface: no Node.js runtime; a monitoring-only shell needs no local capabilities

Architecture Keeps Both Viable:

• SPA uses only standard web APIs (localStorage, fetch, WebSocket, EventSource)
• No Electron-specific APIs (ipcRenderer, remote) or Tauri-specific APIs (invoke) in SPA code
• Desktop shell is a thin wrapper that loads the same embedded UI
• Optionally runs pam serve as a background process (sidecar mode)

Electron Fallback Preserved:

• If Tauri ecosystem gaps emerge (e.g., auto-update issues on macOS), Electron remains a drop-in fallback
• No lock-in: SPA is pure web, can be wrapped in either

Implementation Note (Phase 6):

• Tauri app loads the orchestrator URL (PAM_API_URL — same artifact the browser uses)
• Optionally spawns pam serve as a child process (sidecar mode) — process management only; no execution logic in the shell
• IPC bridge for tray icon, notifications, window management
• Rust core reused directly (no sidecar FFI needed)

Live Pane Rendering

Rendering Approach

Strategy: Server-sent full snapshots, client renders via xterm.js.

Server Behavior:

• Monitors tmux session output
• Detects pane changes (debounced 100ms)
• Sends full snapshot over WS pane WebSocket
• Includes sequence number for ordering

Client Behavior:

• Connects to WS /api/v1/runs/{id}/pane/ws
• Receives {type: "pane_snapshot", seq, capture, timestamp} frames
• Clears and rewrites terminal on each frame
• xterm.js handles ANSI escape sequences

Sequence namespace note (do not conflate): the pane seq on pane_snapshot frames and the run-signal seq carried by progress signals are two separate sequences in unrelated namespaces. The pane seq orders WebSocket snapshot frames for the terminal view; the signal seq orders repeated progress signals at the central ingest edge (030-http-api.md). Neither field is renamed — the distinction is deliberate.

Justification:

• Full snapshot simpler than delta encoding (no diff synchronization)
• Terminal pane size bounded (actual spawn size 220×50 ≈ 11,000 chars, ~11 KB per full snapshot)
• xterm.js efficiently handles full rewrites
• Delta complexity not worth it for small pane size

Throttling and Coalescing

Server-Side:

• Source capture polling is 1 Hz (snapshots emitted only on change)
• Coalesce rapid updates within 100ms window
• Max 10 frames per second for the interactive attach WS relay’s frame coalescing (prevents client overload)
• Priority to latest state (drops intermediate frames if lagging)

Client-Side:

• Request animation frame for rendering (60fps max)
• Skip frames if backpressure detected

Read-Only vs Attach UX Distinction

Read-Only Pane (embedded in task detail page):

• WS /api/v1/runs/{id}/pane/ws
• No input capability
• Always available (no lease enforcement)
• Purpose: Monitoring without interaction

Interactive Attach (run view page):

• WS /api/v1/runs/{id}/attach/ws
• Bidirectional PTY relay
• Single-writer lease (only one client at a time)
• Full terminal takeover
• Purpose: Interactive debugging and intervention

Visual Differentiation:

• Read-only pane: Smaller height, no input prompt visual cues
• Attach pane: Full-height, clear focus border, “Attached” badge

Accessibility Considerations

• Keyboard Navigation: All interactive elements keyboard-accessible (Tab, Enter, Space)
• Screen Reader Support: ARIA labels on dynamic content, live regions for terminal updates
• Color Contrast: WCAG AA compliant (dark/light mode support)
• Focus Management: Logical tab order, visible focus indicators
• Terminal Accessibility: xterm.js provides screen reader announcements for pane changes

Configuration

Status: proposed — consolidated 2026-09-01 from material already designed in 040-cli.md, 020-provider-adapters.md, and 002-data-model.md. No new design decisions were made here; this document is the single place to read the whole configuration surface instead of following links across three docs.

Scope

Consolidated design of configuration for both roles after ADR-012 — the orchestrator host (config.toml) and each worker host (worker.toml, providers.toml, worker token file) — under the per-host platform data dir, plus environment-variable and CLI-flag precedence.

Platform Data Dir

Every config and credential file lives under one per-host directory, resolved via the directories crate (ADR-011):

• macOS: ~/Library/Application Support/pam/
• Linux: ~/.local/share/pam/

Orchestrator Config (config.toml)

[server]
bind = "0.0.0.0"
port = 7898

[database]
url = "postgresql://pam@localhost:5432/pam"

[providers]
catalog_path = "<platform data dir>/providers.toml"

[retention]
events_days = 30

• [server] — bind address and port for the HTTP/WS listener.
• [database] — PostgreSQL DSN; no default, required to serve.
• [providers] — path to the provider catalog TOML (schema owned by 020-provider-adapters.md).
• [retention] — events_days controls how long events-table rows are kept before a best-effort purge at orchestrator startup (default 30 days; see 002-data-model.md § Data Retention).

Worker Config (worker.toml)

[worker]
name = "dev-mac"
api_url = "https://pam.example.internal:7898"

[providers]
catalog_path = "<platform data dir>/providers.toml"

• [worker] — the host’s registered name and the central orchestrator’s base URL it dials out to (outbound-only uplink; no inbound listener).
• [providers] — same catalog-path key as the orchestrator side; each worker host keeps its own providers.toml copy.

Authentication is a separate file, not a config key: the worker token minted by pam host add is stored in worker-token (or PAM_WORKER_TOKEN) — see 061-security.md § Token Lifecycle.

Provider Catalog (providers.toml)

Schema, per-provider fields, and the completion-signal hook config are owned by 020-provider-adapters.md § Provider Catalog — not duplicated here. Default path: <platform data dir>/providers.toml, overridable via the catalog_path key above.

Gateway Config ([gateway] section, optional)

A proxy/gateway base URL and API key, read from the same config.toml. The gateway itself is one centrally-shared, network-reachable instance (e.g. an internal liteLLM deployment), not a local process per host (ADR-013):

[gateway]
url = "https://gateway.internal:8080"
api_key = "sk-gateway-key"

pam only injects these into the provider CLI’s environment (ANTHROPIC_BASE_URL etc.) at spawn time — see 020-provider-adapters.md § Gateway Operations Scope for what is explicitly out of scope (gateway lifecycle, health checks, routing).

Environment Variables

Environment variables are the third-priority source for every key below — below a CLI flag and below a config file value, above the built-in default. See § Precedence Order below for the full four-source chain.

Gateway settings use PAM_GATEWAY_URL / PAM_GATEWAY_API_KEY and are listed separately in 020-provider-adapters.md § Configuration Source.

Precedence Order

All settings now resolve config-file-first — one consistent rule across both groups (resolves OQ-16, which tracked this asymmetry until now):

Core CLI settings (PAM_API_URL, PAM_TOKEN, PAM_WORKER_TOKEN, PAM_BIND, PAM_PORT, PAM_DATABASE_URL, PAM_CONFIG, PAM_WORKER_CONFIG):

1. CLI flag (highest — e.g. --api-url, --config; an explicit, one-shot value typed at invocation time)
2. Config file value
3. Environment variable
4. Built-in default (where one exists)

Gateway settings ([gateway] section) — no CLI flag exists for these, so the chain is:

1. Config file value (highest)
2. PAM_GATEWAY_URL / PAM_GATEWAY_API_KEY
3. Provider-specific env vars (ANTHROPIC_BASE_URL etc.), lowest-priority fallback

Rationale: a config-managed deployment (e.g. Ansible/Terraform writing one config.toml) gets one predictable source of truth, and a value left over in a shell session can no longer silently outrank a value someone just changed in the file.

Open Questions

• OQ-16 (RESOLVED 2026-09-01): resolved — all settings now resolve config-file-first (see § Precedence Order above). Full resolution recorded in 071-risks-and-open-questions.md.

Security

Status: proposed — consolidated 2026-09-01 from material already designed in 030-http-api.md, 010-tmux-executor.md, 020-provider-adapters.md, 002-data-model.md, and 050-ui-dashboard.md. No new design decisions were made here, except where flagged as a new Open Question below.

Scope

Threat model and controls for a multi-host, single-operator platform (ADR-012): transport security (TLS), token classes (user bearer vs worker token), token lifecycle and fingerprint storage, WS query-token exposure, tmux safety boundaries, secret handling in provider env injection, and PostgreSQL credential confinement to the orchestrator.

Transport Security

Served via TLS in deployment; termination strategy (reverse-proxy vs native) is a deployment choice, not fixed here — every token model below assumes TLS protects the transport (030-http-api.md § Bind Configuration). The bind address defaults to 0.0.0.0 (network- exposed by design, ADR-012), so TLS is load-bearing, not optional hardening: without it, both the bearer token and the worker token travel in the clear over a listener other hosts can reach.

Token Classes

Two separate credential classes authenticate against the central orchestrator, never interchangeable:

• User bearer token — a 32-character hex string in the Authorization header, auto-generated on first orchestrator start and stored in the platform data dir token file (PAM_TOKEN overrides it). The CLI reads and injects it automatically. Authenticates the public API surface: users and the pam CLI (030-http-api.md § Authentication Scheme).
• Worker token — minted by the operator (pam host add), never self-registered by a worker. Authenticates the internal /api/internal/worker/* surface (registration, heartbeat, signal ingest) and, on the Attach WebSocket, is passed as a query parameter because browser WebSocket clients cannot set headers — see § WS Query-Token Exposure below. Lifecycle detail (rotation, clone detection) is § Token Lifecycle below.

There is no user account model — a single bearer token stands in for “the operator”, consistent with the single-operator scope in ADR-012.

WS Query-Token Exposure

The Attach WebSocket authenticates with ?token= in the URL, because a browser WebSocket client cannot set an Authorization header on the handshake (030-http-api.md § Live Pane Streaming). This means the token appears in URLs — proxy access logs, browser history, and any tool that logs full request URLs. Avoid logging full WebSocket URLs in application logs. The worker-facing uplink connection is not subject to this constraint: a worker is not a browser and authenticates with a normal Authorization bearer header on the handshake instead (030-http-api.md).

tmux Safety Boundary

The tmux executor runs on a dedicated socket (-L pam), never the user’s default tmux socket, and is bound by four safety rules (010-tmux-executor.md § Safety Rules):

1. Never run kill-server on the default socket — the user’s personal tmux must never be affected; catastrophic cleanup uses tmux -L pam kill-server only, all normal operations use session-specific commands.
2. Validate session names before any tmux invocation — regex ^pam-[0-9a-f]{8}$, applied to every -t argument and to session-list parsing, prevents command injection via user input.
3. Clean up only sessions pam created — two-part check (the session exists in the runs table registry AND its name matches the pam- prefix); orphan cleanup still respects the prefix constraint and never targets a user session, even one that somehow ended up on the pam socket.
4. Restrict send-keys to an allowlisted key set — programmatic input is limited to control sequences (Ctrl+C, Ctrl+D), Enter/Backspace/ Delete, and printable ASCII; shell metacharacters (|, &, ;, $, backticks), terminal-reconfiguring escape sequences, and tmux prefix sequences (Ctrl+B) are rejected. This restriction applies only to programmatic send-keys — a human holding the attach lease via the PTY relay has unrestricted keyboard input, by design.

Explicit NEVER list (from the same source): never run kill-server without -L pam; never use glob patterns in session targeting (kill-session -t "pam-*"); never target a session from an unvalidated user-provided path or workdir; never pass unvalidated input to any -t flag; never send arbitrary shell commands via send-keys; never parse pane output as structured completion data; never access sessions on the default socket.

Secret Handling in Provider Env Injection

Provider adapters substitute {api_key} into a provider CLI’s argv template or environment at spawn time — described as “API key from environment or secure storage” (020-provider-adapters.md § Capability Declaration). Neither the concrete “secure storage” mechanism nor a log- redaction policy for the substituted value is defined anywhere in the current docs set. This is a genuine gap, not a documented default — tracked as OQ-17 below rather than assumed.

PostgreSQL Credential Confinement

Database credentials live on exactly one host: workers never connect to PostgreSQL directly, and all writes flow through the central API/uplink (002-data-model.md § Concurrency Model). The connection DSN is configured once, in the orchestrator’s config.toml [database] section (060-configuration.md); no worker-side config key for it exists.

Shell Least-Privilege (Tauri Desktop Shell)

The desktop shell is a monitoring-only thin client — never an execution node — so it needs no local filesystem or shell capabilities. Tauri v2 capabilities ship with everything disabled: no invoke, no Electron-style ipcRenderer/remote equivalents used from SPA code (050-ui-dashboard.md § Desktop Shell). All user actions are forwarded as API requests only; the orchestrator and its workers perform all actual work.

Token Lifecycle (OQ-10, OQ-14 — confirmed 2026-09-01)

Worker auth model (OQ-10)

Confirmed default: a pre-provisioned worker token over TLS, not mTLS. pam host add mints the token and prints it once; the worker stores it in PAM_WORKER_TOKEN or its local worker-token file and presents it on every registration and heartbeat (030-http-api.md, 040-cli.md). mTLS was rejected as the default because it does not solve the problem OQ-14 actually addresses: a cloned mTLS client certificate aliases a host just as effectively as a cloned bearer token — the two concerns (transport authentication vs. detecting two live processes presenting the same credential) are orthogonal, and mTLS adds certificate provisioning and rotation overhead without closing the OQ-14 gap.

Token rotation procedure (OQ-10)

Run pam host rotate-token <name> (040-cli.md) to mint a fresh token for an already-registered host:

1. The server generates a new token and stores its fingerprint on hosts.token_fingerprint, replacing the previous fingerprint.
2. The new token is printed once, the same one-time-display contract as pam host add. Copy it into the worker host’s worker-token file (or PAM_WORKER_TOKEN) immediately — it is not retrievable again.
3. The old token keeps authenticating until the worker process actually restarts with the new one — there is no server-side grace-period timer and no forced cutover — so a worker left un-restarted after rotation simply keeps using its old (still-valid) token. Restart the worker promptly after rotating to complete the swap.

Rotate a host’s token when: a token may have leaked (compromised host, committed secret, exposed log), as routine credential hygiene, or when retiring a host (rotate, then pam host disable, so a copy of the old token elsewhere stops working).

Instance-suspect clearing (OQ-14)

pam host clear-suspect <name> (040-cli.md) is the operator side of the OQ-14 clone-detection algorithm documented in 002-data-model.md: it clears hosts.instance_suspect and hosts.instance_flagged_at, and accepts whichever instance_id that host most recently presented as the new pinned value. Because clearing the flag also re-pins trust, only run it after confirming out of band (checking which machine is actually supposed to be running that worker token) which of two live presenters is legitimate — clearing the flag on the wrong presenter re-admits the clone to dispatch.

Open Questions

• OQ-17: what is the concrete “secure storage” mechanism for provider API keys ({api_key} substitution), and is there a log-redaction policy for the substituted value? Registered in 071-risks-and-open-questions.md.

Observability

Status: proposed — consolidated 2026-09-01 from material already designed in 010-tmux-executor.md, 030-http-api.md, 002-data-model.md, and 050-ui-dashboard.md. No new design decisions were made here, except where flagged as a new Open Question below.

Scope

Run logs on executing hosts, the central events journal as an audit/activity feed, host liveness (worker heartbeats, last_seen_at), orchestrator/worker tracing (tracing crate usage), and dashboard live views.

Run Logs

Each run keeps an on-disk log on the worker host that executed it — <platform data dir>/runs/<run_id>.log — with raw pane text appended on every poll (010-tmux-executor.md § Disk Persistence). This is deliberately the audit trail of record: run output text is never written to PostgreSQL (§ Data Retention, 002-data-model.md), only structured metadata about it (line count, byte count, timestamp) is journaled as an event.

GET /runs/{id}/logs serves this log through the orchestrator, which never reads the worker’s filesystem directly — the worker relays it over its resident uplink connection as a Log chunk frame ({type: "log_chunk", run_id, text, timestamp}, 030-http-api.md § Worker Uplink), appended incrementally as new text is captured.

Events Journal

The append-only events table is the audit/activity-feed backbone. Topic enumeration, by domain (002-data-model.md § Event Topics):

• Project management: project.created, project.updated, project.archived
• Work items: work_item.created, work_item.updated, work_item.status_changed, work_item.reparented, work_item.deleted
• Runs: run.created, run.spawning, run.started, run.finished, run.failed, run.stopped, run.signal_received, run.output_appended (metadata only, never full text), run.lease_acquired, run.lease_released, run.orphan_detected
• Workers / hosts: worker.registered, worker.offline, worker.instance_suspect, worker.instance_reinstalled

Every run.* payload carries attempt alongside run_id, so a consumer can tell which incarnation of a restarted run an event describes; run.finished and run.failed never overlap for the same exit (one rule, no overlap — see 002-data-model.md § Event Topics for the exact split). Events are retained 30 days by default ([retention] events_days in config.toml, 060-configuration.md), purged best-effort at orchestrator startup.

Host Liveness

Workers send a heartbeat frame every 30 s over their resident uplink connection; the orchestrator marks a host offline after 90 s without one (3 missed beats) and updates hosts.last_seen_at on every heartbeat regardless of derived status (030-http-api.md § Worker Uplink). This 90 s wire threshold is distinct from the worker’s own local reconciliation loop (30 s) and liveness poll (10 s), which never cross the wire (010-tmux-executor.md).

Status is derived, not stored as a free choice: registered means the operator created the host row and no worker has heartbeated yet; online/offline are heartbeat-derived; disabled is sticky — a heartbeat from a disabled host still updates last_seen_at but never flips status, and only POST /hosts/{id}/enable clears it (resolving to online or offline depending on whether last_seen_at is inside the liveness window) (030-http-api.md).

Two liveness-adjacent events (OQ-14, confirmed — see 061-security.md § Instance-suspect clearing) round out the picture: worker.instance_reinstalled (info-level, silent instance_id update) and worker.instance_suspect (host excluded from new dispatch until an operator runs pam host clear-suspect).

SSE Event Stream

GET /api/v1/events/stream is the activity-feed transport (030-http-api.md § Server-Sent Events):

• Optional topic query parameter, glob-capable (run.*)
• Payload envelope: id: <seq> / event: <topic> / data: <json>
• Reconnect via Last-Event-ID header — server resumes from seq + 1; if that seq was already purged, it sends a full snapshot then resumes live
• No server-sent heartbeats on this channel — reconnection relies on TCP keepalive

Dashboard Live Views

The SPA consumes the SSE stream directly for its global Activity Feed drawer — one entry per event, showing timestamp, topic, a clickable resource link, and a short contextual description, filterable by topic prefix (project.* / work_item.* / run.*) (050-ui-dashboard.md § Activity Feed). The connection is a plain web_sys::EventSource against /api/v1/events/stream, with the same Last-Event-ID reconnect cursor described above.

Live terminal panes are a separate transport (WebSocket, not SSE) — see 050-ui-dashboard.md § Live Pane Rendering for the snapshot/coalescing detail, cross-referenced but not duplicated here since it is UI rendering, not journal/audit observability.

Open Questions

• OQ-18: no orchestrator/worker structured-logging (tracing crate) design exists anywhere in the docs set — log levels, output format (plain vs JSON), and whether/how RUST_LOG or an equivalent is exposed are all undecided. Registered in 071-risks-and-open-questions.md.

Testing Strategy

Status: proposed — consolidated 2026-09-01 from material already designed in 004-implementation-reference.md, 070-roadmap.md, and .github/workflows/ci.yml. No new design decisions were made here.

Scope

Test pyramid for the workspace: unit (pam-core), Store parity (PgStore vs MemStore), PostgreSQL dialect/migration tests against an ephemeral container, worker/executor integration against a real tmux, partition-path tests (orchestrator stop/start with signal spool replay), API contract tests, and the runnable phase gates.

Quality Gates

Four gates, run in this order, all must exit 0 — enforced locally before every commit and as the CI test job (005-engineering-rules.md § Commit Gate, 070-roadmap.md § Gate conventions):

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo build --workspace
cargo test --workspace

No coverage percentage is gated — cargo-llvm-cov output is generated as an informational artifact only (see § CI Workflow below), not enforced as a pass/fail threshold.

CI Workflow (.github/workflows/ci.yml)

Three independent jobs on every push/PR to main (004-implementation-reference.md § CI):

• test — the four Quality Gates above as separate steps, toolchain 1.93.1, Cargo.lock-keyed cache. From Phase 1 this job additionally needs an ephemeral PostgreSQL service container for PgStore tests — not yet added, since the scaffold is still pre-ADR-012-pivot (SQLite-backed); code catches up starting Phase 1 (004-implementation-reference.md § What Exists).
• docs — frontmatter presence (implementation: key, README.md navigation files exempt) and relative-link / .mermaid-reference resolution, checked with a stdlib-Python script; a best-effort mermaid-cli parse step runs continue-on-error (Node flakiness must not redden the job, but a genuine syntax error still shows in the step log).
• coverage — cargo-llvm-cov --workspace --lcov uploaded as a 30-day-retained lcov artifact, plus a per-crate summary printed to the log. Informational only (see § Quality Gates above).

Test Pyramid by Phase

Each roadmap phase (070-roadmap.md) ends at a gate expressed as a runnable command sequence or an observable outcome — never a vague quality adjective. Summarized here; the roadmap doc is the source of truth for exact gate steps.

• Phase 0 (scaffold) — 5 crates, each compiling with a smoke test; the four Quality Gates pass workspace-wide.
• Phase 1 (PM domain) — Store parity: the shared, parameterized Store trait test suite runs against both PgStore and MemStore and passes identically; pam task add/list/move/done round-trips against a real PostgreSQL instance (ephemeral container acceptable).
• Phase 2 (worker + tmux executor) — the widest integration surface, five paths against a real tmux: crash-path reconciliation (no duplicate or orphaned sessions after a worker restart), partition path (stop the orchestrator mid-run, restart it — spooled run.started/run.finished signals replay and the run record converges with no manual repair), duplicate-delivery idempotency (the same one-shot signal POSTed twice yields exactly one transition and one journalled event), stale-attempt rejection (a signal stamped with an old attempt changes neither runs nor events), and the tmux safety-boundary path (decoy sessions on both the pam and default sockets survive pam run stop/cleanup untouched).
• Phase 3 (web SPA) — an embedding test (the built binary serves the SPA with web/dist renamed away on disk, proving embedding rather than static-file serving) plus a browser-observable check (board + live pane render with real data).
• Phase 4 (provider adapters) — end-to-end: two real providers each complete a scripted task with distinct completion-signal sources (wrapper-journal vs provider-native hook) visibly different in the run record; pam provider doctor exits 0 on the dev machine.
• Phase 5 (interactive attach) — the single-writer lease contract: one browser’s keystrokes reach the tmux pane and stream to a read-only second viewer; a second concurrent write attempt is rejected with a clear lease error and contributes no input.
• Phase 6 (desktop shell) — the built .app/dmg launches, reaches the same core, and passes the Phase 3 gate through the shell webview; cargo-binstall metadata validates locally (dry run).

Gate Conventions

• All commands run from the repository root on the dev machine (macOS — the primary worker platform per ADR-011; Linux is verified per OQ-9 when an environment is available).
• From Phase 1 on, every gate assumes a reachable PostgreSQL (a local ephemeral container is acceptable) and a locally running orchestrator
  • worker (pam serve + pam worker on the same machine is the dev shape of the multi-host topology).
• CI runs the headless subset only — fmt, clippy, build, test, docs checks. Browser- and tmux-interactive checks (Phases 2, 3, 5, 6) are verified manually at phase exit, not automated in CI.
• A phase flips to done only when every item in its gate has passed; partial completion keeps it in-progress with a note.

Open Questions

• None registered from this consolidation — no contradiction or undocumented decision was found across 004-implementation-reference.md, 070-roadmap.md, and .github/workflows/ci.yml.

Packaging and Desktop Shell

Status: proposed — consolidated 2026-09-01 from material already designed in 001-system-overview.md, 050-ui-dashboard.md, 060-configuration.md, 061-security.md, and 070-roadmap.md. No new design decisions were made here, except where flagged as a new Open Question below.

Scope

Distribution of the single pam binary (cargo-binstall, release process) in its two roles — central orchestrator (pam serve, needs a PostgreSQL DSN) and worker (pam worker) — for the supported worker platforms: macOS (primary) and Linux (secondary); Windows out of scope, mobile excluded (ADR-011/ADR-012). Includes the confirmed Tauri v2 desktop shell (ADR-006; Electron recorded as fallback).

Distribution

One pam binary, two roles selected by subcommand — pam serve (central orchestrator: axum HTTP API + embedded SPA + run dispatch + worker registry, needs a reachable PostgreSQL DSN) and pam worker (outbound-only registration, no inbound listener, supervises tmux sessions on the host’s dedicated pam socket) — the same binary can run both roles on one machine to recover a single-machine deployment (001-system-overview.md § Distribution Topology). Each role reads its own platform data dir (config.toml/token for the orchestrator, worker.toml/provider catalog/per-run logs for a worker), resolved via the directories crate — full paths and precedence rules are 060-configuration.md § Platform Data Dir, not duplicated here.

Platform Targets

Accepted 2026-08-29 (ADR-011): worker hosts target Unix only — macOS primary, Linux secondary; Windows is out of scope (no tmux), mobile is excluded. Code and CI stay platform-neutral (CI already builds and tests on Linux, ubuntu-latest); the interactive tmux gates (spawn, reconcile, attach) remain macOS-verified until a Linux environment is set up for manual verification (OQ-9, 071-risks-and-open-questions.md). The central orchestrator itself has no platform restriction beyond PostgreSQL reachability — ADR-011 binds worker hosts specifically, not pam serve.

Production Build

trunk build --release compiles the Leptos/WASM SPA into web/dist/; rust-embed includes that directory in the compiled binary; Axum serves the embedded files at / (index.html → /, assets/* → /assets/*, all non-API routes fall back to index.html for SPA routing) while API routes stay under /api/v1 unaffected (050-ui-dashboard.md § Production Build). This is what the Phase 3 embedding test proves: renaming web/dist away on disk after the binary is built still serves the SPA, showing embedding rather than static-file serving (063-testing-strategy.md § Test Pyramid by Phase).

Desktop Shell (Tauri v2)

Accepted 2026-08-29 (ADR-006): a Tauri v2 shell around the same embedded SPA, confirmed as a monitoring/viewing thin client that never executes work — every user action is forwarded as an API request only, and the orchestrator + its workers perform all actual work (050-ui-dashboard.md § Desktop Shell). Target platforms mirror the worker split: macOS + Linux desktops, Windows out of scope, mobile excluded; the shell is platform-independent by construction since it just loads the orchestrator URL.

Rationale for Tauri v2 over Electron: web-first (the embedded SPA already works standalone) suffices for single-operator local use; OS webview reuse costs 5-10 MB versus a bundled Chromium’s ~150 MB; the shell reuses the existing Rust core with no Node.js sidecar; and the security surface drops a Node.js runtime entirely, which matters because the shell needs no local capabilities. Electron is a recorded fallback for an ecosystem gap (e.g. a macOS auto-update issue), not a current alternative — the SPA uses only standard web APIs, so either shell stays viable without a rewrite.

Security posture (least privilege): Tauri v2 capabilities ship with everything disabled — no invoke, no Electron-style ipcRenderer/ remote equivalents used from SPA code (061-security.md § Shell Least-Privilege). This follows directly from the monitoring-only role: a client that never executes work locally needs no filesystem or shell capability to disable-by-exception, so the default-disabled posture is the complete posture, not a starting point to relax later.

Implementation is scheduled for Phase 6, not yet built: the Tauri app loads PAM_API_URL, optionally spawns pam serve as a child process (sidecar mode, process management only — the sidecar is still the same pam serve binary, not shell-specific code), bridges IPC for tray icon / notifications / window management, and reuses the Rust core directly (050-ui-dashboard.md § Implementation Note).

Phase 6 Packaging Gate

070-roadmap.md § Phase 6 scopes two deliverables: implementing the Tauri v2 shell as a wrapper (sidecar daemon or linked core; the web UI codebase stays single-source, no shell-specific fork), and cargo-binstall metadata plus release polish (app icon, naming, packaging defaults). The phase gate: a built .app/dmg launches on the dev Mac, starts or attaches to the same core, and passes the Phase 3 embedding gate through the shell webview; cargo-binstall metadata is present and validates locally via a dry run (no network publish required) — cross-ref 063-testing-strategy.md § Test Pyramid by Phase for how this gate fits the overall test pyramid. As of this consolidation, no cargo-binstall metadata exists yet in the workspace Cargo.toml files — Phase 6 has not started in code.

Open Questions

• None registered from this consolidation — no contradiction or undocumented decision was found across 001-system-overview.md, 050-ui-dashboard.md, 060-configuration.md, 061-security.md, and 070-roadmap.md. The earlier apparent ADR-006/ADR-011/ ADR-012 discrepancy in 003-architecture-decisions.md was a false alarm from a plain-text search of that file — the ADR index’s actual content lives in an embedded SVG table (docs/assets/tables-src/003-architecture-decisions/table-01.md), which does record all three ADRs correctly; no doc content was found missing.

Roadmap

Canonical sequencing for project-agent-manager (binary pam). Each phase delivers a working increment and ends at a gate expressed as commands to run or outcomes to observe — never as a vague quality adjective. Beyond Phase 0, dates are deliberately omitted: the contract here is ordering and gates, not scheduling.

Companion doc: 071-risks-and-open-questions.md (risk register, open questions, decision schedule).

1. Product frame (context only)

pam is a multi-host, central-orchestrated platform that combines hierarchical project/task management (Project → Milestone/Epic → Task → Subtask; board columns todo/doing/done/cancelled) with AI task orchestration driving subscription LLM CLIs (claude/codex/grok/gemini) inside tmux sessions on each execution host (dedicated socket pam, session names pam-<8hex>), live-observable from a web dashboard embedded in the orchestrator binary via rust-embed. A central orchestrator (pam serve) owns the API, dashboard, and PostgreSQL; workers (pam worker) on Unix hosts (macOS primary, Linux secondary — ADR-011) execute the sessions over outbound-only connections (ADR-012). Windows and mobile are excluded. Headless -p execution is excluded by design. Details live in 001-system-overview.md and 030-http-api.md / 040-cli.md — this file does not restate them.

2. Phase overview

Status legend: planned / in-progress / done / superseded.

3. Dependencies

graph LR
    P0["Phase 0 — scaffold + docs"] --> P1["Phase 1 — PM domain"]
    P1 --> P2["Phase 2 — tmux executor v1"]
    P1 --> P3["Phase 3 — web SPA v1"]
    P2 --> P4["Phase 4 — provider adapters"]
    P2 --> P5["Phase 5 — interactive attach"]
    P3 --> P5
    P5 --> P6["Phase 6 — desktop shell + packaging"]

In words:

• Phase 1 requires Phase 0 (scaffold to build on).
• Phase 2 requires Phase 1 (runs reference tasks/work items in the store).
• Phase 3 requires Phase 1 (SPA renders PM data). It does NOT require Phase 2: the live-pane gate can be exercised against any session on socket pam, including one started by hand — though in practice Phase 2 will usually be available.
• Phase 4 requires Phase 2 (adapters execute through the worker’s executor).
• Phase 5 requires Phases 2 + 3 (worker sessions to attach to; SPA to host xterm.js).
• Phase 6 requires Phase 5 (the shell wraps the full interactive product; the shell choice is settled — Tauri v2, ADR-006 — but is not exercised before then).

4. Phase details

Phase 0 — Foundation: docs, scaffold, CI (current)

Status: in-progress (started 2026-08-29).

Note (2026-08-29, later same day): the design pivoted to the multi-host topology — PostgreSQL backend, central orchestrator + per-host workers (ADR-012). Phase 0 deliverables (docs, compiling scaffold, CI) keep their shape; the store crate adopts PostgreSQL and the worker command appears at Phase 1/2 as written below.

Scope:

• Docs tree: 001-system-overview.md, 002-data-model.md, 010-tmux-executor.md, 020-provider-adapters.md, 004-implementation-reference.md; 030-http-api.md and 040-cli.md; 050-ui-dashboard.md; roadmap + risks (this pair). Four mermaid diagrams under docs/assets/diagrams/.
• Cargo workspace scaffold: 5 crates (pam-core, pam-store, pam-executor, pam-api, pam-cli), each compiling with a smoke test.
• CI workflow running the full verification chain on every commit.

Gate (runnable, all must exit 0):

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo build --workspace
cargo test --workspace

Plus observable outcomes: docs link/frontmatter checks pass; git log --oneline main shows the Phase 0 commits. Commits are local only — push is deferred to September 2026 (OQ-1 in the risks doc); the remote URL is unconfirmed until then.

Phase 1 — PM domain

Scope:

• pam-core: domain types for the hierarchy (Project → Milestone/Epic → Task → Subtask) and statuses (todo/doing/done/cancelled); board column semantics (a move is a status change).
• pam-store: sqlx migration 0001 implementing the PostgreSQL schema from the 002-data-model.md; Store trait; PgStore; MemStore gated behind a test-support feature.
• pam-cli + pam-api: pam serve, pam migrate, pam project add/list/show/archive, pam task add/list/show/move/board/done — all commands through the central API (ADR-007).

Gate (runnable):

• Against a PostgreSQL instance (ephemeral container acceptable): pam task add → pam task list → pam task move → pam task done round-trips; pam task show reflects each transition.
• Parity: the shared, parameterized Store test suite runs against both PgStore and MemStore and passes identically.
• pam task board renders the todo/doing/done/cancelled columns with tasks placed by status.

Exit note: the pam binary/crate prefix freezes at Phase 1 exit (OQ-2); blocked-as-label and hierarchy depth defaults (OQ-6) must hold or be overturned by then.

Phase 2 — worker + tmux executor v1

Scope:

• Worker agent: pam worker registers outbound with the orchestrator (worker token), heartbeats, and executes dispatched runs; pam host add/list/disable/enable manages the registry.
• Spawn: tmux -L pam new-session -s pam-<8hex> with pam exec run <id> as the session’s foreground wrapper process; the wrapper fetches its run spec from the central API and spools signals across partitions.
• Persisted session registry mapping runs to hosts and session names; reconciliation on worker start adopts known sessions and flags unknown pam-* ones; orchestrator marks hosts offline on heartbeat loss without force-failing their runs.
• pam run start/list/stop/logs/pane and web-relayed attach; local CLI attach only for runs on the local host.
• tmux safety 4-rules enforced in code, per the 010-tmux-executor.md: dedicated socket pam only (never the default socket); session-name regex ^pam-[0-9a-f]{8}$ validated before any tmux command; cleanup limited to own sessions on socket pam; send-keys allowlist.

Gate (runnable):

• Start a run with a stub echo-CLI provider: pam run list shows it with its host, pam run pane shows its output, tmux -L pam list-sessions on that host shows exactly the pam-<8hex> session.
• Crash path: kill -9 the worker, restart it; pam run list reconciles — every run appears exactly once (no duplicates) and every pam-* session on socket pam is accounted for by the registry or cleaned up (no orphans).
• Partition path: stop the orchestrator mid-run, restart it; the wrapper’s spooled run.started/run.finished signals replay and the run record converges without manual repair.
• Duplicate-delivery path: POST the same one-shot signal (e.g. run.finished) twice; the run record shows exactly one status transition and the events journal holds exactly one journalled event for it — signal delivery is idempotent, no phantom transition.
• Stale-attempt path: POST a signal stamped with an attempt older than the run’s current runs.attempt (a dead incarnation’s late replay); afterwards both the runs row and the events journal are unchanged — no status write, no event emitted.
• Safety path: with a decoy session present on socket pam under a non-conforming name, and an unrelated session on the default socket, pam run stop/cleanup leaves both decoys untouched (verify with tmux -L pam list-sessions and tmux list-sessions).

Phase 3 — web SPA v1

Scope:

• Leptos scaffold under web/ (a Cargo workspace member); trunk build --release output web/dist embedded into pam-api via rust-embed.
• axum serves the SPA, the HTTP API per 030-http-api.md, and an SSE endpoint streaming the read-only pane.
• UI: projects list, board, task detail, live read-only pane.
• Auth posture: bearer token over TLS (users/CLI) + separate worker token (OQ-7 superseded by ADR-012; see OQ-10) — no accounts.

Gate (runnable/observable):

• cargo build --workspace succeeds with web/dist embedded: run the built binary with web/dist renamed away on disk — the SPA still serves (proves embedding, not static-file serving).
• Browser at 127.0.0.1:PORT shows the board populated with real data and a live-updating pane from a running session (a session started via pam run start, or any session on socket pam created by hand).

Phase 4 — provider adapters

Scope:

• Provider catalogs for claude/codex/grok confirmed + gemini optional if installed as TOML (argv templates, env mapping, hook config, version constraints), per the 020-provider-adapters.md.
• Env normalization + proxy injection applied by the pam exec run wrapper pre-spawn: ANTHROPIC_BASE_URL / OPENAI_BASE_URL / GEMINI_BASE_URL plus HTTP(S)_PROXY.
• Completion ladder in code: manual > wrapper journal > provider-native hooks POSTing /api/internal/runs/{id}/signals; the pane-idle heuristic is a UI hint only. Provider hooks ship behind a config flag (off by default).
• Restart policy: bounded retries with backoff, configurable per provider.
• pam provider doctor: CLI presence/version against catalog constraints, catalog validity, tmux >= 3.3.

Gate (runnable):

• Two real providers each complete a scripted task end-to-end from pam run start to a recorded completion, with DISTINCT completion signals: one run’s completion captured via wrapper exit (journal), the other via a provider-native hook POST — the signal source is visible and different in the two run records.
• pam provider doctor exits 0 on the dev machine.

Exit note: task-status auto-transition on a trusted signal stays config-gated and OFF by default unless OQ-5 is overturned by this gate.

Phase 5 — interactive attach

Scope:

• WS attach endpoint + PTY relay: portable-pty hosting tmux -L pam attach for the target session.
• Single-writer lease per run: acquire/renew/release; lease state visible via the API; clear rejection error for a second writer.
• xterm.js client rendering bidirectional IO; read-only viewers multiplex the same outbound stream with no input path at the protocol level; activity feed UI.

Gate (runnable/observable):

• Browser A attaches to a run and types; the keystrokes are visible in the tmux pane (verify via tmux -L pam capture-pane -p or by observing output) and stream to Browser B, which remains read-only.
• While A holds the lease, an attach attempt from Browser C (or a re-attach attempt from B) is rejected with a clear lease error (observable HTTP/WS error) and no input flows from the rejected client.

Phase 6 — desktop shell + packaging

Scope:

• Implement the Tauri v2 shell (decision accepted 2026-08-29 — monitoring thin client, never an execution node; comparison record in 050-ui-dashboard.md; formerly OQ-4, resolved). Shell targets macOS and Linux desktops, matching the execution node (ADR-011); the macOS build ships first and the Linux build follows the Linux execution-node verification (OQ-9).
• Implement the chosen shell as a wrapper: sidecar daemon or linked core — the web UI codebase stays single-source.
• cargo-binstall metadata; release polish (app icon, naming, packaging defaults).

Gate (runnable/observable):

• The built .app (or dmg) launches on the dev Mac, starts or attaches to the same core, and passes the Phase 3 gate through the shell webview: board and a live-updating pane visible in the shell window.
• cargo-binstall metadata is present and validates locally (dry run, no network publish required).

5. Deferred / backlog (explicitly out of scope for now)

6. Gate conventions

• All commands run from the repository root on the dev machine (currently macOS — the primary worker platform per ADR-011; the secondary Linux target is verified per OQ-9 when a Linux environment is available).
• From Phase 1 on, gates assume a reachable PostgreSQL (a local ephemeral container is acceptable) and a locally running orchestrator + worker (pam serve + pam worker on the same machine is the dev shape of the multi-host topology).
• Where a gate is an observable outcome (browser, tmux pane), the observation is the verification; record the result by updating the phase status and last_verified in this frontmatter.
• CI runs the headless subset of gates (fmt, clippy, build, test, docs checks). Browser- and tmux-interactive checks are verified manually at phase exit.
• A phase flips to done only when every item in its gate has passed; partial completion keeps it in-progress with a note.

References

• 001-system-overview.md — system architecture
• 002-data-model.md — schema and store design
• 010-tmux-executor.md — executor and tmux safety rules
• 020-provider-adapters.md — catalogs, env injection, completion ladder
• 030-http-api.md, 040-cli.md — interface contracts
• 050-ui-dashboard.md — dashboard IA and shell comparison
• 071-risks-and-open-questions.md — risk register and open questions

Risks and Open Questions

Companion to 070-roadmap.md. Two artifacts live here:

1. The risk register — reviewed at every phase gate; likelihood/impact are updated from observed evidence, not from anxiety.
2. The open questions — each carries a working default that holds until explicitly overturned. Overturning a default requires updating this table and every affected design doc in the same commit.

1. Risk register

Likelihood and impact use a simple Low / Medium / High scale. The mitigation column names the enforcing mechanism: a design rule, a command check, or a specific phase gate in 070-roadmap.md.

Notes on the three structural risks

• R1 (pane contract). This is the highest-leverage discipline in the project: every temptation to “just grep the pane” for completion or status must be redirected to the signal ladder. The pane-idle heuristic exists only to tint a UI badge, and the enforcement point is code review plus the design rule in the tmux-executor doc — there is no lint for intent.
• R3 (provider drift). Likelihood is rated High because subscription CLIs update themselves on their own schedule; the register assumes at least one breaking auto-update per provider per year. The TOML catalog exists largely for this risk: re-pinning or re-templating argv must never require a Rust release.
• R10/R11 (partition and transport). These replaced the old R9 scope guard when the design pivoted to multi-host (ADR-012). The partition risk is bounded by making the worker self-sufficient (local run cache + signal spool) rather than by trying to keep connections alive; the transport risk is bounded by token-class separation and operator-provisioned workers (a worker can never self-register).

2. Open questions

Defaults are current working assumptions, not decisions. “Blocks phase” names the phase whose design must freeze the answer; “none” means the default can hold indefinitely without rework.

3. Decision schedule

When each open question must actually be answered, relative to the roadmap in 070-roadmap.md:

Governance rule: an overturned default is not complete until this table, the affected design doc(s), and any already-written code or contract move in the same commit. A default changed in conversation but not in this file has not changed.