tmux Executor

The tmux executor runs inside the worker (pam worker) on each execution host. The worker receives dispatch from the central orchestrator over its outbound connection; all tmux mechanics below are local to the worker host. The central orchestrator never invokes tmux.

Session Identity & Naming

The tmux executor uses a dedicated socket pam to isolate all project-agent-manager sessions from the user’s default tmux instance. Every tmux invocation uses tmux -L pam ... to ensure complete operational separation.

Session naming: pam-<short_run_id> where <short_run_id> is the first 8 lowercase hexadecimal characters of the run UUID. This format provides blast-radius isolation while maintaining traceability.

Window titles: Set to the task slug for human-readable identification in tmux session lists.

Rationale for dedicated socket: A separate socket prevents accidental interference with the user’s personal tmux sessions. If the worker crashes or is killed, the operator can still manually inspect orphaned sessions on that host via tmux -L pam list-sessions without affecting the default tmux server.

Name validation: Before ANY tmux invocation, session names are validated against the regex ^pam-[0-9a-f]{8}$. This prevents command injection and ensures we only operate on sessions we created.

Spawn Path

Initial Session Creation

When a run transitions from pending to spawning (the central orchestrator dispatches it to a host), the worker on that host creates a new tmux session:

tmux -L pam new-session -d -s pam-<short_run_id> -c <workdir> -x 220 -y 50

Parameters:

• -d: Detach from session (the worker operates headless)
• -s: Session name with pam- prefix
• -c: Initial working directory from run configuration
• -x 220 -y 50: Initial pane dimensions (220 columns × 50 rows)

Foreground Command

The session’s foreground command is NOT the provider CLI directly. Instead, it runs our own wrapper:

pam exec run <run_id>

This wrapper is responsible for:

1. Reading run configuration: Fetch the run spec from the central orchestrator API (the worker injects PAM_API_URL and the run’s auth context into the wrapper environment; the wrapper never holds PostgreSQL credentials — only the central orchestrator connects to the database). The spec includes provider, arguments, environment variables, and working directory.

2. Provider configuration lookup: Load provider details from the host’s local TOML catalog (binary path, argv template, environment mappings, hooks config) — provider CLIs are installed per host, so the catalog lives on the worker host.

3. Environment normalization and injection:

   • Apply environment mappings from the provider catalog
   • Inject proxy/gateway base URLs (ANTHROPIC_BASE_URL, OPENAI_BASE_URL, etc.) if configured
   • Apply standard HTTP(S)_PROXY variables
   • This injection point ensures traffic stays proxied even during interactive attach

4. CLI process spawning: Execute the provider CLI with normalized arguments and environment.

5. Lifecycle signal reporting:

   • After successful spawn: POST a started signal to the central orchestrator at $PAM_API_URL/api/internal/runs/{id}/signals (the orchestrator journals the corresponding run.started event — the wrapper posts signals, never writes the journal itself)
   • After CLI exit: POST a finished signal with the exit code to the same endpoint
   • Attempt and sequence stamping: the wrapper stamps attempt on every signal and, for progress signals, additionally a monotonic seq — both at emission time. The attempt is read from the injected spawn environment (the worker injects the centrally allocated attempt alongside PAM_API_URL); seq is carried only by repeated kinds (progress) — one-shot kinds (started, finished, failed) carry no seq, being deduplicated centrally by (run_id, attempt, kind)
   • If the central orchestrator is unreachable, the wrapper retries with backoff and writes undeliverable signals to the worker-side spool on disk. Because the wrapper is short-lived (it exits after hold-open), the worker — not the wrapper — owns the spool and replays queued signals once the uplink recovers; signals are not dropped. The spool preserves each signal’s attempt/seq verbatim, and the worker MUST NOT renumber signals at replay time — renumbering at replay would make every replayed signal look newest and fabricate ordering that never happened. Ordering and dedup are enforced centrally, at the signal-ingest edge, never worker-side: the worker never queries the database (see 002-data-model.md, processed_signals)

6. Hold-open behavior: After the CLI exits, the wrapper keeps the tmux session alive for a configurable period (default 5 minutes) to allow final output capture and manual inspection. The session displays a message indicating the run has completed.

Session-to-Run Binding

The wrapper identifies its run ID from the command-line argument and uses this to:

• Read its run configuration from the dispatch payload assembled by the worker (run record fields — provider, cwd, task reference — are passed at spawn time as arguments/environment; the wrapper never talks to PostgreSQL, which lives only on the central host per ADR-012)
• Post lifecycle signals to the correct run record
• Include the run ID in all API calls for correlation

Supervision & Reconciliation

Contrast with grok-fleet-orchestrator

In grok-fleet-orchestrator, the GrokRunner supervises child processes directly:

• The fleet worker spawns grok as a direct child subprocess
• The worker owns the process lifecycle and can restart with backoff
• Process supervision uses tokio::process::Child with wait/kill semantics

In project-agent-manager:

• The tmux SERVER owns the processes, not the worker or the central orchestrator
• The worker only controls session creation/termination via tmux commands
• Process supervision is indirect via liveness queries to tmux

Liveness Monitoring

The worker polls its local tmux for liveness using format strings:

tmux -L pam display-message -p -t pam-<short_run_id> -F "#{session_dead}"
tmux -L pam display-message -p -t pam-<short_run_id> -F "#{pane_dead}"

• #{session_dead}: Returns “1” if the session no longer exists
• #{pane_dead}: Returns “1” if the pane has terminated (CLI exited)

These queries run on the liveness poll loop — a worker-local timer, default every 10 seconds. It is cheap by construction: display-message against the host’s own tmux socket, with no call to the central orchestrator. Its job is fast detection of a dead pane or a vanished session so the exited transition is not delayed by the slower reconciliation cadence.

The liveness poll loop is not the reconciliation loop. They are two separate timers with different costs and different periods:

The periods differ because the costs differ. Polling liveness only at the reconciliation period would triple worst-case detection latency for an exited CLI; running a full reconciliation at the liveness period would triple central fetch load for no added signal.

Reconciliation Loop

Trigger conditions:

• On worker startup (before registering with the central orchestrator)
• Periodic timer (default every 30 seconds — the reconciliation cadence, distinct from the 10-second liveness poll above)
• Manual reconciliation command

Reconciliation logic (the worker reconciles its host; the central orchestrator reconciles the fleet — runs whose host is offline):

1. Query desired state: Fetch this host’s runs with status running or spawning from the central orchestrator. For partition tolerance the worker maintains a local run cache persisted on disk (in its data dir, alongside the signal spool) — refreshed on every successful fetch, so desired state survives both a partition and a worker restart happening at the same time.

2. Query actual state: Execute tmux -L pam list-sessions -F "#{session_name}" and filter for names matching ^pam-[0-9a-f]{8}$.

3. Compare and reconcile (the local run cache is the comparison baseline; the worker itself never queries the database):

   • Orphan detection: tmux sessions not present in the local run cache → mark as orphaned, queue for cleanup
   • Missing sessions: Desired runs (centrally recorded, cached locally) without tmux sessions → central transitions the run to failed (if spawning) or to failed with failure_reason = session_lost (if running); where the restart policy applies, central then decides whether to requeue (failed → pending, incrementing attempt) or leave the run terminal
   • Zombie processes: Sessions with pane_dead=1 but run status still running → transition to exited based on the wrapper’s finished signal (which may arrive via spool replay) or timeout
   • Terminal-but-alive sessions: A session whose run record is terminal (exited/failed/stopped) or no longer desired, while the tmux session still lives — produced by stale signals from a previous incarnation, a stop that reached the database but never reached the worker, or manual recovery after a tmux server crash. The session’s 8-hex name prefix maps it back to its run, so this case is distinguishable from true orphans without widening the desired-state fetch. Cleanup routes through safety rule 3’s two-part verification, with the action chosen by the discrimination table below

Terminal-but-alive discrimination. “Terminal run, live session” has two distinct meanings, separated by the incarnation epoch (runs.attempt):

The split matters: killing every terminal-but-alive session indiscriminately would turn a long worker outage into an automatic slaughter of healthy CLI sessions the moment the worker returns.

Central Fleet Reconciliation

The loop above is host-local: each worker reconciles its own host against its local run cache. The central orchestrator separately reconciles the fleet — state no single worker can see. Central-side cases:

1. Host offline (partition). A host whose heartbeat is lost is marked offline; its runs are left untouched at partition time and reconcile on reconnect. The worker’s host-local reconciliation does NOT force-fail runs — any force-fail is the separate central action in case 3 below, never a worker-side behavior.

2. Host-online gating. Central-side inferences about a run are valid only while its host is online (heartbeat current). From the central seat, “no output, no signals” from a live host means the CLI hung; from an offline host it is observationally ambiguous — a partition and a hang are indistinguishable. Output-silence timeouts (the CLI-hang hard timeout, default 30 minutes) are therefore enforced centrally and fire only while the heartbeat is current; while a host is offline, the host-lost bound below governs instead.

3. Host-lost bound (T_dead). A host continuously offline for T_dead (measured from hosts.last_seen_at) is presumed lost: central force-fails that host’s non-terminal runs with failure_reason = host_lost. Confirmed default: 24 hours, configurable (OQ-12, 071-risks-and-open-questions.md). The generous bound is intentional: worker-down ≠ CLI-down — the tmux server owns the processes, not the worker, and surviving sessions are re-adopted on worker restart, so a short bound would convert routine worker downtime into run loss. These host_lost tombstones are requeue-eligible — see § Restart Policy above for the requeue-eligibility statement and its accepted trade-off.

4. Late signals after a host_lost tombstone. A replayed finished(attempt = N) arriving after the tombstone is journaled but does not resurrect the run — the tombstone wins. The operator sees the event and can act; automated resurrection is a possible later enhancement, not a v1 need.

Disabled Host — Worker-Side Behavior

Host disable is an operator exclusion from dispatch, not a shutdown. The API surface and the sticky/derived status semantics are owned by 030-http-api.md (Hosts registry — disable is drain-only); this section states only what the worker on a disabled host does.

A worker whose host is disabled behaves exactly as before, minus new work:

Drain completion is observed, not signalled: the operator watches the host’s active-run count fall to zero (GET /hosts/{id}). Nothing in the worker forces that count down — a disabled host with a long-running CLI stays busy until that CLI finishes on its own or an operator stops the run explicitly.

Restart Policy

This section owns restart policy only — trigger conditions, attempt caps, and backoff. The run status machine itself (the legal edge set, including the canonical restart-requeue edge failed → pending, which increments runs.attempt centrally) is canonical in 002-data-model.md; this document conforms to it and introduces no edges of its own.

Requeue-eligible trigger set (OQ-12, confirmed). All terminal failure_reason values are requeue-eligible, host_lost included — a run tombstoned because its host went dark for T_dead (24h default, § Central Fleet Reconciliation below) is requeued to another host exactly like any other failed run. This closes a documentation gap the restart policy previously left silent (this section named itself as owning “trigger conditions” without ever stating whether host_lost was in scope). The trade-off is accepted, not eliminated: combined with the Terminal-but-alive discrimination rule above — which deliberately does NOT auto-kill an epoch-matching live session found after a host_lost tombstone — a requeue can leave the original host still doing real (or idle) work under the old incarnation while a new incarnation runs elsewhere. That old session is invisible to any automatic signal path (its late signals are discarded as stale-epoch, and it is flagged in the UI rather than killed) until an operator notices the flag and cleans it up manually. Blocking the requeue instead would trade this operator-visible orphan risk for the opposite failure — every run on a host that goes dark stays stuck running/spawning forever with no terminal exit — which the working default already rejected as worse (see OQ-12 in 071-risks-and-open-questions.md).

Restart attempts are capped and follow exponential backoff:

• Maximum attempts: 3 (configurable per task)
• Initial delay: 10 seconds
• Backoff multiplier: 2×
• Maximum delay: 60 seconds

Graceful stop escalation: SIGTERM to pane’s process group → wait up to 10 seconds → SIGKILL to pane process (if still running) → then kill-session for session cleanup.

Kill-before-respawn: before creating a session for a requeued (respawned) run, apply the graceful-stop ladder above to any tmux session already holding the target session name, then proceed with new-session. The session name derives from the run UUID, so an existing session with that name is necessarily a previous incarnation of the same run — safety rule 3’s two-part verification passes by construction. This is mandatory, not polish: cumulative restart backoff totals roughly 130 s even in the worst case, while the previous incarnation’s session survives far longer (wrapper hold-open defaults to 5 minutes after CLI exit; unmapped sessions are held for inspection for 1 hour), so the previous session still exists at every respawn point and the name collides deterministically — without cleanup-before-respawn, every attempt inside the window fails on new-session and the attempt budget is exhausted without a single spawn.

Orphan Handling

Orphan definition: An orphan is a session on socket pam named pam-* that cannot be mapped to a desired run by reconciliation. After worker restart, the run record usually still exists centrally (and in the worker’s local cache) and is recovered; only unmappable sessions are orphans and get cleaned.

Orphaned sessions are:

1. Logged to the events table with run.orphan_detected topic
2. Held for inspection for a configurable period (default 1 hour)
3. Cleaned up via tmux -L pam kill-session -t <session>

Observation

Capture-pane Polling

Terminal output is captured using tmux capture-pane:

tmux -L pam capture-pane -p -e -t pam-<short_run_id>

Parameters:

• -p: Output to stdout (pipe-readable)
• -e: Include escape sequences for ANSI formatting
• -t: Target session

Polling Cadence and Tradeoffs

Default cadence: Once per second (1 Hz) for active sessions. Note: This is the source rate for snapshots (emitted only on content change), while 10 fps is the coalescing cap for the interactive attach WebSocket relay layer—these two limits apply to different layers.

Tradeoffs:

• Higher frequency (2-4 Hz): More responsive UI, higher CPU load, more data transfer
• Lower frequency (0.2-0.5 Hz): Better resource efficiency, laggy terminal feel
• Adaptive: Throttle when no recent activity detected (idle threshold: 30 seconds)

ANSI Handling

The -e flag preserves ANSI escape sequences for colors and formatting. These are:

1. Stored verbatim in the worker’s snapshot cache and uplinked unchanged (the central cache holds the same verbatim bytes)
2. Served by the central orchestrator to WebSocket/SSE clients with appropriate content-type
3. Rendered by the web dashboard using xterm.js or similar terminal emulator

Snapshot Cache (two tiers)

Snapshot caching exists at two points in the pipeline, with different owners:

Worker-side cache (this document’s pipeline):

• Structure: In-memory cache storing the latest pane text per run, used for change detection — each poll compares against the cached snapshot and uplinks a new snapshot only when the content changed.
• Bounds: Maximum 1000 active snapshots, LRU eviction when exceeded.
• Purpose: Keeps the 1 Hz poll off the uplink — tmux is read locally at 1 Hz regardless, but the worker→orchestrator stream carries only changed snapshots.

Central snapshot cache (see 002-data-model.md retention policy and 030-http-api.md pane endpoints):

• The orchestrator holds the latest snapshot per run (received over the uplink) and serves GET /runs/{id}/pane, the read-only /pane/ws fanout, and reconnecting clients from it. Client fanout is a central concern — workers never serve clients directly.

Full snapshot cost (~220×50 ≈ 11,000 chars) is justified by eliminating incremental update complexity.

Cache entry lifecycle (both tiers):

• Created on first capture for a run
• Updated on each poll / each uplink receipt (replace, not append)
• Evicted on LRU when cache full or run completes
• Invalidated on run deletion

Disk persistence: Each run maintains an on-disk log at <platform data dir>/runs/<run_id>.log (~/Library/Application Support/pam/runs/ on macOS, ~/.local/share/pam/runs/ on Linux) with raw pane text appended on each poll. This provides audit trails and debugging material.

Streaming to SSE/WS

Two WebSocket endpoints serve terminal data:

1. Read-only viewer: /api/v1/runs/{id}/pane/ws

   • Multiplexes from the snapshot cache
   • Multiple concurrent viewers allowed
   • Receives full snapshots with sequence numbers (sent only when pane content changes, detected via hash compare)
   • No input channel (display-only)

2. Interactive attach: /api/v1/runs/{id}/attach/ws

   • Single-writer lease enforced
   • Bidirectional byte stream (PTY relay)
   • See “Interactive Attach” section below

Critical Design Constraint

Pane text is display-only, never parsed as a completion protocol. This is a hard lesson from grok-fleet-orchestrator’s ACP (Agent Completion Protocol): never treat a wire/screen format you don’t own as a contract. Completion signals must come from provider hooks or explicit user marks, not screen scraping.

Interactive Attach

Single-writer Lease

Interactive attach uses a lease system to prevent multiple writers from racing:

Lease acquisition:

• Requested via WebSocket upgrade to /api/v1/runs/{id}/attach/ws
• Lease holder stored in-memory with TTL (default 5 minutes)
• Lease refreshed on activity (heartbeat every 30 seconds)

Lease conflicts:

• Second attach attempt is rejected after WebSocket upgrade with error frame {"type":"error","code":"lease_held","holder":<current_holder_info>}
• UI can prompt to “force release” (requires confirmation and shows current holder)

Lease release:

• Explicit release on client disconnect
• TTL expiration on inactivity
• Manual revoke via API or UI

PTY Relay

The attach flow spawns a PTY relay process:

1. The worker spawns a subprocess running tmux -L pam attach -t pam-<short_run_id>
2. This subprocess uses portable-pty to create a PTY
3. The tmux attach process stdout/stdin are bridged to the PTY
4. The PTY byte stream is proxied bidirectionally over the WebSocket

Implementation: Uses portable-pty crate for cross-platform PTY spawning, with the subprocess wrapped in a tokio::process::Child for cleanup on WebSocket close.

Input Safety

Programmatic input via tmux send-keys is restricted to an allowlist for safety:

Allowlisted keys:

• Control sequences: Ctrl+C, Ctrl+D (SIGINT/EOF)
• Enter, Backspace, Delete
• Printable ASCII (0x20-0x7E)

Restricted:

• Shell metacharacters that could execute commands: |, &, ;, $, backticks
• Escape sequences that could reconfigure terminals
• tmux command sequences (e.g., Ctrl+B prefix)

Interactive attach: When a human holds the lease via the PTY relay, input is unrestricted (full keyboard and terminal control). The allowlist only applies to programmatic send-keys operations.

Read-only Viewer Multiplexing

Multiple read-only viewers can attach simultaneously:

• All receive the same pane text stream from the snapshot cache
• No coordination needed beyond cache invalidation on updates
• Each viewer tracks its own “last seen offset” for incremental updates

Safety Rules

The 4 Safety Rules

1. Never run kill-server on the default socket

   • The user’s personal tmux must never be affected
   • Only use tmux -L pam kill-server for catastrophic cleanup
   • All normal operations use session-specific commands

2. Validate session names before ANY tmux invocation

   • Regex: ^pam-[0-9a-f]{8}$
   • Prevents command injection via user input
   • Applied to all -t arguments and session list parsing

3. Cleanup only sessions we created

   • Two-part verification: a) Session exists in our database registry (runs table) b) Session name matches our prefix (pam-)
   • Orphan cleanup still respects prefix constraint
   • Never target user sessions, even if they somehow got on our socket

4. Restrict send-keys to allowlisted set

   • Programmatic input limited to safe keys
   • Prevents remote code execution via terminal injection
   • Interactive attach (human with lease) is unrestricted

Explicit NEVER List

• NEVER run tmux kill-server without the -L pam flag
• NEVER use glob patterns in session targeting (e.g., tmux kill-session -t "pam-*")
• NEVER target sessions based on user-provided paths or workdirs
• NEVER pass unvalidated user input to any -t flag
• NEVER send arbitrary shell commands via send-keys
• NEVER parse pane output as structured completion data
• NEVER access sessions on the default socket (no -L flag)

Failure Modes

Diagrams

stateDiagram-v2
    %% Rendering rule — two planes are overlaid in this one diagram:
    %%   Run-status (DB) states: pending, spawning, running, exited, failed, stopped.
    %%   Edges between them conform to docs/002-data-model.md, which is canon for
    %%   the run status machine.
    %%   tmux-session-plane states: hold_open, orphaned. These describe the tmux
    %%   session itself, carry no runs.status value of their own, and their edges
    %%   are session actions, not DB status transitions.

    [*] --> pending: run created (central assigns host)

    pending --> spawning: worker claims run, spawns session
    spawning --> spawning: session name held by prior incarnation — graceful-stop it, retry
    spawning --> running: wrapper posts run.started event

    running --> running: capture-pane polling continues
    running --> hold_open: CLI exits, wrapper posts run.finished
    running --> failed: wrapper exits unexpectedly
    running --> stopped: user/manual stop
    running --> running: central partition — no state change, signals spool

    hold_open --> exited: hold-open timer expires
    hold_open --> exited: user closes session

    exited --> [*]: session cleanup
    failed --> [*]: session cleanup
    stopped --> [*]: kill-session

    failed --> pending: restart-policy requeue (attempt + 1 at re-dispatch)

    running --> orphaned: worker crashes, session persists
    orphaned --> running: worker restart, session re-adopted (no runs.status change)
    orphaned --> [*]: unmapped after inspection hold — journal-only cleanup

    spawning --> failed: spawn fails (CLI not found, permission error)
    running --> failed: tmux server crash on host
    running --> stopped: force stop via API

    pending --> failed: host presumed lost (T_dead=24h exceeded, host_lost)
    spawning --> failed: host presumed lost (T_dead=24h exceeded, host_lost)
    running --> failed: host presumed lost (T_dead=24h exceeded, host_lost)

    note right of pending
        PostgreSQL: run.status = 'pending'
        host_id assigned by dispatcher
        Entered by dispatch or by restart-policy requeue (from failed)
        No tmux session exists
    end note

    note right of spawning
        PostgreSQL: run.status = 'spawning'
        tmux session created but no run.started event yet
    end note

    note right of running
        PostgreSQL: run.status = 'running'
        tmux session active on worker host, capture-pane polling 1 Hz
    end note

    note right of hold_open
        tmux-session-plane state: the run row is already terminal
        PostgreSQL: run.status = 'exited' or 'failed'
        CLI exited, wrapper holds session open for inspection
        Default 5 minutes, then cleanup
    end note

    note right of orphaned
        tmux-session-plane state — no runs.status value
        Session on socket pam (pam-*) that reconciliation cannot map
        to any desired run; no run row transitions on its account
        Cleanup is journal-only: run.orphan_detected event,
        inspection hold, then kill-session
    end note

    note right of failed
        PostgreSQL: run.status = 'failed'
        Terminal unless the restart policy requeues it
        (failed -> pending while attempts remain; 002 is canon)
        Session cleaned up via tmux -L pam kill-session

        host_lost (OQ-12, confirmed 2026-09-01): a host continuously
        offline for T_dead (default 24h, measured from hosts.last_seen_at)
        is presumed lost; central force-fails its non-terminal runs.
        Requeue-eligible like any other failed run — accepted trade-off:
        the original host's live session (if any) can go undetected
        until an operator notices and cleans it up. See
        010-tmux-executor.md § Restart Policy / § Host-lost bound.
    end note

    note right of stopped
        PostgreSQL: run.status = 'stopped'
        Graceful shutdown via kill-session
    end note

sequenceDiagram
    participant C as central orchestrator<br/>(pam serve)
    participant W as worker agent<br/>(pam worker)
    participant T as tmux -L pam<br/>(worker host)
    participant V as read-only viewers<br/>/pane/ws via central
    participant A as interactive attach<br/>/attach/ws via central
    participant P as provider CLI
    participant H as provider hook<br/>completion signal

    Note over W,T: Capture-pane polling loop (1 Hz, on worker host)
    loop every 1 second
        W->>T: capture-pane -p -e -t pam-<id>
        T-->>W: ANSI pane text
        W->>C: uplink snapshot for run_id (if hash changed)
        C->>V: WS message with full snapshot (sent only on change)
    end

    Note over C,V: Multiple viewers multiplexed from latest snapshot
    V->>C: read latest snapshot
    C-->>V: cached pane text
    V->>V: render in xterm.js terminal

    Note over A,C: Interactive attach flow (relayed through central)
    A->>C: WS upgrade request to /attach/ws
    C->>C: check single-writer lease availability
    C-->>A: 101 Switching Protocols (lease granted)
    C->>W: attach request over worker uplink

    W->>W: spawn PTY relay
    Note over W,T: portable-pty + tmux attach
    W->>T: tmux -L pam attach -t pam-<id>
    T-->>W: bidirectional PTY stream

    A->>C: WS binary message (user input)
    C->>W: relay input
    W->>T: write to PTY stdin
    T->>P: forward to CLI stdin
    P-->>T: CLI stdout
    T-->>W: PTY stdout
    W-->>C: relay output
    C-->>A: WS binary message (terminal output)

    Note over A,C: Lease heartbeat every 30s
    A->>C: WS ping/pong
    C->>C: refresh lease TTL

    Note over A,C: Detach or timeout
    A->>C: WS close or lease timeout
    C->>W: stop attach relay
    W->>T: kill PTY relay
    C->>C: release lease

    Note over H,C: Completion signal flow (primary signal)
    P->>P: CLI completes work
    P->>H: native completion hook fires
    H->>C: POST /api/internal/runs/{id}/signals
    C->>C: emit run.signal_received{kind=completed}
    C->>C: work_item → done (config-gated)
    C->>V: broadcast work_item.status_changed

    Note over W,C: Fallback — wrapper posts event
    P->>P: CLI exits (no native hook)
    Note over W: pam exec run wrapper observes exit
    W->>C: POST /api/internal/runs/{id}/signals<br/>run.finished{exit_code}
    Note over W,C: On partition: spool locally, replay after reconnect
    C->>C: transition to hold-open state
    W->>C: continue snapshot uplink for inspection period

flowchart LR
    subgraph Worker["pam worker (execution host)"]
        Cache[("Worker snapshot cache<br/>change detection")]
        Poll["tmux capture-pane<br/>1 Hz poll"]
    end

    subgraph Tmux["tmux -L pam"]
        Session["pam-<short_run_id>"]
        Pane["tmux pane"]
        CLI["Provider CLI"]
        Wrapper["pam exec run"]
    end

    subgraph CentralG["central orchestrator"]
        Central["serve: API + fanout"]
        CCache[("Central snapshot cache<br/>client serving")]
        DB[(PostgreSQL)]
    end

    subgraph Clients["Web Clients"]
        Viewer["Read-only viewer<br/>/pane/ws"]
        Attach["Interactive attach<br/>/attach/ws"]
    end

    Poll -->|"capture-pane -p -e"| Pane
    Pane -->|"ANSI text"| Cache
    Cache -->|"changed snapshots<br/>uplink only on change"| CCache
    Central -->|"SSE/WS stream"| Viewer
    Central -->|"WS relay"| Attach
    CCache --- Central
    Central --- DB

    Attach -->|"attach relay<br/>through central"| Session
    Session -->|"bidirectional"| Pane
    Pane -->|"stdout/stderr"| CLI

    CLI -->|"exit code"| Wrapper
    Wrapper -->|"finished signal<br/>(spooled on partition)"| Central

    style DB fill:#f5f5f7,stroke:#999
    style Cache fill:#f5f5f7,stroke:#999
    style CCache fill:#f5f5f7,stroke:#999

Cross-References

• Provider Adapters: Completion signal ladder, provider catalog, proxy configuration
• HTTP API Contract: WebSocket protocol, event schema
• Data Model: runs and events table definitions