Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

System Overview

System Boundary

Project Agent Manager (pam) is a multi-host platform for hierarchical project management and AI task orchestration. A central orchestrator owns the API, the web dashboard, and a PostgreSQL database; worker agents on execution hosts run the LLM CLI sessions inside each host’s local tmux. This topology was accepted on 2026-08-29 and deliberately overturns the original single-machine, SQLite premise (ADR-012; the overturn is recorded in 071-risks-and-open-questions.md).

The system consists of:

  • Central orchestrator (pam serve): a single long-lived process hosting the axum HTTP API, the embedded SPA (Rust/WASM — Leptos), run dispatch, the worker registry, and the PostgreSQL-backed store
  • PostgreSQL: the single source of truth — projects, work items, runs, the worker/host registry, and the append-only event journal
  • Workers (pam worker): one per execution host; registers with the central orchestrator over an outbound connection (workers never accept inbound connections — NAT-safe), supervises tmux sessions on the host’s dedicated socket pam, streams pane captures upstream, and relays attach traffic
  • tmux server (per host): a dedicated tmux server on socket pam hosting LLM CLI processes as sessions; CLI processes are children of the host’s tmux server, NOT of the worker
  • LLM CLI processes: long-lived CLI instances (claude, codex, grok, gemini) inside those sessions
  • Optional gateway: liteLLM-style proxy routing CLI traffic (configured per-run via environment injection); when present, it is one centrally-shared, network-reachable instance, not a per-host local process (ADR-013)

Component Architecture

flowchart TB
    subgraph Operator["Operator"]
        Browser["Browser / Tauri Desktop"]
        CLI["pam CLI"]
    end

    subgraph Central["Central Orchestrator — pam serve"]
        API["axum HTTP API · SSE · WS"]
        SPA["Embedded Leptos SPA"]
        Dispatch["Run Dispatcher"]
        Fanout["Event Fan-out — in-process broadcast"]
    end

    PG[("PostgreSQL<br/>single source of truth")]
    Gw["Optional LLM gateway<br/>one shared instance"]

    subgraph HostA["Worker Host — macOS · pam worker"]
        WorkerA["Worker Agent<br/>outbound uplink only"]
        TmuxA["tmux Server<br/>socket: pam"]
        WrapA["pam exec run wrapper"]
        Claude["claude CLI"]
        Codex["codex CLI"]
        LogsA["Run logs · providers.toml<br/>host-local disk"]
    end

    subgraph HostB["Worker Host — Linux · pam worker"]
        WorkerB["Worker Agent<br/>outbound uplink only"]
        TmuxB["tmux Server<br/>socket: pam"]
        WrapB["pam exec run wrapper"]
        Grok["grok CLI"]
        Gemini["gemini CLI — optional"]
    end

    Browser -->|"HTTPS + bearer token"| API
    CLI -->|"HTTPS + bearer token<br/>all commands via API"| API
    API --> SPA
    API --> PG
    Dispatch --> PG
    Fanout --> API

    WorkerA -->|"outbound uplink<br/>fetch runs · post events<br/>worker token + TLS"| API
    WorkerB -->|"outbound uplink<br/>fetch runs · post events<br/>worker token + TLS"| API
    WrapA -->|"POST run signals<br/>retried · spooled on partition"| API
    WrapB -->|"POST run signals<br/>retried · spooled on partition"| API

    WorkerA --> TmuxA
    WorkerA --> LogsA
    TmuxA --> WrapA
    WrapA --> Claude
    WrapA --> Codex

    WorkerB --> TmuxB
    TmuxB --> WrapB
    WrapB --> Grok
    WrapB --> Gemini

    Claude -.->|"optional routing"| Gw
    Codex -.->|"optional routing"| Gw
    Grok -.->|"optional routing"| Gw
    Gemini -.->|"optional routing"| Gw

    classDef operatorStyle fill:#e1f5ff,stroke:#01579b,stroke-width:2px
    classDef centralStyle fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
    classDef dbStyle fill:#fffde7,stroke:#827717,stroke-width:2px
    classDef workerStyle fill:#e0f2f1,stroke:#004d40,stroke-width:2px
    classDef tmuxStyle fill:#fff3e0,stroke:#e65100,stroke-width:2px
    classDef providerStyle fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px
    classDef gatewayStyle fill:#fce4ec,stroke:#880e4f,stroke-width:2px
    classDef storageStyle fill:#f1f8e9,stroke:#33691e,stroke-width:2px

    class Browser,CLI operatorStyle
    class API,SPA,Dispatch,Fanout centralStyle
    class PG dbStyle
    class WorkerA,WorkerB,WrapA,WrapB workerStyle
    class TmuxA,TmuxB tmuxStyle
    class Claude,Codex,Grok,Gemini providerStyle
    class Gw gatewayStyle
    class LogsA storageStyle

The system is organized around three control planes sharing a common append-only event journal:

  1. Project Management State Plane: Projects, work items (milestone/epic/task/subtask), and status board (todo/doing/done)
  2. Run Supervision Plane: Dispatch, spawn, reconcile, restart, and completion signal handling for LLM CLI sessions, distributed across the central orchestrator (dispatch) and workers (execution)
  3. Terminal Observation Plane: Pane capture streaming from workers to the central hub, and attach relay from clients through the hub to workers

Runtime Topology

Network Bindings

  • Central API: configured bind address (default 0.0.0.0:7898), served with TLS in deployment — no longer localhost-only (ADR-012)
  • Worker → central: outbound registration, heartbeat, event/telemetry uplink, and pane/attach relays; the worker holds no listener
  • Authentication: browsers and the CLI use bearer tokens against the central API; workers authenticate with a pre-provisioned worker token (mTLS under review — OQ-10)
  • Single-operator trust model: no multi-tenant isolation or RBAC

Placement

  • Worker hosts: macOS (primary) and Linux (secondary) — the executor is Unix-only by design (ADR-011)
  • Central orchestrator: any host with PostgreSQL reachability; running pam serve and pam worker on the same machine recovers the original single-machine deployment without a separate mode

File Locations

  • Central host, platform data dir (~/Library/Application Support/pam/ on macOS, ~/.local/share/pam/ on Linux): config.toml, auth token files
  • Worker host, platform data dir: worker config.toml, provider catalog, per-run on-disk logs under runs/
  • PostgreSQL: all persistent state (projects, work items, runs, hosts, events)

tmux Integration

Unchanged from the original design, now executed by the worker on each host. The tmux server uses a dedicated socket named pam to isolate pam-managed sessions from system tmux instances:

  • Socket creation: tmux -L pam -S /tmp/tmux-$UID/pam
  • Session naming: pam-<short_run_id> where short_run_id is the first 8 lowercase hex characters of the run UUID
  • CLI processes run as children of the tmux server, allowing detached operation and observation

Event Streaming

  • Worker → central: lifecycle and pane-metadata events flow upstream over the worker’s registration connection
  • Central fan-out: in-process tokio::sync::broadcast for connected clients (the central orchestrator is a single process; PostgreSQL LISTEN/NOTIFY is not required for fan-out)
  • Server-Sent Events endpoint: /api/v1/events/stream
  • Append-only event journal stored in the events table in PostgreSQL for activity feed and audit trail

Control Plane Coordination

The three control planes coordinate through the shared event stream: each plane’s state changes are journaled as events in PostgreSQL at the central hub, and each plane subscribes to the events of the other planes (e.g., terminal observation listens for run.started to begin capture-pane streaming). The authoritative enumeration of which plane emits which topic lives in 002-data-model.md §Event Topics — this document deliberately does not restate the list.

Operational Review Perspective

Reviewing the central/worker split for correctness means checking boundaries that this document describes only at the topology level, not at the decision level. A reviewer should follow these pointers rather than treat this document’s summary as the full picture:

  • Worker authentication. §Network Bindings states the current default (pre-provisioned worker token over TLS), but the token lifecycle — provisioning, rotation, revocation, and the mTLS alternative — is still an open decision, tracked as OQ-10 in 071-risks-and-open-questions.md. Treat OQ-10, not this file, as the source for worker-trust review.
  • Worker uplink protocol. §Network Bindings and §Event Streaming assume an outbound worker→central connection but do not commit to a transport. The transport choice (persistent WebSocket with heartbeat vs. a polling fallback) is tracked as OQ-11 in the same document and blocks Phase 2 implementation — review it before assuming either transport is final.
  • Security documentation gap. 061-security.md is currently a placeholder (see docs/README.md Reading Order). Until it is written, this document’s Network Bindings section is the only committed description of the trust model. A security-focused review should record that as a gap to close, not as evidence the trust model has already been reviewed.

This section adds no new decisions of its own — it exists so an operational review of the central/worker boundary starts from the documents that actually own those decisions.

Non-Goals

The following features are explicitly out of scope for pam:

Superseded non-goals (overturned 2026-08-29, ADR-012): “remote multi-host workers excluded” and “PostgreSQL backend excluded” — both were pillars of the original single-machine design and are now core to the architecture.