server-watcher

server-watcher (binary sw) is ssh-config-driven server infrastructure management for a single operator. It reads the user's ~/.ssh/config, keeps a host catalog in SQLite, probes and collects metrics through the system ssh binary, and exposes the fleet three ways: a local web dashboard with an in-browser terminal (sw serve / sw ui), a REST API, and a stdio MCP server for LLM agents (sw mcp).

What is documented here

• Architecture — the design document: workspace layout (eight crates), run modes, the system-ssh reuse decision, the localhost trust model, storage, the write-path overview, and the frontend/desktop shell.
• Design Intent Review — a conformance review of the implementation against every intent claim in the architecture document, with evidence, performed after Phase 6 completion.
• Write Path and Audit Contracts — the round-2 mutation surface: bidirectional ssh-config editing (stanza splicing, diff preview, optimistic concurrency, atomic writes), SSH key management (scan/generate/delete/deploy), the audit log rules, the stale-host purge policy, and the 16-tool MCP surface.

Key properties

implementation: current-as-of-2026-09-02

Architecture

server-watcher (sw) manages the user's server infrastructure from the one source of truth the user already maintains: ~/.ssh/config. This document is the architecture index; per-area contracts land as follow-up documents and must be linked from here.

Single workspace, eight crates

One Cargo workspace, one version of every dependency, one set of gates. The split follows a single rule: pure vocabulary in sw-core, one crate per runtime concern, nothing shared by accident.

graph TD
    cli[sw-cli<br/>`sw` binary: serve / ui / mcp / sync / token]
    api[sw-api<br/>axum: REST + WS terminal/events + auth + embedded UI]
    mon[sw-monitor<br/>sync service + scheduler + retention]
    mcp[sw-mcp<br/>stdio JSON-RPC MCP server]
    ssh[sw-ssh<br/>system ssh: executor / probe / metrics / PTY]
    store[sw-store<br/>SQLite WAL + migrations + repositories]
    core[sw-core<br/>domain types + DTOs + ssh-config parser + error]
    ui[sw-ui<br/>Leptos CSR, built by trunk into dist/]

    cli --> api
    cli --> mon
    cli --> mcp
    api --> mon
    api --> ssh
    api --> store
    api -.embeds.-> ui
    mcp --> store
    mcp --> ssh
    mon --> ssh
    mon --> store
    ssh --> core
    store --> core
    api --> core
    mcp --> core

Layering rules:

• sw-core depends on nothing internal and no async runtime; it must stay compilable for wasm so the UI can reuse the DTOs and the ssh-config parser.
• sw-store is the only crate that talks SQL; repositories speak sw-core types only.
• sw-ssh is the only crate that spawns ssh or opens sockets.
• sw-ui is a leaf: nothing in Rust depends on it — its build output (crates/sw-ui/dist) is the artifact, embedded by sw-api.

Run modes

The single sw binary exposes:

sw mcp is a separate process by design: an MCP client owns the process's stdio, so the server cannot share a process with the HTTP server. Both processes use the same SQLite database; WAL mode is what makes that concurrent access safe.

System ssh reuse

The fleet is reached through the system ssh binary, never through an embedded SSH library. Rationale: the user's ~/.ssh/config (aliases, ProxyJump chains, Include files), known_hosts, agent and keys keep working unchanged — there is no second source of truth for credentials, no key material in the tool's own config, and no per-host connection setup to maintain. The costs are explicit: process spawn per operation, output is text to be parsed, and remote-side behavior depends on the installed ssh. Both are acceptable at this scale (tens of hosts, minutes-scale polling).

Write path: config editing, keys, audit, purge

Phase 6 made ~/.ssh/config bidirectional: the dashboard (and the MCP tools) can add, update and delete host stanzas through a stanza-splicing editor that preserves every byte it does not replace. The same round added SSH key management (scan / generate / delete / deploy), an append-only audit log for every mutation, and a hard-purge path for stale hosts. The full contracts — concurrency model, backup policy, deploy idempotency, purge rules — live in 003-write-and-audit.md.

Two cross-cutting invariants are architectural, not per-feature:

• One editor — all config writes go through sw-core's pure edit_config (String → String, no fs, wasm-compatible); sw-monitor's ConfigWriter owns the file lifecycle: read → edit → rolling config.sw.bak backup → tmp(0600)+rename atomic write, all under a cross-process File::lock on <config>.sw.lock so sw serve and sw mcp cannot interleave read-modify-write cycles.
• Every mutation is audited — config edits, key operations and host purges insert an audit_log row tagged with the caller's source (api/mcp); the dashboard's /audit page is a plain reader over it.

Localhost trust model

Two layers, no user accounts:

1. Bind guard — sw serve binds loopback only and refuses any other bind target. The API is local by construction.
2. Bearer token — every non-asset request must present the token sw token prints (also sw mcp, via environment). This keeps other local processes and drive-by web pages from driving the API, reading fleet data, or opening terminals.

The WebSocket terminal route carries the same token requirement; terminal input is written to the PTY verbatim — the remote shell is the user's own. The event stream route (GET /api/v1/events) authenticates the same way (bearer header, sw_token cookie, or ?token=, plus the host guard): authenticated BEFORE the upgrade, so a rejected handshake is a plain 401/403, never a dropped socket.

Storage

One SQLite database in WAL mode, embedded migrations compiled in via sqlx::migrate!, all queries runtime-checked (query/query_as functions, no query! macros — nothing reads DATABASE_URL at compile time). Aggregates: hosts (from sync), status history, metrics samples, exec records, sync runs, and the append-only audit_log (migration 0003) that records every config/key/purge mutation with its source (api / mcp). Retention (sw-monitor) prunes and downsamples so the file stays bounded on an always-on machine. Both pool connections enable foreign_keys(true), which is what makes ON DELETE CASCADE real: purging a host row drops its status/metrics/exec history with it.

Frontend and desktop shell

• UI — Leptos CSR built by trunk 0.21.14, pinned. Trunk 0.22 is a beta whose MSRV (1.96.1) is above this workspace's pinned Rust (1.93.1) — that combination cannot compile, which is exactly why the pin is explicit here and in the Phase 3 plan. Output lands in crates/sw-ui/dist, embedded by sw-api with rust-embed (mime-guess on for content types). The dist output is committed (CI never runs trunk): rebuild with trunk build --release in crates/sw-ui, commit the hashed assets, and rebuild so rust-embed picks them up. The wasm entry is declared in index.html as <link data-trunk rel="rust" href="Cargo.toml" /> — a <script data-trunk src="*.rs"> marker would take the script-asset pipeline and copy the Rust source verbatim.
• Live updates — the monitor and sync publish FleetEvents (status flips only, metrics samples, applied syncs) on a tokio::sync::broadcast channel created in sw-cli and shared with the API state; sw-api forwards them to subscribers of /api/v1/events as TEXT frames, and the dashboard bumps its poll tick on every frame, so live data lands without waiting for the next 10-second poll (the poll stays as the fallback; the client reconnects with capped exponential backoff). sw-monitor holds only an Option<Sender> — it never depends on sw-api, and callers that take no channel (MCP, tests) are unaffected.
• Desktop shell — sw ui runs the same axum server inside a Tauri v2 shell. Thread topology is fixed by AppKit: the Tauri event loop owns the main thread (which is why main in sw-cli is a plain fn, not #[tokio::main]), while the store, monitor scheduler and axum run on a dedicated tokio runtime created in src/ui.rs. Startup is ordered so a tauri-plugin-single-instance duplicate exits inside plugin init (never touching the shared port) and setup only builds the webview window after the listener is bound — the dashboard URL (http://127.0.0.1:7420/#token=…, same fragment flow as the browser) cannot 404 on first paint. Closing the window hides it (the status-item menu — Open Dashboard / Quit — is the only real exit), and quitting drops the runtime, which cancels the monitor loops (kill_on_drop ssh children) and closes live PTY masters (SIGHUP to ssh -tt). The shell's static assets are committed: crates/sw-cli/tauri.conf.json (no static window — the window is built at runtime with the token-bearing URL; frontendDist points at the committed gui-stub/ placeholder), crates/sw-cli/icons/ (RGBA PNGs; generate_context! reads them at compile time), and build.rs calls tauri_build::build(). The sw-mcp binary lives in the same crate and links tauri but never initializes it.

Phase plan (summary)

1. ssh-config parser → sw-store schema → API end-to-end (serve, token).
2. sw-monitor: sync service, probe/metrics scheduler, retention + history.
3. sw-ui: Leptos CSR dashboard via trunk.
4. Terminal (PTY → WebSocket) and remote exec.
5. sw-mcp: stdio MCP server.
6. Tauri shell and packaging.

Round 2 (2026-09, complete): config write-back with diff preview and audit → key management (scan/generate/delete/deploy) → key & audit pages in the UI → the config/key/audit tools exposed over MCP (16 tools total, destructive ones dry_run-defaulted) → stale-host purge. Contracts: 003-write-and-audit.md.

Implementation status: reviewed 2026-09-01 against commit 274465b — implementation conforms to every load-bearing intent.

Design Intent Review

A conformance review of the implementation against the intent stated in 001-architecture.md, performed 2026-09-01 at commit 274465b (16 commits on main, all four quality gates passing, workspace line coverage 87.32 % against the 85 % gate).

Method: every intent claim in the architecture document was turned into a checkable statement, then verified against (a) the source tree and build configuration, and (b) live runtime behavior of the running sw ui process and its API. Evidence is cited per row.

What the design is for

The architecture document commits to six load-bearing intents. Everything else in it (tables, phase list) is elaboration of these:

1. One source of truth — the catalog is imported from ~/.ssh/config; the tool never becomes a second place where fleet knowledge lives.
2. Strict layering — eight crates where pure vocabulary (sw-core) sits at the bottom, SQL lives only in sw-store, process/socket I/O only in sw-ssh, and the UI is a leaf whose artifact is dist/.
3. System ssh reuse — reach the fleet through the system ssh binary; no embedded SSH library, no second source of truth for credentials.
4. Localhost trust model — loopback-only bind plus a bearer token on every non-asset request; WS routes authenticate before upgrade.
5. Bounded, dependency-free storage — one SQLite file in WAL mode, compile-time-embedded migrations, runtime-checked queries only, retention keeps it bounded.
6. Reproducible frontend/toolchain — pinned Rust and trunk, committed dist assets, so a fresh checkout builds without npm and CI never runs trunk.

Conformance findings

Notes and minor deviations

1. sw-mcp also depends on sw-monitor (for the sw_sync_ssh_config tool). The architecture diagram shows mcp --> store/ssh/core and omits this edge. It does not violate any stated layering rule — sw-monitor sits below both consumers — but the diagram would be strictly accurate with the edge added.
2. Phase status: the frontmatter of 001-architecture.md says phase-6-complete, which matches reality; the events stream (listed as "Phase 6 optional" in the original plan) landed as commit 274465b and is verified above.
3. Deferred work (environment, not design): git push to the gitea remote is pending external recovery of the remote host; no code change is implied.

Conclusion

The implementation conforms to every load-bearing intent of the design document. The single discrepancy found is one missing (harmless, rule-consistent) edge in the architecture diagram, noted above. No intent is partially met or silently dropped.

implementation: complete

Write Path and Audit Contracts

Contracts for everything that mutates state beyond the read-only fleet view: ssh-config editing, SSH key management, the audit log they report into, and the stale-host purge. Everything here is live and verified against the shipped binary (2026-09).

Config editing

~/.ssh/config stays the source of truth; the app edits it in place under strict preservation rules.

• Stanza splicing, not regeneration — sw-core's edit_config is a pure String → String transform that replaces only the lines of the target Host stanza. Comments between blocks, indentation, directive casing, unknown directives (ProxyJump, …) and the raw ~ in IdentityFile paths survive byte-identical. Regenerating the file was rejected precisely because the parser is lossy in the reverse direction.
• Diff preview → confirm → apply — every write endpoint takes dry_run; a dry run returns the unified diff without touching the file. The UI renders the same diff before its apply button.
• Optimistic concurrency — updates may carry expected_current (the stanza as the caller last saw it). If the stanza changed underneath, the apply answers 409 instead of overwriting. This is stanza-scoped on purpose: an unrelated edit elsewhere in the file does not block yours.
• Cross-process locking — the read → edit → backup → write cycle runs under an exclusive std::fs::File::lock on <config>.sw.lock (0600), so the HTTP server and a concurrent sw mcp process cannot interleave their cycles. In-process, routes additionally serialize on a tokio mutex in the API state. Locking is fail-closed: no lock, no write.
• Atomic write, rolling backup — the new text lands via tmp file (0600) + rename; the previous content is kept as the single rolling ~/.ssh/config.sw.bak (overwritten each apply, never multiplied). The audit log's unified diffs are the history; the backup is only the one-step undo.
• Rejections — multi-pattern Host lines, aliases containing wildcards or whitespace, duplicate adds, updates/deletes of unknown stanzas, and CRLF files are refused outright. Include/Match present in the file produce warnings, not failures.
• EOF normalization — add always terminates the file with exactly one trailing newline; deleting the last stanza also removes the separator blank line ahead of it, so an add → delete round trip is byte-identical (regression-tested).

Endpoint: POST /api/v1/ssh-config/hosts with {dry_run, op: add|update|delete, stanza?, expected_current?, alias?}. MCP: sw_edit_ssh_config (same payload, dry_run defaults to true).

Key management

• Scan — ~/.ssh and ~/CERT (configurable [keys] directories), recursive to depth 3. A file is a key when it has a .pub sibling OR the config references it as an IdentityFile (path-expanded compare — this is what picks up cloud .pem files). .pub files themselves, authorized_keys, known_hosts* and config* are excluded. Metadata only: type/bits/fingerprint (via ssh-keygen -lf), size, mtime, referencing hosts. Private key material never leaves disk.
• Generate — name validated ([A-Za-z0-9._-], no leading dot, no ..), refuses to overwrite; spawns ssh-keygen -t … -N "".
• Delete — usage analysis first; if the key is referenced by any host config, the call returns 409 listing them unless force=true. Removes private + .pub.
• Deploy — validates the .pub content (single line, ssh-/ ecdsa-/sk- prefix, no quotes/newlines — injection guard), then one remote command through the standard executor that is idempotent by marker: SW_ALREADY when the key is already in authorized_keys, SW_ADDED when appended. Recorded in exec_log and the audit log.

Endpoints under /api/v1/keys (GET, POST, DELETE ?force=, /{name}/usage, /{name}/deploy). MCP: sw_list_keys, sw_generate_key, sw_get_key_usage, sw_delete_key, sw_deploy_key.

Audit log

Append-only audit_log table (migration 0003): action, target, detail (unified diff or summary, 8 KB cap), source (api | mcp), created_at (RFC3339 UTC). Actions in use: config_add, config_update, config_delete, key_generate, key_delete, key_deploy, host_purge.

House rules:

• No-ops are not audited — a write that changes nothing (e.g. a purge with zero stale hosts) writes no row and emits no event. The audit log records what changed, not what was attempted.
• Source is injected at the seam — services carry a with_source() builder defaulting to api; the MCP handlers construct theirs with mcp. A route cannot accidentally log as the wrong caller.

Readers: GET /api/v1/audit?limit= (default 50, max 500), the dashboard /audit page, and sw_list_audit over MCP.

Stale-host purge

Hosts that vanished from ~/.ssh/config are flagged missing_from_config = 1 by sync ("stale") but kept — history is worth more than disk. Purging is a deliberate second decision:

• Stale only — DELETE /api/v1/hosts/{alias} hard-deletes a host row iff it is stale; an active host answers 409 pointing at the config delete path (remove the stanza, re-sync, then the row follows).
• Cascade — the four child tables (status checks, metric samples, hourly aggregates, exec log) reference hosts(alias) with ON DELETE CASCADE, and both store connections enable the foreign_keys pragma. One DELETE cleans everything; there is no hand-rolled child cleanup to drift out of sync.
• Bulk — POST /api/v1/hosts/purge-stale purges every stale host in one audited call (target: "*", detail lists the aliases).
• Events — purges publish a hosts_purged FleetEvent so live dashboards refresh.
• UI — the dashboard shows a stale banner (two-step confirm) when stale hosts exist; a stale host's detail page replaces the config-delete danger zone (which would 404 — the stanza is gone) with a "purge from store" zone. MCP mirrors both as sw_delete_host / sw_purge_stale_hosts, dry_run defaulting to true.

MCP tool surface (16 tools)

Read: sw_list_hosts, sw_get_host, sw_get_status, sw_get_metrics, sw_list_exec_history, sw_list_keys, sw_get_key_usage, sw_list_audit · exec: sw_ssh_exec · sync: sw_sync_ssh_config · write (dry_run-defaulted where destructive): sw_edit_ssh_config, sw_generate_key, sw_delete_key, sw_deploy_key, sw_delete_host, sw_purge_stale_hosts.

Errors are domain-mapped: parse/validation → -32602, not-found and conflict → tool errors with recovery hints, the rest → internal. The server is hand-rolled JSON-RPC 2.0 over stdio (no rmcp), sharing the same SQLite file and services as the HTTP API — WAL mode is what makes the two-process access safe.