server-watcher

server-watcher (binary sw) is ssh-config-driven server infrastructure management for a single operator. It reads the user's ~/.ssh/config, keeps a host catalog in SQLite, probes and collects metrics through the system ssh binary, and exposes the fleet three ways: a local web dashboard with an in-browser terminal (sw serve / sw ui), a REST API, and a stdio MCP server for LLM agents (sw mcp).

What is documented here

• Architecture — the design document: workspace layout (eight crates), run modes, the system-ssh reuse decision, the localhost trust model, storage, the write-path overview, and the frontend/desktop shell.

• Design Intent Review — a conformance review of the implementation against every intent claim in the architecture document, with evidence, performed after Phase 6 completion.

• Write Path and Audit Contracts — the round-2 mutation surface: bidirectional ssh-config editing (stanza splicing, diff preview, optimistic concurrency, atomic writes), SSH key management (scan/generate/delete/deploy), the audit log rules, the stale-host purge policy, and the MCP tool surface.

• Alerting Contracts — the transition rule (fire once, not every cycle), what raises and clears each condition, the macOS and webhook channels, and the threshold configuration surface.

• Assistant Contracts (BYOK) — the operator's own LLM key: two wire protocols with vendors as presets, what context each assistant may see, why a suggested command waits for a keystroke, and how a fleet plan becomes an approved fan-out.

Key properties

implementation: current-as-of-2026-09-14

Architecture

server-watcher (sw) manages the user's server infrastructure from the one source of truth the user already maintains: ~/.ssh/config. This document is the architecture index; per-area contracts land as follow-up documents and must be linked from here.

Single workspace, eight crates

One Cargo workspace, one version of every dependency, one set of gates. The split follows a single rule: pure vocabulary in sw-core, one crate per runtime concern, nothing shared by accident.

graph TD
    cli[sw-cli<br/>`sw` binary: serve / ui / mcp / sync / token]
    api[sw-api<br/>axum: REST + WS terminal/events + auth + embedded UI]
    mon[sw-monitor<br/>sync service + scheduler + retention]
    mcp[sw-mcp<br/>stdio JSON-RPC MCP server]
    ssh[sw-ssh<br/>system ssh: executor / probe / metrics / PTY]
    store[sw-store<br/>SQLite WAL + migrations + repositories]
    core[sw-core<br/>domain types + DTOs + ssh-config parser + error]
    ui[sw-ui<br/>Leptos CSR, built by trunk into dist/]

    cli --> api
    cli --> mon
    cli --> mcp
    api --> mon
    api --> ssh
    api --> store
    api -.embeds.-> ui
    mcp --> store
    mcp --> ssh
    mon --> ssh
    mon --> store
    ssh --> core
    store --> core
    api --> core
    mcp --> core

Layering rules:

• sw-core depends on nothing internal and no async runtime; it must stay compilable for wasm so the UI can reuse the DTOs and the ssh-config parser.
• sw-store is the only crate that talks SQL; repositories speak sw-core types only.
• sw-ssh is the only crate that spawns ssh or opens sockets.
• sw-ui is a leaf: nothing in Rust depends on it — its build output (crates/sw-ui/dist) is the artifact, embedded by sw-api.

Run modes

The single sw binary exposes:

sw mcp is a separate process by design: an MCP client owns the process's stdio, so the server cannot share a process with the HTTP server. Both processes use the same SQLite database; WAL mode is what makes that concurrent access safe.

System ssh reuse

The fleet is reached through the system ssh binary, never through an embedded SSH library. Rationale: the user's ~/.ssh/config (aliases, ProxyJump chains, Include files), known_hosts, agent and keys keep working unchanged — there is no second source of truth for credentials, no key material in the tool's own config, and no per-host connection setup to maintain. The costs are explicit: process spawn per operation, output is text to be parsed, and remote-side behavior depends on the installed ssh. Both are acceptable at this scale (tens of hosts, minutes-scale polling).

Write path: config editing, keys, audit, purge

Phase 6 made ~/.ssh/config bidirectional: the dashboard (and the MCP tools) can add, update and delete host stanzas through a stanza-splicing editor that preserves every byte it does not replace. The same round added SSH key management (scan / generate / delete / deploy), an append-only audit log for every mutation, and a hard-purge path for stale hosts. The full contracts — concurrency model, backup policy, deploy idempotency, purge rules — live in 003-write-and-audit.md. Alerting — what the collected metrics are finally for — has its own contracts in 004-alerting.md, and the BYOK assistant, which turns the same data into answers and answers into proposed commands, in 005-assistant.md.

Three cross-cutting invariants are architectural, not per-feature:

• One editor — all config writes go through sw-core's pure edit_config (String → String, no fs, wasm-compatible); sw-monitor's ConfigWriter owns the file lifecycle: read → edit → rolling config.sw.bak backup → tmp(0600)+rename atomic write, all under a cross-process File::lock on <config>.sw.lock so sw serve and sw mcp cannot interleave read-modify-write cycles.
• Every mutation is audited — config edits, key operations, host purges, alert and assistant settings, and every command a fan-out runs insert a row tagged with the caller's source (api/mcp); the dashboard's /audit page is a plain reader over it.
• The catalog holds servers only — ~/.ssh/config entries that exist to pick an identity for a git remote (Host github-as-…) are filtered at the single sync read path, so the API, the CLI and the MCP server import the same fleet. Rows imported before the rule are deleted rather than flagged stale: an authentication alias is not a server that went away.

Localhost trust model

Two layers, no user accounts:

1. Bind guard — sw serve binds loopback only and refuses any other bind target. The API is local by construction.
2. Bearer token — every non-asset request must present the token sw token prints (also sw mcp, via environment). This keeps other local processes and drive-by web pages from driving the API, reading fleet data, or opening terminals.

The WebSocket terminal route carries the same token requirement; terminal input is written to the PTY verbatim — the remote shell is the user's own. The event stream route (GET /api/v1/events) authenticates the same way (bearer header, sw_token cookie, or ?token=, plus the host guard): authenticated BEFORE the upgrade, so a rejected handshake is a plain 401/403, never a dropped socket.

Storage

One SQLite database in WAL mode, embedded migrations compiled in via sqlx::migrate!, all queries runtime-checked (query/query_as functions, no query! macros — nothing reads DATABASE_URL at compile time). Aggregates: hosts (from sync), status history, metrics samples, exec records, sync runs, the append-only audit_log (migration 0003) that records every config/key/purge mutation with its source (api / mcp), the alerting trio (migration 0004): thresholds, live alert state and transition history; and the assistant's own settings (0008/0009), which hold the operator's LLM credentials. Retention (sw-monitor) prunes and downsamples so the file stays bounded on an always-on machine. Both pool connections enable foreign_keys(true), which is what makes ON DELETE CASCADE real: purging a host row drops its status/metrics/exec history with it.

Backups are taken through SQLite, not the filesystem: sw backup asks the store for a VACUUM INTO snapshot, so a copy taken while the server is running is complete rather than a data.db whose recent writes are still in a -wal sidecar. The archive also carries config.toml — the API token and the assistant key live there — so it is written 0600. sw restore moves the files it replaces aside instead of deleting them, and refuses an archive without a matching manifest.

Frontend and desktop shell

• UI — Leptos CSR built by trunk 0.21.14, pinned. Trunk 0.22 is a beta whose MSRV (1.96.1) is above this workspace's pinned Rust (1.93.1) — that combination cannot compile, which is exactly why the pin is explicit here and in the Phase 3 plan. Output lands in crates/sw-ui/dist, embedded by sw-api with rust-embed (mime-guess on for content types). The dist output is committed (CI never runs trunk): rebuild with trunk build --release in crates/sw-ui, commit the hashed assets, and rebuild so rust-embed picks them up. The wasm entry is declared in index.html as <link data-trunk rel="rust" href="Cargo.toml" /> — a <script data-trunk src="*.rs"> marker would take the script-asset pipeline and copy the Rust source verbatim.
• Live updates — the monitor and sync publish FleetEvents (status flips only, metrics samples, applied syncs) on a tokio::sync::broadcast channel created in sw-cli and shared with the API state; sw-api forwards them to subscribers of /api/v1/events as TEXT frames, and the dashboard bumps its poll tick on every frame, so live data lands without waiting for the next 10-second poll (the poll stays as the fallback; the client reconnects with capped exponential backoff). sw-monitor holds only an Option<Sender> — it never depends on sw-api, and callers that take no channel (MCP, tests) are unaffected.
• Terminal — the in-browser terminal attaches to a persistent tmux session on the remote host (tmux new-session -A -s sw), guarded by a command -v tmux check so hosts without it still get a login shell. A build survives a closed tab, and reopening the terminal walks back into the same session rather than starting beside it.
• Desktop shell — sw ui runs the same axum server inside a Tauri v2 shell. Thread topology is fixed by AppKit: the Tauri event loop owns the main thread (which is why main in sw-cli is a plain fn, not #[tokio::main]), while the store, monitor scheduler and axum run on a dedicated tokio runtime created in src/ui.rs. Startup is ordered so a tauri-plugin-single-instance duplicate exits inside plugin init (never touching the shared port) and setup only builds the webview window after the listener is bound — the dashboard URL (http://127.0.0.1:7420/#token=…, same fragment flow as the browser) cannot 404 on first paint. Closing the window hides it (the status-item menu — Open Dashboard / Quit — is the only real exit), and quitting drops the runtime, which cancels the monitor loops (kill_on_drop ssh children) and closes live PTY masters (SIGHUP to ssh -tt). The shell's static assets are committed: crates/sw-cli/tauri.conf.json (no static window — the window is built at runtime with the token-bearing URL; frontendDist points at the committed gui-stub/ placeholder), crates/sw-cli/icons/ (RGBA PNGs; generate_context! reads them at compile time), and build.rs calls tauri_build::build(). The sw-mcp binary lives in the same crate and links tauri but never initializes it.

Phase plan (summary)

1. ssh-config parser → sw-store schema → API end-to-end (serve, token).
2. sw-monitor: sync service, probe/metrics scheduler, retention + history.
3. sw-ui: Leptos CSR dashboard via trunk.
4. Terminal (PTY → WebSocket) and remote exec.
5. sw-mcp: stdio MCP server.
6. Tauri shell and packaging.

Round 2 (2026-09, complete): config write-back with diff preview and audit → key management (scan/generate/delete/deploy) → key & audit pages in the UI → the config/key/audit tools exposed over MCP → stale-host purge → alerting on host-down and resource thresholds, with a connectivity gate → a BYOK assistant beside the terminal and on the dashboard → 22 MCP tools, destructive ones dry_run-defaulted. Contracts: 003-write-and-audit.md, 004-alerting.md, 005-assistant.md.

implementation: reviewed-2026-09-01 reviewed-commit: 274465b

Design Intent Review

A conformance review of the implementation against the intent stated in 001-architecture.md, performed 2026-09-01 at commit 274465b (16 commits on main, all four quality gates passing, workspace line coverage 87.32 % against the 85 % gate).

Method: every intent claim in the architecture document was turned into a checkable statement, then verified against (a) the source tree and build configuration, and (b) live runtime behavior of the running sw ui process and its API. Evidence is cited per row.

What the design is for

The architecture document commits to six load-bearing intents. Everything else in it (tables, phase list) is elaboration of these:

1. One source of truth — the catalog is imported from ~/.ssh/config; the tool never becomes a second place where fleet knowledge lives.
2. Strict layering — eight crates where pure vocabulary (sw-core) sits at the bottom, SQL lives only in sw-store, process/socket I/O only in sw-ssh, and the UI is a leaf whose artifact is dist/.
3. System ssh reuse — reach the fleet through the system ssh binary; no embedded SSH library, no second source of truth for credentials.
4. Localhost trust model — loopback-only bind plus a bearer token on every non-asset request; WS routes authenticate before upgrade.
5. Bounded, dependency-free storage — one SQLite file in WAL mode, compile-time-embedded migrations, runtime-checked queries only, retention keeps it bounded.
6. Reproducible frontend/toolchain — pinned Rust and trunk, committed dist assets, so a fresh checkout builds without npm and CI never runs trunk.

Conformance findings

Notes and minor deviations

1. sw-mcp also depends on sw-monitor (for the sw_sync_ssh_config tool). The architecture diagram shows mcp --> store/ssh/core and omits this edge. It does not violate any stated layering rule — sw-monitor sits below both consumers — but the diagram would be strictly accurate with the edge added.
2. Phase status: the frontmatter of 001-architecture.md says phase-6-complete, which matches reality; the events stream (listed as "Phase 6 optional" in the original plan) landed as commit 274465b and is verified above.
3. Deferred work (environment, not design): git push to the gitea remote is pending external recovery of the remote host; no code change is implied.

Conclusion

The implementation conforms to every load-bearing intent of the design document. The single discrepancy found is one missing (harmless, rule-consistent) edge in the architecture diagram, noted above. No intent is partially met or silently dropped.

implementation: complete

Write Path and Audit Contracts

Contracts for everything that mutates state beyond the read-only fleet view: ssh-config editing, SSH key management, the audit log they report into, and the stale-host purge. Everything here is live and verified against the shipped binary (2026-09).

Config editing

~/.ssh/config stays the source of truth; the app edits it in place under strict preservation rules.

• Stanza splicing, not regeneration — sw-core's edit_config is a pure String → String transform that replaces only the lines of the target Host stanza. Comments between blocks, indentation, directive casing, unknown directives (ProxyJump, …) and the raw ~ in IdentityFile paths survive byte-identical. Regenerating the file was rejected precisely because the parser is lossy in the reverse direction.
• Diff preview → confirm → apply — every write endpoint takes dry_run; a dry run returns the unified diff without touching the file. The UI renders the same diff before its apply button.
• Optimistic concurrency — updates may carry expected_current (the stanza as the caller last saw it). If the stanza changed underneath, the apply answers 409 instead of overwriting. This is stanza-scoped on purpose: an unrelated edit elsewhere in the file does not block yours.
• Cross-process locking — the read → edit → backup → write cycle runs under an exclusive std::fs::File::lock on <config>.sw.lock (0600), so the HTTP server and a concurrent sw mcp process cannot interleave their cycles. In-process, routes additionally serialize on a tokio mutex in the API state. Locking is fail-closed: no lock, no write.
• Atomic write, rolling backup — the new text lands via tmp file (0600) + rename; the previous content is kept as the single rolling ~/.ssh/config.sw.bak (overwritten each apply, never multiplied). The audit log's unified diffs are the history; the backup is only the one-step undo.
• Rejections — multi-pattern Host lines, aliases containing wildcards or whitespace, duplicate adds, updates/deletes of unknown stanzas, and CRLF files are refused outright. Include/Match present in the file produce warnings, not failures.
• EOF normalization — add always terminates the file with exactly one trailing newline; deleting the last stanza also removes the separator blank line ahead of it, so an add → delete round trip is byte-identical (regression-tested).

Endpoint: POST /api/v1/ssh-config/hosts with {dry_run, op: add|update|delete, stanza?, expected_current?, alias?}. MCP: sw_edit_ssh_config (same payload, dry_run defaults to true).

Key management

• Scan — ~/.ssh and ~/CERT (configurable [keys] directories), recursive to depth 3. A file is a key when it has a .pub sibling OR the config references it as an IdentityFile (path-expanded compare — this is what picks up cloud .pem files). .pub files themselves, authorized_keys, known_hosts* and config* are excluded. Metadata only: type/bits/fingerprint (via ssh-keygen -lf), size, mtime, referencing hosts. Private key material never leaves disk.
• Generate — name validated ([A-Za-z0-9._-], no leading dot, no ..), refuses to overwrite; spawns ssh-keygen -t … -N "".
• Delete — usage analysis first; if the key is referenced by any host config, the call returns 409 listing them unless force=true. Removes private + .pub.
• Deploy — validates the .pub content (single line, ssh-/ ecdsa-/sk- prefix, no quotes/newlines — injection guard), then one remote command through the standard executor that is idempotent by marker: SW_ALREADY when the key is already in authorized_keys, SW_ADDED when appended. Recorded in exec_log and the audit log.

Endpoints under /api/v1/keys (GET, POST, DELETE ?force=, /{name}/usage, /{name}/deploy). MCP: sw_list_keys, sw_generate_key, sw_get_key_usage, sw_delete_key, sw_deploy_key.

Audit log

Append-only audit_log table (migration 0003): action, target, detail (unified diff or summary, 8 KB cap), source (api | mcp), created_at (RFC3339 UTC). Actions in use: config_add, config_update, config_delete, key_generate, key_delete, key_deploy, host_purge.

House rules:

• No-ops are not audited — a write that changes nothing (e.g. a purge with zero stale hosts) writes no row and emits no event. The audit log records what changed, not what was attempted.
• Source is injected at the seam — services carry a with_source() builder defaulting to api; the MCP handlers construct theirs with mcp. A route cannot accidentally log as the wrong caller.

Readers: GET /api/v1/audit?limit= (default 50, max 500), the dashboard /audit page, and sw_list_audit over MCP.

Commands are a separate trail. exec_log records every run any surface made — the dashboard's exec panel, an MCP sw_ssh_exec, and each host of an assistant fan-out — with its exit code, duration and source. It is read by GET /api/v1/exec-history?limit=&alias= (fleet-wide unless an alias narrows it), the /exec page, and sw_list_exec_history. A missing exit code means the run never completed (timeout, or ssh died), which is a different fact from exiting non-zero.

Stale-host purge

Hosts that vanished from ~/.ssh/config are flagged missing_from_config = 1 by sync ("stale") but kept — history is worth more than disk. Purging is a deliberate second decision:

• Stale only — DELETE /api/v1/hosts/{alias} hard-deletes a host row iff it is stale; an active host answers 409 pointing at the config delete path (remove the stanza, re-sync, then the row follows).
• Cascade — the four child tables (status checks, metric samples, hourly aggregates, exec log) reference hosts(alias) with ON DELETE CASCADE, and both store connections enable the foreign_keys pragma. One DELETE cleans everything; there is no hand-rolled child cleanup to drift out of sync.
• Bulk — POST /api/v1/hosts/purge-stale purges every stale host in one audited call (target: "*", detail lists the aliases).
• Events — purges publish a hosts_purged FleetEvent so live dashboards refresh.
• UI — the dashboard shows a stale banner (two-step confirm) when stale hosts exist; a stale host's detail page replaces the config-delete danger zone (which would 404 — the stanza is gone) with a "purge from store" zone. MCP mirrors both as sw_delete_host / sw_purge_stale_hosts, dry_run defaulting to true.

MCP tool surface (22 tools)

• Read — sw_list_hosts, sw_get_host, sw_get_status, sw_get_status_history, sw_get_metrics, sw_list_exec_history, sw_list_keys, sw_get_key_usage, sw_list_audit, sw_list_alerts.
• Act on a host — sw_ssh_exec, sw_check_host (probe now, recorded like a scheduled probe), sw_patch_host (tags, group, notes, enabled — columns sync never writes, so an agent's tidying survives the next import).
• Diagnose this machine — sw_check_network. An agent should ask it before concluding a fleet is down: while the verdict is not online, host-down alerting is frozen by design and every host looks unreachable. See 004-alerting.md.
• Change configuration (dry_run defaults true where a mistake costs something) — sw_sync_ssh_config, sw_edit_ssh_config, sw_generate_key, sw_delete_key, sw_deploy_key, sw_delete_host, sw_purge_stale_hosts, sw_set_alert_config.

Every applied change writes an audit row with source='mcp', which is how an operator finds out that an agent moved something — most importantly a threshold, where the failure mode is silence rather than an error.

The assistant is deliberately absent: an agent already has the tools the assistant would call, and proxying one model through another buys nothing.

Errors are domain-mapped: parse/validation → -32602, not-found and conflict → tool errors with recovery hints, the rest → internal. The server is hand-rolled JSON-RPC 2.0 over stdio (no rmcp), sharing the same SQLite file and services as the HTTP API — WAL mode is what makes the two-process access safe.

implementation: complete

Alerting Contracts

How server-watcher decides something is wrong, who it tells, and what it refuses to do. Monitoring collected status and metrics from Phase 2; this round turned that data into notifications.

The transition rule

An alert fires once per transition, never per cycle:

• A condition that becomes true raises an alert and notifies.
• The same condition still being true on the next cycle refreshes the stored measurement and notifies nobody.
• The condition going away clears the alert and notifies once more.

The alert_state row IS that memory: present means firing, absent means armed. This is why a host down for a week produces one notification rather than one every minute, and why an entry in the dashboard's active list can be read as "still firing right now".

Conditions

Two deliberate restraints:

• Not enough evidence is not an alert. A host with fewer stored probes than down_confirm_cycles cannot satisfy the rule, so a freshly added host does not alert on its first miss.
• A missing reading is not a recovery. A sample that carries no CPU figure leaves the CPU condition exactly as it was. Absent data never clears an alert, and never raises one.

Down evaluation runs after the probe row is stored, because the streak is counted from stored rows; resource evaluation runs on the sample the cycle just wrote.

The connectivity gate

down is the one condition that can be wrong about the whole fleet at once: when this machine loses its link, every probe fails and the fleet looks dead. So down alerts are decided per cycle, not per host, and a cycle where nothing answered is not believed until the machine's own connectivity is checked.

Two stages, cheapest first, and only in that all-failed case — a healthy fleet never triggers any outbound traffic:

1. Route check — a UDP socket "connected" to a public address. No packet is sent (UDP connect only fixes the peer), but the kernel must pick a route, so an unplugged cable or a sleeping interface fails here for free.
2. Endpoint race — short TCP connects to several endpoints at once (default 8.8.8.8:53,1.1.1.1:443); the first to answer wins. More than one on purpose: measured on a real link, 1.1.1.1:443 took 75 seconds to answer while 8.8.8.8:53 replied instantly, and an endpoint that slow would have been read as "offline". An empty list disables this stage, leaving the purely local verdict.

When the machine is offline, down evaluation freezes: nothing is raised, and — just as importantly — nothing is cleared, so alerts that were already firing survive the outage instead of replaying as a wall of recoveries when the link returns. The verdict is mirrored to GET /api/v1/alerts/network and the dashboard shows a banner, because a frozen state must not look like a healthy fleet. No notification is sent: the machine being offline is something the operator can already see.

One more restraint, learned in production: a lone probe is not evidence. The scheduler only probes hosts whose timer is due, and backoff means a long-down host is routinely probed by itself. Treating that single failure as a link outage froze alerting fleet-wide, so the gate now requires a cycle to cover at least two hosts before "they all failed" counts as doubt about the network.

Checking on demand

The gate only runs inside a cycle where every host failed, which is the wrong time to find out whether the check itself agrees with you. So the same check is available directly:

• POST /api/v1/alerts/network/check — runs it now, records the verdict, returns it.
• sw net — the same thing from the terminal. It opens the store directly rather than calling the API, so it answers while no server is running — exactly when someone asks whether the machine is online.
• The dashboard's /alerts page leads with a network panel and a "지금 확인" button.

All three write the same verdict the banner reads, so the UI never contradicts what the caller just saw.

Channels

Reachability is silent by default. down alerts are recorded (state, history, events, dashboard) but do not notify unless down_notify_enabled is turned on: down/up pairs are the noisiest kind — a flaky link, a reboot, a machine that is simply meant to be off — and the dashboard already shows what is unreachable. Resource thresholds always notify, because a filling disk is not something a glance at the table catches in time.

Both channels are optional and independent, and both are dispatched on a detached task — a hung webhook or a slow osascript cannot delay a monitoring cycle, and a delivery failure is logged, never retried, never fatal.

• macOS notification — osascript -e 'display notification …', spawned with no shell. The body is reduced to an ASCII-safe alphabet rather than escaped: AppleScript string literals have no escape that survives -e, and alert text is machine-generated, so nothing meaningful is lost and nothing injectable survives.
• Webhook — a JSON POST with a 5s timeout: {"event":"raised"|"cleared","kind":"down"|"cpu"|"mem"|"disk","host":…,"value":…,"threshold":…,"ts":…}. Enabled only when a URL with an http:// or https:// scheme is set.

With both channels off, evaluation still runs: alerts appear in the dashboard and the history, they just do not push anywhere.

Configuration

One singleton row (alert_config), edited from the dashboard's /alerts page or PUT /api/v1/alerts/config, and re-read by the evaluator on every pass — a threshold change takes effect on the next cycle, with no restart.

Validation (server-side, mirrored in the form): thresholds are percentages in 1–100, down_confirm_cycles is 1–10, an enabled webhook requires a schemed URL, and connectivity_endpoint entries must carry a port. down_notify_enabled and connectivity_endpoint are both optional in the payload and keep their stored value when omitted, so an older client cannot silently re-enable the noisiest alert kind or redirect the connectivity check. A save that changes nothing is a no-op and is not audited, matching the config-writer's rule that the audit log records what changed; a real change lands as alerts_config_update.

Surfaces

• GET/PUT /api/v1/alerts/config, GET /api/v1/alerts/active, GET /api/v1/alerts/history?limit= (default 50, max 500), GET /api/v1/alerts/network, POST /api/v1/alerts/network/check.
• CLI: sw net.
• Dashboard: a banner counting firing alerts, and /alerts — active list, threshold form, and transition history.
• MCP: sw_list_alerts reads the state; sw_set_alert_config changes thresholds and channels. The worry that kept thresholds read-only at first — an agent silencing a real alert — is handled rather than avoided: dry_run defaults to true, and an applied change writes an audit row with source='mcp', so moving a threshold is visible instead of merely quiet. sw_check_network answers the "is it me or them?" question from an agent's side.
• Events: alert_raised / alert_cleared on the /api/v1/events stream, so an open dashboard reflects a transition without waiting for its poll.

Retention

alert_state cascades with its host, so purging a stale host cannot leave an orphan alert firing forever. alert_history keeps its host_id as plain text: the record outlives the row it describes.

implementation: complete

Assistant Contracts (BYOK)

The assistant answers questions about servers using the operator's own LLM key. It exists in two places with deliberately different powers: beside a terminal, where it talks about one machine, and on the dashboard, where it talks about the estate.

Everything below follows from one rule: the model proposes, the operator disposes. Nothing the model says executes itself, except through a setting the operator turned on knowingly.

Providers are configuration, not code

Two wire protocols cover the field:

A vendor is a base URL and a model name, so the dashboard ships presets rather than the server shipping adapters: Anthropic and Z.AI (whose compatible endpoint speaks the Anthropic shape) use the first; OpenAI, OpenRouter, vLLM and ollama use the second. Adding a vendor that speaks either shape needs no code.

The key

Stored in the operator's own SQLite file, next to the API token, on a tool that binds loopback only. Three rules keep it there:

• GET /api/v1/assistant/config returns a view — every setting except the key, plus has_api_key so the form can say "저장됨".
• A PUT that omits the key keeps the stored one. The dashboard never receives the key, so it cannot echo it back, and saving an unrelated setting cannot wipe the secret.
• The audit row for a settings change records endpoint, model and flags — never the key.

Host assistant

Beside the terminal. Context is assembled from what the operator allows, each with its own switch:

• host facts and the latest metrics (send_metrics)
• the terminal screen, last 2000 characters, read from xterm in the browser (send_terminal)
• recent commands from exec_log (send_exec_history)

The reply may carry one command, marked by a CMD: line the server strips out. The dashboard types it onto the shell prompt without a newline — the operator's Enter is what runs it.

auto_run (off by default, migration 0009) removes that keystroke. The server decides per reply: auto-run applies only when the setting is on AND a command was actually suggested, so the dashboard cannot run anything on its own initiative, and the run happens once when the answer arrives rather than from a view closure that re-renders.

Fleet assistant

On the dashboard, collapsed until asked for. Context is a snapshot: fleet counts, every host's state with its tags and flags, and what is firing. Down hosts sort first, so truncation never hides exactly what the question is about.

A fleet answer may carry a plan — a CMD: line plus a HOSTS: line naming targets. The plan is rendered, never executed:

• Asking and running are separate endpoints (POST /assistant/fleet and POST /assistant/fleet/run). The run request restates the hosts, so what executes is what the dashboard showed, not whatever the model said last.
• auto_run deliberately does not reach here. "Run this after I read it" and "run this on twenty servers" are different decisions, and the second always asks.
• Targets are checked against the catalog. Invented aliases come back in unknown_hosts — a misunderstanding is shown rather than quietly shrinking the run — and the run endpoint refuses unknown, disabled or stale hosts outright rather than skipping them.
• Fan-out is capped: 50 hosts per run, 8 at a time, a per-host timeout. Every host's run lands in exec_log through the same path a single exec uses, so a batch is as auditable as a keystroke.

Failure modes

Errors map to the status a caller can act on: 412 when the assistant is off or has no key (fixable in settings), 404 for an unknown host, 502 when the LLM endpoint itself failed, 400 for an empty or oversized question. A timeout is 60 seconds — long enough for a real answer, short enough not to hold a tab open forever.

Surfaces

• GET/PUT /api/v1/assistant/config
• POST /api/v1/hosts/{alias}/assistant
• POST /api/v1/assistant/fleet, POST /api/v1/assistant/fleet/run
• Dashboard: the fleet panel; host detail: the strip under the terminal; /settings: the single home for provider, key and context switches.
• MCP: none. An agent already has the tools the assistant would call, and proxying one model through another buys nothing.