implementation: complete

Alerting Contracts

How server-watcher decides something is wrong, who it tells, and what it refuses to do. Monitoring collected status and metrics from Phase 2; this round turned that data into notifications.

The transition rule

An alert fires once per transition, never per cycle:

• A condition that becomes true raises an alert and notifies.
• The same condition still being true on the next cycle refreshes the stored measurement and notifies nobody.
• The condition going away clears the alert and notifies once more.

The alert_state row IS that memory: present means firing, absent means armed. This is why a host down for a week produces one notification rather than one every minute, and why an entry in the dashboard's active list can be read as "still firing right now".

Conditions

Two deliberate restraints:

• Not enough evidence is not an alert. A host with fewer stored probes than down_confirm_cycles cannot satisfy the rule, so a freshly added host does not alert on its first miss.
• A missing reading is not a recovery. A sample that carries no CPU figure leaves the CPU condition exactly as it was. Absent data never clears an alert, and never raises one.

Down evaluation runs after the probe row is stored, because the streak is counted from stored rows; resource evaluation runs on the sample the cycle just wrote.

The connectivity gate

down is the one condition that can be wrong about the whole fleet at once: when this machine loses its link, every probe fails and the fleet looks dead. So down alerts are decided per cycle, not per host, and a cycle where nothing answered is not believed until the machine's own connectivity is checked.

Two stages, cheapest first, and only in that all-failed case — a healthy fleet never triggers any outbound traffic:

1. Route check — a UDP socket "connected" to a public address. No packet is sent (UDP connect only fixes the peer), but the kernel must pick a route, so an unplugged cable or a sleeping interface fails here for free.
2. Endpoint race — short TCP connects to several endpoints at once (default 8.8.8.8:53,1.1.1.1:443); the first to answer wins. More than one on purpose: measured on a real link, 1.1.1.1:443 took 75 seconds to answer while 8.8.8.8:53 replied instantly, and an endpoint that slow would have been read as "offline". An empty list disables this stage, leaving the purely local verdict.

When the machine is offline, down evaluation freezes: nothing is raised, and — just as importantly — nothing is cleared, so alerts that were already firing survive the outage instead of replaying as a wall of recoveries when the link returns. The verdict is mirrored to GET /api/v1/alerts/network and the dashboard shows a banner, because a frozen state must not look like a healthy fleet. No notification is sent: the machine being offline is something the operator can already see.

One more restraint, learned in production: a lone probe is not evidence. The scheduler only probes hosts whose timer is due, and backoff means a long-down host is routinely probed by itself. Treating that single failure as a link outage froze alerting fleet-wide, so the gate now requires a cycle to cover at least two hosts before "they all failed" counts as doubt about the network.

Checking on demand

The gate only runs inside a cycle where every host failed, which is the wrong time to find out whether the check itself agrees with you. So the same check is available directly:

• POST /api/v1/alerts/network/check — runs it now, records the verdict, returns it.
• sw net — the same thing from the terminal. It opens the store directly rather than calling the API, so it answers while no server is running — exactly when someone asks whether the machine is online.
• The dashboard's /alerts page leads with a network panel and a "지금 확인" button.

All three write the same verdict the banner reads, so the UI never contradicts what the caller just saw.

Channels

Reachability is silent by default. down alerts are recorded (state, history, events, dashboard) but do not notify unless down_notify_enabled is turned on: down/up pairs are the noisiest kind — a flaky link, a reboot, a machine that is simply meant to be off — and the dashboard already shows what is unreachable. Resource thresholds always notify, because a filling disk is not something a glance at the table catches in time.

Both channels are optional and independent, and both are dispatched on a detached task — a hung webhook or a slow osascript cannot delay a monitoring cycle, and a delivery failure is logged, never retried, never fatal.

• macOS notification — osascript -e 'display notification …', spawned with no shell. The body is reduced to an ASCII-safe alphabet rather than escaped: AppleScript string literals have no escape that survives -e, and alert text is machine-generated, so nothing meaningful is lost and nothing injectable survives.
• Webhook — a JSON POST with a 5s timeout: {"event":"raised"|"cleared","kind":"down"|"cpu"|"mem"|"disk","host":…,"value":…,"threshold":…,"ts":…}. Enabled only when a URL with an http:// or https:// scheme is set.

With both channels off, evaluation still runs: alerts appear in the dashboard and the history, they just do not push anywhere.

Configuration

One singleton row (alert_config), edited from the dashboard's /alerts page or PUT /api/v1/alerts/config, and re-read by the evaluator on every pass — a threshold change takes effect on the next cycle, with no restart.

Validation (server-side, mirrored in the form): thresholds are percentages in 1–100, down_confirm_cycles is 1–10, an enabled webhook requires a schemed URL, and connectivity_endpoint entries must carry a port. down_notify_enabled and connectivity_endpoint are both optional in the payload and keep their stored value when omitted, so an older client cannot silently re-enable the noisiest alert kind or redirect the connectivity check. A save that changes nothing is a no-op and is not audited, matching the config-writer's rule that the audit log records what changed; a real change lands as alerts_config_update.

Surfaces

• GET/PUT /api/v1/alerts/config, GET /api/v1/alerts/active, GET /api/v1/alerts/history?limit= (default 50, max 500), GET /api/v1/alerts/network, POST /api/v1/alerts/network/check.
• CLI: sw net.
• Dashboard: a banner counting firing alerts, and /alerts — active list, threshold form, and transition history.
• MCP: sw_list_alerts reads the state; sw_set_alert_config changes thresholds and channels. The worry that kept thresholds read-only at first — an agent silencing a real alert — is handled rather than avoided: dry_run defaults to true, and an applied change writes an audit row with source='mcp', so moving a threshold is visible instead of merely quiet. sw_check_network answers the "is it me or them?" question from an agent's side.
• Events: alert_raised / alert_cleared on the /api/v1/events stream, so an open dashboard reflects a transition without waiting for its poll.

Retention

alert_state cascades with its host, so purging a stale host cannot leave an orphan alert firing forever. alert_history keeps its host_id as plain text: the record outlives the row it describes.