# Fleet — always-on agents with a console you can read Design spec, 2026-09-20. Status: awaiting operator review. ## 1. Purpose Seven narrow, unattended agents plus the two things they all depend on: a watchdog that notices when an agent is silent, stale, or over budget, and a console where every run is laid out step by step so the operator can see what the agent did, what the model was told and what it answered, what it produced, and exactly where anything failed. The design follows the research verdict (reviews.thomaspeng.ca/24-7-agents): agents are read-only or draft-only, every irreversible action is a human decision made in one approval queue, constraints live in the execution path rather than the prompt, and "the job ran" is never confused with "the job produced something". ## 2. What is in scope | Agent | Cadence | Output | Human gate | |---|---|---|---| | `watchdog` | every 5 min, no LLM | alerts, kill of over-cap runs, fleet status | none (safety valve) | | `console` | always on (web) | per-agent pages, run traces, approval queue | every queue item | | `brain` | nightly 02:30 UTC + Sunday 20:00 UTC | files the note drop folder; weekly review note | none | | `digest` | daily 11:00 UTC | ranked digest page + queue item with ratings | reading it | | `relay` | daily 05:00 UTC + on CI failure of a fleet PR | draft PRs for dependency/CVE upgrades and flaky-test quarantine | every merge | | `smoke` | every 5 min + within 2 min of a unit restart | pass/fail per host, incident packet on failure | rollback | | `receipts` | daily 12:00 UTC | flagged renewals and price increases; monthly summary | everything | Out of scope now, deliberately: inbox triage (operator deferred), any agent that sends, merges, deploys, pays, or deletes. ## 3. Layout ``` ~/projects/fleet/ pyproject.toml # uv workspace; one venv at .venv (absolute path in units) shared/fleet/ # the runtime every agent imports config.py # TOML loading, allowlists, caps; fails loudly on missing keys trace.py # run/step/event recording (JSONL + SQLite index) heartbeat.py # per-agent heartbeat file + PID spend.py # token/dollar ledger from claude -p JSON output llm.py # ClaudeRunner: subprocess claude -p with hard limits untrusted.py # wrap external content as inert data queue.py # client for the approval queue (insert, resolve, list) notify.py # Telegram via `sudo -n notify-general` clock.py # UTC only watchdog/ console/ brain/ digest/ relay/ smoke/ receipts/ /__main__.py # entry point: `python -m ` /logic.py # pure functions, no I/O, unit-tested without mocks /io.py # fetches, files, subprocesses /config.toml # defaults; overridden by /etc/personal/fleet/.toml /tests/ deploy/ units/personal-fleet-.service, .timer caddy/fleet.caddy install.sh # idempotent; copies units, enables timers, verifies var/ # runtime state (gitignored): traces, heartbeats, ledgers, sqlite docs/ ``` Every file carries a header block saying what it does and why it exists. Exported functions carry docstrings with purpose, params, return, and side effects. Booleans are predicates, functions are verbs. No bare `except`. Full type hints. ## 4. The shared runtime ### 4.1 Run trace (the thing the console renders) Every invocation of an agent is a **run** with a UUID. A run is a sequence of **steps**; a step is a sequence of **events**. The trace is the single source of truth for the console: if it is not in the trace, the console cannot show it, so agents record generously. Event kinds: - `run.started` / `run.finished` (status: succeeded, failed, killed, produced_nothing) - `step.started` / `step.finished` (name, summary, duration) - `fetch` (what was requested, how many items came back, bytes, latency) - `llm.call` (model, prompt text in full, system prompt, max_turns, budget) - `llm.stream` (each assistant message, tool_use, tool_result, and thinking block the CLI emits, in order, with timestamps) - `llm.result` (final text, input/output tokens, cost, turns used, stop reason) - `decision` (a structured "why": what the agent chose, the alternatives, the rule that decided it — e.g. "skipped PR: path touches auth/ (denylist)") - `output` (path or URL produced, content hash) - `queue.item` (id of the item placed in the approval queue) - `error` (exception type, message, full traceback, the step it occurred in) - `heartbeat` (written every 30 s while a run is alive) Storage: `var/traces//.jsonl` appended as events happen (so a running run is readable mid-flight), plus a SQLite index `var/fleet.db` with tables `runs`, `steps`, `errors`, `spend` for listing and search. Prompts and responses are stored verbatim; the console redacts nothing because the box is the operator's, but the `untrusted` wrapper marks external content so the console can render it in a distinct style. ### 4.2 ClaudeRunner One way to call the model, used by every agent: ``` claude -p --model --output-format stream-json --verbose --max-turns --max-budget-usd .json` (`run_id`, `pid`, `started_at`, `last_beat_at`, `step`) every 30 s from a background thread and removes it on exit. Each `llm.result` appends to `var/spend.jsonl`; the ledger answers "spend today for agent X" and "spend today for the fleet". Caps live in config: per-run turns, per-run dollars, per-agent per-day dollars, fleet per-day dollars. ### 4.5 Queue client `queue.put(kind, title, body_markdown, payload, actions)` inserts an item and returns its id. `kind` is one of `draft_pr`, `digest`, `alert`, `proposal`, `receipt_flag`. `actions` is the list of buttons the console offers (approve, discard, snooze, plus kind-specific ones). Resolution is recorded with who, when, and which action. Handlers for approve are registered per kind and are the only code paths that touch external systems on the operator's behalf; none of them merges, sends, deploys, or pays. ## 5. The console FastAPI + Jinja + htmx, no build step, port **3110**, bound to 127.0.0.1, served at `https://fleet.thomaspeng.ca` behind the existing GitHub SSO gateway (`forward_auth 127.0.0.1:4180`, same import the dashboard uses). Runs as `tpeng` because the approve handlers need the operator's `gh` auth; the SSO gate and loopback bind are the boundary, and the spec says so rather than pretending otherwise. Pages: - `/` — fleet overview: one card per agent with status (ok / stale / silent / over budget / failed), last run outcome, next scheduled run, heartbeat age, spend today vs cap, count of unresolved queue items. The watchdog's verdict is what the card shows. - `/agents/` — the agent page the operator asked for. Header repeats the card. Then: **runs** (table: started, duration, status, steps, cost, one-line summary, error badge). Then **config in effect** (allowlists, feeds, caps, model) as the run saw it. Then **decisions** across recent runs (the `decision` events, filterable), so "why did it skip that" is answerable without opening a run. - `/agents//runs/` — the run detail. A vertical timeline of steps; each step expands to its events. `llm.call` shows the full prompt with untrusted blocks visually distinct; `llm.stream` shows the model's messages, tool calls, and tool results in order as the CLI emitted them; `llm.result` shows tokens and cost. `error` events are pinned red with the traceback expanded. `output` events link to the produced file or URL. A running run streams new events into the page via server-sent events until `run.finished`. - `/queue` — unresolved items first, grouped by kind, with the actions the item declared; resolved items in a collapsible history with who resolved and when. Approving a `draft_pr` marks the PR ready for review on GitHub and comments the decision; it never merges. Approving a `digest` item records ratings. `alert` items have acknowledge only. - `/spend` — per-agent and fleet spend by day, against caps. Telegram: each queue insert and each watchdog alert sends one line via `sudo -n notify-general` with a link into the console. No two-way actions in Telegram. Errors are first-class: an unhandled exception anywhere in an agent is caught at the entry point, recorded as an `error` event with traceback, the run is marked `failed`, the heartbeat is removed, and the process exits non-zero so systemd records it too. The console shows the same traceback the journal would, next to the step that raised it and the prompt or fetch that preceded it. ## 6. The watchdog No model. Every 5 minutes: 1. **Silent:** for each agent, compare the last `run.started` in the index with the timer's expected cadence (from the registry in `watchdog/config.toml`). More than 1.5 intervals without a run → alert `silent`. 2. **Stale:** a run is alive (heartbeat file exists) but `last_beat_at` is older than 3 minutes → alert `stale`; older than 10 minutes → kill by PID (SIGTERM, then SIGKILL after 30 s), mark run `killed`, alert. 3. **Produced nothing:** the last finished run has no `output` or `queue.item` event and the agent is one that should produce (all but watchdog and smoke on green) → alert `produced_nothing`. This is the check the research says most fleets lack. 4. **Over budget:** agent or fleet daily spend at or above cap → kill any alive run of that agent, mark, alert. Caps are read from config, not from the anomaly heuristics. 5. **Runaway wall clock:** a run older than its agent's `max_run_seconds` → kill. 6. Write `var/status.json` (what the console cards render) and its own heartbeat. Alerts dedupe: one Telegram line per (agent, condition) until the condition clears, then a recovery line. The watchdog is the only agent with a kill action and holds no other credential. A test proves that a missing heartbeat produces an alert and that a fake over-cap ledger produces a kill of a dummy sleeping process. ## 7. The agents ### 7.1 brain - Drop folder: `~/notes/_inbox/` (inside the second brain's `notes` root, so nothing in the second brain changes). Nightly: for each file, the model proposes a destination folder under `~/notes/`, a title, frontmatter (`source`, `filed_at`, `tags`), and up to five `[[wikilinks]]` to existing notes found by grepping titles. The file is moved and rewritten; the original is kept under `~/notes/_inbox/.filed/` for 30 days. The second brain's 10-minute reindex picks the result up. - Sunday: one review note `~/notes/reviews/YYYY-WW.md` built from: git log across `~/projects` and `~/roux` for the week (commits by the operator, repos touched), queue decisions (what was approved and discarded), digest ratings, fleet outputs, and the watchdog's incident list. Opus writes it; the prompt asks for what changed, what was decided, what was ignored, and one question worth answering next week. No calendar or email access. - Decision events: destination chosen and why; links added and why. ### 7.2 digest - `feeds.toml`: RSS/Atom, arXiv queries, plain pages (diffed), and the existing ai-news-brief fetchers imported as a package where they have no DeepSeek dependency (rss, hn, github, anthropic). Seeded with the operator's AI/Claude sources plus placeholders for two more topics the operator names. - Pipeline: fetch → normalise → dedupe against `var/digest/seen.db` (URL + title hash, 30-day window) → score each item 1–5 with Sonnet in batches of 20, prompt includes the last 20 liked and 20 disliked exemplars (the feedback loop) → compose an HTML page at `~/review-files/fleet-digest/YYYY-MM-DD.html` → queue item of kind `digest` whose actions are per-item thumbs up/down. Ratings persist in `var/digest/ratings.db`. - Decision events: score and one-line reason per item; items dropped as duplicates. - Produced-nothing is legitimate when no feed had new items; the run records a `decision` saying so and the watchdog treats a `no_new_items` decision as output. ### 7.3 relay - `repos.toml` allowlist, seeded from the operator's GitHub repos that have CI, minus `honesty-index*` and anything the operator marks active. Each entry: owner/name, default branch, test command, path denylist (defaults: `auth`, `crypto`, `payment`, `billing`, `secrets`, `.github/workflows`), max diff lines (default 400). - Daily: `gh api` for open Dependabot alerts and Renovate/Dependabot PRs without a fleet review. For each alert without an upstream PR: clone to a temp worktree, bump the dependency, run the test command in the repo's own toolchain, and if green open a **draft** PR titled `fleet: upgrade → ` with the alert link, the test output summary, and a Sonnet-written explanation of the changelog delta. Queue item `draft_pr`. If tests fail: no PR; queue `proposal` with the failure and the diff. - On CI failure of a fleet-authored PR (poll every 15 min, or webhook later): up to three attempts where Sonnet reads the failing job log (untrusted) and proposes a fix restricted to the same files; each attempt is a commit on the PR branch; after three, stop and queue a `proposal`. Never force-push. Never touch a PR a human has commented on since the last fleet commit. - Flaky tests: from CI run history, a test that failed then passed on rerun twice in 14 days → a draft PR that adds a skip-with-reason marker to that test only (test files only, enforced by path check before commit). Queue `draft_pr`. - Execution-path guardrails: the branch name must start with `fleet/`; the push helper refuses any other branch and refuses the default branch; the diff is measured before push and rejected over the cap; denylisted paths are rejected before commit. These checks are functions in `logic.py` with tests, called from `io.py`. ### 7.4 smoke - `hosts.toml` allowlist seeded from `/etc/caddy/sites/**` hostnames the operator owns, minus the honesty index and preview hosts; each entry: URL, expected status, one expected text fragment, optional key asset path, the systemd unit behind it. - Every 5 minutes: GET each URL with a 10 s timeout; compare status, fragment, and asset status to the baseline; record latency. On the first failure, retry once after 20 s before declaring. Also watch `ActiveEnterTimestamp` of each unit; when it changes, run that host's check within 2 minutes and record a `deploy_observed` decision. - On failure: Telegram line, queue `alert`, and an **incident packet** as the run's output: last 80 lines of the unit's journal, last 10 commits of the project directory if known, Caddy access log tail for the host, and a Sonnet-written "likely cause and first thing to check" from those inputs (all untrusted). Recovery is recorded when the host passes again. The agent has no restart or rollback action. ### 7.5 receipts - IMAP over TLS to Gmail with an app password stored in `/etc/personal/fleet/receipts.env` (0600), read-only mailbox open, only the label `Receipts` (a Gmail filter the operator creates routes receipts there). Fallback if app passwords are unavailable: a CSV drop folder `~/notes/_receipts/` with the same downstream logic. - Daily: fetch messages since the last UID; for each, Sonnet extracts merchant, amount, currency, date, whether it is recurring, and the next renewal date if stated, from the body wrapped as untrusted; store in `var/receipts/ledger.db`. - Flags: a merchant seen for the first time as recurring; an amount higher than the previous charge from the same merchant (price increase); a stated renewal within 14 days. Each flag is a queue item `receipt_flag`. First of the month: a summary page of recurring charges and the year-to-date total per merchant, served only through the SSO-gated console (never on the public reviews host). - Nothing here can cancel, pay, or contact a merchant. The employment constraint on finance tooling was raised with the operator, who chose this receipt-only scope. ## 8. Deployment - One uv-managed venv at `~/projects/fleet/.venv`; units reference the absolute interpreter path. Units run as `tpeng` with `WorkingDirectory=~/projects/fleet`, `Environment=PYTHONUNBUFFERED=1`, `TimeoutStartSec=`. `sudo -n notify-general` is the one privileged call, already restricted by sudoers. - Timers: `personal-fleet-.timer` with `Persistent=true` and `RandomizedDelaySec` of 60 s so cron-minute pile-ups don't happen. The console is a plain service. - Caddy: `deploy/caddy/fleet.caddy` installed to `/etc/caddy/sites/personal/`, importing the same SSO snippet as `dashboard.caddy`. Reload is the only step needing sudo and is done by `install.sh` with a visible prompt. - `install.sh` is idempotent, prints every unit it touches, and ends by curling the console over HTTPS and asserting the served page matches disk. - `var/` is 0700; SQLite files 0600. Digest pages go to the public reviews host (they are news); receipts pages are served only through the SSO-gated console. ## 9. Testing - `logic.py` in every agent is pure and tested with real fixtures captured from the box (a real Dependabot alert JSON, a real feed, a real receipt email, a real journal tail), never hand-written stand-ins, per the operator's parser-guard lesson. - Every new test is proven to fail first (the test is written against the behaviour before the implementation exists). - `trace.py` has a round-trip test: write events, read the run back, render the timeline model the console uses. - Watchdog has the two prove-fail tests in §6. - `relay` guardrails: tests that a denylisted path, an oversized diff, and a non-fleet branch are each rejected before any git write. - One end-to-end smoke test per agent that runs the entry point in `--dry-run` mode against fixtures and asserts a trace with `run.finished` and at least one `output` or `decision` event. - Full suite runs once before each commit; filtered runs while iterating. ## 10. Build plan and ownership Coupled work gets one owner; independent work fans out. 1. **Main thread, sequential:** `shared/` runtime, `watchdog`, `console` (they define the interfaces everything else writes to). Includes `deploy/` and `install.sh`. 2. **Fan-out, five Sonnet implementers in one batch,** each owning one of `brain`, `digest`, `relay`, `smoke`, `receipts` against the shared interfaces and this spec, each producing tests and a dry-run trace. Concurrency held at five because another session is active on the box. 3. **Main thread:** review each agent's diff and dry-run trace in the console, integrate, install timers, run one real cycle of each, verify the console over HTTPS, and hand over. ## 11. What only the operator can do None of these block the start of the build. 1. Create the Gmail filter that labels receipts `Receipts`, and a Gmail app password for the receipts agent (or say app passwords are blocked and the CSV drop folder is used instead). 2. Confirm `fleet.thomaspeng.ca` as the console host, or name another. 3. Name two topics beyond AI/Claude for the digest, or leave it seeded as is. 4. Prune the two allowlists (`relay/repos.toml`, `smoke/hosts.toml`) after they are seeded. ## 12. Non-goals and standing rules - No agent sends email, merges, deploys, restarts, pays, cancels, deletes user data, or posts to a third-party platform. Adding any such action is a new spec. - No API key. Plan auth only. - No public control surface: the console binds to loopback and sits behind SSO. - No prompt-only constraints: every cap, denylist, and gate is enforced in code with a test. - The honesty index (`~/projects/honesty-index*`, its services, its repos) is excluded from every allowlist by default.