Swarm Self-Improvement Proposal

Date: 2026-05-27 Author: adhoc-swarm-research (read-only scoping pass, ~1–2h) Status: Proposal — no code changes made. Owner directive 15:29 today. Scope: What's actually been hurting us this week, what off-the-shelf tooling would buy us if we adopted it, and the smallest set of incremental fixes that would move the needle without rewriting the swarm.

1. Pain points we've actually hit (concrete, dated)

These are the failure modes the bus, the handoffs, and the audit docs document — not speculative. Each is paired with the cheapest fix later in §3.

1.1 Hung one-shot agents ("ghosts")

• Evidence: the 5/25 ci-backend-retarget filler hung 45h before finishing the rollout (referenced verbatim in the 11:52 valideerleeftijd bus line and in HANDOFF-2026-05-20). We killed 4+ stuck one-shots in this session alone (manager/orchestrator/worker shells left sitting on a dead JSONL stream).
• Why: claude -p invoked via start-agent.sh/run-agent.sh has no wall-clock budget. When the model stalls (network blip, 429, "Not logged in", or a quiet stdin), the wrapper process sits forever; nothing reaps it.
• Symptom: the manager re-queues "the same" task assuming the prior worker exited, producing duplicate work / racing PRs.

1.2 Shared-working-tree races

• Evidence: 2026-05-24 14:56 — two auditPic one-shots running concurrently in the same checkout collided on the working tree; we pivoted the manager to serial-mode for auditPic on the spot. The shared /Users/sarkout/projects/prive/<repo> working copy is mutated by git checkout, npm install, ./mvnw …, and Helm templating, none of which are concurrency-safe.
• Why: we run agents from the same checkout. Each project has exactly one working tree.
• Symptom: half-applied branch checkouts, dirty git status blocking commits, intermittent test failures.

1.3 Orchestrator-duplicate / replay misfires

• Evidence: 15:49 today — orchestrator restart was about to replay an already-completed directive; we caught it before it dispatched.
• Why: the orchestrator/manager rehydrates by tailing bus.log and the inbox, but has no idempotency key on the work it's seen. A restart can re-execute "the latest unfinished-looking message."
• Symptom: double-dispatch of fixes (one already merged), wasted Claude budget, occasionally a stale PR re-opening.

1.4 Claude 5h-budget stalls under heavy parallelism

• Evidence: 44h dead window 5/25 → 5/27 caused by the scheduled-agent loop hitting HTTP 429 + "Not logged in" from the CLI session expiring, with no agent able to refresh the credential. Documented in HANDOFF-2026-05-20 + bus lines around 5/25.
• Why: the 5h Anthropic credit window and the CLI session token are both soft, silent failures: the wrapper exits 0, the bus shows nothing, the manager assumes "the task finished cleanly."
• Symptom: the whole swarm goes quiet; we only notice when the human looks.

1.5 Mid-verify stale snapshots → phantom bugfix dispatches

• Evidence: #126 phantom — manager dispatched a "fix" for behavior the running pod no longer exhibited because the bug report was based on an older deploy. We killed it manually.
• Why: the manager reasons over bus text, not over a fresh kubectl get pods / git log -1 snapshot. Reports decay in seconds in a deploy-heavy loop, but the bus line stays.
• Symptom: workers chase fixed bugs, burn budget, sometimes regress the fix.

1.6 Local-first ladder vs VPS-final boundary

• Evidence: 2026-05-24 saturation outage — the VPS k3s node went over ~95% memory because workers used it for per-iteration redeploys. Owner directive that day added the local-first ladder to RECIPE.md (rung-1 compose → rung-2 docker-desktop k8s → rung-3 VPS as final-promotion-only).
• Why: the ladder is documented but not enforced by any check — a worker can still target --context transip-k3s whenever it wants.
• Symptom: capacity outages, owner-blocking work.

1.7 Crossed ACK pattern (race on directives)

• Evidence: 14:41 today — an ACK to one directive crossed two newer directives the owner had already sent. The agent operated on a 3-messages-old view of the world.
• Why: the orchestrator processes the bus serially, but the owner inbox has its own queue; without a "last-seen-owner-message" cursor, the orchestrator can ACK an old item while newer ones sit unread.
• Symptom: owner repeats themselves; trust erodes.

1.8 Owner-channel-vs-bus gap (now mostly closed)

• Evidence: owner messages used to land in a separate stream the orchestrator was not actively tailing — closed today by the owner-messages hook + owner-messages-watchdog.log.
• Status: keep this as the reference pattern for any future side-channel (Slack, email, etc.).

2. External tools / frameworks worth evaluating

Pragmatic assessment: what would this buy us beyond bus.log + .swarm/inbox? Most agent frameworks are designed for LLM-internal multi-agent chat, not "supervise N long-running shell wrappers on a laptop+VPS." That changes the verdict for several of them.

2.1 Process supervisors

| Tool | Verdict | Why | |---|---|---| | timeout (coreutils) wrapper | ✅ Adopt now | One-line fix for §1.1 ghosts. timeout 1800 claude -p … and we're done. Zero new deps. | | launchd (already used for designqa plist) | ✅ Keep, extend | macOS-native, restarts on crash, logs to file. Use it for orchestrator/manager/watchdog instead of start-claude.sh nohup. We already have one plist (com.glorylabs.swarm.designqa.plist) — pattern is proven. | | supervisord | ❌ Skip | Cross-platform but redundant with launchd on macOS; adds a Python dep and a config-file dialect. | | systemd timers | ❌ N/A | Linux-only; the supervising machine is a Mac. (VPS uses systemd, but agents don't run there.) | | just / make | ⚠️ Optional | Better ergonomics for the run-*.sh family, but not a supervisor. Defer. |

2.2 Job queues / event buses

| Tool | Verdict | Why | |---|---|---| | Redis Streams | ⚠️ Maybe — Phase 2 | Persistent, append-only, has consumer groups + ack/retry semantics (which bus.log doesn't). Would solve §1.3 (idempotency via message-id ACKs) and §1.7 (cursors). We already run Redis locally per project. Cost: every agent needs a Redis client; bus.log greppability is lost (need a bus tail CLI). | | NATS JetStream | ❌ Skip | Strictly more powerful than Redis Streams but adds an unfamiliar daemon. Redis is already on every dev box. | | Temporal | ❌ Skip | Brilliant for durable workflows; massive overkill for "run claude -p with a 30-min budget." Adds a server, a worker SDK, and a learning curve for ~3 humans. | | PostgreSQL LISTEN/NOTIFY | ❌ Skip | Same shape as Redis Streams without the consumer-group ergonomics. |

Bottom line: stay on bus.log until idempotency (§1.3) actually re-bites us. The grep-ability and "human can read it" property of a flat file is worth a lot.

2.3 Agent frameworks

| Tool | Verdict | Why | |---|---|---| | OpenAI Swarm (now archived; succeeded by Agents SDK) | ❌ Skip | Designed for handoffs between LLM agents in one process. We're orchestrating shell processes across machines and Claude sessions — different problem. | | LangGraph | ❌ Skip | State-machine for a single Python process. Doesn't model "ephemeral one-shot subprocess with a 5h credit budget." Adopting it = rewriting the swarm in Python around a graph runtime; the orchestration we have is the graph. | | CrewAI / AutoGen / LlamaIndex Workflows | ❌ Skip | Same shape as LangGraph: designed to coordinate LLM calls within an app, not to be a control plane for long-running CLI agents on different machines. They also assume OpenAI-shaped API access — re-plumbing them onto claude -p is non-trivial and brittle. | | Claude Code's own Task / Agent tool | ✅ Already using | The Agent/TaskCreate tooling in this very session is a structured agent framework — and it has proper lifecycle + cancel + result capture, which bus.log doesn't. Worth leaning harder on Claude-Code Agents for short-lived research/audit tasks instead of spinning ad-hoc one-shots. |

Bottom line: the gap isn't "we need an agent framework." It's "we need a 30-line supervisor with timeouts and idempotency." Any framework large enough to be useful is also large enough to recreate the bugs we just fixed.

2.4 Observability

| Tool | Verdict | Why | |---|---|---| | Langfuse (self-hosted) | ⚠️ Worth a half-day spike | Would give us per-agent cost + latency + token traces with a real UI. Today we have zero visibility into "which agent burned the 44h budget." Self-hostable, free tier ample. Risk: requires routing claude calls through a proxy that Langfuse can capture — claude-code CLI doesn't natively emit OTLP. | | Helicone | ⚠️ Same shape | Hosted; cheaper to start. Same proxy concern. | | OpenTelemetry | ❌ Premature | We don't have a service mesh of agents — we have ~10 shell scripts. OTel for a shell-script fleet is a tarpit. | | A tiny claude-usage.log aggregator | ✅ Adopt | We already have adhoc-claude-usage.log in .swarm/logs/. A 20-line awk summary + a daily cron post to the bus would catch §1.4 (budget stalls) without any new dep. |

2.5 Git-worktree-by-default

• Verdict: ✅ Adopt for ephemeral workers.
• Why: directly solves §1.2 (working-tree races). Each one-shot agent runs in its own throwaway worktree (git worktree add /tmp/swarm-<repo>-<ts> HEAD). The orchestrator/manager (persistent) stays on the primary checkout.
• Cost: ~10 lines in run-agent.sh. Cleanup: git worktree prune daily. Disk: negligible (worktrees share .git).
• Caveat: worktrees can't share a branch, so if two workers really need the same branch they must coordinate via the manager — which is the desired behavior anyway.

3. Light incremental fixes (no framework rewrite)

These are the high-leverage, low-blast-radius fixes. Roughly ordered by effort/value.

3.1 Hard timeout on every one-shot (fixes §1.1)

• Wrap claude -p … invocations in timeout --signal=TERM --kill-after=30s 1800 … (30-min budget, configurable per-task).
• On timeout, emit a bus line: <agent> -> manager: TIMEOUT after 1800s — task <id> killed, requeue=manual.
• ~5 lines in .swarm/run-agent.sh. Highest ROI fix in this document.

3.2 Per-repo claim-lock with PID + heartbeat (fixes §1.2 if §3.7 not adopted)

• Before mutating a repo's working tree, an agent takes <repo>/.swarm-lock ({pid, agent_id, started_at, heartbeat_at}). Heartbeats every 60s.
• Stale lock (heartbeat > 5 min OR pid not alive) ⇒ next agent steals it and notes the steal in the bus.
• Lower priority if we adopt §3.7 (worktrees) — worktrees make this unnecessary for ephemerals. Lock would still be useful for the persistent agents.

3.3 Structured bus messages (JSON-line, dual-write) (fixes §1.3, §1.7)

• Keep the human-readable line, but also write bus.jsonl with {ts, from, to, type, task_id, msg_id, in_reply_to, body}.
• task_id derived from a content hash of the task statement → automatic duplicate-dispatch detection (§3.6).
• Manager/orchestrator process bus.jsonl; humans still tail bus.log. Dual-write keeps the grep-ability we love.

3.4 Cursor file for owner messages (closes §1.7)

• Orchestrator persists last_owner_msg_seen_ts next to owner-messages-watchdog.log. On rehydrate, refuses to ACK anything older than the last unread owner message.

3.5 Verify-before-dispatch snapshot (fixes §1.5)

• Manager rule: before dispatching a "fix bug X" task, run a 30s sanity probe (kubectl get pods, gh pr view, curl /actuator/health) and embed the snapshot in the task body. The worker compares against current state at start and aborts if the snapshot is stale > 5 min.
• This is just a manager-prompt change + a small snapshot.sh.

3.6 Task fingerprint hash (fixes §1.3)

• task_id = sha1(repo + normalized_task_body) written into the inbox filename. run-agent.sh refuses to launch if task_id is already in .swarm/completed/ from the last 60 min.
• Cheap, additive, doesn't break the existing flow.

3.7 Git-worktree-by-default for ephemerals (fixes §1.2 cleanly)

• See §2.5. ~10 lines in run-agent.sh. Highest ROI after §3.1.

3.8 Ladder enforcement guard (fixes §1.6)

• A pre-action wrapper checks: if KUBECONFIG_CONTEXT == transip-k3s and the task wasn't explicitly tagged FINAL-PROMOTION, refuse and post NEED-HUMAN ladder-violation. One bash function in RECIPE.md-aligned helpers.

3.9 Budget-stall canary (fixes §1.4)

• Watchdog runs a 5-min claude -p 'echo ok' heartbeat. If it returns 429 / "Not logged in", it posts NEED-HUMAN credentials to the orchestrator immediately — no waiting for the next scheduled agent to fail silently.

3.10 Bus-replay test harness (fixes §1.3 + general regressions)

• Capture a 1-hour bus slice as a fixture. Add a tiny replay.sh that feeds it to a dry-run orchestrator and asserts no duplicate dispatches, no APPROVAL-NEEDED dropped, no crossed ACKs. Run before any change to MANAGER-PROMPT.md / ORCHESTRATOR-PROMPT.md.

4. Recommended path (priority order)

Do now (this week, ~4 hours total)

1. §3.1 — Hard timeout wrapper. Single biggest win. 5 lines. Eliminates ghost agents.
2. §3.7 — Git-worktree-by-default for ephemerals. Eliminates the working-tree race class entirely.
3. §3.4 — Owner-message cursor. Closes the crossed-ACK pattern; tiny change.
4. §3.9 — Budget-stall canary. Stops 44h dead windows.

Do next (next 1–2 weeks, ~1 day total)

5. §3.6 — Task fingerprint hash. Prevents duplicate dispatch on restart.
6. §3.5 — Verify-before-dispatch snapshot. Stops phantom-bugfix tasks.
7. §3.8 — Ladder enforcement guard. Hard-stops VPS-saturation regressions.
8. §3.10 — Bus-replay test harness. Catches prompt-edit regressions.

Worth a spike (~half day, then decide)

9. §2.4 — Langfuse self-hosted for per-agent cost+latency. Only commit if the spike shows we can wrap claude -p cleanly.

Skip / defer

• All agent frameworks (CrewAI, LangGraph, AutoGen, LlamaIndex Workflows, OpenAI Swarm/Agents SDK). They model the wrong problem; adopting one means rebuilding the swarm in their shape for no incremental capability. The Claude-Code Agent/TaskCreate tooling we already have is the right "framework" for short tasks.
• Temporal, NATS JetStream, full OTel. Overkill for ~10 shell scripts on one Mac + one VPS.
• §3.2 (PID locks) if §3.7 lands — worktrees subsume the use case for ephemerals.
• §3.3 structured bus messages unless §3.6 isn't enough — JSON-line dual-write is more invasive than the other fixes and the value is mostly downstream of duplicate-detection, which §3.6 already gives us.

5. What we are explicitly NOT proposing

• Rewriting the swarm in Python.
• Replacing bus.log as the human-facing log (humans must keep being able to tail -F it).
• Adopting any framework that requires routing claude -p through a different SDK.
• Moving persistent agents (manager/orchestrator/watchdog) onto the VPS. They stay on the laptop where the human is.
• Adding a UI / dashboard. The bus is the UI. Until that hurts more than it helps, leave it alone.

6. Open questions for the owner

1. Per-task timeout default — 30 min reasonable, or do some legitimate tasks (full-repo migration, secret rotation) need 60+? Suggest 30 min default with a # TIMEOUT: 3600 task-header override.
2. Worktree cleanup policy — keep last N successful worktrees for forensics, or prune aggressively? Suggest: keep failed-run worktrees, prune successful ones immediately.
3. Langfuse spike — worth a half day this week, or park until we hit another budget-stall outage?

End of proposal. No code touched. Recommended next step: a single PR implementing §3.1 + §3.7 (the two highest-ROI fixes), behind a SWARM_V2=1 env flag so we can A/B the rollout against the existing wrappers.