Compare commits

...
122 Commits
Author SHA1 Message Date
Huskies Agent aa9306912d huskies: regen source-map.json
Publish sled artifact / publish-dev-artifact (push) Canceled after 0s
2026-07-18 11:46:52 +00:00
Huskies Agent cebe9e2737 huskies: merge 1213 story Chat "stop" command that immediately aborts the in-flight LLM turn 2026-07-18 11:46:52 +00:00
TimmyandClaude Opus 4.8 b91e2d53ff docs: refresh README release section
Update the stale 0.7.1 example to 0.14.0, note the Linux arm64 target (the sleds' arch), and mention the branch+tag push added in bcac9266.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-18 12:28:58 +01:00
Timmy fdfa4bac08 Bump version to 0.14.0 2026-07-18 12:20:02 +01:00
TimmyandClaude Opus 4.8 bcac92669a script/release: push branch and tag together so master doesn't lag the tag
Previously the script pushed only the tag, leaving origin/master behind the release by the version-bump commit (the tag pointed at an unpushed commit). Push the current branch and the tag atomically with --atomic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-18 12:18:40 +01:00
Huskies Agent a43148ee13 huskies: regen source-map.json 2026-07-18 11:00:28 +00:00
Huskies Agent 71f3fa09c4 huskies: merge 1211 story Deterministic crash notification when the agent PTY dies mid-turn 2026-07-18 11:00:28 +00:00
Huskies Agent 153333d055 huskies: regen source-map.json 2026-07-18 10:45:40 +00:00
Huskies Agent b0f19eb0c5 huskies: merge 1212 story Sled chat messages show "workspace" instead of the real project name 2026-07-18 10:45:40 +00:00
Huskies Agent 36ec6d0f93 huskies: regen source-map.json 2026-07-18 10:12:31 +00:00
Huskies Agent 3f37a6cf34 huskies: merge 1210 story Gateway-side "digging in" notification for long tool-only turns 2026-07-18 10:12:31 +00:00
TimmyandClaude Opus 4.8 98f8825701 Fix compact no-op in gateway mode: add compact to GATEWAY_LOCAL_COMMANDS
Root cause of compact still failing after 1192/1205: in gateway mode,
on_room_message proxies any command not in GATEWAY_LOCAL_COMMANDS to the
active project's sled (proxy_bot_command) and returns — and that proxy
runs BEFORE the local compact interception (try_handle_compact_command).
'compact' was missing from the allowlist, so the gateway shipped it to a
sled (which has no compact command → no-op 'Command succeeded with no
response text'), and the real handler was never reached.

'reset' was already in the list, which is why reset worked in gateway
mode and compact did not. Add 'compact' as its sibling.

Why 1192 and 1205 both missed this: their tests drive
try_handle_compact_command directly, bypassing the gateway-proxy seam
that only exists on the full on_room_message path in gateway mode.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-18 09:37:05 +01:00
Huskies Agent 4ab34d3be2 huskies: regen source-map.json 2026-07-18 02:21:05 +00:00
Huskies Agent 8a7bff71aa huskies: merge 1209 story Gateway lifecycle &amp; telemetry MCP: gateway_info, restart_gateway, gateway_logs, start_story, chat_telemetry 2026-07-18 02:21:05 +00:00
Huskies Agent 3b10b29ef5 huskies: regen source-map.json 2026-07-18 02:05:03 +00:00
Huskies Agent 883f15045b huskies: merge 1204 story Slim CRDT-backed pipeline_query MCP tool (project / stages / fields / include_archived) 2026-07-18 02:05:03 +00:00
Huskies Agent cd2b417962 huskies: regen source-map.json 2026-07-18 01:54:05 +00:00
Huskies Agent 405d29d933 huskies: merge 1208 story Ops/LLM sessions can reach gateway-mode + cross-project MCP (the biggest shell-fallback cause) 2026-07-18 01:54:05 +00:00
Huskies Agent bb16f915f3 huskies: regen source-map.json 2026-07-18 01:43:24 +00:00
Huskies Agent 25bc952dff huskies: merge 1207 story fleet_resources MCP tool: on-demand host + per-container disk, load, and CPU/mem 2026-07-18 01:43:24 +00:00
Huskies Agent 5243854129 huskies: merge 1205 bug compact STILL swallowed after 1192 — interception is placed AFTER the registry dispatch, not before 2026-07-18 01:23:52 +00:00
Huskies Agent 486146ab34 huskies: regen source-map.json 2026-07-18 01:14:14 +00:00
Huskies Agent fba9b09d3e huskies: merge 1206 story fleet_identity MCP tool: read sled pins vs live signed identity, and re-pin via TOFU 2026-07-18 01:14:14 +00:00
Huskies Agent 8d2ad6424b huskies: regen source-map.json 2026-07-17 23:23:52 +00:00
Huskies Agent 75c4a8de33 huskies: merge 1203 bug Gateway status gives no response — 1187 made it proxy-only with no local resolution or error surfacing 2026-07-17 23:23:52 +00:00
Huskies Agent fd83ff2f53 huskies: merge 1202 bug Flaky test: matrix pull pull_rejects_tampered_artifact_sha256_mismatch fails nondeterministically in merge gates 2026-07-17 21:45:53 +00:00
Huskies Agent 38bdfa2ab4 huskies: regen source-map.json 2026-07-17 20:57:21 +00:00
Huskies Agent 5b340e7b20 huskies: merge 1199 story Orphaned build-dir GC: reclaim dead worktree targets without touching warm caches 2026-07-17 20:57:21 +00:00
Huskies Agent 2bd41d980c huskies: regen source-map.json 2026-07-17 19:43:27 +00:00
Huskies Agent 80efb7fcfc huskies: merge 1201 story Chat notification when a new work item is filed 2026-07-17 19:43:27 +00:00
Huskies Agent b9af302baf huskies: regen source-map.json 2026-07-17 19:25:52 +00:00
Huskies Agent c1523e8acf huskies: merge 1200 story Low-disk warning: the fleet tells the operator before the disk takes it down 2026-07-17 19:25:52 +00:00
Huskies Agent 82865956d2 huskies: merge 1195 bug Chat show renders metadata from stale content-text YAML instead of CRDT registers 2026-07-17 19:04:56 +00:00
Huskies Agent ecbed641bb huskies: regen source-map.json 2026-07-17 18:31:36 +00:00
Huskies Agent db27d0dbf3 huskies: merge 1198 bug Failed agents are never reaped — a dead pool entry blocks respawn indefinitely 2026-07-17 18:31:36 +00:00
Huskies Agent 2335fc0bbb huskies: merge 1194 bug db shadow-table tests flake when SHADOW_DB_PATH is not initialized 2026-07-17 18:06:34 +00:00
Huskies Agent ba1617934a huskies: merge 1190 bug Startup version announcement only fires on the trampoline path — normal restarts still say just "Timmy is online." 2026-07-17 17:42:23 +00:00
Huskies Agent c91ebb810c huskies: regen source-map.json 2026-07-17 17:33:55 +00:00
Huskies Agent b9d130bf64 huskies: merge 1193 story overview chat command: active work across all connected sleds 2026-07-17 17:33:55 +00:00
Huskies Agent 9db8a5006a huskies: merge 1196 bug Inactivity watchdog kills agents mid tool-call: awaited MCP calls produce no PTY output 2026-07-17 16:21:51 +00:00
Huskies Agent d4dde5d436 huskies: regen source-map.json 2026-07-17 14:24:56 +00:00
Huskies Agent a2b7b62960 huskies: merge 1191 bug /identity node_id field reports the CRDT id, not the node_identity.key the signature uses 2026-07-17 14:24:55 +00:00
Huskies Agent d77399aced huskies: regen source-map.json 2026-07-17 13:46:11 +00:00
Huskies Agent 6174828e18 huskies: merge 1192 bug compact command is swallowed by the registry placeholder before its real handler runs 2026-07-17 13:46:11 +00:00
Huskies Agent e991c6ac63 huskies: regen source-map.json 2026-07-17 13:32:30 +00:00
Huskies Agent bb5d7879ff huskies: merge 1189 story Gitea Actions workflow: build the sled artifact on master merges and publish to the dev channel 2026-07-17 13:32:30 +00:00
TimmyandClaude Fable 5 3ed3fdd6b0 Fold GatesFailed auto-retry into the block subscriber's shared budget
Code review of 1185 (merge 0f4b0c95) found the retry subscriber's central
invariant did not hold: the MergeFailure->Merge bounce caused by its own
retry reset both its attempt counter and the block subscriber's counter,
so the shared merge_failure_block_threshold budget was unreachable and a
deterministic gates failure retried forever.

One subscriber now owns one counter driving both policies:

- Counter survives PipelineEvent::MergeRetryStarted bounces (finding 1);
  a third consecutive failure blocks even with retries in between.
- Mixed failure kinds share the single budget (finding 5).
- Retries respect recovery: no counting or scheduling while a mergemaster
  is active, and perform_auto_retry re-checks before firing (finding 2).
- perform_auto_retry applies the same eligibility gates as
  assign_merge_stage (review hold, frozen, blocked, unmet deps) so freeze
  now stops a retry loop (finding 4).
- Per-story scheduling generations invalidate stale sleeping timers
  (finding 6).
- One-shot startup scan schedules a catch-up retry for stories already
  parked in GatesFailed, so restarts no longer strand them (finding 3);
  kept out of the periodic reconciler to avoid re-retrying exhausted
  stories every tick.
- Chat is notified only after the merge actually starts; a failed trigger
  logs instead of claiming a retry ran (finding 7).
- Config reads moved onto spawn_blocking (finding 8, bug 1170 class).

Deletes merge_failure_retry_subscriber.rs; notification plumbing
(WatcherEvent::MergeAutoRetry et al) is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-17 13:36:45 +01:00
Huskies Agent c717ae7041 huskies: regen source-map.json 2026-07-17 12:11:27 +00:00
Huskies Agent 0f4b0c9536 huskies: merge 1185 story Bounded auto-retry for GatesFailed merges before requiring human intervention 2026-07-17 12:11:27 +00:00
Huskies Agent 5d672f5bf6 huskies: regen source-map.json 2026-07-17 12:05:04 +00:00
Huskies Agent 043c77f077 huskies: merge 1186 story compact chat command: distill session context deterministically, then reset with a seed 2026-07-17 12:05:04 +00:00
Huskies Agent b241661941 huskies: regen source-map.json 2026-07-17 11:58:37 +00:00
Huskies Agent dac0278218 huskies: merge 1188 story Merge tool results return a summary, not the full gate log (35KB per call) 2026-07-17 11:58:37 +00:00
Huskies Agent ea8e6edc94 huskies: regen source-map.json 2026-07-17 11:49:43 +00:00
Huskies Agent eda14976d0 huskies: merge 1187 story status chat command shows that project's status 2026-07-17 11:49:43 +00:00
Huskies Agent 0d26ac5a2a huskies: regen source-map.json 2026-07-17 11:39:47 +00:00
Huskies Agent 39dd6e0151 huskies: merge 1184 bug Stage transitions never reach the gateway relay: StatusEvent::StageTransition is published only in tests 2026-07-17 11:39:47 +00:00
Huskies Agent 1d71877af6 huskies: merge 1182 bug Sled uplinks all register as 'workspace' — containers never get HUSKIES_PROJECT_NAME 2026-07-17 10:46:22 +00:00
Huskies Agent ae47dd29e8 huskies: merge 1183 story Config parse error hints at misplaced top-level keys after [[component]] 2026-07-17 10:35:40 +00:00
Huskies Agent faa7825799 huskies: merge 1181 refactor Dependency freshness sweep: bump manifest minimums, refresh lockfile, drop dead serde_yaml 2026-07-16 19:07:01 +00:00
Huskies Agent 97ed97885a huskies: regen source-map.json 2026-07-16 18:10:14 +00:00
Huskies Agent de7d22cb9f huskies: merge 1180 story Sled↔gateway goes WS-only: remove the deprecated HTTP fallback paths 2026-07-16 18:10:14 +00:00
Timmy 0e684fb06f Upgrading Docker to include newer node, so front-end builds 2026-07-16 19:03:48 +01:00
Huskies Agent e0ed35adf1 huskies: regen source-map.json 2026-07-16 17:20:20 +00:00
Huskies Agent aebe75cd04 huskies: merge 1179 story Token-authenticated sled→gateway WS uplink (remote-gateway ready) 2026-07-16 17:20:20 +00:00
Huskies Agent 0738005863 huskies: regen source-map.json 2026-07-16 16:22:46 +00:00
Huskies Agent efd0097f76 huskies: merge 1176 bug base_branch fallback hardcodes master instead of auto-detecting 2026-07-16 16:22:46 +00:00
Huskies Agent a4e2d4bc25 huskies: merge 1177 bug project.toml scaffold puts top-level keys after [[component]] so uncommenting them silently no-ops 2026-07-16 16:13:17 +00:00
TimmyandClaude Fable 5 0eacffa08d Install Node 22 from NodeSource in project base image
Bookworm's apt nodejs is 18.x; frontend toolchains after the 2026-07-15
dependency upgrades (vite 7) require Node >= 20, so any sled building a
frontend via build.rs failed. NodeSource nodejs bundles npm, so the
separate apt npm package is dropped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-16 16:39:51 +01:00
Huskies Agent e5df6232cf huskies: merge 1178 bug MergeFailureFinal is a trap state: successful re-merge cannot mark the story done 2026-07-16 14:19:55 +00:00
Huskies Agent 61acf98909 huskies: regen source-map.json 2026-07-16 14:03:31 +00:00
Huskies Agent 77e0394195 huskies: merge 1169 story Gateway pulls signed artifacts from a release channel into its local store 2026-07-16 14:03:31 +00:00
Huskies Agent 1e0e581bd7 huskies: regen source-map.json 2026-07-16 13:26:03 +00:00
Huskies Agent 0ac68afa4c huskies: merge 1173 story Identity-aware fleet checks: cryptographic node identity in upgrade and health probes 2026-07-16 13:26:03 +00:00
Huskies Agent 489c415fd9 huskies: regen source-map.json 2026-07-16 13:19:20 +00:00
Huskies Agent 18b065f77a huskies: merge 1163 story Replace perm_rx lock-as-presence-signal with a permission router 2026-07-16 13:19:20 +00:00
Huskies Agent 6f8a8ffd87 huskies: merge 1175 bug Flaky tests intermittently fail merge gates 2026-07-16 13:03:05 +00:00
TimmyandClaude Fable 5 6a1ee8377d docs: update stale libsqlite3-sys pin comment for sqlx 0.9 stable
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-16 11:46:26 +01:00
Huskies Agent a464febdcb huskies: merge 1174 story Gateway startup announcement reports version and model 2026-07-16 10:11:55 +00:00
Huskies Agent 8ae2eaaad3 huskies: merge 1160 bug Chat bot crash-loops on poisoned Claude Code session resume 2026-07-16 09:51:44 +00:00
Huskies Agent 23f2934e1f huskies: merge 1172 story Add Docker log rotation to project container launch args 2026-07-16 09:44:48 +00:00
Huskies Agent 69df921856 huskies: merge 1170 bug Full tokio runtime stall after unblock → merge auto-assign 2026-07-16 09:10:46 +00:00
TimmyandClaude Fable 5 f73689cfd8 Fix CRDT self-deadlock: read_llm_session re-locked the state mutex
read_llm_session acquired the CRDT_STATE mutex, then called
extract_llm_session_view while holding the guard — which called
our_node_id(), which locks the same non-reentrant std::sync::Mutex.
The thread deadlocks itself and parks forever HOLDING the lock; every
other CRDT user then queues behind it. With light traffic that's a
partial wedge (MCP `show`/content reads hang while /health stays
green); during a CRDT-write burst (unblock → auto-assign) enough
tasks pile up to pin every tokio worker: liveness heartbeat stops,
/health dies, full sled freeze. Root cause of bug 1170's repeated
sled freezes, confirmed by live gdb capture: thread parked in
lock_contended at presence::our_node_id ← read_llm_session ←
event_matches_persona, with all other threads queued on CRDT reads.

Fix: extract_llm_session_view now takes local_sled_id as a parameter;
read_llm_session computes it from the guard it already holds. The
trigger path (event_matches_persona on persona-subscribed WS events)
explains the raciness — it needs a chat/persona event racing a
pipeline transition.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-16 09:46:36 +01:00
Huskies Agent 1a15347b02 huskies: merge 1165 refactor Upgrade matrix-sdk 0.17 to 0.18 2026-07-16 05:14:46 +00:00
Huskies Agent a4bf11dbc1 huskies: merge 1166 refactor Upgrade ed25519-dalek 2 to 3 2026-07-16 01:08:34 +00:00
Huskies Agent 182771192b huskies: merge 1168 refactor Upgrade tokio-tungstenite 0.29 to 0.30 2026-07-16 00:53:47 +00:00
Huskies Agent 0a53c1ff0a huskies: merge 1167 refactor Upgrade garde 0.22 to 0.23 2026-07-16 00:38:34 +00:00
Huskies Agent ca05bd7224 huskies: merge 1164 refactor Lockfile refresh + sqlx alpha-to-stable 2026-07-16 00:13:38 +00:00
Huskies Agent aa977482db huskies: regen source-map.json 2026-07-16 00:03:44 +00:00
Huskies Agent e67eff17ad huskies: merge 1158 story Reject boilerplate user stories at save time 2026-07-16 00:03:44 +00:00
TimmyandClaude Fable 5 6cceec9c26 Disable CRDT debug logging in default features — fixes runtime stalls
logging-list/logging-json were in bft-json-crdt's default feature set,
so every production build printed multi-KB debug dumps on every CRDT
op — executed INSIDE the global CRDT_STATE mutex. A stdout write that
stalls while holding that lock blocks every task touching the CRDT
(tick loop, watchers, MCP, RPC), each one pinning an OS worker thread
until the tokio pool is exhausted: liveness heartbeat stops, /health
dies, zero CPU. This is the mechanism behind bug 1170 (two full-sled
freezes on huskies-server, both seconds after a CRDT write burst, the
second insert's dump truncated mid-print in the log).

The features remain available for CRDT debugging via explicit opt-in.

Also fixes a latent race in persist_tx_send_success_emits_no_warn:
it counted [crdt_persist] warns in the process-global log buffer,
which parallel tests also write to; the debug prints had been acting
as an accidental serializer. Now filters for its own story id.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 18:37:47 +01:00
TimmyandClaude Fable 5 be0c88c801 Pre-check binary writability before accepting an upgrade request
/api/upgrade now verifies the target can be replaced (create + remove
the swap's tmp file) before returning 202. A sled that cannot write
its own binary — e.g. a container predating the /opt/huskies/bin
layout — fails phase 1 of `upgrade all` loudly instead of returning
202, staying healthy, and silently remaining on the old version, which
is exactly what happened on the first fleet deploy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 17:53:56 +01:00
TimmyandClaude Fable 5 ba0a38d403 Fix WS deadlock: never block on perm_rx in the chat handler
Since story 884 the Matrix permission listener (and the sled uplink,
when configured) hold services.perm_rx for the process lifetime. The
WS chat handler's blocking `lock().await` on that same mutex therefore
parked the entire WS connection loop forever on StartChat: chat_fut was
never polled, RPC frames on the socket were never answered, and
everything queued behind the dispatcher's serial session lock —
wedging /mcp and /rpc while /health stayed green.

Use try_lock instead: if another task already owns permission routing,
run the chat without the local permission-forwarding select arm (a
pending future keeps the select shape unchanged).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 17:17:24 +01:00
TimmyandClaude Fable 5 01b24ff2ae Drop the drain check from upgrade — agent death is routine
Agents die all the time; the pipeline's retry machinery re-queues
their work. Skipping busy sleds just created version skew and manual
retries for no real protection. `upgrade all` now sweeps every sled
unconditionally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 16:54:55 +01:00
TimmyandClaude Fable 5 f39c4b7c4b Remove all alternate update paths — fleet redeploy is release + upgrade all
Killed:
- rebuild_and_restart (in-container cargo self-compile): the MCP tool,
  the `rebuild` chat command in all four transports, the web-ui bot
  command, and the underlying function. This was the path that caused
  the exec() deadlocks.
- upgrade_sled gateway MCP tool: second entry point to sled upgrades,
  defaulted to serving the gateway's own macOS binary to Linux sleds.
- GET /api/huskies-binary (both sled and gateway route trees): served
  current_exe(), wrong platform when the gateway is macOS. Superseded
  by /api/artifacts/ which now also serves on the gateway route tree.
- `huskies upgrade` CLI subcommand and --source flag: third way of
  doing the same download-and-replace. Escape hatch for a bricked sled
  is `docker cp` + restart.

Kept, distinct jobs: `project-rebuild` (container/image updates),
`rebuild gateway` + script/local-release (gateway self-update).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 16:46:11 +01:00
TimmyandClaude Fable 5 83f941b77e Revert /app/target binary fallback in entrypoint
Superseded: the running fleet was created by `new project` (no /app
mount), and the one-way upgrade path now replaces the binary at its
canonical install location (/opt/huskies/bin/huskies) directly, so the
entrypoint needs no fallback logic.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 16:28:32 +01:00
TimmyandClaude Fable 5 e8253a06b7 Add release chat command — build sled binary and publish artifact
Finds the registered project carrying the huskies source tree, runs
`cargo build --release` inside its container (dedicated
CARGO_TARGET_DIR=target/sled-release so container builds stop
clobbering host target/release), then atomically publishes the binary
to ~/.huskies/artifacts/ with a .hash sidecar for convergence checks.

Full fleet redeploy is now chat-only: `release` then `upgrade all` —
no laptop access needed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 16:27:32 +01:00
TimmyandClaude Fable 5 18ba57a7b0 Add upgrade all chat command with drain check and convergence verify
- `upgrade all` sweeps every registered sled in sequence, streaming
  per-sled phase markers and reporting a summary.
- Binary source is now the gateway's own artifact store via
  host.docker.internal (was: unresolvable `gateway` hostname serving
  the gateway's macOS binary to Linux sleds — would have bricked them).
- Sleds with active claude processes are skipped, never killed.
- After reconnect, /api/version git_hash is compared against the
  published artifact's .hash sidecar; divergence is reported loudly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 16:22:11 +01:00
TimmyandClaude Fable 5 07b9e1605d Serve sled binary artifacts from ~/.huskies/artifacts/
GET /api/artifacts/:filename with filename validation (no path
components, no dotfiles). Sleds only ever download binaries from their
own gateway; this endpoint is where the gateway serves them from,
replacing the current_exe()-based /api/huskies-binary which serves the
gateway's own (macOS) binary — wrong platform for Linux sleds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 16:12:00 +01:00
TimmyandClaude Fable 5 4d22171d16 Install sled binary in huskies-owned dir so upgrades work without root
/opt/huskies/bin/huskies (chowned to the huskies user) with a symlink
from /usr/local/bin/huskies. Atomic replace needs write permission on
the directory for the tmp-write + rename, which root-owned
/usr/local/bin can't provide to the server process.

resolve_target_path() now prefers /opt/huskies/bin/huskies over
current_exe(), which can point at a stale location (e.g.
/workspace/target/release/huskies after a historical in-container
rebuild) that the entrypoint would never launch after a restart.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 16:06:31 +01:00
TimmyandClaude Fable 5 a1ae532c6e Add /api/version endpoint for upgrade convergence checks
Reports crate version + compile-time BUILD_GIT_HASH as JSON. The
gateway will poll this after `upgrade all` to verify each sled is
actually running the published artifact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
2026-07-15 16:03:49 +01:00
Timmy 2a7db16a77 Fixed exec bug 2026-07-15 14:55:27 +01:00
TimmyandClaude Opus 4.6 549e0349d7 Fix rebuild_and_restart in Docker project containers
CARGO_MANIFEST_DIR is baked at image build time as /app/server, but
project containers (Dockerfile.base) don't copy /app — the source is
bind-mounted at /workspace instead. Fall back to project_root when
the compile-time path doesn't exist.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-14 18:42:55 +01:00
Huskies Agent 16d4294f08 huskies: regen source-map.json 2026-06-29 19:38:13 +00:00
Huskies Agent c733fb2bf9 huskies: merge 1162 refactor Update workspace Cargo.toml dependencies to latest compatible versions 2026-06-29 19:38:13 +00:00
Timmy 4c965c73b2 Added CRDT snapshotting 2026-06-29 20:11:52 +01:00
Timmy 3342d129c2 Unreachable container times out gracefully 2026-06-29 19:22:47 +01:00
Timmy 32646c6256 Catching a docker rm problem 2026-06-29 18:27:50 +01:00
Timmy 0a0ab65908 Adding doc comments. 2026-06-29 17:02:17 +01:00
Timmy feb35ddd10 Converted all external tool calling to async 2026-06-29 16:59:54 +01:00
Timmy 75f41088b1 Dealing with bot failures 2026-06-29 15:34:14 +01:00
Timmy b662a7da95 Stack detection fix 2026-06-29 13:14:06 +01:00
Timmy 146205c83b Added docs comment 2026-06-29 12:45:25 +01:00
Timmy 705f5bcc89 Adding show story mcp 2026-06-29 12:42:45 +01:00
dave 8285a98f80 huskies: regen source-map.json 2026-05-20 01:18:59 +00:00
dave 2fb935e726 huskies: merge 1156 story Periodic liveness tick so runtime freezes have a precise timestamp 2026-05-20 01:18:59 +00:00
dave 7be3bf1dbf huskies: regen source-map.json 2026-05-20 00:42:25 +00:00
dave 2a5359051e huskies: merge 1154 story Extend gateway_health with a relay-working signal — is each sled actually delivering events? 2026-05-20 00:42:24 +00:00
dave a3ac09f8a3 huskies: regen source-map.json 2026-05-20 00:23:44 +00:00
dave 846b3e1b4c huskies: merge 1153 story huskies projects chat command — list every registered project with port and status 2026-05-20 00:23:44 +00:00
dave 0c207981e9 huskies: regen source-map.json 2026-05-19 23:54:05 +00:00
dave e7456d3391 huskies: merge 1155 story Bracket logging around install_pre_commit_hook to diagnose bug 1151 freezes 2026-05-19 23:54:05 +00:00
236 changed files with 20895 additions and 5398 deletions
+53
View File
@@ -0,0 +1,53 @@
# Gitea Actions workflows
## `release-artifact.yml`
Triggers on every push to `master`. Builds the `linux-arm64` sled binary and
publishes it to the "dev" release channel via `script/ci-publish-artifact`.
### Runner registration
The job targets the `arm64-mac` runner label. Register an `act_runner` on an
Apple Silicon macOS host that has `cargo`/`rustc` and `curl` on `PATH`:
```sh
act_runner register \
--instance https://code.crashlabs.io \
--token <runner-registration-token> \
--labels arm64-mac
act_runner daemon
```
The registration token comes from the repo's **Settings → Actions →
Runners → Create new Runner** page in Gitea. Without a runner carrying the
`arm64-mac` label, jobs from this workflow queue indefinitely.
### Secrets
Configure these under the repo's **Settings → Actions → Secrets**. Never
commit credentials — the workflow only ever references them via
`${{ secrets.* }}`.
| Secret | Purpose |
| --- | --- |
| `HUSKIES_CHANNEL_URL` | Base URL of the dev release channel host. |
| `HUSKIES_CHANNEL_TOKEN` | Bearer token authorised to publish artifacts to that channel. |
### Channel host contract
`script/ci-publish-artifact` expects the channel host at
`HUSKIES_CHANNEL_URL` to implement:
- `POST {HUSKIES_CHANNEL_URL}/<artifact-name>` — accepts the raw artifact
bytes as the request body. Requires `Authorization: Bearer <token>` and
`X-Git-Hash: <short-git-hash>` headers. Non-2xx responses in the 4xx range
(including 401/403) are treated as permanent failures; 5xx responses and
network errors are retried with backoff.
- `GET {HUSKIES_CHANNEL_URL}/manifest.json` — returns a JSON object with a
`git_hash` field reflecting the most recently published artifact.
Requires `Authorization: Bearer <token>`.
This is a separate, unsigned channel distinct from the Ed25519-signed
release channels the `pull <channel>` gateway command consumes (see
`server/src/service/gateway/release_manifest.rs`) — CI has no safe place to
hold a channel signing key, so the dev channel trusts the bearer token alone.
+33
View File
@@ -0,0 +1,33 @@
name: Publish sled artifact
on:
push:
branches:
- master
jobs:
publish-dev-artifact:
# Registered on an arm64 macOS act_runner host — see
# .gitea/workflows/README.md for registration instructions.
runs-on: [arm64-mac]
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build linux-arm64 sled binary
env:
# Dedicated target dir so this CI build never clobbers a developer's
# incremental target/release build on the shared runner host, mirroring
# SLED_TARGET_DIR in server/src/chat/transport/matrix/release.rs.
CARGO_TARGET_DIR: target/ci-release
run: cargo build --release -p huskies
- name: Stage artifact
run: cp target/ci-release/release/huskies target/ci-release/release/huskies-linux-arm64
- name: Publish to dev channel
env:
HUSKIES_CHANNEL_URL: ${{ secrets.HUSKIES_CHANNEL_URL }}
HUSKIES_CHANNEL_TOKEN: ${{ secrets.HUSKIES_CHANNEL_TOKEN }}
run: script/ci-publish-artifact target/ci-release/release/huskies-linux-arm64 "$(git rev-parse --short HEAD)"
+225 -39
View File
@@ -113,6 +113,12 @@
"crates/bft-json-crdt/tests/byzantine.rs": [],
"crates/bft-json-crdt/tests/commutative.rs": [],
"crates/bft-json-crdt/tests/kleppmann_trace.rs": [],
"crates/release-manifest/src/lib.rs": [
"struct ReleaseManifest",
"fn canonical_bytes",
"struct SignedManifest"
],
"crates/release-tool/src/main.rs": [],
"crates/source-map-gen/src/lib.rs": [
"struct CheckFailure",
"fn to_direction",
@@ -516,6 +522,7 @@
"struct MergeReport"
],
"server/src/agents/merge/squash/mod.rs": [
"fn merge_lock_is_free",
"fn run_squash_merge",
"fn cleanup_merge_workspace"
],
@@ -611,6 +618,9 @@
"server/src/agents/pool/auto_assign/watchdog/orphan.rs": [
"fn check_orphaned_agents"
],
"server/src/agents/pool/auto_assign/watchdog/reap.rs": [
"fn reap_failed_agents"
],
"server/src/agents/pool/auto_assign/watchdog/tests/limits_tests.rs": [],
"server/src/agents/pool/auto_assign/watchdog/tests/mod.rs": [
"fn write_fake_session_log",
@@ -619,6 +629,7 @@
"fn write_project_config"
],
"server/src/agents/pool/auto_assign/watchdog/tests/orphan_tests.rs": [],
"server/src/agents/pool/auto_assign/watchdog/tests/reap_tests.rs": [],
"server/src/agents/pool/cost_rollup_subscriber.rs": [
"fn reconcile_cost_rollup",
"fn spawn_cost_rollup_subscriber",
@@ -685,6 +696,7 @@
"fn available_agents_for_stage",
"fn list_running_merges",
"fn list_agents",
"fn list_agents_nonblocking",
"fn subscribe",
"fn drain_events",
"fn get_log_info"
@@ -904,8 +916,26 @@
"server/src/chat/commands/unreleased.rs": [
"fn handle_unreleased"
],
"server/src/chat/compact/digest.rs": [
"fn build_digest"
],
"server/src/chat/compact/mod.rs": [
"mod digest",
"mod transcript",
"struct CompactCommand",
"fn extract_compact_command",
"struct CompactOutcome",
"enum CompactError",
"fn compact_session",
"fn handle_compact_for_key",
"fn frame_seed_for_prompt"
],
"server/src/chat/compact/transcript.rs": [
"fn transcript_path"
],
"server/src/chat/dispatcher.rs": [
"type SpawnFn",
"enum StopOutcome",
"struct ChatDispatcher",
"fn new",
"fn submit",
@@ -921,6 +951,7 @@
],
"server/src/chat/mod.rs": [
"mod commands",
"mod compact",
"mod dispatcher",
"mod history",
"mod lookup",
@@ -955,7 +986,8 @@
],
"server/src/chat/transport/discord/meta.rs": [
"struct DiscordTransport",
"fn new"
"fn new",
"fn with_api_base"
],
"server/src/chat/transport/discord/mod.rs": [
"mod commands",
@@ -979,7 +1011,9 @@
"fn effective_project_root",
"fn is_gateway",
"fn active_project_url",
"fn proxy_bot_command"
"fn project_url",
"fn proxy_bot_command",
"fn run_proxy_bot_command"
],
"server/src/chat/transport/matrix/bot/format.rs": [
"fn format_startup_announcement",
@@ -1003,6 +1037,7 @@
"fn is_addressed_to_other"
],
"server/src/chat/transport/matrix/bot/messages/handle_message.rs": [
"fn spawn_digging_in_watcher",
"fn handle_message"
],
"server/src/chat/transport/matrix/bot/messages/mod.rs": [
@@ -1040,6 +1075,9 @@
"fn handle_cleanup_worktrees"
],
"server/src/chat/transport/matrix/commands.rs": [],
"server/src/chat/transport/matrix/compact.rs": [
"fn handle_compact"
],
"server/src/chat/transport/matrix/config/loading.rs": [
"fn load",
"fn effective_room_ids",
@@ -1051,9 +1089,11 @@
"server/src/chat/transport/matrix/config/types.rs": [
"fn default_history_size",
"fn default_permission_timeout_secs",
"fn default_aggregated_notifications_poll_interval_secs",
"fn default_aggregated_notifications_enabled",
"fn default_coalesce_window_ms",
"fn default_compact_seed_max_bytes",
"fn default_cache_read_suggest_threshold",
"fn default_compact_suggest_cooldown_secs",
"fn default_digging_in_threshold_secs",
"fn default_transport",
"fn default_whatsapp_provider",
"struct BotConfig"
@@ -1080,17 +1120,22 @@
"mod assign",
"mod cleanup_worktrees",
"mod commands",
"mod compact",
"mod config",
"mod delete",
"mod health",
"mod htop",
"mod new_project",
"mod project_rebuild",
"mod projects",
"mod pull",
"mod rebuild",
"mod release",
"mod reset",
"mod rmtree",
"mod sled_upgrade",
"mod start",
"mod stop",
"mod transport_impl",
"fn spawn_bot"
],
@@ -1098,6 +1143,7 @@
"struct NewProjectCommand",
"fn extract_new_project_command",
"fn apply_project_config",
"fn stacks_dir",
"fn detect_stack",
"fn image_for_stack",
"fn resolve_git_identity",
@@ -1112,11 +1158,21 @@
"fn extract_project_rebuild_command",
"fn handle_project_rebuild"
],
"server/src/chat/transport/matrix/projects.rs": [
"fn run_projects_list"
],
"server/src/chat/transport/matrix/pull.rs": [
"fn extract_pull_command",
"fn handle_pull"
],
"server/src/chat/transport/matrix/rebuild.rs": [
"struct RebuildCommand",
"fn extract_rebuild_command",
"fn extract_rebuild_gateway_command",
"fn handle_rebuild"
"fn extract_rebuild_gateway_command"
],
"server/src/chat/transport/matrix/release.rs": [
"fn extract_release_command",
"fn find_builder_project",
"fn handle_release"
],
"server/src/chat/transport/matrix/reset.rs": [
"struct ResetCommand",
@@ -1132,6 +1188,7 @@
"enum UpgradeCommand",
"fn extract_upgrade_command",
"fn handle_upgrade_list_projects",
"fn handle_upgrade_all",
"fn handle_sled_upgrade"
],
"server/src/chat/transport/matrix/start.rs": [
@@ -1139,6 +1196,10 @@
"fn extract_start_command",
"fn handle_start"
],
"server/src/chat/transport/matrix/stop.rs": [
"struct StopCommand",
"fn extract_stop_command"
],
"server/src/chat/transport/matrix/transport_impl.rs": [
"struct MatrixTransport",
"fn new"
@@ -1256,6 +1317,7 @@
"fn drain_complete_paragraphs",
"fn normalize_line_breaks"
],
"server/src/ci_publish_artifact.rs": [],
"server/src/cli.rs": [
"struct CliArgs",
"fn parse_cli_args",
@@ -1271,6 +1333,7 @@
"mod agent_name",
"struct ProjectConfig",
"struct WatcherConfig",
"struct DiskWatchConfig",
"struct ComponentConfig",
"struct AgentConfig",
"fn load",
@@ -1677,10 +1740,6 @@
"struct AppContext",
"fn new_test"
],
"server/src/http/events.rs": [
"struct EventsQuery",
"fn events_handler"
],
"server/src/http/gateway/jsonrpc.rs": [
"struct JsonRpcRequest",
"struct JsonRpcResponse",
@@ -1715,9 +1774,13 @@
"fn gateway_event_push_handler"
],
"server/src/http/identity.rs": [
"struct IdentityQuery",
"struct IdentityResponse",
"fn identity_handler"
],
"server/src/http/mcp/agent_tools/gc.rs": [
"fn tool_gc"
],
"server/src/http/mcp/agent_tools/inspection.rs": [
"fn tool_get_agent_output",
"fn tool_get_agent_config",
@@ -1725,6 +1788,7 @@
],
"server/src/http/mcp/agent_tools/lifecycle.rs": [
"fn tool_start_agent",
"fn tool_start_story",
"fn tool_stop_agent",
"fn tool_list_agents",
"fn tool_wait_for_agent"
@@ -1738,9 +1802,11 @@
"fn tool_get_editor_command",
"fn get_worktree_commits"
],
"server/src/http/mcp/diagnostics/chat_telemetry.rs": [
"fn tool_chat_telemetry"
],
"server/src/http/mcp/diagnostics/mod.rs": [
"fn tool_get_server_logs",
"fn tool_rebuild_and_restart",
"fn tool_move_story",
"fn tool_dump_crdt",
"fn tool_get_version",
@@ -1819,7 +1885,7 @@
"fn tool_run_check"
],
"server/src/http/mcp/status_tools.rs": [
"fn tool_status"
"fn tool_show"
],
"server/src/http/mcp/story_tools/bug.rs": [
"fn tool_create_bug",
@@ -1869,7 +1935,8 @@
"server/src/http/mcp/story_tools/story/query.rs": [
"fn tool_validate_stories",
"fn tool_list_upcoming",
"fn tool_get_pipeline_status"
"fn tool_get_pipeline_status",
"fn tool_pipeline_query"
],
"server/src/http/mcp/story_tools/story/update.rs": [
"fn tool_update_story",
@@ -1912,7 +1979,6 @@
"server/src/http/mod.rs": [
"mod agents_sse",
"mod context",
"mod events",
"mod identity",
"mod mcp",
"mod oauth",
@@ -1925,11 +1991,14 @@
"fn write_port_file",
"fn remove_port_file",
"fn health_handler",
"fn version_handler",
"fn build_routes",
"fn rpc_http_handler",
"fn debug_crdt_handler",
"fn upgrade_trigger_handler",
"fn serve_binary_handler"
"const SLED_ARTIFACT_NAME",
"fn artifacts_dir",
"fn serve_artifact_handler"
],
"server/src/http/oauth.rs": [
"fn oauth_authorize",
@@ -2216,6 +2285,7 @@
"server/src/llm/providers/claude_code/events/tests.rs": [],
"server/src/llm/providers/claude_code/mod.rs": [
"struct ClaudeCodeResult",
"const CANCELLED",
"struct ClaudeCodeProvider",
"fn new",
"fn chat_stream"
@@ -2303,7 +2373,8 @@
"struct NodeIdentity",
"fn load_or_create_keypair_file",
"fn init_identity",
"fn get_identity"
"fn get_identity",
"fn sign_challenge_with_identity"
],
"server/src/pidfile.rs": [
"struct PidfileGuard",
@@ -2346,6 +2417,8 @@
"struct AuditLogSubscriber",
"fn reconcile_audit_log",
"fn spawn_audit_log_subscriber",
"struct StatusBroadcastSubscriber",
"fn spawn_status_broadcast_subscriber",
"struct MatrixBotSubscriber",
"struct FileRendererSubscriber",
"struct PipelineItemsSubscriber",
@@ -2402,7 +2475,7 @@
"fn new",
"fn notify_startup",
"fn notify",
"fn rebuild_and_restart"
"fn drain_and_exit"
],
"server/src/service/agents/cost_rollup.rs": [
"struct CostRollup",
@@ -2474,7 +2547,6 @@
"fn call_start",
"fn call_delete",
"fn call_rmtree",
"fn call_rebuild",
"fn call_timer",
"fn call_htop",
"fn call_sync"
@@ -2492,6 +2564,11 @@
"fn parse_start",
"fn parse_number"
],
"server/src/service/chat_telemetry.rs": [
"struct ChatTurnTelemetry",
"fn record",
"fn recent"
],
"server/src/service/common/item_id.rs": [
"fn extract_item_number",
"fn has_valid_id_prefix"
@@ -2511,6 +2588,22 @@
"fn generate_permission_rule",
"fn is_dominated_by_wildcard"
],
"server/src/service/disk_watch/io.rs": [
"fn free_space_bytes",
"fn dir_size_bytes",
"fn check_and_notify"
],
"server/src/service/disk_watch/mod.rs": [
"mod io",
"enum DiskLevel",
"fn as_str",
"fn classify_level",
"fn is_recovered",
"fn should_send",
"struct DiskWatchState",
"enum DiskAction",
"fn decide_action"
],
"server/src/service/event_triggers/mod.rs": [
"mod store",
"struct TriggerPredicate",
@@ -2532,23 +2625,12 @@
"fn parse_mode"
],
"server/src/service/events/buffer.rs": [
"const MAX_BUFFER_SIZE",
"enum StoredEvent",
"fn timestamp_ms",
"struct EventBuffer",
"fn new",
"fn push",
"fn events_since"
],
"server/src/service/events/io.rs": [
"fn now_ms",
"fn subscribe_to_watcher"
"fn timestamp_ms"
],
"server/src/service/events/mod.rs": [
"mod buffer",
"mod io",
"enum Error",
"fn events_since"
"enum Error"
],
"server/src/service/file_io/io.rs": [
"fn read_file",
@@ -2576,25 +2658,42 @@
"fn exec_shell"
],
"server/src/service/gateway/aggregation.rs": [
"fn format_aggregate_status_compact"
"fn format_aggregate_status_compact",
"fn format_overview_compact",
"fn format_identity_reports",
"fn find_project_containing_story"
],
"server/src/service/gateway/config.rs": [
"struct ProjectEntry",
"fn with_url",
"fn has_url",
"struct ReleaseChannelConfig",
"struct GatewayConfig",
"fn validate_config",
"fn validate_project_exists",
"fn toml_string",
"fn serialize_bot_config"
],
"server/src/service/gateway/identity.rs": [
"struct IdentityProbeResponse",
"enum IdentityCheck",
"fn check_identity",
"struct SledIdentityReport",
"fn build_identity_report"
],
"server/src/service/gateway/io.rs": [
"fn load_config",
"fn save_config",
"fn read_bot_config_raw",
"struct BotConfigFields",
"fn write_bot_config",
"fn proxy_mcp_call",
"fn probe_identity",
"fn probe_sled_identity_report",
"fn repin_sled_identity",
"fn fetch_release_manifest",
"fn download_channel_artifact",
"fn install_pulled_artifact",
"fn read_installed_manifest",
"fn proxy_mcp_call_sse",
"fn fetch_one_project_pipeline_status",
"fn fetch_all_project_pipeline_statuses",
@@ -2608,17 +2707,21 @@
"fn scaffold_project",
"fn init_wizard_state",
"fn spawn_gateway_broadcaster_forwarder",
"fn spawn_gateway_notification_poller",
"type ActiveProject",
"fn spawn_gateway_bot"
],
"server/src/service/gateway/mod.rs": [
"mod aggregation",
"mod config",
"mod identity",
"mod io",
"mod polling",
"mod release_manifest",
"mod resources",
"fn gateway_uptime_secs",
"struct GatewayStatusEvent",
"const HEARTBEAT_MAX_AGE_MS",
"const RELAY_MAX_AGE_SECS",
"const MCP_VIA_WS_TIMEOUT_MS",
"struct SledConnection",
"fn is_alive",
@@ -2630,8 +2733,10 @@
"fn active_url",
"fn register_sled_connection",
"fn deregister_sled_connection",
"fn sled_connection_for",
"fn active_sled_connection",
"fn proxy_active_mcp",
"fn proxy_mcp_for_project",
"fn switch_project",
"fn generate_join_token",
"fn register_agent",
@@ -2644,12 +2749,50 @@
"fn init_project",
"fn broadcast_status_event",
"fn subscribe_status_events",
"fn fleet_identity_read",
"fn fleet_identity_repin",
"fn save_bot_config_and_restart"
],
"server/src/service/gateway/polling.rs": [
"fn format_gateway_event",
"fn format_gateway_audit_line"
],
"server/src/service/gateway/release_manifest.rs": [
"fn verify_manifest_signature",
"fn verify_artifact_sha256",
"fn check_rollback"
],
"server/src/service/gateway/resources/io.rs": [
"fn collect_host_stats",
"fn collect_container_stats",
"fn collect_project_dir_sizes",
"fn collect_fleet_resources"
],
"server/src/service/gateway/resources/mod.rs": [
"mod io",
"struct HostStats",
"struct ContainerStats",
"struct ProjectDirSizes",
"struct ResourceThresholds",
"struct ResourceFlag",
"struct FleetResources",
"fn flag_host",
"fn parse_docker_cpu_percent",
"fn parse_docker_size",
"fn parse_docker_mem_usage",
"fn parse_meminfo"
],
"server/src/service/gc/io.rs": [
"fn run_gc_pass"
],
"server/src/service/gc/mod.rs": [
"mod io",
"struct GcCandidate",
"struct GcReport",
"fn select_orphaned_worktree_targets",
"fn select_merge_workspace",
"fn should_auto_run"
],
"server/src/service/git_ops/io.rs": [
"fn validate_worktree_path",
"fn run_git",
@@ -2670,29 +2813,42 @@
"server/src/service/git_ops/porcelain.rs": [
"fn parse_git_status_porcelain"
],
"server/src/service/merge/io.rs": [],
"server/src/service/merge/io.rs": [
"fn write_merge_report"
],
"server/src/service/merge/mod.rs": [
"mod io",
"mod status",
"mod summary",
"enum Error"
],
"server/src/service/merge/status.rs": [
"fn format_merge_status_message"
],
"server/src/service/merge/summary.rs": [
"const SUMMARY_TAIL_LINES",
"fn extract_failing_test_names",
"fn summarize_merge_result",
"fn summarize_merge_failure_kind"
],
"server/src/service/mod.rs": [
"mod agents",
"mod anthropic",
"mod bot_command",
"mod chat_telemetry",
"mod common",
"mod diagnostics",
"mod disk_watch",
"mod event_triggers",
"mod events",
"mod file_io",
"mod gateway",
"mod gc",
"mod git_ops",
"mod merge",
"mod notifications",
"mod oauth",
"mod permission_router",
"mod pipeline",
"mod project",
"mod qa",
@@ -2713,9 +2869,10 @@
"const RATE_LIMIT_DEBOUNCE",
"const STAGE_TRANSITION_DEBOUNCE",
"const AGENT_EVENT_DEBOUNCE",
"const NEW_ITEM_COALESCE_WINDOW",
"fn should_send_rate_limit"
],
"server/src/service/notifications/format.rs": [
"server/src/service/notifications/format/mod.rs": [
"fn stage_display_name",
"fn format_stage_notification",
"fn format_error_notification",
@@ -2725,10 +2882,18 @@
"fn format_oauth_accounts_exhausted",
"fn format_agent_started_notification",
"fn format_agent_completed_notification",
"fn format_agent_crashed_notification",
"struct NewItemInfo",
"fn format_new_item_notification",
"fn format_new_items_notification",
"fn format_merge_auto_retry_notification",
"fn format_disk_warning_notification",
"fn format_disk_recovery_notification",
"fn project_display_name",
"const MERGE_FAILURE_TAIL_LINES",
"fn truncate_gate_output"
],
"server/src/service/notifications/format/tests.rs": [],
"server/src/service/notifications/io/listener.rs": [
"fn spawn_notification_listener"
],
@@ -2739,7 +2904,8 @@
],
"server/src/service/notifications/io/mod.rs": [
"mod stage_subscriber",
"fn read_story_name"
"fn read_story_name",
"fn resolve_project_display_name"
],
"server/src/service/notifications/io/stage_subscriber.rs": [
"fn spawn_stage_notification_subscriber"
@@ -2797,6 +2963,21 @@
"fn percent_encode",
"fn build_authorize_url"
],
"server/src/service/permission_router.rs": [
"const RESPONDER_CHANNEL_CAPACITY",
"struct ResponderRegistry",
"fn new",
"fn is_empty",
"fn register",
"fn dispatch",
"struct ResponderGuard",
"fn spawn_permission_router",
"struct PendingPermReplies",
"fn new",
"fn insert",
"fn resolve_oldest",
"fn remove_by_request_id"
],
"server/src/service/pipeline/mod.rs": [
"fn aggregate_pipeline_counts"
],
@@ -3088,6 +3269,8 @@
],
"server/src/sled_uplink.rs": [
"struct UplinkConfig",
"fn to_ws_url",
"fn build_uplink_url",
"const HEARTBEAT_INTERVAL_SECS",
"struct UplinkEnvelope",
"fn spawn_uplink_task"
@@ -3112,6 +3295,7 @@
"server/src/startup/tick_loop.rs": [
"fn spawn_event_bridges",
"fn spawn_tick_loop",
"fn spawn_liveness_tick",
"fn spawn_gateway_relay",
"fn spawn_event_trigger_subscriber",
"fn run_reconcile_pass",
@@ -3138,7 +3322,7 @@
"server/src/upgrade.rs": [
"fn fetch_and_replace_binary",
"fn upgrade_and_reexec",
"fn run_cli_upgrade",
"fn preflight_target_writable",
"fn resolve_target_path"
],
"server/src/validation/error.rs": [
@@ -3150,6 +3334,7 @@
"const NAME_MAX_LEN",
"const DESCRIPTION_MAX_LEN",
"const AC_MAX_LEN",
"fn check_boilerplate_user_story",
"struct StoryName",
"fn parse",
"struct AcceptanceCriterion",
@@ -3235,6 +3420,7 @@
"server/src/worktree/git.rs": [
"fn branch_name",
"fn detect_base_branch",
"fn resolve_base_branch",
"fn configure_sparse_checkout",
"fn create_worktree_sync",
"fn remove_worktree_sync",
@@ -0,0 +1,46 @@
# CRDT Snapshot Compaction
## Problem
The huskies project CRDT has grown to 55K ops / 276MB in `pipeline.db`.
Every container restart replays all operations in a tight synchronous loop
on the tokio runtime (`crdt_state/state/init.rs:68-78`), taking 27+ minutes
and freezing the runtime so the HTTP server never becomes ready.
### Op bloat
55K ops across only 4,154 sequence numbers (~13 ops per seq on average).
Many zeroed-out `MergeJobCrdt` entries appear to be tombstones never cleaned
up. Some individual ops are up to 523KB. Average op size is 4.6KB.
## Proposed Fix
### 1. Snapshot (checkpoint)
After replaying all ops, serialize the materialized CRDT state to a
checkpoint blob (e.g. a `crdt_snapshot` table or a separate file). On next
startup, load the snapshot and only replay ops with `rowid > snapshot_rowid`.
At snapshot time, back up the database file so corruption is recoverable.
### 2. Op pruning / compaction
Delete ops that are superseded by the snapshot. Tombstoned/deleted items with
all-zero fields contribute nothing to materialized state and can be dropped
from the log once snapshotted.
### 3. Immediate fix: spawn_blocking
Move the replay loop to `tokio::task::spawn_blocking` so the HTTP server and
liveness ticks are not starved during replay. This does not reduce replay
time but prevents the runtime freeze.
## Acceptance Criteria
- Startup loads a snapshot when available and only replays ops newer than the snapshot sequence
- A snapshot is written after full CRDT replay completes (or periodically in background)
- DB backup is created at each snapshot time
- Startup time for 55K ops drops from 27+ minutes to under 30 seconds
- Dead/zeroed CRDT ops are pruned during compaction
- CRDT sync protocol continues to work correctly across nodes after compaction
- The replay loop runs in spawn_blocking so it does not freeze the tokio runtime
@@ -0,0 +1,127 @@
# Story 1208: Cross-Project MCP for Ops/LLM Sessions
## 1. Problem Statement
An ops/LLM session connects to a `huskies --gateway` instance's `/mcp`
endpoint. Before this story, the *only* way to act on a specific registered
project was:
1. Call `switch_project` (mutates the gateway's shared, global
`GatewayState.active_project`), then
2. Call the ordinary project-level tool (`create_story`, `get_story_todos`,
`show`, …), which the gateway silently proxies to whichever project is
currently active.
This has two problems:
- **Race condition**: `active_project` is one value shared by every
connected client. Two concurrent ops sessions targeting different projects
will step on each other's `switch_project` calls.
- **No true "read a named project once" path**: for a single lookup against
a project that isn't the current default, a caller had to mutate shared
state just to read something, then (optionally) switch back.
The practical consequence (and the reason this story exists) is that
operators and LLM agents fall back to hand-crafting raw JSON-RPC requests
directly against a project's own container port, bypassing the gateway
entirely — the "shell-fallback" this story is named for.
## 2. Chosen Mechanism: Per-Call `project` Argument
Any `tools/call` request for a non-gateway (proxied) tool may now include an
optional top-level `project` field inside `arguments`:
```json
{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {
"name": "create_story",
"arguments": {
"name": "Fix login bug",
"acceptance_criteria": ["..."],
"origin": "...",
"project": "robot-studio"
}
}
}
```
- If `project` is present and non-empty, the gateway looks it up in
`projects.toml` (`GatewayState.projects`) and proxies the call directly to
that project's live sled-uplink WebSocket connection —
`GatewayState::proxy_mcp_for_project` in
`server/src/service/gateway/mod.rs`. `GatewayState.active_project` is
**not read or mutated** by this path.
- If `project` is absent (the common case, and all pre-existing behavior),
the call proxies to whichever project is currently active, exactly as
before — full backward compatibility with existing sessions and
`switch_project`-based workflows.
- An unknown project name returns a JSON-RPC `-32602` (invalid params)
error listing the registered project names. A known project with no live
WS-uplink connection returns `-32603` naming the sled, matching the
existing `active_project` proxy error shape.
Implementation: `server/src/http/gateway/mcp.rs`
(`gateway_mcp_post_handler`'s `tools/call` branch,
`proxy_and_respond_for_project`) and
`server/src/service/gateway/mod.rs` (`GatewayState::sled_connection_for`,
`GatewayState::proxy_mcp_for_project`, generalized from the existing
`active_sled_connection` / `proxy_active_mcp`).
### Schema discoverability
`tools/list` merges gateway tools with the active project's own tool list.
Every merged (proxied) tool's `inputSchema.properties` gets a `project`
property injected (`inject_project_arg_schema` in `http/gateway/mcp.rs`) so
MCP clients that validate call arguments against the declared schema before
sending don't strip or reject the extra field. This is additive only — no
existing property, and no `required` list, is touched.
### Why not mirror every tool at the gateway level?
Rejected alternative: define a `project_create_story`, `project_show`, etc.
for every project-level tool at the gateway. This was rejected because it
duplicates ~15+ tool schemas and dispatch arms and drifts out of sync every
time a project-level tool's schema changes. A single per-call argument that
every proxied tool call can carry scales to new project-level tools for
free.
## 3. Fleet-Wide Reads (AC 2)
These already existed as gateway-level tools before this story and needed
no code change — listed here for completeness of the "how an ops session
connects" picture:
| Tool | Purpose |
|------|---------|
| `list_projects` | Every registered project: name, url, ssh_port, host_path, adopted/built-in marker, active marker. No liveness check. |
| `gateway_health` | Per-project health (WS heartbeat or HTTP poll) plus CRDT event-relay staleness. |
| `aggregate_pipeline_status` | Pipeline stage counts and blocked/failing items across every registered project, fetched in parallel. |
| `fleet_identity` | (Story 1206) Per-sled identity pin vs. live signed identity, and TOFU re-pin. |
## 4. How an Ops Session Should Connect
1. Point the MCP client at the gateway's `/mcp` endpoint
(`http://<gateway-host>:<port>/mcp`), the same endpoint local agents use
— there is no separate "ops" endpoint.
2. Call `tools/list` to see the merged tool surface (gateway tools + the
active project's tools, each carrying the optional `project` schema
property).
3. For a one-off call against a specific project, pass `project: "<name>"`
inside `arguments` on that call — no `switch_project` required, and no
risk of racing another session's active-project selection.
4. For fleet-wide questions (is anything down, what's blocked everywhere),
use `list_projects`, `gateway_health`, or `aggregate_pipeline_status`
directly; they already scan every registered project.
5. `switch_project` remains available for sessions that want a persistent
default (e.g. an interactive chat session working one project at a
time) — it is unaffected by this change.
## 5. Design Review Note (AC 4)
This document captures the chosen approach (per-call `project` argument,
generalized proxy functions, additive schema injection) as required by AC 4.
No new gateway-level tool surface was added for AC 1 — the existing proxy
path was extended instead, minimizing new schema/dispatch surface area and
keeping every future project-level tool automatically cross-project-capable.
@@ -0,0 +1,245 @@
# LLM Context From Events
Design overview for making any LLM-driven chat persona (Timmy at the
gateway, Sally at a single sled, future personas) aware of huskies
events without the user having to re-narrate them.
## Goal
**Update the LLM's context non-intrusively when a state transition
happens.** No new LLM turn is fired; events are simply visible to the
LLM the next time the user (or anything else) does cause it to run.
The LLM should never need to be told what already happened inside
huskies.
## Guiding Principle
**Transports have nothing to do with LLMs.** A transport (Matrix bot,
web UI, CLI, future TUIs) is a pure courier — it relays user text in,
LLM text out, and never owns LLM-facing state. Anything the LLM needs
to know lives in huskies, behind a single `assemble_prompt_context`
helper that the transport calls. Adding a new transport must require
zero changes to the event-awareness path.
## Three things this doc is NOT
1. **Triggers**`on StoryMerged{1122} do Rebuild`. These are
deterministic subscribers; they should never invoke the LLM. Covered
in a separate design.
2. **Proactive wake** — running an LLM turn *because* an event fired,
without the user typing. Costs tokens, risks ramble. Explicitly out
of scope here; a separate decision to make later.
3. **A transport feature** — this design assumes any transport that
invokes the LLM uses the same context-assembly helper. Matrix bot,
web UI, CLI all funnel through it.
## Why Past Attempts Have Failed
- **Buffer lived on the transport**, not on huskies. The current
`BotContext.pending_pipeline_events` (`server/src/chat/transport/matrix/bot/context.rs:103-116`)
is Matrix-only; web UI users see nothing of the kind, and the buffer
dies with the bot process.
- **Process-local, RAM-only**. Server rebuild → buffer empty. Any
events between the old binary's last user turn and the new binary's
first are silently lost.
- **Unbounded `mpsc` channels drop under lag.** The server logs
routinely show `[xxx-sub] Subscriber lagged, skipped N event(s)`.
When the subscriber feeding the buffer falls behind, events vanish
without being recorded.
- **No end-to-end test.** Nothing asserts "fire event E, send user
message M, the LLM's prompt contains E."
- **No cross-process aggregation.** Events in a sled have no path to
the gateway-side LLM context without bespoke plumbing per event type.
## Architecture at a Glance
```
┌────────────┐ ┌────────────┐ ┌────────────┐
│ Sled A │ │ Sled B │ │ Sled C │
│ event_log/ │ │ event_log/ │ │ event_log/ │ ◄── source of truth
└─────┬──────┘ └─────┬──────┘ └─────┬──────┘ (CRDT-backed)
│ │ │
└───────────────┼───────────────┘
┌────────────────────┐
│ Gateway aggregator │ ◄── tail-merges all sled logs
│ event_view/ │ into a single ordered stream
└─────────┬──────────┘
┌────────────────────────┐
│ Per-LLM-session state │ ◄── scope filter +
│ sessions/<id>/ │ high-water mark per stream
└─────────┬──────────────┘
┌────────────────────────┐
│ assemble_prompt_context│ ◄── single helper used by
│ (session_id) -> Str │ every transport before
└─────────┬──────────────┘ each LLM turn
┌──────────────┼──────────────┐
▼ ▼ ▼
Matrix bot Web UI CLI / TUI
```
## Event Model — Reuse What Already Exists
There is no need to invent a parallel event taxonomy. Huskies already
has a complete typed enum and a single broadcast bus:
- `server/src/pipeline_state/transition.rs` defines `PipelineEvent`
with **30 variants** covering every state-machine transition
(`DepsMet`, `GatesStarted/Passed/Failed`, `QaSkipped`,
`MergeSucceeded/Failed/FailedFinal`, `Accepted`, `Block/Unblock`,
`Abandon`, `Supersede`, `ReviewHold/Cleared`, `Reject`, `Triage`,
`Close`, `Demote`, `Freeze/Unfreeze`, `MergemasterAttempted`,
`FixupRequested`, `ReQueuedForQa`, `MergeAborted`,
`HotfixRequested`, `MergeRetryStarted`).
- The same module defines `ExecutionEvent` with 7 variants for agent
lifecycle (`SpawnRequested`, `SpawnedSuccessfully`, `Heartbeat`,
`HitRateLimit`, `Exited`, `Stopped`, `Reset`).
- Every transition fires a `TransitionFired` event on a single internal
bus. Ten subscribers already consume it (audit-log,
worktree-create-sub, worktree-cleanup-sub, merge-failure-sub,
merge-block-sub, done-archive-sub, content-gc, cost-rollup-sub,
stage-notification-sub, event-triggers).
**The LLM context injector is just the 11th subscriber on the same
bus.** It writes typed events into the per-sled CRDT event log
described below; everything downstream reuses the existing taxonomy.
Each persisted entry carries:
```
struct LoggedEvent {
id: EventId, // monotonic per sled
sled_id: SledId,
timestamp: UnixSeconds,
transition: TransitionFired, // story_id + from + to + PipelineEvent
// (or ExecutionEvent — see open question)
}
```
The few events that genuinely don't fit the pipeline state machine
(e.g. `ProjectAdopted`, `Rebuilt`, `GatewayHealthChanged`) live in a
small, separately-enumerated `InfraEvent` enum, but the same log and
the same subscriber pattern still apply.
## Session Model
An LLM session is a first-class CRDT entity:
```
struct LlmSession {
id: SessionId,
persona: Persona, // "Timmy", "Sally", ...
scope: ScopeFilter, // { sleds: All } | { sleds: Set<SledId> }
high_water: BTreeMap<SledId, EventId>, // per-stream
created: UnixSeconds,
}
```
The session id is what the transport carries; it's not the Matrix room,
not the web socket id. A given Matrix room may map to one session; a
web UI tab may map to another. Multiple transports for the same human
can share a session if you want — that's a separate UX call.
## Prompt Assembly Contract
Every transport calls one helper before invoking the LLM:
```
fn assemble_prompt_context(session_id: SessionId) -> String
```
Behavior:
1. Read the session's scope filter and high-water marks.
2. Fetch events from the gateway aggregator that match the scope and
are newer than the high-water marks.
3. Render them as a single `<system-reminder>` block, ordered by sled
then timestamp.
4. Advance the high-water marks to the latest event seen, atomically
with the LLM-turn-start CRDT op (so a crash mid-turn doesn't double-
inject).
5. Return the rendered block (empty string if no new events).
The transport prepends the result to the user's prompt and invokes the
LLM as usual.
## Persistence & Reliability Rules
- **Event log is CRDT-backed.** Survives sled restart.
- **High-water marks are CRDT-backed.** Survives gateway restart.
- **Aggregator uses bounded queues with drop-oldest semantics**, and
every drop logs `[event-agg] dropped N events for session <id>; client
must re-fetch from <high-water>`. The aggregator never silently
swallows events — if the queue is full, the session gets a sentinel
event `EventStreamGap { from, to }` so the LLM can see it missed
context.
- **End-to-end test required**: `fire(Event::StoryMerged{1122}) → user
sends "what's going on?" → assembled prompt contains "1122 merged"`.
## Multi-Persona Scoping
The same machinery serves both Timmy and Sally:
| Persona | Scope filter | Notes |
|---------|------------------------------------|--------------------------------|
| Timmy | `{ sleds: All }` | Gateway-wide; aware of every sled |
| Sally | `{ sleds: { huskies-server } }` | Single-sled; sled-local events only |
| Manny | `{ sleds: { huskies, ketflix } }` | Hypothetical; subset |
Sally never has to know Timmy exists, and vice versa. Their sessions
advance their own high-water marks against the same underlying log.
## Decisions
| Decision | Choice | Alternative |
|------------------------|-------------------------------------|----------------------------------------------|
| Event publication | Each sled owns its log | Single global log: cross-sled bottleneck |
| Aggregation | Gateway tail-merges | Each session pulls from each sled directly: N×M fanout |
| Buffer location | CRDT-persisted | In-process: lost on rebuild (current bug) |
| Event identity | Typed enum | Strings: structured-log creep, no compile-time safety |
| Drop semantics | Drop-oldest + `EventStreamGap` | Silent drop (current bug): LLM lies confidently |
| Session ↔ transport | Session is separate from transport | One per transport: web tab + Matrix get different views |
| Proactive LLM wake | OUT OF SCOPE | Wake on every event: cost + ramble |
## Open Questions
1. **Session lifecycle**. How are sessions created and garbage-
collected? Created on first transport message? GC'd after N days
idle?
2. **Event retention**. How long are events kept in the log? Forever
feels wrong; "since last terminal session turn" feels right but
needs care for multi-session readers.
3. **Multi-transport same session**. Should one human's Matrix and web
UI share a session by default, or always be separate?
4. **Render budget**. If 500 events accumulated between turns, do we
render all 500 or summarize? A `summarize_events` fallback path is
probably worth designing in from the start.
5. **Aggregator placement when there is no gateway**. A standalone
single-sled install has no gateway — does the sled itself host the
aggregator? (Probably yes; trivially "aggregates" its own log.)
## Phasing
- **Phase 0 (now):** this design doc.
- **Phase 1:** typed `Event` enum + per-sled CRDT-backed event log;
one publisher subscribes to existing pipeline transitions and writes
`StoryStaged` / `StoryMerged` / `StoryMergeFailed`.
- **Phase 2:** `LlmSession` CRDT entity + `assemble_prompt_context`
helper, wired into the Matrix bot's `handle_message` (replaces the
existing `pending_pipeline_events` Vec). End-to-end test covering the
fire-event → user-turn → prompt-contains-event contract.
- **Phase 3:** Gateway aggregator over multiple sleds; Timmy's session
scoped to `All`. Sally's session scoped to a single sled.
- **Phase 4:** Web UI and any other transports migrated onto
`assemble_prompt_context`; the Matrix-specific Vec deleted.
- **Phase 5:** Bounded queues + `EventStreamGap` sentinel; observability
for `assemble_prompt_context` runs (events injected, gaps observed).
Each phase ships independently. Phase 2 alone delivers the user-facing
fix: Timmy sees what merged when you next say anything, without you
needing to re-narrate.
Generated
+536 -895
View File
File diff suppressed because it is too large Load Diff
+46 -36
View File
@@ -1,58 +1,68 @@
[workspace]
members = ["server", "crates/bft-json-crdt", "crates/source-map-gen"]
members = [
"server",
"crates/bft-json-crdt",
"crates/source-map-gen",
"crates/release-manifest",
"crates/release-tool",
]
resolver = "3"
[workspace.dependencies]
async-stream = "0.3"
async-stream = "0.3.6"
async-trait = "0.1.89"
bytes = "1"
chrono = { version = "0.4.44", features = ["serde"] }
chrono-tz = "0.10"
bytes = "1.12.1"
chrono = { version = "0.4.45", features = ["serde"] }
chrono-tz = "0.10.4"
eventsource-stream = "0.2.3"
futures = "0.3"
futures = "0.3.32"
homedir = "0.3.6"
ignore = "0.4.25"
mime_guess = "2"
ignore = "0.4.29"
mime_guess = "2.0.5"
notify = "8.2.0"
poem = { version = "3", features = ["websocket", "test"] }
poem = { version = "3.1.12", features = ["websocket", "test"] }
portable-pty = "0.9.0"
reqwest = { version = "0.13.3", features = ["json", "stream"] }
rust-embed = "8"
ed25519-dalek = { version = "2", default-features = false, features = ["rand_core"] }
reqwest = { version = "0.13.4", features = ["json", "stream"] }
rust-embed = "8.12.0"
ed25519-dalek = { version = "3.0.0", default-features = false, features = ["rand_core"] }
indexmap = { version = "2.14.0", features = ["serde"] }
rand = "0.10"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_urlencoded = "0.7"
sha1 = "0.11"
rand = "0.10.2"
serde = { version = "1.0.228", features = ["derive"] }
serde_json = "1.0.150"
serde_urlencoded = "0.7.1"
sha1 = "0.11.0"
sha2 = "0.11.0"
hmac = "0.13"
subtle = "2"
base64 = "0.22"
serde_yaml = "0.9"
strip-ansi-escapes = "0.2"
tempfile = "3"
tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync"] }
toml = "1.1.2"
uuid = { version = "1.23.1", features = ["v4", "serde"] }
tokio-tungstenite = { version = "0.29.0", features = ["connect", "rustls-tls-native-roots"] }
hmac = "0.13.0"
subtle = "2.6.1"
base64 = "0.22.1"
strip-ansi-escapes = "0.2.1"
tempfile = "3.27.0"
tokio = { version = "1.52.4", features = ["rt-multi-thread", "macros", "sync"] }
toml = "1.1.3"
uuid = { version = "1.24.0", features = ["v4", "serde"] }
tokio-tungstenite = { version = "0.30.0", features = ["connect", "rustls-tls-native-roots"] }
walkdir = "2.5.0"
filetime = "0.2"
matrix-sdk = { version = "0.17", default-features = false, features = [
filetime = "0.2.29"
# 0.18 is current as of this pin (verified via `cargo search matrix-sdk`).
# matrix-sdk-sqlite 0.18 pulls in rusqlite 0.37, which requires libsqlite3-sys
# ==0.35.x — see the pin rationale in server/Cargo.toml. Bumping matrix-sdk
# past 0.18 may require re-checking that ceiling.
matrix-sdk = { version = "0.18", default-features = false, features = [
"sqlite",
"e2e-encryption",
] }
pulldown-cmark = { version = "0.13.3", default-features = false, features = [
pulldown-cmark = { version = "0.13.4", default-features = false, features = [
"html",
] }
regex = "1"
libc = "0.2"
nutype = { version = "0.7", features = ["serde"] }
garde = { version = "0.22", features = ["derive"] }
ammonia = "4.1"
sqlx = { version = "=0.9.0-alpha.1", default-features = false, features = [
regex = "1.13.1"
libc = "0.2.186"
nutype = { version = "0.7.0", features = ["serde"] }
garde = { version = "0.23", features = ["derive"] }
ammonia = "4.1.3"
sqlx = { version = "0.9.0", default-features = false, features = [
"runtime-tokio",
"sqlite",
"macros",
"migrate",
] }
serde_yaml = "0.9.34"
+2 -2
View File
@@ -95,10 +95,10 @@ Internal architecture documentation lives in [`docs/architecture/`](docs/archite
Requires a Gitea API token in `.env` (`GITEA_TOKEN=your_token`).
```bash
script/release 0.7.1
script/release 0.14.0
```
This bumps version in `Cargo.toml` and `package.json`, builds macOS arm64 and Linux amd64 binaries, tags the repo, and publishes a Gitea release with changelog and binaries attached.
This bumps version in `Cargo.toml` and `package.json`, builds macOS arm64, Linux amd64, and Linux arm64 binaries, tags the repo, pushes the branch and tag, and publishes a Gitea release with changelog and binaries attached.
## Multi-node CRDT sync (rendezvous)
+11 -5
View File
@@ -6,8 +6,14 @@ edition = "2021"
[lib]
crate-type = ["lib"]
# The logging-* features print multi-KB debug dumps on every CRDT op — and
# they execute inside the global CRDT state mutex in the server, so a stalled
# stdout write while holding that lock can pin every tokio worker and freeze
# the whole process (bug 1170). They are development tools: opt in explicitly
# with `--features bft-json-crdt/logging-list` when debugging CRDT internals.
# Never enable them in a production build.
[features]
default = ["bft", "logging-list", "logging-json"]
default = ["bft"]
logging-list = ["logging-base"]
logging-json = ["logging-base"]
logging-base = []
@@ -15,18 +21,18 @@ bft = []
[dependencies]
bft-crdt-derive = { path = "bft-crdt-derive" }
colored = "3"
colored = "3.1.1"
ed25519-dalek = { workspace = true }
indexmap = { workspace = true, features = ["serde"] }
rand = { workspace = true }
random_color = "1"
random_color = "1.1.0"
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true, features = ["preserve_order"] }
serde_with = "3"
serde_with = "3.21.0"
sha2 = { workspace = true }
[dev-dependencies]
criterion = { version = "0.8", features = ["html_reports"] }
criterion = { version = "0.8.2", features = ["html_reports"] }
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true, features = ["preserve_order"] }
@@ -8,8 +8,8 @@ publish = false
proc-macro = true
[dependencies]
indexmap = { version = "2.2.6", features = ["serde"] }
proc-macro2 = "1.0.47"
proc-macro-crate = "3"
quote = "1.0.21"
syn = { version = "2", features = ["full"] }
indexmap = { version = "2.14.0", features = ["serde"] }
proc-macro2 = "1.0.106"
proc-macro-crate = "3.5.0"
quote = "1.0.46"
syn = { version = "2.0.119", features = ["full"] }
+5
View File
@@ -20,6 +20,10 @@ use std::{
/// An RGA-like list CRDT that can store a CRDT-like datatype
#[derive(Clone, Serialize, Deserialize)]
#[serde(bound(
serialize = "T: serde::Serialize",
deserialize = "T: serde::de::DeserializeOwned"
))]
pub struct ListCrdt<T>
where
T: CrdtNode,
@@ -32,6 +36,7 @@ where
pub ops: Vec<Op<T>>,
/// Queue of messages where K is the ID of the message yet to arrive
/// and V is the list of operations depending on it
#[serde(skip)]
message_q: HashMap<OpId, Vec<Op<T>>>,
/// The sequence number of this node
our_seq: SequenceNumber,
+6 -1
View File
@@ -7,6 +7,7 @@
use crate::debug::DebugView;
use crate::json_crdt::{CrdtNode, JsonValue, OpState};
use crate::op::{join_path, print_path, Op, PathSegment, SequenceNumber};
use serde::{Deserialize, Serialize};
use std::cmp::{max, Ordering};
use std::fmt::Debug;
@@ -14,7 +15,11 @@ use crate::keypair::AuthorId;
/// A simple delete-wins, last-writer-wins (LWW) register CRDT.
/// Basically only for adding support for primitives within a more complex CRDT
#[derive(Clone)]
#[derive(Clone, Serialize, Deserialize)]
#[serde(bound(
serialize = "T: serde::Serialize",
deserialize = "T: serde::de::DeserializeOwned"
))]
pub struct LwwRegisterCrdt<T>
where
T: CrdtNode,
+8
View File
@@ -0,0 +1,8 @@
[package]
name = "release-manifest"
version = "0.1.0"
edition = "2024"
[dependencies]
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true }
+126
View File
@@ -0,0 +1,126 @@
//! Shared release-manifest type for the signed release-channel pull pipeline.
//!
//! The publisher tool (`crates/release-tool`) builds a [`ReleaseManifest`],
//! serializes it to canonical bytes, signs those bytes with the channel's
//! Ed25519 private key, and publishes the resulting [`SignedManifest`] as
//! `manifest.json` on the release channel. The gateway (`huskies-server`)
//! fetches that file, re-serializes the embedded manifest with
//! [`ReleaseManifest::canonical_bytes`], and verifies the signature against
//! its pinned public key before trusting anything in it.
//!
//! Keeping the type in its own dependency-light crate lets both sides agree
//! on the exact byte representation to sign/verify without the server crate
//! ever linking signing code, and without the publisher tool depending on
//! the full `huskies` server crate.
use serde::{Deserialize, Serialize};
/// The signed payload describing one published release artifact.
///
/// Field order is significant: [`ReleaseManifest::canonical_bytes`] relies on
/// `serde_json`'s struct serialization preserving declaration order, so the
/// signer and verifier always agree on the exact bytes being signed.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ReleaseManifest {
/// Filename of the artifact within the channel (e.g. `huskies-linux-arm64`).
pub artifact: String,
/// Lowercase hex sha256 digest of the artifact's bytes.
pub sha256: String,
/// Version identifier — the short git commit hash the artifact was built from.
pub version: String,
/// Release channel name this manifest was signed for (e.g. `stable`).
pub channel: String,
/// Unix timestamp (seconds) the manifest was signed at.
///
/// Used for rollback/replay detection: a pull refuses any manifest whose
/// timestamp is not strictly newer than the currently installed one.
pub timestamp: i64,
}
impl ReleaseManifest {
/// Serialize this manifest deterministically for signing and verification.
///
/// Both the publisher and the gateway construct this independently from
/// their own in-memory `ReleaseManifest` value — the manifest.json file's
/// exact on-disk byte layout is never itself the signed payload.
pub fn canonical_bytes(&self) -> Vec<u8> {
serde_json::to_vec(self).expect("ReleaseManifest serialization cannot fail")
}
}
/// A [`ReleaseManifest`] plus its Ed25519 signature (lowercase hex), as
/// published to a release channel's `manifest.json`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SignedManifest {
/// The manifest describing the published artifact.
pub manifest: ReleaseManifest,
/// Hex-encoded Ed25519 signature over `manifest.canonical_bytes()`.
pub signature: String,
}
#[cfg(test)]
mod tests {
use super::*;
fn sample() -> ReleaseManifest {
ReleaseManifest {
artifact: "huskies-linux-arm64".to_string(),
sha256: "a".repeat(64),
version: "abc1234".to_string(),
channel: "stable".to_string(),
timestamp: 1_700_000_000,
}
}
#[test]
fn canonical_bytes_is_deterministic() {
let m = sample();
assert_eq!(m.canonical_bytes(), m.canonical_bytes());
}
#[test]
fn canonical_bytes_changes_with_any_field() {
let m1 = sample();
let mut m2 = sample();
m2.timestamp += 1;
assert_ne!(m1.canonical_bytes(), m2.canonical_bytes());
}
#[test]
fn signed_manifest_roundtrips_through_json() {
let signed = SignedManifest {
manifest: sample(),
signature: "deadbeef".to_string(),
};
let json = serde_json::to_string(&signed).unwrap();
let parsed: SignedManifest = serde_json::from_str(&json).unwrap();
assert_eq!(parsed.manifest, signed.manifest);
assert_eq!(parsed.signature, signed.signature);
}
#[test]
fn manifest_missing_field_fails_to_parse() {
let bad = serde_json::json!({
"artifact": "huskies-linux-arm64",
"sha256": "a".repeat(64),
"version": "abc1234",
"channel": "stable"
// timestamp missing
});
let result: Result<ReleaseManifest, _> = serde_json::from_value(bad);
assert!(
result.is_err(),
"manifest missing a field must fail to parse"
);
}
#[test]
fn signed_manifest_missing_signature_fails_to_parse() {
let bad = serde_json::json!({ "manifest": sample() });
let result: Result<SignedManifest, _> = serde_json::from_value(bad);
assert!(
result.is_err(),
"signed manifest missing signature must fail to parse"
);
}
}
+18
View File
@@ -0,0 +1,18 @@
[package]
name = "release-tool"
version = "0.1.0"
edition = "2024"
[[bin]]
name = "release-tool"
path = "src/main.rs"
[dependencies]
release-manifest = { path = "../release-manifest" }
ed25519-dalek = { workspace = true }
sha2 = { workspace = true }
rand = { workspace = true }
serde_json = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
+311
View File
@@ -0,0 +1,311 @@
//! `release-tool` — offline publisher CLI for signed release channels.
//!
//! Generates a release-channel Ed25519 keypair and signs release manifests
//! for a channel's `manifest.json`. This binary is intentionally its own
//! crate, depending only on [`release_manifest`] and `ed25519-dalek` — it
//! never links against the `huskies` server crate, so the running gateway
//! has no code path that can read a channel's private signing key. Run this
//! tool offline (or in a separate publish pipeline) and copy only the
//! resulting public key hex into the gateway's `projects.toml`.
//!
//! Usage:
//! ```text
//! release-tool keygen <key-out-path>
//! release-tool sign --key <path> --artifact <path> --version <str> --channel <str> --out <path> [--timestamp <unix-secs>]
//! ```
use ed25519_dalek::{Signer, SigningKey};
use rand::Rng;
use release_manifest::{ReleaseManifest, SignedManifest};
use sha2::{Digest, Sha256};
use std::path::{Path, PathBuf};
fn main() {
let args: Vec<String> = std::env::args().collect();
let result = match args.get(1).map(String::as_str) {
Some("keygen") => run_keygen(&args[2..]),
Some("sign") => run_sign(&args[2..]),
_ => Err(
"usage: release-tool keygen <key-out-path> | release-tool sign --key <path> \
--artifact <path> --version <str> --channel <str> --out <path> [--timestamp <unix-secs>]"
.to_string(),
),
};
if let Err(e) = result {
eprintln!("error: {e}");
std::process::exit(1);
}
}
// ── keygen ───────────────────────────────────────────────────────────────────
fn run_keygen(args: &[String]) -> Result<(), String> {
let key_path = args.first().ok_or("keygen requires a key-out-path")?;
let signing_key = generate_signing_key();
write_seed_file(Path::new(key_path), &signing_key)?;
let pubkey_hex = hex_encode(signing_key.verifying_key().as_bytes());
println!("Wrote private key seed to {key_path}");
println!("Pinned release public key (paste into projects.toml as `pubkey`):");
println!("{pubkey_hex}");
Ok(())
}
fn generate_signing_key() -> SigningKey {
let mut seed = [0u8; 32];
rand::rng().fill_bytes(&mut seed);
SigningKey::from_bytes(&seed)
}
fn write_seed_file(path: &Path, signing_key: &SigningKey) -> Result<(), String> {
if let Some(parent) = path.parent()
&& !parent.as_os_str().is_empty()
{
std::fs::create_dir_all(parent)
.map_err(|e| format!("cannot create {}: {e}", parent.display()))?;
}
#[cfg(unix)]
{
use std::io::Write;
use std::os::unix::fs::OpenOptionsExt;
let mut file = std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.mode(0o600)
.open(path)
.map_err(|e| format!("cannot create {}: {e}", path.display()))?;
file.write_all(&signing_key.to_bytes())
.map_err(|e| format!("cannot write {}: {e}", path.display()))
}
#[cfg(not(unix))]
{
std::fs::write(path, signing_key.to_bytes())
.map_err(|e| format!("cannot write {}: {e}", path.display()))
}
}
fn load_seed_file(path: &Path) -> Result<SigningKey, String> {
let bytes = std::fs::read(path).map_err(|e| format!("cannot read {}: {e}", path.display()))?;
let seed: [u8; 32] = bytes
.try_into()
.map_err(|_| format!("{} must contain exactly 32 bytes", path.display()))?;
Ok(SigningKey::from_bytes(&seed))
}
// ── sign ─────────────────────────────────────────────────────────────────────
/// Parsed `sign` subcommand arguments.
struct SignArgs {
key: PathBuf,
artifact: PathBuf,
version: String,
channel: String,
out: PathBuf,
timestamp: Option<i64>,
}
fn parse_sign_args(args: &[String]) -> Result<SignArgs, String> {
let mut key = None;
let mut artifact = None;
let mut version = None;
let mut channel = None;
let mut out = None;
let mut timestamp = None;
let mut i = 0;
while i < args.len() {
let flag = args[i].as_str();
let value = args
.get(i + 1)
.ok_or_else(|| format!("missing value for {flag}"))?;
match flag {
"--key" => key = Some(PathBuf::from(value)),
"--artifact" => artifact = Some(PathBuf::from(value)),
"--version" => version = Some(value.clone()),
"--channel" => channel = Some(value.clone()),
"--out" => out = Some(PathBuf::from(value)),
"--timestamp" => {
timestamp = Some(
value
.parse::<i64>()
.map_err(|_| format!("--timestamp must be an integer, got `{value}`"))?,
)
}
other => return Err(format!("unknown flag `{other}`")),
}
i += 2;
}
Ok(SignArgs {
key: key.ok_or("--key is required")?,
artifact: artifact.ok_or("--artifact is required")?,
version: version.ok_or("--version is required")?,
channel: channel.ok_or("--channel is required")?,
out: out.ok_or("--out is required")?,
timestamp,
})
}
fn run_sign(args: &[String]) -> Result<(), String> {
let parsed = parse_sign_args(args)?;
let signing_key = load_seed_file(&parsed.key)?;
let artifact_bytes = std::fs::read(&parsed.artifact)
.map_err(|e| format!("cannot read {}: {e}", parsed.artifact.display()))?;
let artifact_name = parsed
.artifact
.file_name()
.and_then(|n| n.to_str())
.ok_or("--artifact path has no filename")?
.to_string();
let timestamp = match parsed.timestamp {
Some(t) => t,
None => std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_err(|e| format!("system clock before epoch: {e}"))?
.as_secs() as i64,
};
let signed = sign_manifest(
&signing_key,
artifact_name,
&artifact_bytes,
parsed.version,
parsed.channel,
timestamp,
);
let json =
serde_json::to_string_pretty(&signed).map_err(|e| format!("serialise manifest: {e}"))?;
std::fs::write(&parsed.out, json)
.map_err(|e| format!("cannot write {}: {e}", parsed.out.display()))?;
println!("Signed manifest written to {}", parsed.out.display());
Ok(())
}
/// Build and sign a [`SignedManifest`] for the given artifact bytes.
///
/// Pure aside from the signature computation — split out from `run_sign` so
/// tests can exercise it without touching the filesystem.
fn sign_manifest(
signing_key: &SigningKey,
artifact: String,
artifact_bytes: &[u8],
version: String,
channel: String,
timestamp: i64,
) -> SignedManifest {
let mut hasher = Sha256::new();
hasher.update(artifact_bytes);
let sha256 = hex_encode(&hasher.finalize());
let manifest = ReleaseManifest {
artifact,
sha256,
version,
channel,
timestamp,
};
let signature = hex_encode(&signing_key.sign(&manifest.canonical_bytes()).to_bytes());
SignedManifest {
manifest,
signature,
}
}
// ── helpers ──────────────────────────────────────────────────────────────────
fn hex_encode(bytes: &[u8]) -> String {
bytes.iter().map(|b| format!("{b:02x}")).collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn keygen_then_sign_produces_verifiable_signature() {
let tmp = tempfile::tempdir().unwrap();
let key_path = tmp.path().join("channel.key");
let signing_key = generate_signing_key();
write_seed_file(&key_path, &signing_key).unwrap();
let loaded = load_seed_file(&key_path).unwrap();
assert_eq!(loaded.verifying_key(), signing_key.verifying_key());
let signed = sign_manifest(
&loaded,
"huskies-linux-arm64".to_string(),
b"fake binary contents",
"abc1234".to_string(),
"stable".to_string(),
1_700_000_000,
);
// Verify with ed25519-dalek directly, mirroring how the gateway verifies.
use ed25519_dalek::Verifier;
let sig_bytes: [u8; 64] = hex_bytes(&signed.signature).try_into().unwrap();
let sig = ed25519_dalek::Signature::from_bytes(&sig_bytes);
assert!(
signing_key
.verifying_key()
.verify(&signed.manifest.canonical_bytes(), &sig)
.is_ok(),
"signature produced by sign_manifest must verify against the signing key's pubkey"
);
}
#[test]
fn sign_manifest_hashes_artifact_bytes() {
let signing_key = generate_signing_key();
let signed = sign_manifest(
&signing_key,
"art".to_string(),
b"hello world",
"v1".to_string(),
"stable".to_string(),
1,
);
let mut hasher = Sha256::new();
hasher.update(b"hello world");
let expected = hex_encode(&hasher.finalize());
assert_eq!(signed.manifest.sha256, expected);
}
#[test]
fn parse_sign_args_rejects_missing_required_flag() {
let args: Vec<String> = vec!["--key".into(), "k".into()];
assert!(parse_sign_args(&args).is_err());
}
#[test]
fn parse_sign_args_accepts_all_flags() {
let args: Vec<String> = vec![
"--key".into(),
"k".into(),
"--artifact".into(),
"a".into(),
"--version".into(),
"v1".into(),
"--channel".into(),
"stable".into(),
"--out".into(),
"o".into(),
"--timestamp".into(),
"42".into(),
];
let parsed = parse_sign_args(&args).unwrap();
assert_eq!(parsed.timestamp, Some(42));
assert_eq!(parsed.channel, "stable");
}
fn hex_bytes(s: &str) -> Vec<u8> {
(0..s.len())
.step_by(2)
.map(|i| u8::from_str_radix(&s[i..i + 2], 16).unwrap())
.collect()
}
}
+4 -6
View File
@@ -111,10 +111,9 @@ fn parse_pub_item(line: &str) -> Option<(String, String)> {
let rest = if let Some(r) = trimmed.strip_prefix("pub(") {
let end = r.find(')')?;
r[end + 1..].trim_start()
} else if let Some(r) = trimmed.strip_prefix("pub ") {
r.trim_start()
} else {
return None;
let r = trimmed.strip_prefix("pub ")?;
r.trim_start()
};
// Handle "async fn"
@@ -139,10 +138,9 @@ fn parse_pub_item(line: &str) -> Option<(String, String)> {
("const", r.trim_start())
} else if let Some(r) = rest.strip_prefix("static ") {
("static", r.trim_start())
} else if let Some(r) = rest.strip_prefix("mod ") {
("mod", r.trim_start())
} else {
return None;
let r = rest.strip_prefix("mod ")?;
("mod", r.trim_start())
};
let name: String = name_part
+4 -6
View File
@@ -123,10 +123,9 @@ fn parse_exported_item(line: &str) -> Option<(String, String)> {
// Strip "export default" or "export"
let rest = if let Some(r) = trimmed.strip_prefix("export default ") {
r.trim_start()
} else if let Some(r) = trimmed.strip_prefix("export ") {
r.trim_start()
} else {
return None;
let r = trimmed.strip_prefix("export ")?;
r.trim_start()
};
// Strip optional "async"
@@ -148,10 +147,9 @@ fn parse_exported_item(line: &str) -> Option<(String, String)> {
("const", r.trim_start())
} else if let Some(r) = rest.strip_prefix("let ") {
("let", r.trim_start())
} else if let Some(r) = rest.strip_prefix("enum ") {
("enum", r.trim_start())
} else {
return None;
let r = rest.strip_prefix("enum ")?;
("enum", r.trim_start())
};
let name: String = name_part
+1 -1
View File
@@ -7,7 +7,7 @@
#
# Tested with: OrbStack (recommended on macOS), Docker Desktop (slower bind mounts)
FROM rust:1.93-bookworm AS base
FROM rust:1.94-bookworm AS base
# Clippy and rustfmt are needed at runtime for acceptance gates
RUN rustup component add clippy rustfmt
+15 -3
View File
@@ -27,13 +27,23 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
procps \
openssh-server \
sudo \
nodejs \
npm \
&& rm -rf /var/lib/apt/lists/*
# Node.js 22.x from NodeSource — bookworm's apt nodejs is 18.x, which modern
# frontend toolchains (vite ≥ 7) refuse to run under. Project builds that
# shell out to npm (e.g. build.rs frontend steps) need ≥ 20.
RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
&& apt-get install -y --no-install-recommends nodejs \
&& npm install -g @anthropic-ai/claude-code \
&& rm -rf /var/lib/apt/lists/*
# Copy the huskies binary and entrypoint from the main image.
COPY --from=huskies-src /usr/local/bin/huskies /usr/local/bin/huskies
# The binary lives in /opt/huskies/bin (chowned to the huskies user below) so
# the in-process upgrade path (`POST /api/upgrade`) can atomically replace it
# without root: rename() needs write permission on the *directory*, not just
# the file. /usr/local/bin/huskies stays a symlink so PATH resolution and the
# entrypoint CMD are unchanged.
COPY --from=huskies-src /usr/local/bin/huskies /opt/huskies/bin/huskies
COPY --from=huskies-src /usr/local/bin/entrypoint.sh /usr/local/bin/entrypoint.sh
# Non-root user — Claude Code refuses --dangerously-skip-permissions as root.
@@ -44,6 +54,8 @@ RUN groupadd -r huskies \
&& mkdir -p /home/huskies/.ssh \
&& chmod 700 /home/huskies/.ssh \
&& chown -R huskies:huskies /home/huskies \
&& chown -R huskies:huskies /opt/huskies \
&& ln -s /opt/huskies/bin/huskies /usr/local/bin/huskies \
&& mkdir -p /workspace \
&& chown huskies:huskies /workspace \
&& git config --global init.defaultBranch master \
+6 -5
View File
@@ -13,19 +13,20 @@
USER root
# OpenJDK 21 (current LTS) and Maven for build support.
# OpenJDK 17 (Bookworm's default LTS) and Maven for build support.
RUN apt-get update && apt-get install -y --no-install-recommends \
openjdk-21-jdk-headless \
openjdk-17-jdk-headless \
maven \
&& rm -rf /var/lib/apt/lists/*
ENV JAVA_HOME="/usr/lib/jvm/java-21-openjdk-amd64"
RUN ln -s /usr/lib/jvm/java-17-openjdk-* /usr/lib/jvm/java-17-openjdk
ENV JAVA_HOME="/usr/lib/jvm/java-17-openjdk"
# Eclipse JDT Language Server — canonical LSP for Java/JVM (Java, Kotlin, Groovy).
# Pin to a specific release; update JDTLS_VERSION + JDTLS_BUILD for upgrades.
# All releases: https://github.com/eclipse-jdtls/eclipse.jdt.ls/releases
ENV JDTLS_VERSION="1.38.0" \
JDTLS_BUILD="202503271418"
ENV JDTLS_VERSION="1.60.0" \
JDTLS_BUILD="202606262232"
RUN mkdir -p /opt/jdtls \
&& curl -fsSL \
"https://download.eclipse.org/jdtls/milestones/${JDTLS_VERSION}/jdt-language-server-${JDTLS_VERSION}-${JDTLS_BUILD}.tar.gz" \
+4
View File
@@ -14,10 +14,14 @@
USER root
# Build tools required by rustup and many Rust crates.
# libudev-dev: serial/USB device crates (libudev-sys, serialport).
# libclang-dev: bindgen-based crates; cargo test is skipped when absent.
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
pkg-config \
libssl-dev \
libudev-dev \
libclang-dev \
&& rm -rf /var/lib/apt/lists/*
ENV RUSTUP_HOME="/home/huskies/.rustup" \
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "huskies",
"version": "0.13.0",
"version": "0.14.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "huskies",
"version": "0.13.0",
"version": "0.14.0",
"dependencies": {
"@types/react-syntax-highlighter": "^15.5.13",
"react": "^19.1.0",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "huskies",
"private": true,
"version": "0.13.0",
"version": "0.14.0",
"type": "module",
"scripts": {
"dev": "vite",
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env bash
# script/ci-publish-artifact — upload a built sled artifact to a release
# channel and verify the channel's manifest reflects the published commit.
#
# Usage: script/ci-publish-artifact <artifact-path> [git-hash]
#
# git-hash defaults to `git rev-parse --short HEAD` when omitted.
#
# Required env:
# HUSKIES_CHANNEL_URL Base URL of the release channel (Gitea secret).
# HUSKIES_CHANNEL_TOKEN Bearer token authorised to publish (Gitea secret).
#
# Protocol against the channel host:
# POST {HUSKIES_CHANNEL_URL}/<artifact-name>
# Headers: Authorization: Bearer <token>, X-Git-Hash: <hash>
# Body: raw artifact bytes
# GET {HUSKIES_CHANNEL_URL}/manifest.json
# Headers: Authorization: Bearer <token>
# Body: JSON object with a "git_hash" field
#
# 5xx responses and network errors are retried with backoff; 4xx responses
# (including auth failures) fail immediately since retrying won't fix them.
set -euo pipefail
ARTIFACT_PATH="${1:?Usage: script/ci-publish-artifact <artifact-path> [git-hash]}"
GIT_HASH="${2:-$(git rev-parse --short HEAD)}"
if [ -z "${HUSKIES_CHANNEL_URL:-}" ]; then
echo "Error: HUSKIES_CHANNEL_URL is not set." >&2
exit 1
fi
if [ -z "${HUSKIES_CHANNEL_TOKEN:-}" ]; then
echo "Error: HUSKIES_CHANNEL_TOKEN is not set." >&2
exit 1
fi
if [ ! -f "$ARTIFACT_PATH" ]; then
echo "Error: artifact not found at $ARTIFACT_PATH" >&2
exit 1
fi
ARTIFACT_NAME="$(basename "$ARTIFACT_PATH")"
CHANNEL_URL="${HUSKIES_CHANNEL_URL%/}"
UPLOAD_URL="${CHANNEL_URL}/${ARTIFACT_NAME}"
MANIFEST_URL="${CHANNEL_URL}/manifest.json"
UPLOAD_MAX_ATTEMPTS="${HUSKIES_CI_PUBLISH_MAX_ATTEMPTS:-3}"
UPLOAD_BACKOFF_SECS="${HUSKIES_CI_PUBLISH_BACKOFF_SECS:-1}"
MANIFEST_MAX_ATTEMPTS=3
MANIFEST_BACKOFF_SECS=1
RESPONSE_FILE="$(mktemp)"
trap 'rm -f "$RESPONSE_FILE"' EXIT
# ── Upload ────────────────────────────────────────────────────────────────
attempt=1
while :; do
echo "==> Uploading ${ARTIFACT_NAME} (${GIT_HASH}), attempt ${attempt}/${UPLOAD_MAX_ATTEMPTS}..."
HTTP_CODE=$(curl -sS --connect-timeout 10 --max-time 60 \
-o "$RESPONSE_FILE" -w "%{http_code}" \
-X POST \
-H "Authorization: Bearer ${HUSKIES_CHANNEL_TOKEN}" \
-H "X-Git-Hash: ${GIT_HASH}" \
--data-binary "@${ARTIFACT_PATH}" \
"${UPLOAD_URL}") || HTTP_CODE="000"
RESPONSE_BODY="$(cat "$RESPONSE_FILE" 2>/dev/null || true)"
case "$HTTP_CODE" in
2??)
echo "==> Upload succeeded (HTTP ${HTTP_CODE})."
break
;;
401|403)
echo "Error: upload rejected — authentication failed (HTTP ${HTTP_CODE})." >&2
echo "Response: ${RESPONSE_BODY}" >&2
exit 1
;;
4??)
echo "Error: upload rejected by the channel (HTTP ${HTTP_CODE}); not retrying a client error." >&2
echo "Response: ${RESPONSE_BODY}" >&2
exit 1
;;
esac
if [ "$attempt" -ge "$UPLOAD_MAX_ATTEMPTS" ]; then
echo "Error: upload failed after ${UPLOAD_MAX_ATTEMPTS} attempts (last HTTP ${HTTP_CODE})." >&2
echo "Response: ${RESPONSE_BODY}" >&2
exit 1
fi
echo "==> Transient failure (HTTP ${HTTP_CODE}); retrying in ${UPLOAD_BACKOFF_SECS}s..."
sleep "$UPLOAD_BACKOFF_SECS"
attempt=$((attempt + 1))
UPLOAD_BACKOFF_SECS=$((UPLOAD_BACKOFF_SECS * 2))
done
# ── Verify manifest ──────────────────────────────────────────────────────
attempt=1
while :; do
echo "==> Verifying channel manifest reflects ${GIT_HASH} (attempt ${attempt}/${MANIFEST_MAX_ATTEMPTS})..."
MANIFEST_JSON=$(curl -sS --connect-timeout 10 --max-time 30 \
-H "Authorization: Bearer ${HUSKIES_CHANNEL_TOKEN}" \
"${MANIFEST_URL}") || MANIFEST_JSON=""
MANIFEST_HASH=$(printf '%s' "$MANIFEST_JSON" \
| python3 -c "import sys,json; print(json.load(sys.stdin).get('git_hash',''))" 2>/dev/null || echo "")
if [ "$MANIFEST_HASH" = "$GIT_HASH" ]; then
echo "==> Published ${ARTIFACT_NAME} (${GIT_HASH}) to ${CHANNEL_URL}; manifest verified."
exit 0
fi
if [ "$attempt" -ge "$MANIFEST_MAX_ATTEMPTS" ]; then
echo "Error: manifest mismatch — channel reports git_hash '${MANIFEST_HASH}', expected '${GIT_HASH}'." >&2
echo "Manifest: ${MANIFEST_JSON}" >&2
exit 1
fi
sleep "$MANIFEST_BACKOFF_SECS"
attempt=$((attempt + 1))
MANIFEST_BACKOFF_SECS=$((MANIFEST_BACKOFF_SECS * 2))
done
+7 -1
View File
@@ -281,7 +281,13 @@ echo "$RELEASE_BODY"
# ── Tag & Push ─────────────────────────────────────────────────
echo "==> Tagging ${TAG}..."
git tag -a "$TAG" -m "Release ${TAG}"
git push origin "$TAG"
# Push the branch (with the version-bump commit) and the tag together, so
# the remote branch never lags the release tag. --atomic means both refs
# land or neither does, avoiding a pushed tag pointing at an unpushed commit.
BRANCH="$(git rev-parse --abbrev-ref HEAD)"
echo "==> Pushing ${BRANCH} and ${TAG}..."
git push --atomic origin "$BRANCH" "$TAG"
# ── Create Gitea Release ──────────────────────────────────────
echo "==> Creating release on Gitea..."
+7 -4
View File
@@ -1,6 +1,6 @@
[package]
name = "huskies"
version = "0.13.0"
version = "0.14.0"
edition = "2024"
build = "build.rs"
@@ -40,13 +40,15 @@ tokio-tungstenite = { workspace = true }
# against a missing system libsqlite3.
#
# The 0.35 pin is the ceiling: rusqlite 0.37 (matrix-sdk-sqlite) requires
# 0.35.x exactly, and sqlx-sqlite 0.9.0-alpha.1 requires >=0.30, <0.36. Bumping
# this needs one of those upstreams to widen their range first.
# 0.35.x exactly, and sqlx-sqlite 0.9.0 requires >=0.30.1, <0.38. Bumping this
# needs matrix-sdk to move to a newer rusqlite first; sqlx then allows up to
# 0.37.x.
libsqlite3-sys = { version = "0.35.0", features = ["bundled"] }
sqlx = { workspace = true }
wait-timeout = "0.2.1"
bft-json-crdt = { path = "../crates/bft-json-crdt", default-features = false, features = ["bft"] }
source-map-gen = { path = "../crates/source-map-gen" }
release-manifest = { path = "../crates/release-manifest" }
ed25519-dalek = { workspace = true }
rand = { workspace = true }
nutype = { workspace = true }
@@ -62,5 +64,6 @@ check-cfg = ["cfg(feature, values(\"logging-base\"))"]
[dev-dependencies]
tempfile = { workspace = true }
mockito = "1"
mockito = "1.7.2"
filetime = { workspace = true }
serde_yaml = { workspace = true }
@@ -0,0 +1,9 @@
-- Stores a serialized CRDT state snapshot so startup can skip replaying
-- the full op log. Only the single most recent snapshot row is kept.
CREATE TABLE IF NOT EXISTS crdt_snapshot (
id INTEGER PRIMARY KEY CHECK (id = 1),
at_seq INTEGER NOT NULL,
max_rowid INTEGER NOT NULL,
state_json TEXT NOT NULL,
created_at TEXT NOT NULL
);
+7 -2
View File
@@ -61,6 +61,11 @@ pub(super) fn build_agent_app_context(
);
let (reconciliation_tx, _) = broadcast::channel(64);
let (perm_tx, perm_rx) = tokio::sync::mpsc::unbounded_channel();
let permission_registry = crate::service::permission_router::ResponderRegistry::new();
crate::service::permission_router::spawn_permission_router(
perm_rx,
Arc::clone(&permission_registry),
);
let timer_store = Arc::new(crate::service::timer::TimerStore::load(
project_root.join(".huskies").join("timers.json"),
));
@@ -74,8 +79,8 @@ pub(super) fn build_agent_app_context(
bot_name: "Agent".to_string(),
bot_user_id: String::new(),
ambient_rooms: Arc::new(std::sync::Mutex::new(std::collections::HashSet::new())),
perm_rx: Arc::new(tokio::sync::Mutex::new(perm_rx)),
pending_perm_replies: Arc::new(tokio::sync::Mutex::new(std::collections::HashMap::new())),
permission_registry,
pending_perm_replies: crate::service::permission_router::PendingPermReplies::new(),
permission_timeout_secs: 120,
status: agents.status_broadcaster(),
chat_dispatcher: Arc::new(crate::chat::dispatcher::ChatDispatcher::new(1_500)),
+2 -2
View File
@@ -159,7 +159,7 @@ pub(super) async fn detect_conflicts(
our_claims.remove(&story_id);
// Stop any local agent for this story by looking up its name.
if let Ok(agent_list) = agents.list_agents() {
if let Ok(agent_list) = agents.list_agents().await {
for info in agent_list {
if info.story_id == story_id {
let _ = agents
@@ -219,7 +219,7 @@ pub(super) fn reclaim_timed_out_work(_project_root: &Path) {
/// Check for completed agents, push their feature branches to the remote,
/// and report completion via CRDT.
pub(super) async fn check_completions_and_push(agents: &AgentPool, _project_root: &Path) {
let Ok(agent_list) = agents.list_agents() else {
let Ok(agent_list) = agents.list_agents().await else {
return;
};
+19
View File
@@ -213,6 +213,15 @@ pub async fn run(
// Track which stories we've claimed so we can detect conflicts.
let mut our_claims: HashMap<String, f64> = HashMap::new();
// Low-disk-space watchdog (story 1200 AC1): tracks rate-limit/recovery
// state across loop iterations. Thresholds come from the config loaded
// at startup; host_id identifies this sled in chat messages and the
// gateway dedupe key.
let mut disk_watch_state = crate::service::disk_watch::DiskWatchState::default();
let disk_watch_host_id =
crdt_state::our_node_id().unwrap_or_else(|| "unknown-host".to_string());
let disk_watch_status = agents.status_broadcaster();
// Main loop: heartbeat, scan, claim, detect conflicts.
let mut interval = tokio::time::interval(std::time::Duration::from_secs(SCAN_INTERVAL_SECS));
loop {
@@ -221,6 +230,16 @@ pub async fn run(
// Write heartbeat.
write_heartbeat(&rendezvous_url, port);
// Low-disk-space check (story 1200 AC1): every tick period.
crate::service::disk_watch::io::check_and_notify(
&project_root,
&config.disk_watch,
&mut disk_watch_state,
&watcher_tx,
&disk_watch_status,
&disk_watch_host_id,
);
// Scan CRDT for claimable work.
scan_and_claim(&agents, &project_root, &mut our_claims).await;
+29 -1
View File
@@ -121,7 +121,7 @@ pub fn move_story_to_done(story_id: &str) -> Result<(), String> {
Stage::Merge { .. } => PipelineEvent::MergeSucceeded {
merge_commit: GitSha("accepted".to_string()),
},
Stage::MergeFailure { .. } => PipelineEvent::Accepted,
Stage::MergeFailure { .. } | Stage::MergeFailureFinal { .. } => PipelineEvent::Accepted,
Stage::Coding { .. } | Stage::Qa | Stage::Backlog => PipelineEvent::Close,
_ => {
return Err(format!(
@@ -627,6 +627,34 @@ mod tests {
);
}
/// Regression test (story 1178): a story in `Stage::MergeFailureFinal`
/// whose merge is later retried and succeeds must be movable to Done.
/// Before this fix, `move_story_to_done` had no arm for
/// `MergeFailureFinal`, so it always returned an error even after a
/// real, successful re-merge — the exact "trap state" this story fixes.
#[test]
fn move_story_to_done_from_merge_failure_final_succeeds() {
crate::db::ensure_content_store();
crate::db::write_item_with_content(
"99952_story_merge_failure_final",
"merge_failure_final",
"---\nname: Merge Failure Final Test\n---\n# Story\n",
crate::db::ItemMeta::named("Merge Failure Final Test"),
);
move_story_to_done("99952_story_merge_failure_final")
.expect("move_story_to_done should succeed from MergeFailureFinal");
let item = crate::pipeline_state::read_typed("99952_story_merge_failure_final")
.expect("CRDT read should succeed")
.expect("item should exist in CRDT");
assert_eq!(
item.stage.dir_name(),
"done",
"item should be in done after move from MergeFailureFinal"
);
}
// ── item_type_from_id tests ────────────────────────────────────────────────
#[test]
+9 -1
View File
@@ -4,7 +4,7 @@ use serde::{Deserialize, Serialize};
mod squash;
pub(crate) use squash::run_squash_merge;
pub(crate) use squash::{merge_lock_is_free, run_squash_merge};
/// Typed outcome of a completed squash-merge operation.
///
@@ -103,6 +103,9 @@ pub struct MergeJob {
/// than the current server's boot time. This survives `rebuild_and_restart`
/// (which re-execs and keeps the same PID).
pub server_start_time: f64,
/// Unix timestamp (seconds) when this merge job started, used to compute
/// elapsed time for a still-`Running` job.
pub started_at: f64,
}
/// Result of a mergemaster merge operation.
@@ -113,6 +116,11 @@ pub struct MergeReport {
pub result: MergeResult,
pub worktree_cleaned_up: bool,
pub story_archived: bool,
/// Path (relative to the project root) of the full untruncated report
/// written by `service::merge::io::write_merge_report`, if the write
/// succeeded.
#[serde(default)]
pub report_path: Option<String>,
}
#[cfg(test)]
+22 -9
View File
@@ -17,6 +17,27 @@ use crate::config::ProjectConfig;
/// causing `git cherry-pick merge-queue/…` to fail with "bad revision".
static MERGE_LOCK: Mutex<()> = Mutex::new(());
/// Returns `true` when no squash-merge is currently running, i.e. the merge
/// lock is free.
///
/// Used by the build-directory GC pass (story 1199) to decide whether
/// `.huskies/merge_workspace` is safe to reclaim. This is a best-effort,
/// momentary check — the lock is not held across the reclaim itself, so a
/// merge that starts immediately afterward can still race with GC. That's
/// acceptable: the GC pass tolerates races and skips on error rather than
/// failing the whole pass.
pub(crate) fn merge_lock_is_free() -> bool {
MERGE_LOCK.try_lock().is_ok()
}
/// Resolve the base branch for `project_root` from config, or auto-detect it.
fn resolve_base_branch(project_root: &Path) -> String {
let configured = crate::config::ProjectConfig::load(project_root)
.ok()
.and_then(|c| c.base_branch);
crate::worktree::resolve_base_branch(project_root, configured.as_deref())
}
pub(crate) fn run_squash_merge(
project_root: &Path,
branch: &str,
@@ -31,10 +52,7 @@ pub(crate) fn run_squash_merge(
// A zero-commit branch produces an empty squash and a silent "nothing to
// commit" failure. Catch it early with a grep-able error before any merge
// work starts.
let base_branch = crate::config::ProjectConfig::load(project_root)
.ok()
.and_then(|c| c.base_branch.clone())
.unwrap_or_else(|| "master".to_string());
let base_branch = resolve_base_branch(project_root);
let ahead_out = Command::new("git")
.args(["rev-list", "--count", &format!("{base_branch}..{branch}")])
@@ -316,11 +334,6 @@ pub(crate) fn run_squash_merge(
.map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string())
.unwrap_or_default();
let base_branch = crate::config::ProjectConfig::load(project_root)
.ok()
.and_then(|c| c.base_branch.clone())
.unwrap_or_else(|| "master".to_string());
if current_branch != base_branch {
all_output.push_str(&format!(
"=== VERIFICATION FAILED: expected branch '{base_branch}' but HEAD is on \
@@ -178,6 +178,79 @@ async fn squash_merge_clean_merge_succeeds() {
);
}
#[tokio::test]
async fn squash_merge_succeeds_on_main_based_repo_with_base_branch_unset() {
use std::fs;
use tempfile::tempdir;
let tmp = tempdir().unwrap();
let repo = tmp.path();
// Repo whose default branch is `main` — no `master` branch exists at all,
// and no `.huskies/project.toml` sets `base_branch`. run_squash_merge must
// auto-detect `main` instead of assuming `master` (bug 1176).
Command::new("git")
.args(["init", "-b", "main"])
.current_dir(repo)
.output()
.unwrap();
Command::new("git")
.args(["config", "user.email", "test@test.com"])
.current_dir(repo)
.output()
.unwrap();
Command::new("git")
.args(["config", "user.name", "Test"])
.current_dir(repo)
.output()
.unwrap();
Command::new("git")
.args(["commit", "--allow-empty", "-m", "init"])
.current_dir(repo)
.output()
.unwrap();
Command::new("git")
.args(["checkout", "-b", "feature/story-main_test"])
.current_dir(repo)
.output()
.unwrap();
fs::write(repo.join("new_file.txt"), "new content").unwrap();
Command::new("git")
.args(["add", "."])
.current_dir(repo)
.output()
.unwrap();
Command::new("git")
.args(["commit", "-m", "add new file"])
.current_dir(repo)
.output()
.unwrap();
Command::new("git")
.args(["checkout", "main"])
.current_dir(repo)
.output()
.unwrap();
let result = run_squash_merge(repo, "feature/story-main_test", "main_test").unwrap();
assert!(
matches!(
result,
super::MergeResult::Success {
conflicts_resolved: false,
..
}
),
"clean merge should succeed on a main-based repo; got: {result:?}"
);
assert!(
repo.join("new_file.txt").exists(),
"merged file should exist on main"
);
}
#[tokio::test]
async fn squash_merge_nonexistent_branch_fails() {
use tempfile::tempdir;
@@ -73,7 +73,7 @@ mod tests {
// task eventually fails.
pool.auto_assign_available_work(tmp.path()).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let has_pending = agents.values().any(|a| {
a.agent_name == "coder-1"
&& matches!(a.status, AgentStatus::Pending | AgentStatus::Running)
@@ -115,7 +115,7 @@ mod tests {
pool.auto_assign_available_work(root).await;
// No agent should have been started for the spike.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
assert!(
agents.is_empty(),
"No agents should be assigned to a spike with review_hold"
@@ -155,7 +155,7 @@ mod tests {
pool.auto_assign_available_work(tmp.path()).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
// coder-1 must NOT have been assigned to the QA story (wrong stage).
let coder_assigned_to_qa = agents.iter().any(|(key, a)| {
key.contains("9930_story_qa1")
@@ -209,7 +209,7 @@ mod tests {
pool.auto_assign_available_work(tmp.path()).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
// coder-1 should have been picked (it matches the stage and is preferred).
let coder1_assigned = agents.values().any(|a| {
a.agent_name == "coder-1"
@@ -262,7 +262,7 @@ mod tests {
// Must not panic.
pool.auto_assign_available_work(tmp.path()).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
// No agent should be assigned to the specific QA story (coder-1 may
// be assigned to leaked 2_current items from the global CRDT store).
let assigned_to_qa_story = agents.iter().any(|(key, a)| {
@@ -301,7 +301,7 @@ mod tests {
let pool = AgentPool::new_test(3001);
pool.auto_assign_available_work(root).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
// Filter to only agents assigned to our specific story to avoid
// interference from other tests sharing the global CRDT store.
let assigned_to_our_story = agents.iter().any(|(key, a)| {
@@ -347,7 +347,7 @@ mod tests {
let pool = AgentPool::new_test(3001);
pool.auto_assign_available_work(root).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let has_pending = agents.values().any(|a| {
matches!(
a.status,
@@ -553,7 +553,7 @@ mod tests {
let _ = tokio::join!(t1, t2);
// At most one Pending/Running entry should exist for coder-1.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let active_coder_count = agents
.values()
.filter(|a| {
@@ -602,7 +602,7 @@ mod tests {
pool.auto_assign_available_work(tmp.path()).await;
let count_after_first = {
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
agents
.iter()
.filter(|(key, a)| {
@@ -616,7 +616,7 @@ mod tests {
pool.auto_assign_available_work(tmp.path()).await;
let count_after_second = {
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
agents
.iter()
.filter(|(key, a)| {
+123 -27
View File
@@ -65,27 +65,43 @@ impl AgentPool {
// AC6: Detect empty-diff stories before starting the merge pipeline.
// If the worktree has no commits on the feature branch, block the
// story immediately via the state machine — no merge job needed.
if let Some(wt_path) = worktree::find_worktree_path(project_root, story_id)
&& !crate::agents::gates::worktree_has_committed_work(&wt_path)
{
let empty_diff_reason = "Feature branch has no code changes — the coder agent \
did not produce any commits.";
slog_warn!(
"[auto-assign] Story '{story_id}' in 4_merge/ has no commits \
on feature branch. Blocking via state machine."
);
if let Err(e) =
crate::agents::lifecycle::transition_to_blocked(story_id, empty_diff_reason)
{
slog_error!("[auto-assign] Failed to transition '{story_id}' to Blocked: {e}");
//
// Bug 1170: worktree_has_committed_work shells out to `git log`
// synchronously. assign_merge_stage runs on the shared tokio
// runtime (it's invoked reactively on every CRDT transition, incl.
// unblock), so calling it inline here blocked a runtime worker
// thread for the duration of the git subprocess — with a story
// whose worktree/agent had crashed, that call could hang
// indefinitely and stall /health and the liveness heartbeat along
// with it. Run it on the blocking-thread pool instead.
if let Some(wt_path) = worktree::find_worktree_path(project_root, story_id) {
let has_commits = tokio::task::spawn_blocking(move || {
crate::agents::gates::worktree_has_committed_work(&wt_path)
})
.await
.unwrap_or(false);
if !has_commits {
let empty_diff_reason = "Feature branch has no code changes — the coder agent \
did not produce any commits.";
slog_warn!(
"[auto-assign] Story '{story_id}' in 4_merge/ has no commits \
on feature branch. Blocking via state machine."
);
if let Err(e) =
crate::agents::lifecycle::transition_to_blocked(story_id, empty_diff_reason)
{
slog_error!(
"[auto-assign] Failed to transition '{story_id}' to Blocked: {e}"
);
}
let _ = self
.watcher_tx
.send(crate::io::watcher::WatcherEvent::StoryBlocked {
story_id: story_id.to_string(),
reason: empty_diff_reason.to_string(),
});
continue;
}
let _ = self
.watcher_tx
.send(crate::io::watcher::WatcherEvent::StoryBlocked {
story_id: story_id.to_string(),
reason: empty_diff_reason.to_string(),
});
continue;
}
// Skip if a merge job is already running for this story (e.g. triggered
@@ -99,13 +115,7 @@ impl AgentPool {
// Skip if an explicit mergemaster LLM agent is already running
// (operator-driven failure recovery path).
let has_mergemaster = {
let agents = match self.agents.lock() {
Ok(a) => a,
Err(e) => {
slog_error!("[auto-assign] Failed to lock agents: {e}");
break;
}
};
let agents = self.agents.lock().await;
is_story_assigned_for_stage(config, &agents, story_id, &PipelineStage::Mergemaster)
};
if has_mergemaster {
@@ -117,3 +127,89 @@ impl AgentPool {
}
}
}
#[cfg(test)]
mod tests {
use super::super::super::AgentPool;
use crate::config::ProjectConfig;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
/// Bug 1170 regression: `assign_merge_stage` used to call
/// `worktree_has_committed_work` (which shells out to `git`) directly on
/// the async runtime. For a story with a crashed/unassignable agent
/// sitting in `4_merge/`, that synchronous subprocess call had no yield
/// point, so on a runtime with few worker threads it starved every other
/// task — including the liveness heartbeat and `/health` — for the whole
/// scan. After wrapping the call in `spawn_blocking`, the executor stays
/// free to interleave other work while the git subprocess runs
/// off-runtime.
///
/// This reproduces the unblock → merge-auto-assign path: a story sits in
/// `4_merge/` with no active agent entry (the crashed/unassignable case)
/// and a worktree directory that isn't a real git repo, forcing every
/// `git` invocation in the scan to fail — but only after paying the
/// process fork/exec cost, which is what stalls a non-yielding runtime.
#[tokio::test(flavor = "multi_thread", worker_threads = 1)]
async fn assign_merge_stage_does_not_stall_liveness_heartbeat() {
crate::db::ensure_content_store();
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path().to_path_buf();
std::fs::create_dir_all(root.join(".huskies")).unwrap();
std::fs::write(root.join(".huskies/project.toml"), "").unwrap();
let worktrees_dir = root.join(".huskies/worktrees");
std::fs::create_dir_all(&worktrees_dir).unwrap();
// Simulate several stories stuck in 4_merge/ with a crashed/unassignable
// agent: each has a worktree directory (so find_worktree_path succeeds
// and the git-shelling check runs) but is not a real git repo and has
// no active agent entry in the pool.
const STORY_COUNT: usize = 25;
for i in 0..STORY_COUNT {
let story_id = format!("11700_merge_{i:03}");
std::fs::create_dir_all(worktrees_dir.join(&story_id)).unwrap();
crate::db::write_item_with_content(
&story_id,
"4_merge",
"---\nname: Crashed Merge\n---\n",
crate::db::ItemMeta::named("Crashed Merge"),
);
}
let pool = AgentPool::new_test(3200);
let config = ProjectConfig::load(&root).unwrap_or_default();
// Stand in for the liveness heartbeat (tick_loop.rs's
// spawn_liveness_tick) and /health polling: a tight-interval task
// racing the merge scan on the single-worker-thread runtime.
let ticks = Arc::new(AtomicU64::new(0));
let ticks_clone = Arc::clone(&ticks);
let heartbeat = tokio::spawn(async move {
loop {
tokio::time::sleep(std::time::Duration::from_millis(2)).await;
ticks_clone.fetch_add(1, Ordering::SeqCst);
}
});
let start = std::time::Instant::now();
pool.assign_merge_stage(&root, &config).await;
let elapsed = start.elapsed();
heartbeat.abort();
let observed_ticks = ticks.load(Ordering::SeqCst);
// With a 2ms heartbeat cadence, an unstalled runtime should have
// fired roughly elapsed/2ms ticks. Require at least a quarter of that
// as a generous floor — a stalled runtime (pre-fix) produces ~0 ticks
// because the single worker thread never yields during the scan.
let expected_min_ticks = (elapsed.as_millis() / 2 / 4) as u64;
assert!(
observed_ticks >= expected_min_ticks,
"liveness heartbeat stalled during assign_merge_stage: {observed_ticks} tick(s) \
over {elapsed:?} (expected at least ~{expected_min_ticks}); the merge scan likely \
blocked the tokio runtime instead of yielding via spawn_blocking"
);
}
}
@@ -1,18 +1,43 @@
//! TransitionFired subscriber that auto-blocks stories after N consecutive MergeFailure transitions.
//! TransitionFired subscriber that owns the consecutive-MergeFailure budget:
//! auto-blocks stories at the threshold and auto-retries `GatesFailed`
//! failures below it.
//!
//! Listens on the pipeline transition broadcast channel and, for each story,
//! counts how many times it has entered [`Stage::MergeFailure`] consecutively.
//! When the count reaches the configurable threshold (default 3), the story is
//! transitioned to [`Stage::Blocked`] with a reason that names the failure kind.
//! One counter drives two policies sharing the `merge_failure_block_threshold`
//! budget (default 3):
//!
//! The counter for a story resets whenever a non-`MergeFailure` transition fires
//! for that story (e.g. after a successful merge or a `FixupRequested` demotion
//! back to coding).
//! - **Below the threshold**, a `GatesFailed` failure schedules a delayed
//! re-trigger of the deterministic server-side merge (story 1185) — gates
//! failures are dominated by transients (flaky tests, stale base) that a
//! plain re-run fixes. Other kinds still count toward the budget but are
//! not retried: `ConflictDetected` has its own mergemaster recovery path via
//! [`super::merge_failure_subscriber`]; `EmptyDiff`/`NoCommits`/`Other`
//! require human intervention.
//! - **At the threshold**, the story is transitioned to [`Stage::Blocked`]
//! with a reason naming the failure kind.
//!
//! The counter resets when the story leaves `MergeFailure` for a real reason
//! (successful merge, `FixupRequested`, `Block`), but **not** on
//! [`PipelineEvent::MergeRetryStarted`] — that is the `MergeFailure → Merge`
//! bounce a retry itself causes. Treating it as a reset made the budget
//! unreachable and let a deterministic gates failure retry forever (1185
//! review finding 1); counting across the bounce is what makes the budget
//! real.
//!
//! Bug 1025: while a mergemaster is actively running on the story, its
//! iteration loop (squash → fail → fix → retry) generates multiple
//! MergeFailure transitions. Those are NOT consecutive give-ups — they are
//! recovery iterations in progress. We neither count nor schedule retries
//! while a mergemaster is in the pool for the story.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::Mutex;
use std::time::Duration;
use crate::io::watcher::WatcherEvent;
use crate::pipeline_state::{MergeFailureKind, PipelineEvent, Stage, Status, StoryId};
use crate::slog;
use crate::slog_warn;
@@ -20,42 +45,102 @@ use crate::slog_warn;
use super::super::super::PipelineStage;
use super::super::AgentPool;
use super::scan::is_story_assigned_for_stage;
use super::story_checks::{
has_review_hold, has_unmet_dependencies, is_story_blocked, is_story_frozen,
};
/// Reconcile: no-op for the merge-failure block subscriber.
/// Delay before an auto-retry re-triggers the server-side merge for a
/// `GatesFailed` failure. Gives transient conditions (a concurrently landing
/// master merge, an exhausted runner) a moment to clear; retrying instantly
/// would just replay the same failure.
const AUTO_RETRY_DELAY: Duration = Duration::from_secs(30);
/// Per-story scheduling generation, shared between the subscriber loop and the
/// delayed retry tasks it spawns.
///
/// The block subscriber maintains an in-memory per-story consecutive-failure counter
/// that cannot be reconstructed from CRDT state alone (only the current stage is
/// stored, not the history of how many times each story failed). Eventual consistency
/// is guaranteed by the live subscriber reacting to each new `MergeFailure` event;
/// the periodic reconciler cannot add value here without risking spurious blocks.
/// Every scheduled retry captures the generation current at schedule time; the
/// timer only acts if that generation is still current when it fires. The
/// subscriber bumps the generation on every (re)schedule and clears the entry
/// on counter reset, so stale timers left over from an earlier failure cycle
/// become no-ops instead of firing unaccounted retries (1185 review finding 6).
type Generations = Arc<Mutex<HashMap<String, u64>>>;
/// What the subscriber decided to do about one transition. Split out from the
/// event loop so the counter/budget policy is synchronous and unit-testable.
#[derive(Debug, PartialEq, Eq)]
enum Decision {
/// Nothing to do (not a MergeFailure, recovery in progress, retry bounce,
/// budget disabled, or a non-retryable kind below the threshold).
Nothing,
/// Schedule a delayed auto-retry: this is consecutive failure `attempt` of
/// a `budget`-sized budget, and the kind is `GatesFailed`.
ScheduleRetry { attempt: u32, budget: u32 },
/// The budget is exhausted: block the story.
Block { count: u32 },
}
/// Reconcile: no-op for the periodic pass.
///
/// The consecutive-failure counter is in-memory and cannot be reconstructed
/// from CRDT state (only the current stage is stored, not the failure
/// history). Restart catch-up for stories already parked in
/// `MergeFailure{GatesFailed}` is handled once, at subscriber startup, by
/// [`reconcile_stranded_gates_failed`] — running it from the periodic
/// reconciler instead would re-schedule retries for budget-exhausted stories
/// on every tick, reintroducing the unbounded-retry bug the startup-only scan
/// avoids.
pub(crate) fn reconcile_merge_failure_block() {}
/// Spawn a background task that blocks stories after N consecutive `MergeFailure` transitions.
///
/// Subscribes to the pipeline transition broadcast channel and tracks a per-story
/// consecutive-failure counter. When a story's count reaches the threshold configured
/// in `project.toml` (`merge_failure_block_threshold`, default 3), the story is
/// transitioned to `Stage::Blocked` with a reason that names the failure kind.
///
/// The counter resets when the story leaves `MergeFailure` (e.g. on `FixupRequested`,
/// `ReQueuedForQa`, or a successful merge via `Unblock → Merge → Done`).
///
/// Bug 1025: while a mergemaster is actively running on the story, its
/// iteration loop (squash → fail → fix → retry) generates multiple
/// MergeFailure transitions. Those are NOT consecutive give-ups — they are
/// recovery iterations in progress. We skip counter increments while a
/// mergemaster is in the pool for the story; the counter only increments on
/// transitions that happen with no recovery agent attached.
/// Spawn the background task that owns the consecutive-MergeFailure budget:
/// auto-retry for `GatesFailed` below the threshold, auto-block at it.
pub(crate) fn spawn_merge_failure_block_subscriber(pool: Arc<AgentPool>, project_root: PathBuf) {
let mut rx = crate::pipeline_state::subscribe_transitions();
tokio::spawn(async move {
let mut counters: HashMap<StoryId, (u32, MergeFailureKind)> = HashMap::new();
let generations: Generations = Generations::default();
// One-shot restart catch-up: stories already sitting in GatesFailed
// when the process starts will never fire another transition on their
// own, so without this they'd silently lose auto-retry coverage
// (1185 review finding 3).
reconcile_stranded_gates_failed(&pool, &project_root, &mut counters, &generations).await;
loop {
match rx.recv().await {
Ok(fired) => {
let recovery_running =
is_mergemaster_running(&pool, &project_root, &fired.story_id.0);
on_transition(&project_root, &fired, &mut counters, recovery_running);
is_mergemaster_running(&pool, &project_root, &fired.story_id.0).await;
let threshold = load_threshold(&project_root).await;
match decide(&fired, &mut counters, recovery_running, threshold) {
Decision::Nothing => {
// A real departure from MergeFailure also
// invalidates any pending retry timer.
if fired.after.status() != Status::MergeFailure
&& !matches!(fired.event, PipelineEvent::MergeRetryStarted)
{
invalidate_generation(&generations, &fired.story_id.0);
}
}
Decision::ScheduleRetry { attempt, budget } => {
schedule_auto_retry(
Arc::clone(&pool),
project_root.clone(),
fired.story_id.0.clone(),
attempt,
budget,
Arc::clone(&generations),
);
}
Decision::Block { count } => {
let kind = counters
.get(&fired.story_id)
.map(|(_, k)| k.clone())
.unwrap_or(MergeFailureKind::Other(String::new()));
apply_block(&fired.story_id, count, &kind);
counters.remove(&fired.story_id);
invalidate_generation(&generations, &fired.story_id.0);
}
}
}
Err(tokio::sync::broadcast::error::RecvError::Lagged(n)) => {
slog_warn!(
@@ -69,94 +154,294 @@ pub(crate) fn spawn_merge_failure_block_subscriber(pool: Arc<AgentPool>, project
});
}
/// Return true if a mergemaster agent is currently in the pool for `story_id`.
/// Used to suppress counter increments while recovery is actively iterating
/// (bug 1025).
fn is_mergemaster_running(pool: &AgentPool, project_root: &Path, story_id: &str) -> bool {
let config = match crate::config::ProjectConfig::load(project_root) {
Ok(c) => c,
Err(_) => return false,
};
let agents = match pool.agents.lock() {
Ok(a) => a,
Err(_) => return false,
};
is_story_assigned_for_stage(&config, &agents, story_id, &PipelineStage::Mergemaster)
}
/// Handle a single transition event: update counters and emit Block if threshold is reached.
/// Pure budget policy: given a fired transition and the per-story counter,
/// decide whether to do nothing, schedule a `GatesFailed` auto-retry, or
/// block the story.
///
/// `recovery_running`: when `true`, a mergemaster is currently in the pool for
/// the story and the failure is part of an in-flight recovery loop. We do NOT
/// increment the consecutive-failure counter in that case (bug 1025).
fn on_transition(
project_root: &Path,
/// `threshold == 0` disables both policies (feature off), matching the
/// pre-1185 block-subscriber behaviour.
fn decide(
fired: &crate::pipeline_state::TransitionFired,
counters: &mut HashMap<StoryId, (u32, MergeFailureKind)>,
recovery_running: bool,
) {
threshold: u32,
) -> Decision {
// Story 1086: gate on the typed `Status` projection — `Status::MergeFailure`
// is precisely the set of stages we count toward the block threshold. We
// still need the variant pattern below to read `kind`.
// is precisely the set of stages we count toward the budget.
if fired.after.status() != Status::MergeFailure {
// The MergeFailure → Merge bounce caused by a retry (auto or manual)
// is not a recovery: the budget must survive it, or a deterministic
// failure retries forever (1185 review finding 1).
if !matches!(fired.event, PipelineEvent::MergeRetryStarted) {
counters.remove(&fired.story_id);
}
return Decision::Nothing;
}
let Stage::MergeFailure { kind, .. } = &fired.after else {
counters.remove(&fired.story_id);
return Decision::Nothing;
};
if recovery_running {
slog!(
"[merge-block-sub] Story '{}' MergeFailure while mergemaster is running; \
not counting toward budget (recovery in progress).",
fired.story_id.0
);
return Decision::Nothing;
}
let entry = counters
.entry(fired.story_id.clone())
.or_insert_with(|| (0, kind.clone()));
entry.0 += 1;
entry.1 = kind.clone();
let count = entry.0;
if threshold == 0 {
return Decision::Nothing;
}
if count >= threshold {
return Decision::Block { count };
}
if matches!(kind, MergeFailureKind::GatesFailed(_)) {
return Decision::ScheduleRetry {
attempt: count,
budget: threshold,
};
}
Decision::Nothing
}
/// Transition `story_id` to `Blocked` with a reason naming the failure kind.
fn apply_block(story_id: &StoryId, count: u32, kind: &MergeFailureKind) {
let kind_str = failure_kind_label(kind);
let reason =
format!("Auto-blocked after {count} consecutive MergeFailure ({kind_str}) transitions.");
let story_id = story_id.0.as_str();
slog!(
"[merge-block-sub] Story '{story_id}' reached {count} consecutive \
MergeFailure ({kind_str}); blocking."
);
if let Err(e) =
crate::pipeline_state::apply_transition(story_id, PipelineEvent::Block { reason }, None)
{
slog_warn!("[merge-block-sub] Failed to block '{story_id}': {e}");
}
}
/// Spawn the delayed auto-retry task for one `GatesFailed` failure.
///
/// Bumps the story's scheduling generation so any earlier still-sleeping timer
/// for the story becomes stale and skips itself.
fn schedule_auto_retry(
pool: Arc<AgentPool>,
project_root: PathBuf,
story_id: String,
attempt: u32,
budget: u32,
generations: Generations,
) {
let generation = next_generation(&generations, &story_id);
slog!(
"[merge-block-sub] Story '{story_id}' GatesFailed (attempt {attempt}/{budget}); \
scheduling auto-retry in {AUTO_RETRY_DELAY:?}."
);
tokio::spawn(async move {
tokio::time::sleep(AUTO_RETRY_DELAY).await;
perform_auto_retry(
&pool,
&project_root,
&story_id,
attempt,
budget,
&generations,
generation,
)
.await;
});
}
/// Fire one auto-retry: re-check that acting is still correct, start the
/// server-side merge, and notify chat only when the merge actually started
/// (1185 review finding 7).
///
/// Guards, in order:
/// - the scheduling generation is still current (no newer schedule/reset);
/// - the story is still parked in `MergeFailure{GatesFailed}`;
/// - the story is not frozen/blocked/on hold/dep-blocked — the same
/// eligibility gates `assign_merge_stage` applies (1185 review finding 4);
/// - no mergemaster is actively recovering the story (1185 review finding 2).
async fn perform_auto_retry(
pool: &Arc<AgentPool>,
project_root: &Path,
story_id: &str,
attempt: u32,
budget: u32,
generations: &Generations,
generation: u64,
) {
if !is_generation_current(generations, story_id, generation) {
slog!(
"[merge-block-sub] Story '{story_id}' auto-retry ({attempt}/{budget}) is stale \
(superseded or reset); skipping."
);
return;
}
match &fired.after {
Stage::MergeFailure { kind, .. } => {
if recovery_running {
slog!(
"[merge-block-sub] Story '{}' MergeFailure while mergemaster is running; \
not counting toward block threshold (recovery in progress).",
fired.story_id.0
);
return;
}
let entry = counters
.entry(fired.story_id.clone())
.or_insert_with(|| (0, kind.clone()));
entry.0 += 1;
entry.1 = kind.clone();
let count = entry.0;
let threshold = load_threshold(project_root);
if threshold == 0 {
return;
let still_gates_failed = matches!(
crate::pipeline_state::read_typed(story_id),
Ok(Some(item)) if matches!(
item.stage,
Stage::MergeFailure {
kind: MergeFailureKind::GatesFailed(_),
..
}
)
);
if !still_gates_failed {
slog!(
"[merge-block-sub] Story '{story_id}' left GatesFailed before auto-retry \
({attempt}/{budget}) fired; skipping."
);
return;
}
if count >= threshold {
let kind_str = failure_kind_label(kind);
let reason = format!(
"Auto-blocked after {count} consecutive MergeFailure ({kind_str}) transitions."
);
let story_id = fired.story_id.0.as_str();
slog!(
"[merge-block-sub] Story '{story_id}' reached {count} consecutive \
MergeFailure ({kind_str}); blocking."
);
if let Err(e) = crate::pipeline_state::apply_transition(
story_id,
PipelineEvent::Block { reason },
None,
) {
slog_warn!("[merge-block-sub] Failed to block '{story_id}': {e}");
} else {
counters.remove(&fired.story_id);
}
}
if has_review_hold(story_id)
|| is_story_frozen(story_id)
|| is_story_blocked(story_id)
|| has_unmet_dependencies(story_id)
{
slog!(
"[merge-block-sub] Story '{story_id}' is held/frozen/blocked/dep-blocked; \
skipping auto-retry ({attempt}/{budget})."
);
return;
}
if is_mergemaster_running(pool, project_root, story_id).await {
slog!(
"[merge-block-sub] Story '{story_id}' has an active mergemaster; \
skipping auto-retry ({attempt}/{budget}) recovery owns the story."
);
return;
}
match pool.start_merge_agent_work(project_root, story_id) {
Ok(()) => {
slog!(
"[merge-block-sub] Auto-retrying merge for '{story_id}' \
(attempt {attempt}/{budget})."
);
let _ = pool.watcher_tx.send(WatcherEvent::MergeAutoRetry {
story_id: story_id.to_string(),
attempt,
budget,
});
}
_ => {
counters.remove(&fired.story_id);
Err(e) => {
slog_warn!(
"[merge-block-sub] Auto-retry for '{story_id}' ({attempt}/{budget}) \
could not start: {e}; not notifying."
);
}
}
}
/// Load the threshold from project config, falling back to the compiled default.
fn load_threshold(project_root: &Path) -> u32 {
crate::config::ProjectConfig::load(project_root)
.map(|c| c.merge_failure_block_threshold)
.unwrap_or(3)
/// One-shot startup scan: schedule a first auto-retry for every story already
/// parked in `MergeFailure{GatesFailed}`.
///
/// The pre-restart attempt count is unrecoverable, so the counter restarts at
/// 1 — worst case a story gets up to `threshold - 1` extra retries across a
/// restart, still bounded per process lifetime.
async fn reconcile_stranded_gates_failed(
pool: &Arc<AgentPool>,
project_root: &Path,
counters: &mut HashMap<StoryId, (u32, MergeFailureKind)>,
generations: &Generations,
) {
let threshold = load_threshold(project_root).await;
if threshold == 0 {
return;
}
for item in crate::pipeline_state::read_all_typed() {
let Stage::MergeFailure { kind, .. } = &item.stage else {
continue;
};
if !matches!(kind, MergeFailureKind::GatesFailed(_)) {
continue;
}
counters.insert(item.story_id.clone(), (1, kind.clone()));
slog!(
"[merge-block-sub] Story '{}' found parked in GatesFailed at startup; \
scheduling catch-up auto-retry (attempt 1/{threshold}).",
item.story_id.0
);
schedule_auto_retry(
Arc::clone(pool),
project_root.to_path_buf(),
item.story_id.0.clone(),
1,
threshold,
Arc::clone(generations),
);
}
}
/// Bump and return the scheduling generation for `story_id`.
fn next_generation(generations: &Generations, story_id: &str) -> u64 {
let mut map = generations.lock().unwrap_or_else(|p| p.into_inner());
let entry = map.entry(story_id.to_string()).or_insert(0);
*entry += 1;
*entry
}
/// Drop the generation entry for `story_id`, making every pending timer stale.
fn invalidate_generation(generations: &Generations, story_id: &str) {
generations
.lock()
.unwrap_or_else(|p| p.into_inner())
.remove(story_id);
}
/// True when `expected` is still the current scheduling generation.
fn is_generation_current(generations: &Generations, story_id: &str, expected: u64) -> bool {
generations
.lock()
.unwrap_or_else(|p| p.into_inner())
.get(story_id)
== Some(&expected)
}
/// Return true if a mergemaster agent is currently in the pool for `story_id`.
/// Used to suppress counting and retries while recovery is actively iterating
/// (bug 1025).
async fn is_mergemaster_running(pool: &AgentPool, project_root: &Path, story_id: &str) -> bool {
let root = project_root.to_path_buf();
let config = match tokio::task::spawn_blocking(move || {
crate::config::ProjectConfig::load(&root)
})
.await
{
Ok(Ok(c)) => c,
_ => return false,
};
let agents = pool.agents.lock().await;
is_story_assigned_for_stage(&config, &agents, story_id, &PipelineStage::Mergemaster)
}
/// Load the budget from project config off the async runtime (the read is
/// synchronous filesystem I/O — bug 1170 class), falling back to the compiled
/// default.
async fn load_threshold(project_root: &Path) -> u32 {
let root = project_root.to_path_buf();
tokio::task::spawn_blocking(move || {
crate::config::ProjectConfig::load(&root)
.map(|c| c.merge_failure_block_threshold)
.unwrap_or(3)
})
.await
.unwrap_or(3)
}
/// Short human-readable label for a [`MergeFailureKind`] variant.
@@ -178,11 +463,7 @@ mod tests {
use crate::pipeline_state::{BranchName, PipelineEvent, Stage, StoryId, TransitionFired};
use std::num::NonZeroU32;
fn setup_project(tmp: &tempfile::TempDir) {
let sk = tmp.path().join(".huskies");
std::fs::create_dir_all(&sk).unwrap();
std::fs::write(sk.join("project.toml"), "[[agent]]\nname = \"coder\"\n").unwrap();
}
const THRESHOLD: u32 = 3;
fn seed_at_merge(story_id: &str) {
crate::crdt_state::init_for_test();
@@ -233,218 +514,232 @@ mod tests {
}
}
/// AC3 (threshold-not-reached): 2 consecutive failures below threshold of 3 must NOT block.
#[test]
fn below_threshold_does_not_block() {
let tmp = tempfile::tempdir().unwrap();
setup_project(&tmp);
let story_id = "1018_below";
seed_at_merge(story_id);
// Transition to MergeFailure once to establish the stage.
crate::agents::lifecycle::transition_to_merge_failure(
story_id,
MergeFailureKind::GatesFailed("error".to_string()),
)
.expect("initial MergeFailure transition");
let mut counters: HashMap<StoryId, (u32, MergeFailureKind)> = HashMap::new();
let kind = MergeFailureKind::GatesFailed("error".to_string());
// Fire 2 MergeFailure events (default threshold is 3).
for _ in 0..2 {
let fired = make_merge_failure_fired(story_id, kind.clone());
on_transition(tmp.path(), &fired, &mut counters, false);
/// The MergeFailure → Merge bounce a retry causes.
fn make_retry_started_fired(story_id: &str) -> TransitionFired {
TransitionFired {
story_id: StoryId(story_id.to_string()),
before: Stage::MergeFailure {
kind: MergeFailureKind::GatesFailed("error".to_string()),
feature_branch: BranchName("feature/test".to_string()),
commits_ahead: NonZeroU32::new(1).unwrap(),
},
after: Stage::Merge {
feature_branch: BranchName("feature/test".to_string()),
commits_ahead: NonZeroU32::new(1).unwrap(),
claim: None,
retries: 1,
server_start_time: None,
},
event: PipelineEvent::MergeRetryStarted,
at: chrono::Utc::now(),
}
}
// Story must still be in MergeFailure (not Blocked).
let item = crate::pipeline_state::read_typed(story_id)
.expect("read")
.expect("item");
assert!(
matches!(item.stage, Stage::MergeFailure { .. }),
"story must still be in MergeFailure after 2 failures (threshold 3): {:?}",
item.stage
fn gates_failed() -> MergeFailureKind {
MergeFailureKind::GatesFailed("error".to_string())
}
/// Below the threshold, GatesFailed schedules a retry with the right
/// attempt numbering.
#[test]
fn gates_failed_below_threshold_schedules_retry() {
let mut counters = HashMap::new();
let fired = make_merge_failure_fired("t_sched", gates_failed());
assert_eq!(
decide(&fired, &mut counters, false, THRESHOLD),
Decision::ScheduleRetry {
attempt: 1,
budget: THRESHOLD
}
);
assert_eq!(
decide(&fired, &mut counters, false, THRESHOLD),
Decision::ScheduleRetry {
attempt: 2,
budget: THRESHOLD
}
);
}
/// AC3 (threshold-reached): 3 consecutive failures at threshold of 3 must block.
/// 1185 review finding 1 (regression): the retry's own MergeFailure→Merge
/// bounce must NOT reset the counter — the third consecutive failure
/// blocks even though retries happened in between.
#[test]
fn at_threshold_blocks_with_failure_kind_in_reason() {
let tmp = tempfile::tempdir().unwrap();
setup_project(&tmp);
fn merge_retry_started_does_not_reset_counter() {
let mut counters = HashMap::new();
let story = "t_no_reset";
let fail = make_merge_failure_fired(story, gates_failed());
let bounce = make_retry_started_fired(story);
assert!(matches!(
decide(&fail, &mut counters, false, THRESHOLD),
Decision::ScheduleRetry { attempt: 1, .. }
));
assert_eq!(
decide(&bounce, &mut counters, false, THRESHOLD),
Decision::Nothing
);
assert!(matches!(
decide(&fail, &mut counters, false, THRESHOLD),
Decision::ScheduleRetry { attempt: 2, .. }
));
assert_eq!(
decide(&bounce, &mut counters, false, THRESHOLD),
Decision::Nothing
);
// Third consecutive failure: budget exhausted despite the bounces.
assert_eq!(
decide(&fail, &mut counters, false, THRESHOLD),
Decision::Block { count: 3 }
);
}
/// A real departure (FixupRequested → Coding) still resets the counter.
#[test]
fn real_departure_resets_counter() {
let mut counters = HashMap::new();
let story = "t_reset";
let fail = make_merge_failure_fired(story, gates_failed());
decide(&fail, &mut counters, false, THRESHOLD);
decide(&fail, &mut counters, false, THRESHOLD);
assert_eq!(
counters.get(&StoryId(story.to_string())).map(|e| e.0),
Some(2)
);
decide(&make_coding_fired(story), &mut counters, false, THRESHOLD);
assert!(!counters.contains_key(&StoryId(story.to_string())));
// Fresh failures start a fresh budget.
assert!(matches!(
decide(&fail, &mut counters, false, THRESHOLD),
Decision::ScheduleRetry { attempt: 1, .. }
));
}
/// Non-GatesFailed kinds count toward the block budget but never schedule
/// a retry (ConflictDetected has its own mergemaster path; the rest need
/// humans).
#[test]
fn non_gates_failed_counts_but_does_not_retry() {
let mut counters = HashMap::new();
let story = "t_conflict";
let conflict = make_merge_failure_fired(story, MergeFailureKind::ConflictDetected(None));
assert_eq!(
decide(&conflict, &mut counters, false, THRESHOLD),
Decision::Nothing
);
assert_eq!(
counters.get(&StoryId(story.to_string())).map(|e| e.0),
Some(1)
);
assert_eq!(
decide(&conflict, &mut counters, false, THRESHOLD),
Decision::Nothing
);
assert_eq!(
decide(&conflict, &mut counters, false, THRESHOLD),
Decision::Block { count: 3 }
);
}
/// Mixed kinds share one budget: GatesFailed and ConflictDetected
/// interleavings block at the same total count (1185 review finding 5).
#[test]
fn mixed_kinds_share_one_budget() {
let mut counters = HashMap::new();
let story = "t_mixed";
let fail = make_merge_failure_fired(story, gates_failed());
let conflict = make_merge_failure_fired(story, MergeFailureKind::ConflictDetected(None));
assert!(matches!(
decide(&fail, &mut counters, false, THRESHOLD),
Decision::ScheduleRetry { attempt: 1, .. }
));
assert_eq!(
decide(&conflict, &mut counters, false, THRESHOLD),
Decision::Nothing
);
assert_eq!(
decide(&fail, &mut counters, false, THRESHOLD),
Decision::Block { count: 3 }
);
}
/// Bug 1025: recovery in progress neither counts nor schedules.
#[test]
fn mergemaster_running_suppresses_counting_and_retry() {
let mut counters = HashMap::new();
let story = "t_recovery";
let fail = make_merge_failure_fired(story, gates_failed());
for _ in 0..3 {
assert_eq!(
decide(&fail, &mut counters, true, THRESHOLD),
Decision::Nothing
);
}
assert!(!counters.contains_key(&StoryId(story.to_string())));
}
/// threshold == 0 disables both policies.
#[test]
fn threshold_zero_disables_block_and_retry() {
let mut counters = HashMap::new();
let fail = make_merge_failure_fired("t_disabled", gates_failed());
for _ in 0..5 {
assert_eq!(decide(&fail, &mut counters, false, 0), Decision::Nothing);
}
}
/// Applying a Block decision transitions the story and names the kind.
#[test]
fn apply_block_blocks_with_failure_kind_in_reason() {
let story_id = "1018_at_threshold";
seed_at_merge(story_id);
crate::agents::lifecycle::transition_to_merge_failure(
story_id,
MergeFailureKind::GatesFailed("fmt error".to_string()),
)
.expect("initial MergeFailure transition");
let mut counters: HashMap<StoryId, (u32, MergeFailureKind)> = HashMap::new();
let kind = MergeFailureKind::GatesFailed("fmt error".to_string());
// Fire 3 MergeFailure events — the 3rd must trigger the block.
for _ in 0..3 {
let fired = make_merge_failure_fired(story_id, kind.clone());
on_transition(tmp.path(), &fired, &mut counters, false);
}
apply_block(
&StoryId(story_id.to_string()),
3,
&MergeFailureKind::GatesFailed("fmt error".to_string()),
);
let item = crate::pipeline_state::read_typed(story_id)
.expect("read")
.expect("item");
assert!(
matches!(item.stage, Stage::Blocked { .. }),
"story must be Blocked after 3 consecutive MergeFailures: {:?}",
item.stage
);
// The block reason must name the failure kind.
if let Stage::Blocked { reason } = &item.stage {
assert!(
reason.contains("GatesFailed"),
"block reason must name the failure kind: {reason}"
);
match &item.stage {
Stage::Blocked { reason } => {
assert!(
reason.contains("GatesFailed"),
"block reason must name the failure kind: {reason}"
);
}
other => panic!("story must be Blocked: {other:?}"),
}
}
/// AC3 (reset): counter clears after a non-MergeFailure transition.
///
/// 2 failures → FixupRequested reset → 2 more failures: still below threshold, no block.
/// 1185 review finding 6 (regression): a newer schedule or a reset makes
/// earlier timers stale.
#[test]
fn counter_resets_on_non_merge_failure_transition() {
let tmp = tempfile::tempdir().unwrap();
setup_project(&tmp);
let story_id = "1018_reset";
seed_at_merge(story_id);
fn stale_generations_are_not_current() {
let generations: Generations = Generations::default();
let g1 = next_generation(&generations, "s");
assert!(is_generation_current(&generations, "s", g1));
crate::agents::lifecycle::transition_to_merge_failure(
story_id,
MergeFailureKind::ConflictDetected(None),
)
.expect("initial MergeFailure transition");
let g2 = next_generation(&generations, "s");
assert!(!is_generation_current(&generations, "s", g1));
assert!(is_generation_current(&generations, "s", g2));
let mut counters: HashMap<StoryId, (u32, MergeFailureKind)> = HashMap::new();
let kind = MergeFailureKind::ConflictDetected(None);
// Fire 2 MergeFailure events.
for _ in 0..2 {
let fired = make_merge_failure_fired(story_id, kind.clone());
on_transition(tmp.path(), &fired, &mut counters, false);
}
assert_eq!(
counters.get(&StoryId(story_id.to_string())).map(|e| e.0),
Some(2),
"counter must be 2 after 2 failures"
);
// Simulate FixupRequested (non-MergeFailure transition).
let reset_fired = make_coding_fired(story_id);
on_transition(tmp.path(), &reset_fired, &mut counters, false);
assert!(
!counters.contains_key(&StoryId(story_id.to_string())),
"counter must be cleared after non-MergeFailure transition"
);
// Re-seed to MergeFailure so we can apply the block transition.
crate::agents::lifecycle::transition_to_merge_failure(
story_id,
MergeFailureKind::ConflictDetected(None),
)
.expect("re-enter MergeFailure after reset");
// Fire 2 more MergeFailure events — still below threshold.
for _ in 0..2 {
let fired = make_merge_failure_fired(story_id, kind.clone());
on_transition(tmp.path(), &fired, &mut counters, false);
}
let item = crate::pipeline_state::read_typed(story_id)
.expect("read")
.expect("item");
assert!(
matches!(item.stage, Stage::MergeFailure { .. }),
"story must still be in MergeFailure after reset + 2 new failures: {:?}",
item.stage
);
}
/// Bug 1025: while a mergemaster is running, MergeFailure transitions are
/// recovery iterations, not consecutive give-ups. 3 failures with
/// `recovery_running=true` must NOT block.
#[test]
fn mergemaster_running_suppresses_block() {
let tmp = tempfile::tempdir().unwrap();
setup_project(&tmp);
let story_id = "1025_recovery_running";
seed_at_merge(story_id);
crate::agents::lifecycle::transition_to_merge_failure(
story_id,
MergeFailureKind::ConflictDetected(None),
)
.expect("initial MergeFailure transition");
let mut counters: HashMap<StoryId, (u32, MergeFailureKind)> = HashMap::new();
let kind = MergeFailureKind::ConflictDetected(None);
// Fire 3 MergeFailure events WHILE a mergemaster is running (gated).
for _ in 0..3 {
let fired = make_merge_failure_fired(story_id, kind.clone());
on_transition(tmp.path(), &fired, &mut counters, true);
}
// Counter must NOT have incremented at all — recovery in progress.
assert!(
!counters.contains_key(&StoryId(story_id.to_string())),
"counter must not increment while mergemaster is running"
);
// And the story must still be in MergeFailure (not Blocked).
let item = crate::pipeline_state::read_typed(story_id)
.expect("read")
.expect("item");
assert!(
matches!(item.stage, Stage::MergeFailure { .. }),
"story must NOT be blocked while mergemaster is running (recovery in progress): {:?}",
item.stage
);
}
/// Bug 1025 regression guard: the genuinely-stuck case (no mergemaster
/// running) still blocks at the threshold, so the original 1018 behaviour
/// is preserved.
#[test]
fn no_mergemaster_still_blocks_at_threshold() {
let tmp = tempfile::tempdir().unwrap();
setup_project(&tmp);
let story_id = "1025_genuine_stuck";
seed_at_merge(story_id);
crate::agents::lifecycle::transition_to_merge_failure(
story_id,
MergeFailureKind::ConflictDetected(None),
)
.expect("initial MergeFailure transition");
let mut counters: HashMap<StoryId, (u32, MergeFailureKind)> = HashMap::new();
let kind = MergeFailureKind::ConflictDetected(None);
// Fire 3 MergeFailure events with NO mergemaster (recovery_running=false).
for _ in 0..3 {
let fired = make_merge_failure_fired(story_id, kind.clone());
on_transition(tmp.path(), &fired, &mut counters, false);
}
// Story must be Blocked (genuine-stuck case unchanged).
let item = crate::pipeline_state::read_typed(story_id)
.expect("read")
.expect("item");
assert!(
matches!(item.stage, Stage::Blocked { .. }),
"story must still block when no mergemaster is running: {:?}",
item.stage
);
invalidate_generation(&generations, "s");
assert!(!is_generation_current(&generations, "s", g2));
}
}
@@ -100,15 +100,7 @@ async fn on_merge_failure_transition(
};
let agent_name = {
let agents = match pool.agents.lock() {
Ok(a) => a,
Err(e) => {
slog_warn!(
"[merge-failure-sub] Failed to lock agent pool for '{story_id}': {e}"
);
return;
}
};
let agents = pool.agents.lock().await;
if is_story_assigned_for_stage(
&config,
&agents,
@@ -228,7 +220,7 @@ mod tests {
);
on_merge_failure_transition(&pool, tmp.path(), &fired).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.lock().await;
assert!(
agents.iter().any(|(key, a)| {
key.contains(story_id)
@@ -259,7 +251,7 @@ mod tests {
// Give the subscriber time to run (it should do nothing).
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.lock().await;
let spawned = agents.iter().any(|(key, a)| {
key.contains(story_id)
&& a.agent_name == "mergemaster"
@@ -287,7 +279,7 @@ mod tests {
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.lock().await;
let spawned = agents.iter().any(|(key, a)| {
key.contains(story_id)
&& a.agent_name == "mergemaster"
@@ -315,7 +307,7 @@ mod tests {
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.lock().await;
let spawned = agents.iter().any(|(key, a)| {
key.contains(story_id)
&& a.agent_name == "mergemaster"
@@ -343,7 +335,7 @@ mod tests {
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.lock().await;
let spawned = agents.iter().any(|(key, a)| {
key.contains(story_id)
&& a.agent_name == "mergemaster"
@@ -374,7 +366,7 @@ mod tests {
// First call — spawns mergemaster (agent enters Pending).
on_merge_failure_transition(&pool, tmp.path(), &fired).await;
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.lock().await;
assert!(
agents.iter().any(|(key, a)| {
key.contains(story_id)
@@ -388,7 +380,7 @@ mod tests {
// Second call (self-loop) — agent is still Pending; guard must prevent double-spawn.
on_merge_failure_transition(&pool, tmp.path(), &fired).await;
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.lock().await;
let active_count = agents
.iter()
.filter(|(key, a)| {
+2 -1
View File
@@ -4,7 +4,8 @@
mod auto_assign;
mod backlog;
mod merge;
/// TransitionFired subscriber that auto-blocks stories after N consecutive MergeFailure transitions.
/// TransitionFired subscriber owning the consecutive-MergeFailure budget:
/// auto-retries GatesFailed below the threshold, auto-blocks at it.
pub(crate) mod merge_failure_block_subscriber;
/// TransitionFired subscriber that auto-spawns mergemaster on ConflictDetected merge failures.
pub(crate) mod merge_failure_subscriber;
+2 -15
View File
@@ -5,7 +5,6 @@ use std::path::Path;
use crate::config::ProjectConfig;
use crate::pipeline_state::Stage;
use crate::slog;
use crate::slog_error;
use super::super::super::PipelineStage;
use super::super::AgentPool;
@@ -80,13 +79,7 @@ impl AgentPool {
if *stage == PipelineStage::Coder
&& let Some(max) = config.max_coders
{
let agents_lock = match self.agents.lock() {
Ok(a) => a,
Err(e) => {
slog_error!("[auto-assign] Failed to lock agents: {e}");
break;
}
};
let agents_lock = self.agents.lock().await;
let active = count_active_agents_for_stage(config, &agents_lock, stage);
if active >= max {
slog!(
@@ -102,13 +95,7 @@ impl AgentPool {
// stage_mismatch=true means the preferred agent's stage doesn't match the
// pipeline stage, so we fell back to a generic stage agent.
let (already_assigned, free_agent, preferred_busy, stage_mismatch) = {
let agents = match self.agents.lock() {
Ok(a) => a,
Err(e) => {
slog_error!("[auto-assign] Failed to lock agents: {e}");
break;
}
};
let agents = self.agents.lock().await;
let assigned = is_story_assigned_for_stage(config, &agents, story_id, stage);
if assigned {
(true, None, false, false)
+6 -6
View File
@@ -256,7 +256,7 @@ mod tests {
let pool = AgentPool::new_test(3001);
pool.inject_test_agent("42_story_foo", "coder-1", AgentStatus::Running);
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
assert!(is_story_assigned_for_stage(
&config,
&agents,
@@ -285,7 +285,7 @@ mod tests {
let pool = AgentPool::new_test(3001);
pool.inject_test_agent("42_story_foo", "coder-1", AgentStatus::Completed);
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
// Completed agents don't count as assigned
assert!(!is_story_assigned_for_stage(
&config,
@@ -309,7 +309,7 @@ stage = "qa"
let pool = AgentPool::new_test(3001);
pool.inject_test_agent("42_story_foo", "qa-2", AgentStatus::Running);
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
// qa-2 with stage=qa should be recognised as a QA agent
assert!(
is_story_assigned_for_stage(&config, &agents, "42_story_foo", &PipelineStage::Qa),
@@ -338,7 +338,7 @@ name = "coder-2"
pool.inject_test_agent("s1", "coder-1", AgentStatus::Running);
pool.inject_test_agent("s2", "coder-2", AgentStatus::Running);
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let free = find_free_agent_for_stage(&config, &agents, &PipelineStage::Coder);
assert!(free.is_none(), "no free coders should be available");
}
@@ -361,7 +361,7 @@ name = "coder-3"
// coder-1 is busy, coder-2 is free
pool.inject_test_agent("s1", "coder-1", AgentStatus::Running);
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let free = find_free_agent_for_stage(&config, &agents, &PipelineStage::Coder);
assert_eq!(
free,
@@ -384,7 +384,7 @@ name = "coder-1"
// coder-1 completed its previous story — it's free for a new one
pool.inject_test_agent("s1", "coder-1", AgentStatus::Completed);
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let free = find_free_agent_for_stage(&config, &agents, &PipelineStage::Coder);
assert_eq!(free, Some("coder-1"), "completed coder-1 should be free");
}
@@ -2,7 +2,7 @@
use std::collections::HashMap;
use std::path::Path;
use std::sync::Mutex;
use tokio::sync::Mutex;
use tokio::sync::broadcast;
use crate::agents::pool::StoryAgent;
@@ -83,7 +83,7 @@ pub(crate) fn count_turns_in_log(path: &Path) -> u64 {
/// Turns and budget are counted from the **current session's** log file
/// only — prior sessions are excluded so that restart counts from earlier
/// runs do not accumulate against the limits.
pub(super) fn check_agent_limits(
pub(super) async fn check_agent_limits(
agents: &Mutex<HashMap<String, StoryAgent>>,
project_root: &Path,
) -> Vec<(String, TerminationReason)> {
@@ -94,10 +94,7 @@ pub(super) fn check_agent_limits(
// Snapshot running agents: (key, story_id, agent_name, tx, log_session_id).
let running: Vec<RunningAgentSnapshot> = {
let lock = match agents.lock() {
Ok(l) => l,
Err(_) => return Vec::new(),
};
let lock = agents.lock().await;
lock.iter()
.filter(|(_, agent)| agent.status == AgentStatus::Running)
.map(|(key, agent)| {
@@ -4,9 +4,11 @@
mod budget;
mod limits;
mod orphan;
mod reap;
#[cfg(test)]
mod tests;
use std::collections::HashSet;
use std::path::Path;
use crate::agents::AgentStatus;
@@ -18,6 +20,7 @@ use crate::slog_warn;
use super::super::AgentPool;
use limits::check_agent_limits;
use orphan::check_orphaned_agents;
use reap::reap_failed_agents;
pub(crate) use budget::{compute_budget_from_logs, compute_budget_from_single_log};
pub(crate) use limits::{count_turns_in_log, resolve_session_log};
@@ -25,8 +28,8 @@ pub(crate) use limits::{count_turns_in_log, resolve_session_log};
impl AgentPool {
/// Run a single watchdog pass synchronously (test helper).
#[cfg(test)]
pub fn run_watchdog_once(&self) {
check_orphaned_agents(&self.agents);
pub async fn run_watchdog_once(&self) {
check_orphaned_agents(&self.agents).await;
}
/// Run one watchdog pass: detect orphans, enforce limits, kill offenders.
@@ -39,21 +42,24 @@ impl AgentPool {
/// `retry_count` is incremented and the story stays in `2_current/` for
/// re-attempt. This prevents the original kill-respawn loop (bug 646)
/// while restoring the `max_retries` semantic for turn/budget overruns.
pub fn run_watchdog_pass(&self, project_root: Option<&Path>) -> usize {
let orphaned = check_orphaned_agents(&self.agents);
pub async fn run_watchdog_pass(&self, project_root: Option<&Path>) -> usize {
let orphaned = check_orphaned_agents(&self.agents).await;
if let Some(root) = project_root {
let terminated = check_agent_limits(&self.agents, root);
let terminated = check_agent_limits(&self.agents, root).await;
let config = ProjectConfig::load(root).unwrap_or_default();
let mut just_terminated: HashSet<String> = HashSet::new();
for (key, reason) in &terminated {
just_terminated.insert(key.clone());
// Step 1: snapshot the agent's worktree path so we can find every
// process running in it (claude + any subprocesses). This must
// happen BEFORE we mutate the agent record so we can read the
// worktree info safely.
let worktree_path = self.agents.lock().ok().and_then(|lock| {
let worktree_path = {
let lock = self.agents.lock().await;
lock.get(key)
.and_then(|a| a.worktree_info.as_ref().map(|wt| wt.path.clone()))
});
};
// Step 2: SIGKILL every process running in the worktree and
// BLOCK until verified gone. The previous mechanism — portable_pty's
@@ -85,23 +91,24 @@ impl AgentPool {
"[watchdog] No worktree path recorded for '{key}'; cannot tree-kill, \
falling back to portable_pty SIGHUP (likely no-op for claude-code)."
);
self.kill_child_for_key(key);
self.kill_child_for_key(key).await;
}
// Step 3: NOW update the agent record. The process is verified
// gone (or we logged that SIGKILL didn't take effect, which is
// exceptional), so flipping status away from Running can no
// longer open a window for a concurrent spawn.
if let Ok(mut lock) = self.agents.lock()
&& let Some(agent) = lock.get_mut(key)
{
agent.status = AgentStatus::Failed;
agent.termination_reason = Some(reason.clone());
if let Some(handle) = agent.task_handle.take() {
// Best-effort abort of the outer tokio task. The PTY
// blocking thread already returned (claude is dead),
// so this is bookkeeping rather than load-bearing.
handle.abort();
let mut lock = self.agents.lock().await;
if let Some(agent) = lock.get_mut(key) {
agent.status = AgentStatus::Failed;
agent.termination_reason = Some(reason.clone());
if let Some(handle) = agent.task_handle.take() {
// Best-effort abort of the outer tokio task. The PTY
// blocking thread already returned (claude is dead),
// so this is bookkeeping rather than load-bearing.
handle.abort();
}
}
}
@@ -130,7 +137,16 @@ impl AgentPool {
if !terminated.is_empty() {
Self::notify_agent_state_changed(&self.watcher_tx);
}
return orphaned + terminated.len();
// Bug 1198: reap any other Failed pool entry with no live process
// — orphan-detected above, or left behind by a spawn error
// (inactivity-watchdog kill, worktree timeout, runtime error)
// that never routed through the retry/respawn path. Entries the
// limits loop above just processed are excluded so their retry
// count isn't bumped twice.
let reaped = reap_failed_agents(self, root, &config, &just_terminated).await;
return orphaned + terminated.len() + reaped;
}
orphaned
@@ -1,7 +1,7 @@
//! Orphan detection: marks running agents whose backing task has exited.
use std::collections::HashMap;
use std::sync::Mutex;
use tokio::sync::Mutex;
use tokio::sync::broadcast;
use crate::agents::pool::StoryAgent;
@@ -15,11 +15,8 @@ use crate::slog;
/// without updating the agent status — for example when the process is killed
/// externally and the PTY master fd returns EOF before our inactivity timeout
/// fires, but some other edge case prevents the normal cleanup path from running.
pub(super) fn check_orphaned_agents(agents: &Mutex<HashMap<String, StoryAgent>>) -> usize {
let mut lock = match agents.lock() {
Ok(l) => l,
Err(_) => return 0,
};
pub(super) async fn check_orphaned_agents(agents: &Mutex<HashMap<String, StoryAgent>>) -> usize {
let mut lock = agents.lock().await;
// Collect orphaned entries: Running or Pending agents whose task handle is finished.
// Pending agents can be orphaned if worktree creation panics before setting status.
@@ -0,0 +1,99 @@
//! Reap: removes stale `Failed` pool entries left behind by orphan detection,
//! watchdog kills, or internal spawn failures, and respawns the story's agent
//! (or blocks the story once its retry budget is exhausted).
//!
//! `check_orphaned_agents` only scans `Running`/`Pending` entries, so once an
//! entry is marked `Failed` it becomes invisible to every later watchdog
//! pass. Without this step a `Failed` entry with no live process sits in the
//! pool forever: `list_agents` keeps showing it, and the story never gets a
//! new agent unless some unrelated CRDT transition happens to trigger a
//! system-wide auto-assign scan.
use std::collections::HashSet;
use std::path::Path;
use crate::agents::AgentStatus;
use crate::agents::pool::AgentPool;
use crate::agents::pool::pipeline::should_block_story;
use crate::config::ProjectConfig;
use crate::io::watcher::WatcherEvent;
use crate::{slog, slog_warn};
/// Reap every `Failed` pool entry with no live process, except keys in
/// `exclude_keys` (already handled by the caller's own retry/block logic in
/// this same pass — e.g. limit-exceeded kills).
///
/// For each reaped entry: removes it from the pool (so `list_agents` stops
/// showing it), increments the story's retry count via [`should_block_story`]
/// and, unless that blocks the story, respawns the agent by name.
/// `start_agent`'s own session-store lookup resumes the prior session
/// automatically whenever one was recorded — no explicit session plumbing
/// needed here.
pub(super) async fn reap_failed_agents(
pool: &AgentPool,
project_root: &Path,
config: &ProjectConfig,
exclude_keys: &HashSet<String>,
) -> usize {
let dead: Vec<(String, String, String)> = {
let mut agents = pool.agents.lock().await;
let keys: Vec<String> = agents
.iter()
.filter(|(key, agent)| {
agent.status == AgentStatus::Failed
&& !exclude_keys.contains(*key)
&& agent
.task_handle
.as_ref()
.map(|h| h.is_finished())
.unwrap_or(true)
})
.map(|(key, _)| key.clone())
.collect();
keys.into_iter()
.filter_map(|key| {
agents.remove(&key).map(|agent| {
let story_id = key
.rsplit_once(':')
.map(|(s, _)| s.to_string())
.unwrap_or_else(|| key.clone());
(key, story_id, agent.agent_name)
})
})
.collect()
};
let count = dead.len();
for (key, story_id, agent_name) in dead {
if let Some(block_reason) = should_block_story(&story_id, config.max_retries, "watchdog") {
let _ = pool.watcher_tx.send(WatcherEvent::StoryBlocked {
story_id: story_id.clone(),
reason: block_reason,
});
slog!(
"[watchdog] Story '{story_id}' blocked after exceeding retry limit \
(reaped dead pool entry '{key}')."
);
continue;
}
slog!(
"[watchdog] Reaping dead pool entry '{key}'; respawning '{agent_name}' \
for '{story_id}'."
);
if let Err(e) = pool
.start_agent(project_root, &story_id, Some(&agent_name), None, None)
.await
{
slog_warn!(
"[watchdog] Failed to respawn '{agent_name}' for '{story_id}' after reap: {e}"
);
}
}
if count > 0 {
AgentPool::notify_agent_state_changed(&pool.watcher_tx);
}
count
}
@@ -10,8 +10,8 @@ use crate::agents::{AgentEvent, AgentStatus, TerminationReason};
// ── Limit enforcement integration tests (bug 624) ────────────────────────
#[test]
fn watchdog_terminates_agent_exceeding_turn_limit() {
#[tokio::test]
async fn watchdog_terminates_agent_exceeding_turn_limit() {
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
@@ -37,12 +37,12 @@ max_turns = 10
);
let mut rx = tx.subscribe();
let found = pool.run_watchdog_pass(Some(root));
let found = pool.run_watchdog_pass(Some(root)).await;
assert!(found >= 1, "watchdog should detect the over-limit agent");
// Agent should now be Failed with TurnLimit reason.
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key("story_a", "coder-1");
let agent = agents.get(&key).unwrap();
assert_eq!(agent.status, AgentStatus::Failed);
@@ -60,8 +60,8 @@ max_turns = 10
);
}
#[test]
fn watchdog_terminates_agent_exceeding_budget_limit() {
#[tokio::test]
async fn watchdog_terminates_agent_exceeding_budget_limit() {
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
@@ -87,11 +87,11 @@ max_budget_usd = 5.00
);
let mut rx = tx.subscribe();
let found = pool.run_watchdog_pass(Some(root));
let found = pool.run_watchdog_pass(Some(root)).await;
assert!(found >= 1, "watchdog should detect the over-budget agent");
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key("story_b", "coder-1");
let agent = agents.get(&key).unwrap();
assert_eq!(agent.status, AgentStatus::Failed);
@@ -106,8 +106,8 @@ max_budget_usd = 5.00
assert!(matches!(event, AgentEvent::Error { .. }));
}
#[test]
fn watchdog_does_not_terminate_agent_under_limits() {
#[tokio::test]
async fn watchdog_does_not_terminate_agent_under_limits() {
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
@@ -133,11 +133,11 @@ max_budget_usd = 10.00
// has 25 turns < 50 so no violation).
pool.inject_test_agent_with_session("story_c", "coder-1", AgentStatus::Running, "sess-ok");
let found = pool.run_watchdog_pass(Some(root));
let found = pool.run_watchdog_pass(Some(root)).await;
assert_eq!(found, 0, "agent under limits should not be terminated");
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key("story_c", "coder-1");
let agent = agents.get(&key).unwrap();
assert_eq!(
@@ -153,8 +153,8 @@ max_budget_usd = 10.00
/// coder-1 with max_turns=50, max_budget_usd=5.00 ran 5.6× over the turn
/// limit (280 turns). The watchdog must terminate at the turn limit (turns
/// hit first in the observed trace), with reason TurnLimit.
#[test]
fn regression_bug624_coder1_story623_trajectory() {
#[tokio::test]
async fn regression_bug624_coder1_story623_trajectory() {
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
@@ -183,11 +183,11 @@ max_budget_usd = 5.00
);
let mut rx = tx.subscribe();
let found = pool.run_watchdog_pass(Some(root));
let found = pool.run_watchdog_pass(Some(root)).await;
assert!(found >= 1, "watchdog must catch the turn-limit violation");
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key("story_623", "coder-1");
let agent = agents.get(&key).unwrap();
assert_eq!(agent.status, AgentStatus::Failed);
@@ -218,8 +218,8 @@ max_budget_usd = 5.00
///
/// This test seeds a single session that legitimately exceeds the limit
/// and uses `max_retries = 1` so that the first violation blocks.
#[test]
fn watchdog_marks_story_blocked_after_limit_termination() {
#[tokio::test]
async fn watchdog_marks_story_blocked_after_limit_termination() {
crate::db::ensure_content_store();
crate::crdt_state::init_for_test();
@@ -263,7 +263,7 @@ max_turns = 10
"sess-runaway",
);
let found = pool.run_watchdog_pass(Some(root));
let found = pool.run_watchdog_pass(Some(root)).await;
assert!(found >= 1, "watchdog should detect the over-limit agent");
// With max_retries=1, the first violation blocks immediately via the state machine.
@@ -278,7 +278,7 @@ max_turns = 10
// Sanity: the agent itself is also Failed with the right reason.
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key(story_id, "coder-1");
let agent = agents.get(&key).unwrap();
assert_eq!(agent.status, AgentStatus::Failed);
@@ -297,8 +297,8 @@ max_turns = 10
/// fresh session_id whose log has fewer events than `max_turns`.
/// Assert the agent is NOT terminated (per-session count is under the
/// limit) AND the story is NOT marked blocked.
#[test]
fn per_session_counting_does_not_terminate_under_limit() {
#[tokio::test]
async fn per_session_counting_does_not_terminate_under_limit() {
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
@@ -323,14 +323,14 @@ max_turns = 10
let pool = AgentPool::new_test(3001);
pool.inject_test_agent_with_session("story_d", "coder-1", AgentStatus::Running, "new-sess");
let found = pool.run_watchdog_pass(Some(root));
let found = pool.run_watchdog_pass(Some(root)).await;
assert_eq!(
found, 0,
"agent under per-session limit should NOT be terminated"
);
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key("story_d", "coder-1");
let agent = agents.get(&key).unwrap();
assert_eq!(
@@ -345,8 +345,8 @@ max_turns = 10
/// Same setup as per_session_counting_does_not_terminate_under_limit, but
/// the new agent's own session log exceeds `max_turns`. Assert the agent
/// IS terminated AND (with max_retries=1) the story IS marked blocked.
#[test]
fn per_session_counting_terminates_over_limit() {
#[tokio::test]
async fn per_session_counting_terminates_over_limit() {
crate::db::ensure_content_store();
crate::crdt_state::init_for_test();
@@ -390,14 +390,14 @@ max_turns = 10
pool.inject_test_agent_with_session(story_id, "coder-1", AgentStatus::Running, "new-sess");
let mut rx = tx.subscribe();
let found = pool.run_watchdog_pass(Some(root));
let found = pool.run_watchdog_pass(Some(root)).await;
assert!(
found >= 1,
"agent over per-session limit must be terminated"
);
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key(story_id, "coder-1");
let agent = agents.get(&key).unwrap();
assert_eq!(agent.status, AgentStatus::Failed);
@@ -423,8 +423,8 @@ max_turns = 10
/// `max_turns`. After session 1: retry_count=1, NOT blocked. After
/// session 2: retry_count=2, NOT blocked. After session 3:
/// retry_count=3 >= max_retries, story IS blocked.
#[test]
fn watchdog_retry_semantic_blocks_after_max_retries() {
#[tokio::test]
async fn watchdog_retry_semantic_blocks_after_max_retries() {
crate::db::ensure_content_store();
let tmp = tempfile::tempdir().unwrap();
@@ -453,7 +453,7 @@ max_turns = 10
write_fake_session_log(root, story_id, "coder-1", "session-1", 12);
let pool = AgentPool::new_test(3001);
pool.inject_test_agent_with_session(story_id, "coder-1", AgentStatus::Running, "session-1");
pool.run_watchdog_pass(Some(root));
pool.run_watchdog_pass(Some(root)).await;
let item = crate::crdt_state::read_item(story_id).expect("story must be in CRDT");
assert_eq!(
@@ -473,7 +473,7 @@ max_turns = 10
write_fake_session_log(root, story_id, "coder-1", "session-2", 12);
let pool = AgentPool::new_test(3001);
pool.inject_test_agent_with_session(story_id, "coder-1", AgentStatus::Running, "session-2");
pool.run_watchdog_pass(Some(root));
pool.run_watchdog_pass(Some(root)).await;
let item = crate::crdt_state::read_item(story_id).expect("story must be in CRDT");
assert_eq!(
@@ -493,7 +493,7 @@ max_turns = 10
write_fake_session_log(root, story_id, "coder-1", "session-3", 12);
let pool = AgentPool::new_test(3001);
pool.inject_test_agent_with_session(story_id, "coder-1", AgentStatus::Running, "session-3");
pool.run_watchdog_pass(Some(root));
pool.run_watchdog_pass(Some(root)).await;
let item = crate::crdt_state::read_item(story_id).expect("story must be in CRDT");
assert_eq!(
@@ -518,8 +518,8 @@ max_turns = 10
/// must not count against the watchdog's turn budget. A session log with
/// 5 tool turns and 30 narration turns reports turns_used == 5, so an
/// agent with max_tool_turns = 10 stays Running.
#[test]
fn watchdog_does_not_count_narration_only_turns() {
#[tokio::test]
async fn watchdog_does_not_count_narration_only_turns() {
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
@@ -542,13 +542,13 @@ max_turns = 200
let pool = AgentPool::new_test(3001);
pool.inject_test_agent_with_session("story_923", "coder-1", AgentStatus::Running, "sess-narr");
let found = pool.run_watchdog_pass(Some(root));
let found = pool.run_watchdog_pass(Some(root)).await;
assert_eq!(
found, 0,
"agent must not be terminated: only 5 tool turns of a 10-turn budget"
);
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key("story_923", "coder-1");
let agent = agents.get(&key).unwrap();
assert_eq!(agent.status, AgentStatus::Running);
@@ -558,8 +558,8 @@ max_turns = 200
/// Story 923: max_tool_turns takes precedence over max_turns when both are
/// set. With max_tool_turns = 3 and max_turns = 200, an agent that has 4
/// tool turns is killed even though total turns (4) is far below max_turns.
#[test]
fn watchdog_max_tool_turns_overrides_max_turns() {
#[tokio::test]
async fn watchdog_max_tool_turns_overrides_max_turns() {
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
@@ -585,13 +585,13 @@ max_turns = 200
);
let mut rx = tx.subscribe();
let found = pool.run_watchdog_pass(Some(root));
let found = pool.run_watchdog_pass(Some(root)).await;
assert!(
found >= 1,
"watchdog must terminate when tool turns exceed max_tool_turns"
);
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key("story_923b", "coder-1");
let agent = agents.get(&key).unwrap();
assert_eq!(agent.status, AgentStatus::Failed);
@@ -4,6 +4,7 @@ use std::path::Path;
mod limits_tests;
mod orphan_tests;
mod reap_tests;
/// Write a fake session log file with `n` tool-using assistant turn entries.
///
@@ -20,18 +20,18 @@ async fn check_orphaned_agents_returns_count_of_orphaned_agents() {
pool.inject_test_agent_with_handle("story_a", "coder", AgentStatus::Running, h1);
pool.inject_test_agent_with_handle("story_b", "coder", AgentStatus::Running, h2);
let found = check_orphaned_agents(&pool.agents);
let found = check_orphaned_agents(&pool.agents).await;
assert_eq!(found, 2, "should detect both orphaned agents");
}
#[test]
fn check_orphaned_agents_returns_zero_when_no_orphans() {
#[tokio::test]
async fn check_orphaned_agents_returns_zero_when_no_orphans() {
let pool = AgentPool::new_test(3001);
// Inject agents in terminal states — not orphaned.
pool.inject_test_agent("story_a", "coder", AgentStatus::Completed);
pool.inject_test_agent("story_b", "qa", AgentStatus::Failed);
let found = check_orphaned_agents(&pool.agents);
let found = check_orphaned_agents(&pool.agents).await;
assert_eq!(
found, 0,
"no orphans should be detected for terminal agents"
@@ -53,10 +53,10 @@ async fn watchdog_detects_orphaned_running_agent() {
pool.inject_test_agent_with_handle("orphan_story", "coder", AgentStatus::Running, handle);
let mut rx = tx.subscribe();
pool.run_watchdog_once();
pool.run_watchdog_once().await;
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key("orphan_story", "coder");
let agent = agents.get(&key).unwrap();
assert_eq!(
@@ -87,13 +87,13 @@ async fn watchdog_orphan_detection_returns_nonzero_enabling_auto_assign() {
// Before watchdog: agent is Running.
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key("orphan_story", "coder");
assert_eq!(agents.get(&key).unwrap().status, AgentStatus::Running);
}
// Run watchdog pass — should return 1 (orphan found).
let found = check_orphaned_agents(&pool.agents);
let found = check_orphaned_agents(&pool.agents).await;
assert_eq!(
found, 1,
"watchdog must return 1 for a single orphaned agent"
@@ -101,7 +101,7 @@ async fn watchdog_orphan_detection_returns_nonzero_enabling_auto_assign() {
// After watchdog: agent is Failed.
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = composite_key("orphan_story", "coder");
assert_eq!(
agents.get(&key).unwrap().status,
@@ -0,0 +1,238 @@
//! Regression tests for the reap pass (bug 1198): a Failed pool entry with
//! no live process must be respawned by the next watchdog pass, with its
//! story's retry count bumped, and must stop appearing in list_agents.
use super::super::super::super::{AgentPool, composite_key};
use super::{write_fake_session_log, write_project_config};
use crate::agents::AgentStatus;
/// AC1 + AC4: a Failed entry with no live process (simulating e.g. the
/// inactivity watchdog kill landing in spawn.rs's generic Err arm) is
/// respawned by the very next `run_watchdog_pass` — no manual
/// stop_agent/start_agent needed.
#[tokio::test]
async fn reap_respawns_stale_failed_agent() {
crate::db::ensure_content_store();
crate::crdt_state::init_for_test();
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
write_project_config(
root,
r#"
[[agent]]
name = "coder-1"
runtime = "claude-code"
"#,
);
let story_id = "1198_story_stuck";
crate::db::write_content(
crate::db::ContentKey::Story(story_id),
"---\nname: Stuck Story\n---\n",
);
crate::crdt_state::write_item_str(story_id, "2_current", Some("Stuck Story"), None, None, None);
// Inject a Failed entry with no task_handle — exactly what spawn.rs's
// generic Err arm leaves behind after e.g. an inactivity-watchdog kill.
let pool = AgentPool::new_test(3001);
pool.inject_test_agent(story_id, "coder-1", AgentStatus::Failed);
let found = pool.run_watchdog_pass(Some(root)).await;
assert!(found >= 1, "reap should count the stale Failed entry");
// A fresh entry must exist for the same story — the agent respawned.
let agents = pool.agents.try_lock().unwrap();
let key = composite_key(story_id, "coder-1");
let agent = agents.get(&key).expect("agent must have been respawned");
assert_ne!(
agent.status,
AgentStatus::Failed,
"respawned entry must not still be Failed"
);
}
/// AC2: reaping a stale Failed entry bumps the story's retry count via the
/// existing should_block_story path.
#[tokio::test]
async fn reap_increments_retry_count() {
crate::db::ensure_content_store();
crate::crdt_state::init_for_test();
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
write_project_config(
root,
r#"
max_retries = 5
[[agent]]
name = "coder-1"
runtime = "claude-code"
"#,
);
let story_id = "1198_story_retry";
crate::db::write_content(
crate::db::ContentKey::Story(story_id),
"---\nname: Retry Story\n---\n",
);
crate::crdt_state::write_item_str(story_id, "2_current", Some("Retry Story"), None, None, None);
let pool = AgentPool::new_test(3001);
pool.inject_test_agent(story_id, "coder-1", AgentStatus::Failed);
pool.run_watchdog_pass(Some(root)).await;
let item = crate::crdt_state::read_item(story_id).expect("story must be in CRDT");
assert_eq!(
item.retry_count(),
1,
"reaping a stale Failed entry must bump retry_count exactly once"
);
}
/// AC2: exhausting max_retries via a reap blocks the story via the existing
/// should_block_story path (same mechanism as the limits watchdog).
#[tokio::test]
async fn reap_blocks_story_after_max_retries() {
crate::db::ensure_content_store();
crate::crdt_state::init_for_test();
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
write_project_config(
root,
r#"
max_retries = 1
[[agent]]
name = "coder-1"
runtime = "claude-code"
"#,
);
let story_id = "1198_story_block";
crate::db::write_content(
crate::db::ContentKey::Story(story_id),
"---\nname: Block Story\n---\n",
);
crate::crdt_state::write_item_str(story_id, "2_current", Some("Block Story"), None, None, None);
let pool = AgentPool::new_test(3001);
pool.inject_test_agent(story_id, "coder-1", AgentStatus::Failed);
pool.run_watchdog_pass(Some(root)).await;
let item = crate::crdt_state::read_item(story_id).expect("story must be in CRDT");
assert_eq!(
item.stage().dir_name(),
"blocked",
"story must be blocked after exhausting max_retries=1 via reap"
);
}
/// AC3: list_agents never shows a Failed entry after the next watchdog pass
/// reaps it — regardless of whether the story blocks or respawns.
#[tokio::test]
async fn reap_removes_failed_entry_from_list_agents() {
crate::db::ensure_content_store();
crate::crdt_state::init_for_test();
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
write_project_config(
root,
r#"
max_retries = 1
[[agent]]
name = "coder-1"
runtime = "claude-code"
"#,
);
let story_id = "1198_story_listing";
crate::db::write_content(
crate::db::ContentKey::Story(story_id),
"---\nname: Listing Story\n---\n",
);
crate::crdt_state::write_item_str(
story_id,
"2_current",
Some("Listing Story"),
None,
None,
None,
);
let pool = AgentPool::new_test(3001);
pool.inject_test_agent(story_id, "coder-1", AgentStatus::Failed);
pool.run_watchdog_pass(Some(root)).await;
let listed = pool.list_agents().await.unwrap();
assert!(
!listed
.iter()
.any(|a| a.story_id == story_id && a.status == AgentStatus::Failed),
"list_agents must not show a Failed entry for '{story_id}' after the next watchdog pass"
);
}
/// The limits-termination path (turn/budget overrun) must not have its
/// retry_count double-bumped by the reap pass running in the same
/// `run_watchdog_pass` call.
#[tokio::test]
async fn reap_does_not_double_bump_limits_terminated_agent() {
crate::db::ensure_content_store();
crate::crdt_state::init_for_test();
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path();
write_project_config(
root,
r#"
max_retries = 5
[[agent]]
name = "coder-1"
runtime = "claude-code"
max_turns = 10
"#,
);
let story_id = "1198_story_no_double_bump";
crate::db::write_content(
crate::db::ContentKey::Story(story_id),
"---\nname: No Double Bump\n---\n",
);
crate::crdt_state::write_item_str(
story_id,
"2_current",
Some("No Double Bump"),
None,
None,
None,
);
write_fake_session_log(root, story_id, "coder-1", "sess-overrun", 12);
let pool = AgentPool::new_test(3001);
pool.inject_test_agent_with_session(story_id, "coder-1", AgentStatus::Running, "sess-overrun");
pool.run_watchdog_pass(Some(root)).await;
let item = crate::crdt_state::read_item(story_id).expect("story must be in CRDT");
assert_eq!(
item.retry_count(),
1,
"a single limit-termination pass must bump retry_count by exactly 1, \
not twice (once from the limits branch, once from reap)"
);
}
+7 -5
View File
@@ -19,8 +19,8 @@ mod test_helpers;
use crate::io::watcher::WatcherEvent;
use crate::service::status::StatusBroadcaster;
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use tokio::sync::broadcast;
use std::sync::Arc;
use tokio::sync::{Mutex, broadcast};
// Bring pool-internal types into pool's namespace so that sub-modules
// (auto_assign, pipeline, etc.) can access them via `use super::...`.
@@ -87,10 +87,12 @@ impl AgentPool {
_ => continue,
};
let key = composite_key(&story_id, &agent_name);
if let Ok(mut agents) = agents_clone.lock()
&& let Some(agent) = agents.get_mut(&key)
{
agent.throttled = Some(crate::agents::AgentExecution::Throttled { until });
let mut agents = agents_clone.lock().await;
if let Some(agent) = agents.get_mut(&key) {
agent.throttled =
Some(crate::agents::AgentExecution::Throttled { until });
}
}
let _ = watcher_tx_clone.send(WatcherEvent::AgentStateChanged);
}
@@ -3,7 +3,7 @@
use std::collections::HashMap;
use std::path::PathBuf;
use std::sync::{Arc, Mutex};
use std::sync::Arc;
use tokio::sync::broadcast;
@@ -16,7 +16,7 @@ use std::path::Path;
/// type cycle between `start_agent` and `run_server_owned_completion`.
#[allow(clippy::too_many_arguments)]
pub(crate) fn spawn_pipeline_advance(
agents: Arc<Mutex<HashMap<String, StoryAgent>>>,
agents: Arc<tokio::sync::Mutex<HashMap<String, StoryAgent>>>,
port: u16,
story_id: &str,
agent_name: &str,
@@ -694,7 +694,7 @@ impl AgentPool {
if let Err(e) = crate::agents::lifecycle::move_story_to_done(story_id) {
slog_error!("[pipeline] Failed to move '{story_id}' to done: {e}");
}
self.remove_agents_for_story(story_id);
self.remove_agents_for_story(story_id).await;
crate::crdt_state::delete_merge_job(story_id);
// TODO: Re-enable worktree cleanup once we have persistent agent logs.
// Removing worktrees destroys evidence needed to debug empty-commit agents.
@@ -104,7 +104,7 @@ async fn mergemaster_blocks_and_sends_story_blocked_when_no_commits_ahead() {
);
// No mergemaster agent should have been started.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let mergemaster_started = agents
.values()
.any(|a| a.agent_name.contains("mergemaster"));
@@ -162,7 +162,7 @@ stage = "qa"
// Verify that 293 cannot get a QA agent right now (QA is busy).
{
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
assert!(
!is_agent_free(&agents, "qa"),
"qa should be busy on story 292"
@@ -172,7 +172,7 @@ stage = "qa"
// Simulate QA completing on story 292: remove the agent from the pool
// (as run_server_owned_completion does) then run pipeline advance.
{
let mut agents = pool.agents.lock().unwrap();
let mut agents = pool.agents.try_lock().unwrap();
agents.remove(&composite_key("292_story_first", "qa"));
}
@@ -193,7 +193,7 @@ stage = "qa"
.await;
// After pipeline advance, auto_assign should have started QA on story 293.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let qa_on_293 = agents.values().any(|a| {
a.agent_name == "qa" && matches!(a.status, AgentStatus::Pending | AgentStatus::Running)
});
@@ -278,7 +278,7 @@ async fn stale_mergemaster_advance_for_done_story_is_noop() {
.await;
// No agents should have been started.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
assert!(
agents.is_empty(),
"No agents should be started for a stale advance on a done story. \
@@ -871,7 +871,7 @@ stage = "coder"
.await;
// The coder must be re-spawned — Pending or Running.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let coder_restarted = agents.values().any(|a| {
a.agent_name == "coder-1" && matches!(a.status, AgentStatus::Pending | AgentStatus::Running)
});
@@ -957,7 +957,7 @@ stage = "coder"
.await;
// The recovery respawn must have been issued — coder-1 should be Pending/Running.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let coder_restarted = agents.values().any(|a| {
a.agent_name == "coder-1" && matches!(a.status, AgentStatus::Pending | AgentStatus::Running)
});
@@ -1328,7 +1328,7 @@ async fn coder_completion_with_test_evidence_and_zero_commits_does_not_advance()
);
// No QA or merge agent should have been started.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let qa_or_merge_started = agents
.values()
.any(|a| a.agent_name.contains("qa") || a.agent_name.contains("merge"));
@@ -28,7 +28,7 @@ impl AgentPool {
// Verify agent exists, is Running, and grab its worktree path.
let worktree_path = {
let agents = self.agents.lock().map_err(|e| e.to_string())?;
let agents = self.agents.lock().await;
let agent = agents
.get(&key)
.ok_or_else(|| format!("No agent '{agent_name}' for story '{story_id}'"))?;
@@ -82,7 +82,7 @@ impl AgentPool {
merge_failure_reported_for_advance,
session_id_for_advance,
) = {
let mut agents = self.agents.lock().map_err(|e| e.to_string())?;
let mut agents = self.agents.lock().await;
let agent = agents.get_mut(&key).ok_or_else(|| {
format!("Agent '{agent_name}' for story '{story_id}' disappeared during gate check")
})?;
@@ -2,7 +2,8 @@
use crate::io::watcher::WatcherEvent;
use crate::slog;
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::sync::Arc;
use tokio::sync::Mutex;
use tokio::sync::broadcast;
use super::super::super::super::{AgentEvent, CompletionReport, PipelineStage, pipeline_stage};
@@ -45,10 +46,7 @@ pub(in crate::agents::pool) async fn run_server_owned_completion(
// Guard: skip if completion was already recorded (legacy path).
{
let lock = match agents.lock() {
Ok(a) => a,
Err(_) => return,
};
let lock = agents.lock().await;
match lock.get(&key) {
Some(agent) if agent.completion.is_some() => {
slog!(
@@ -64,10 +62,7 @@ pub(in crate::agents::pool) async fn run_server_owned_completion(
// Get worktree path for running gates.
let worktree_path = {
let lock = match agents.lock() {
Ok(a) => a,
Err(_) => return,
};
let lock = agents.lock().await;
lock.get(&key)
.and_then(|a| a.worktree_info.as_ref().map(|wt| wt.path.clone()))
};
@@ -192,10 +187,7 @@ pub(in crate::agents::pool) async fn run_server_owned_completion(
// Store completion report, extract data for pipeline advance, then
// remove the entry so completed agents never appear in list_agents.
let (tx, project_root_for_advance, wt_path_for_advance, merge_failure_reported_for_advance) = {
let mut lock = match agents.lock() {
Ok(a) => a,
Err(_) => return,
};
let mut lock = agents.lock().await;
let agent = match lock.get_mut(&key) {
Some(a) => a,
None => return,
@@ -108,7 +108,7 @@ async fn server_owned_completion_skips_when_already_completed() {
);
// Subscribe before calling so we can check if Done event was emitted.
let mut rx = pool.subscribe("s10", "coder-1").unwrap();
let mut rx = pool.subscribe("s10", "coder-1").await.unwrap();
run_server_owned_completion(
&pool.agents,
@@ -121,7 +121,7 @@ async fn server_owned_completion_skips_when_already_completed() {
.await;
// Status should remain Completed (unchanged) — no gate re-run.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = super::super::super::composite_key("s10", "coder-1");
let agent = agents.get(&key).unwrap();
assert_eq!(agent.status, AgentStatus::Completed);
@@ -147,7 +147,7 @@ async fn server_owned_completion_runs_gates_on_clean_worktree() {
let pool = AgentPool::new_test(3001);
pool.inject_test_agent_with_path("s11", "coder-1", AgentStatus::Running, repo.to_path_buf());
let mut rx = pool.subscribe("s11", "coder-1").unwrap();
let mut rx = pool.subscribe("s11", "coder-1").await.unwrap();
run_server_owned_completion(
&pool.agents,
@@ -160,7 +160,7 @@ async fn server_owned_completion_runs_gates_on_clean_worktree() {
.await;
// Agent entry should be removed from the map after completion.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = super::super::super::composite_key("s11", "coder-1");
assert!(
agents.get(&key).is_none(),
@@ -192,7 +192,7 @@ async fn server_owned_completion_fails_on_dirty_worktree() {
let pool = AgentPool::new_test(3001);
pool.inject_test_agent_with_path("s12", "coder-1", AgentStatus::Running, repo.to_path_buf());
let mut rx = pool.subscribe("s12", "coder-1").unwrap();
let mut rx = pool.subscribe("s12", "coder-1").await.unwrap();
run_server_owned_completion(
&pool.agents,
@@ -205,7 +205,7 @@ async fn server_owned_completion_fails_on_dirty_worktree() {
.await;
// Agent entry should be removed from the map after completion (even on failure).
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = super::super::super::composite_key("s12", "coder-1");
assert!(
agents.get(&key).is_none(),
@@ -307,7 +307,7 @@ async fn server_owned_completion_is_noop_for_mergemaster() {
// The agent entry should remain in the pool (lifecycle cleanup is the
// caller's responsibility, not run_server_owned_completion's).
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
let key = super::super::super::composite_key("99_story_merge445", "mergemaster");
assert!(
agents.get(&key).is_some(),
@@ -361,7 +361,7 @@ async fn server_owned_completion_preserves_dirty_worktree_with_committed_work()
let pool = AgentPool::new_test(3001);
pool.inject_test_agent_with_path("645_test", "coder-1", AgentStatus::Running, wt_path.clone());
let mut rx = pool.subscribe("645_test", "coder-1").unwrap();
let mut rx = pool.subscribe("645_test", "coder-1").await.unwrap();
run_server_owned_completion(
&pool.agents,
@@ -34,35 +34,29 @@ impl AgentPool {
/// If the agent was already removed from the pool (race: `remove_agents_for_story`
/// ran first) this is a no-op; the `ContentKey::MergeSuccess` DB key written
/// by the caller acts as the authoritative fallback in that case.
pub fn set_merge_success_reported(&self, story_id: &str) {
match self.agents.lock() {
Ok(mut lock) => {
let found = lock.iter_mut().find(|(key, agent)| {
let key_story_id = key
.rsplit_once(':')
.map(|(sid, _)| sid)
.unwrap_or(key.as_str());
key_story_id == story_id
&& pipeline_stage(&agent.agent_name) == PipelineStage::Mergemaster
});
match found {
Some((_, agent)) => {
agent.merge_success_reported = true;
slog!(
"[pipeline] Merge success flag set for '{story_id}:{}'",
agent.agent_name
);
}
None => {
slog!(
"[pipeline] set_merge_success_reported: no running mergemaster \
for '{story_id}' DB key is the authoritative fallback"
);
}
}
pub async fn set_merge_success_reported(&self, story_id: &str) {
let mut lock = self.agents.lock().await;
let found = lock.iter_mut().find(|(key, agent)| {
let key_story_id = key
.rsplit_once(':')
.map(|(sid, _)| sid)
.unwrap_or(key.as_str());
key_story_id == story_id
&& pipeline_stage(&agent.agent_name) == PipelineStage::Mergemaster
});
match found {
Some((_, agent)) => {
agent.merge_success_reported = true;
slog!(
"[pipeline] Merge success flag set for '{story_id}:{}'",
agent.agent_name
);
}
Err(e) => {
slog_error!("[pipeline] set_merge_success_reported: could not lock agents: {e}");
None => {
slog!(
"[pipeline] set_merge_success_reported: no running mergemaster \
for '{story_id}' DB key is the authoritative fallback"
);
}
}
}
@@ -74,35 +68,29 @@ impl AgentPool {
/// that `run_pipeline_advance` can block advancement to `5_done/` even when
/// the server-owned gate check returns `gates_passed=true` (those gates run
/// in the feature-branch worktree, not on master).
pub fn set_merge_failure_reported(&self, story_id: &str) {
match self.agents.lock() {
Ok(mut lock) => {
let found = lock.iter_mut().find(|(key, agent)| {
let key_story_id = key
.rsplit_once(':')
.map(|(sid, _)| sid)
.unwrap_or(key.as_str());
key_story_id == story_id
&& pipeline_stage(&agent.agent_name) == PipelineStage::Mergemaster
});
match found {
Some((_, agent)) => {
agent.merge_failure_reported = true;
slog!(
"[pipeline] Merge failure flag set for '{story_id}:{}'",
agent.agent_name
);
}
None => {
slog_warn!(
"[pipeline] set_merge_failure_reported: no running mergemaster found \
for story '{story_id}' flag not set"
);
}
}
pub async fn set_merge_failure_reported(&self, story_id: &str) {
let mut lock = self.agents.lock().await;
let found = lock.iter_mut().find(|(key, agent)| {
let key_story_id = key
.rsplit_once(':')
.map(|(sid, _)| sid)
.unwrap_or(key.as_str());
key_story_id == story_id
&& pipeline_stage(&agent.agent_name) == PipelineStage::Mergemaster
});
match found {
Some((_, agent)) => {
agent.merge_failure_reported = true;
slog!(
"[pipeline] Merge failure flag set for '{story_id}:{}'",
agent.agent_name
);
}
Err(e) => {
slog_error!("[pipeline] set_merge_failure_reported: could not lock agents: {e}");
None => {
slog_warn!(
"[pipeline] set_merge_failure_reported: no running mergemaster found \
for story '{story_id}' flag not set"
);
}
}
}
+46 -13
View File
@@ -32,7 +32,19 @@ impl AgentPool {
/// Called at the top of [`start_merge_agent_work`] to unblock retries,
/// and also by the periodic background reaper in the tick loop so stale
/// entries are cleaned up even when no new merge is triggered.
///
/// A job's `server_start` round-trips through JSON text (see
/// [`encode_server_start_time`]/[`decode_server_start_time`]), and
/// `serde_json`'s float parser is not guaranteed bit-exact for
/// high-precision Unix timestamps — it can decode a value a couple of
/// ULPs below the original. Comparing with a bare `<` against a
/// freshly-read `current_boot` would then occasionally treat a job
/// written by *this very server instance* as belonging to a previous
/// one. Real server restarts are always seconds apart at minimum, so a
/// generous tolerance absorbs that noise without weakening genuine
/// stale-boot detection.
pub(crate) fn reap_stale_merge_jobs(&self) {
const STALE_TOLERANCE_SECS: f64 = 1.0;
if let Some(jobs) = crate::crdt_state::read_all_merge_jobs() {
let current_boot = server_start_time();
for job in jobs {
@@ -40,7 +52,7 @@ impl AgentPool {
continue;
}
let stale = match decode_server_start_time(job.error.as_deref()) {
Some(t) => t < current_boot,
Some(t) => t < current_boot - STALE_TOLERANCE_SECS,
None => true, // Legacy (pid-encoded) or malformed: stale
};
if stale {
@@ -205,7 +217,7 @@ impl AgentPool {
// retry_count=1 so maybe_inject_gate_failure injects gate output
// into --append-system-prompt on the fixup spawn.
// transition_to_merge_failure also writes ContentKey::GateOutput.
let display = kind.display_reason();
let display = crate::service::merge::summarize_merge_failure_kind(&kind);
let _ =
crate::agents::lifecycle::transition_to_merge_failure(sid.as_str(), kind);
match crate::agents::lifecycle::move_story_to_stage(&sid, "current") {
@@ -246,7 +258,7 @@ impl AgentPool {
// Transition through the state machine (Merge → MergeFailure).
// Only send the notification when the stage actually changed; if the
// story was already in MergeFailure (self-loop), suppress the duplicate.
let display = kind.display_reason();
let display = crate::service::merge::summarize_merge_failure_kind(&kind);
let should_notify = match crate::agents::lifecycle::transition_to_merge_failure(
sid.as_str(),
kind,
@@ -283,7 +295,7 @@ impl AgentPool {
&& let Ok(ref r) = report
&& r.story_archived
{
pool.set_merge_success_reported(&sid);
pool.set_merge_success_reported(&sid).await;
crate::db::write_content(crate::db::ContentKey::MergeSuccess(&sid), "1");
}
@@ -340,24 +352,44 @@ impl AgentPool {
merge_result,
crate::agents::merge::MergeResult::Success { .. }
) {
let report_path = crate::service::merge::io::write_merge_report(
project_root,
story_id,
merge_result.output(),
);
return Ok(crate::agents::merge::MergeReport {
story_id: story_id.to_string(),
result: merge_result,
worktree_cleaned_up: false,
story_archived: false,
report_path,
});
}
let story_archived = crate::agents::lifecycle::move_story_to_done(story_id).is_ok();
if story_archived {
self.remove_agents_for_story(story_id);
}
let report_path = crate::service::merge::io::write_merge_report(
project_root,
story_id,
merge_result.output(),
);
let worktree_cleaned_up = if wt_path.exists() {
let config = crate::config::ProjectConfig::load(project_root).unwrap_or_default();
worktree::remove_worktree_by_story_id(project_root, story_id, &config)
.await
.is_ok()
let story_archived = crate::agents::lifecycle::move_story_to_done(story_id).is_ok();
// Story 1178: only delete the feature branch once the state transition
// to Done is confirmed. Deleting it unconditionally here meant a
// successful squash merge whose CRDT transition failed (e.g. the story
// was in a stage `move_story_to_done` didn't yet handle) would still
// lose its feature branch, making the failure unrecoverable — the
// story couldn't be retried because the branch it needed was gone.
let worktree_cleaned_up = if story_archived {
self.remove_agents_for_story(story_id).await;
if wt_path.exists() {
let config = crate::config::ProjectConfig::load(project_root).unwrap_or_default();
worktree::remove_worktree_by_story_id(project_root, story_id, &config)
.await
.is_ok()
} else {
false
}
} else {
false
};
@@ -369,6 +401,7 @@ impl AgentPool {
result: merge_result,
worktree_cleaned_up,
story_archived,
report_path,
})
}
}
@@ -29,6 +29,7 @@ impl AgentPool {
},
worktree_cleaned_up: false,
story_archived: false,
report_path: None,
});
(crate::agents::merge::MergeJobStatus::Completed(report), 0.0)
}
@@ -41,6 +42,7 @@ impl AgentPool {
story_id: story_id.to_string(),
status,
server_start_time,
started_at: view.started_at,
})
}
}
+109 -2
View File
@@ -113,6 +113,22 @@ async fn stale_running_merge_job_is_cleared_and_retry_succeeds() {
result.is_ok(),
"start_merge_agent_work must succeed after stale Running job is cleared; got: {result:?}"
);
// start_merge_agent_work spawns the actual pipeline as a background
// tokio task and returns immediately. Wait for it to reach a terminal
// state before the test ends: otherwise the task keeps running after
// `_serial` is released and can still be touching CRDT state
// (write_merge_job / delete_merge_job) while the next merge-pipeline
// test has already called init_for_test(), corrupting that test's
// thread-local state.
loop {
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
if let Some(job) = pool.get_merge_status("77_story_stale")
&& !matches!(job.status, MergeJobStatus::Running)
{
break;
}
}
}
// ── story 852: periodic background reaper ────────────────────────────────
@@ -149,7 +165,7 @@ async fn reap_stale_merge_jobs_removes_old_running_entry_without_merge() {
);
// No agents must have been spawned (no merge was triggered).
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
assert!(
agents.is_empty(),
"reap must not spawn any agents; got {} agent(s)",
@@ -393,6 +409,97 @@ async fn merge_agent_work_succeeds_on_clean_branch() {
}
}
/// Regression test (story 1178, AC3): when the squash merge itself succeeds
/// but the CRDT state transition to Done fails (here: no CRDT entry exists
/// for the story, so `move_story_to_done` errors and `story_archived` is
/// false), the feature branch must NOT be deleted — otherwise a retry has
/// nothing to merge from.
#[tokio::test]
async fn merge_success_without_story_archived_keeps_feature_branch() {
let _serial = serial_test_lock();
use std::fs;
use tempfile::tempdir;
crate::crdt_state::init_for_test();
let tmp = tempdir().unwrap();
let repo = tmp.path();
init_git_repo(repo);
let branch = "feature/story-1178_branch_kept";
Command::new("git")
.args(["checkout", "-b", branch])
.current_dir(repo)
.output()
.unwrap();
fs::write(repo.join("feature.txt"), "feature content").unwrap();
Command::new("git")
.args(["add", "."])
.current_dir(repo)
.output()
.unwrap();
Command::new("git")
.args(["commit", "-m", "add feature"])
.current_dir(repo)
.output()
.unwrap();
Command::new("git")
.args(["checkout", "master"])
.current_dir(repo)
.output()
.unwrap();
let merge_dir = repo.join(".huskies/work/4_merge");
fs::create_dir_all(&merge_dir).unwrap();
fs::write(
merge_dir.join("1178_branch_kept.md"),
"---\nname: Branch Kept Test\n---\n",
)
.unwrap();
Command::new("git")
.args(["add", "."])
.current_dir(repo)
.output()
.unwrap();
Command::new("git")
.args(["commit", "-m", "add story in merge"])
.current_dir(repo)
.output()
.unwrap();
let pool = Arc::new(AgentPool::new_test(3001));
// Note: no CRDT entry is written for this story, so `move_story_to_done`
// will fail with NotFound — `story_archived` will be false regardless of
// git merge outcome. That is exactly the scenario this test protects.
let job = run_merge_to_completion(&pool, repo, "1178_branch_kept").await;
let MergeJobStatus::Completed(report) = &job.status else {
panic!("expected a completed job, got: {:?}", job.status);
};
if matches!(
report.result,
crate::agents::merge::MergeResult::Success { .. }
) {
assert!(
!report.story_archived,
"story_archived should be false: no CRDT entry exists for this story"
);
assert!(
!report.worktree_cleaned_up,
"worktree/branch must not be cleaned up when story_archived is false"
);
let branch_check = Command::new("git")
.args(["rev-parse", "--verify", branch])
.current_dir(repo)
.output()
.unwrap();
assert!(
branch_check.status.success(),
"feature branch '{branch}' must still exist after a merge whose state \
transition to Done failed, so a retry stays possible"
);
}
}
// ── quality gate ordering test ────────────────────────────────
/// Regression test for bug 142: quality gates must run BEFORE the fast-forward
@@ -811,7 +918,7 @@ async fn server_side_merge_happy_path_advances_to_done() {
}
// Verify no LLM agent was spawned.
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.try_lock().unwrap();
assert!(
agents.is_empty(),
"no LLM agents should be spawned for deterministic merge; pool has {} agents",
@@ -2,10 +2,10 @@
/// Wall-clock time captured the first time this server process touches the
/// merge subsystem. Used to detect merge_jobs left over from a previous
/// server instance: a re-exec on `rebuild_and_restart` keeps the same PID,
/// so PID alone cannot distinguish "current" vs "previous" server. This
/// timestamp is fresh per-process (the static is reset by execve) and is
/// the source of truth for stale-merge detection.
/// server instance: PIDs can collide across restarts (PID 1 in a container
/// is always the server), so PID alone cannot distinguish "current" vs
/// "previous" server. This timestamp is fresh per-process and is the source
/// of truth for stale-merge detection.
static SERVER_START_TIME: std::sync::OnceLock<f64> = std::sync::OnceLock::new();
/// Return this server process's start time (lazily captured on first call).
+13 -17
View File
@@ -25,11 +25,9 @@ impl AgentPool {
/// continuing to run after the server exits. Collects each agent's worktree
/// path, then SIGKILLs every process running inside that path and verifies
/// termination before returning.
pub fn kill_all_children(&self) {
pub async fn kill_all_children(&self) {
let worktree_paths: Vec<(String, std::path::PathBuf)> = {
let Ok(agents) = self.agents.lock() else {
return;
};
let agents = self.agents.lock().await;
agents
.iter()
.filter_map(|(key, agent)| {
@@ -69,11 +67,9 @@ impl AgentPool {
/// Fallback used by `stop_agent` when no worktree path is recorded for the
/// agent. Also the primary kill path for any caller that has only a composite
/// key and not a worktree path directly.
pub(super) fn kill_child_for_key(&self, key: &str) {
pub(super) async fn kill_child_for_key(&self, key: &str) {
let worktree_path = {
let Ok(agents) = self.agents.lock() else {
return;
};
let agents = self.agents.lock().await;
agents
.get(key)
.and_then(|a| a.worktree_info.as_ref().map(|wt| wt.path.clone()))
@@ -124,18 +120,18 @@ mod tests {
.unwrap_or(false)
}
#[test]
fn kill_all_children_is_safe_on_empty_pool() {
#[tokio::test]
async fn kill_all_children_is_safe_on_empty_pool() {
let pool = AgentPool::new_test(3001);
pool.kill_all_children(); // must not panic
pool.kill_all_children().await; // must not panic
}
/// AC 4 — `kill_child_for_key` SIGKILLs the single agent's process and
/// verifies it is gone within 2 s. The sleeper has the worktree path in
/// its argv[0] so `pgrep -f` can locate it, mirroring how claude-code is
/// launched with `--directory <worktree>` in production.
#[test]
fn kill_child_for_key_kills_real_process() {
#[tokio::test]
async fn kill_child_for_key_kills_real_process() {
use std::os::unix::process::CommandExt;
let pool = AgentPool::new_test(3002);
@@ -165,7 +161,7 @@ mod tests {
"sleeper pid {pid} should be running before kill_child_for_key"
);
pool.kill_child_for_key("story-1090-kill:coder");
pool.kill_child_for_key("story-1090-kill:coder").await;
let _ = child.wait(); // reap zombie so ps -p returns false
assert!(
@@ -176,8 +172,8 @@ mod tests {
/// AC 5 — `kill_all_children` SIGKILLs all agents' processes. Two agents
/// with distinct worktree paths are injected; both must be gone after the call.
#[test]
fn kill_all_children_kills_multiple_real_processes() {
#[tokio::test]
async fn kill_all_children_kills_multiple_real_processes() {
use std::os::unix::process::CommandExt;
let pool = AgentPool::new_test(3003);
@@ -213,7 +209,7 @@ mod tests {
);
}
pool.kill_all_children();
pool.kill_all_children().await;
for (pid, child, _tmp) in &mut sleepers {
let _ = child.wait(); // reap zombie
+45 -16
View File
@@ -10,12 +10,12 @@ use super::types::{agent_info_from_entry, composite_key};
impl AgentPool {
/// Return the names of configured agents for `stage` that are not currently
/// running or pending.
pub fn available_agents_for_stage(
pub async fn available_agents_for_stage(
&self,
config: &ProjectConfig,
stage: &PipelineStage,
) -> Result<Vec<String>, String> {
let agents = self.agents.lock().map_err(|e| e.to_string())?;
let agents = self.agents.lock().await;
Ok(config
.agent
.iter()
@@ -44,8 +44,8 @@ impl AgentPool {
}
/// List all agents with their status.
pub fn list_agents(&self) -> Result<Vec<AgentInfo>, String> {
let agents = self.agents.lock().map_err(|e| e.to_string())?;
pub async fn list_agents(&self) -> Result<Vec<AgentInfo>, String> {
let agents = self.agents.lock().await;
Ok(agents
.iter()
.map(|(key, agent)| {
@@ -59,14 +59,35 @@ impl AgentPool {
.collect())
}
/// Best-effort agent list for sync callers (chat commands, rendering).
///
/// Uses `try_lock()` so it never blocks the thread. Returns an empty list
/// if the lock is currently held — callers that refresh periodically (htop,
/// pipeline board) tolerate this gracefully.
pub fn list_agents_nonblocking(&self) -> Vec<AgentInfo> {
let Ok(agents) = self.agents.try_lock() else {
return Vec::new();
};
agents
.iter()
.map(|(key, agent)| {
let story_id = key
.rsplit_once(':')
.map(|(sid, _)| sid.to_string())
.unwrap_or_else(|| key.clone());
agent_info_from_entry(&story_id, agent)
})
.collect()
}
/// Subscribe to events for a story agent.
pub fn subscribe(
pub async fn subscribe(
&self,
story_id: &str,
agent_name: &str,
) -> Result<broadcast::Receiver<AgentEvent>, String> {
let key = composite_key(story_id, agent_name);
let agents = self.agents.lock().map_err(|e| e.to_string())?;
let agents = self.agents.lock().await;
let agent = agents
.get(&key)
.ok_or_else(|| format!("No agent '{agent_name}' for story '{story_id}'"))?;
@@ -74,13 +95,13 @@ impl AgentPool {
}
/// Drain accumulated events for polling. Returns all events since the last drain.
pub fn drain_events(
pub async fn drain_events(
&self,
story_id: &str,
agent_name: &str,
) -> Result<Vec<AgentEvent>, String> {
let key = composite_key(story_id, agent_name);
let agents = self.agents.lock().map_err(|e| e.to_string())?;
let agents = self.agents.lock().await;
let agent = agents
.get(&key)
.ok_or_else(|| format!("No agent '{agent_name}' for story '{story_id}'"))?;
@@ -91,9 +112,13 @@ impl AgentPool {
/// Get the log session ID and project root for an agent, if available.
///
/// Used by MCP tools to find the persistent log file for a completed agent.
pub fn get_log_info(&self, story_id: &str, agent_name: &str) -> Option<(String, PathBuf)> {
pub async fn get_log_info(
&self,
story_id: &str,
agent_name: &str,
) -> Option<(String, PathBuf)> {
let key = composite_key(story_id, agent_name);
let agents = self.agents.lock().ok()?;
let agents = self.agents.lock().await;
let agent = agents.get(&key)?;
let session_id = agent.log_session_id.clone()?;
let project_root = agent.project_root.clone()?;
@@ -111,8 +136,8 @@ mod tests {
ProjectConfig::parse(toml_str).unwrap()
}
#[test]
fn available_agents_for_stage_returns_idle_agents() {
#[tokio::test]
async fn available_agents_for_stage_returns_idle_agents() {
let config = make_config(
r#"
[[agent]]
@@ -133,17 +158,19 @@ stage = "qa"
let available = pool
.available_agents_for_stage(&config, &PipelineStage::Coder)
.await
.unwrap();
assert_eq!(available, vec!["coder-2"]);
let available_qa = pool
.available_agents_for_stage(&config, &PipelineStage::Qa)
.await
.unwrap();
assert_eq!(available_qa, vec!["qa"]);
}
#[test]
fn available_agents_for_stage_returns_empty_when_all_busy() {
#[tokio::test]
async fn available_agents_for_stage_returns_empty_when_all_busy() {
let config = make_config(
r#"
[[agent]]
@@ -156,12 +183,13 @@ stage = "coder"
let available = pool
.available_agents_for_stage(&config, &PipelineStage::Coder)
.await
.unwrap();
assert!(available.is_empty());
}
#[test]
fn available_agents_for_stage_ignores_completed_agents() {
#[tokio::test]
async fn available_agents_for_stage_ignores_completed_agents() {
let config = make_config(
r#"
[[agent]]
@@ -174,6 +202,7 @@ stage = "coder"
let available = pool
.available_agents_for_stage(&config, &PipelineStage::Coder)
.await
.unwrap();
assert_eq!(available, vec!["coder-1"]);
}
+27 -24
View File
@@ -3,8 +3,10 @@
use crate::agent_log::AgentLogWriter;
use crate::config::ProjectConfig;
use crate::slog_error;
use std::future::Future;
use std::path::Path;
use std::sync::{Arc, Mutex};
use std::pin::Pin;
use std::sync::Arc;
use tokio::sync::broadcast;
use super::super::runtime::{
@@ -38,48 +40,48 @@ impl AgentPool {
/// `resume_context` (if any) is sent as the new message. This lets
/// the agent re-enter the previous conversation without re-reading
/// CLAUDE.md and README, satisfying story 543.
pub async fn start_agent(
&self,
project_root: &Path,
story_id: &str,
agent_name: Option<&str>,
resume_context: Option<&str>,
pub fn start_agent<'a>(
&'a self,
project_root: &'a Path,
story_id: &'a str,
agent_name: Option<&'a str>,
resume_context: Option<&'a str>,
session_id_to_resume: Option<String>,
) -> Result<AgentInfo, String> {
self.start_agent_inner(
) -> Pin<Box<dyn Future<Output = Result<AgentInfo, String>> + Send + 'a>> {
Box::pin(self.start_agent_inner(
project_root,
story_id,
agent_name,
resume_context,
session_id_to_resume,
None,
)
))
}
/// Start an agent with an `AppContext` for direct MCP tool dispatch.
///
/// API-based runtimes (Gemini, OpenAI) need the `AppContext` to invoke MCP
/// tools without an HTTP round-trip. CLI-based runtimes (Claude Code) do not.
pub fn start_agent_with_ctx(
&self,
project_root: &Path,
story_id: &str,
agent_name: Option<&str>,
resume_context: Option<&str>,
pub fn start_agent_with_ctx<'a>(
&'a self,
project_root: &'a Path,
story_id: &'a str,
agent_name: Option<&'a str>,
resume_context: Option<&'a str>,
session_id_to_resume: Option<String>,
app_ctx: Arc<crate::http::context::AppContext>,
) -> Result<AgentInfo, String> {
self.start_agent_inner(
) -> Pin<Box<dyn Future<Output = Result<AgentInfo, String>> + Send + 'a>> {
Box::pin(self.start_agent_inner(
project_root,
story_id,
agent_name,
resume_context,
session_id_to_resume,
Some(app_ctx),
)
))
}
fn start_agent_inner(
async fn start_agent_inner(
&self,
project_root: &Path,
story_id: &str,
@@ -100,7 +102,8 @@ impl AgentPool {
// Create name-independent shared resources before the lock so they are
// ready for the atomic check-and-insert (story 132).
let (tx, _) = broadcast::channel::<AgentEvent>(1024);
let event_log: Arc<Mutex<Vec<AgentEvent>>> = Arc::new(Mutex::new(Vec::new()));
let event_log: Arc<std::sync::Mutex<Vec<AgentEvent>>> =
Arc::new(std::sync::Mutex::new(Vec::new()));
let log_session_id = uuid::Uuid::new_v4().to_string();
// Create the per-session status buffer subscribed to this project's
@@ -149,7 +152,7 @@ impl AgentPool {
// agent turn (story 736).
let prior_events: Option<String>;
{
let mut agents = self.agents.lock().map_err(|e| e.to_string())?;
let mut agents = self.agents.lock().await;
resolved_name = match agent_name {
Some(name) => name.to_string(),
@@ -371,7 +374,7 @@ impl AgentPool {
// the atomic resolution above).
let log_writer =
match AgentLogWriter::new(project_root, story_id, &resolved_name, &log_session_id) {
Ok(w) => Some(Arc::new(Mutex::new(w))),
Ok(w) => Some(Arc::new(std::sync::Mutex::new(w))),
Err(e) => {
eprintln!(
"[agents] Failed to create log writer for {story_id}:{resolved_name}: {e}"
@@ -436,7 +439,7 @@ impl AgentPool {
// Store the task handle while the agent is still Pending.
{
let mut agents = self.agents.lock().map_err(|e| e.to_string())?;
let mut agents = self.agents.lock().await;
if let Some(agent) = agents.get_mut(&key) {
agent.task_handle = Some(handle);
}
+74 -72
View File
@@ -146,7 +146,7 @@ pub(super) async fn run_agent_spawn(
story_id: String,
agent_name: String,
tx: broadcast::Sender<AgentEvent>,
agents: Arc<Mutex<HashMap<String, StoryAgent>>>,
agents: Arc<tokio::sync::Mutex<HashMap<String, StoryAgent>>>,
key: String,
event_log: Arc<Mutex<Vec<AgentEvent>>>,
port: u16,
@@ -189,10 +189,10 @@ pub(super) async fn run_agent_spawn(
let wt_info = {
let wt_path = crate::worktree::worktree_path(&project_root_clone, &sid);
let branch = format!("feature/story-{sid}");
let base_branch = config_clone
.base_branch
.clone()
.unwrap_or_else(|| crate::worktree::detect_base_branch(&project_root_clone));
let base_branch = crate::worktree::resolve_base_branch(
&project_root_clone,
config_clone.base_branch.as_deref(),
);
let deadline =
tokio::time::Instant::now() + std::time::Duration::from_secs(worktree_wait_secs);
loop {
@@ -218,10 +218,11 @@ pub(super) async fn run_agent_spawn(
log.push(event.clone());
}
let _ = tx_clone.send(event);
if let Ok(mut agents) = agents_ref.lock()
&& let Some(agent) = agents.get_mut(&key_clone)
{
agent.status = AgentStatus::Failed;
let mut agents = agents_ref.lock().await;
if let Some(agent) = agents.get_mut(&key_clone) {
agent.status = AgentStatus::Failed;
}
}
AgentPool::notify_agent_state_changed(&watcher_tx_clone);
return;
@@ -233,21 +234,33 @@ pub(super) async fn run_agent_spawn(
// Step 1.1: Install the pre-commit quality-gate hook in the worktree.
// Non-fatal — if installation fails the agent can still run; the failure
// is logged so the operator can investigate.
if let Err(e) = crate::worktree::install_pre_commit_hook(&wt_info.path) {
slog_error!("[agents] pre-commit hook install failed for {sid}: {e}");
// Runs in spawn_blocking because install_pre_commit_hook executes
// synchronous git-config subprocesses that would otherwise pin a
// tokio worker thread and contribute to runtime starvation under
// concurrent agent spawns.
{
let hook_path = wt_info.path.clone();
let hook_result = tokio::task::spawn_blocking(move || {
crate::worktree::install_pre_commit_hook(&hook_path)
})
.await
.unwrap_or_else(|e| Err(format!("spawn_blocking panicked: {e}")));
if let Err(e) = hook_result {
slog_error!("[agents] pre-commit hook install failed for {sid}: {e}");
}
}
// Step 2: store worktree info and render agent command/args/prompt.
let wt_path_str = wt_info.path.to_string_lossy().to_string();
{
if let Ok(mut agents) = agents_ref.lock()
&& let Some(agent) = agents.get_mut(&key_clone)
{
let mut agents = agents_ref.lock().await;
if let Some(agent) = agents.get_mut(&key_clone) {
agent.worktree_info = Some(wt_info.clone());
}
}
let (command, mut args, mut prompt) = match config_clone.render_agent_args(
&project_root_clone,
&wt_path_str,
&sid,
Some(&aname),
@@ -266,10 +279,11 @@ pub(super) async fn run_agent_spawn(
log.push(event.clone());
}
let _ = tx_clone.send(event);
if let Ok(mut agents) = agents_ref.lock()
&& let Some(agent) = agents.get_mut(&key_clone)
{
agent.status = AgentStatus::Failed;
let mut agents = agents_ref.lock().await;
if let Some(agent) = agents.get_mut(&key_clone) {
agent.status = AgentStatus::Failed;
}
}
AgentPool::notify_agent_state_changed(&watcher_tx_clone);
return;
@@ -358,9 +372,8 @@ pub(super) async fn run_agent_spawn(
// Step 3: transition to Running now that the worktree is ready.
{
if let Ok(mut agents) = agents_ref.lock()
&& let Some(agent) = agents.get_mut(&key_clone)
{
let mut agents = agents_ref.lock().await;
if let Some(agent) = agents.get_mut(&key_clone) {
agent.status = AgentStatus::Running;
}
}
@@ -457,25 +470,26 @@ pub(super) async fn run_agent_spawn(
match run_result {
Ok(result) => {
// Persist token usage if the agent reported it.
if let Some(ref usage) = result.token_usage
&& let Ok(agents) = agents_ref.lock()
&& let Some(agent) = agents.get(&key_clone)
&& let Some(ref pr) = agent.project_root
{
let model_for_record = config_clone
.find_agent(&aname)
.and_then(|a| a.model.clone());
let record = crate::agents::token_usage::build_record(
&sid,
&aname,
model_for_record,
usage.clone(),
);
if let Err(e) = crate::agents::token_usage::append_record(pr, &record) {
slog_error!(
"[agents] Failed to persist token usage for \
{sid}:{aname}: {e}"
if let Some(ref usage) = result.token_usage {
let agents = agents_ref.lock().await;
if let Some(agent) = agents.get(&key_clone)
&& let Some(ref pr) = agent.project_root
{
let model_for_record = config_clone
.find_agent(&aname)
.and_then(|a| a.model.clone());
let record = crate::agents::token_usage::build_record(
&sid,
&aname,
model_for_record,
usage.clone(),
);
if let Err(e) = crate::agents::token_usage::append_record(pr, &record) {
slog_error!(
"[agents] Failed to persist token usage for \
{sid}:{aname}: {e}"
);
}
}
}
@@ -526,10 +540,7 @@ pub(super) async fn run_agent_spawn(
// Remove the agent entry from the pool and emit Done so that
// any caller blocked on wait_for_agent is unblocked.
let tx_done = {
let mut lock = match agents_ref.lock() {
Ok(a) => a,
Err(_) => return,
};
let mut lock = agents_ref.lock().await;
if let Some(agent) = lock.remove(&key_clone) {
agent.tx
} else {
@@ -608,10 +619,7 @@ pub(super) async fn run_agent_spawn(
if stage == PipelineStage::Mergemaster {
let (tx_done, done_session_id, merge_failure_reported, merge_success_reported) = {
let mut lock = match agents_ref.lock() {
Ok(a) => a,
Err(_) => return,
};
let mut lock = agents_ref.lock().await;
if let Some(agent) = lock.remove(&key_clone) {
(
agent.tx,
@@ -648,15 +656,14 @@ pub(super) async fn run_agent_spawn(
// Do NOT send WorkItem/reassign — story is already Done.
// Drain one queued ConflictDetected story now that this
// mergemaster slot is free (story 1044).
if let Some((candidate_id, candidate_agent)) =
crate::config::ProjectConfig::load(&project_root_clone)
.ok()
.and_then(|cfg| {
agents_ref.lock().ok().as_ref().and_then(|agts| {
pick_queued_conflict_detected(&cfg, agts, &sid)
})
})
{
let candidate =
if let Ok(cfg) = crate::config::ProjectConfig::load(&project_root_clone) {
let agts = agents_ref.lock().await;
pick_queued_conflict_detected(&cfg, &agts, &sid)
} else {
None
};
if let Some((candidate_id, candidate_agent)) = candidate {
slog!(
"[agents] Mergemaster exit for '{sid}' (success): \
queued ConflictDetected story '{candidate_id}' found; \
@@ -766,17 +773,14 @@ pub(super) async fn run_agent_spawn(
});
// Drain one queued ConflictDetected story now that this
// mergemaster slot is free (story 1044).
if let Some((candidate_id, candidate_agent)) =
crate::config::ProjectConfig::load(&project_root_clone)
.ok()
.and_then(|cfg| {
agents_ref
.lock()
.ok()
.as_ref()
.and_then(|agts| pick_queued_conflict_detected(&cfg, agts, &sid))
})
{
let candidate =
if let Ok(cfg) = crate::config::ProjectConfig::load(&project_root_clone) {
let agts = agents_ref.lock().await;
pick_queued_conflict_detected(&cfg, &agts, &sid)
} else {
None
};
if let Some((candidate_id, candidate_agent)) = candidate {
slog!(
"[agents] Mergemaster exit for '{sid}': queued ConflictDetected \
story '{candidate_id}' found; spawning '{candidate_agent}'."
@@ -833,10 +837,7 @@ pub(super) async fn run_agent_spawn(
// Remove agent from the pool and unblock any wait_for_agent callers.
let tx_done = {
let mut lock = match agents_ref.lock() {
Ok(a) => a,
Err(_) => return,
};
let mut lock = agents_ref.lock().await;
if let Some(agent) = lock.remove(&key_clone) {
agent.tx
} else {
@@ -931,10 +932,11 @@ pub(super) async fn run_agent_spawn(
log.push(event.clone());
}
let _ = tx_clone.send(event);
if let Ok(mut agents) = agents_ref.lock()
&& let Some(agent) = agents.get_mut(&key_clone)
{
agent.status = AgentStatus::Failed;
let mut agents = agents_ref.lock().await;
if let Some(agent) = agents.get_mut(&key_clone) {
agent.status = AgentStatus::Failed;
}
}
AgentPool::notify_agent_state_changed(&watcher_tx_clone);
}
@@ -109,7 +109,7 @@ async fn start_agent_cleans_up_pending_entry_on_failure() {
"agent must transition to Failed after worktree creation error"
);
let agents = pool.agents.lock().unwrap();
let agents = pool.agents.lock().await;
let failed_entry = agents
.values()
.find(|a| a.agent_name == "coder-1" && a.status == AgentStatus::Failed);
@@ -121,6 +121,7 @@ async fn start_agent_cleans_up_pending_entry_on_failure() {
let events = pool
.drain_events("50_story_test", "coder-1")
.await
.expect("drain_events should succeed");
let has_error_event = events.iter().any(|e| matches!(e, AgentEvent::Error { .. }));
assert!(
@@ -736,7 +737,7 @@ async fn reconcile_canonical_agents_stops_stale_coder_in_qa_stage() {
let pool = AgentPool::new_test(3099);
pool.inject_test_agent("777_story_reconcile", "coder-1", AgentStatus::Running);
let before = pool.list_agents().unwrap();
let before = pool.list_agents().await.unwrap();
assert!(
before.iter().any(|a| a.agent_name == "coder-1"
&& matches!(a.status, AgentStatus::Running | AgentStatus::Pending)),
@@ -745,7 +746,7 @@ async fn reconcile_canonical_agents_stops_stale_coder_in_qa_stage() {
pool.reconcile_canonical_agents(root).await;
let after = pool.list_agents().unwrap();
let after = pool.list_agents().await.unwrap();
let still_active = after.iter().any(|a| {
a.story_id == "777_story_reconcile"
&& a.agent_name == "coder-1"
@@ -786,7 +787,7 @@ async fn reconcile_canonical_agents_leaves_correct_stage_agent_alone() {
pool.reconcile_canonical_agents(root).await;
let after = pool.list_agents().unwrap();
let after = pool.list_agents().await.unwrap();
let still_active = after.iter().any(|a| {
a.story_id == "555_story_correct"
&& a.agent_name == "coder-1"
@@ -851,7 +852,7 @@ async fn regression_1100_stale_coder_blocks_mergemaster_then_reconciler_clears()
pool.reconcile_canonical_agents(root).await;
// coder-1 must be gone from the active pool.
let remaining = pool.list_agents().unwrap();
let remaining = pool.list_agents().await.unwrap();
assert!(
!remaining.iter().any(|a| {
a.story_id == "1100_reg"
+18 -27
View File
@@ -1,7 +1,6 @@
//! Agent stop — terminates a running agent while preserving its worktree.
use crate::process_kill::{pids_matching, sigkill_pids_and_verify};
use crate::slog;
use crate::slog_error;
use crate::slog_warn;
use std::path::Path;
@@ -40,7 +39,7 @@ impl AgentPool {
// Step 1: snapshot the worktree path (no status mutation yet).
let worktree_info = {
let agents = self.agents.lock().map_err(|e| e.to_string())?;
let agents = self.agents.lock().await;
let agent = agents
.get(&key)
.ok_or_else(|| format!("No agent '{agent_name}' for story '{story_id}'"))?;
@@ -71,12 +70,12 @@ impl AgentPool {
"[stop_agent] No worktree path recorded for '{key}'; cannot tree-kill, \
falling back to portable_pty SIGHUP (likely no-op for claude-code)."
);
self.kill_child_for_key(&key);
self.kill_child_for_key(&key).await;
}
// Step 3: now safe to mutate. Status flip and handle abort.
let (task_handle, tx) = {
let mut agents = self.agents.lock().map_err(|e| e.to_string())?;
let mut agents = self.agents.lock().await;
let agent = agents
.get_mut(&key)
.ok_or_else(|| format!("No agent '{agent_name}' for story '{story_id}'"))?;
@@ -107,7 +106,7 @@ impl AgentPool {
// Remove from map.
{
let mut agents = self.agents.lock().map_err(|e| e.to_string())?;
let mut agents = self.agents.lock().await;
agents.remove(&key);
}
@@ -138,9 +137,7 @@ impl AgentPool {
// Snapshot active LLM agents without holding the lock during async stops.
let snapshot: Vec<(String, String, PipelineStage)> = {
let Ok(agents) = self.agents.lock() else {
return;
};
let agents = self.agents.lock().await;
agents
.iter()
.filter_map(|(key, a)| {
@@ -197,14 +194,8 @@ impl AgentPool {
///
/// Called when a story is archived so that stale entries don't accumulate.
/// Returns the number of entries removed.
pub fn remove_agents_for_story(&self, story_id: &str) -> usize {
let mut agents = match self.agents.lock() {
Ok(a) => a,
Err(e) => {
slog_error!("[agents] Failed to lock pool for cleanup of '{story_id}': {e}");
return 0;
}
};
pub async fn remove_agents_for_story(&self, story_id: &str) -> usize {
let mut agents = self.agents.lock().await;
let prefix = format!("{story_id}:");
let keys_to_remove: Vec<String> = agents
.keys()
@@ -229,30 +220,30 @@ mod tests {
// ── remove_agents_for_story tests ────────────────────────────────────────
#[test]
fn remove_agents_for_story_removes_all_entries() {
#[tokio::test]
async fn remove_agents_for_story_removes_all_entries() {
let pool = AgentPool::new_test(3001);
pool.inject_test_agent("story_a", "coder-1", AgentStatus::Completed);
pool.inject_test_agent("story_a", "qa", AgentStatus::Failed);
pool.inject_test_agent("story_b", "coder-1", AgentStatus::Running);
let removed = pool.remove_agents_for_story("story_a");
let removed = pool.remove_agents_for_story("story_a").await;
assert_eq!(removed, 2, "should remove both agents for story_a");
let agents = pool.list_agents().unwrap();
let agents = pool.list_agents().await.unwrap();
assert_eq!(agents.len(), 1, "only story_b agent should remain");
assert_eq!(agents[0].story_id, "story_b");
}
#[test]
fn remove_agents_for_story_returns_zero_when_no_match() {
#[tokio::test]
async fn remove_agents_for_story_returns_zero_when_no_match() {
let pool = AgentPool::new_test(3001);
pool.inject_test_agent("story_a", "coder-1", AgentStatus::Running);
let removed = pool.remove_agents_for_story("nonexistent");
let removed = pool.remove_agents_for_story("nonexistent").await;
assert_eq!(removed, 0);
let agents = pool.list_agents().unwrap();
let agents = pool.list_agents().await.unwrap();
assert_eq!(agents.len(), 1, "existing agents should not be affected");
}
@@ -283,12 +274,12 @@ mod tests {
pool.inject_test_agent("60_story_cleanup", "qa", AgentStatus::Completed);
pool.inject_test_agent("61_story_other", "coder-1", AgentStatus::Running);
assert_eq!(pool.list_agents().unwrap().len(), 3);
assert_eq!(pool.list_agents().await.unwrap().len(), 3);
move_story_to_done("60_story_cleanup").unwrap();
pool.remove_agents_for_story("60_story_cleanup");
pool.remove_agents_for_story("60_story_cleanup").await;
let remaining = pool.list_agents().unwrap();
let remaining = pool.list_agents().await.unwrap();
assert_eq!(
remaining.len(),
1,
+8 -8
View File
@@ -20,7 +20,7 @@ impl AgentPool {
) -> broadcast::Sender<AgentEvent> {
let (tx, _) = broadcast::channel::<AgentEvent>(64);
let key = composite_key(story_id, agent_name);
let mut agents = self.agents.lock().unwrap();
let mut agents = self.agents.try_lock().unwrap();
agents.insert(
key,
StoryAgent {
@@ -55,7 +55,7 @@ impl AgentPool {
) -> broadcast::Sender<AgentEvent> {
let (tx, _) = broadcast::channel::<AgentEvent>(64);
let key = composite_key(story_id, agent_name);
let mut agents = self.agents.lock().unwrap();
let mut agents = self.agents.try_lock().unwrap();
agents.insert(
key,
StoryAgent {
@@ -95,7 +95,7 @@ impl AgentPool {
) -> broadcast::Sender<AgentEvent> {
let (tx, _) = broadcast::channel::<AgentEvent>(64);
let key = composite_key(story_id, agent_name);
let mut agents = self.agents.lock().unwrap();
let mut agents = self.agents.try_lock().unwrap();
agents.insert(
key,
StoryAgent {
@@ -130,7 +130,7 @@ impl AgentPool {
) -> broadcast::Sender<AgentEvent> {
let (tx, _) = broadcast::channel::<AgentEvent>(64);
let key = composite_key(story_id, agent_name);
let mut agents = self.agents.lock().unwrap();
let mut agents = self.agents.try_lock().unwrap();
agents.insert(
key,
StoryAgent {
@@ -165,7 +165,7 @@ impl AgentPool {
) -> broadcast::Sender<AgentEvent> {
let (tx, _) = broadcast::channel::<AgentEvent>(64);
let key = composite_key(story_id, agent_name);
let mut agents = self.agents.lock().unwrap();
let mut agents = self.agents.try_lock().unwrap();
agents.insert(
key,
StoryAgent {
@@ -197,7 +197,7 @@ impl AgentPool {
story_id: &str,
agent_name: &str,
) -> Option<Vec<BufferedItem>> {
let agents = self.agents.lock().unwrap();
let agents = self.agents.try_lock().unwrap();
let key = composite_key(story_id, agent_name);
agents
.get(&key)
@@ -219,7 +219,7 @@ impl AgentPool {
) -> broadcast::Sender<AgentEvent> {
let (tx, _) = broadcast::channel::<AgentEvent>(64);
let key = composite_key(story_id, agent_name);
let mut agents = self.agents.lock().unwrap();
let mut agents = self.agents.try_lock().unwrap();
agents.insert(
key,
StoryAgent {
@@ -258,7 +258,7 @@ impl AgentPool {
) -> broadcast::Sender<AgentEvent> {
let (tx, _) = broadcast::channel::<AgentEvent>(64);
let key = composite_key(story_id, agent_name);
let mut agents = self.agents.lock().unwrap();
let mut agents = self.agents.try_lock().unwrap();
agents.insert(
key,
StoryAgent {
+7 -5
View File
@@ -3,8 +3,8 @@ use crate::slog;
use crate::worktree::WorktreeInfo;
use std::collections::HashMap;
use std::path::PathBuf;
use std::sync::{Arc, Mutex};
use tokio::sync::broadcast;
use std::sync::Arc;
use tokio::sync::{Mutex, broadcast};
use super::super::{AgentEvent, AgentInfo, AgentStatus, CompletionReport};
@@ -45,8 +45,10 @@ impl PendingGuard {
impl Drop for PendingGuard {
fn drop(&mut self) {
if self.armed
&& let Ok(mut agents) = self.agents.lock()
if !self.armed {
return;
}
if let Ok(mut agents) = self.agents.try_lock()
&& agents
.get(&self.key)
.is_some_and(|a| a.status == AgentStatus::Pending)
@@ -68,7 +70,7 @@ pub(super) struct StoryAgent {
pub(super) tx: broadcast::Sender<AgentEvent>,
pub(super) task_handle: Option<tokio::task::JoinHandle<()>>,
/// Accumulated events for polling via get_agent_output.
pub(super) event_log: Arc<Mutex<Vec<AgentEvent>>>,
pub(super) event_log: Arc<std::sync::Mutex<Vec<AgentEvent>>>,
/// Set when the agent calls report_completion.
pub(super) completion: Option<CompletionReport>,
/// Project root, stored for pipeline advancement after completion.
+5 -5
View File
@@ -17,11 +17,11 @@ impl AgentPool {
) -> Result<AgentInfo, String> {
// Subscribe before checking status so we don't miss the terminal event
// if the agent completes in the window between the two operations.
let mut rx = self.subscribe(story_id, agent_name)?;
let mut rx = self.subscribe(story_id, agent_name).await?;
// Return immediately if already in a terminal state.
{
let agents = self.agents.lock().map_err(|e| e.to_string())?;
let agents = self.agents.lock().await;
let key = composite_key(story_id, agent_name);
if let Some(agent) = agents.get(&key)
&& matches!(agent.status, AgentStatus::Completed | AgentStatus::Failed)
@@ -48,7 +48,7 @@ impl AgentPool {
_ => false,
};
if is_terminal {
let agents = self.agents.lock().map_err(|e| e.to_string())?;
let agents = self.agents.lock().await;
let key = composite_key(story_id, agent_name);
return Ok(if let Some(agent) = agents.get(&key) {
agent_info_from_entry(story_id, agent)
@@ -78,7 +78,7 @@ impl AgentPool {
}
Ok(Err(broadcast::error::RecvError::Lagged(_))) => {
// Missed some buffered events — check current status before resuming.
let agents = self.agents.lock().map_err(|e| e.to_string())?;
let agents = self.agents.lock().await;
let key = composite_key(story_id, agent_name);
if let Some(agent) = agents.get(&key)
&& matches!(agent.status, AgentStatus::Completed | AgentStatus::Failed)
@@ -89,7 +89,7 @@ impl AgentPool {
}
Ok(Err(broadcast::error::RecvError::Closed)) => {
// Channel closed: no more events will arrive. Return current state.
let agents = self.agents.lock().map_err(|e| e.to_string())?;
let agents = self.agents.lock().await;
let key = composite_key(story_id, agent_name);
if let Some(agent) = agents.get(&key) {
return Ok(agent_info_from_entry(story_id, agent));
@@ -130,11 +130,20 @@ pub(crate) async fn on_coding_transition(project_root: &Path, port: u16, story_i
info.path.display()
);
let hook_path = info.path.clone();
slog!("[worktree] starting pre-commit hook install for '{story_id}'");
let hook_result = tokio::task::spawn_blocking(move || {
crate::worktree::install_pre_commit_hook(&hook_path)
})
.await
.unwrap_or_else(|e| Err(format!("spawn_blocking panicked: {e}")));
match &hook_result {
Ok(()) => {
slog!("[worktree] finished pre-commit hook install for '{story_id}' result=ok")
}
Err(e) => slog!(
"[worktree] finished pre-commit hook install for '{story_id}' result=err: {e}"
),
}
if let Err(e) = hook_result {
slog_warn!(
"[worktree-create-sub] Pre-commit hook install failed for '{story_id}': {e}"
@@ -148,6 +157,12 @@ pub(crate) async fn on_coding_transition(project_root: &Path, port: u16, story_i
}
/// Remove the worktree and feature branch for `story_id` after it reaches a terminal stage.
///
/// Story 1199: bytes-reclaimed accounting for the worktree's `target/` dir
/// happens one layer down, in `remove_worktree` — the single choke point
/// shared by this subscriber, the `remove_worktree` MCP tool, and
/// `worktree::cleanup`/`sweep`. Logging it there (once) avoids a duplicate
/// log line here.
pub(crate) async fn on_terminal_transition(project_root: &Path, story_id: &str) {
let config = match crate::config::ProjectConfig::load(project_root) {
Ok(c) => c,
@@ -261,6 +276,30 @@ mod tests {
);
}
/// Story 1199 AC4: on_terminal_transition reclaims the worktree's target/
/// dir (and everything else in the worktree) when removing it.
#[tokio::test]
async fn terminal_transition_reclaims_target_dir() {
let tmp = TempDir::new().unwrap();
let root = setup_project(&tmp);
let story_id = "1006_test_reclaim_target";
on_coding_transition(&root, 3001, story_id).await;
let wt_path = crate::worktree::worktree_path(&root, story_id);
let target_dir = wt_path.join("target");
fs::create_dir_all(&target_dir).unwrap();
fs::write(target_dir.join("build_artifact.bin"), vec![0u8; 512]).unwrap();
assert!(target_dir.exists(), "target/ must exist before cleanup");
on_terminal_transition(&root, story_id).await;
assert!(
!target_dir.exists(),
"target/ dir must be reclaimed when the worktree is removed"
);
assert!(!wt_path.exists());
}
/// AC2: on_terminal_transition is a no-op (non-fatal) when no worktree exists.
#[tokio::test]
async fn terminal_transition_noop_when_no_worktree() {
+220
View File
@@ -247,6 +247,104 @@ mod tests {
);
}
// ── story 1196: in-flight MCP tool calls suspend the inactivity deadline ──
/// A `tool_use` block in an `assistant` message marks a call as in
/// flight; the inactivity deadline must be suspended while it is
/// outstanding (so a slow MCP tool like `run_tests` doesn't get the
/// agent killed), and must resume as soon as the matching `tool_result`
/// arrives in a `user` message.
///
/// Script: emits a tool_use, sleeps 2s (would fail a 1s timeout without
/// suspension), emits the matching tool_result, then sleeps 2s again
/// (now with no call in flight, the 1s timeout must fire).
#[tokio::test]
async fn tool_call_in_flight_suspends_then_resumes_inactivity_deadline() {
use std::os::unix::fs::PermissionsExt;
let tmp = tempfile::tempdir().unwrap();
let script = tmp.path().join("tool_call_then_silence.sh");
let body = "#!/bin/sh\n\
printf '%s\\n' '{\"type\":\"assistant\",\"message\":{\"content\":[{\"type\":\"tool_use\",\"id\":\"tool1\",\"name\":\"run_tests\",\"input\":{}}]}}'\n\
sleep 2\n\
printf '%s\\n' '{\"type\":\"user\",\"message\":{\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"tool1\",\"content\":\"ok\"}]}}'\n\
sleep 2\n";
std::fs::write(&script, body).unwrap();
std::fs::set_permissions(&script, std::fs::Permissions::from_mode(0o755)).unwrap();
let (tx, _rx) = broadcast::channel::<AgentEvent>(64);
let (watcher_tx, _watcher_rx) = broadcast::channel::<WatcherEvent>(16);
let event_log = Arc::new(Mutex::new(Vec::new()));
let result = run_agent_pty_streaming(
"1196_story_tool_call_in_flight",
"coder-1",
"sh",
&[script.to_string_lossy().to_string()],
"--",
"/tmp",
&tx,
&event_log,
None,
1, // inactivity_timeout_secs = 1s
watcher_tx,
None,
None,
)
.await;
match result {
Err(err) => assert!(
err.contains("inactivity timeout"),
"expected an inactivity timeout error after the tool call resolved, got: {err}"
),
Ok(_) => panic!(
"agent must still be killed once the tool call resolves and \
the process falls genuinely silent again"
),
}
}
/// A genuinely hung agent (no output at all, no tool call in flight)
/// must still be killed by the inactivity watchdog after the configured
/// timeout — the suspension added for in-flight tool calls must not
/// mask a real hang.
#[tokio::test]
async fn genuinely_silent_agent_with_no_in_flight_call_is_killed() {
use std::os::unix::fs::PermissionsExt;
let tmp = tempfile::tempdir().unwrap();
let script = tmp.path().join("silent.sh");
std::fs::write(&script, "#!/bin/sh\nsleep 2\n").unwrap();
std::fs::set_permissions(&script, std::fs::Permissions::from_mode(0o755)).unwrap();
let (tx, _rx) = broadcast::channel::<AgentEvent>(64);
let (watcher_tx, _watcher_rx) = broadcast::channel::<WatcherEvent>(16);
let event_log = Arc::new(Mutex::new(Vec::new()));
let result = run_agent_pty_streaming(
"1196_story_genuine_hang",
"coder-1",
"sh",
&[script.to_string_lossy().to_string()],
"--",
"/tmp",
&tx,
&event_log,
None,
1, // inactivity_timeout_secs = 1s
watcher_tx,
None,
None,
)
.await;
match result {
Err(err) => assert!(err.contains("inactivity timeout")),
Ok(_) => panic!("a genuinely silent agent with no in-flight tool call must be killed"),
}
}
#[test]
fn test_emit_event_writes_to_log_writer() {
let tmp = tempfile::tempdir().unwrap();
@@ -503,4 +601,126 @@ mod tests {
"Expected RateLimitWarning for status=allowed, got: {evt:?}"
);
}
// ── story 1211: deterministic crash notification on mid-turn death ──────
/// AC1/AC3/AC4/AC5: a child killed mid-turn (inactivity watchdog, never
/// emits a `"result"` event) produces exactly one `AgentCrashed` watcher
/// event, carrying the exit code and the last line the child printed
/// before it died. The watchdog-kill path used to `return Err(...)`
/// directly from inside the receive loop, bypassing the bottom
/// cleanup/notification code entirely — this test guards against that
/// regression.
#[tokio::test]
async fn killed_mid_turn_sends_exactly_one_crash_notification() {
use std::os::unix::fs::PermissionsExt;
let tmp = tempfile::tempdir().unwrap();
let script = tmp.path().join("crash_then_hang.sh");
std::fs::write(
&script,
"#!/bin/sh\nprintf '%s\\n' 'assertion failed: output.write(&bytes).is_ok()'\nsleep 5\n",
)
.unwrap();
std::fs::set_permissions(&script, std::fs::Permissions::from_mode(0o755)).unwrap();
let (tx, _rx) = broadcast::channel::<AgentEvent>(64);
let (watcher_tx, mut watcher_rx) = broadcast::channel::<WatcherEvent>(16);
let event_log = Arc::new(Mutex::new(Vec::new()));
let result = run_agent_pty_streaming(
"1211_story_crash",
"coder-1",
"sh",
&[script.to_string_lossy().to_string()],
"--",
"/tmp",
&tx,
&event_log,
None,
1, // inactivity_timeout_secs = 1s
watcher_tx,
None,
None,
)
.await;
assert!(
result.is_err(),
"a watchdog-killed agent must still return the inactivity timeout error"
);
let evt = watcher_rx
.try_recv()
.expect("Expected exactly one AgentCrashed watcher event");
match evt {
WatcherEvent::AgentCrashed {
story_id,
agent_name,
exit_code,
last_error_line,
} => {
assert_eq!(story_id, "1211_story_crash");
assert_eq!(agent_name, "coder-1");
assert!(exit_code.is_some(), "exit code should be captured");
assert_eq!(
last_error_line.as_deref(),
Some("assertion failed: output.write(&bytes).is_ok()"),
"last line before death should be captured"
);
}
other => panic!("Expected AgentCrashed, got: {other:?}"),
}
// AC5: exactly one crash notification — no second event queued.
assert!(
watcher_rx.try_recv().is_err(),
"must not emit a duplicate AgentCrashed for the same death event"
);
}
/// AC2: a turn that ends cleanly (a `"result"` event observed) followed
/// by a normal child exit must produce zero `AgentCrashed` notifications.
#[tokio::test]
async fn clean_turn_end_sends_zero_crash_notifications() {
use std::os::unix::fs::PermissionsExt;
let tmp = tempfile::tempdir().unwrap();
let script = tmp.path().join("clean_result.sh");
std::fs::write(
&script,
"#!/bin/sh\nprintf '%s\\n' '{\"type\":\"result\",\"subtype\":\"success\"}'\n",
)
.unwrap();
std::fs::set_permissions(&script, std::fs::Permissions::from_mode(0o755)).unwrap();
let (tx, _rx) = broadcast::channel::<AgentEvent>(64);
let (watcher_tx, mut watcher_rx) = broadcast::channel::<WatcherEvent>(16);
let event_log = Arc::new(Mutex::new(Vec::new()));
let result = run_agent_pty_streaming(
"1211_story_clean",
"coder-1",
"sh",
&[script.to_string_lossy().to_string()],
"--",
"/tmp",
&tx,
&event_log,
None,
0, // no inactivity timeout
watcher_tx,
None,
None,
)
.await;
assert!(result.is_ok(), "PTY run should succeed: {:?}", result.err());
assert!(result.unwrap().exit_ok, "clean exit should report exit_ok");
assert!(
watcher_rx.try_recv().is_err(),
"a clean turn end followed by normal exit must not send AgentCrashed"
);
}
}
+92 -12
View File
@@ -246,14 +246,41 @@ fn run_agent_pty_blocking(
// can distinguish a rate-limit exit from a genuine no-progress exit (bug 1053).
let mut rate_limit_hard_block_seen = false;
let mut rate_limit_reset_at_captured: Option<chrono::DateTime<chrono::Utc>> = None;
// Tool-use ids from `assistant` messages that haven't yet seen a matching
// `tool_result` in a `user` message. While an MCP tool call (e.g.
// run_tests) is in flight, the server can legitimately run for many
// minutes with no PTY output at all — the CLI is blocked waiting on the
// MCP response, not hung. The inactivity deadline is suspended entirely
// while this set is non-empty, and resumes as soon as the matching
// tool_result clears the last in-flight id (story 1196).
let mut tool_calls_in_flight: std::collections::HashSet<String> =
std::collections::HashSet::new();
// Tracks whether a `"result"` event was observed — the CLI's signal that
// the current turn completed normally (see
// llm/providers/claude_code/events/mod.rs). If the child dies before
// this is set, the death is a mid-turn crash (AC1/AC2 of story 1211)
// rather than a clean completion.
let mut saw_result_event = false;
// Most recent non-empty line emitted by the child before it died — used
// as the "last error line" in the crash notification (AC4) since panics
// and assertion failures print to the PTY's combined stdout/stderr.
let mut last_line: Option<String> = None;
// Set when the inactivity watchdog kills the child (AC3 of story 1211):
// tracked instead of returning early so the death still flows through
// the single crash-detection funnel below before the function returns.
let mut timed_out = false;
loop {
let effective_timeout = base_timeout.map(|base| {
let extra = block_until
.and_then(|t| (t - chrono::Utc::now()).to_std().ok())
.unwrap_or(std::time::Duration::ZERO);
base + extra
});
let effective_timeout = if !tool_calls_in_flight.is_empty() {
None
} else {
base_timeout.map(|base| {
let extra = block_until
.and_then(|t| (t - chrono::Utc::now()).to_std().ok())
.unwrap_or(std::time::Duration::ZERO);
base + extra
})
};
let recv_result = match effective_timeout {
Some(dur) => line_rx.recv_timeout(dur),
@@ -278,10 +305,8 @@ fn run_agent_pty_blocking(
{inactivity_timeout_secs}s with no output. Killing process."
);
let _ = child.kill();
let _ = child.wait();
return Err(format!(
"Agent inactivity timeout: no output received for {inactivity_timeout_secs}s"
));
timed_out = true;
break;
}
};
@@ -289,6 +314,7 @@ fn run_agent_pty_blocking(
if trimmed.is_empty() {
continue;
}
last_line = Some(trimmed.to_string());
// Try to parse as JSON
let json: serde_json::Value = match serde_json::from_str(trimmed) {
@@ -339,8 +365,31 @@ fn run_agent_pty_blocking(
}
// Complete assistant events are skipped for content extraction
// because thinking and text already arrived via stream_event.
// The raw JSON is still forwarded as AgentJson below.
"assistant" | "user" => {}
// The raw JSON is still forwarded as AgentJson below. A tool_use
// block marks an MCP call as in flight so the inactivity deadline
// is suspended until its tool_result arrives (story 1196).
"assistant" => {
if let Some(blocks) = json.pointer("/message/content").and_then(|c| c.as_array()) {
for block in blocks {
if block.get("type").and_then(|t| t.as_str()) == Some("tool_use")
&& let Some(id) = block.get("id").and_then(|i| i.as_str())
{
tool_calls_in_flight.insert(id.to_string());
}
}
}
}
"user" => {
if let Some(blocks) = json.pointer("/message/content").and_then(|c| c.as_array()) {
for block in blocks {
if block.get("type").and_then(|t| t.as_str()) == Some("tool_result")
&& let Some(id) = block.get("tool_use_id").and_then(|i| i.as_str())
{
tool_calls_in_flight.remove(id);
}
}
}
}
"rate_limit_event" => {
let rate_limit_info = json.get("rate_limit_info");
let status = rate_limit_info
@@ -404,6 +453,10 @@ fn run_agent_pty_blocking(
}
}
"result" => {
// A "result" event signals the CLI turn completed — mark the
// turn as clean so the post-loop crash check (AC1/AC2 of
// story 1211) does not treat this exit as a mid-turn death.
saw_result_event = true;
// Extract token usage from the result event.
if let Some(usage) = TokenUsage::from_result_event(&json) {
slog!(
@@ -445,6 +498,7 @@ fn run_agent_pty_blocking(
false
}
};
let exit_code = wait_result.as_ref().ok().map(|status| status.exit_code());
// Wait for the reader thread to finish so it releases the cloned PTY
// master fd before we return. Without this, the next PTY spawn for the
@@ -453,6 +507,32 @@ fn run_agent_pty_blocking(
slog!("[agent:{story_id}:{agent_name}] Reader thread panicked: {e:?}");
}
// AC1-AC3 (story 1211): single crash-detection funnel. Every exit path
// above (EOF, reader disconnect, IO error, or watchdog kill on timeout)
// breaks out of the loop into this one spot instead of returning early,
// so a mid-turn death is detected and notified exactly once regardless
// of which path triggered it (AC5). A turn is "clean" once a `"result"`
// event has been observed; anything else was killed or crashed before
// finishing its turn.
if !saw_result_event {
slog_warn!(
"[agent:{story_id}:{agent_name}] Agent died mid-turn (no result event observed); \
exit_code={exit_code:?}, last_line={last_line:?}"
);
let _ = watcher_tx.send(WatcherEvent::AgentCrashed {
story_id: story_id.to_string(),
agent_name: agent_name.to_string(),
exit_code,
last_error_line: last_line.clone(),
});
}
if timed_out {
return Err(format!(
"Agent inactivity timeout: no output received for {inactivity_timeout_secs}s"
));
}
// Log whether session was created — Session: None indicates CLI died
// before emitting any events (possible causes: rate limit, budget
// exhaustion, PTY write failure, CLI crash).
+3 -12
View File
@@ -90,19 +90,10 @@ fn find_story_id(num_str: &str) -> Option<String> {
/// Return the configured base branch, or auto-detect it from the project root HEAD.
fn resolve_base_branch(project_root: &Path) -> String {
crate::config::ProjectConfig::load(project_root)
let configured = crate::config::ProjectConfig::load(project_root)
.ok()
.and_then(|c| c.base_branch)
.unwrap_or_else(|| {
Command::new("git")
.args(["rev-parse", "--abbrev-ref", "HEAD"])
.current_dir(project_root)
.output()
.ok()
.filter(|o| o.status.success())
.map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string())
.unwrap_or_else(|| "master".to_string())
})
.and_then(|c| c.base_branch);
crate::worktree::resolve_base_branch(project_root, configured.as_deref())
}
/// Run a git command in `dir`, returning trimmed stdout (empty string on failure).
+42 -8
View File
@@ -225,9 +225,14 @@ pub fn commands() -> &'static [BotCommand] {
handler: handle_reset_fallback,
},
BotCommand {
name: "rebuild",
description: "Rebuild the server binary and restart",
handler: handle_rebuild_fallback,
name: "compact",
description: "Distill the session transcript into a seed, then reset (unlike `reset`, keeps distilled context for the next message)",
handler: handle_compact_fallback,
},
BotCommand {
name: "stop",
description: "Cancel the in-flight LLM turn for this room (synonyms: `halt`, `abort`)",
handler: handle_stop_fallback,
},
BotCommand {
name: "timer",
@@ -284,6 +289,11 @@ pub fn commands() -> &'static [BotCommand] {
description: "Rebuild a project's Docker image and swap the container (gateway only): `project-rebuild <name> [--timeout <secs>] [--force]`",
handler: handle_project_rebuild_fallback,
},
BotCommand {
name: "projects",
description: "List all registered gateway projects: name, url, ssh port, host path, and adopted/built-in marker (gateway only)",
handler: handle_projects_fallback,
},
]
}
@@ -420,13 +430,27 @@ fn handle_reset_fallback(_ctx: &CommandContext) -> Option<String> {
None
}
/// Fallback handler for the `rebuild` command when it is not intercepted by
/// the async handler in `on_room_message`. In practice this is never called —
/// rebuild is detected and handled before `try_handle_command` is invoked.
/// Fallback handler for the `compact` command when it is not intercepted by
/// the async handler beforehand. For Matrix, `on_room_message` detects and
/// handles `compact` before `try_handle_command` is invoked, so this is
/// never called. Discord/Slack/WhatsApp check for `compact` only after their
/// own `try_handle_command` dispatch, so this handler does run there — it
/// deliberately always returns `None` so their subsequent async check gets a
/// chance to run the real handler instead of the LLM. The entry exists in
/// the registry so `help` lists it.
///
/// Returns `None` to prevent the LLM from receiving "compact" as a prompt.
fn handle_compact_fallback(_ctx: &CommandContext) -> Option<String> {
None
}
/// Fallback handler for the `stop` command when it is not intercepted by the
/// async handler in `on_room_message`. In practice this is never called —
/// stop is detected and handled before `try_handle_command` is invoked.
/// The entry exists in the registry only so `help` lists it.
///
/// Returns `None` to prevent the LLM from receiving "rebuild" as a prompt.
fn handle_rebuild_fallback(_ctx: &CommandContext) -> Option<String> {
/// Returns `None` to prevent the LLM from receiving "stop" as a prompt.
fn handle_stop_fallback(_ctx: &CommandContext) -> Option<String> {
None
}
@@ -461,6 +485,16 @@ fn handle_health_fallback(_ctx: &CommandContext) -> Option<String> {
None
}
/// Fallback handler for the `projects` command when it is not intercepted by the
/// async gateway handler in `on_room_message`. In practice this is never called —
/// `projects` is detected and handled before `try_handle_command` runs in gateway
/// mode. The entry exists in the registry so `help` lists it.
///
/// Returns `None` to prevent the LLM from receiving the raw command text.
fn handle_projects_fallback(_ctx: &CommandContext) -> Option<String> {
None
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
+142 -50
View File
@@ -3,66 +3,84 @@
use super::CommandContext;
use crate::io::story_metadata::QaMode;
/// Strip YAML front matter and return a summary of useful fields + the remaining body.
/// Strip a leading YAML front-matter block (`---\n...\n---`) and return the
/// remaining body text.
///
/// The front-matter block itself is no longer parsed for display — story
/// 1195 found that field extracted here (agent, depends_on, blocked,
/// retry_count, qa) can go stale relative to the CRDT registers that are the
/// actual source of truth. Metadata display now comes exclusively from
/// [`crdt_metadata_summary`].
#[allow(clippy::string_slice)] // indices from find("\n---") on ASCII delimiter; "---" and "\n---" are ASCII-only
fn strip_front_matter(text: &str) -> (String, String) {
fn strip_front_matter(text: &str) -> String {
let trimmed = text.trim_start();
if !trimmed.starts_with("---") {
return (String::new(), text.to_string());
return text.to_string();
}
// Find the closing ---
if let Some(end) = trimmed[3..].find("\n---") {
let yaml_block = &trimmed[3..3 + end].trim();
let body = &trimmed[3 + end + 4..]; // skip past closing ---
// Extract useful fields from YAML (simple line-based parsing)
let mut parts = Vec::new();
for line in yaml_block.lines() {
let line = line.trim();
if line.starts_with("depends_on:") {
let val = line.trim_start_matches("depends_on:").trim();
if !val.is_empty() && val != "[]" {
parts.push(format!("**Depends on:** {val}"));
}
} else if line.starts_with("agent:") {
let val = line.trim_start_matches("agent:").trim().trim_matches('"');
if !val.is_empty() {
parts.push(format!("**Agent:** {val}"));
}
} else if line.starts_with("blocked:") {
let val = line.trim_start_matches("blocked:").trim();
if val == "true" {
parts.push("**Blocked:** yes".to_string());
}
} else if line.starts_with("retry_count:") {
let val = line.trim_start_matches("retry_count:").trim();
if val != "0" && !val.is_empty() {
parts.push(format!("**Retries:** {val}"));
}
} else if line.starts_with("qa:") {
let val = line.trim_start_matches("qa:").trim().trim_matches('"');
if let Some(QaMode::Human) = QaMode::from_str(val) {
parts.push("**QA:** human review required".to_string());
}
} else if line.starts_with("merge_failure:") {
let val = line
.trim_start_matches("merge_failure:")
.trim()
.trim_matches('"');
if !val.is_empty() {
parts.push(format!("**Merge failure:** {val}"));
}
}
}
(parts.join(" · "), body.to_string())
trimmed[3 + end + 4..].to_string() // skip past closing ---
} else {
// No closing ---, return as-is
(String::new(), text.to_string())
text.to_string()
}
}
/// Build the metadata summary line from CRDT registers: agent, depends_on,
/// blocked, retry count, QA mode, and merge failure detail.
///
/// Register values are the sole source here — no fallback to content-text
/// YAML, which can go stale relative to the CRDT (story 1195). Mirrors the
/// CRDT-first pattern already used by `status <number>` (`triage.rs`).
fn crdt_metadata_summary(story_id: &str) -> String {
let Some(item) = crate::crdt_state::read_item(story_id) else {
return String::new();
};
let mut parts = Vec::new();
if matches!(
item.stage(),
crate::pipeline_state::Stage::Blocked { .. }
| crate::pipeline_state::Stage::MergeFailure { .. }
| crate::pipeline_state::Stage::MergeFailureFinal { .. }
| crate::pipeline_state::Stage::Archived {
reason: crate::pipeline_state::ArchiveReason::Blocked { .. },
..
}
) {
parts.push("**Blocked:** yes".to_string());
}
if let Some(agent) = item.agent() {
parts.push(format!("**Agent:** {agent}"));
}
let deps = item.depends_on();
if !deps.is_empty() {
let nums: Vec<String> = deps.iter().map(|n| format!("#{n}")).collect();
parts.push(format!("**Depends on:** {}", nums.join(", ")));
}
let rc = item.retry_count();
if rc > 0 {
parts.push(format!("**Retries:** {rc}"));
}
if let Some(QaMode::Human) = item.qa_mode() {
parts.push("**QA:** human review required".to_string());
}
if let Some(job) = crate::crdt_state::read_merge_job(story_id)
&& let Some(err) = job.error
{
parts.push(format!("**Merge failure:** {err}"));
}
parts.join(" · ")
}
/// Display the full markdown text of a work item identified by its numeric ID.
///
/// Lookup priority: CRDT → content store → filesystem (Story 512).
@@ -90,8 +108,11 @@ pub(super) fn handle_show(ctx: &CommandContext) -> Option<String> {
format!("Story {story_id} found in pipeline but its content is unavailable.")
});
// Strip front matter block and extract useful metadata to show inline.
let (front_matter_summary, body) = strip_front_matter(&text);
// Strip front matter block from the displayed body; source the metadata
// summary separately from CRDT registers, which win over any stale
// content-text YAML (story 1195).
let body = strip_front_matter(&text);
let front_matter_summary = crdt_metadata_summary(&story_id);
// Convert markdown headings to bold text for consistent rendering across
// Matrix clients. Element X doesn't style <h2> tags distinctly, but bold
@@ -254,6 +275,77 @@ mod tests {
);
}
/// Story 1195, AC 1 + 2: `show` must source metadata from the CRDT
/// registers, and a CRDT-only write (e.g. an agent pin via `update_story`)
/// must be visible immediately — no content-text rewrite required.
#[test]
fn show_command_reflects_crdt_only_agent_pin_immediately() {
crate::crdt_state::init_for_test();
let tmp = tempfile::TempDir::new().unwrap();
// Use a high story number to avoid collisions with other tests in the
// global content store.
write_story_file(
tmp.path(),
"2_current",
"9904_story_pin_test.md",
"---\nname: Pin Test\n---\n\n# Story\n\nBody text.",
None,
);
crate::crdt_state::write_item_str(
"9904_story_pin_test",
"2_current",
Some("Pin Test"),
None,
None,
None,
);
// Set the pin via the same CRDT-only register write `update_story`
// uses (crdt_state::set_agent) — the content body is never touched.
crate::crdt_state::set_agent("9904_story_pin_test", "coder-1".parse().ok());
let output = show_cmd_with_root(tmp.path(), "9904").unwrap();
assert!(
output.contains("**Agent:** coder-1"),
"show should reflect the CRDT-only agent pin immediately: {output}"
);
}
/// Story 1195, AC 1: register values must win over stale content-text
/// YAML — a front-matter `agent:` value that no longer matches the CRDT
/// register must not leak into the displayed summary.
#[test]
fn show_command_prefers_crdt_agent_over_stale_yaml_front_matter() {
crate::crdt_state::init_for_test();
let tmp = tempfile::TempDir::new().unwrap();
write_story_file(
tmp.path(),
"2_current",
"9905_story_stale_yaml.md",
"---\nname: Stale YAML\nagent: coder-1\n---\n\n# Story\n\nBody text.",
None,
);
crate::crdt_state::write_item_str(
"9905_story_stale_yaml",
"2_current",
Some("Stale YAML"),
None,
None,
None,
);
crate::crdt_state::set_agent("9905_story_stale_yaml", "coder-2".parse().ok());
let output = show_cmd_with_root(tmp.path(), "9905").unwrap();
assert!(
output.contains("**Agent:** coder-2"),
"show should display the live CRDT agent: {output}"
);
assert!(
!output.contains("**Agent:** coder-1"),
"show must not surface the stale YAML agent value: {output}"
);
}
#[test]
fn show_command_case_insensitive() {
let result = super::super::tests::try_cmd_addressed(
+1 -1
View File
@@ -90,7 +90,7 @@ pub(crate) fn build_status_from_items(
items: &[PipelineItem],
) -> String {
// Build a map from story_id → active AgentInfo for quick lookup.
let active_agents = agents.list_agents().unwrap_or_default();
let active_agents = agents.list_agents_nonblocking();
let active_map: HashMap<String, &crate::agents::AgentInfo> = active_agents
.iter()
.filter(|a| matches!(a.status, AgentStatus::Running | AgentStatus::Pending))
+263
View File
@@ -0,0 +1,263 @@
//! Deterministic, pure extraction of a size-capped digest from a Claude Code
//! session transcript (JSONL), with no LLM call involved.
/// Build a deterministic digest of a Claude Code session transcript.
///
/// Reads NDJSON `jsonl` (one Claude Code transcript event per line) and keeps
/// only:
/// - `user` entries whose `message.content` is a plain string (real user
/// text), included verbatim.
/// - `assistant` entries' final text content, included verbatim, plus each
/// `tool_use` block reduced to `name(one-line json args)`.
///
/// Excluded: `tool_result` content (found in `user` entries whose content is
/// an array), `thinking` blocks, and any event whose top-level `type` is not
/// `user` or `assistant` (e.g. `queue-operation`, `attachment`, `summary`,
/// `system`, `stream_event`). Sidechain entries (subagent turns) are also
/// excluded so the digest reflects only the main conversation thread.
///
/// Lines that fail to parse as JSON are skipped rather than treated as fatal.
///
/// When the joined digest exceeds `max_bytes`, entries are dropped from the
/// oldest end first so the most recent content survives the cap.
pub fn build_digest(jsonl: &str, max_bytes: usize) -> String {
let entries = extract_entries(jsonl);
cap_to_bytes(&entries, max_bytes)
}
/// Parse `jsonl` into an ordered list of digest lines, applying the
/// extraction/exclusion rules. Pure — no truncation is applied here.
fn extract_entries(jsonl: &str) -> Vec<String> {
let mut entries = Vec::new();
for line in jsonl.lines() {
let trimmed = line.trim();
if trimmed.is_empty() {
continue;
}
let Ok(value) = serde_json::from_str::<serde_json::Value>(trimmed) else {
continue;
};
if value.get("isSidechain").and_then(|v| v.as_bool()) == Some(true) {
continue;
}
match value.get("type").and_then(|t| t.as_str()) {
Some("user") => {
if let Some(text) = extract_user_text(&value) {
entries.push(format!("User: {text}"));
}
}
Some("assistant") => {
entries.extend(extract_assistant_lines(&value));
}
_ => {}
}
}
entries
}
/// Extract verbatim user text from a `user`-typed transcript entry.
///
/// Returns `None` when the message content is an array (tool results) rather
/// than a plain string — tool results are excluded from the digest.
fn extract_user_text(value: &serde_json::Value) -> Option<String> {
let content = value.get("message")?.get("content")?;
content.as_str().map(str::to_string)
}
/// Extract digest lines from an `assistant`-typed transcript entry: the final
/// text reply verbatim, followed by one line per `tool_use` block. `thinking`
/// blocks are skipped.
fn extract_assistant_lines(value: &serde_json::Value) -> Vec<String> {
let mut lines = Vec::new();
let Some(content) = value
.get("message")
.and_then(|m| m.get("content"))
.and_then(|c| c.as_array())
else {
return lines;
};
let mut text_parts = Vec::new();
for block in content {
match block.get("type").and_then(|t| t.as_str()) {
Some("text") => {
if let Some(text) = block.get("text").and_then(|t| t.as_str()) {
text_parts.push(text);
}
}
Some("tool_use") => {
let name = block.get("name").and_then(|n| n.as_str()).unwrap_or("tool");
let args = block
.get("input")
.map(serde_json::Value::to_string)
.unwrap_or_default();
lines.push(format!("Tool: {name}({args})"));
}
// "thinking" and any other block types are excluded.
_ => {}
}
}
if !text_parts.is_empty() {
lines.insert(0, format!("Assistant: {}", text_parts.join("\n")));
}
lines
}
/// Join `entries` with newlines, dropping the oldest entries first so the
/// digest stays within `max_bytes`. Always keeps at least the single newest
/// entry, even if it alone exceeds the cap.
fn cap_to_bytes(entries: &[String], max_bytes: usize) -> String {
let joined = entries.join("\n");
if joined.len() <= max_bytes {
return joined;
}
let mut kept: Vec<&str> = Vec::new();
let mut total = 0usize;
for entry in entries.iter().rev() {
let addition = entry.len() + if kept.is_empty() { 0 } else { 1 };
if total + addition > max_bytes && !kept.is_empty() {
break;
}
total += addition;
kept.push(entry);
}
kept.reverse();
kept.join("\n")
}
#[cfg(test)]
mod tests {
use super::*;
// -- extraction / exclusion rules ---------------------------------------
#[test]
fn extracts_plain_user_text_verbatim() {
let jsonl = r#"{"type":"user","message":{"role":"user","content":"hello there"}}"#;
let digest = build_digest(jsonl, 10_000);
assert_eq!(digest, "User: hello there");
}
#[test]
fn extracts_assistant_final_text_verbatim() {
let jsonl = r#"{"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":"here is my answer"}]}}"#;
let digest = build_digest(jsonl, 10_000);
assert_eq!(digest, "Assistant: here is my answer");
}
#[test]
fn extracts_tool_use_as_name_and_one_line_args() {
let jsonl = r#"{"type":"assistant","message":{"role":"assistant","content":[{"type":"tool_use","name":"Bash","input":{"command":"ls -la"}}]}}"#;
let digest = build_digest(jsonl, 10_000);
assert_eq!(digest, r#"Tool: Bash({"command":"ls -la"})"#);
assert!(!digest.contains('\n'), "tool args must be one line");
}
#[test]
fn excludes_tool_result_user_entries() {
// A user entry carrying a tool_result array (not plain string content)
// must be excluded entirely from the digest.
let jsonl = r#"{"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"abc","content":"file contents..."}]}}"#;
let digest = build_digest(jsonl, 10_000);
assert_eq!(digest, "", "tool_result entries must be excluded");
}
#[test]
fn excludes_thinking_blocks() {
let jsonl = r#"{"type":"assistant","message":{"role":"assistant","content":[{"type":"thinking","thinking":"let me consider..."},{"type":"text","text":"final answer"}]}}"#;
let digest = build_digest(jsonl, 10_000);
assert_eq!(digest, "Assistant: final answer");
assert!(!digest.contains("let me consider"));
}
#[test]
fn excludes_non_user_assistant_event_types() {
let jsonl = "\
{\"type\":\"queue-operation\",\"operation\":\"enqueue\",\"content\":\"noise\"}
{\"type\":\"attachment\",\"attachment\":{\"type\":\"skill_listing\"}}
{\"type\":\"summary\",\"summary\":\"irrelevant\"}
{\"type\":\"system\",\"content\":\"system noise\"}
{\"type\":\"user\",\"message\":{\"role\":\"user\",\"content\":\"real message\"}}";
let digest = build_digest(jsonl, 10_000);
assert_eq!(digest, "User: real message");
}
#[test]
fn excludes_sidechain_entries() {
let jsonl = r#"{"type":"user","isSidechain":true,"message":{"role":"user","content":"subagent chatter"}}"#;
let digest = build_digest(jsonl, 10_000);
assert_eq!(digest, "", "sidechain (subagent) entries must be excluded");
}
#[test]
fn deterministic_across_repeated_calls() {
let jsonl = "\
{\"type\":\"user\",\"message\":{\"role\":\"user\",\"content\":\"a\"}}
{\"type\":\"assistant\",\"message\":{\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"b\"}]}}";
let first = build_digest(jsonl, 10_000);
let second = build_digest(jsonl, 10_000);
assert_eq!(first, second);
}
// -- cap-truncation keeping newest ---------------------------------------
#[test]
fn cap_truncation_keeps_newest_entries() {
let jsonl = "\
{\"type\":\"user\",\"message\":{\"role\":\"user\",\"content\":\"oldest message\"}}
{\"type\":\"assistant\",\"message\":{\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"middle reply\"}]}}
{\"type\":\"user\",\"message\":{\"role\":\"user\",\"content\":\"newest message\"}}";
// Cap small enough to only fit the last entry.
let digest = build_digest(jsonl, 20);
assert_eq!(digest, "User: newest message");
assert!(!digest.contains("oldest"));
}
#[test]
fn cap_truncation_keeps_at_least_one_entry_even_if_oversized() {
let jsonl = r#"{"type":"user","message":{"role":"user","content":"this single message is longer than the cap"}}"#;
let digest = build_digest(jsonl, 5);
assert!(
digest.contains("this single message"),
"must keep the single newest entry even if it exceeds max_bytes: {digest}"
);
}
#[test]
fn no_truncation_when_under_cap() {
let jsonl = r#"{"type":"user","message":{"role":"user","content":"short"}}"#;
let digest = build_digest(jsonl, 10_000);
assert_eq!(digest, "User: short");
}
// -- malformed-line tolerance ---------------------------------------------
#[test]
fn tolerates_malformed_lines_between_valid_ones() {
let jsonl = "\
{\"type\":\"user\",\"message\":{\"role\":\"user\",\"content\":\"first\"}}
not json at all {{{
{\"type\":\"assistant\",\"message\":{\"role\":\"assistant\",\"content\":[{\"type\":\"text\",\"text\":\"second\"}]}}";
let digest = build_digest(jsonl, 10_000);
assert_eq!(digest, "User: first\nAssistant: second");
}
#[test]
fn tolerates_empty_lines() {
let jsonl = "\n{\"type\":\"user\",\"message\":{\"role\":\"user\",\"content\":\"hi\"}}\n\n";
let digest = build_digest(jsonl, 10_000);
assert_eq!(digest, "User: hi");
}
#[test]
fn empty_input_yields_empty_digest() {
assert_eq!(build_digest("", 10_000), "");
}
#[test]
fn all_malformed_yields_empty_digest() {
let jsonl = "garbage\nmore garbage\n{not valid";
assert_eq!(build_digest(jsonl, 10_000), "");
}
}
+296
View File
@@ -0,0 +1,296 @@
//! `compact` chat command: deterministically distill a Claude Code session
//! transcript into a size-capped digest, write it as a seed, and clear the
//! session so the next turn starts fresh with only the distilled context.
//!
//! Transport-agnostic: parsing (`extract_compact_command`) and orchestration
//! (`compact_session`) live here so every chat transport (Matrix, WhatsApp,
//! Slack, Discord) shares the same behaviour. Each transport wires this into
//! its own conversation-history type.
/// Pure extraction of a deterministic digest from session transcript lines.
pub mod digest;
/// Resolves a Claude Code session transcript's path on disk from its id.
pub(crate) mod transcript;
use std::collections::HashMap;
use std::fs;
use std::path::{Path, PathBuf};
use crate::chat::transport::matrix::RoomConversation;
use crate::chat::util::strip_bot_mention;
/// A parsed `compact` command.
#[derive(Debug, PartialEq)]
pub struct CompactCommand;
/// Parse a `compact` command from a raw message body.
///
/// Mirrors [`crate::chat::transport::matrix::reset::extract_reset_command`]:
/// strips the bot mention prefix and checks whether the command word is
/// `compact`. Returns `None` when the message is not a compact command.
pub fn extract_compact_command(
message: &str,
bot_name: &str,
bot_user_id: &str,
) -> Option<CompactCommand> {
let stripped = strip_bot_mention(message, bot_name, bot_user_id);
let trimmed = stripped
.trim()
.trim_start_matches(|c: char| !c.is_alphanumeric());
let cmd = match trimmed.split_once(char::is_whitespace) {
Some((c, _)) => c,
None => trimmed,
};
if cmd.eq_ignore_ascii_case("compact") {
Some(CompactCommand)
} else {
None
}
}
/// Result of a successful compaction: the digest text plus before/after
/// sizes (in bytes) for the reply message.
pub struct CompactOutcome {
/// The distilled digest text, written to the seed file and set as the
/// pending seed for the next spawned session.
pub digest: String,
/// Size in bytes of the original session transcript.
pub before_bytes: u64,
/// Size in bytes of the distilled digest.
pub after_bytes: u64,
/// Path the digest was written to.
pub seed_path: PathBuf,
}
/// Why a compaction attempt could not proceed.
#[derive(Debug)]
pub enum CompactError {
/// The session transcript file could not be read (missing, permissions,
/// or otherwise unreadable).
TranscriptUnreadable(String),
}
impl std::fmt::Display for CompactError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
CompactError::TranscriptUnreadable(e) => {
write!(f, "session transcript is unreadable: {e}")
}
}
}
}
/// Distill the session transcript at `transcript_path` into a size-capped
/// digest, and write it to a per-room seed file under
/// `project_root/.huskies/chat_seeds/`.
///
/// Callers are responsible for checking that a session is actually active
/// before calling this (AC5: no active session must be handled by the caller
/// without invoking this function, since there would be no transcript path
/// to resolve). Use [`transcript::transcript_path`] to resolve
/// `transcript_path` from a cwd + session_id.
pub fn compact_session(
project_root: &Path,
transcript_path: &Path,
room_key: &str,
max_bytes: usize,
) -> Result<CompactOutcome, CompactError> {
let jsonl = fs::read_to_string(transcript_path)
.map_err(|e| CompactError::TranscriptUnreadable(e.to_string()))?;
let before_bytes = jsonl.len() as u64;
let digest_text = digest::build_digest(&jsonl, max_bytes);
let after_bytes = digest_text.len() as u64;
let seed_path = seed_file_path(project_root, room_key);
if let Some(parent) = seed_path.parent() {
fs::create_dir_all(parent)
.map_err(|e| CompactError::TranscriptUnreadable(e.to_string()))?;
}
fs::write(&seed_path, &digest_text)
.map_err(|e| CompactError::TranscriptUnreadable(e.to_string()))?;
Ok(CompactOutcome {
digest: digest_text,
before_bytes,
after_bytes,
seed_path,
})
}
/// Handle a `compact` command for any transport that keys its conversation
/// history by a plain string (Discord channel id, Slack channel id, WhatsApp
/// phone number, …), rather than Matrix's `OwnedRoomId`.
///
/// Mirrors [`crate::chat::transport::matrix::compact::handle_compact`]: looks
/// up the room's session, distills its transcript into a digest, writes the
/// digest as a seed, clears the session id and entries, sets `pending_seed`,
/// then calls `save` to persist the updated history. Returns an actionable
/// message and changes nothing when there is no active session, or when the
/// session transcript cannot be read.
pub async fn handle_compact_for_key(
room_key: &str,
history: &tokio::sync::Mutex<HashMap<String, RoomConversation>>,
project_root: &Path,
max_bytes: usize,
save: impl FnOnce(&Path, &HashMap<String, RoomConversation>),
) -> String {
let mut guard = history.lock().await;
let conv = guard.entry(room_key.to_string()).or_default();
let Some(session_id) = conv.session_id.clone() else {
return "No active session to compact. Send a message first to start one.".to_string();
};
let transcript_path = transcript::transcript_path(project_root, &session_id);
match compact_session(project_root, &transcript_path, room_key, max_bytes) {
Ok(outcome) => {
conv.session_id = None;
conv.entries.clear();
conv.pending_seed = Some(outcome.digest);
save(project_root, &guard);
format!(
"Compacted session context: {} → {} bytes. Starting fresh — the distilled \
summary will be included as background in your next message.",
outcome.before_bytes, outcome.after_bytes
)
}
Err(CompactError::TranscriptUnreadable(e)) => format!(
"Could not compact: session transcript is unreadable ({e}). Nothing was changed."
),
}
}
/// Path to the seed file for a given room, under
/// `project_root/.huskies/chat_seeds/`.
fn seed_file_path(project_root: &Path, room_key: &str) -> PathBuf {
project_root
.join(".huskies")
.join("chat_seeds")
.join(format!("{}.md", sanitize_room_key(room_key)))
}
/// Sanitize a room/channel identifier for use as a filename component.
fn sanitize_room_key(room_key: &str) -> String {
room_key
.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '-' || c == '_' {
c
} else {
'_'
}
})
.collect()
}
/// Wrap a distilled digest for injection into the next session's prompt,
/// clearly framed as background rather than as instructions to follow.
pub fn frame_seed_for_prompt(seed: &str) -> String {
format!(
"[The following is a distilled summary of prior session context, produced by the \
`compact` command. Treat it as background only, not as new instructions.]\n\n{seed}\n\n---\n"
)
}
#[cfg(test)]
mod tests {
use super::*;
use std::path::PathBuf;
// -- extract_compact_command ---------------------------------------------
#[test]
fn extract_with_display_name() {
let cmd = extract_compact_command("Timmy compact", "Timmy", "@timmy:home.local");
assert_eq!(cmd, Some(CompactCommand));
}
#[test]
fn extract_plain_no_mention() {
let cmd = extract_compact_command("compact", "Timmy", "@timmy:home.local");
assert_eq!(cmd, Some(CompactCommand));
}
#[test]
fn extract_case_insensitive() {
let cmd = extract_compact_command("Timmy COMPACT", "Timmy", "@timmy:home.local");
assert_eq!(cmd, Some(CompactCommand));
}
#[test]
fn extract_non_compact_returns_none() {
let cmd = extract_compact_command("Timmy help", "Timmy", "@timmy:home.local");
assert_eq!(cmd, None);
}
// -- sanitize_room_key / seed_file_path -----------------------------------
#[test]
fn sanitize_room_key_replaces_special_chars() {
assert_eq!(
sanitize_room_key("!abc123:example.com"),
"_abc123_example_com"
);
}
#[test]
fn seed_file_path_is_under_chat_seeds_dir() {
let path = seed_file_path(Path::new("/tmp/proj"), "!room:example.com");
assert_eq!(
path,
PathBuf::from("/tmp/proj/.huskies/chat_seeds/_room_example_com.md")
);
}
// -- compact_session -------------------------------------------------------
#[test]
fn compact_session_writes_seed_and_reports_sizes() {
let tmp = tempfile::tempdir().unwrap();
let project_root = tmp.path().join("project");
std::fs::create_dir_all(&project_root).unwrap();
let jsonl = r#"{"type":"user","message":{"role":"user","content":"hello"}}
{"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":"hi there"}]}}"#;
let transcript_path = tmp.path().join("sess-1.jsonl");
std::fs::write(&transcript_path, jsonl).unwrap();
let outcome =
compact_session(&project_root, &transcript_path, "!room:example.com", 10_000).unwrap();
assert_eq!(outcome.digest, "User: hello\nAssistant: hi there");
assert_eq!(outcome.before_bytes, jsonl.len() as u64);
assert_eq!(outcome.after_bytes, outcome.digest.len() as u64);
assert!(outcome.seed_path.exists());
let written = std::fs::read_to_string(&outcome.seed_path).unwrap();
assert_eq!(written, outcome.digest);
}
#[test]
fn compact_session_errors_on_missing_transcript() {
let tmp = tempfile::tempdir().unwrap();
let project_root = tmp.path().join("project");
std::fs::create_dir_all(&project_root).unwrap();
let result = compact_session(
&project_root,
&tmp.path().join("no-such-session.jsonl"),
"room",
10_000,
);
assert!(matches!(result, Err(CompactError::TranscriptUnreadable(_))));
}
// -- frame_seed_for_prompt --------------------------------------------------
#[test]
fn frame_seed_for_prompt_marks_it_as_background() {
let framed = frame_seed_for_prompt("User: hi\nAssistant: hello");
assert!(framed.contains("distilled summary"));
assert!(framed.contains("User: hi\nAssistant: hello"));
}
}
+65
View File
@@ -0,0 +1,65 @@
//! Resolves the on-disk path to a Claude Code session transcript JSONL file.
use std::path::{Path, PathBuf};
/// Resolve the path to a Claude Code session transcript.
///
/// Claude Code stores each session's transcript at
/// `$HOME/.claude/projects/<mangled-cwd>/<session_id>.jsonl`, where
/// `<mangled-cwd>` is the absolute working directory with every `/` and `.`
/// replaced by `-` (e.g. `/workspace/.huskies/worktrees/1186` becomes
/// `-workspace--huskies-worktrees-1186`).
pub fn transcript_path(cwd: &Path, session_id: &str) -> PathBuf {
let home = std::env::var("HOME").unwrap_or_else(|_| "/home/huskies".to_string());
transcript_path_under_home(Path::new(&home), cwd, session_id)
}
/// Same as [`transcript_path`] but takes an explicit `$HOME` directory,
/// keeping the path-joining logic testable without mutating process env vars.
fn transcript_path_under_home(home: &Path, cwd: &Path, session_id: &str) -> PathBuf {
let mangled = mangle_cwd(&cwd.to_string_lossy());
home.join(".claude")
.join("projects")
.join(mangled)
.join(format!("{session_id}.jsonl"))
}
/// Replace every `/` and `.` in an absolute path string with `-`, matching
/// the directory-naming convention Claude Code uses under `~/.claude/projects/`.
fn mangle_cwd(cwd: &str) -> String {
cwd.chars()
.map(|c| if c == '/' || c == '.' { '-' } else { c })
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn mangles_worktree_path_matching_observed_convention() {
// Verified against this very worktree's own transcript directory.
assert_eq!(
mangle_cwd("/workspace/.huskies/worktrees/1186"),
"-workspace--huskies-worktrees-1186"
);
}
#[test]
fn mangles_simple_home_path() {
assert_eq!(mangle_cwd("/home/huskies"), "-home-huskies");
}
#[test]
fn transcript_path_joins_home_projects_dir_and_session_file() {
let path = transcript_path_under_home(
Path::new("/home/testuser"),
Path::new("/workspace/proj"),
"abc-123",
);
assert_eq!(
path,
PathBuf::from("/home/testuser/.claude/projects/-workspace-proj/abc-123.jsonl")
);
}
}
+78 -11
View File
@@ -19,6 +19,7 @@
use crate::slog;
use std::collections::HashMap;
use std::pin::Pin;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio::sync::{mpsc, watch};
@@ -47,6 +48,20 @@ enum SessionMsg {
struct SessionHandle {
tx: mpsc::UnboundedSender<SessionMsg>,
/// `true` whenever the session task is coalescing, running, or draining a
/// batch — i.e. there is something for [`ChatDispatcher::stop`] to cancel.
/// `false` while the task is idle in Phase 1, waiting for a message.
active: Arc<AtomicBool>,
}
/// Result of a [`ChatDispatcher::stop`] call.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum StopOutcome {
/// A turn was actively coalescing, running, or draining pending messages
/// for the session, and a cancellation was sent.
Cancelled,
/// The session doesn't exist, or exists but has no turn in flight.
NothingRunning,
}
/// Coalescing, serialising dispatcher for chat-to-LLM message routing.
@@ -79,8 +94,14 @@ impl ChatDispatcher {
let coalesce_ms = self.coalesce_ms;
let handle = guard.entry(session_key.clone()).or_insert_with(|| {
let (tx, rx) = mpsc::unbounded_channel();
tokio::spawn(session_task(session_key.clone(), rx, coalesce_ms));
SessionHandle { tx }
let active = Arc::new(AtomicBool::new(false));
tokio::spawn(session_task(
session_key.clone(),
rx,
coalesce_ms,
Arc::clone(&active),
));
SessionHandle { tx, active }
});
let _ = handle.tx.send(SessionMsg::UserMessage {
text: message,
@@ -90,15 +111,19 @@ impl ChatDispatcher {
/// Stop the active LLM run for `session_key` and clear its pending queue.
///
/// Returns `true` if the session existed (whether or not anything was
/// actually running), `false` if no session for that key has been created.
pub fn stop(&self, session_key: &str) -> bool {
/// Returns [`StopOutcome::Cancelled`] and sends the cancellation only when
/// a turn is actually coalescing, running, or draining pending messages;
/// otherwise returns [`StopOutcome::NothingRunning`] without sending
/// anything (covers both "no session was ever created" and "session
/// exists but is idle").
pub fn stop(&self, session_key: &str) -> StopOutcome {
let guard = self.sessions.lock().unwrap();
if let Some(handle) = guard.get(session_key) {
let _ = handle.tx.send(SessionMsg::Stop);
true
} else {
false
match guard.get(session_key) {
Some(handle) if handle.active.load(Ordering::SeqCst) => {
let _ = handle.tx.send(SessionMsg::Stop);
StopOutcome::Cancelled
}
_ => StopOutcome::NothingRunning,
}
}
}
@@ -119,6 +144,7 @@ async fn session_task(
session_key: String,
mut rx: mpsc::UnboundedReceiver<SessionMsg>,
coalesce_ms: u64,
active: Arc<AtomicBool>,
) {
let coalesce_dur = Duration::from_millis(coalesce_ms);
@@ -131,6 +157,9 @@ async fn session_task(
Some(SessionMsg::UserMessage { text, factory }) => break (text, factory),
}
};
// From here until we loop back to Phase 1, there is something in
// flight (coalescing, running, or draining) for `stop()` to cancel.
active.store(true, Ordering::SeqCst);
// ── Phase 2: coalesce window (debounce) ──────────────────────────────
let mut batch: Vec<String> = vec![first_text];
@@ -160,6 +189,7 @@ async fn session_task(
}
if batch.is_empty() {
active.store(false, Ordering::SeqCst);
continue; // Stop received during coalesce — restart
}
@@ -209,6 +239,7 @@ async fn session_task(
}
if stopped || pending_texts.is_empty() {
active.store(false, Ordering::SeqCst);
break; // back to Phase 1
}
@@ -351,7 +382,12 @@ mod tests {
);
// Stop immediately.
dispatcher.stop(&session);
let outcome = dispatcher.stop(&session);
assert_eq!(
outcome,
StopOutcome::Cancelled,
"a run was active, so stop must report Cancelled"
);
// Wait longer than the run would have taken if not stopped.
tokio::time::sleep(Duration::from_millis(700)).await;
@@ -364,4 +400,35 @@ mod tests {
"stop should discard pending; got {count} spawns"
);
}
/// Stopping a session key that was never submitted reports NothingRunning
/// rather than silently no-opping.
#[tokio::test]
async fn stop_on_unknown_session_reports_nothing_running() {
let dispatcher = ChatDispatcher::new(30);
assert_eq!(
dispatcher.stop("never-seen-room"),
StopOutcome::NothingRunning
);
}
/// Stopping a session that exists but has no active turn (its one run
/// already completed) reports NothingRunning, not Cancelled.
#[tokio::test]
async fn stop_on_idle_session_reports_nothing_running() {
let spawn_count = Arc::new(AtomicUsize::new(0));
let dispatcher = ChatDispatcher::new(20);
let session = "room4".to_string();
dispatcher.submit(
session.clone(),
"hello".to_string(),
make_factory(Arc::clone(&spawn_count), 20),
);
// Wait long enough for the coalesce window and the run to finish.
tokio::time::sleep(Duration::from_millis(200)).await;
assert_eq!(dispatcher.stop(&session), StopOutcome::NothingRunning);
}
}
+1
View File
@@ -128,6 +128,7 @@ mod tests {
content: "hi there!".to_string(),
},
],
..Default::default()
},
);
+2
View File
@@ -6,6 +6,8 @@
/// Bot command registry and dispatch — parses and routes incoming chat messages.
pub mod commands;
/// `compact` command: deterministic session-transcript distillation and seeding.
pub mod compact;
/// Protocol-agnostic chat dispatcher — coalesce window and per-session serial lock.
pub mod dispatcher;
/// Chat history utilities — loading and serialising conversation history.
+139 -41
View File
@@ -47,8 +47,12 @@ pub(super) async fn handle_incoming_message(
// If there is a pending permission prompt for this channel, interpret the
// message as a yes/no response.
{
let mut pending = ctx.services.pending_perm_replies.lock().await;
if let Some(tx) = pending.remove(channel) {
if let Some(tx) = ctx
.services
.pending_perm_replies
.resolve_oldest(channel)
.await
{
let decision = if is_permission_approval(message) {
PermissionDecision::Approve
} else {
@@ -161,27 +165,6 @@ pub(super) async fn handle_incoming_message(
return;
}
if crate::chat::transport::matrix::rebuild::extract_rebuild_command(
message,
&ctx.services.bot_name,
&ctx.services.bot_user_id,
)
.is_some()
{
slog!("[discord] Handling rebuild command from {user} in {channel}");
let ack = "Rebuilding server… this may take a moment.";
let _ = ctx.transport.send_message(channel, ack, "").await;
let response = crate::chat::transport::matrix::rebuild::handle_rebuild(
&ctx.services.bot_name,
&ctx.services.project_root,
&ctx.services.agents,
)
.await;
let response = markdown_to_discord(&response);
let _ = ctx.transport.send_message(channel, &response, "").await;
return;
}
if let Some(rmtree_cmd) = crate::chat::transport::matrix::rmtree::extract_rmtree_command(
message,
&ctx.services.bot_name,
@@ -233,6 +216,27 @@ pub(super) async fn handle_incoming_message(
return;
}
if crate::chat::compact::extract_compact_command(
message,
&ctx.services.bot_name,
&ctx.services.bot_user_id,
)
.is_some()
{
slog!("[discord] Handling compact command from {user} in {channel}");
let response = crate::chat::compact::handle_compact_for_key(
channel,
&ctx.history,
&ctx.services.project_root,
8_000,
save_discord_history,
)
.await;
let response = markdown_to_discord(&response);
let _ = ctx.transport.send_message(channel, &response, "").await;
return;
}
if let Some(start_cmd) = crate::chat::transport::matrix::start::extract_start_command(
message,
&ctx.services.bot_name,
@@ -362,6 +366,7 @@ async fn handle_llm_message(ctx: &DiscordContext, channel: &str, user: &str, use
&project_root_str,
resume_session_id.as_deref(),
None,
None,
&mut cancel_rx,
move |token| {
let mut buf = buffer_for_callback.lock().unwrap();
@@ -377,14 +382,14 @@ async fn handle_llm_message(ctx: &DiscordContext, channel: &str, user: &str, use
);
tokio::pin!(chat_fut);
// Lock the permission receiver for the duration of this chat session.
let mut perm_rx_guard = ctx.services.perm_rx.lock().await;
// Register as a permission responder for the duration of this chat turn.
let (_perm_guard, mut perm_rx) = ctx.services.permission_registry.register();
let result = loop {
tokio::select! {
r = &mut chat_fut => break r,
Some(perm_fwd) = perm_rx_guard.recv() => {
Some(perm_fwd) = perm_rx.recv() => {
let prompt_msg = format!(
"**Permission Request**\n\nTool: `{}`\n```json\n{}\n```\n\nReply **yes** to approve or **no** to deny.",
perm_fwd.tool_name,
@@ -394,20 +399,22 @@ async fn handle_llm_message(ctx: &DiscordContext, channel: &str, user: &str, use
let formatted = markdown_to_discord(&prompt_msg);
let _ = ctx.transport.send_message(channel, &formatted, "").await;
// Keyed by request_id (not just channel) so a second
// concurrent request doesn't drop the first's sender.
ctx.services
.pending_perm_replies
.lock()
.await
.insert(channel.to_string(), perm_fwd.response_tx);
.insert(channel.to_string(), perm_fwd.request_id.clone(), perm_fwd.response_tx)
.await;
// Spawn a timeout task: auto-deny if the user does not respond.
let pending = Arc::clone(&ctx.services.pending_perm_replies);
let timeout_channel = channel.to_string();
let timeout_request_id = perm_fwd.request_id.clone();
let timeout_transport = Arc::clone(&ctx.transport) as Arc<dyn ChatTransport>;
let timeout_secs = ctx.services.permission_timeout_secs;
tokio::spawn(async move {
tokio::time::sleep(std::time::Duration::from_secs(timeout_secs)).await;
if let Some(tx) = pending.lock().await.remove(&timeout_channel) {
if let Some(tx) = pending.remove_by_request_id(&timeout_channel, &timeout_request_id).await {
let _ = tx.send(PermissionDecision::Deny);
let msg = "Permission request timed out — denied (fail-closed).";
let _ = timeout_transport.send_message(&timeout_channel, msg, "").await;
@@ -416,7 +423,6 @@ async fn handle_llm_message(ctx: &DiscordContext, channel: &str, user: &str, use
}
}
};
drop(perm_rx_guard);
// Flush remaining text.
let remaining = buffer.lock().unwrap().trim().to_string();
@@ -426,6 +432,7 @@ async fn handle_llm_message(ctx: &DiscordContext, channel: &str, user: &str, use
Ok(ClaudeCodeResult {
messages,
session_id,
..
}) => {
let reply = if !remaining.is_empty() {
let _ = msg_tx.send(remaining.clone());
@@ -524,16 +531,6 @@ mod tests {
assert!(result.unwrap().contains("Pipeline Status"));
}
#[test]
fn rebuild_command_extracted_from_discord_message() {
let result = crate::chat::transport::matrix::rebuild::extract_rebuild_command(
"Huskies rebuild",
"Huskies",
"discord-bot",
);
assert!(result.is_some(), "'Huskies rebuild' should be recognised");
}
#[test]
fn reset_command_extracted_from_discord_message() {
let result = crate::chat::transport::matrix::reset::extract_reset_command(
@@ -593,6 +590,7 @@ mod tests {
sender: "user123".to_string(),
content: "previous message".to_string(),
}],
..Default::default()
},
);
m
@@ -653,4 +651,104 @@ mod tests {
"assembled prompt must contain user message; got: {prompt}"
);
}
/// Regression test for story 1192: `compact` must not be swallowed by the
/// registry placeholder. Drives the real entrypoint
/// (`handle_incoming_message`, not `handle_compact_for_key` directly) so a
/// future regression in the dispatch order is caught, and asserts the
/// real handler ran: reply mentions the byte-size confirmation, and the
/// history's session_id/entries were cleared with pending_seed set.
#[tokio::test]
async fn compact_command_runs_through_full_dispatch_and_clears_session() {
use crate::chat::transport::matrix::{
ConversationEntry, ConversationRole, RoomConversation,
};
use std::collections::HashSet;
use std::sync::Arc;
let channel = "555444333";
let session_id = "sess-discord-compact";
let tmp = tempfile::tempdir().unwrap();
let project_root = tmp.path().join("project");
std::fs::create_dir_all(&project_root).unwrap();
let home = tempfile::tempdir().unwrap();
// SAFETY: this test owns HOME for its duration; no other test in this
// process reads HOME concurrently with this call.
unsafe {
std::env::set_var("HOME", home.path());
}
let transcript_dir =
crate::chat::compact::transcript::transcript_path(&project_root, session_id)
.parent()
.unwrap()
.to_path_buf();
std::fs::create_dir_all(&transcript_dir).unwrap();
let jsonl = r#"{"type":"user","message":{"role":"user","content":"hello"}}
{"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":"hi there"}]}}"#;
std::fs::write(transcript_dir.join(format!("{session_id}.jsonl")), jsonl).unwrap();
let history: DiscordConversationHistory = Arc::new(TokioMutex::new({
let mut m = HashMap::new();
m.insert(
channel.to_string(),
RoomConversation {
session_id: Some(session_id.to_string()),
entries: vec![ConversationEntry {
role: ConversationRole::User,
sender: "user123".to_string(),
content: "hi".to_string(),
}],
pending_seed: None,
last_compact_suggested_at_ms: None,
},
);
m
}));
let mut server = mockito::Server::new_async().await;
let mock = server
.mock("POST", format!("/channels/{channel}/messages").as_str())
.match_body(mockito::Matcher::Regex(
"Compacted session context".to_string(),
))
.with_body(r#"{"id": "1"}"#)
.create_async()
.await;
let services =
crate::services::Services::new_test(project_root.clone(), "Huskies".to_string());
let ctx = DiscordContext {
services,
bot_token: "test-token".to_string(),
transport: Arc::new(DiscordTransport::with_api_base(
"test-token".to_string(),
server.url(),
)),
history: history.clone(),
history_size: 20,
channel_ids: HashSet::new(),
allowed_users: HashSet::new(),
};
handle_incoming_message(&ctx, channel, "user123", "compact").await;
mock.assert_async().await;
let guard = history.lock().await;
let conv = guard.get(channel).unwrap();
assert!(
conv.session_id.is_none(),
"session_id must be cleared after compact"
);
assert!(
conv.entries.is_empty(),
"entries must be cleared after compact"
);
assert_eq!(
conv.pending_seed.as_deref(),
Some("User: hello\nAssistant: hi there"),
"pending_seed must hold the distilled digest"
);
}
}
@@ -56,6 +56,7 @@ mod tests {
content: "hi there!".to_string(),
},
],
..Default::default()
},
);
+3 -1
View File
@@ -34,8 +34,10 @@ impl DiscordTransport {
}
}
/// Creates a `DiscordTransport` pointed at a custom API base URL, for
/// tests that mock the Discord API instead of hitting discord.com.
#[cfg(test)]
fn with_api_base(bot_token: String, api_base: String) -> Self {
pub(crate) fn with_api_base(bot_token: String, api_base: String) -> Self {
Self {
bot_token,
client: reqwest::Client::new(),
@@ -111,6 +111,7 @@ pub async fn handle_assign(
// Check whether a coder is already running on this story.
let running_coders: Vec<_> = agents
.list_agents()
.await
.unwrap_or_default()
.into_iter()
.filter(|a| {
+80 -22
View File
@@ -94,6 +94,14 @@ pub struct BotContext {
/// The `new project` command writes here so HTTP handlers see the new entry
/// immediately without requiring a gateway restart. `None` in standalone mode.
pub gateway_projects_store: Option<Arc<RwLock<BTreeMap<String, ProjectEntry>>>>,
/// In gateway mode: shared configured release channels (story 1169).
///
/// Read by the `pull <channel>` command to resolve a channel's
/// `base_url`, pinned `pubkey`, and optional `bearer_token`. `None` in
/// standalone mode.
pub gateway_channels_store: Option<
Arc<RwLock<BTreeMap<String, crate::service::gateway::config::ReleaseChannelConfig>>>,
>,
/// Bounded FIFO set of already-handled incoming event IDs.
///
/// The Matrix sync loop can replay events on reconnect. This set ensures
@@ -109,6 +117,23 @@ pub struct BotContext {
/// configured room. Updated atomically on every `on_room_message` call so
/// the `health` command can detect a stale or dead sync loop.
pub last_matrix_event_ms: Arc<AtomicI64>,
/// Optional model override from bot.toml. Passed as `--model` to the
/// `claude` CLI when set.
pub model: Option<String>,
/// Maximum size in bytes of the digest the `compact` command writes as a
/// seed file. From `bot.toml`'s `compact_seed_max_bytes`.
pub compact_seed_max_bytes: usize,
/// `cache_read_input_tokens` threshold above which the bot suggests
/// running `compact` after a turn. From `bot.toml`'s
/// `cache_read_suggest_threshold`.
pub cache_read_suggest_threshold: u64,
/// Minimum seconds between repeated `compact` suggestions for the same
/// room. From `bot.toml`'s `compact_suggest_cooldown_secs`.
pub compact_suggest_cooldown_secs: i64,
/// Seconds a turn may spend on tool calls before emitting any
/// user-facing text before the bot posts a "digging in" notice. From
/// `bot.toml`'s `digging_in_threshold_secs`.
pub digging_in_threshold_secs: u64,
}
impl BotContext {
@@ -136,11 +161,16 @@ impl BotContext {
pub async fn active_project_url(&self) -> Option<String> {
let ap = self.gateway_active_project.as_ref()?;
let name = ap.read().await.clone();
self.project_url(&name).await
}
/// Return the base URL for a named project from the live gateway store.
pub async fn project_url(&self, name: &str) -> Option<String> {
let store = self.gateway_projects_store.as_ref()?;
store
.read()
.await
.get(&name)
.get(name)
.and_then(|entry| entry.url.clone())
}
@@ -151,18 +181,28 @@ impl BotContext {
/// `rpc_response` frame. Returns an error message string if the
/// connection or command fails.
pub async fn proxy_bot_command(&self, command: &str, args: &str) -> Option<String> {
let base_url = self.active_project_url().await?;
Some(Self::run_proxy_bot_command(&base_url, command, args).await)
}
/// Run the `bot.command` WebSocket RPC call against `base_url` and return
/// the Markdown response, or an error message string on failure.
///
/// `pub(crate)` so callers that need to target a project by name (rather
/// than always the active one, as [`Self::proxy_bot_command`] does — e.g.
/// the `status <project>` command) can resolve their own URL and reuse
/// this transport logic.
pub(crate) async fn run_proxy_bot_command(base_url: &str, command: &str, args: &str) -> String {
use futures::{SinkExt, StreamExt};
use tokio_tungstenite::tungstenite::Message as WsMsg;
let base_url = self.active_project_url().await?;
// Convert http(s):// → ws(s)://
let ws_base = if let Some(rest) = base_url.strip_prefix("https://") {
format!("wss://{rest}")
} else if let Some(rest) = base_url.strip_prefix("http://") {
format!("ws://{rest}")
} else {
base_url.clone()
base_url.to_string()
};
let ws_url = format!("{ws_base}/ws");
@@ -177,25 +217,36 @@ impl BotContext {
});
let request_text = match serde_json::to_string(&request) {
Ok(t) => t,
Err(e) => return Some(format!("Failed to serialize RPC request: {e}")),
Err(e) => return format!("Failed to serialize RPC request: {e}"),
};
let ws_stream = match tokio_tungstenite::connect_async(&ws_url).await {
Ok((stream, _)) => stream,
Err(e) => {
return Some(format!(
"Failed to connect to project server at {ws_url}: {e}"
));
let connect_timeout = std::time::Duration::from_secs(5);
let ws_stream = match tokio::time::timeout(
connect_timeout,
tokio_tungstenite::connect_async(&ws_url),
)
.await
{
Ok(Ok((stream, _))) => stream,
Ok(Err(e)) => {
return format!("Failed to connect to project server at {ws_url}: {e}");
}
Err(_) => {
return format!(
"Project server at {ws_url} is unreachable (connect timed out after {connect_timeout:?})"
);
}
};
let (mut sink, mut stream) = ws_stream.split();
if let Err(e) = sink.send(WsMsg::Text(request_text.into())).await {
return Some(format!("Failed to send RPC request: {e}"));
return format!("Failed to send RPC request: {e}");
}
while let Some(msg) = stream.next().await {
let response_timeout = std::time::Duration::from_secs(30);
let deadline = tokio::time::Instant::now() + response_timeout;
while let Ok(Some(msg)) = tokio::time::timeout_at(deadline, stream.next()).await {
match msg {
Ok(WsMsg::Text(text)) => {
let Ok(frame) = serde_json::from_str::<serde_json::Value>(&text) else {
@@ -219,24 +270,24 @@ impl BotContext {
.and_then(|r| r.get("response"))
.and_then(|v| v.as_str())
.map(String::from)
.or_else(|| {
Some("Command succeeded with no response text".to_string())
.unwrap_or_else(|| {
"Command succeeded with no response text".to_string()
});
} else {
let err = frame
.get("error")
.and_then(|v| v.as_str())
.unwrap_or("unknown error");
return Some(format!("Project server command failed: {err}"));
return format!("Project server command failed: {err}");
}
}
Ok(WsMsg::Close(_)) => break,
Err(e) => return Some(format!("WebSocket error: {e}")),
Err(e) => return format!("WebSocket error: {e}"),
_ => continue,
}
}
Some("Connection closed before receiving command response".to_string())
"Project server did not respond in time (connection closed or timed out)".to_string()
}
}
@@ -247,9 +298,9 @@ impl BotContext {
#[cfg(test)]
mod tests {
use super::*;
use crate::service::permission_router::{PendingPermReplies, ResponderRegistry};
use std::collections::HashMap;
use std::path::PathBuf;
use tokio::sync::mpsc;
fn make_user_id(s: &str) -> OwnedUserId {
s.parse().unwrap()
@@ -257,15 +308,14 @@ mod tests {
/// Build a test `Services` bundle with the given project root.
fn test_services(project_root: PathBuf) -> Arc<Services> {
let (_perm_tx, perm_rx) = mpsc::unbounded_channel();
Arc::new(Services {
project_root,
agents: Arc::new(crate::agents::AgentPool::new_test(3000)),
bot_name: "Assistant".to_string(),
bot_user_id: "@bot:example.com".to_string(),
ambient_rooms: Arc::new(std::sync::Mutex::new(std::collections::HashSet::new())),
perm_rx: Arc::new(TokioMutex::new(perm_rx)),
pending_perm_replies: Arc::new(TokioMutex::new(HashMap::new())),
permission_registry: ResponderRegistry::new(),
pending_perm_replies: PendingPermReplies::new(),
permission_timeout_secs: 120,
status: Arc::new(crate::service::status::StatusBroadcaster::new()),
chat_dispatcher: Arc::new(crate::chat::dispatcher::ChatDispatcher::new(1_500)),
@@ -300,11 +350,17 @@ mod tests {
)),
gateway_active_project,
gateway_projects_store,
gateway_channels_store: None,
handled_incoming_event_ids: Arc::new(TokioMutex::new(SeenEventIds::new(
SEEN_EVENT_IDS_CAP,
))),
gateway_port: None,
last_matrix_event_ms: Arc::new(AtomicI64::new(chrono::Utc::now().timestamp_millis())),
model: None,
compact_seed_max_bytes: 8_000,
cache_read_suggest_threshold: 50_000,
compact_suggest_cooldown_secs: 3_600,
digging_in_threshold_secs: 15,
}
}
@@ -470,6 +526,7 @@ mod tests {
auth_token: None,
ssh_port: None,
host_path: None,
expected_node_id: None,
},
)])));
let ctx = test_bot_context(services, Some(Arc::clone(&active)), Some(store));
@@ -514,6 +571,7 @@ mod tests {
auth_token: None,
ssh_port: None,
host_path: None,
expected_node_id: None,
},
);
+106 -14
View File
@@ -1,20 +1,38 @@
//! Matrix message formatting — Markdown-to-HTML conversion and startup announcements.
use pulldown_cmark::{Options, Parser, html};
/// Format the startup greeting the bot sends to each room when it comes online.
/// Format the startup greeting the bot sends to each room when it comes online
/// via a normal (non-trampoline) process start.
///
/// Uses the bot's configured display name so the message reads naturally
/// (e.g. "Timmy is online.").
pub fn format_startup_announcement(bot_name: &str) -> String {
format!("{bot_name} is online.")
/// Uses the bot's configured display name plus the same version/git-hash/model
/// details as [`format_gateway_ready_announcement`], so a plain restart carries
/// the same diagnostic information as a trampoline-triggered rebuild instead of
/// just "Timmy is online.". `git_hash` should already have been resolved to
/// `"unknown"` by the caller if unavailable. `model` is `None` when `bot.toml`
/// has no `model` override configured, in which case the CLI's own default
/// applies.
pub fn format_startup_announcement(bot_name: &str, git_hash: &str, model: Option<&str>) -> String {
format!(
"{bot_name} is online — gateway {} ({git_hash}) — model: {}",
env!("CARGO_PKG_VERSION"),
model.unwrap_or("default")
)
}
/// Format the ready announcement sent after a successful gateway trampoline restart.
///
/// Returns "gateway X.Y.Z ready" using the compiled-in crate version so the
/// operator can confirm which binary is running after a rebuild.
pub fn format_gateway_ready_announcement() -> String {
format!("gateway {} ready", env!("CARGO_PKG_VERSION"))
/// Returns "gateway X.Y.Z (git_hash) ready — model: <model>" using the compiled-in
/// crate version (same source as `/api/version`), the given git hash, and the
/// configured chat model, so the operator can confirm which binary and model are
/// running after a rebuild. `git_hash` should already have been resolved to
/// `"unknown"` by the caller if unavailable. `model` is `None` when `bot.toml`
/// has no `model` override configured, in which case the CLI's own default applies.
pub fn format_gateway_ready_announcement(git_hash: &str, model: Option<&str>) -> String {
format!(
"gateway {} ({git_hash}) ready — model: {}",
env!("CARGO_PKG_VERSION"),
model.unwrap_or("default")
)
}
/// Format the failure announcement sent when the trampoline rolls back to the
@@ -158,15 +176,89 @@ mod tests {
#[test]
fn startup_announcement_uses_bot_name() {
assert_eq!(format_startup_announcement("Timmy"), "Timmy is online.");
let msg = format_startup_announcement("Timmy", "abc1234", Some("claude-sonnet-5"));
assert!(
msg.starts_with("Timmy is online"),
"expected bot name: {msg}"
);
}
#[test]
fn startup_announcement_uses_configured_display_name_not_hardcoded() {
assert_eq!(format_startup_announcement("HAL"), "HAL is online.");
assert_eq!(
format_startup_announcement("Assistant"),
"Assistant is online."
let msg = format_startup_announcement("HAL", "abc1234", Some("claude-sonnet-5"));
assert!(msg.starts_with("HAL is online"), "expected bot name: {msg}");
let msg = format_startup_announcement("Assistant", "abc1234", Some("claude-sonnet-5"));
assert!(
msg.starts_with("Assistant is online"),
"expected bot name: {msg}"
);
}
#[test]
fn startup_announcement_includes_version_hash_and_model() {
let msg = format_startup_announcement("Timmy", "abc1234", Some("claude-sonnet-5"));
assert!(
msg.contains(env!("CARGO_PKG_VERSION")),
"expected crate version in announcement: {msg}"
);
assert!(msg.contains("abc1234"), "expected git hash: {msg}");
assert!(
msg.contains("claude-sonnet-5"),
"expected configured model: {msg}"
);
}
#[test]
fn startup_announcement_renders_unknown_git_hash_without_suppressing_message() {
let msg = format_startup_announcement("Timmy", "unknown", Some("claude-sonnet-5"));
assert!(
msg.contains("unknown"),
"expected 'unknown' fallback for missing git hash: {msg}"
);
assert!(
msg.starts_with("Timmy is online"),
"message must still be sent: {msg}"
);
}
#[test]
fn startup_announcement_falls_back_to_default_when_model_absent() {
let msg = format_startup_announcement("Timmy", "abc1234", None);
assert!(
msg.contains("default"),
"expected 'default' fallback when no model configured: {msg}"
);
}
#[test]
fn gateway_ready_announcement_includes_version_hash_and_model() {
let msg = format_gateway_ready_announcement("abc1234", Some("claude-sonnet-5"));
assert!(
msg.contains(env!("CARGO_PKG_VERSION")),
"expected crate version in announcement: {msg}"
);
assert!(msg.contains("abc1234"), "expected git hash: {msg}");
assert!(
msg.contains("claude-sonnet-5"),
"expected configured model: {msg}"
);
}
#[test]
fn gateway_ready_announcement_renders_unknown_git_hash_cleanly() {
let msg = format_gateway_ready_announcement("unknown", Some("claude-sonnet-5"));
assert!(
msg.contains("unknown"),
"expected 'unknown' fallback for missing git hash: {msg}"
);
}
#[test]
fn gateway_ready_announcement_falls_back_to_default_when_model_absent() {
let msg = format_gateway_ready_announcement("abc1234", None);
assert!(
msg.contains("default"),
"expected 'default' fallback when no model configured: {msg}"
);
}
@@ -36,6 +36,17 @@ pub struct RoomConversation {
pub session_id: Option<String>,
/// Rolling conversation entries (used for turn counting and persistence).
pub entries: Vec<ConversationEntry>,
/// A distilled digest produced by the `compact` command, waiting to be
/// injected as background context into the next spawned session's
/// prompt. Cleared immediately after the first turn that uses it, so it
/// is never re-injected.
#[serde(skip_serializing_if = "Option::is_none")]
pub pending_seed: Option<String>,
/// Timestamp (ms since Unix epoch) of the last time this room was sent a
/// "consider running `compact`" suggestion, used to rate-limit repeat
/// suggestions.
#[serde(skip_serializing_if = "Option::is_none")]
pub last_compact_suggested_at_ms: Option<i64>,
}
/// Per-room conversation state, keyed by room ID (serialised as string).
@@ -1,12 +1,14 @@
//! Matrix handle_message — runs the LLM turn for a verified incoming message and
//! streams the assistant reply back to the room.
use crate::chat::ChatTransport;
use crate::chat::util::drain_complete_paragraphs;
use crate::llm::providers::claude_code::{ClaudeCodeProvider, ClaudeCodeResult};
use crate::llm::providers::claude_code::{CANCELLED, ClaudeCodeProvider, ClaudeCodeResult};
use crate::slog;
use matrix_sdk::ruma::OwnedRoomId;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use tokio::sync::watch;
use super::super::context::BotContext;
@@ -15,6 +17,33 @@ use super::super::history::{ConversationEntry, ConversationRole, save_history};
use super::format_user_prompt;
/// Text posted to the room by [`spawn_digging_in_watcher`] when a turn runs
/// long without emitting any user-facing text.
const DIGGING_IN_MESSAGE: &str = "Still digging in — this turn is taking a bit longer than usual.";
/// Spawns a background watcher that posts a single "digging in" notice to
/// `room_id` if `threshold` elapses before `sent_any_text` becomes `true`.
///
/// Callers must abort the returned [`tokio::task::JoinHandle`] once the turn
/// completes so a turn that finishes under the threshold — with or without
/// text — never triggers the notice after the fact.
pub(in crate::chat::transport::matrix::bot) fn spawn_digging_in_watcher(
transport: Arc<dyn ChatTransport>,
room_id: String,
sent_any_text: Arc<AtomicBool>,
threshold: Duration,
) -> tokio::task::JoinHandle<()> {
tokio::spawn(async move {
tokio::time::sleep(threshold).await;
if !sent_any_text.load(Ordering::Relaxed) {
let html = markdown_to_html(DIGGING_IN_MESSAGE);
let _ = transport
.send_message(&room_id, DIGGING_IN_MESSAGE, &html)
.await;
}
})
}
pub(in crate::chat::transport::matrix::bot) async fn handle_message(
room_id_str: String,
room_id: OwnedRoomId,
@@ -31,6 +60,23 @@ pub(in crate::chat::transport::matrix::bot) async fn handle_message(
guard.get(&room_id).and_then(|conv| conv.session_id.clone())
};
// Drain any pending `compact` seed for this room so it is injected into
// the prompt exactly once, then persist the cleared state immediately —
// a crash mid-turn must not cause it to be re-injected on the next try.
let pending_seed: Option<String> = {
let mut guard = ctx.history.lock().await;
let conv = guard.entry(room_id.clone()).or_default();
let seed = conv.pending_seed.take();
if seed.is_some() {
save_history(&ctx.services.project_root, &guard);
}
seed
};
let seed_prefix = pending_seed
.as_deref()
.map(crate::chat::compact::frame_seed_for_prompt)
.unwrap_or_default();
// Pull new pipeline-transition events from the CRDT event log for this
// persona and atomically advance the high-water marks so the same events
// are not re-injected on the next turn. All transports share the same
@@ -49,7 +95,7 @@ pub(in crate::chat::transport::matrix::bot) async fn handle_message(
String::new()
};
let prompt = format!(
"{event_log_ctx}[Your name is {bot_name}. Refer to yourself as {bot_name}, not Claude.]\n{active_project_ctx}\n{}",
"{event_log_ctx}{seed_prefix}[Your name is {bot_name}. Refer to yourself as {bot_name}, not Claude.]\n{active_project_ctx}\n{}",
format_user_prompt(&sender, &user_message)
);
@@ -101,6 +147,7 @@ pub(in crate::chat::transport::matrix::bot) async fn handle_message(
&project_root_str,
resume_session_id.as_deref(),
None,
ctx.model.as_deref(),
&mut cancel_rx,
move |token| {
let mut buf = buffer_for_callback.lock().unwrap();
@@ -117,19 +164,32 @@ pub(in crate::chat::transport::matrix::bot) async fn handle_message(
);
tokio::pin!(chat_fut);
// Notify the room if this turn spends longer than the configured
// threshold on tool calls before emitting any user-facing text, so a
// long silent turn doesn't look hung. Aborted below once the turn ends.
let digging_in_task = spawn_digging_in_watcher(
Arc::clone(&ctx.transport),
room_id_str.clone(),
Arc::clone(&sent_any_chunk),
Duration::from_secs(ctx.digging_in_threshold_secs),
);
// Permission requests are handled by the persistent permission_listener
// task spawned at bot startup (story 884) — they no longer route through
// per-message handlers. Just await chat_fut.
let result = (&mut chat_fut).await;
digging_in_task.abort();
// Flush any remaining text that didn't end with a paragraph boundary.
let remaining = buffer.lock().unwrap().trim().to_string();
let did_send_any = sent_any_chunk.load(Ordering::Relaxed);
let (assistant_reply, new_session_id) = match result {
let mut was_cancelled = false;
let (assistant_reply, new_session_id, turn_usage) = match result {
Ok(ClaudeCodeResult {
messages,
session_id,
usage,
}) => {
let reply = if !remaining.is_empty() {
let _ = msg_tx.send(remaining.clone());
@@ -152,7 +212,16 @@ pub(in crate::chat::transport::matrix::bot) async fn handle_message(
remaining
};
slog!("[matrix-bot] session_id from chat_stream: {:?}", session_id);
(reply, session_id)
(reply, session_id, usage)
}
Err(e) if e == CANCELLED => {
// A user-initiated "stop" — not a crash. The stop handler already
// sent a confirmation, so don't post a second message here, and
// don't run the crash-recovery path below (which would otherwise
// clear the room's session_id as if the session were poisoned).
slog!("[matrix-bot] LLM turn cancelled via stop for room {room_id}");
was_cancelled = true;
(String::new(), None, None)
}
Err(e) => {
slog!("[matrix-bot] LLM error: {e}");
@@ -162,7 +231,7 @@ pub(in crate::chat::transport::matrix::bot) async fn handle_message(
format!("Error processing your request: {e}")
};
let _ = msg_tx.send(err_msg.clone());
(err_msg, None)
(err_msg, None, None)
}
};
@@ -171,11 +240,16 @@ pub(in crate::chat::transport::matrix::bot) async fn handle_message(
drop(msg_tx);
let _ = post_task.await;
if was_cancelled {
return;
}
// Record this exchange in the per-room conversation history and persist
// the session ID so the next turn resumes with structured API messages.
let mut compact_suggestion: Option<String> = None;
if !assistant_reply.starts_with("Error processing") {
let mut guard = ctx.history.lock().await;
let conv = guard.entry(room_id).or_default();
let conv = guard.entry(room_id.clone()).or_default();
// Store the session ID so the next turn uses --resume.
slog!(
@@ -207,11 +281,221 @@ pub(in crate::chat::transport::matrix::bot) async fn handle_message(
conv.entries.drain(..excess);
}
// When this turn's cache_read usage crosses the configured threshold,
// queue a rate-limited suggestion to run `compact`. Rate-limited via
// `last_compact_suggested_at_ms` so a busy room isn't nagged every turn.
if let Some(usage) = &turn_usage
&& usage.cache_read_input_tokens > ctx.cache_read_suggest_threshold
{
let now_ms = chrono::Utc::now().timestamp_millis();
let cooldown_ms = ctx.compact_suggest_cooldown_secs.saturating_mul(1_000);
let due = conv
.last_compact_suggested_at_ms
.is_none_or(|last| now_ms - last >= cooldown_ms);
if due {
conv.last_compact_suggested_at_ms = Some(now_ms);
compact_suggestion = Some(format!(
"This turn read {} cache tokens. Consider running `compact` to distill the \
session and reduce context size.",
usage.cache_read_input_tokens
));
}
}
// Persist to disk so history survives server restarts.
save_history(&ctx.services.project_root, &guard);
} else {
// Clear session_id on error so a poisoned session (e.g. pending
// tool call with no response) doesn't cause a crash loop on resume.
let mut guard = ctx.history.lock().await;
if let Some(conv) = guard.get_mut(&room_id)
&& conv.session_id.is_some()
{
slog!("[matrix-bot] clearing session_id after error to prevent crash loop");
conv.session_id = None;
save_history(&ctx.services.project_root, &guard);
}
}
if let Some(suggestion) = compact_suggestion {
let html = markdown_to_html(&suggestion);
let _ = ctx
.transport
.send_message(&room_id_str, &suggestion, &html)
.await;
}
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
use crate::chat::MessageId;
use async_trait::async_trait;
/// Captures every message sent through it, for assertion in tests.
struct CapturingTransport {
sent: std::sync::Mutex<Vec<(String, String)>>,
}
impl CapturingTransport {
fn new() -> Self {
Self {
sent: std::sync::Mutex::new(Vec::new()),
}
}
fn sent_count(&self) -> usize {
self.sent.lock().unwrap().len()
}
fn last_message(&self) -> Option<String> {
self.sent
.lock()
.unwrap()
.last()
.map(|(_, plain)| plain.clone())
}
}
#[async_trait]
impl ChatTransport for CapturingTransport {
async fn send_message(
&self,
room_id: &str,
plain: &str,
_html: &str,
) -> Result<MessageId, String> {
self.sent
.lock()
.unwrap()
.push((room_id.to_string(), plain.to_string()));
Ok("msg-id".to_string())
}
async fn edit_message(
&self,
_room_id: &str,
_original_message_id: &str,
_plain: &str,
_html: &str,
) -> Result<(), String> {
Ok(())
}
async fn send_typing(&self, _room_id: &str, _typing: bool) -> Result<(), String> {
Ok(())
}
}
/// AC 1: a turn that runs longer than the threshold without sending any
/// text gets exactly one "digging in" notice.
#[tokio::test]
async fn digging_in_fires_after_threshold_when_no_text_sent() {
let transport = Arc::new(CapturingTransport::new());
let sent_any_text = Arc::new(AtomicBool::new(false));
let handle = spawn_digging_in_watcher(
transport.clone() as Arc<dyn ChatTransport>,
"!room:example.com".to_string(),
Arc::clone(&sent_any_text),
Duration::from_millis(30),
);
handle.await.unwrap();
assert_eq!(transport.sent_count(), 1);
assert!(
transport
.last_message()
.unwrap()
.to_lowercase()
.contains("digging in"),
"notice should mention 'digging in'"
);
}
/// AC 2: if text is sent before the threshold elapses, the watcher must
/// not post anything.
#[tokio::test]
async fn digging_in_does_not_fire_when_text_sent_before_threshold() {
let transport = Arc::new(CapturingTransport::new());
let sent_any_text = Arc::new(AtomicBool::new(false));
let handle = spawn_digging_in_watcher(
transport.clone() as Arc<dyn ChatTransport>,
"!room:example.com".to_string(),
Arc::clone(&sent_any_text),
Duration::from_millis(30),
);
sent_any_text.store(true, Ordering::Relaxed);
handle.await.unwrap();
assert_eq!(transport.sent_count(), 0);
}
/// AC 2: a fast turn that completes (and is aborted by its caller)
/// before the threshold elapses must not post anything, even if it
/// never sent any text either.
#[tokio::test]
async fn digging_in_does_not_fire_when_aborted_before_threshold() {
let transport = Arc::new(CapturingTransport::new());
let sent_any_text = Arc::new(AtomicBool::new(false));
let handle = spawn_digging_in_watcher(
transport.clone() as Arc<dyn ChatTransport>,
"!room:example.com".to_string(),
Arc::clone(&sent_any_text),
Duration::from_millis(200),
);
handle.abort();
tokio::time::sleep(Duration::from_millis(250)).await;
assert_eq!(transport.sent_count(), 0);
}
/// AC 3: the notice fires at most once per turn — even waiting well past
/// the threshold never produces a second message.
#[tokio::test]
async fn digging_in_fires_at_most_once_even_after_extra_wait() {
let transport = Arc::new(CapturingTransport::new());
let sent_any_text = Arc::new(AtomicBool::new(false));
let handle = spawn_digging_in_watcher(
transport.clone() as Arc<dyn ChatTransport>,
"!room:example.com".to_string(),
Arc::clone(&sent_any_text),
Duration::from_millis(20),
);
handle.await.unwrap();
tokio::time::sleep(Duration::from_millis(100)).await;
assert_eq!(transport.sent_count(), 1);
}
/// AC 5: state is fresh per call — spawning a second, independent
/// watcher (as a new turn would) does not inherit any state from a
/// prior watcher's `sent_any_text` flag.
#[tokio::test]
async fn digging_in_state_does_not_leak_across_turns() {
let transport = Arc::new(CapturingTransport::new());
// First "turn": text sent before threshold, so no notice.
let first_sent_any_text = Arc::new(AtomicBool::new(false));
let first_handle = spawn_digging_in_watcher(
transport.clone() as Arc<dyn ChatTransport>,
"!room:example.com".to_string(),
Arc::clone(&first_sent_any_text),
Duration::from_millis(30),
);
first_sent_any_text.store(true, Ordering::Relaxed);
first_handle.await.unwrap();
assert_eq!(transport.sent_count(), 0);
// Second "turn": fresh flag, no text sent — must fire independently
// of the first turn's outcome.
let second_sent_any_text = Arc::new(AtomicBool::new(false));
let second_handle = spawn_digging_in_watcher(
transport.clone() as Arc<dyn ChatTransport>,
"!room:example.com".to_string(),
Arc::clone(&second_sent_any_text),
Duration::from_millis(30),
);
second_handle.await.unwrap();
assert_eq!(transport.sent_count(), 1);
}
}
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -9,8 +9,8 @@ pub mod history;
pub mod mentions;
/// Message handlers — processes incoming Matrix room messages.
pub mod messages;
/// Permission listener — holds perm_rx for the bot's lifetime and forwards
/// permission requests to the configured Matrix room.
/// Permission listener — registers as a permission responder for the bot's
/// lifetime and forwards permission requests to the configured Matrix room.
pub mod permission_listener;
/// Bot run loop — the main async task that drives the Matrix sync loop.
pub mod run;
@@ -1,11 +1,11 @@
//! Background task that holds `perm_rx` for the bot's lifetime and forwards
//! permission requests to the configured Matrix room.
//! Background task that registers as a permission responder for the bot's
//! lifetime and forwards permission requests to the configured Matrix room.
//!
//! Before story 884, each chat message handler acquired `perm_rx` for the
//! duration of one chat_fut and dropped it afterwards. That meant whenever
//! the bot wasn't actively responding, `prompt_permission` auto-denied any
//! spawned coder bash call as "no interactive session" — making unattended
//! coder work impossible. This task holds the lock continuously while the
//! coder work impossible. This task stays registered continuously while the
//! bot is connected, so requests can flow at any time.
use crate::chat::ChatTransport;
@@ -20,13 +20,13 @@ use tokio::sync::Mutex as TokioMutex;
use super::format::markdown_to_html;
/// Spawn a background task that holds `services.perm_rx` for the bot's
/// lifetime and forwards each incoming permission request to `target_room`
/// as a chat message. Replies (yes/no) are resolved by the existing
/// `on_room_message` handler via `pending_perm_replies`.
/// Spawn a background task that registers with `services.permission_registry`
/// for the bot's lifetime and forwards each incoming permission request to
/// `target_room` as a chat message. Replies (yes/no) are resolved by the
/// existing `on_room_message` handler via `pending_perm_replies`.
///
/// Returns the JoinHandle so the caller can keep ownership; the task exits
/// only when the `perm_rx` channel is closed (bot shutdown).
/// only when its private channel closes (bot shutdown).
pub fn spawn_permission_listener(
services: Arc<Services>,
transport: Arc<dyn ChatTransport>,
@@ -34,7 +34,7 @@ pub fn spawn_permission_listener(
bot_sent_event_ids: Arc<TokioMutex<HashSet<OwnedEventId>>>,
) -> tokio::task::JoinHandle<()> {
tokio::spawn(async move {
let mut perm_rx = services.perm_rx.lock().await;
let (_responder_guard, mut perm_rx) = services.permission_registry.register();
let target_room_str = target_room.as_str().to_string();
slog!("[matrix-bot] permission listener started; forwarding requests to {target_room_str}");
@@ -57,24 +57,33 @@ pub fn spawn_permission_listener(
}
// Store the MCP oneshot sender so on_room_message can resolve it
// when the user replies yes/no in the target room.
// when the user replies yes/no in the target room. Keyed by
// request_id (not just room) so a second concurrent request for
// the same room doesn't drop the first's sender.
services
.pending_perm_replies
.lock()
.await
.insert(target_room.to_string(), perm_fwd.response_tx);
.insert(
target_room.to_string(),
perm_fwd.request_id.clone(),
perm_fwd.response_tx,
)
.await;
// Spawn a per-request timeout: auto-deny if the user does not
// respond within `permission_timeout_secs`.
let pending = Arc::clone(&services.pending_perm_replies);
let timeout_room_key = target_room.to_string();
let timeout_request_id = perm_fwd.request_id.clone();
let timeout_transport = Arc::clone(&transport);
let timeout_room_str = target_room_str.clone();
let timeout_sent_ids = Arc::clone(&bot_sent_event_ids);
let timeout_secs = services.permission_timeout_secs;
tokio::spawn(async move {
tokio::time::sleep(Duration::from_secs(timeout_secs)).await;
if let Some(tx) = pending.lock().await.remove(&timeout_room_key) {
if let Some(tx) = pending
.remove_by_request_id(&timeout_room_key, &timeout_request_id)
.await
{
let _ = tx.send(PermissionDecision::Deny);
let msg = "Permission request timed out — denied (fail-closed).";
let html = markdown_to_html(msg);
@@ -89,7 +98,7 @@ pub fn spawn_permission_listener(
});
}
slog!("[matrix-bot] permission listener exiting (perm_rx channel closed)");
slog!("[matrix-bot] permission listener exiting (channel closed)");
})
}
@@ -102,11 +111,11 @@ mod tests {
use super::*;
use crate::http::context::PermissionForward;
use crate::service::permission_router::PendingPermReplies;
use crate::services::Services;
use async_trait::async_trait;
use serde_json::json;
use std::collections::HashMap;
use tokio::sync::{mpsc, oneshot};
use tokio::sync::oneshot;
struct RecordingTransport {
sent: Arc<std::sync::Mutex<Vec<(String, String)>>>,
@@ -138,26 +147,40 @@ mod tests {
}
}
fn test_services_with_tx() -> (Arc<Services>, mpsc::UnboundedSender<PermissionForward>) {
let (perm_tx, perm_rx) = mpsc::unbounded_channel();
let services = Arc::new(Services {
/// Poll `cond` until it returns `true` or `timeout` elapses, sleeping
/// briefly between checks. Used instead of a single fixed sleep so tests
/// don't flake under slow/loaded CI where a spawned task hasn't yet run.
async fn wait_until(mut cond: impl FnMut() -> bool, timeout: std::time::Duration) -> bool {
let start = tokio::time::Instant::now();
loop {
if cond() {
return true;
}
if start.elapsed() > timeout {
return false;
}
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
}
}
fn test_services() -> Arc<Services> {
Arc::new(Services {
project_root: std::path::PathBuf::from("/tmp/test"),
agents: Arc::new(crate::agents::AgentPool::new_test(3000)),
bot_name: "Assistant".to_string(),
bot_user_id: "@bot:example.com".to_string(),
ambient_rooms: Arc::new(std::sync::Mutex::new(std::collections::HashSet::new())),
perm_rx: Arc::new(TokioMutex::new(perm_rx)),
pending_perm_replies: Arc::new(TokioMutex::new(HashMap::new())),
permission_registry: crate::service::permission_router::ResponderRegistry::new(),
pending_perm_replies: PendingPermReplies::new(),
permission_timeout_secs: 120,
status: Arc::new(crate::service::status::StatusBroadcaster::new()),
chat_dispatcher: Arc::new(crate::chat::dispatcher::ChatDispatcher::new(1_500)),
});
(services, perm_tx)
})
}
#[tokio::test]
async fn listener_forwards_request_to_target_room_when_no_chat_in_flight() {
let (services, perm_tx) = test_services_with_tx();
let services = test_services();
let sent: Arc<std::sync::Mutex<Vec<(String, String)>>> =
Arc::new(std::sync::Mutex::new(Vec::new()));
let transport: Arc<dyn crate::chat::ChatTransport> = Arc::new(RecordingTransport {
@@ -173,21 +196,33 @@ mod tests {
Arc::clone(&bot_sent_event_ids),
);
// Yield so the listener task acquires perm_rx and starts recv'ing.
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
// Wait until the listener task registers and starts recv'ing.
assert!(
wait_until(
|| !services.permission_registry.is_empty(),
std::time::Duration::from_secs(2)
)
.await,
"listener never registered as a responder"
);
let (response_tx, _response_rx) = oneshot::channel();
perm_tx
.send(PermissionForward {
request_id: "req-1".to_string(),
tool_name: "Bash".to_string(),
tool_input: json!({"command": "cargo test"}),
response_tx,
})
.expect("send PermissionForward");
services.permission_registry.dispatch(PermissionForward {
request_id: "req-1".to_string(),
tool_name: "Bash".to_string(),
tool_input: json!({"command": "cargo test"}),
response_tx,
});
// Give the listener a moment to process.
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
// Wait for the listener to process the forward.
assert!(
wait_until(
|| !sent.lock().unwrap().is_empty(),
std::time::Duration::from_secs(2)
)
.await,
"listener never sent the permission prompt"
);
// The transport must have received exactly one send_message to the
// target room with the prompt content.
@@ -205,17 +240,126 @@ mod tests {
recorded[0].1
);
// pending_perm_replies must contain an entry keyed by the target room
// (so the user-reply handler can resolve the request when they reply).
let pending = services.pending_perm_replies.lock().await;
// pending_perm_replies must contain an entry resolvable for the target
// room (so the user-reply handler can resolve the request when they
// reply). The insert happens just after send_message returns, so poll
// briefly rather than assuming it's already visible.
let mut resolved = None;
for _ in 0..50 {
resolved = services
.pending_perm_replies
.resolve_oldest(target_room.as_str())
.await;
if resolved.is_some() {
break;
}
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
}
assert!(
pending.contains_key(target_room.as_str()),
resolved.is_some(),
"pending_perm_replies missing entry for target room"
);
// bot_sent_event_ids must have recorded the prompt's event ID so the
// bot does not echo its own prompt back as user input.
let sent_ids = bot_sent_event_ids.lock().await;
assert_eq!(sent_ids.len(), 1, "expected one sent event ID recorded");
assert!(
wait_until(
|| bot_sent_event_ids
.try_lock()
.map(|s| s.len() == 1)
.unwrap_or(false),
std::time::Duration::from_secs(2)
)
.await,
"expected one sent event ID recorded"
);
}
/// Regression: two concurrent permission requests for the same target
/// room must both be resolvable — keying `pending_perm_replies` by
/// request_id (rather than overwriting a single room-keyed entry) means
/// the first request's oneshot sender is never dropped by the second.
#[tokio::test]
async fn two_concurrent_requests_for_same_room_both_resolve() {
let services = test_services();
let sent: Arc<std::sync::Mutex<Vec<(String, String)>>> =
Arc::new(std::sync::Mutex::new(Vec::new()));
let transport: Arc<dyn crate::chat::ChatTransport> = Arc::new(RecordingTransport {
sent: Arc::clone(&sent),
});
let target_room: OwnedRoomId = "!perm:example.com".parse().unwrap();
let bot_sent_event_ids = Arc::new(TokioMutex::new(HashSet::new()));
spawn_permission_listener(
Arc::clone(&services),
Arc::clone(&transport),
target_room.clone(),
Arc::clone(&bot_sent_event_ids),
);
assert!(
wait_until(
|| !services.permission_registry.is_empty(),
std::time::Duration::from_secs(2)
)
.await,
"listener never registered as a responder"
);
let (tx1, rx1) = oneshot::channel();
services.permission_registry.dispatch(PermissionForward {
request_id: "req-a".to_string(),
tool_name: "Bash".to_string(),
tool_input: json!({}),
response_tx: tx1,
});
let (tx2, rx2) = oneshot::channel();
services.permission_registry.dispatch(PermissionForward {
request_id: "req-b".to_string(),
tool_name: "Write".to_string(),
tool_input: json!({}),
response_tx: tx2,
});
// Wait until the listener has processed both forwards.
assert!(
wait_until(
|| sent.lock().unwrap().len() >= 2,
std::time::Duration::from_secs(2)
)
.await,
"listener never sent both permission prompts"
);
// Poll rather than assuming the pending_perm_replies insert (which
// happens just after send_message returns) is already visible.
let mut first = None;
for _ in 0..50 {
first = services
.pending_perm_replies
.resolve_oldest(target_room.as_str())
.await;
if first.is_some() {
break;
}
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
}
let first = first.expect("first request's sender must not have been dropped");
let _ = first.send(PermissionDecision::Approve);
assert_eq!(rx1.await.unwrap(), PermissionDecision::Approve);
let mut second = None;
for _ in 0..50 {
second = services
.pending_perm_replies
.resolve_oldest(target_room.as_str())
.await;
if second.is_some() {
break;
}
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
}
let second = second.expect("second request's sender must still be present");
let _ = second.send(PermissionDecision::Deny);
assert_eq!(rx2.await.unwrap(), PermissionDecision::Deny);
}
}
+32 -34
View File
@@ -40,6 +40,16 @@ pub async fn run_bot(
tokio::sync::broadcast::Receiver<crate::service::gateway::GatewayStatusEvent>,
>,
gateway_port: Option<u16>,
gateway_channels_store: Option<
Arc<
RwLock<
std::collections::BTreeMap<
String,
crate::service::gateway::config::ReleaseChannelConfig,
>,
>,
>,
>,
) -> Result<(), String> {
let project_root = &services.project_root;
let store_path = project_root.join(".huskies").join("matrix_store");
@@ -180,21 +190,6 @@ pub async fn run_bot(
let notif_room_ids = target_room_ids.clone();
let notif_project_root = project_root.clone();
let announce_room_ids = target_room_ids.clone();
// Clone values needed by the gateway notification poller (only used in gateway mode).
let poller_room_ids: Vec<String> = target_room_ids.iter().map(|r| r.to_string()).collect();
let poller_project_urls: std::collections::BTreeMap<String, String> =
if let Some(ref store) = gateway_projects_store {
store
.read()
.await
.iter()
.filter_map(|(name, entry)| entry.url.clone().map(|url| (name.clone(), url)))
.collect()
} else {
std::collections::BTreeMap::new()
};
let poller_poll_interval = config.aggregated_notifications_poll_interval_secs;
let poller_enabled = config.aggregated_notifications_enabled;
let persisted = load_history(project_root);
slog!(
@@ -299,7 +294,7 @@ pub async fn run_bot(
let bot_sent_event_ids: Arc<TokioMutex<HashSet<matrix_sdk::ruma::OwnedEventId>>> =
Arc::new(TokioMutex::new(HashSet::new()));
// Spawn the permission listener: holds `perm_rx` for the bot's lifetime
// Spawn the permission listener: registers as a responder for the bot's lifetime
// and forwards permission requests to the first configured room. Story
// 884 — replaces the per-message lock acquire previously done in
// handle_message.rs, so spawned coders' bash calls reach chat even when
@@ -332,11 +327,17 @@ pub async fn run_bot(
timer_store,
gateway_active_project,
gateway_projects_store,
gateway_channels_store,
handled_incoming_event_ids: Arc::new(TokioMutex::new(super::context::SeenEventIds::new(
super::context::SEEN_EVENT_IDS_CAP,
))),
gateway_port,
last_matrix_event_ms: Arc::new(AtomicI64::new(chrono::Utc::now().timestamp_millis())),
model: config.model.clone(),
compact_seed_max_bytes: config.compact_seed_max_bytes,
cache_read_suggest_threshold: config.cache_read_suggest_threshold,
compact_suggest_cooldown_secs: config.compact_suggest_cooldown_secs,
digging_in_threshold_secs: config.digging_in_threshold_secs,
};
slog!(
@@ -359,21 +360,9 @@ pub async fn run_bot(
notif_project_root,
);
// In gateway mode, spawn the cross-project notification poller.
// It polls every registered project's `/api/events` endpoint and forwards
// new events to the configured gateway rooms with a `[project-name]` prefix.
// The poller is controlled by the gateway-level `aggregated_notifications_enabled`
// flag in bot.toml — set it to `false` to disable without touching per-project configs.
if !poller_project_urls.is_empty() && poller_enabled {
crate::gateway::spawn_gateway_notification_poller(
Arc::clone(&transport),
poller_room_ids,
poller_project_urls,
poller_poll_interval,
);
}
// Forwarder task: post gateway events to Matrix rooms with `[project-name]` prefix.
// Project nodes push events over the WS uplink (story 899/1179); the
// gateway no longer polls per-project `/api/events` over HTTP (story 1180).
if let Some(event_rx) = gateway_event_rx_for_forwarder {
let broadcast_room_ids: Vec<String> =
announce_room_ids.iter().map(|r| r.to_string()).collect();
@@ -412,15 +401,24 @@ pub async fn run_bot(
// reconnects internally so this code is never reached again on a network
// blip or sync resumption.
//
// When started by the trampoline the message is specialised:
// - HUSKIES_TRAMPOLINE_STARTED=1 → "gateway X.Y.Z ready"
// A normal start and a trampoline-triggered restart both announce version,
// git hash, and configured model; the trampoline path is specialised further:
// - HUSKIES_TRAMPOLINE_STARTED=1 → "gateway X.Y.Z (git_hash) ready — model: ..."
// - HUSKIES_TRAMPOLINE_FAILURE=<reason> → rollback failure notice
// - otherwise (normal start) → "{bot_name} is online — gateway X.Y.Z (git_hash) — model: ..."
let announce_msg = if let Ok(reason) = std::env::var("HUSKIES_TRAMPOLINE_FAILURE") {
super::format::format_gateway_rollback_announcement(&reason)
} else if std::env::var("HUSKIES_TRAMPOLINE_STARTED").is_ok() {
super::format::format_gateway_ready_announcement()
super::format::format_gateway_ready_announcement(
option_env!("BUILD_GIT_HASH").unwrap_or("unknown"),
config.model.as_deref(),
)
} else {
format_startup_announcement(&announce_bot_name)
format_startup_announcement(
&announce_bot_name,
option_env!("BUILD_GIT_HASH").unwrap_or("unknown"),
config.model.as_deref(),
)
};
let announce_html = markdown_to_html(&announce_msg);
slog!("[matrix-bot] Sending startup announcement: {announce_msg}");
+190
View File
@@ -0,0 +1,190 @@
//! `compact` command: distills the current room's Claude Code session
//! transcript into a size-capped digest, writes it as a seed, and clears the
//! session so the next turn starts fresh with only the distilled context.
//!
//! Parsing and the transcript→digest→seed pipeline are transport-agnostic
//! and live in [`crate::chat::compact`]; this module is the Matrix-specific
//! glue around the shared [`ConversationHistory`], mirroring [`super::reset`].
use crate::chat::compact::{self, CompactError};
use crate::chat::transport::matrix::bot::{ConversationHistory, RoomConversation};
use matrix_sdk::ruma::OwnedRoomId;
use std::path::Path;
/// Re-exported so Matrix call sites read as `compact::extract_compact_command`,
/// matching the sibling `reset` module's shape.
pub use crate::chat::compact::extract_compact_command;
/// Handle a `compact` command: distill the room's session transcript into a
/// digest, write it as a seed for the next turn, clear the session ID and
/// entries (like `reset`), and return a confirmation with before/after sizes.
///
/// `project_root` doubles as the working directory the `claude` CLI was
/// spawned with for this room (matrix's `handle_message` always runs Claude
/// Code in `services.project_root` — the gateway config dir in gateway mode,
/// or the project root in standalone mode — so the session transcript lives
/// under the matching `~/.claude/projects/<mangled project_root>/` directory).
///
/// Returns an actionable message and changes nothing when there is no active
/// session, or when the session transcript cannot be read.
pub async fn handle_compact(
room_id: &OwnedRoomId,
history: &ConversationHistory,
project_root: &Path,
max_bytes: usize,
) -> String {
let mut guard = history.lock().await;
let conv = guard
.entry(room_id.clone())
.or_insert_with(RoomConversation::default);
let Some(session_id) = conv.session_id.clone() else {
return "No active session to compact. Send a message first to start one.".to_string();
};
let transcript_path = compact::transcript::transcript_path(project_root, &session_id);
match compact::compact_session(project_root, &transcript_path, room_id.as_ref(), max_bytes) {
Ok(outcome) => {
conv.session_id = None;
conv.entries.clear();
conv.pending_seed = Some(outcome.digest);
crate::chat::transport::matrix::bot::save_history(project_root, &guard);
crate::slog!(
"[matrix-bot] compact: room {room_id} {} -> {} bytes, seed written to {}",
outcome.before_bytes,
outcome.after_bytes,
outcome.seed_path.display()
);
format!(
"Compacted session context: {} → {} bytes. Starting fresh — the distilled \
summary will be included as background in your next message.",
outcome.before_bytes, outcome.after_bytes
)
}
Err(CompactError::TranscriptUnreadable(e)) => format!(
"Could not compact: session transcript is unreadable ({e}). Nothing was changed."
),
}
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
use crate::chat::transport::matrix::bot::{ConversationEntry, ConversationRole};
use std::collections::HashMap;
use std::sync::Arc;
use tokio::sync::Mutex as TokioMutex;
fn room_id() -> OwnedRoomId {
"!test:example.com".parse().unwrap()
}
#[tokio::test]
async fn no_active_session_returns_actionable_message_and_changes_nothing() {
let history: ConversationHistory = Arc::new(TokioMutex::new(HashMap::new()));
let tmp = tempfile::tempdir().unwrap();
let response = handle_compact(&room_id(), &history, tmp.path(), 8_000).await;
assert!(
response.contains("No active session"),
"expected actionable no-session message: {response}"
);
let guard = history.lock().await;
assert!(
guard.get(&room_id()).is_none_or(|c| c.entries.is_empty()),
"no entries should be created for a compact with no session"
);
}
#[tokio::test]
async fn unreadable_transcript_returns_actionable_message_and_preserves_session() {
let history: ConversationHistory = Arc::new(TokioMutex::new({
let mut m = HashMap::new();
m.insert(
room_id(),
RoomConversation {
session_id: Some("missing-session".to_string()),
entries: vec![],
pending_seed: None,
last_compact_suggested_at_ms: None,
},
);
m
}));
let tmp = tempfile::tempdir().unwrap();
let response = handle_compact(&room_id(), &history, tmp.path(), 8_000).await;
assert!(
response.contains("unreadable"),
"expected actionable unreadable-transcript message: {response}"
);
let guard = history.lock().await;
let conv = guard.get(&room_id()).unwrap();
assert_eq!(
conv.session_id.as_deref(),
Some("missing-session"),
"session_id must be preserved when compaction fails"
);
}
#[tokio::test]
async fn successful_compact_clears_session_and_sets_pending_seed() {
let session_id = "sess-compact-1";
let project_root = tempfile::tempdir().unwrap();
let home = tempfile::tempdir().unwrap();
// SAFETY: this test owns HOME for its duration; no other test in this
// process reads HOME concurrently with this call.
unsafe {
std::env::set_var("HOME", home.path());
}
let transcript_dir = compact::transcript::transcript_path(project_root.path(), session_id)
.parent()
.unwrap()
.to_path_buf();
std::fs::create_dir_all(&transcript_dir).unwrap();
let jsonl = r#"{"type":"user","message":{"role":"user","content":"hello"}}
{"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":"hi there"}]}}"#;
std::fs::write(transcript_dir.join(format!("{session_id}.jsonl")), jsonl).unwrap();
let history: ConversationHistory = Arc::new(TokioMutex::new({
let mut m = HashMap::new();
m.insert(
room_id(),
RoomConversation {
session_id: Some(session_id.to_string()),
entries: vec![ConversationEntry {
role: ConversationRole::User,
sender: "@alice:example.com".to_string(),
content: "hi".to_string(),
}],
pending_seed: None,
last_compact_suggested_at_ms: None,
},
);
m
}));
let response = handle_compact(&room_id(), &history, project_root.path(), 8_000).await;
assert!(
response.contains("Compacted session context"),
"response should confirm compaction with sizes: {response}"
);
let guard = history.lock().await;
let conv = guard.get(&room_id()).unwrap();
assert!(conv.session_id.is_none(), "session_id must be cleared");
assert!(conv.entries.is_empty(), "entries must be cleared");
assert_eq!(
conv.pending_seed.as_deref(),
Some("User: hello\nAssistant: hi there"),
"pending_seed must hold the distilled digest"
);
}
}
@@ -172,6 +172,49 @@ history_size = 50
assert_eq!(config.history_size, 50);
}
/// AC4: `digging_in_threshold_secs` defaults to 15 when unset in bot.toml.
#[test]
fn load_uses_default_digging_in_threshold_secs() {
let tmp = tempfile::tempdir().unwrap();
let sk = tmp.path().join(".huskies");
fs::create_dir_all(&sk).unwrap();
fs::write(
sk.join("bot.toml"),
r#"
homeserver = "https://matrix.example.com"
username = "@bot:example.com"
password = "secret"
room_ids = ["!abc:example.com"]
enabled = true
"#,
)
.unwrap();
let config = BotConfig::load(tmp.path()).unwrap();
assert_eq!(config.digging_in_threshold_secs, 15);
}
/// AC4: `digging_in_threshold_secs` can be overridden in bot.toml.
#[test]
fn load_respects_custom_digging_in_threshold_secs() {
let tmp = tempfile::tempdir().unwrap();
let sk = tmp.path().join(".huskies");
fs::create_dir_all(&sk).unwrap();
fs::write(
sk.join("bot.toml"),
r#"
homeserver = "https://matrix.example.com"
username = "@bot:example.com"
password = "secret"
room_ids = ["!abc:example.com"]
enabled = true
digging_in_threshold_secs = 30
"#,
)
.unwrap();
let config = BotConfig::load(tmp.path()).unwrap();
assert_eq!(config.digging_in_threshold_secs, 30);
}
#[test]
fn load_reads_display_name() {
let tmp = tempfile::tempdir().unwrap();
@@ -282,47 +325,6 @@ require_verified_devices = true
);
}
#[test]
fn aggregated_notifications_enabled_defaults_to_true() {
let tmp = tempfile::tempdir().unwrap();
let sk = tmp.path().join(".huskies");
fs::create_dir_all(&sk).unwrap();
fs::write(
sk.join("bot.toml"),
r#"
homeserver = "https://matrix.example.com"
username = "@bot:example.com"
password = "secret"
room_ids = ["!abc:example.com"]
enabled = true
"#,
)
.unwrap();
let config = BotConfig::load(tmp.path()).unwrap();
assert!(config.aggregated_notifications_enabled);
}
#[test]
fn aggregated_notifications_enabled_can_be_set_to_false() {
let tmp = tempfile::tempdir().unwrap();
let sk = tmp.path().join(".huskies");
fs::create_dir_all(&sk).unwrap();
fs::write(
sk.join("bot.toml"),
r#"
homeserver = "https://matrix.example.com"
username = "@bot:example.com"
password = "secret"
room_ids = ["!abc:example.com"]
enabled = true
aggregated_notifications_enabled = false
"#,
)
.unwrap();
let config = BotConfig::load(tmp.path()).unwrap();
assert!(!config.aggregated_notifications_enabled);
}
#[test]
fn load_reads_ambient_rooms() {
let tmp = tempfile::tempdir().unwrap();
@@ -9,19 +9,34 @@ pub(super) fn default_permission_timeout_secs() -> u64 {
120
}
pub(super) fn default_aggregated_notifications_poll_interval_secs() -> u64 {
5
}
pub(super) fn default_aggregated_notifications_enabled() -> bool {
true
}
/// Default coalesce window for the chat dispatcher (1 500 ms).
pub(super) fn default_coalesce_window_ms() -> u64 {
1_500
}
/// Default cap (bytes) on the digest the `compact` command writes as a seed.
pub(super) fn default_compact_seed_max_bytes() -> usize {
8_000
}
/// Default cache_read token threshold above which the bot suggests running
/// `compact` after a turn.
pub(super) fn default_cache_read_suggest_threshold() -> u64 {
50_000
}
/// Default cooldown (seconds) between repeated `compact` suggestions for the
/// same room.
pub(super) fn default_compact_suggest_cooldown_secs() -> i64 {
3_600
}
/// Default threshold (seconds) a turn may spend on tool calls before
/// emitting any user-facing text before the bot posts a "digging in" notice.
pub(super) fn default_digging_in_threshold_secs() -> u64 {
15
}
pub(super) fn default_transport() -> String {
"matrix".to_string()
}
@@ -71,10 +86,10 @@ pub struct BotConfig {
/// (fail-closed). Defaults to 120 seconds.
#[serde(default = "default_permission_timeout_secs")]
pub permission_timeout_secs: u64,
/// Previously used to select an Anthropic model. Now ignored — the bot
/// uses Claude Code which manages its own model selection. Kept for
/// backwards compatibility so existing bot.toml files still parse.
#[allow(dead_code)]
/// Claude Code model override. When set, passed as `--model <value>` to
/// the `claude` CLI so the bot uses a specific model instead of the CLI's
/// configured default.
#[serde(default)]
pub model: Option<String>,
/// Display name the bot uses to identify itself in conversations.
/// If unset, the bot falls back to "Assistant".
@@ -173,26 +188,6 @@ pub struct BotConfig {
#[serde(default)]
pub discord_allowed_users: Vec<String>,
/// How often (in seconds) the gateway polls each project server's
/// `/api/events` endpoint to aggregate cross-project notifications.
///
/// Only used when the gateway's bot is enabled. Defaults to 5 seconds.
#[serde(default = "default_aggregated_notifications_poll_interval_secs")]
pub aggregated_notifications_poll_interval_secs: u64,
/// Whether the gateway-level aggregated cross-project notification stream
/// is enabled. When `false`, the gateway will not poll per-project
/// servers for events even if the bot is otherwise enabled.
///
/// Set this in the **gateway's** `bot.toml` (not in per-project configs).
/// Adding a new project to `projects.toml` never requires touching
/// per-project bot configs — the aggregated stream picks it up
/// automatically once this flag is `true` (the default).
///
/// Defaults to `true`.
#[serde(default = "default_aggregated_notifications_enabled")]
pub aggregated_notifications_enabled: bool,
/// Duration in milliseconds of the chat dispatcher's coalesce window.
///
/// Messages for the same session arriving within this window are
@@ -218,4 +213,26 @@ pub struct BotConfig {
/// `git config user.email` when absent.
#[serde(default)]
pub git_user_email: Option<String>,
/// Maximum size in bytes of the digest the `compact` command writes as a
/// seed file. Older content is dropped first so the most recent
/// conversation survives the cap. Defaults to 8000 bytes.
#[serde(default = "default_compact_seed_max_bytes")]
pub compact_seed_max_bytes: usize,
/// Number of `cache_read_input_tokens` in a single turn above which the
/// bot suggests running `compact`. Defaults to 50 000.
#[serde(default = "default_cache_read_suggest_threshold")]
pub cache_read_suggest_threshold: u64,
/// Minimum seconds between repeated `compact` suggestions for the same
/// room, so a busy room isn't spammed every turn. Defaults to 3600 (1h).
#[serde(default = "default_compact_suggest_cooldown_secs")]
pub compact_suggest_cooldown_secs: i64,
/// Seconds a turn may spend on tool calls before emitting any
/// user-facing text before the bot posts a "digging in" notice to the
/// room, so a silent long-running turn doesn't look hung. Defaults to 15.
#[serde(default = "default_digging_in_threshold_secs")]
pub digging_in_threshold_secs: u64,
}

Some files were not shown because too many files have changed in this diff Show More