Story 1130 added HUSKIES_HOST=0.0.0.0 so the server INSIDE a project
container binds to all interfaces, but the host-side `docker -p`
mapping was still `127.0.0.1:{port}:3001` and `127.0.0.1:{ssh_port}:22`
— reachable from the docker host only, blocking remote MCP clients
and out-of-host SSH onto the project container.
Switch host-side mapping to 0.0.0.0 for both the MCP and SSH ports so
project containers spawned via `new project` are reachable from
anywhere that can route to the docker host. Existing containers
created before this commit retain their localhost-only mapping and
need to be recreated to pick up the change.
Add a regression test asserting both -p arguments use 0.0.0.0 and
reject any 127.0.0.1 restriction in the mapping.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous attempt (c1318964) used npm ci + npm install --include=optional
--no-save, which still missed rolldown's platform-specific native
binding (@rolldown/binding-linux-arm64-gnu) — the runtime build still
fails with `Cannot find native binding`.
Wipe both the lockfile and node_modules so npm install resolves the
dependency tree fresh for the build platform. The lockfile mutation
stays inside the container image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`npm ci` alone hits npm/cli#4828: optional platform-specific bindings
(e.g. @rolldown/binding-linux-arm64-gnu introduced by 1119's vite 5→8
upgrade) listed in package-lock.json for the lockfile author's
platform are not fetched for the build platform. The sled rebuild
fails with `Cannot find native binding`.
Follow `npm ci` with `npm install --include=optional --no-save` so the
build platform's native binding is fetched without mutating the
lockfile.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A 0-byte pipeline.db sometimes appears at the repo root, left over
from old code paths. Current master correctly opens it at
.huskies/pipeline.db via project_root.join() in
server/src/startup/project.rs:280 — no relative-path opener exists.
This is purely defensive so any future regression doesn't sneak into
commits. Stops 1123 from being a coder task.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Story 1113 added `#[derive(RustEmbed)] #[folder = "../frontend/dist"]`
plus a unit test that calls `EmbeddedAssets::iter()`. The macro only
generates `iter()` when the folder exists at compile time, so the Rust
build now has a hard compile-time dependency on `frontend/dist/`.
`script/test` ran `cargo clippy` (line 48) before the frontend build
(line 53+). In a fresh merge worktree with no `frontend/dist/`, clippy
failed immediately on the `iter()` call and the script exited before
`npm run build` ever ran — the gate could never self-heal. Blocked
1116's merge today; would block every future merge.
Move the frontend build above all cargo invocations. Verified by
running script/test in a fresh worktree with `node_modules` and
`frontend/dist` removed: 385/385 frontend tests + cargo tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The merge gate classifier was matching trigger keywords like
`missing_doc_comments` inside passing-test name lines
(e.g. `test agents::gates::tests::classify_lint_from_missing_doc_comments ... ok`),
causing every gate failure to be mis-classified as Lint and bounced
back to a fixup coder. Strip `test … … ok` lines before scanning for
lint triggers. Also removes the temporary diagnostic block in
runner.rs that confirmed the bug.
Applied directly to master because the 1101 feature branch carried
stale work from an earlier incarnation of the story that semantically
conflicted with master's later diagnostic commit (`is_fixup` deleted
on the branch, referenced on master).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the architecture for going from "new project" chat command to
a running, container-isolated, editor-accessible huskies project.
Covers the three personas (chat-only / editor-using / multi-project),
the container template (base + stack overlay + project bind mount),
build sandbox model (host stays clean, all dep-code in container),
editor-agnostic SSH access, git integration, and a 5-phase rollout.
Source for upcoming bootstrap stories.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bug 1102 was created today with origin={kind:user, id:""} because
build_origin silently defaulted id to empty when the caller didn't pass
one — we couldn't tell who filed it. Bug 1088's origin field is useless
as audit if every caller can omit themselves.
Changes:
- build_origin (server/src/http/mcp/story_tools/mod.rs) now returns
Result<String, String> and rejects missing/empty/whitespace-only id
with an instructional error pointing at bug 1102 / story 1104.
- 5 create_* tool handlers (bug, spike, refactor, epic, story) now
resolve origin BEFORE create_*_file so an attribution-less call
leaves no half-state behind.
- 5 tool input schemas now advertise origin as a required object via
a shared origin_schema() helper. The schema description gives every
caller (coder agent, chat bot, user, system) a concrete example so
the LLM populates the field correctly on first sight.
- Test fixtures pass origin = {kind:"test", id:"test-suite"}.
Story 1104 (signed actions) is the longer-term replacement; this is the
quick attribution win agreed for master ahead of that design work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 1101's reframed AC1: when a non-success merge runs, log the typed
GateFailureKind, the matched classifier-trigger substring (if any) and
~90 chars of surrounding context. Fires on every gate failure regardless
of routing, so the next fixup-loop bounce will tell us which substring is
fooling classify() into Fmt|Lint|SourceMapCheck on what's actually a Test
failure.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The test asserted msg_count == 0 on a process-global broadcast channel
(TRANSITION_TX is a single OnceLock<Sender> shared across the test
binary), so any concurrent test calling apply_transition could land
events in our receiver between the drain and the post-reconcile check.
Observed failure: 3 stray transitions from parallel tests.
Drop the strict count check. The real "never floods" invariant is
captured by the Lagged check alone: 1000 seeded items must not overflow
the 256-slot channel, which can only hold if the reconcile path
bypasses the broadcast (AC4). The sibling test
`reconcile_pass_scales_to_1000_items_without_lagged_divergence` already
uses this Lagged-only pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
stop_agent had the same order-of-operations bug fixed in the watchdog:
status flipped to Failed before the claude process was verified gone,
opening the idempotency window that allowed a duplicate spawn to race
in alongside the surviving process.
Now follows the three-step protocol:
1. Read worktree path under a read-only lock (no mutation).
2. SIGKILL the worktree's process tree via process_kill and block
until verified gone — start_agent's Running/Pending whitelist
continues to reject duplicate spawns throughout.
3. Only then mutate the agent record, abort the task handle, and
drop the child_killers entry.
Falls back to the old portable_pty SIGHUP path (with a warning) when
no worktree was recorded, matching the watchdog's behaviour.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `crate::process_kill` — reliable SIGKILL-with-verify primitives used
across the server in place of the various ad-hoc kill paths that ignored
their kill-effective return values. The module exposes three pieces:
- `sigkill_pids_and_verify(pids)`: SIGKILL each pid and block (up to 2s)
until every pid is verified gone. Returns survivors if not.
- `pids_matching(pattern)`: pgrep -f wrapper.
- `descendant_pids(root)`: recursive pgrep -P walker for process trees.
Wires the watchdog's limit-termination path through it, and reorders the
protocol to fix the duplicate-coder bug observed on story 1086 (2026-05-15):
Before: check_agent_limits set status=Failed before the kill ran. The
kill itself was `portable_pty::ChildKiller::kill()`, which sends SIGHUP
on Unix — claude-code ignores SIGHUP, so the process kept running while
the agent record was already marked terminated. The idempotency check
in `start_agent` whitelists Running/Pending, so the next auto-assign
pass spawned a fresh agent alongside the still-alive prior one. Two
claude PIDs sharing one session_id, racing on the same worktree.
After: status update is moved OUT of check_agent_limits and into the
caller AFTER the kill is verified. The kill itself is now SIGKILL-the-
process-tree-in-the-worktree, with explicit verification that every pid
is gone. The idempotency window is closed.
The existing watchdog test suite (14 tests) still passes; 7 new tests
cover the process_kill primitives directly.
`agents/pool/process.rs`'s `kill_all_children` and `kill_child_for_key`
still use the old portable_pty SIGHUP path — they have the same bug but
in lower-impact code paths (shutdown, operator stop). They will be
migrated under a separate story to keep this commit focused.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Local-only scratchpad for tracking suspected duplicate-Timmy /
duplicate-create_story incidents while we hunt the cause.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Story 1066 (merged 2026-05-14 23:39) introduced a periodic reconciler that
calls `reconcile_worktree_create` every 30 seconds (default
`reconcile_interval_secs`). The reconciler's docstring promises it is a no-op
for stories whose worktree already exists — but the implementation calls
`create_worktree`, whose reuse path was running `run_setup_commands`
unconditionally. Setup includes destructive `npm ci` (rm -rf node_modules
then reinstall), so every Coding story got `npm ci` fired every 30 seconds.
When story 1086 hit a gate-failure retry loop on 2026-05-15, the merge gate's
own `npm install`/`npm run build` raced one of these reconciler-driven
`npm ci` runs that was wiping node_modules — leaving `.bin/tsc` as a broken
symlink pointing into a half-populated `typescript/` package and producing
`sh: 1: tsc: not found`. 37 npm ci fires for 1086 in 5 hours against only
3 real Coding transitions, a 12x amplification driven entirely by the
30-second reconcile cadence.
Fix: align `create_worktree`'s behaviour with the contract `reconcile_worktree_create`
already documents — reuse is a no-op for setup commands. Sparse checkout
and `.mcp.json` rewrite still run (both cheap and idempotent).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The orientation bundle was 96 KB per coder spawn with 85 KB of that being
source-map.json — a static symbol listing that drowned out the workflow rules
in AGENT.md and likely explains why PLAN.md ceremony is being skipped (the
instruction is ~5% of the bundle, buried under a wall of symbols). Agents are
excellent at grep on demand, so the source map adds little value as a preloaded
cheat sheet. File stays on disk for the merge-time source-map-check doc-coverage
gate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Catches up master with entries added by stories that merged in a binary
predating 1065 (merge-pipeline source-map regen): ErrorBoundary,
WsConnectivity, transition_merge_failure_to_retry, and others.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge 1061 added a replay_current_pipeline_state() call to the broadcast::Lagged
branch, but replay broadcasts one event per CRDT item (~997) into a 256-slot
channel, deterministically re-overflowing it and triggering another Lagged. The
loop pinned CPU and likely caused today's machine crash. Revert to the pre-1061
behaviour of logging and continuing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`subscribe_to_watcher` was pushing StoredEvents into the event
buffer with story_name hardcoded to String::new(), so /api/events
polled by the gateway always omitted the title. The 1035 fix
patched the other path (gateway_relay status_to_stored) but left
this one bleeding empty strings.
Lookup happens once at the subscriber boundary rather than at all
44 watcher emit sites — the story_id is already in hand and
crdt_state::read_item is the canonical name source.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dep is declared only to flip on the `bundled` feature for the
static musl build, and 0.35 is the ceiling forced by rusqlite 0.37
(matrix-sdk-sqlite) and sqlx-sqlite 0.9.0-alpha.1. Future readers
no longer have to reconstruct that from cargo-tree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Covers the two binaries (source-map-check, source-map-regen), the
library entry points, and the why — that .huskies/source-map.json
is embedded directly in every autonomous coder's orientation
prompt, so determinism and freshness are load-bearing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Toolchain bump surfaced new lints (derivable_impls,
unnecessary_unwrap, unnecessary_sort_by, while_let_loop,
collapsible_match, unnecessary_option_map_or_else, cmp_owned)
across bft-json-crdt and huskies-server. All fixed mechanically.
Cargo.toml: dropped the no-longer-existing `rustls-tls` matrix-sdk
feature, then chased through the 0.17 API breakage:
- Relation::Reply is now a tuple variant wrapping Reply, not a
struct variant with `in_reply_to`
- UserIdentifier::UserIdOrLocalpart removed — use
UserIdentifier::Matrix(MatrixUserIdentifier::new(..))
- SendMessageLikeEventResult no longer exposes event_id directly;
it's now on the inner `response` field
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
matrix-sdk 0.17 requires Rust 1.93 (uses Duration::from_mins, declares
rust-version = "1.93"). The container was on 1.90, which is why stories
1022 and 1028 both bounced off the matrix-sdk upgrade despite the host
having Rust 1.93 — the rustup update on the host doesn't propagate into
the build container.
Bumping the FROM rust:1.93-bookworm so the next container rebuild ships
1.93, unblocking matrix-sdk 0.17 upgrades and the rand@0.8 transitive
elimination that comes with it.
cargo-machete dropped eventsource-stream, indexmap, serde_yaml, and
strip-ansi-escapes from server/Cargo.toml in 1027 (4fad2838), but the
Cargo.lock didn't regenerate as part of that merge. The lockfile was
sitting dirty on master, blocking subsequent cherry-picks (1026 hit
'Your local changes to the following files would be overwritten by
merge: Cargo.lock').
This commit is the missing lockfile catch-up — drops the four crates
(and their transitives nom + minimal-lexical) from the lock.
1018's merge_failure_block_subscriber counted every MergeFailure transition
toward the 3-strike block threshold, but mergemaster's recovery iterations
(squash → fail → fix → retry) emit multiple MergeFailure transitions while
making real progress. Story 997 was blocked at 10:59:46 while mergemaster
was still resolving conflicts and would have succeeded a minute later.
Fix: pass the AgentPool to the subscriber. When a mergemaster agent is in
the pool for the story, MergeFailure transitions are recovery iterations
in progress and do NOT increment the consecutive-failure counter. Block
only fires for the genuinely-stuck case (no recovery agent attached and N
consecutive failures accumulate).
Tests:
- mergemaster_running_suppresses_block: 3 failures with recovery_running=true
→ counter stays empty, story stays in MergeFailure
- no_mergemaster_still_blocks_at_threshold: 3 failures with recovery_running=false
→ blocks (1018 behaviour preserved)
All 2938 tests pass.
The progress-aware no-progress cap (3 consecutive byte-identical diffs)
doesn't catch the degenerate pattern where the agent keeps making
DIFFERENT file edits each session but never commits — every respawn
resets the no-progress counter, infinite loop, budget burns.
Adds ContentKey::CommitRecoveryTotalAttempts: an absolute counter that
increments on every commit-recovery respawn regardless of progress.
TOTAL_ATTEMPTS_CAP = 8; when hit, block with reason 'agent flapped — N
respawns without ever committing'.
Two caps now bound the recovery loop:
- NO_PROGRESS_CAP (3): catches stuck-agent (same diff repeatedly)
- TOTAL_ATTEMPTS_CAP (8): catches flapping-agent (different diffs, no commits)
Easy to tune the constant lower if we see runaway in practice.
All 2936 tests pass.
The existing commit-recovery path blocked stories on the 2nd consecutive
exit-without-commit. For long sweep refactors (e.g. story 997, the typed
retries payload migration), claude-code's session-length boundary
naturally terminates the coder mid-sweep before it can commit — even
though substantial file-edit progress is being made each session. The
old cap-of-1 misclassified normal mid-flight progress as 'agent declined
to commit'.
New behaviour:
- Each commit-recovery respawn captures a worktree-diff byte-length
fingerprint (git diff master | wc -c).
- If the fingerprint differs from the previous attempt the agent made
file-edit progress, the no-progress counter resets to 1.
- If the fingerprint is byte-identical (no new edits between exits),
increment the no-progress counter.
- Block only when the counter reaches NO_PROGRESS_CAP (3) — i.e. three
consecutive respawns where the agent did literally nothing.
Adds ContentKey::CommitRecoveryDiffFingerprint to store the prior
fingerprint. Updates the existing block-test to reflect the new cap
semantics; existing 'first respawn issued' test continues to pass.
All 2935 tests pass.
Temporary diagnostic added to reap_stale_merge_jobs to surface the t,
current_boot, and decoded values being compared on every reap pass.
Will revert once the disappearance bug is understood.
When deterministic-merge produces a clean git squash but the post-squash
compile fails (typical when master gained a Stage payload field after the
feature branch forked — e.g. story 1018 hit `error[E0063]: missing field
plan` after 1010's PlanState landed), the failure is morally a merge
conflict that git's diff3 missed: the conflicting literal lives in a
different file from the type definition that changed on master. Routing
it as GatesFailed left mergemaster idle and the story stuck.
Changes:
- gates.rs GateFailureKind::classify: detect rustc compile errors
(`error[E\d+]`) as Build instead of falling through to Test. Clippy
errors (`error[clippy::...]`) still classify as Lint.
- agents/merge/mod.rs: new MergeResult::to_merge_failure_kind() method.
GateFailure with failure_kind=Build maps to ConflictDetected (so the
existing 998 subscriber auto-spawns mergemaster). Other gate failures
stay GatesFailed.
- agents/pool/pipeline/merge/runner.rs: replace the inline match with a
call to the new method.
Tests: 6 new unit tests covering the classifier branch and every
to_merge_failure_kind arm. All 2932 tests pass.
A mid-merge server restart used to silently kill the merge: the
in-flight tokio task died with the process, reap_stale_merge_jobs ran
on the new boot, saw the Running entry from the previous boot, and
simply deleted it. Mergemaster polling `get_merge_status` then saw
"Merge job disappeared", treated it as a strike, and after three
restarts escalated the story to MergeFailureFinal — even though no
real merge failure ever happened (this is what trapped story 998
during the bug 1001 iteration cycle).
Reap now also fires a `WatcherEvent::WorkItem reassign` for the
cleared story so the auto-assign watcher loop re-runs
start_merge_agent_work on the fresh boot. The story is still in
4_merge/; the merge resumes automatically. The change is contained to
the reap path — start_merge_agent_work's own behaviour is unchanged.
Added regression test
reap_stale_merge_jobs_emits_reassign_watcher_event that asserts the
new event fires. Existing
reap_stale_merge_jobs_removes_old_running_entry_without_merge still
passes (the "without_merge" guarantee is about agent spawning, not
about absence of watcher events).
Also exposes AgentPool::watcher_tx() as pub(crate) so the merge
runner can fan out re-dispatch events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MergeFailureFinal was unreachable from move_story: the only transitions
out were Freeze (→ Frozen) and a self-loop on MergemasterAttempted, so
once mergemaster exhausted its 3-retry budget the only way to get a
story coding again was to delete + recreate it.
The respawn budget is a mergemaster bookkeeping detail, not a hard
ceiling. A human operator inspecting a Final story can reasonably
decide the gate failure is fixable, so this adds the same
FixupRequested → Coding edge that already exists for plain
MergeFailure.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The recovery tool was a one-shot migration aid for the half-written
items that existed before the Stage 1 allocator fix. The three live
orphans (989/1000/1001) have been migrated; the Stage 1 fix prevents
new half-writes; the tool's job is done.
Removes the MCP wrapper, schema, dispatch case, and tools-list
assertion. The db::recover module itself stays in-process (under
`#[allow(dead_code)]`) so it can be re-exposed quickly if the bug
ever resurfaces — its regression tests still run as part of the
default suite.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first dry-run against the live pipeline surfaced 735 orphans (35
tombstoned half-writes, 700 stale content rows with no CRDT entry —
mostly artefacts of the pre-numeric-id era). Bulk-recovering would
resurrect a lot of stories the user deliberately purged in the past.
Add an optional `story_ids` filter that restricts both discovery (in
dry-run) and recovery to a named subset, so the operator can target
the specific recent half-writes without touching anything else. The
new test asserts the filter is honoured.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds db::recover, a discovery + recovery layer for pipeline items that
got half-written before the Stage 1 fix landed (content in content
store + SQLite shadow, no live CRDT entry). For each orphan, the
content body is re-anchored to a fresh non-tombstoned id and the old
id's content row is cleared.
Exposed as the recover_half_written_items MCP tool. dry_run defaults
to true so the caller can review what would change before mutating.
YAML front-matter parsing is hand-rolled and scoped to the three
fields the create_*_file path emits (name, type, depends_on). It
tolerates missing or malformed lines by falling back to safe
defaults; the orphan is recovered with the best metadata we can pull
from the body and the rest is left to the operator to fix up.
The discovery step is read-only and idempotent. Recovery is also
idempotent in the sense that once an orphan is lifted, the next
discovery pass won't see it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause: db::next_item_number scanned the visible CRDT index and the
content store but not the tombstone set, so it would hand out a numeric
ID whose CRDT entry had been tombstoned. crdt_state::write_item then
silently no-op'd the insert (tombstone-match guard) while the content
store and SQLite shadow happily accepted the row, producing a split-
brain half-write that was invisible to every CRDT-driven read path and
couldn't be cleaned up by delete_story / purge_story.
This change closes the loop:
- crdt_state::read::{is_tombstoned, tombstoned_ids} expose the
tombstone set so callers outside crdt_state can consult it.
- db::next_item_number now scans tombstoned_ids() too. The allocator
skips past tombstoned numeric IDs instead of treating their slots as
free.
- write_item logs a WARN when it rejects a write for a tombstoned ID
(was silent). The warn is a tripwire — if the allocator ever lets one
slip through again we'll see it in the log.
- create_item_in_backlog adds two defence-in-depth checks:
(a) before any write, reject if the allocator returned a
tombstoned ID;
(b) after the writes, call read_item to confirm the CRDT entry
materialised. If not, roll back the content-store + shadow-DB
rows via db::delete_item and return Err.
Regression tests cover the allocator skip, the is_tombstoned accessor,
and the create_item_in_backlog rollback path.
Out of scope for this commit:
- Recovery of the already-half-written items currently in the running
pipeline (989, 1000, 1001) — Stage 2/3 of the plan, handled
separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Distinct icons in StagePanel/GatewayPanel/render.rs status output for
blocked-with-running-recovery (robot), blocked-with-queued-recovery (hourglass),
and blocked-cold (red circle). All 2822 tests pass.
keepalive_connection_survives_with_pong_responses set ping_ms=100,
timeout_ms=250, so the server's pong-deadline fired ~560ms after the
first ping — only ~60ms past the end of the test's 500ms await window.
Under CI scheduler jitter that 60ms slack was insufficient and the
server timer fired inside the test window, closing the connection
mid-await and producing a flake.
Bump timeout_ms to 2000ms so the pong-deadline cannot fire within
the test window under any realistic jitter. ping_ms stays at 100ms
so the test still exercises multiple ping/pong rounds in the same
wall-clock budget.
Test still passes locally; was hitting 964's merge gate as a flake.
Reorders the gate so fmt --check, duplicate-module scan, clippy, and
doc-coverage run before the frontend build and the multi-minute test
suites. set -euo pipefail short-circuits on the first failure, so a
fmt or clippy drift now fails in seconds instead of after a 30s
frontend build.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "living document" rule was soft and got ignored — coders wrote
PLAN.md once at session start and then drifted away from it. Tie the
update to a trigger they already do (the wip/final commit), and call
out stale "Current state" as a process failure.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 12 tests in `agents::pool::pipeline::merge::tests` share a
process-wide `server_start_time` (a `OnceLock` captured the first time
the merge subsystem runs) and the global merge-job CRDT log. Default
cargo parallelism has caught at least one interleaving on the merge
gate's Docker scheduler where `stale_running_merge_job_is_cleared_and_retry_succeeds`
flakes — `delete_merge_job` from one test lands while another is mid-
assertion. Couldn't reproduce locally despite many tries.
Each test now acquires a poison-tolerant `std::sync::Mutex` at entry,
so the 12 tests run serially relative to each other while the rest of
the suite (2862 tests) stays parallel. Module-level
`#![allow(clippy::await_holding_lock)]` covers the deliberate sync
guard across `.await`s.
Targeted isolation — not a global `--test-threads=1`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five tests pin down the contract of `migrate_legacy_stage_strings`:
rewrite of all pre-934 directory-style strings to clean wire form,
the lossy `7_frozen` → backlog + frozen-flag collapse, no-op on
already-clean items, idempotence, and graceful behaviour before
CRDT init. A test-only `seed_with_raw_stage` helper bypasses the
boundary normalisers (which can't produce legacy strings) by writing
directly to the CRDT register — the same shape we'll see in real
pre-migration data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The state machine's `Stage` enum becomes the source of truth for pipeline
state. Six stages of work land together:
1. Clean wire vocabulary (`coding`, `merge`, `merge_failure`, ...) replaces
legacy directory-style strings (`2_current`, `4_merge`, ...) on the wire.
`Stage::from_dir` accepted both during deployment; new writes always
emit the clean form via `stage_dir_name`. Lexicographic `dir >= "5_done"`
checks in lifecycle.rs become typed `matches!` checks since the new
vocabulary doesn't sort in pipeline order.
2. `crdt_state::write_item` takes typed `&Stage`, serialising via
`stage_dir_name` at the CRDT boundary. `#[cfg(test)] write_item_str`
parses legacy strings for test fixtures.
3. `WorkItem::stage()` returns typed `crdt_state::Stage`; `stage_str()`
is gone from the public API. Projection dispatches on the typed enum.
4. `frozen` becomes an orthogonal CRDT register. `Stage::Frozen` and
`PipelineEvent::Freeze`/`Unfreeze` are removed; `transition_to_frozen`/
`unfrozen` set the flag directly without touching the stage register.
5. Watcher sweep and `tool_update_story`'s `blocked` setter route through
`apply_transition` so the typed transition table validates every
stage change. `update_story` gains a `frozen` field for symmetry.
6. One-shot startup migration rewrites pre-934 directory-style stage
registers (and sets `frozen=true` on items previously at `7_frozen`).
`Stage::from_dir` drops legacy aliases. The db boundary keeps a small
normaliser so callers with legacy strings (MCP, tests) still work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Final 929 sweep: every YAML-shaped helper is gone. No production code
parses or writes YAML front matter anywhere.
Surface removed:
- db/yaml_legacy.rs (FrontMatter/StoryMetadata structs, parse_front_matter,
set_front_matter_field, yaml_residue marker) — file deleted.
- ItemMeta::from_yaml — deleted; callers pass typed ItemMeta::named(...) or
ItemMeta::default() and use typed CRDT setters (set_depends_on,
set_blocked, set_retry_count, set_agent, set_qa_mode, set_review_hold,
set_item_type, set_epic, set_mergemaster_attempted) for the rest.
- write_coverage_baseline_to_story_file + read_coverage_percent_from_json —
the coverage_baseline YAML field was write-only (nothing read it back);
removed along with its caller in agent_tools/lifecycle.rs.
- update_story_in_file's generic `front_matter` HashMap parameter —
tool_update_story now intercepts every known field name and routes it
to a typed CRDT setter; unknown keys are rejected with an explicit error
pointing at the typed setters. The function only takes user_story /
description sections now.
- All 117 ItemMeta::from_yaml callsites migrated. Where tests previously
passed a YAML-shaped content blob and relied on the helper to extract
name/depends_on/blocked/agent/qa, they now pass:
write_item_with_content(id, stage, content, ItemMeta::named("Foo"))
crate::crdt_state::set_depends_on(id, &[...]) // when needed
crate::crdt_state::set_blocked(id, true) // when needed
crate::crdt_state::set_agent(id, Some("...")) // when needed
- write_story_content + write_story_file (test helper) now take an
explicit `name: Option<&str>` instead of parsing it from content.
- db::ops::move_item_stage stopped re-parsing YAML on every stage
transition; metadata is read straight from the CRDT view when mirroring
the row into SQLite.
New CRDT setters added for symmetry:
- crdt_state::set_name (mirrors set_agent — explicit name updates).
cargo fmt --check, clippy --all-targets -- -D warnings, and the
2830-test suite all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every stage transition was reading the content body's YAML front matter to
derive name/agent/blocked/depends_on, then writing those values straight
back into the CRDT registers — but the CRDT was already the source of
truth for all of these fields. The reparse was at best a no-op and at
worst could regress the CRDT to stale YAML values during transitions on
items whose YAML was out of date.
Now move_item_stage:
- writes the new stage to the CRDT with None for every other field, so
write_item leaves existing registers untouched.
- reads name/agent/blocked/depends_on back from the CRDT view when
mirroring the row into the SQLite shadow table (still needed because
the shadow stores a denormalised snapshot for read-side queries).
The yaml_legacy::parse_front_matter import is gone from db/ops.rs; the
only path still using it on the production side is ItemMeta::from_yaml,
which is a caller convenience (mostly used in test fixtures).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After 932 (review_hold register) and 933 (item_type + epic registers), the
remaining production yaml_legacy callers all had typed CRDT equivalents.
Migrated:
- agents/lifecycle.rs:
- transition_to_merge_failure writes to MergeJob.error CRDT entry instead
of YAML body. The legacy `merge_failure: "..."` front-matter write is gone.
- reject_story_from_qa inlines the QA-rejection notes append; no longer
needs yaml_legacy::write_rejection_notes_to_content.
- fields_to_clear_transform helper deleted along with all five callers —
blocked/retry_count/merge_failure are typed CRDT fields now, so clearing
the equivalent YAML keys is redundant.
- http/workflow/pipeline.rs:
- load_pipeline_state reads merge_failure from MergeJob.error (mirrors
status_tools.rs).
- validate_story_dirs checks the typed CRDT `name` register instead of
parsing YAML front matter.
- http/mcp/status_tools.rs: review_hold reads the typed CRDT register
(yaml_residue wrap was the last one in this file).
- http/mcp/story_tools/criteria.rs: story_name reads from CRDT.
- service/agents/mod.rs::get_work_item_content: name/agent come from CRDT.
- service/notifications/io/mod.rs::read_story_name: same.
- http/workflow/bug_ops/{bug,refactor}.rs: name-fallback paths drop YAML
parsing in favour of the CRDT-derived item.name.
Dead helpers removed from db/yaml_legacy.rs:
yaml_residue, write_merge_failure_in_content, write_rejection_notes_to_content,
clear_front_matter_field_in_content, write_review_hold_in_content,
clear_front_matter_field, write_review_hold (the last four shipped in 932).
Remaining surface: FrontMatter / StoryMetadata structs, parse_front_matter,
set_front_matter_field — kept for `coverage_baseline` writes via
test_results.rs and the generic update_story front_matter escape hatch.
Test fixtures rewritten to seed the CRDT register instead of relying on
YAML parsing during write_item_with_content:
- has_review_hold_returns_* tests
- item_type_from_id_uses_crdt_register_for_numeric_ids
- tool_list_epics_shows_member_rollup
- get_work_item_content (both copies — http/agents + service/agents)
- validate_story_dirs_missing_name_in_crdt
- server_side_merge_*_sets_merge_failure (assert MergeJob.error, not YAML)
cargo fmt --check, clippy --all-targets -- -D warnings, and the
2856-test suite all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the YAML-only `type: epic` / `epic: <id>` front-matter fields with
typed CRDT registers on PipelineItemCrdt. The epic-mechanism MCP tools
(`tool_list_epics`, `tool_show_epic`), the epic-context injection in agent
spawn, and the type-classifier helpers (`item_type_from_id`, `is_bug_item`,
`is_refactor_item`) now all read from the CRDT.
Schema:
- PipelineItemCrdt: `item_type: LwwRegisterCrdt<String>` and
`epic: LwwRegisterCrdt<String>` registers.
- WorkItem: typed `item_type()` and `epic()` accessors returning `Option<&str>`.
- crdt_state::set_item_type(story_id, Option<&str>) and
crdt_state::set_epic(story_id, Option<&str>) typed setters.
Write paths populate the new registers:
- create_story_file / create_bug_file / create_spike_file /
create_refactor_file / create_epic_file — each calls set_item_type after
write_story_content.
- tool_update_story intercepts `epic` and `type` fields and routes them to
the typed setters (same pattern as qa / depends_on).
Read paths migrated off yaml_legacy:
- http/mcp/story_tools/epic.rs: tool_list_epics + tool_show_epic.
- agents/lifecycle.rs::item_type_from_id (numeric-only IDs).
- agents/pool/start/spawn.rs epic-context injection.
- http/workflow/bug_ops/bug.rs::is_bug_item, refactor.rs::is_refactor_item.
- http/workflow/pipeline.rs::load_pipeline_state — review_hold/qa/epic_id
all come from the CRDT now; only merge_failure is still YAML (sweep in
929 stage 10).
All `yaml_residue(...)` wraps for item_type / epic are removed; the
remaining residue marker doc no longer references 933.
cargo fmt --check, clippy --all-targets -- -D warnings, and the 2857-test
suite all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
review_hold is now a typed bool register on PipelineItemCrdt alongside
blocked / mergemaster_attempted. Exposed via the typed setter
`crdt_state::set_review_hold(story_id, value)` and the
`WorkItem::review_hold()` accessor. Replaces the legacy
`review_hold: true` YAML front-matter field.
Migrated callers:
- http/mcp/qa_tools.rs::tool_approve_qa — clear via set_review_hold(false)
- agents/lifecycle.rs::reject_story_from_qa — clear via set_review_hold(false)
- agents/pool/pipeline/advance/helpers.rs::write_review_hold_to_store
— set via set_review_hold(true), no more content rewrite
- agents/pool/auto_assign/reconcile.rs (two callsites) — set via
set_review_hold(true) instead of FS YAML write
- agents/pool/auto_assign/story_checks.rs::has_review_hold — reads the
typed register instead of conflating with Stage::Frozen (real bug fix:
the legacy implementation returned `stage.is_frozen()`, which made
the auto-assigner treat *every* held-for-review item as frozen even
when it wasn't actually parked at the freeze stage).
Dead yaml_legacy helpers removed:
- write_review_hold(path), write_review_hold_in_content(content)
- clear_front_matter_field(path) — last caller was the qa_tools wrap
The yaml_residue marker doc now only mentions 933; the 932 line is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
io/watcher and io/watcher/sweep were already CRDT-only — the watcher only
watches .huskies/{project,agents}.toml, work-item events come from CRDT
subscribe — so the remaining FS shadow reader was the bug-503 archived-dep
warning in story_tools/story/create.rs (via check_archived_deps_from_list,
which scanned .huskies/work/6_archived/). Migrate that call to the
CRDT-direct `dep_is_archived_crdt`. Drop the now-unused helper and the
four dead imports in bug/spike/refactor/criteria.rs that referenced it.
io/story_metadata/deps.rs is reduced to a module-level comment pointing
callers at the crdt_state helpers; nothing in io/ now scans the FS shadow
tree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The startup reconciler still pokes review_hold into the on-disk story file
when promoting human-QA items, because no CRDT register exists yet for
review_hold (filed as sub-story 932). The two write-side callsites in
reconcile.rs were the last bare yaml_legacy:: calls in production write
paths; wrap them in yaml_residue so the gap shows up in
`grep -rn yaml_residue` like the other 932/933 markers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
transition_to_frozen and transition_to_unfrozen no longer touch YAML; both
now just call apply_transition with no content_transform. Pairs with the
stage-6 read-side change in projection.rs.
Story 934 will obviate the entire resume_to mechanism by making frozen a
flag orthogonal to Stage (story stays in its current Stage when frozen).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
projection::project_stage was the last yaml_legacy reader in pipeline_state.
Drop the read_content+parse_front_matter detour for the "7_frozen" case and
always default resume_to to Stage::Coding. The YAML write side in apply.rs
goes in stage 7.
Story 934 (sibling refactor) will replace Stage::Frozen-with-payload with a
frozen flag orthogonal to Stage, so a story frozen in Qa stays in Stage::Qa
rather than encoding a "where to resume" target. After 934 lands the
resume_to payload disappears entirely.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrate the last three callers of the FS-scanning dependency helpers to the
CRDT-direct equivalents and delete the dead helpers:
- agents/pool/auto_assign/story_checks.rs: has_unmet_dependencies and
check_archived_dependencies now wrap check_unmet_deps_crdt /
check_archived_deps_crdt directly. Tests rewritten to seed the CRDT.
- http/mcp/story_tools/story/update.rs: bug-503 archived-dep warning now
reads from CRDT instead of scanning 6_archived.
- agents/pool/pipeline/advance/helpers.rs: resolve_qa_mode_from_store is
CRDT-only (the FS fallback for content-store-empty stories is gone).
- io/story_metadata/parser.rs: resolve_qa_mode_from_content removed.
- io/story_metadata/deps.rs: check_unmet_deps and dep_is_done deleted,
along with the unused check_unmet_deps_from_list helper.
- io/story_metadata/mod.rs: re-exports trimmed accordingly.
check_archived_deps_from_list survives because story-creation still calls
it before the CRDT entry exists (used from story_tools/story/create.rs).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Read-side migrations:
- agents/pool/auto_assign/backlog.rs: depends_on check now reads from
WorkItem.depends_on() instead of parse_front_matter.
- agents/pool/auto_assign/story_checks.rs: read_story_front_matter_agent
drops its YAML fallback — post-891 the CRDT entry is reliable, and
removing the fallback makes the contract honest. The now-unused
read_story_contents helper goes too.
- agents/pool/start/validation.rs: same shape — YAML fallback removed,
CRDT register is the only source for agent pinning.
- agents/pool/start/spawn.rs: epic-context injection wraps the
parse_front_matter call in `yaml_residue(...)` since `meta.epic` has no
CRDT analog (sub-story 933).
- agents/lifecycle.rs: item_type_from_id (numeric-only ID path) wraps its
parse_front_matter in `yaml_residue(...)` for the same reason (933).
The write-side `fields_to_clear_transform` calls in lifecycle.rs are
left for stage 8, when FS-shadow writes are deleted wholesale.
Test fix:
- start_agent_returns_error_when_front_matter_agent_busy now seeds the
CRDT entry (write_item with agent="coder-opus") instead of relying on
parse_front_matter reading the YAML on disk.
Filed earlier:
- 932 (review_hold register) — note: this turns out to be a real class-1
bug: write_review_hold_to_store still writes YAML but has_review_hold
reads Stage::Frozen, so the write goes into a void. 932 is the correct
fix.
All 2861 tests pass; fmt + clippy clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three MCP files touched:
- status_tools.rs (story-status JSON dump): every field with a CRDT
equivalent now reads from WorkItem (name, agent, blocked, qa_mode,
retry_count, depends_on, claimed_by, claimed_at) or MergeJob.error
(merge_failure detail). One field — review_hold — has no CRDT register
yet (sub-story 932) and is wrapped in `yaml_residue(parse_front_matter(...))`
so the gap is visible at every code-search.
- qa_tools.rs:
• tool_approve_qa wraps the legacy `clear_front_matter_field("review_hold")`
write in `yaml_residue(...)` pending sub-story 932.
• tool_reject_qa now reads the agent name from the CRDT WorkItem instead
of parsing front matter on disk.
- story_tools/epic.rs: the entire epic feature (item_type, epic link)
has no CRDT analog — sub-story 933. Every parse_front_matter call here
is wrapped in `yaml_residue(...)`.
Also: new identity wrapper `db::yaml_legacy::yaml_residue<T>(v: T) -> T`
that marks a yaml_legacy callsite blocked on a CRDT-register gap. Pure
identity at runtime; the distinctive name makes the residue grep-findable
(`grep -rn yaml_residue`). Sub-stories 932 and 933 enumerate the gaps.
Filed:
- 932: Add CRDT register for review_hold
- 933: Add CRDT registers for the epic mechanism
All 2854 tests pass; fmt + clippy clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
delete.rs, start.rs, assign.rs all looked up the story name by reading
the content from disk/store and parsing the front matter. Replaced with
`crdt_state::read_item(&story_id).and_then(|w| w.name())`. Each callsite's
fallback chain ("CRDT → content store → filesystem") still locates the
story_id; only the name extraction moved off YAML.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each chat command that previously read parse_front_matter for story
metadata (name, agent, depends_on, blocked, retry_count, merge_failure,
qa_mode) now reads from the typed CRDT API:
- WorkItem (via crdt_state::read_item) for pipeline-item registers.
- MergeJobView (via crdt_state::read_merge_job) for the merge failure
detail text, which has its own LWW-map CRDT entry.
Files migrated: depends.rs, freeze.rs, move_story.rs, overview.rs,
status/render.rs, triage.rs, unblock.rs, unreleased.rs.
unblock.rs: also removes the legacy front-matter cleanup branch that
fired when the typed Blocked→Coding transition failed. Post-929 there
is no YAML on disk to clean; the fallback now just resets retry_count
in the CRDT.
triage.rs: drops the YAML-only `review_hold` and `coverage_baseline`
fields from the dump. These have no CRDT register and were never
load-bearing on the triage output; if needed later, add a CRDT register
and surface it back.
Tests:
- The three status/render merge-failure rendering tests now seed a
MergeJob CRDT entry via write_merge_job instead of writing YAML.
- The unblock test that asserted YAML cleanup on disk is now an assertion
on the CRDT registers (blocked=false, retry_count=0). Also re-seeded
in `2_blocked` stage so the typed Blocked → Coding transition actually
fires (not the fallback path).
All 2855 tests pass; fmt clean; clippy clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Observed: stories 917, 918, 920, 910 all turn-limit-killed despite producing
real commits. Tally across their session logs shows 30–55% of assistant
turns were pure narration ("I'll read X next", "Now let me check Y") with
no tool_use. At 80 max_turns the effective work budget was ~44 tool calls,
not enough for a typical bug fix's edit + test + check_criterion cycle.
Changes:
- New optional AgentConfig field max_tool_turns. When set the watchdog
uses it instead of max_turns; only assistant messages whose
data.message.content has at least one tool_use block count.
- count_turns_in_log in agents/pool/auto_assign/watchdog/limits.rs
filters on tool_use. Existing test helper write_fake_session_log now
emits tool_use blocks; added write_fake_mixed_session_log for the
narration regression test.
- agents.toml: coders/coder-opus get max_turns=200 (claude-code's own
--max-turns cap, sized to never bite before the watchdog) and
max_tool_turns=80. qa: 120 / 40. mergemaster: 250 / 100. Budgets
unchanged — the dollar cap remains the runaway-loop backstop, with
~$3-5 worst-case waste if an agent narrates indefinitely.
- Two new regression tests:
* watchdog_does_not_count_narration_only_turns: 5 tool + 30 narration
under max_tool_turns=10 stays Running.
* watchdog_max_tool_turns_overrides_max_turns: 4 tool turns at
max_tool_turns=3 / max_turns=200 still terminates with TurnLimit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
agent_log::format::format_log_entry_as_text was truncating long tool_result
strings via the naive byte slice `&content_str[..500]`. When byte 500 fell
inside a multi-byte UTF-8 codepoint (box-drawing chars like '─', smart
quotes, emoji), the slice panicked, propagating up through the MCP
get_agent_output dispatcher and surfacing as an internal-error response.
This blocked any diagnostic readout of a coder's session that had emitted
tool output containing those chars.
Walk back to the nearest char boundary with `is_char_boundary` before
slicing. Regression test asserts the formatter doesn't panic on a 599-byte
string with a 3-byte '─' straddling byte 500.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original 90b31fc8 test computed reset_at = now + 3s in the test thread,
then relied on the script spawning fast enough that the rate_limit_event
arrived while reset_at was still meaningfully in the future. Under
cargo-test load the spawn could take long enough that block_until - now
clamped to 0 and the inactivity timeout killed the script before its sleep
finished. Pin reset_at to 2099-01-01 (matching the existing
rate_limit_hard_block_sends_watcher_hard_block_event test) so the
extension is essentially infinite and the assertion isolates the
extension-vs-no-extension behavior from wall-clock slack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When claude-code emits a rate_limit_event with status != allowed_warning,
the subprocess waits internally for the limit to clear before retrying. No
PTY output flows during that window, so the inactivity timeout in the PTY
runner would fire and kill the agent — mergemaster especially, whose
15-minute inactivity window is shorter than typical rate-limit backoffs.
Track `block_until = Some(reset_at)` on hard-block events and add the
remaining time-until-reset to the per-iteration recv timeout. Once reset_at
passes (or an earlier emit arrives), the extension implicitly drops to 0
and the base inactivity timeout resumes. Turn/budget counts aren't affected
— they come from the session log and only advance when API calls actually
complete, so a stalled retry doesn't burn either.
Regression test in agents/pty/mod.rs spawns a script that emits a hard-block
with reset_at = now+3s, sleeps 3s, then exits, with inactivity_timeout_secs
= 1. Without the fix the runner kills the script at 1s; with the fix the
deadline is bumped past the sleep and the run completes cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause was not the persist channel (the test-mode channel is unbounded
and its receiver is leaked, so sends never fail). It was that `ALL_OPS` and
`VECTOR_CLOCK` were process-wide `OnceLock` globals while `CRDT_STATE` was
already thread-local — so one test thread's `apply_compaction` would prune
another test thread's freshly-written ops out of the shared journal, and
the subsequent `all_ops_json()` read in `compaction_reduces_ops` would
return fewer than the 5 it had just written.
Mirror the pattern already used for `CRDT_STATE` and `SnapshotState`: in
`cfg(test)` use thread-local `OnceLock<Mutex<...>>`s for the op journal and
vector clock, accessed via new `all_ops_lock()` / `vector_clock_lock()`
helpers. Production code path is unchanged (still the global statics set
during `init()`).
Touches ops/read/snapshot call sites to go through the helpers. Note in
passing that this overlaps backlog story 518; that story is about the
production-side persist path, this is the cfg(test)-only journal-isolation
slice.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 903 (run_tests attach instead of respawn) + 904 (MCP progress
notifications + SSE) together eliminate the transport-timeout error
mode from the agent's point of view: long test runs complete without
the MCP client ever observing a tool-call error. Production
verification (see d64f1e94 / ddc4228b deploy at 14:30 UTC today)
confirmed 78s and 65s test runs completing in single processes with
no respawn churn and no retry needed.
The "If run_tests errors with a transport timeout, call it again"
sentence in coder-1/2/3/opus system_prompts (added belt-and-braces
in a97a10fb) is now redundant. Removing it tightens the agent's
mental model down to: call run_tests, wait for the result. No
error-handling branch, no retry semantics to internalise.
This closes the last open AC on story 904.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two latent bugs in `service/gateway/io.rs::spawn_gateway_bot`, exposed
today after a long-overdue gateway rebuild:
1. The permission channel sender was bound as `_perm_tx` (the underscore
prefix signalling "unused") and dropped at function return. The
matrix bot's permission_listener task — which holds `perm_rx` for
its lifetime per story 884 — then saw the channel close immediately
and exited with "perm_rx channel closed" 1s after starting. Net
effect: the listener was effectively absent on every gateway boot,
so non-MCP tool permission requests had no destination at all
(separate from the architectural mismatch that 898 will fix; this
was a strictly worse "listener never even ran" version of the same
problem). Bind as `perm_tx` and `mem::forget` it to keep the
channel open for the gateway's lifetime, mirroring the existing
`shutdown_tx` pattern two lines below.
2. `bot_name` was hardcoded to `"Assistant"`, ignoring
`bot.toml::display_name`. So the gateway's matrix bot announced
itself as "Assistant" and treated user messages addressed to
"Timmy" (the actual configured display_name) as unaddressed,
silently dropping them. `ambient_rooms` and
`permission_timeout_secs` were similarly ignored. Load
`BotConfig::load(config_dir)` and apply the same field plumbing
the standard-mode initialisation in `main.rs:211-232` already
uses.
Symptoms seen in production today:
- gateway.log: `Sending startup announcement: Assistant is online.`
followed by repeated `Ignoring unaddressed message from
@yossarian:crashlabs.io` lines.
- gateway.log: `permission listener started` immediately followed
(same timestamp) by `permission listener exiting (perm_rx channel
closed)`.
After this lands, rebuild the gateway binary and restart so it picks
up `bot.toml` correctly and the listener stays alive for the bot's
lifetime.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to bug 903. The attach fix made run_tests retries safe, but
agents still observed the underlying MCP transport timeout as a
tool-call error and had to handle it via retry. Implement the proper
fix: MCP `notifications/progress` events keep the client's transport
timer alive so the call never errors from the agent's perspective.
What changed:
server/src/http/mcp/progress.rs (new)
- `ProgressEmitter` (progressToken + mpsc sender) installed in a
`tokio::task_local!` scope by the SSE response path.
- `emit_progress(progress, total, message)` builds a JSON-RPC
`notifications/progress` message and sends it via the channel.
No-op when no emitter is in scope (plain JSON path / tests / API
runtimes), so tool handlers can call it unconditionally.
server/src/http/mcp/mod.rs
- mcp_post_handler now detects `Accept: text/event-stream` AND a
`params._meta.progressToken` on tools/call. When both are present,
routes through `sse_tools_call` instead of the plain JSON path.
- sse_tools_call: spawns the dispatch task with the emitter installed,
builds an SSE stream that interleaves incoming progress events with
the final JSON-RPC response, with a 15s keep-alive interval as a
backstop for tools that don't emit their own progress.
- Plain JSON behaviour is unchanged for non-SSE clients and for
everything other than tools/call.
server/src/http/mcp/shell_tools/script.rs
- tool_run_tests poll loop emits `notifications/progress` every 25s
of elapsed time (well below the typical ~60s MCP transport
timeout). Attached callers (the bug 903 fix path) also emit so
their MCP socket stays alive while waiting for the in-flight job.
- Output filtering: on a passing run the response now returns a
one-line summary ("All N tests passed.") instead of the full
`cargo test` stdout, which was pure noise that burned agent
tokens. Failure output is unchanged (truncated tail with the
`failures:` section and final test_result line). CRDT entry
stores the same filtered value so attached callers see it too.
Tests (3 new):
- emit_progress_no_op_without_emitter — calling outside scope is safe
- emit_progress_sends_notification_when_emitter_installed — full path
- emit_progress_omits_optional_fields — total/message optional
Not changed: coder system_prompts still tell agents to retry on
transport-timeout errors. That advice is now belt-and-braces — if
claude-code's HTTP MCP client honours progress notifications, no agent
will ever observe the error; if not, retry is still safe post-903. We
can drop the retry advice once we've observed the SSE path working in
the field.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-d64f1e94 the "call run_tests again — it attaches" guidance was a
lie (every call killed the prior job and spawned a fresh one). With
the attach fix in place, the contract is now real and safe to depend
on. Tighten the wording so agents see exactly what to do:
OLD: "Do not use ScheduleWakeup to wait for run_tests; if run_tests
appears to time out, call run_tests again — it attaches to the
in-flight test job and blocks until completion."
NEW: "If run_tests errors with a transport timeout, call it again —
it's idempotent and attaches to the same in-flight test job,
so retries are safe and eventually return a pass/fail result."
Improvements:
- "errors with a transport timeout" matches what the agent literally
observes (a tool-call error), not the vague "appears to time out".
- Explicit on idempotency so agents understand why retry is safe and
don't worry about double-running the suite.
- Drops the ScheduleWakeup clause — already enforced via the
`disallowed_tools` setting on coder-1/2/3/opus, so the prompt
reminder was redundant.
Applied uniformly across coder-1, coder-2, coder-3, coder-opus.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 903: every `run_tests` MCP call killed the prior `cargo test` child
for the same worktree and spawned a fresh one. Combined with the
~60s MCP client-side timeout and the 896 agent prompt that told agents
to "call run_tests again — it attaches to the in-flight test job",
this produced a respawn loop: agent calls, MCP times out at 60s, agent
retries, run_tests kills the running build and starts a new one. The
test suite never reaches the finish line.
Server log evidence: "Started test job for <worktree> (pid N)" with a
new PID every ~60-90s for the same worktree.
Fix: when `run_tests` is called and a job is already in flight for that
worktree, ATTACH to it instead of killing+respawning. The original job's
poll loop already writes the final status to the CRDT `test_jobs`
collection; attached callers just poll that CRDT entry (the same
pattern `get_test_result` uses) and return the result when the
in-flight job transitions out of "running". The 896 prompt's claim is
now actually true.
Worktrees remain isolated from each other and may run `cargo test`
concurrently — there is no cross-worktree serialisation. The single
invariant is "at most one test job per worktree at a time".
New test: `tool_run_tests_concurrent_calls_attach_to_single_job`
spawns two concurrent calls on the same worktree against a 2s
`sleep`-based script and asserts total elapsed stays close to 2s
(attach) rather than 4s (respawn).
Note: the cross-worktree linker-OOM symptom Timmy reported in the
field was downstream of the respawn loop. Killed-but-not-fully-reaped
cargo invocations stack memory pressure beyond the nominal N
worktrees. With the attach fix, each worktree runs exactly one
in-flight build at a time and old builds finish cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline gap: the state machine refused `move_story(... target='backlog')`
from a Blocked story, leaving stuck items with no way to be parked while
waiting on dependent fixes — operators had to either Unblock (which
re-enters the active flow) or Archive (which loses the item).
Extend the existing Demote rule so `Blocked + Demote → Backlog` is a
legal transition, alongside the existing `Coding/Qa/Merge + Demote`.
Also update `map_stage_move_to_event` in agents/lifecycle.rs so the
chat/MCP `move_story` API recognises Blocked → backlog and routes it
through `PipelineEvent::Demote`.
Tests:
- `blocked_demote_returns_to_backlog` — happy path.
- `cannot_demote_from_done` / `cannot_demote_from_upcoming` — sanity
checks that the broadened rule does NOT permit Demote from
terminal or pre-triage stages.
Pattern follows 892 (MergeFailure → Done) and 893 (MergeFailure →
Coding) — pure transition.rs extension plus matching event mapping in
lifecycle.rs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 902: the Step 0 "resume from worktree state" instruction told coders
to call git_status / git_log / git_diff to discover prior session work,
which they then extended into hunting for the story `.md` file on disk
via find / ls — pointless post-865, since story content lives only in
the CRDT.
Update Step 0 in coder-1, coder-2, coder-3, and coder-opus to add an
explicit instruction: "To read story content, ACs, or description, call
the `get_story_todos` MCP tool — do NOT search for a story `.md` file
on disk; story content is CRDT-only."
Single substring replacement covers all four agents (identical Step 0
across them).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 901: `unblock_story` (and the chat `unblock` command) routed through
`parse_front_matter` and errored with "Missing front matter" on any
post-865 story (story content is now CRDT-only with no YAML on disk).
In `chat/commands/unblock.rs::unblock_by_story_id`:
- Drop the early `parse_front_matter` gate.
- Read story name and blocked state from the CRDT register API instead
of parsed YAML (`crdt_state::read_item`, `pipeline_state::read_typed`).
- Keep the legacy fallback cleanup, but gate it on the content actually
starting with a `---` YAML block, so CRDT-only stories don't hit a
parse error there either.
- Remove the now-unused `parse_front_matter` import.
Surfaced a second sub-bug: even when the state-machine transition
fired (`Blocked + Unblock → Coding`), the CRDT `blocked` register was
never explicitly cleared. Pre-865 the YAML-strip content_transform
cleared it as a side effect; post-865 there is no YAML to strip.
- Add `crdt_state::set_blocked(story_id, bool)` parallel to
`set_retry_count`. Wired through `crdt_state::write` and the
crate-level re-export.
- `agents::lifecycle::transition_to_unblocked` now calls
`set_blocked(story_id, false)` alongside `set_retry_count(0)` so
the legacy register stays in sync with the typed stage.
Test: `unblock_command_works_on_crdt_only_story_no_yaml` seeds a CRDT
entry with no YAML on disk, runs unblock, asserts success + cleared
blocked + retry_count=0. All 10 existing unblock tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude Code rejects "Bash(:*)" with "Prefix cannot be empty before :*" —
the rule is silently skipped, which since 5b48f0d0 left no Bash entry
in the allowlist at all. Every coder agent's Bash call has been
auto-denying since that commit landed (~840 of 1.4k denials in the sled
log).
The canonical form for "allow all bash commands" is the tool name alone:
"Bash" (no parens). Apply it in three places that 5b48f0d0 touched:
- .claude/settings.json (project root, inherited by new worktrees)
- server/src/io/fs/scaffold/templates.rs (huskies init template)
- server/src/io/fs/scaffold/tests.rs (assertion now checks "Bash")
The gateway settings.json at ~/Desktop/huskies/.claude/settings.json and
the four live worktrees (810, 888, 890, 894) were also corrected — not
in this commit since they live outside the repo.
Surfaced via /doctor; reported with rule "Invalid permission rule
Bash(:*) was skipped: Prefix cannot be empty before :*".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-command allowlist (Bash(cargo:*), Bash(git:*), …) misses any tool
a coder agent reaches for outside the curated set — ./script/*, make, curl,
jq, docker, test, [, etc. Each miss hits prompt_permission, which auto-denies
on the sled because no listener holds perm_rx (the matrix bot lives in the
gateway). 1,377 such denies in the sled log over the past week, accounting
for most of the recent throughput slowdown.
Replace the curated list with a single Bash(:*) wildcard in:
- .claude/settings.json (project root, picked up on git worktree add)
- server/src/io/fs/scaffold/templates.rs (used only by huskies init when
no .claude/settings.json already exists)
Update scaffold/tests.rs to assert the wildcard rather than a fixed set
of patterns; the per-command gate offered no real safety in this trusted
single-user deployment, since the prompt was never going to reach a human
anyway (that's the bug).
Stopgap until story 898 lands the proper sled→gateway permission
forwarding — at which point the wildcard can be narrowed back if desired.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
855 deleted the HTTP /mcp route and pointed agents at ws://...crdt-sync,
but Claude Code's .mcp.json doesn't speak ws:// and the rendezvous WS
never had MCP method handlers wired up — so every spawned Claude Code
agent (gateway-routed and local) booted with zero huskies tools and
died on --permission-prompt-tool=mcp__huskies__prompt_permission.
Restore mcp_post_handler / mcp_get_handler / handle_initialize, re-add
the /mcp route, and revert all three .mcp.json writers to emit
http://localhost:{port}/mcp with explicit "type": "http". Reuses the
already-extracted gateway::jsonrpc types and the surviving
dispatch_tool_call / list_tools surfaces — net add ~140 lines.
Federation work is unaffected: /crdt-sync continues to do CRDT sync,
which is what it was actually doing. MCP-over-WebSocket for cross-LAN
agents was never wired up by 855 and can be done as a proper follow-up
with a regression test that boots a real claude and verifies tool
registration.
Verified end-to-end: /mcp initialize, tools/list (74 tools incl.
prompt_permission), and tools/call all respond correctly from inside
the rebuilt container.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In gateway mode the bot's Claude Code CLI was spawned with cwd set to
a nonexistent project subdirectory (gateway_config_dir/project_name).
This meant it couldn't find .mcp.json and had no MCP tools available.
Now the bot uses the gateway config directory as cwd in gateway mode,
where the auto-generated .mcp.json points to the gateway's MCP proxy.
Also fixes cargo fmt formatting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The gateway proxy was sending every message's first word to the project
server's /api/bot/command endpoint, then displaying the "Unknown command"
response before falling through to the LLM. Now the proxy only fires
when the first word matches a known bot command.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
In gateway mode the bot has no local CRDT or project filesystem, so all
bot commands (status, backlog, start, assign, etc.) returned empty or
broken results. Now the gateway bot proxies non-local commands via HTTP
to the active project's /api/bot/command endpoint, which already exists
on every project server.
Only a small set of gateway-local commands (help, ambient, reset, switch)
are still handled directly by the gateway. Everything else is forwarded
automatically, so new commands added in the future will work through the
proxy without additional gateway changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When replaying old CRDT ops that predate new struct fields (e.g.
claimed_by, claim_ts added by story 479), node_from would call
.unwrap() on None and panic during init. Now defaults to an empty
CrdtNode::new() for missing fields, allowing schema evolution without
breaking replay of historical ops.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Squash merge of story 504: add MCP regression tests for non-string
front_matter values (arrays, bools, integers). The schema change itself
was already on master. Fixed the array assertion to match YAML's
space-after-comma inline sequence format.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The post-520 migration changed validate_story_dirs to read from
pipeline_state::read_all_typed() (the process-global CRDT singleton),
ignoring its root: &Path argument. This broke test isolation — tests
creating a tempdir saw dozens of results from ambient CRDT state,
causing non-deterministic failures that blocked every mergemaster gate.
Remove the CRDT singleton block and rely on the filesystem shadow scan
that already uses the root argument correctly. 1845/1845 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TimmyandClaude Opus 4.7