Observed: stories 917, 918, 920, 910 all turn-limit-killed despite producing
real commits. Tally across their session logs shows 30–55% of assistant
turns were pure narration ("I'll read X next", "Now let me check Y") with
no tool_use. At 80 max_turns the effective work budget was ~44 tool calls,
not enough for a typical bug fix's edit + test + check_criterion cycle.
Changes:
- New optional AgentConfig field max_tool_turns. When set the watchdog
uses it instead of max_turns; only assistant messages whose
data.message.content has at least one tool_use block count.
- count_turns_in_log in agents/pool/auto_assign/watchdog/limits.rs
filters on tool_use. Existing test helper write_fake_session_log now
emits tool_use blocks; added write_fake_mixed_session_log for the
narration regression test.
- agents.toml: coders/coder-opus get max_turns=200 (claude-code's own
--max-turns cap, sized to never bite before the watchdog) and
max_tool_turns=80. qa: 120 / 40. mergemaster: 250 / 100. Budgets
unchanged — the dollar cap remains the runaway-loop backstop, with
~$3-5 worst-case waste if an agent narrates indefinitely.
- Two new regression tests:
* watchdog_does_not_count_narration_only_turns: 5 tool + 30 narration
under max_tool_turns=10 stays Running.
* watchdog_max_tool_turns_overrides_max_turns: 4 tool turns at
max_tool_turns=3 / max_turns=200 still terminates with TurnLimit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
agent_log::format::format_log_entry_as_text was truncating long tool_result
strings via the naive byte slice `&content_str[..500]`. When byte 500 fell
inside a multi-byte UTF-8 codepoint (box-drawing chars like '─', smart
quotes, emoji), the slice panicked, propagating up through the MCP
get_agent_output dispatcher and surfacing as an internal-error response.
This blocked any diagnostic readout of a coder's session that had emitted
tool output containing those chars.
Walk back to the nearest char boundary with `is_char_boundary` before
slicing. Regression test asserts the formatter doesn't panic on a 599-byte
string with a 3-byte '─' straddling byte 500.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original 90b31fc8 test computed reset_at = now + 3s in the test thread,
then relied on the script spawning fast enough that the rate_limit_event
arrived while reset_at was still meaningfully in the future. Under
cargo-test load the spawn could take long enough that block_until - now
clamped to 0 and the inactivity timeout killed the script before its sleep
finished. Pin reset_at to 2099-01-01 (matching the existing
rate_limit_hard_block_sends_watcher_hard_block_event test) so the
extension is essentially infinite and the assertion isolates the
extension-vs-no-extension behavior from wall-clock slack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When claude-code emits a rate_limit_event with status != allowed_warning,
the subprocess waits internally for the limit to clear before retrying. No
PTY output flows during that window, so the inactivity timeout in the PTY
runner would fire and kill the agent — mergemaster especially, whose
15-minute inactivity window is shorter than typical rate-limit backoffs.
Track `block_until = Some(reset_at)` on hard-block events and add the
remaining time-until-reset to the per-iteration recv timeout. Once reset_at
passes (or an earlier emit arrives), the extension implicitly drops to 0
and the base inactivity timeout resumes. Turn/budget counts aren't affected
— they come from the session log and only advance when API calls actually
complete, so a stalled retry doesn't burn either.
Regression test in agents/pty/mod.rs spawns a script that emits a hard-block
with reset_at = now+3s, sleeps 3s, then exits, with inactivity_timeout_secs
= 1. Without the fix the runner kills the script at 1s; with the fix the
deadline is bumped past the sleep and the run completes cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause was not the persist channel (the test-mode channel is unbounded
and its receiver is leaked, so sends never fail). It was that `ALL_OPS` and
`VECTOR_CLOCK` were process-wide `OnceLock` globals while `CRDT_STATE` was
already thread-local — so one test thread's `apply_compaction` would prune
another test thread's freshly-written ops out of the shared journal, and
the subsequent `all_ops_json()` read in `compaction_reduces_ops` would
return fewer than the 5 it had just written.
Mirror the pattern already used for `CRDT_STATE` and `SnapshotState`: in
`cfg(test)` use thread-local `OnceLock<Mutex<...>>`s for the op journal and
vector clock, accessed via new `all_ops_lock()` / `vector_clock_lock()`
helpers. Production code path is unchanged (still the global statics set
during `init()`).
Touches ops/read/snapshot call sites to go through the helpers. Note in
passing that this overlaps backlog story 518; that story is about the
production-side persist path, this is the cfg(test)-only journal-isolation
slice.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 903 (run_tests attach instead of respawn) + 904 (MCP progress
notifications + SSE) together eliminate the transport-timeout error
mode from the agent's point of view: long test runs complete without
the MCP client ever observing a tool-call error. Production
verification (see d64f1e94 / ddc4228b deploy at 14:30 UTC today)
confirmed 78s and 65s test runs completing in single processes with
no respawn churn and no retry needed.
The "If run_tests errors with a transport timeout, call it again"
sentence in coder-1/2/3/opus system_prompts (added belt-and-braces
in a97a10fb) is now redundant. Removing it tightens the agent's
mental model down to: call run_tests, wait for the result. No
error-handling branch, no retry semantics to internalise.
This closes the last open AC on story 904.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two latent bugs in `service/gateway/io.rs::spawn_gateway_bot`, exposed
today after a long-overdue gateway rebuild:
1. The permission channel sender was bound as `_perm_tx` (the underscore
prefix signalling "unused") and dropped at function return. The
matrix bot's permission_listener task — which holds `perm_rx` for
its lifetime per story 884 — then saw the channel close immediately
and exited with "perm_rx channel closed" 1s after starting. Net
effect: the listener was effectively absent on every gateway boot,
so non-MCP tool permission requests had no destination at all
(separate from the architectural mismatch that 898 will fix; this
was a strictly worse "listener never even ran" version of the same
problem). Bind as `perm_tx` and `mem::forget` it to keep the
channel open for the gateway's lifetime, mirroring the existing
`shutdown_tx` pattern two lines below.
2. `bot_name` was hardcoded to `"Assistant"`, ignoring
`bot.toml::display_name`. So the gateway's matrix bot announced
itself as "Assistant" and treated user messages addressed to
"Timmy" (the actual configured display_name) as unaddressed,
silently dropping them. `ambient_rooms` and
`permission_timeout_secs` were similarly ignored. Load
`BotConfig::load(config_dir)` and apply the same field plumbing
the standard-mode initialisation in `main.rs:211-232` already
uses.
Symptoms seen in production today:
- gateway.log: `Sending startup announcement: Assistant is online.`
followed by repeated `Ignoring unaddressed message from
@yossarian:crashlabs.io` lines.
- gateway.log: `permission listener started` immediately followed
(same timestamp) by `permission listener exiting (perm_rx channel
closed)`.
After this lands, rebuild the gateway binary and restart so it picks
up `bot.toml` correctly and the listener stays alive for the bot's
lifetime.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to bug 903. The attach fix made run_tests retries safe, but
agents still observed the underlying MCP transport timeout as a
tool-call error and had to handle it via retry. Implement the proper
fix: MCP `notifications/progress` events keep the client's transport
timer alive so the call never errors from the agent's perspective.
What changed:
server/src/http/mcp/progress.rs (new)
- `ProgressEmitter` (progressToken + mpsc sender) installed in a
`tokio::task_local!` scope by the SSE response path.
- `emit_progress(progress, total, message)` builds a JSON-RPC
`notifications/progress` message and sends it via the channel.
No-op when no emitter is in scope (plain JSON path / tests / API
runtimes), so tool handlers can call it unconditionally.
server/src/http/mcp/mod.rs
- mcp_post_handler now detects `Accept: text/event-stream` AND a
`params._meta.progressToken` on tools/call. When both are present,
routes through `sse_tools_call` instead of the plain JSON path.
- sse_tools_call: spawns the dispatch task with the emitter installed,
builds an SSE stream that interleaves incoming progress events with
the final JSON-RPC response, with a 15s keep-alive interval as a
backstop for tools that don't emit their own progress.
- Plain JSON behaviour is unchanged for non-SSE clients and for
everything other than tools/call.
server/src/http/mcp/shell_tools/script.rs
- tool_run_tests poll loop emits `notifications/progress` every 25s
of elapsed time (well below the typical ~60s MCP transport
timeout). Attached callers (the bug 903 fix path) also emit so
their MCP socket stays alive while waiting for the in-flight job.
- Output filtering: on a passing run the response now returns a
one-line summary ("All N tests passed.") instead of the full
`cargo test` stdout, which was pure noise that burned agent
tokens. Failure output is unchanged (truncated tail with the
`failures:` section and final test_result line). CRDT entry
stores the same filtered value so attached callers see it too.
Tests (3 new):
- emit_progress_no_op_without_emitter — calling outside scope is safe
- emit_progress_sends_notification_when_emitter_installed — full path
- emit_progress_omits_optional_fields — total/message optional
Not changed: coder system_prompts still tell agents to retry on
transport-timeout errors. That advice is now belt-and-braces — if
claude-code's HTTP MCP client honours progress notifications, no agent
will ever observe the error; if not, retry is still safe post-903. We
can drop the retry advice once we've observed the SSE path working in
the field.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-d64f1e94 the "call run_tests again — it attaches" guidance was a
lie (every call killed the prior job and spawned a fresh one). With
the attach fix in place, the contract is now real and safe to depend
on. Tighten the wording so agents see exactly what to do:
OLD: "Do not use ScheduleWakeup to wait for run_tests; if run_tests
appears to time out, call run_tests again — it attaches to the
in-flight test job and blocks until completion."
NEW: "If run_tests errors with a transport timeout, call it again —
it's idempotent and attaches to the same in-flight test job,
so retries are safe and eventually return a pass/fail result."
Improvements:
- "errors with a transport timeout" matches what the agent literally
observes (a tool-call error), not the vague "appears to time out".
- Explicit on idempotency so agents understand why retry is safe and
don't worry about double-running the suite.
- Drops the ScheduleWakeup clause — already enforced via the
`disallowed_tools` setting on coder-1/2/3/opus, so the prompt
reminder was redundant.
Applied uniformly across coder-1, coder-2, coder-3, coder-opus.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 903: every `run_tests` MCP call killed the prior `cargo test` child
for the same worktree and spawned a fresh one. Combined with the
~60s MCP client-side timeout and the 896 agent prompt that told agents
to "call run_tests again — it attaches to the in-flight test job",
this produced a respawn loop: agent calls, MCP times out at 60s, agent
retries, run_tests kills the running build and starts a new one. The
test suite never reaches the finish line.
Server log evidence: "Started test job for <worktree> (pid N)" with a
new PID every ~60-90s for the same worktree.
Fix: when `run_tests` is called and a job is already in flight for that
worktree, ATTACH to it instead of killing+respawning. The original job's
poll loop already writes the final status to the CRDT `test_jobs`
collection; attached callers just poll that CRDT entry (the same
pattern `get_test_result` uses) and return the result when the
in-flight job transitions out of "running". The 896 prompt's claim is
now actually true.
Worktrees remain isolated from each other and may run `cargo test`
concurrently — there is no cross-worktree serialisation. The single
invariant is "at most one test job per worktree at a time".
New test: `tool_run_tests_concurrent_calls_attach_to_single_job`
spawns two concurrent calls on the same worktree against a 2s
`sleep`-based script and asserts total elapsed stays close to 2s
(attach) rather than 4s (respawn).
Note: the cross-worktree linker-OOM symptom Timmy reported in the
field was downstream of the respawn loop. Killed-but-not-fully-reaped
cargo invocations stack memory pressure beyond the nominal N
worktrees. With the attach fix, each worktree runs exactly one
in-flight build at a time and old builds finish cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline gap: the state machine refused `move_story(... target='backlog')`
from a Blocked story, leaving stuck items with no way to be parked while
waiting on dependent fixes — operators had to either Unblock (which
re-enters the active flow) or Archive (which loses the item).
Extend the existing Demote rule so `Blocked + Demote → Backlog` is a
legal transition, alongside the existing `Coding/Qa/Merge + Demote`.
Also update `map_stage_move_to_event` in agents/lifecycle.rs so the
chat/MCP `move_story` API recognises Blocked → backlog and routes it
through `PipelineEvent::Demote`.
Tests:
- `blocked_demote_returns_to_backlog` — happy path.
- `cannot_demote_from_done` / `cannot_demote_from_upcoming` — sanity
checks that the broadened rule does NOT permit Demote from
terminal or pre-triage stages.
Pattern follows 892 (MergeFailure → Done) and 893 (MergeFailure →
Coding) — pure transition.rs extension plus matching event mapping in
lifecycle.rs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 902: the Step 0 "resume from worktree state" instruction told coders
to call git_status / git_log / git_diff to discover prior session work,
which they then extended into hunting for the story `.md` file on disk
via find / ls — pointless post-865, since story content lives only in
the CRDT.
Update Step 0 in coder-1, coder-2, coder-3, and coder-opus to add an
explicit instruction: "To read story content, ACs, or description, call
the `get_story_todos` MCP tool — do NOT search for a story `.md` file
on disk; story content is CRDT-only."
Single substring replacement covers all four agents (identical Step 0
across them).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 901: `unblock_story` (and the chat `unblock` command) routed through
`parse_front_matter` and errored with "Missing front matter" on any
post-865 story (story content is now CRDT-only with no YAML on disk).
In `chat/commands/unblock.rs::unblock_by_story_id`:
- Drop the early `parse_front_matter` gate.
- Read story name and blocked state from the CRDT register API instead
of parsed YAML (`crdt_state::read_item`, `pipeline_state::read_typed`).
- Keep the legacy fallback cleanup, but gate it on the content actually
starting with a `---` YAML block, so CRDT-only stories don't hit a
parse error there either.
- Remove the now-unused `parse_front_matter` import.
Surfaced a second sub-bug: even when the state-machine transition
fired (`Blocked + Unblock → Coding`), the CRDT `blocked` register was
never explicitly cleared. Pre-865 the YAML-strip content_transform
cleared it as a side effect; post-865 there is no YAML to strip.
- Add `crdt_state::set_blocked(story_id, bool)` parallel to
`set_retry_count`. Wired through `crdt_state::write` and the
crate-level re-export.
- `agents::lifecycle::transition_to_unblocked` now calls
`set_blocked(story_id, false)` alongside `set_retry_count(0)` so
the legacy register stays in sync with the typed stage.
Test: `unblock_command_works_on_crdt_only_story_no_yaml` seeds a CRDT
entry with no YAML on disk, runs unblock, asserts success + cleared
blocked + retry_count=0. All 10 existing unblock tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude Code rejects "Bash(:*)" with "Prefix cannot be empty before :*" —
the rule is silently skipped, which since 5b48f0d0 left no Bash entry
in the allowlist at all. Every coder agent's Bash call has been
auto-denying since that commit landed (~840 of 1.4k denials in the sled
log).
The canonical form for "allow all bash commands" is the tool name alone:
"Bash" (no parens). Apply it in three places that 5b48f0d0 touched:
- .claude/settings.json (project root, inherited by new worktrees)
- server/src/io/fs/scaffold/templates.rs (huskies init template)
- server/src/io/fs/scaffold/tests.rs (assertion now checks "Bash")
The gateway settings.json at ~/Desktop/huskies/.claude/settings.json and
the four live worktrees (810, 888, 890, 894) were also corrected — not
in this commit since they live outside the repo.
Surfaced via /doctor; reported with rule "Invalid permission rule
Bash(:*) was skipped: Prefix cannot be empty before :*".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-command allowlist (Bash(cargo:*), Bash(git:*), …) misses any tool
a coder agent reaches for outside the curated set — ./script/*, make, curl,
jq, docker, test, [, etc. Each miss hits prompt_permission, which auto-denies
on the sled because no listener holds perm_rx (the matrix bot lives in the
gateway). 1,377 such denies in the sled log over the past week, accounting
for most of the recent throughput slowdown.
Replace the curated list with a single Bash(:*) wildcard in:
- .claude/settings.json (project root, picked up on git worktree add)
- server/src/io/fs/scaffold/templates.rs (used only by huskies init when
no .claude/settings.json already exists)
Update scaffold/tests.rs to assert the wildcard rather than a fixed set
of patterns; the per-command gate offered no real safety in this trusted
single-user deployment, since the prompt was never going to reach a human
anyway (that's the bug).
Stopgap until story 898 lands the proper sled→gateway permission
forwarding — at which point the wildcard can be narrowed back if desired.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
855 deleted the HTTP /mcp route and pointed agents at ws://...crdt-sync,
but Claude Code's .mcp.json doesn't speak ws:// and the rendezvous WS
never had MCP method handlers wired up — so every spawned Claude Code
agent (gateway-routed and local) booted with zero huskies tools and
died on --permission-prompt-tool=mcp__huskies__prompt_permission.
Restore mcp_post_handler / mcp_get_handler / handle_initialize, re-add
the /mcp route, and revert all three .mcp.json writers to emit
http://localhost:{port}/mcp with explicit "type": "http". Reuses the
already-extracted gateway::jsonrpc types and the surviving
dispatch_tool_call / list_tools surfaces — net add ~140 lines.
Federation work is unaffected: /crdt-sync continues to do CRDT sync,
which is what it was actually doing. MCP-over-WebSocket for cross-LAN
agents was never wired up by 855 and can be done as a proper follow-up
with a regression test that boots a real claude and verifies tool
registration.
Verified end-to-end: /mcp initialize, tools/list (74 tools incl.
prompt_permission), and tools/call all respond correctly from inside
the rebuilt container.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In gateway mode the bot's Claude Code CLI was spawned with cwd set to
a nonexistent project subdirectory (gateway_config_dir/project_name).
This meant it couldn't find .mcp.json and had no MCP tools available.
Now the bot uses the gateway config directory as cwd in gateway mode,
where the auto-generated .mcp.json points to the gateway's MCP proxy.
Also fixes cargo fmt formatting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The gateway proxy was sending every message's first word to the project
server's /api/bot/command endpoint, then displaying the "Unknown command"
response before falling through to the LLM. Now the proxy only fires
when the first word matches a known bot command.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
In gateway mode the bot has no local CRDT or project filesystem, so all
bot commands (status, backlog, start, assign, etc.) returned empty or
broken results. Now the gateway bot proxies non-local commands via HTTP
to the active project's /api/bot/command endpoint, which already exists
on every project server.
Only a small set of gateway-local commands (help, ambient, reset, switch)
are still handled directly by the gateway. Everything else is forwarded
automatically, so new commands added in the future will work through the
proxy without additional gateway changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When replaying old CRDT ops that predate new struct fields (e.g.
claimed_by, claim_ts added by story 479), node_from would call
.unwrap() on None and panic during init. Now defaults to an empty
CrdtNode::new() for missing fields, allowing schema evolution without
breaking replay of historical ops.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Squash merge of story 504: add MCP regression tests for non-string
front_matter values (arrays, bools, integers). The schema change itself
was already on master. Fixed the array assertion to match YAML's
space-after-comma inline sequence format.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The post-520 migration changed validate_story_dirs to read from
pipeline_state::read_all_typed() (the process-global CRDT singleton),
ignoring its root: &Path argument. This broke test isolation — tests
creating a tempdir saw dozens of results from ambient CRDT state,
causing non-deterministic failures that blocked every mergemaster gate.
Remove the CRDT singleton block and rely on the filesystem shadow scan
that already uses the root argument correctly. 1845/1845 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds the foundational capability to clear a story from the running
server's in-memory CRDT state without restarting the process. This is
story 521, motivated by the 2026-04-09 incident where stories 478 and
503 kept resurrecting from in-memory CRDT after every sqlite delete /
worktree removal / timers.json clear. The only previous remedy was a
full docker restart.
Changes:
- server/src/crdt_state.rs: new `pub fn evict_item(story_id: &str)`.
Looks up the item's CRDT OpId via the visible-index map, calls the
bft-json-crdt list `delete()` primitive to construct a tombstone op,
runs it through the existing `apply_and_persist` machinery (which
signs, applies to the in-memory CRDT, and queues for persistence to
crdt_ops), rebuilds the story_id → visible_index map, and drops the
in-memory CONTENT_STORE entry. The tombstone survives a restart
because it's persisted as a real CRDT op.
- server/src/http/mcp/story_tools.rs: new `tool_purge_story` MCP
handler that takes a story_id and calls evict_item. Deliberately
minimal — does NOT touch agents, worktrees, pipeline_items shadow
table, timers.json, or filesystem shadows. Compose with stop_agent,
remove_worktree, etc. for a full purge. Story 514 (delete_story
full cleanup) is the future "do it all" tool.
- server/src/http/mcp/mod.rs: registers the `purge_story` tool in the
tools list and dispatch table.
Usage:
mcp__huskies__purge_story story_id="<full_story_id>"
Returns a string confirming the eviction. The story will no longer
appear in get_pipeline_status, list_agents, or any other API that
reads from the in-memory CRDT view, and on the next server restart
the persisted tombstone op will keep it from being reconstructed.
This is a prerequisite for story 514 (delete_story full cleanup) and
useful for any "kill it with fire" operator need.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bundles in-progress work from a parallel Claude session toward fixing
bug 501 (rate-limit retry timer doesn't cancel on stop_agent / move_story
/ successful completion). This commit lands the foundation but the MCP
tool wiring is still TODO.
- server/src/chat/timer.rs: defense-in-depth check in tick_once that
skips firing a timer for stories already past 3_qa (3_qa, 4_merge,
5_done, 6_archived). The primary cancellation path will be in the
MCP tools; this guards races where a timer was scheduled before the
story was advanced and the tool didn't get a chance to cancel it.
- server/src/http/context.rs: adds `timer_store: Arc<TimerStore>` field
on AppContext so MCP tools (move_story, stop_agent, ...) can reach
the shared timer store and cancel pending entries when the user
intervenes manually. The test helper is updated to construct one.
- server/src/main.rs: wires up a TimerStore instance in the AppContext
initialiser so the binary actually compiles after the context.rs
field addition. TODO: the matrix bot's spawn_bot still creates its
own TimerStore instance (in chat/transport/matrix/bot/run.rs:220-227)
rather than consuming the shared one — that refactor is the next
step in the bug 501 fix.
What is NOT in this commit and is needed to actually fix bug 501:
- The MCP tool side (move_story, stop_agent, delete_story) does not
yet call timer_store.cancel(story_id) when invoked
- The matrix bot's spawn_bot does not yet consume the shared
timer_store from AppContext — it still creates its own
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The statig version was missing the per-node ExecutionState machine that
the bare version has. This commit adds it as a sub-module so its
generated `State` enum doesn't collide with the top-level PipelineMachine's
`State` enum.
Adds:
- ExecutionEvent enum (top-level, alongside PipelineEvent)
- mod execution { … } sub-module containing ExecutionMachine
- States: idle, pending, running, rate_limited, completed
- Cross-cutting `any` superstate that handles Stopped/Reset → Idle
- 6 new tests covering the happy path, rate-limit + resume, and
stop-from-anywhere via the superstate
Also adds a small note about how statig's `#[action]` entry/exit hooks
would replace the bare version's external EventBus pattern (without
implementing it — we'd pick one or the other based on whether side
effects should live inside or outside the state machine).
Test count: 11 → 17 (all passing).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds the markdown shadows for stories filed during today's stress-test
session, plus a SESSION_HANDOFF document for picking up the work in
a future session.
New stories (510-521):
510 — bug: stale 1_backlog filesystem shadows get re-promoted by timers
511 — bug: CRDT lamport clock resets to 1 on restart (FIXED in 99557635)
512 — story: migrate chat commands from filesystem lookup to CRDT/DB
513 — story: startup reconcile pass for state-machine drift detection
514 — story: delete_story should do a full cleanup
515 — story: debug MCP tool to dump in-memory CRDT state
516 — story: update_story.description should create the section if missing
517 — story: remove filesystem-shadow fallback paths from lifecycle.rs
518 — story: apply_and_persist should log persist_tx send failures
519 — story: mergemaster should fail loudly on no-op merges (mostly
obviated by Stage::Merge { commits_ahead: NonZeroU32 } in 520)
520 — story: typed pipeline state machine in Rust (sketches added in f7d69cde)
521 — story: MCP capability to write a CRDT tombstone for a story
Refactor 436 (unify story stuck states) is marked superseded by 520
via front_matter — its functionality is now part of the
Stage::Archived { reason: ArchiveReason } enum in story 520's design.
The SESSION_HANDOFF_2026-04-09.md document captures: the four-state-machine
drift situation that motivated story 520, today's bug fixes (502 + 511),
the off-leash rogue commit incident (forensic tag rogue-commit-2026-04-09-ac9f3ecf
preserved), the recommended next-session priority order, and useful
diagnostic recipes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two parallel scratch experiments under server/examples/ exploring the
typed Rust state machine that should replace huskies's current
stringly-typed CRDT representation (story 520).
- pipeline_state_sketch_bare.rs — hand-rolled, plain enums + match
- pipeline_state_sketch_statig.rs — using the statig crate
Both sketches:
- Define the same Stage enum (Backlog, Coding, Qa, Merge, Done, Archived)
- Define ArchiveReason (subsumes refactor 436's blocked/merge_failure/review_hold)
- Define ExecutionState (per-node, separate from synced Stage) — bare only
- Define PipelineEvent and the valid transitions
- Make bug 519 unrepresentable: Stage::Merge requires NonZeroU32 commits_ahead
- Make bug 502 unrepresentable: Coder agents can't be assigned to Merge state
- Have happy-path tests, retry-loop tests, and invalid-transition tests
Differences:
- Bare uses pure pattern matching, no framework. ~720 lines.
- Statig uses #[state_machine] proc macro and gets free hierarchical
states via the `active` superstate that factors out the cross-cutting
Block / ReviewHold / Abandon / Supersede transitions across the four
active stages. ~440 lines, 11 passing tests.
Run with:
cargo run --example pipeline_state_sketch_bare -p huskies
cargo run --example pipeline_state_sketch_statig -p huskies
cargo test --example pipeline_state_sketch_bare -p huskies
cargo test --example pipeline_state_sketch_statig -p huskies
Adds statig 0.3 as a dev-dependency in server/Cargo.toml. Cargo.lock
updated to include statig + statig-macro and their transitive deps.
Not wired into the main codebase. Once we agree on which version to
adopt, story 520 promotes the chosen sketch into a real
server/src/pipeline_state.rs module.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The CRDT lamport seq is per-author and per-field, not globally
monotonic. Replaying by `seq ASC` causes field-update ops (which
have low per-field seq counters like 1, 2, 3) to be applied
BEFORE the list-insert ops they reference (which have higher
per-list seq counters like N for the Nth item ever inserted).
The field updates fail with ErrPathMismatch because the target
item doesn't exist yet, the field counter is never advanced,
and subsequent writes silently lose state.
Concretely on 2026-04-09 we observed: post-restart writes were
being persisted at seq=1,2,3,4,5,6,7 even though pre-restart
seq had reached 492. On the next replay, those low-seq field
updates would be applied before their seq=485+ creation ops,
silently dropping the updates. This was the load-bearing
"why does state keep flapping" bug today.
Fix: replay by `rowid ASC` (SQLite insertion order) instead.
Rowid preserves the causal order ops were originally applied
in, so field updates always come after the item insert they
reference.
Adds a regression test that constructs the exact scenario:
inserts a story (op gets seq=6), updates its stage (op gets
seq=1 because field counter starts at 0), persists both ops
in causal order, then replays both seq ASC (reproduces the
bug — stage update is lost) and rowid ASC (the fix — stage
update is preserved).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Manual squash-merge of feature/story-478_… into master after the in-pipeline
mergemaster runs failed silently. The 478 agent did substantial real work
across multiple respawn cycles before being interrupted; commits on the
feature branch were intact and verified high-quality but never merged via
the normal pipeline path due to compounding bugs:
- The first mergemaster attempt ran ($0.82 in tokens) and exited "Done"
cleanly but didn't push anything to master — likely the worktree was
briefly on master rather than the feature branch when the merge_agent_work
MCP tool ran, so it found nothing to merge.
- Subsequent timer fires defaulted to spawning coders instead of resuming
mergemaster, burning more tokens for no progress.
- Bug 510 (split-brain shadows yanking done stories back to current) and
bug 501 (timers don't cancel on stop/completion) compounded the cost.
What this commit lands:
- server/src/crdt_sync.rs (new, ~518 lines): GET /crdt-sync WebSocket
handler that subscribes to locally-applied SignedOps and streams them as
binary frames. Per-peer bounded queue (256 ops) drops slow peers.
- server/src/crdt_state.rs: new public functions subscribe_ops(),
all_ops_json(), apply_remote_op() backing the sync handler. Adds the
CRDT_OP_TX broadcast channel (capacity 1024).
- server/src/main.rs: wires up the sync subsystem at startup.
- server/src/http/mod.rs: registers the new endpoint.
- server/src/config.rs: adds optional rendezvous field for outbound peers.
- server/src/worktree.rs: minor changes from the original branch.
- server/Cargo.toml: cfg lint suppression for CrdtNode derive.
- crates/bft-json-crdt/src/debug.rs: fix unused-variable warnings.
Resolved a trivial test-mod merge conflict in crdt_state.rs (both 478 and
503 added new tests at the end of the test module — kept both sets).
Note: this is the squash of the original 478 work that the user explicitly
authorized landing. The earlier rogue commit ac9f3ecf — which added a
DIFFERENT, broken implementation of the same feature directly to master
under the user's identity without consent — was reverted earlier in this
session. The forensic tags rogue-commit-2026-04-09-ac9f3ecf and
pre-502-reset-2026-04-09 still exist for incident audit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
start_agent unconditionally called move_story_to_current at the top of
its body, before the agent-stage check. When called for mergemaster (or
qa) on a story in 4_merge/ AND a stale 1_backlog/ shadow of the story
existed (post-491/492 split-brain artifact), the move would find the
shadow and yank it to 2_current/, find_active_story_stage would then
report 2_current/, the stage check would expect a Coder agent, and
mergemaster would be rejected — leaving the story in 2_current/ to be
re-promoted by the next auto-assign tick. Infinite loop.
Gate the move so it only fires for Coder-stage agents. QA and
Mergemaster now attach to the story at its existing stage.
Adds a regression test that reproduces the split-brain scenario by
seeding both 4_merge/ and 1_backlog/ copies of the same story and
asserting (1) the stage check does not reject mergemaster, and (2) the
4_merge/ copy is preserved (i.e. not demoted to 2_current/).
Observed live on 2026-04-09 while story 478 was looping. Filed as
bug 502.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rename all references from storkit to huskies across the codebase:
- .storkit/ directory → .huskies/
- Binary name, Cargo package name, Docker image references
- Server code, frontend code, config files, scripts
- Fix script/test to build frontend before cargo clippy/test
so merge worktrees have frontend/dist available for RustEmbed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add three HTTP endpoints for OAuth login without terminal access:
- GET /oauth/authorize — generates PKCE params, redirects to
claude.com/cai/oauth/authorize with code=true and full scopes
- GET /callback — exchanges auth code for tokens via JSON POST to
platform.claude.com/v1/oauth/token, writes ~/.claude/.credentials.json
- GET /oauth/status — returns current credential state as JSON
Uses SHA-256 (sha2 crate) for PKCE code challenge. The authorize URL
targets claude.com/cai/ (not platform.claude.com) which is required
for Max/Pro subscriptions to grant user:inference scope.
Users visit http://localhost:3001/oauth/authorize in their browser
to authenticate. Matrix/WhatsApp can send this link when auth fails.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Detect authentication_failed errors from the Claude Code PTY stream
and automatically refresh the OAuth access token using the stored
refresh token in ~/.claude/.credentials.json.
- New module server/src/llm/oauth.rs: reads credentials, calls
platform.claude.com/v1/oauth/token with JSON body, writes back
- PTY provider detects "error":"authentication_failed" via AtomicBool
- chat_stream retries once after successful refresh
- Clear error message if refresh also fails
On success the retry is transparent. On failure the user sees:
"OAuth session expired. Please run claude login to re-authenticate."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Archive completed bug 397
- Create story 399 (CLI port flag)
- Remove @biomejs/cli-darwin-arm64 and @rollup/rollup-darwin-arm64
from package.json (breaks Docker builds on Linux)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TimmyandClaude Opus 4.7