Pipeline gap: the state machine refused `move_story(... target='backlog')`
from a Blocked story, leaving stuck items with no way to be parked while
waiting on dependent fixes — operators had to either Unblock (which
re-enters the active flow) or Archive (which loses the item).
Extend the existing Demote rule so `Blocked + Demote → Backlog` is a
legal transition, alongside the existing `Coding/Qa/Merge + Demote`.
Also update `map_stage_move_to_event` in agents/lifecycle.rs so the
chat/MCP `move_story` API recognises Blocked → backlog and routes it
through `PipelineEvent::Demote`.
Tests:
- `blocked_demote_returns_to_backlog` — happy path.
- `cannot_demote_from_done` / `cannot_demote_from_upcoming` — sanity
checks that the broadened rule does NOT permit Demote from
terminal or pre-triage stages.
Pattern follows 892 (MergeFailure → Done) and 893 (MergeFailure →
Coding) — pure transition.rs extension plus matching event mapping in
lifecycle.rs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 902: the Step 0 "resume from worktree state" instruction told coders
to call git_status / git_log / git_diff to discover prior session work,
which they then extended into hunting for the story `.md` file on disk
via find / ls — pointless post-865, since story content lives only in
the CRDT.
Update Step 0 in coder-1, coder-2, coder-3, and coder-opus to add an
explicit instruction: "To read story content, ACs, or description, call
the `get_story_todos` MCP tool — do NOT search for a story `.md` file
on disk; story content is CRDT-only."
Single substring replacement covers all four agents (identical Step 0
across them).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug 901: `unblock_story` (and the chat `unblock` command) routed through
`parse_front_matter` and errored with "Missing front matter" on any
post-865 story (story content is now CRDT-only with no YAML on disk).
In `chat/commands/unblock.rs::unblock_by_story_id`:
- Drop the early `parse_front_matter` gate.
- Read story name and blocked state from the CRDT register API instead
of parsed YAML (`crdt_state::read_item`, `pipeline_state::read_typed`).
- Keep the legacy fallback cleanup, but gate it on the content actually
starting with a `---` YAML block, so CRDT-only stories don't hit a
parse error there either.
- Remove the now-unused `parse_front_matter` import.
Surfaced a second sub-bug: even when the state-machine transition
fired (`Blocked + Unblock → Coding`), the CRDT `blocked` register was
never explicitly cleared. Pre-865 the YAML-strip content_transform
cleared it as a side effect; post-865 there is no YAML to strip.
- Add `crdt_state::set_blocked(story_id, bool)` parallel to
`set_retry_count`. Wired through `crdt_state::write` and the
crate-level re-export.
- `agents::lifecycle::transition_to_unblocked` now calls
`set_blocked(story_id, false)` alongside `set_retry_count(0)` so
the legacy register stays in sync with the typed stage.
Test: `unblock_command_works_on_crdt_only_story_no_yaml` seeds a CRDT
entry with no YAML on disk, runs unblock, asserts success + cleared
blocked + retry_count=0. All 10 existing unblock tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Claude Code rejects "Bash(:*)" with "Prefix cannot be empty before :*" —
the rule is silently skipped, which since 5b48f0d0 left no Bash entry
in the allowlist at all. Every coder agent's Bash call has been
auto-denying since that commit landed (~840 of 1.4k denials in the sled
log).
The canonical form for "allow all bash commands" is the tool name alone:
"Bash" (no parens). Apply it in three places that 5b48f0d0 touched:
- .claude/settings.json (project root, inherited by new worktrees)
- server/src/io/fs/scaffold/templates.rs (huskies init template)
- server/src/io/fs/scaffold/tests.rs (assertion now checks "Bash")
The gateway settings.json at ~/Desktop/huskies/.claude/settings.json and
the four live worktrees (810, 888, 890, 894) were also corrected — not
in this commit since they live outside the repo.
Surfaced via /doctor; reported with rule "Invalid permission rule
Bash(:*) was skipped: Prefix cannot be empty before :*".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-command allowlist (Bash(cargo:*), Bash(git:*), …) misses any tool
a coder agent reaches for outside the curated set — ./script/*, make, curl,
jq, docker, test, [, etc. Each miss hits prompt_permission, which auto-denies
on the sled because no listener holds perm_rx (the matrix bot lives in the
gateway). 1,377 such denies in the sled log over the past week, accounting
for most of the recent throughput slowdown.
Replace the curated list with a single Bash(:*) wildcard in:
- .claude/settings.json (project root, picked up on git worktree add)
- server/src/io/fs/scaffold/templates.rs (used only by huskies init when
no .claude/settings.json already exists)
Update scaffold/tests.rs to assert the wildcard rather than a fixed set
of patterns; the per-command gate offered no real safety in this trusted
single-user deployment, since the prompt was never going to reach a human
anyway (that's the bug).
Stopgap until story 898 lands the proper sled→gateway permission
forwarding — at which point the wildcard can be narrowed back if desired.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
855 deleted the HTTP /mcp route and pointed agents at ws://...crdt-sync,
but Claude Code's .mcp.json doesn't speak ws:// and the rendezvous WS
never had MCP method handlers wired up — so every spawned Claude Code
agent (gateway-routed and local) booted with zero huskies tools and
died on --permission-prompt-tool=mcp__huskies__prompt_permission.
Restore mcp_post_handler / mcp_get_handler / handle_initialize, re-add
the /mcp route, and revert all three .mcp.json writers to emit
http://localhost:{port}/mcp with explicit "type": "http". Reuses the
already-extracted gateway::jsonrpc types and the surviving
dispatch_tool_call / list_tools surfaces — net add ~140 lines.
Federation work is unaffected: /crdt-sync continues to do CRDT sync,
which is what it was actually doing. MCP-over-WebSocket for cross-LAN
agents was never wired up by 855 and can be done as a proper follow-up
with a regression test that boots a real claude and verifies tool
registration.
Verified end-to-end: /mcp initialize, tools/list (74 tools incl.
prompt_permission), and tools/call all respond correctly from inside
the rebuilt container.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In gateway mode the bot's Claude Code CLI was spawned with cwd set to
a nonexistent project subdirectory (gateway_config_dir/project_name).
This meant it couldn't find .mcp.json and had no MCP tools available.
Now the bot uses the gateway config directory as cwd in gateway mode,
where the auto-generated .mcp.json points to the gateway's MCP proxy.
Also fixes cargo fmt formatting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The gateway proxy was sending every message's first word to the project
server's /api/bot/command endpoint, then displaying the "Unknown command"
response before falling through to the LLM. Now the proxy only fires
when the first word matches a known bot command.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
In gateway mode the bot has no local CRDT or project filesystem, so all
bot commands (status, backlog, start, assign, etc.) returned empty or
broken results. Now the gateway bot proxies non-local commands via HTTP
to the active project's /api/bot/command endpoint, which already exists
on every project server.
Only a small set of gateway-local commands (help, ambient, reset, switch)
are still handled directly by the gateway. Everything else is forwarded
automatically, so new commands added in the future will work through the
proxy without additional gateway changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When replaying old CRDT ops that predate new struct fields (e.g.
claimed_by, claim_ts added by story 479), node_from would call
.unwrap() on None and panic during init. Now defaults to an empty
CrdtNode::new() for missing fields, allowing schema evolution without
breaking replay of historical ops.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Squash merge of story 504: add MCP regression tests for non-string
front_matter values (arrays, bools, integers). The schema change itself
was already on master. Fixed the array assertion to match YAML's
space-after-comma inline sequence format.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The post-520 migration changed validate_story_dirs to read from
pipeline_state::read_all_typed() (the process-global CRDT singleton),
ignoring its root: &Path argument. This broke test isolation — tests
creating a tempdir saw dozens of results from ambient CRDT state,
causing non-deterministic failures that blocked every mergemaster gate.
Remove the CRDT singleton block and rely on the filesystem shadow scan
that already uses the root argument correctly. 1845/1845 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds the foundational capability to clear a story from the running
server's in-memory CRDT state without restarting the process. This is
story 521, motivated by the 2026-04-09 incident where stories 478 and
503 kept resurrecting from in-memory CRDT after every sqlite delete /
worktree removal / timers.json clear. The only previous remedy was a
full docker restart.
Changes:
- server/src/crdt_state.rs: new `pub fn evict_item(story_id: &str)`.
Looks up the item's CRDT OpId via the visible-index map, calls the
bft-json-crdt list `delete()` primitive to construct a tombstone op,
runs it through the existing `apply_and_persist` machinery (which
signs, applies to the in-memory CRDT, and queues for persistence to
crdt_ops), rebuilds the story_id → visible_index map, and drops the
in-memory CONTENT_STORE entry. The tombstone survives a restart
because it's persisted as a real CRDT op.
- server/src/http/mcp/story_tools.rs: new `tool_purge_story` MCP
handler that takes a story_id and calls evict_item. Deliberately
minimal — does NOT touch agents, worktrees, pipeline_items shadow
table, timers.json, or filesystem shadows. Compose with stop_agent,
remove_worktree, etc. for a full purge. Story 514 (delete_story
full cleanup) is the future "do it all" tool.
- server/src/http/mcp/mod.rs: registers the `purge_story` tool in the
tools list and dispatch table.
Usage:
mcp__huskies__purge_story story_id="<full_story_id>"
Returns a string confirming the eviction. The story will no longer
appear in get_pipeline_status, list_agents, or any other API that
reads from the in-memory CRDT view, and on the next server restart
the persisted tombstone op will keep it from being reconstructed.
This is a prerequisite for story 514 (delete_story full cleanup) and
useful for any "kill it with fire" operator need.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bundles in-progress work from a parallel Claude session toward fixing
bug 501 (rate-limit retry timer doesn't cancel on stop_agent / move_story
/ successful completion). This commit lands the foundation but the MCP
tool wiring is still TODO.
- server/src/chat/timer.rs: defense-in-depth check in tick_once that
skips firing a timer for stories already past 3_qa (3_qa, 4_merge,
5_done, 6_archived). The primary cancellation path will be in the
MCP tools; this guards races where a timer was scheduled before the
story was advanced and the tool didn't get a chance to cancel it.
- server/src/http/context.rs: adds `timer_store: Arc<TimerStore>` field
on AppContext so MCP tools (move_story, stop_agent, ...) can reach
the shared timer store and cancel pending entries when the user
intervenes manually. The test helper is updated to construct one.
- server/src/main.rs: wires up a TimerStore instance in the AppContext
initialiser so the binary actually compiles after the context.rs
field addition. TODO: the matrix bot's spawn_bot still creates its
own TimerStore instance (in chat/transport/matrix/bot/run.rs:220-227)
rather than consuming the shared one — that refactor is the next
step in the bug 501 fix.
What is NOT in this commit and is needed to actually fix bug 501:
- The MCP tool side (move_story, stop_agent, delete_story) does not
yet call timer_store.cancel(story_id) when invoked
- The matrix bot's spawn_bot does not yet consume the shared
timer_store from AppContext — it still creates its own
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The statig version was missing the per-node ExecutionState machine that
the bare version has. This commit adds it as a sub-module so its
generated `State` enum doesn't collide with the top-level PipelineMachine's
`State` enum.
Adds:
- ExecutionEvent enum (top-level, alongside PipelineEvent)
- mod execution { … } sub-module containing ExecutionMachine
- States: idle, pending, running, rate_limited, completed
- Cross-cutting `any` superstate that handles Stopped/Reset → Idle
- 6 new tests covering the happy path, rate-limit + resume, and
stop-from-anywhere via the superstate
Also adds a small note about how statig's `#[action]` entry/exit hooks
would replace the bare version's external EventBus pattern (without
implementing it — we'd pick one or the other based on whether side
effects should live inside or outside the state machine).
Test count: 11 → 17 (all passing).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds the markdown shadows for stories filed during today's stress-test
session, plus a SESSION_HANDOFF document for picking up the work in
a future session.
New stories (510-521):
510 — bug: stale 1_backlog filesystem shadows get re-promoted by timers
511 — bug: CRDT lamport clock resets to 1 on restart (FIXED in 99557635)
512 — story: migrate chat commands from filesystem lookup to CRDT/DB
513 — story: startup reconcile pass for state-machine drift detection
514 — story: delete_story should do a full cleanup
515 — story: debug MCP tool to dump in-memory CRDT state
516 — story: update_story.description should create the section if missing
517 — story: remove filesystem-shadow fallback paths from lifecycle.rs
518 — story: apply_and_persist should log persist_tx send failures
519 — story: mergemaster should fail loudly on no-op merges (mostly
obviated by Stage::Merge { commits_ahead: NonZeroU32 } in 520)
520 — story: typed pipeline state machine in Rust (sketches added in f7d69cde)
521 — story: MCP capability to write a CRDT tombstone for a story
Refactor 436 (unify story stuck states) is marked superseded by 520
via front_matter — its functionality is now part of the
Stage::Archived { reason: ArchiveReason } enum in story 520's design.
The SESSION_HANDOFF_2026-04-09.md document captures: the four-state-machine
drift situation that motivated story 520, today's bug fixes (502 + 511),
the off-leash rogue commit incident (forensic tag rogue-commit-2026-04-09-ac9f3ecf
preserved), the recommended next-session priority order, and useful
diagnostic recipes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two parallel scratch experiments under server/examples/ exploring the
typed Rust state machine that should replace huskies's current
stringly-typed CRDT representation (story 520).
- pipeline_state_sketch_bare.rs — hand-rolled, plain enums + match
- pipeline_state_sketch_statig.rs — using the statig crate
Both sketches:
- Define the same Stage enum (Backlog, Coding, Qa, Merge, Done, Archived)
- Define ArchiveReason (subsumes refactor 436's blocked/merge_failure/review_hold)
- Define ExecutionState (per-node, separate from synced Stage) — bare only
- Define PipelineEvent and the valid transitions
- Make bug 519 unrepresentable: Stage::Merge requires NonZeroU32 commits_ahead
- Make bug 502 unrepresentable: Coder agents can't be assigned to Merge state
- Have happy-path tests, retry-loop tests, and invalid-transition tests
Differences:
- Bare uses pure pattern matching, no framework. ~720 lines.
- Statig uses #[state_machine] proc macro and gets free hierarchical
states via the `active` superstate that factors out the cross-cutting
Block / ReviewHold / Abandon / Supersede transitions across the four
active stages. ~440 lines, 11 passing tests.
Run with:
cargo run --example pipeline_state_sketch_bare -p huskies
cargo run --example pipeline_state_sketch_statig -p huskies
cargo test --example pipeline_state_sketch_bare -p huskies
cargo test --example pipeline_state_sketch_statig -p huskies
Adds statig 0.3 as a dev-dependency in server/Cargo.toml. Cargo.lock
updated to include statig + statig-macro and their transitive deps.
Not wired into the main codebase. Once we agree on which version to
adopt, story 520 promotes the chosen sketch into a real
server/src/pipeline_state.rs module.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The CRDT lamport seq is per-author and per-field, not globally
monotonic. Replaying by `seq ASC` causes field-update ops (which
have low per-field seq counters like 1, 2, 3) to be applied
BEFORE the list-insert ops they reference (which have higher
per-list seq counters like N for the Nth item ever inserted).
The field updates fail with ErrPathMismatch because the target
item doesn't exist yet, the field counter is never advanced,
and subsequent writes silently lose state.
Concretely on 2026-04-09 we observed: post-restart writes were
being persisted at seq=1,2,3,4,5,6,7 even though pre-restart
seq had reached 492. On the next replay, those low-seq field
updates would be applied before their seq=485+ creation ops,
silently dropping the updates. This was the load-bearing
"why does state keep flapping" bug today.
Fix: replay by `rowid ASC` (SQLite insertion order) instead.
Rowid preserves the causal order ops were originally applied
in, so field updates always come after the item insert they
reference.
Adds a regression test that constructs the exact scenario:
inserts a story (op gets seq=6), updates its stage (op gets
seq=1 because field counter starts at 0), persists both ops
in causal order, then replays both seq ASC (reproduces the
bug — stage update is lost) and rowid ASC (the fix — stage
update is preserved).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Manual squash-merge of feature/story-478_… into master after the in-pipeline
mergemaster runs failed silently. The 478 agent did substantial real work
across multiple respawn cycles before being interrupted; commits on the
feature branch were intact and verified high-quality but never merged via
the normal pipeline path due to compounding bugs:
- The first mergemaster attempt ran ($0.82 in tokens) and exited "Done"
cleanly but didn't push anything to master — likely the worktree was
briefly on master rather than the feature branch when the merge_agent_work
MCP tool ran, so it found nothing to merge.
- Subsequent timer fires defaulted to spawning coders instead of resuming
mergemaster, burning more tokens for no progress.
- Bug 510 (split-brain shadows yanking done stories back to current) and
bug 501 (timers don't cancel on stop/completion) compounded the cost.
What this commit lands:
- server/src/crdt_sync.rs (new, ~518 lines): GET /crdt-sync WebSocket
handler that subscribes to locally-applied SignedOps and streams them as
binary frames. Per-peer bounded queue (256 ops) drops slow peers.
- server/src/crdt_state.rs: new public functions subscribe_ops(),
all_ops_json(), apply_remote_op() backing the sync handler. Adds the
CRDT_OP_TX broadcast channel (capacity 1024).
- server/src/main.rs: wires up the sync subsystem at startup.
- server/src/http/mod.rs: registers the new endpoint.
- server/src/config.rs: adds optional rendezvous field for outbound peers.
- server/src/worktree.rs: minor changes from the original branch.
- server/Cargo.toml: cfg lint suppression for CrdtNode derive.
- crates/bft-json-crdt/src/debug.rs: fix unused-variable warnings.
Resolved a trivial test-mod merge conflict in crdt_state.rs (both 478 and
503 added new tests at the end of the test module — kept both sets).
Note: this is the squash of the original 478 work that the user explicitly
authorized landing. The earlier rogue commit ac9f3ecf — which added a
DIFFERENT, broken implementation of the same feature directly to master
under the user's identity without consent — was reverted earlier in this
session. The forensic tags rogue-commit-2026-04-09-ac9f3ecf and
pre-502-reset-2026-04-09 still exist for incident audit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
start_agent unconditionally called move_story_to_current at the top of
its body, before the agent-stage check. When called for mergemaster (or
qa) on a story in 4_merge/ AND a stale 1_backlog/ shadow of the story
existed (post-491/492 split-brain artifact), the move would find the
shadow and yank it to 2_current/, find_active_story_stage would then
report 2_current/, the stage check would expect a Coder agent, and
mergemaster would be rejected — leaving the story in 2_current/ to be
re-promoted by the next auto-assign tick. Infinite loop.
Gate the move so it only fires for Coder-stage agents. QA and
Mergemaster now attach to the story at its existing stage.
Adds a regression test that reproduces the split-brain scenario by
seeding both 4_merge/ and 1_backlog/ copies of the same story and
asserting (1) the stage check does not reject mergemaster, and (2) the
4_merge/ copy is preserved (i.e. not demoted to 2_current/).
Observed live on 2026-04-09 while story 478 was looping. Filed as
bug 502.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rename all references from storkit to huskies across the codebase:
- .storkit/ directory → .huskies/
- Binary name, Cargo package name, Docker image references
- Server code, frontend code, config files, scripts
- Fix script/test to build frontend before cargo clippy/test
so merge worktrees have frontend/dist available for RustEmbed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add three HTTP endpoints for OAuth login without terminal access:
- GET /oauth/authorize — generates PKCE params, redirects to
claude.com/cai/oauth/authorize with code=true and full scopes
- GET /callback — exchanges auth code for tokens via JSON POST to
platform.claude.com/v1/oauth/token, writes ~/.claude/.credentials.json
- GET /oauth/status — returns current credential state as JSON
Uses SHA-256 (sha2 crate) for PKCE code challenge. The authorize URL
targets claude.com/cai/ (not platform.claude.com) which is required
for Max/Pro subscriptions to grant user:inference scope.
Users visit http://localhost:3001/oauth/authorize in their browser
to authenticate. Matrix/WhatsApp can send this link when auth fails.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Detect authentication_failed errors from the Claude Code PTY stream
and automatically refresh the OAuth access token using the stored
refresh token in ~/.claude/.credentials.json.
- New module server/src/llm/oauth.rs: reads credentials, calls
platform.claude.com/v1/oauth/token with JSON body, writes back
- PTY provider detects "error":"authentication_failed" via AtomicBool
- chat_stream retries once after successful refresh
- Clear error message if refresh also fails
On success the retry is transparent. On failure the user sees:
"OAuth session expired. Please run claude login to re-authenticate."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Archive completed bug 397
- Create story 399 (CLI port flag)
- Remove @biomejs/cli-darwin-arm64 and @rollup/rollup-darwin-arm64
from package.json (breaks Docker builds on Linux)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TimmyandClaude Opus 4.7