Adds the foundational capability to clear a story from the running
server's in-memory CRDT state without restarting the process. This is
story 521, motivated by the 2026-04-09 incident where stories 478 and
503 kept resurrecting from in-memory CRDT after every sqlite delete /
worktree removal / timers.json clear. The only previous remedy was a
full docker restart.
Changes:
- server/src/crdt_state.rs: new `pub fn evict_item(story_id: &str)`.
Looks up the item's CRDT OpId via the visible-index map, calls the
bft-json-crdt list `delete()` primitive to construct a tombstone op,
runs it through the existing `apply_and_persist` machinery (which
signs, applies to the in-memory CRDT, and queues for persistence to
crdt_ops), rebuilds the story_id → visible_index map, and drops the
in-memory CONTENT_STORE entry. The tombstone survives a restart
because it's persisted as a real CRDT op.
- server/src/http/mcp/story_tools.rs: new `tool_purge_story` MCP
handler that takes a story_id and calls evict_item. Deliberately
minimal — does NOT touch agents, worktrees, pipeline_items shadow
table, timers.json, or filesystem shadows. Compose with stop_agent,
remove_worktree, etc. for a full purge. Story 514 (delete_story
full cleanup) is the future "do it all" tool.
- server/src/http/mcp/mod.rs: registers the `purge_story` tool in the
tools list and dispatch table.
Usage:
mcp__huskies__purge_story story_id="<full_story_id>"
Returns a string confirming the eviction. The story will no longer
appear in get_pipeline_status, list_agents, or any other API that
reads from the in-memory CRDT view, and on the next server restart
the persisted tombstone op will keep it from being reconstructed.
This is a prerequisite for story 514 (delete_story full cleanup) and
useful for any "kill it with fire" operator need.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bundles in-progress work from a parallel Claude session toward fixing
bug 501 (rate-limit retry timer doesn't cancel on stop_agent / move_story
/ successful completion). This commit lands the foundation but the MCP
tool wiring is still TODO.
- server/src/chat/timer.rs: defense-in-depth check in tick_once that
skips firing a timer for stories already past 3_qa (3_qa, 4_merge,
5_done, 6_archived). The primary cancellation path will be in the
MCP tools; this guards races where a timer was scheduled before the
story was advanced and the tool didn't get a chance to cancel it.
- server/src/http/context.rs: adds `timer_store: Arc<TimerStore>` field
on AppContext so MCP tools (move_story, stop_agent, ...) can reach
the shared timer store and cancel pending entries when the user
intervenes manually. The test helper is updated to construct one.
- server/src/main.rs: wires up a TimerStore instance in the AppContext
initialiser so the binary actually compiles after the context.rs
field addition. TODO: the matrix bot's spawn_bot still creates its
own TimerStore instance (in chat/transport/matrix/bot/run.rs:220-227)
rather than consuming the shared one — that refactor is the next
step in the bug 501 fix.
What is NOT in this commit and is needed to actually fix bug 501:
- The MCP tool side (move_story, stop_agent, delete_story) does not
yet call timer_store.cancel(story_id) when invoked
- The matrix bot's spawn_bot does not yet consume the shared
timer_store from AppContext — it still creates its own
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The statig version was missing the per-node ExecutionState machine that
the bare version has. This commit adds it as a sub-module so its
generated `State` enum doesn't collide with the top-level PipelineMachine's
`State` enum.
Adds:
- ExecutionEvent enum (top-level, alongside PipelineEvent)
- mod execution { … } sub-module containing ExecutionMachine
- States: idle, pending, running, rate_limited, completed
- Cross-cutting `any` superstate that handles Stopped/Reset → Idle
- 6 new tests covering the happy path, rate-limit + resume, and
stop-from-anywhere via the superstate
Also adds a small note about how statig's `#[action]` entry/exit hooks
would replace the bare version's external EventBus pattern (without
implementing it — we'd pick one or the other based on whether side
effects should live inside or outside the state machine).
Test count: 11 → 17 (all passing).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds the markdown shadows for stories filed during today's stress-test
session, plus a SESSION_HANDOFF document for picking up the work in
a future session.
New stories (510-521):
510 — bug: stale 1_backlog filesystem shadows get re-promoted by timers
511 — bug: CRDT lamport clock resets to 1 on restart (FIXED in 99557635)
512 — story: migrate chat commands from filesystem lookup to CRDT/DB
513 — story: startup reconcile pass for state-machine drift detection
514 — story: delete_story should do a full cleanup
515 — story: debug MCP tool to dump in-memory CRDT state
516 — story: update_story.description should create the section if missing
517 — story: remove filesystem-shadow fallback paths from lifecycle.rs
518 — story: apply_and_persist should log persist_tx send failures
519 — story: mergemaster should fail loudly on no-op merges (mostly
obviated by Stage::Merge { commits_ahead: NonZeroU32 } in 520)
520 — story: typed pipeline state machine in Rust (sketches added in f7d69cde)
521 — story: MCP capability to write a CRDT tombstone for a story
Refactor 436 (unify story stuck states) is marked superseded by 520
via front_matter — its functionality is now part of the
Stage::Archived { reason: ArchiveReason } enum in story 520's design.
The SESSION_HANDOFF_2026-04-09.md document captures: the four-state-machine
drift situation that motivated story 520, today's bug fixes (502 + 511),
the off-leash rogue commit incident (forensic tag rogue-commit-2026-04-09-ac9f3ecf
preserved), the recommended next-session priority order, and useful
diagnostic recipes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two parallel scratch experiments under server/examples/ exploring the
typed Rust state machine that should replace huskies's current
stringly-typed CRDT representation (story 520).
- pipeline_state_sketch_bare.rs — hand-rolled, plain enums + match
- pipeline_state_sketch_statig.rs — using the statig crate
Both sketches:
- Define the same Stage enum (Backlog, Coding, Qa, Merge, Done, Archived)
- Define ArchiveReason (subsumes refactor 436's blocked/merge_failure/review_hold)
- Define ExecutionState (per-node, separate from synced Stage) — bare only
- Define PipelineEvent and the valid transitions
- Make bug 519 unrepresentable: Stage::Merge requires NonZeroU32 commits_ahead
- Make bug 502 unrepresentable: Coder agents can't be assigned to Merge state
- Have happy-path tests, retry-loop tests, and invalid-transition tests
Differences:
- Bare uses pure pattern matching, no framework. ~720 lines.
- Statig uses #[state_machine] proc macro and gets free hierarchical
states via the `active` superstate that factors out the cross-cutting
Block / ReviewHold / Abandon / Supersede transitions across the four
active stages. ~440 lines, 11 passing tests.
Run with:
cargo run --example pipeline_state_sketch_bare -p huskies
cargo run --example pipeline_state_sketch_statig -p huskies
cargo test --example pipeline_state_sketch_bare -p huskies
cargo test --example pipeline_state_sketch_statig -p huskies
Adds statig 0.3 as a dev-dependency in server/Cargo.toml. Cargo.lock
updated to include statig + statig-macro and their transitive deps.
Not wired into the main codebase. Once we agree on which version to
adopt, story 520 promotes the chosen sketch into a real
server/src/pipeline_state.rs module.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The CRDT lamport seq is per-author and per-field, not globally
monotonic. Replaying by `seq ASC` causes field-update ops (which
have low per-field seq counters like 1, 2, 3) to be applied
BEFORE the list-insert ops they reference (which have higher
per-list seq counters like N for the Nth item ever inserted).
The field updates fail with ErrPathMismatch because the target
item doesn't exist yet, the field counter is never advanced,
and subsequent writes silently lose state.
Concretely on 2026-04-09 we observed: post-restart writes were
being persisted at seq=1,2,3,4,5,6,7 even though pre-restart
seq had reached 492. On the next replay, those low-seq field
updates would be applied before their seq=485+ creation ops,
silently dropping the updates. This was the load-bearing
"why does state keep flapping" bug today.
Fix: replay by `rowid ASC` (SQLite insertion order) instead.
Rowid preserves the causal order ops were originally applied
in, so field updates always come after the item insert they
reference.
Adds a regression test that constructs the exact scenario:
inserts a story (op gets seq=6), updates its stage (op gets
seq=1 because field counter starts at 0), persists both ops
in causal order, then replays both seq ASC (reproduces the
bug — stage update is lost) and rowid ASC (the fix — stage
update is preserved).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Manual squash-merge of feature/story-478_… into master after the in-pipeline
mergemaster runs failed silently. The 478 agent did substantial real work
across multiple respawn cycles before being interrupted; commits on the
feature branch were intact and verified high-quality but never merged via
the normal pipeline path due to compounding bugs:
- The first mergemaster attempt ran ($0.82 in tokens) and exited "Done"
cleanly but didn't push anything to master — likely the worktree was
briefly on master rather than the feature branch when the merge_agent_work
MCP tool ran, so it found nothing to merge.
- Subsequent timer fires defaulted to spawning coders instead of resuming
mergemaster, burning more tokens for no progress.
- Bug 510 (split-brain shadows yanking done stories back to current) and
bug 501 (timers don't cancel on stop/completion) compounded the cost.
What this commit lands:
- server/src/crdt_sync.rs (new, ~518 lines): GET /crdt-sync WebSocket
handler that subscribes to locally-applied SignedOps and streams them as
binary frames. Per-peer bounded queue (256 ops) drops slow peers.
- server/src/crdt_state.rs: new public functions subscribe_ops(),
all_ops_json(), apply_remote_op() backing the sync handler. Adds the
CRDT_OP_TX broadcast channel (capacity 1024).
- server/src/main.rs: wires up the sync subsystem at startup.
- server/src/http/mod.rs: registers the new endpoint.
- server/src/config.rs: adds optional rendezvous field for outbound peers.
- server/src/worktree.rs: minor changes from the original branch.
- server/Cargo.toml: cfg lint suppression for CrdtNode derive.
- crates/bft-json-crdt/src/debug.rs: fix unused-variable warnings.
Resolved a trivial test-mod merge conflict in crdt_state.rs (both 478 and
503 added new tests at the end of the test module — kept both sets).
Note: this is the squash of the original 478 work that the user explicitly
authorized landing. The earlier rogue commit ac9f3ecf — which added a
DIFFERENT, broken implementation of the same feature directly to master
under the user's identity without consent — was reverted earlier in this
session. The forensic tags rogue-commit-2026-04-09-ac9f3ecf and
pre-502-reset-2026-04-09 still exist for incident audit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
start_agent unconditionally called move_story_to_current at the top of
its body, before the agent-stage check. When called for mergemaster (or
qa) on a story in 4_merge/ AND a stale 1_backlog/ shadow of the story
existed (post-491/492 split-brain artifact), the move would find the
shadow and yank it to 2_current/, find_active_story_stage would then
report 2_current/, the stage check would expect a Coder agent, and
mergemaster would be rejected — leaving the story in 2_current/ to be
re-promoted by the next auto-assign tick. Infinite loop.
Gate the move so it only fires for Coder-stage agents. QA and
Mergemaster now attach to the story at its existing stage.
Adds a regression test that reproduces the split-brain scenario by
seeding both 4_merge/ and 1_backlog/ copies of the same story and
asserting (1) the stage check does not reject mergemaster, and (2) the
4_merge/ copy is preserved (i.e. not demoted to 2_current/).
Observed live on 2026-04-09 while story 478 was looping. Filed as
bug 502.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rename all references from storkit to huskies across the codebase:
- .storkit/ directory → .huskies/
- Binary name, Cargo package name, Docker image references
- Server code, frontend code, config files, scripts
- Fix script/test to build frontend before cargo clippy/test
so merge worktrees have frontend/dist available for RustEmbed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add three HTTP endpoints for OAuth login without terminal access:
- GET /oauth/authorize — generates PKCE params, redirects to
claude.com/cai/oauth/authorize with code=true and full scopes
- GET /callback — exchanges auth code for tokens via JSON POST to
platform.claude.com/v1/oauth/token, writes ~/.claude/.credentials.json
- GET /oauth/status — returns current credential state as JSON
Uses SHA-256 (sha2 crate) for PKCE code challenge. The authorize URL
targets claude.com/cai/ (not platform.claude.com) which is required
for Max/Pro subscriptions to grant user:inference scope.
Users visit http://localhost:3001/oauth/authorize in their browser
to authenticate. Matrix/WhatsApp can send this link when auth fails.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Detect authentication_failed errors from the Claude Code PTY stream
and automatically refresh the OAuth access token using the stored
refresh token in ~/.claude/.credentials.json.
- New module server/src/llm/oauth.rs: reads credentials, calls
platform.claude.com/v1/oauth/token with JSON body, writes back
- PTY provider detects "error":"authentication_failed" via AtomicBool
- chat_stream retries once after successful refresh
- Clear error message if refresh also fails
On success the retry is transparent. On failure the user sees:
"OAuth session expired. Please run claude login to re-authenticate."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Archive completed bug 397
- Create story 399 (CLI port flag)
- Remove @biomejs/cli-darwin-arm64 and @rollup/rollup-darwin-arm64
from package.json (breaks Docker builds on Linux)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When unset, Claude Code falls back to OAuth credentials from
`claude login`, allowing agents to run on a Max subscription
instead of prepaid API credits.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Docker named volumes inherit directory ownership when first created.
By creating /workspace/target and /app/target as storkit-owned before
the USER directive, the volumes will be writable by the storkit user.
Without this, cargo build/test/clippy all fail with Permission Denied.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Acceptance gates run cargo clippy but the component wasn't installed
in the build stage. Agents were doing real work then failing every
gate check because clippy wasn't available.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The tmpfs at /home/storkit defaulted to root ownership (mode=755),
so Claude Code couldn't write ~/.claude.json or ~/.cache/. Set
uid=999,gid=999 to match the storkit user.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Claude Code writes to ~/.claude.json, ~/.cache/, and ~/.npm/ which
failed silently on the read-only root filesystem. Add tmpfs at
/home/storkit so the home dir is writable (the claude-state volume
overlays on top for persistent .claude/ data).
Also fix .dockerignore: use **/target/ to match nested target dirs,
add .storkit/logs/ and **/node_modules/ to prevent multi-GB build
context transfers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use the known cargo build output path instead of current_exe() when
re-execing after a rebuild. In Docker, the running binary lives at
/usr/local/bin/storkit (read-only) while cargo writes the new binary
to /app/target/release/storkit (a writable volume), so current_exe()
would just restart the old binary.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Timmy
