huskies/.huskies/specs/tech/STATE_MACHINE.md

Pipeline State Machine

This document describes the huskies pipeline state machine in two halves: (a) the model that runs in production today, and (b) transitions, refinements, and corrections we have identified as needed but not yet implemented.

The codebase is in a deliberate transitional state: a typed CRDT state machine exists at server/src/pipeline_state.rs (introduced by story 520) with strict Rust enums for every stage, archive reason, execution state, and event. It is fully defined and tested but not yet called from non-test code (#![allow(dead_code)] at the top of the module). Consumers will migrate incrementally.

The model that is actually doing work is the older filesystem-stage-string + front-matter-flag model. Section (a) below documents both representations and the migration intent.

---

(a) The current state machine

Stages (production: filesystem string; future: typed enum)

Filesystem (production)	Typed (future)	Meaning
work/1_backlog/	Stage::Backlog	Story exists, waiting for dependencies or auto-assign promotion
work/2_current/	Stage::Coding	Coder agent is running (or about to)
work/3_qa/	Stage::Qa	Coder finished; gates / human review running
work/4_merge/	Stage::Merge { feature_branch, commits_ahead: NonZeroU32 }	Gates passed, mergemaster ready to squash
work/5_done/	Stage::Done { merged_at, merge_commit }	Mergemaster squashed to master
work/6_archived/	Stage::Archived { archived_at, reason: ArchiveReason }	Out of the active flow

5_done auto-sweeps to 6_archived after four hours. The typed Stage::Done variant always carries the merge SHA and timestamp; Stage::Merge's commits_ahead: NonZeroU32 makes "Merge with nothing to merge" structurally impossible (eliminates bug 519).

Archive reasons (pipeline_state.rs::ArchiveReason)

The typed model already enumerates the reasons a story can leave the active flow (subsumes the legacy blocked, merge_failure, and review_hold front-matter fields per story 436):

• Completed — happy-path
• Abandoned — user explicitly abandoned
• Superseded { by: StoryId } — replaced by another story
• Blocked { reason: String } — manually blocked, awaiting human resolution
• MergeFailed { reason: String } — mergemaster gave up after retry budget
• ReviewHeld { reason: String } — held for human review at user request

Per-node execution state (pipeline_state.rs::ExecutionState)

Stage is shared/CRDT-replicated. Execution state is per-node and lives under each node's pubkey in the CRDT, so there are no inter-author merge conflicts:

• Idle
• Pending { agent, since } — worktree being created, agent about to start
• Running { agent, started_at, last_heartbeat }
• RateLimited { agent, resume_at }
• Completed { agent, exit_code, completed_at }

Pipeline events (pipeline_state.rs::PipelineEvent)

The typed model defines every event that drives a Stage transition. Each variant carries the data needed to construct the destination state, so a transition function can never accidentally land in an underspecified state:

• DepsMet — dependencies met; promote from backlog
• GatesStarted — coder starting gates
• GatesPassed { feature_branch, commits_ahead }
• GatesFailed { reason }
• QaSkipped { feature_branch, commits_ahead } — qa-mode = "server"; skip QA, go to merge
• MergeSucceeded { merge_commit }
• MergeFailedFinal { reason }
• Accepted — Done → Archived(Completed)

Transitions (current production = MCP verb shape)

Backlog → Coding (a.k.a. backlog → 2_current)

• Auto path: AgentPool::auto_assign_available_work calls promote_ready_backlog_stories. A backlog story is promoted iff (a) it has an explicit non-empty depends_on AND (b) every dep is in 5_done or 6_archived. Stories with no depends_on are NOT auto-promoted — they wait for human scheduling.
  • Implemented in server/src/agents/pool/auto_assign/auto_assign.rs::promote_ready_backlog_stories.
• Manual path: mcp__huskies__move_story story_id=X target_stage=current, or mcp__huskies__start_agent (which moves the story to current as a side-effect of starting an agent).
• Archived-dep warning: if a dep was satisfied via 6_archived rather than 5_done (e.g. abandoned/superseded), the auto-assigner logs a prominent warning so the user can see the promotion was triggered by an archived dep.

Coding → Qa (current → 3_qa)

• Triggered when the coder agent finishes (gates start running).
• mcp__huskies__request_qa is the manual verb.

Qa → Coding (qa → current — rejection path)

• mcp__huskies__reject_qa story_id=X notes="..." moves qa → current, clears review_hold, and writes the rejection notes (agents/lifecycle.rs:210).
• Used when a qa agent fails or a human reviewer rejects the work.

Qa → Merge (qa → 4_merge)

• Triggered when QA gates pass. mcp__huskies__move_story_to_merge is the dedicated verb.
• For server-mode QA: typed-side PipelineEvent::QaSkipped allows going from Coding → Merge directly without entering Qa.

Merge → Done (merge → 5_done)

• Mergemaster picks up a story in 4_merge/, squashes the feature branch onto master, then transitions to 5_done.
• mcp__huskies__move_story_to_merge queues; mergemaster does the actual work.

Done → Archived(Completed) (5_done → 6_archived)

• Auto-sweep after four hours, OR
• mcp__huskies__accept_story (immediate manual archive).

Any-stage → Archived(other reasons)

• Abandoned / Superseded: today done by mcp__huskies__move_story target_stage=done (no first-class verbs for these reasons; see (b) below).
• Blocked: blocked: true flag in front matter is set on retry-limit exceedance. mcp__huskies__unblock_story clears the flag and resets retry_count.
• MergeFailed: written to front matter when mergemaster fails; auto-assign skips these stories (has_merge_failure check).
• ReviewHeld: review_hold: true flag is set automatically on spike completion; auto-assign skips these stories until the flag is cleared.

Tombstone / purge

• mcp__huskies__delete_story and mcp__huskies__purge_story permanently remove. Purge writes a CRDT tombstone.

Auto-assign skip conditions (current production)

auto_assign_available_work walks 2_current/, 3_qa/, 4_merge/ in order and attempts to dispatch a free agent to each unassigned story. It skips any story that:

1. Has review_hold: true in front matter (spikes after QA, manual hold).
2. Is frozen (is_story_frozen — pipeline advancement suspended for this story).
3. Has blocked: true (retry limit exceeded; cleared via unblock_story).
4. Has unmet depends_on dependencies.
5. (Merge stage only) Has a recorded merge failure (has_merge_failure).
6. (Merge stage only) Has an empty diff on the feature branch — auto-writes merge_failure and blocks immediately rather than wasting a mergemaster turn.

Front-matter fields that gate transitions

Field	Type	Effect
depends_on	list of story IDs	Blocks backlog → current promotion until all deps are in 5_done or 6_archived
agent	string (e.g. coder-opus)	Pins the preferred agent for next assignment
review_hold	bool	Auto-assign skips this story; cleared by reject_qa or manual unblock
blocked	bool	Auto-assign skips this story; cleared by unblock_story
frozen	bool	Auto-assign skips this story; manual unfreeze required
merge_failure	string	Auto-assign skips merge-stage agents on this story
retry_count	int	Local-only (not in CRDT); incremented by orchestrator

Spike-specific behavior

Per the typical lifecycle, a spike runs through current → qa like any work item, then stops in qa awaiting human review (spikes skip merge). This is implemented via review_hold: true being written automatically when a spike's qa gates pass. The user accepts (move qa → done) or rejects (move qa → current). Spikes do NOT auto-promote to merge.

Mergemaster lifecycle

The mergemaster agent only runs against stories in 4_merge/. It:

1. Verifies the feature branch has commits (or the story is auto-blocked).
2. Squashes the feature branch onto master with a deterministic commit message.
3. Transitions the story to 5_done with merged_at and merge_commit.
4. On failure beyond the retry budget, writes merge_failure and blocks the story (auto-assign then skips it).

Agent terminated with committed work (bug 645 recovery path)

When a coder agent terminates abnormally (e.g. the Claude Code CLI's output.write(&bytes).is_ok() PTY write assertion fires mid-session), the server-owned completion path detects the crash and checks for surviving work:

1. If the worktree is dirty but has commits ahead of master, reset the uncommitted files (git checkout . && git clean -fd) and run gates against the committed code.
2. If gates still fail but git log master..HEAD shows commits and cargo check passes, advance to QA instead of entering the retry/block path. This is the "work survived" check, implemented in server/src/agents/pool/pipeline/advance.rs.
3. Agents that die WITHOUT committed work (no commits ahead of master) still follow the existing retry → block path unchanged.

This prevents false-positive blocking of stories where the agent completed meaningful work before crashing.

Watchdog (current production)

The "watchdog" at server/src/agents/pool/auto_assign/watchdog.rs runs every 30 ticks of the unified background loop. Today it does one thing: detect orphaned agents whose tokio task is is_finished() but whose status is still Running or Pending, and mark them Failed with an AgentEvent::Error emission. Bug 624 (now merged) extends it to also enforce max_turns and max_budget_usd limits — an agent over either limit is killed via the existing kill_child_for_key path and recorded with a typed termination reason.

---

(b) Transitions and behaviors that don't yet exist (or are only partially wired)

Migration of consumers off legacy strings to typed Stage enum

The biggest outstanding piece. pipeline_state.rs is #![allow(dead_code)]. Every consumer (auto-assign, mergemaster, MCP tools, chat commands) still works with stage strings ("2_current", "4_merge") and front-matter flags. The projection layer (TryFrom<PipelineItemView> for PipelineItem and friends) exists but isn't called outside tests. Migration is intentionally incremental.

Opportunity: pick a leaf consumer (e.g. one MCP tool that reads the stage string) and migrate it to read Stage instead. Pattern repeats outward until all consumers go through the typed projection and the legacy stage-string code can be deleted.

First-class verbs for archive reasons

ArchiveReason already has six variants but only Completed (via accept_story) and Blocked (via the blocked: true flag) have dedicated MCP verbs. Today, Abandoned, Superseded, MergeFailed, and ReviewHeld are reached either via move_story target_stage=done (which doesn't carry the reason) or via setting front-matter flags on the live story.

Missing transitions:

• mcp__huskies__supersede_story story_id=X by=Y — sets stage to Archived { reason: Superseded { by: Y } }. Today we use move_story → done, losing the by reference. (Came up 2026-04-25 with spike 621 → refactor 623.)
• mcp__huskies__abandon_story story_id=X reason="..." — sets Archived { reason: Abandoned }. Today done via move_story → done or purge_story.
• mcp__huskies__hold_for_review story_id=X reason="..." — explicitly puts a story in Archived { reason: ReviewHeld } rather than relying on the auto-set review_hold flag.

Type-conversion transitions

Spike → story conversion is a real workflow (we do it when a spike's scope grows into an implementation story). Today, converting type via update_story front_matter={"type": "story"} does not bootstrap the ## Acceptance Criteria section, and add_criterion then permanently fails on that story (see bug 625 filed 2026-04-25). The type field passed via front_matter is also silently dropped — same silent-drop bug class as acceptance_criteria. The state machine should treat type conversion as a transition with side effects — at minimum, ensuring the AC section exists when transitioning to a type that requires it, and the displayed type reflects the new value (today the display chip is parsed from the immutable story_id prefix; story 578 in backlog will fix this by switching to numeric-only IDs).

Limit-based agent termination (turn / budget)

Pre-624 master: max_turns and max_budget_usd per-agent config were read by the metric tool (tool_get_agent_remaining_turns_and_budget) but not enforced anywhere. Observed coder-1 running 282/50 turns and $10.05/$5.00 USD on story 623 before a human stopped it (bug 624, now merged).

The bug 624 fix adds enforcement to the watchdog. The state-machine impact: introduces a new agent-termination path distinct from "Failed (orphan)" — something like Failed(LimitExceeded { kind: Turns | Budget }). The ExecutionState enum may want a corresponding terminal variant so it can be distinguished from generic Failed.

Pinned-agent honoring under contention

When a story has agent: coder-opus pinned but coder-opus is busy, today's auto-assign behavior is to leave the story unassigned — but if a human stops the running attempt and the story sits in current/, auto-assign re-grabs it with the default coder rather than waiting for the pinned agent. Observed multiple times on 2026-04-25 with story 623: pinning coder-opus did not prevent coder-1 (sonnet) from being auto-assigned during opus's busy window.

Missing behavior: auto-assign should treat a pinned agent as a hard filter ("only this agent can take this story"), not a preference. Today the workaround is to also set depends_on on a phantom story, or move the story back to backlog and let the dependency system gate it.

Honoring the blocked flag (bug 559)

559_bug_mergemaster_ignores_blocked_flag_and_keeps_respawning_on_blocked_stories is in backlog. Even though blocked: true is documented as a skip condition in auto_assign_available_work, mergemaster's spawn path apparently checks something different (or earlier) and respawns on blocked merge-stage stories. The state machine should make Stage::Archived { reason: Blocked } a single authoritative source so no consumer can incidentally bypass it.

Formal "ghost story recovery" transition

The move_story MCP tool description mentions "recovering a ghost story by moving it back to current" as a valid use. Ghost stories are CRDT entries with no corresponding filesystem stage directory (or the inverse). Today this is an update_story + move_story ad-hoc dance. A first-class recover_ghost_story verb that reconciles the CRDT and filesystem would formalize the recovery path.

Operator-level visibility / observability

There is no UI, CLI, or doc that shows "the state machine as a diagram." The typed enums are the closest thing to a canonical specification, but they aren't rendered anywhere a human can see at a glance: which stages exist, which transitions are valid, which events trigger them. A generated state diagram (graphviz or mermaid, dumped into this doc on each release) would help both new contributors and operators triaging stuck pipelines.

Review-hold cleanup verb

review_hold: true is set automatically on spike completion. Clearing it is done as a side effect of reject_qa (which also moves the story qa → current) or by manually editing front matter. There is no clean "I have reviewed this, release the hold" verb that doesn't also move the story.

Cross-node concurrency for execution state

ExecutionState is per-node (keyed by pubkey) so two nodes can't fight over who's running an agent. But there is no formal transition that says "node A hands the story to node B" if node A goes offline. The state machine's distributed semantics for this case are not yet specified.

---

How to update this document

Whenever you discover a transition that doesn't yet exist, or a flag that behaves surprisingly, add it to section (b) with:

• A short description of the desired behavior
• Citation of the work item or incident that surfaced it
• Pointer to the place in pipeline_state.rs where it should be modeled (or note "needs a new variant" if it doesn't fit any existing enum yet)

When a transition from (b) ships, move it to (a) with the relevant file:line citations.