---
name: "Typed pipeline state machine in Rust (foundation: replaces stringly-typed CRDT views with strict enums, subsumes 436)"
---

# Story 520: Typed pipeline state machine in Rust (foundation: replaces stringly-typed CRDT views with strict enums, subsumes 436)

## User Story

**Note:** content stuffed into user_story per bug 509 workaround.

---

## Context

Today huskies represents pipeline state as a loose JSON document inside the BFT JSON CRDT. Each story has fields like `stage: String`, `agent: String`, `retry_count: f64`, `blocked: bool`, `depends_on: String` (JSON-encoded list, double-encoded). This stringly-typed representation allows **many impossible states** to be representable in the data model:

- `stage = "9_invalid"` — typo, no compile error
- `stage = "5_done"` + `blocked = true` — a done story is blocked? what does that mean?
- `stage = "4_merge"` with no commits ahead of master — the silent mergemaster failure mode (today's story 478)
- A coder agent assigned to a story in `4_merge` — bug 502, the loop we fought all day today
- `retry_count = 3.7` — fractional retry counts (it's an f64 because that's what JSON CRDTs do)
- `agent = "coder-1"` AND `stage = "1_backlog"` — backlog story has an agent? sentinel encoding via empty string

Multiple bugs filed today (501, 502, 510, 511) exist *because* the type system can't enforce the pipeline invariants. **Patching individual symptoms forever is the wrong strategy.** The right strategy is to make impossible states unrepresentable at the Rust type level, using a typed state machine layered on top of the loose CRDT. The CRDT can stay loose at the persistence layer (it has to be — that's what makes it merge correctly across nodes), but every consumer above the CRDT operates on strict typed enums.

## Real user story

As a developer working on huskies, I want the pipeline state to be expressed as a strict Rust state machine where impossible states and impossible transitions are compile-time errors, so future bugs in this category become structural rather than runtime drift.

## Design

### Two enum hierarchies

**Synced state (CRDT-backed, converges across nodes):**

```rust
enum Stage {
    Backlog,
    Current,
    Qa,
    Merge { feature_branch: BranchName, commits_ahead: NonZeroU32 },
    Done { merged_at: DateTime<Utc>, merge_commit: GitSha },
    Archived { archived_at: DateTime<Utc>, reason: ArchiveReason },
}

enum ArchiveReason {
    Completed,                              // normal accept_story → archived
    Abandoned,                              // user explicitly abandoned
    Superseded { by: StoryId },
    Blocked { reason: String },             // was bug 436's `blocked: true`
    MergeFailed { reason: String },         // was bug 436's `merge_failure`
    ReviewHeld { reason: String },          // was bug 436's `review_hold`
}

struct PipelineItem {
    story_id: StoryId,                      // newtype, validated
    name: String,
    stage: Stage,                           // typed enum, all variants are valid by construction
    depends_on: Vec<StoryId>,               // parsed, not stringified
    retry_count: u32,                       // not f64
    // No more separate `blocked`, `merge_failure`, `review_hold` — folded into Stage::Archived
}
```

**Per-node execution state (CRDT-backed under node_id key, local-authored but globally-readable):**

```rust
enum ExecutionState {
    Idle,
    Pending     { agent: AgentName, since: DateTime<Utc> },
    Running     { agent: AgentName, started_at: DateTime<Utc>, last_heartbeat: DateTime<Utc> },
    RateLimited { agent: AgentName, resume_at: DateTime<Utc> },
    Completed   { agent: AgentName, exit_code: i32, completed_at: DateTime<Utc> },
}

// In the CRDT document, ExecutionState is stored under each node's pubkey:
//   crdt.execution_state: { node_pubkey → { story_id → ExecutionState } }
```

The execution state lives in the CRDT under **each node's pubkey**. Each node only writes to entries where `node_pubkey == self.pubkey`, so there's no merge conflict — concurrent writes from the same author follow LWW, concurrent writes from different authors target different entries entirely. All nodes can READ all execution states across the mesh.

**This per-node-keyed CRDT pattern enables:**

- **Cross-node observability** — matrix bot can show "node A is running coder-1 on story X, node B is rate-limited on story Y"
- **Heartbeat detection** — if a node hasn't updated its execution_state in N minutes, the entry is "stale" (laptop closed, process crashed, oom kill, etc.)
- **Foundation for story 479** (CRDT work claiming) — a node knows what other nodes are doing *before* claiming work
- **Stuck job recovery** — if node A's heartbeat dies mid-run, node B can see the stuck state and decide whether to take over
- **Crash forensics** — the last persisted ExecutionState before a crash is preserved in CRDT, accessible from any node

### The transition function

```rust
fn transition(
    state: PipelineItem,
    event: PipelineEvent,
) -> Result<PipelineItem, TransitionError>
```

Pure function. Takes the current state and an event, returns either the new state or a TransitionError. The compiler enforces that the result of every transition is structurally valid — you can't construct a `Stage::Merge` without `commits_ahead: NonZeroU32`, you can't construct a `Stage::Done` without a `merge_commit: GitSha`, etc.

**The set of valid transitions is small** (roughly 10):

- `Backlog → Current` — deps met, auto-assign promotes
- `Current → Qa` — gates start
- `Current → Merge` — qa: server, gates auto-pass
- `Qa → Merge` — gates pass
- `Qa → Current` — gates fail, retry
- `Merge → Done` — mergemaster squash succeeds; *requires `Merge.commits_ahead > 0`*
- `Done → Archived(Completed)` — accept_story
- `* → Archived(Blocked / MergeFailed / ReviewHeld)` — stuck-state move
- `Archived(Blocked) → Backlog` — unblock

Anything else is a `TransitionError`. The compiler refuses to compile code that constructs invalid transitions.

### The event subscriber pattern

State changes fire events on a bus. Side-effect handlers subscribe independently:

```rust
type TransitionEvent = (PipelineItem /* before */, PipelineItem /* after */);

bus.subscribe("matrix-bot",       |before, after| matrix_bot.notify_stage_change(before, after));
bus.subscribe("filesystem",       |before, after| fs_renderer.update(after));
bus.subscribe("pipeline-table",   |before, after| pipeline_items_table.upsert(after));
bus.subscribe("auto-assign",      |before, after| auto_assign.poke_if_relevant(after));
bus.subscribe("web-ui-broadcast", |before, after| ws_clients.broadcast(after));
```

Each subscriber is independent and concerns itself only with its own dispatch. Adding a new side effect = adding a new subscriber, not editing the transition function. **The "many things happen on state changes" complexity moves out of the state machine and into the bus consumers**, where each piece is testable in isolation.

### Projection layer (loose CRDT ↔ typed Rust)

The bft-json-crdt JSON document is the persistence layer. The typed enums are the application layer. A projection function bridges them at one carefully-controlled boundary:

```rust
impl TryFrom<&PipelineItemCrdt> for PipelineItem {
    type Error = ProjectionError;
    fn try_from(crdt: &PipelineItemCrdt) -> Result<Self, ProjectionError> { ... }
}

impl From<&PipelineItem> for PipelineItemCrdt {
    fn from(item: &PipelineItem) -> Self { ... }
}
```

When the CRDT contains data the typed layer can't parse (e.g. a stage value from a future huskies version, OR a merge that produces an inconsistent intermediate state), `try_from` returns a `ProjectionError`. The error surfaces to the caller — it doesn't silently propagate as garbage. The validation happens at exactly one point: the projection boundary.

## What this subsumes

**Story 436** ("Unify story stuck states into a single status field") is subsumed by the `Stage::Archived { reason: ArchiveReason }` variant. The unified status field IS the `ArchiveReason` enum. Story 436 is marked superseded by this story.

## What this enables (concrete bug eliminations)

- **Bug 502** becomes unrepresentable: there's no way to construct `Stage::Merge` with a Coder agent — Coder agents only attach to `Current` / `Qa` stages, and that constraint is in the type signature of the transition function.
- **Bug 510** becomes irrelevant: there's no "stage='1_backlog' filesystem shadow vs stage='5_done' DB" drift, because Stage is a typed enum with a single source of truth (the CRDT), and projections are derived deterministically.
- **Bug 519** (mergemaster silent on no-op merge) becomes unrepresentable: `Stage::Merge` requires `commits_ahead: NonZeroU32`. You can't construct a Merge state with zero commits.
- **Bug 511** (lamport seq reset) becomes detectable: the projection layer notices when CRDT data fails to parse cleanly and surfaces a ProjectionError instead of silently producing garbage in-memory state.
- **Story 479** (CRDT work claiming) has a clean foundation: ExecutionState gives every node visibility into what every other node is doing, including stale-heartbeat detection.
- **Future state machine bugs become compile errors**, not runtime drift.

## Implementation order

1. Define the `Stage`, `ArchiveReason`, `ExecutionState`, `PipelineItem` types in a new module (e.g. `server/src/pipeline_state.rs`).
2. Implement the projection layer (try_from / from for PipelineItemCrdt).
3. Implement the `transition` function with exhaustive valid transitions.
4. Implement the event bus.
5. Migrate consumers ONE AT A TIME — chat commands, lifecycle, API, auto-assign, matrix bot. Each migration is isolated; the compiler tells you when you've missed something.
6. Once nothing reads the loose `PipelineItemView` anymore, delete it.
7. Story 436 closes when this lands.

## Acceptance Criteria

- [ ] A new module (e.g. server/src/pipeline_state.rs) defines the Stage, ArchiveReason, ExecutionState, and PipelineItem types with the variants described in the design
- [ ] Stage::Merge has a NonZeroU32 commits_ahead field (so the bug 519 silent no-op merge is unrepresentable)
- [ ] Stage::Done has GitSha merge_commit and DateTime merged_at fields (so a 'done' story always has merge metadata)
- [ ] ArchiveReason enum subsumes the old blocked / merge_failure / review_hold front matter fields, with a sub-reason variant for each
- [ ] PipelineItem.depends_on is Vec<StoryId>, not String (no more JSON-as-string)
- [ ] ExecutionState lives in the CRDT under per-node-pubkey keys; each node only writes to its own subspace (validated by CRDT signature check)
- [ ] Last_heartbeat field is updated periodically by the running node so other nodes can detect stale entries
- [ ] A pure transition(state, event) -> Result<PipelineItem, TransitionError> function exists and is exhaustively pattern-matched
- [ ] Every valid transition listed in the design (~10) is implemented and unit-tested with both success and error cases
- [ ] The TryFrom<&PipelineItemCrdt> for PipelineItem projection function handles every currently-valid CRDT state and returns a structured ProjectionError for invalid ones (instead of silently propagating garbage)
- [ ] An event bus pattern is in place where matrix bot, filesystem renderer, pipeline_items materialiser, auto-assign, and web UI broadcaster are independent subscribers
- [ ] All call sites that previously read item.stage as a string or used the blocked / merge_failure / review_hold fields are migrated to the typed enum API
- [ ] Story 436 is closed as superseded by this story
- [ ] Bug 502 has a regression test that confirms the type system prevents the loop (the test should be a compile-fail test if possible)
- [ ] Bug 510 (filesystem shadow split-brain) no longer reproduces after this lands, because the typed state machine has a single source of truth
- [ ] Documentation in README.md or a new ARCHITECTURE.md explains the type hierarchy, the transition function, the event bus pattern, and the per-node ExecutionState convention

## Out of Scope

- TBD
