---
name: "Stop auto-committing intermediate pipeline moves"
agent: "coder-opus"
review_hold: true
---

# Spike 92: Stop auto-committing intermediate pipeline moves

## Goal

Determine how to stop the filesystem watcher from auto-committing every pipeline stage move (upcoming -> current -> qa -> merge -> done -> archive) while still committing at terminal states (creation in upcoming, acceptance in done and archived). This keeps git history clean while preserving cross-machine portability for completed work.

## Context

The watcher in `server/src/io/watcher.rs` currently auto-commits every file change in `.story_kit/work/`. A single story run generates 5+ commits just from pipeline moves:
- `huskies: create 42_story_foo`
- `huskies: start 42_story_foo`
- `huskies: queue 42_story_foo for QA`
- `huskies: queue 42_story_foo for merge`
- `huskies: accept 42_story_foo`

Since story runs complete relatively quickly, the intermediate state (current/qa/merge) is transient and doesn't need to be committed. Only creation and archival are meaningful checkpoints.

## Questions to Answer

1. Can we filter `stage_metadata()` to only commit for `1_upcoming` and `6_archived` stages while still broadcasting `WatcherEvent`s for all stages (so the frontend stays in sync)?
2. Should we keep `git add -A .story_kit/work/` for the committed stages, or narrow it to only the specific file?
3. What happens if the server crashes mid-pipeline? Uncommitted moves are lost — is this acceptable given the story can just be re-run?
4. Should intermediate moves be `.gitignore`d at the directory level, or is filtering in the watcher sufficient?
5. Do any other parts of the system (agent worktree setup, merge_agent_work, sparse checkout) depend on intermediate pipeline files being committed to master?

## Findings

### Q1: Can we filter to only commit terminal stages?

**Yes.** The fix is in `flush_pending()`, not `stage_metadata()`. We add a `should_commit_stage()` predicate that returns `true` only for `1_upcoming` and `6_archived`. The event broadcast path is decoupled from the commit path — `flush_pending()` always broadcasts a `WatcherEvent` regardless of whether it commits.

Prototype implemented: added `COMMIT_WORTHY_STAGES` constant and `should_commit_stage()` function. The change is ~15 lines including the constant, predicate, and conditional in `flush_pending()`.

### Q2: Keep `git add -A .story_kit/work/` or narrow to specific file?

**Keep `git add -A .story_kit/work/`.** When committing a terminal stage (e.g. `6_archived`), the file has been moved from a previous stage (e.g. `5_done`). Using `-A` on the whole work directory captures both the addition in the new stage and the deletion from the old stage in a single commit. Narrowing to the specific file would miss the deletion side of the move.

### Q3: Server crash mid-pipeline — acceptable?

**Yes.** If the server crashes while a story is in `2_current`, `3_qa`, or `4_merge`, the file is lost from git but:
- The story file still exists on the filesystem (it's just not committed)
- The agent's work is in its own feature branch/worktree (independent of pipeline file state)
- The story can be re-queued from `1_upcoming` which IS committed
- Pipeline state is transient by nature — it reflects "what's happening right now", not permanent record

### Q4: `.gitignore` vs watcher filtering?

**Watcher filtering is sufficient.** `.gitignore` approach (Option C) has downsides:
- `git status` won't show pipeline state, making debugging harder
- If you ever need to commit an intermediate state (e.g. for a new feature), you'd have to fight `.gitignore`
- Watcher filtering is explicit and easy to understand — a constant lists the commit-worthy stages
- No risk of accidentally ignoring files that should be tracked

### Q5: Dependencies on intermediate pipeline commits?

**None found.** Thorough investigation confirmed:

1. **`merge_agent_work`** (`agents/merge.rs`): Creates a temporary `merge-queue/` branch and worktree. Reads the feature branch, not pipeline files. After merge, calls `move_story_to_archived()` which is a filesystem operation.

2. **Agent worktree setup** (`worktree.rs`): Creates worktrees from feature branches. Sparse checkout is a no-op (disabled). Does not read pipeline file state from git.

3. **MCP tool handlers** (`agents/lifecycle.rs`): `move_story_to_current()`, `move_story_to_merge()`, `move_story_to_qa()`, `move_story_to_archived()` — all pure filesystem `fs::rename()` operations. None perform git commits.

4. **Frontend** (`http/workflow.rs`): `load_pipeline_state()` reads directories from the filesystem directly via `fs::read_dir()`. Never calls git. WebSocket events keep the frontend in sync.

5. **No git inspection commands** reference pipeline stage directories anywhere in the codebase.

### Edge Cases

- **Multi-machine sync:** Only `1_upcoming` and `6_archived` are committed. If you push/pull, you'll see story creation and archival but not intermediate pipeline state. This is correct — intermediate state is machine-local runtime state.
- **Manual git operations:** `git status` will show uncommitted files in intermediate stages. This is actually helpful for debugging — you can see what's in the pipeline without grepping git log.
- **Sweep (5_done → 6_archived):** The sweep moves files to `6_archived`, which triggers a watcher event that WILL commit (since `6_archived` is a terminal stage). This naturally captures the final state.

## Approach to Investigate

### Option A: Filter in `flush_pending()` ← **RECOMMENDED**
- In `flush_pending()`, still broadcast the `WatcherEvent` for all stages
- Only call `git_add_work_and_commit()` for stages `1_upcoming` and `6_archived`
- Simplest change — ~15 lines modified in `watcher.rs`

### Option B: Two-tier watcher
- Split into "commit-worthy" events (create, archive) and "notify-only" events (start, qa, merge)
- Commit-worthy events go through git
- Notify-only events just broadcast to WebSocket clients
- More explicit but same end result as Option A

### Option C: .gitignore intermediate directories
- Add `2_current/`, `3_qa/`, `4_merge/` to `.gitignore`
- Watcher still sees events (gitignore doesn't affect filesystem watching)
- Git naturally ignores them
- Risk: harder to debug, `git status` won't show pipeline state

## Recommendation

**Option A is viable and implemented.** The prototype is in `server/src/io/watcher.rs`:
- Added `COMMIT_WORTHY_STAGES` constant: `["1_upcoming", "6_archived"]`
- Added `should_commit_stage()` predicate
- Modified `flush_pending()` to conditionally commit based on stage, while always broadcasting events
- All 872 tests pass, clippy clean

A full story run will now produce only 2 pipeline commits instead of 5+:
- `huskies: create 42_story_foo` (creation in `1_upcoming`)
- `huskies: accept 42_story_foo` (archival in `6_archived`)

The intermediate moves (`start`, `queue for QA`, `queue for merge`, `done`) are still broadcast to WebSocket clients for real-time frontend updates, but no longer clutter git history.

## Acceptance Criteria

- [x] Spike document updated with findings and recommendation
- [x] If Option A is viable: prototype the change and verify git log is clean during a full story run
- [x] Confirm frontend still receives real-time pipeline updates for all stages
- [x] Confirm no other system depends on intermediate pipeline commits being on master
- [x] Identify any edge cases (server crash, manual git operations, multi-machine sync)
