story-kit: create 92_spike_stop_auto_committing_intermediate_pipeline_moves

.story_kit/work/5_done/92_spike_stop_auto_committing_intermediate_pipeline_moves.md

@@ -1,119 +0,0 @@
----
-name: "Stop auto-committing intermediate pipeline moves"
-agent: "coder-opus"
-review_hold: true
----
-
-# Spike 92: Stop auto-committing intermediate pipeline moves
-
-## Goal
-
-Determine how to stop the filesystem watcher from auto-committing every pipeline stage move (upcoming -> current -> qa -> merge -> done -> archive) while still committing at terminal states (creation in upcoming, acceptance in done and archived). This keeps git history clean while preserving cross-machine portability for completed work.
-
-## Context
-
-The watcher in `server/src/io/watcher.rs` currently auto-commits every file change in `.story_kit/work/`. A single story run generates 5+ commits just from pipeline moves:
-- `story-kit: create 42_story_foo`
-- `story-kit: start 42_story_foo`
-- `story-kit: queue 42_story_foo for QA`
-- `story-kit: queue 42_story_foo for merge`
-- `story-kit: accept 42_story_foo`
-
-Since story runs complete relatively quickly, the intermediate state (current/qa/merge) is transient and doesn't need to be committed. Only creation and archival are meaningful checkpoints.
-
-## Questions to Answer
-
-1. Can we filter `stage_metadata()` to only commit for `1_upcoming` and `6_archived` stages while still broadcasting `WatcherEvent`s for all stages (so the frontend stays in sync)?
-2. Should we keep `git add -A .story_kit/work/` for the committed stages, or narrow it to only the specific file?
-3. What happens if the server crashes mid-pipeline? Uncommitted moves are lost — is this acceptable given the story can just be re-run?
-4. Should intermediate moves be `.gitignore`d at the directory level, or is filtering in the watcher sufficient?
-5. Do any other parts of the system (agent worktree setup, merge_agent_work, sparse checkout) depend on intermediate pipeline files being committed to master?
-
-## Findings
-
-### Q1: Can we filter to only commit terminal stages?
-
-**Yes.** The fix is in `flush_pending()`, not `stage_metadata()`. We add a `should_commit_stage()` predicate that returns `true` only for `1_upcoming` and `6_archived`. The event broadcast path is decoupled from the commit path — `flush_pending()` always broadcasts a `WatcherEvent` regardless of whether it commits.
-
-Prototype implemented: added `COMMIT_WORTHY_STAGES` constant and `should_commit_stage()` function. The change is ~15 lines including the constant, predicate, and conditional in `flush_pending()`.
-
-### Q2: Keep `git add -A .story_kit/work/` or narrow to specific file?
-
-**Keep `git add -A .story_kit/work/`.** When committing a terminal stage (e.g. `6_archived`), the file has been moved from a previous stage (e.g. `5_done`). Using `-A` on the whole work directory captures both the addition in the new stage and the deletion from the old stage in a single commit. Narrowing to the specific file would miss the deletion side of the move.
-
-### Q3: Server crash mid-pipeline — acceptable?
-
-**Yes.** If the server crashes while a story is in `2_current`, `3_qa`, or `4_merge`, the file is lost from git but:
-- The story file still exists on the filesystem (it's just not committed)
-- The agent's work is in its own feature branch/worktree (independent of pipeline file state)
-- The story can be re-queued from `1_upcoming` which IS committed
-- Pipeline state is transient by nature — it reflects "what's happening right now", not permanent record
-
-### Q4: `.gitignore` vs watcher filtering?
-
-**Watcher filtering is sufficient.** `.gitignore` approach (Option C) has downsides:
-- `git status` won't show pipeline state, making debugging harder
-- If you ever need to commit an intermediate state (e.g. for a new feature), you'd have to fight `.gitignore`
-- Watcher filtering is explicit and easy to understand — a constant lists the commit-worthy stages
-- No risk of accidentally ignoring files that should be tracked
-
-### Q5: Dependencies on intermediate pipeline commits?
-
-**None found.** Thorough investigation confirmed:
-
-1. **`merge_agent_work`** (`agents/merge.rs`): Creates a temporary `merge-queue/` branch and worktree. Reads the feature branch, not pipeline files. After merge, calls `move_story_to_archived()` which is a filesystem operation.
-
-2. **Agent worktree setup** (`worktree.rs`): Creates worktrees from feature branches. Sparse checkout is a no-op (disabled). Does not read pipeline file state from git.
-
-3. **MCP tool handlers** (`agents/lifecycle.rs`): `move_story_to_current()`, `move_story_to_merge()`, `move_story_to_qa()`, `move_story_to_archived()` — all pure filesystem `fs::rename()` operations. None perform git commits.
-
-4. **Frontend** (`http/workflow.rs`): `load_pipeline_state()` reads directories from the filesystem directly via `fs::read_dir()`. Never calls git. WebSocket events keep the frontend in sync.
-
-5. **No git inspection commands** reference pipeline stage directories anywhere in the codebase.
-
-### Edge Cases
-
-- **Multi-machine sync:** Only `1_upcoming` and `6_archived` are committed. If you push/pull, you'll see story creation and archival but not intermediate pipeline state. This is correct — intermediate state is machine-local runtime state.
-- **Manual git operations:** `git status` will show uncommitted files in intermediate stages. This is actually helpful for debugging — you can see what's in the pipeline without grepping git log.
-- **Sweep (5_done → 6_archived):** The sweep moves files to `6_archived`, which triggers a watcher event that WILL commit (since `6_archived` is a terminal stage). This naturally captures the final state.
-
-## Approach to Investigate
-
-### Option A: Filter in `flush_pending()` ← **RECOMMENDED**
-- In `flush_pending()`, still broadcast the `WatcherEvent` for all stages
-- Only call `git_add_work_and_commit()` for stages `1_upcoming` and `6_archived`
-- Simplest change — ~15 lines modified in `watcher.rs`
-
-### Option B: Two-tier watcher
-- Split into "commit-worthy" events (create, archive) and "notify-only" events (start, qa, merge)
-- Commit-worthy events go through git
-- Notify-only events just broadcast to WebSocket clients
-- More explicit but same end result as Option A
-
-### Option C: .gitignore intermediate directories
-- Add `2_current/`, `3_qa/`, `4_merge/` to `.gitignore`
-- Watcher still sees events (gitignore doesn't affect filesystem watching)
-- Git naturally ignores them
-- Risk: harder to debug, `git status` won't show pipeline state
-
-## Recommendation
-
-**Option A is viable and implemented.** The prototype is in `server/src/io/watcher.rs`:
-- Added `COMMIT_WORTHY_STAGES` constant: `["1_upcoming", "6_archived"]`
-- Added `should_commit_stage()` predicate
-- Modified `flush_pending()` to conditionally commit based on stage, while always broadcasting events
-- All 872 tests pass, clippy clean
-
-A full story run will now produce only 2 pipeline commits instead of 5+:
-- `story-kit: create 42_story_foo` (creation in `1_upcoming`)
-- `story-kit: accept 42_story_foo` (archival in `6_archived`)
-
-The intermediate moves (`start`, `queue for QA`, `queue for merge`, `done`) are still broadcast to WebSocket clients for real-time frontend updates, but no longer clutter git history.
-
-## Acceptance Criteria
-
-- [x] Spike document updated with findings and recommendation
-- [x] If Option A is viable: prototype the change and verify git log is clean during a full story run
-- [x] Confirm frontend still receives real-time pipeline updates for all stages
-- [x] Confirm no other system depends on intermediate pipeline commits being on master
-- [x] Identify any edge cases (server crash, manual git operations, multi-machine sync)