---
name: "Startup reconcile pass that detects drift between CRDT, pipeline_items, and filesystem shadows"
---

# Story 513: Startup reconcile pass that detects drift between CRDT, pipeline_items, and filesystem shadows

## User Story

**Note:** content stuffed into user_story per bug 509 workaround.

---

## Context

Post-491/492, huskies has **four places state lives** that can drift apart:

1. `crdt_ops` table — the persisted CRDT operation log (intended source of truth)
2. In-memory CRDT view — `state.crdt.doc.items` reconstructed from `crdt_ops` on startup, mutated by `apply_and_persist` during runtime
3. `pipeline_items` table — a shadow / materialised view, written to as a shadow alongside CRDT writes
4. Filesystem shadows in `.huskies/work/N_stage/*.md` — legacy rendering, still written by some paths and read by others

There is currently **no reconcile pass** that detects drift between them. We've watched this drift bite repeatedly today: stories appear in some views and not others, lifecycle moves happen in one but not another, my direct sqlite UPDATE was invisible to the API, etc. Each individual view looks "fine" in isolation, but the drift only becomes visible when a user notices a story behaving inconsistently.

## Real user story

As a developer or operator running huskies, I want a startup reconcile pass that compares all four state sources and either reconciles them automatically (preferred) or logs structured warnings about the drift, so I can detect and diagnose state corruption before it causes user-visible bugs.

## Observed 2026-04-09

Throughout this session we observed: 478 in pipeline_items but missing from CRDT (after a direct sqlite insert), 503 in CRDT at stage=merge but pipeline_items at stage=5_done (after my UPDATE), filesystem shadows in `1_backlog/` for stories that were already in 5_done in the DB (bug 510), etc. None of these were detected by huskies itself — they were all only found by running ad-hoc `SELECT` queries during incident response.

## Scope

This is the *detection* story, not the *fix-the-drift* story. The reconcile pass should:

- Run at startup (after CRDT replay, before serving requests)
- Compare each story's stage across all four sources
- Emit structured log lines for each drift type (CRDT-only, FS-only, DB-only, stage mismatch, etc.)
- Optionally surface a count to the matrix bot startup announcement (e.g. "⚠️ 3 stories have CRDT/DB drift — see logs")

The actual *fix-the-drift* logic (what to do when drift is detected) is a separate, larger story.

## Acceptance Criteria

- [ ] At server startup, after CRDT replay, a reconcile_state() function runs that walks all four state sources and detects drift
- [ ] Each drift type is logged with a structured line: e.g. `[reconcile] DRIFT story=X crdt_stage=Y db_stage=Z fs_stage=W` (or `MISSING` for absent)
- [ ] If any drift is detected, the matrix bot startup announcement includes a count and a suggestion to check the server logs
- [ ] The reconcile pass completes in < 1 second for a typical pipeline (~100 stories) so it doesn't slow startup meaningfully
- [ ] Tests cover: no drift (clean state), CRDT-only story, DB-only story, FS-only story, stage mismatch between CRDT and DB
- [ ] Documentation in README.md explains the reconcile pass and what each drift type means
- [ ] The pass is opt-out via a config flag in case it produces noise during the migration window

## Out of Scope

- TBD
