---
name: "CRDT lamport clock (inner.seq) resets to 1 on server restart instead of resuming from max(own_author_seq) + 1"
---

# Bug 511: CRDT lamport clock (inner.seq) resets to 1 on server restart instead of resuming from max(own_author_seq) + 1

## Description

When the huskies server restarts (e.g. via `rebuild_and_restart`), the local node's CRDT lamport clock — `inner.seq` on each `SignedOp` — appears to reset to 1 instead of resuming from `MAX(seq) + 1` for the local author's own previously-persisted ops.

**Discovered live on 2026-04-09** while inspecting the `crdt_ops` table after a `rebuild_and_restart`. Pre-restart ops were at seqs 485-492 (creation ops for stories 503-510). Post-restart ops were being persisted at seqs 1, 2, 3, 4, 5, 6, 7 — visible by sorting `crdt_ops` by `created_at DESC`:

```
created_at                              | seq
2026-04-09T18:49:56  → seq 7      ← post-restart
2026-04-09T18:38:22  → seq 6      ← post-restart
2026-04-09T18:37:45  → seq 492    ← pre-restart, last write before restart
2026-04-09T18:31:32  → seq 4      ← post-restart
2026-04-09T18:27:04  → seq 3      ← post-restart
```

So the local node, which had reached seq=492 before the restart, started writing new ops at seq=1 after the restart and is now climbing from there. This means **new ops have lower seqs than existing ops from the same author**.

## Why this matters

In a BFT JSON CRDT, `inner.seq` is the local lamport clock used for causality tracking. The library assumes per-author seqs are monotonically increasing — newer ops from the same author have higher seqs than older ops. Several things break when this invariant is violated:

1. **Causality / ordering on remote replay.** When a peer (or this same node after another restart) replays the persisted ops in `seq` order, the post-restart ops will be applied *before* the pre-restart ops, even though they happened later. This can produce a non-deterministic state and can cause field updates to "go backwards" — e.g. a story that was moved current → done pre-restart, then nothing post-restart, would correctly end at "done"; but if you also did a post-restart action, the seq ordering would re-play it in the wrong order.

2. **Op ID collisions.** The op id is a hash of the op contents (including author, seq, content). If a post-restart op happens to be structurally identical to a pre-restart op (e.g. "set stage to 1_backlog" with the same author and same seq=3), the op ids could collide. The persistence path uses `INSERT INTO crdt_ops ... ON CONFLICT(op_id) DO NOTHING`, which would *silently drop* the new op. (We have not yet observed this happen, but it's a latent risk.)

3. **Sync between nodes will desync.** Once the WebSocket sync layer (story 478, just merged) is exchanging ops between nodes, a restart on one node will produce ops with seqs that look "old" to the other node, and the receiving node may de-dupe or mis-order them. This will manifest as silent state divergence in multi-node deployments, which is exactly what the sync layer is supposed to prevent.

4. **Today's pipeline state confusion.** The 8 stories I created in this session (503-510) are at seqs 485-492 in the persisted CRDT. Their post-restart lifecycle moves are at seqs 1-15. If we replay the CRDT from disk in seq order, the lifecycle moves will be applied to *empty* state before the creation ops have run, and will silently no-op (because they reference content indexes that don't exist yet). On the next restart after this state, the in-memory view will show the stories in their *creation* state, not their post-restart-lifecycle state — i.e. all 8 stories will appear "stuck at 1_backlog" again. **This may well be the cause of bug 510's split-brain symptom**.

## Where the bug lives

`server/src/crdt_state.rs::init()` (around lines 80-115) replays persisted ops to reconstruct state, then constructs a fresh `CrdtState { crdt, keypair, index, persist_tx }`. The `BaseCrdt::new(&keypair)` call constructs a fresh CRDT with a fresh internal seq counter. The replay re-applies ops via `crdt.apply(signed_op)` which presumably updates the doc but does NOT advance the local seq counter (because `apply()` is for *remote* ops).

After replay, the local seq counter is at 0 (or wherever the BFT CRDT library defaults). The next call to `apply_and_persist` produces an op with `inner.seq = 1` (or whatever the next-counter value is) — even though there are already ops at seq 485+ from this author in the persisted state.

The fix is to inspect `MAX(inner.seq)` for ops where `author == local_keypair.public()` after the replay, and seed the BFT CRDT's local seq counter from `max + 1`. The exact API for "seed the seq counter" depends on the bft-json-crdt library — may need a small upstream change if not already exposed.

## How to Reproduce

1. Start a fresh huskies server with an empty database. Verify `crdt_ops` is empty.
2. Create several stories via `create_story` or similar — observe ops being persisted at incrementing seqs (1, 2, 3, ...).
3. Note the highest seq via `sqlite3 .huskies/pipeline.db "SELECT MAX(seq) FROM crdt_ops;"` — call this N.
4. Stop the server and start it again (or `rebuild_and_restart`).
5. Create another story via `create_story`.
6. Query `SELECT seq, created_at FROM crdt_ops ORDER BY created_at DESC LIMIT 5;`

## Actual Result

The new op (just created in step 5) is persisted with `seq = 1` (or some small value), NOT `seq = N + 1`. The lamport clock has been reset.

Concretely on 2026-04-09 we observed seqs in `crdt_ops` ordered by `created_at` DESC of: 7, 6, 492, 4, 3 — i.e. post-restart writes were at seqs 3, 4, 6, 7 even though the highest pre-restart seq was 492.

## Expected Result

After restart, the local node's seq counter must resume from `MAX(inner.seq)` across all persisted ops where `author == local_keypair.public()`, plus 1. The next op written by the local node should have `seq = N + 1` where N is the previous local high-water mark.

Equivalent stated: `inner.seq` on the local author's ops must be monotonically increasing across the entire lifetime of the local node's keypair, not just within a single process invocation.

## Acceptance Criteria

- [ ] After a server restart, the next CRDT op written by the local node has seq = MAX(local_author_seq from crdt_ops) + 1, not 1
- [ ] Regression test: seed crdt_ops with an op at seq=100 by the local author, restart the CRDT subsystem (or call init() in a test harness), trigger a write_item, assert the new op has seq=101
- [ ] Regression test: a brand-new node (no pre-existing ops) still starts at seq=1 (no off-by-one introduced by the fix)
- [ ] Inter-node test: simulate two nodes A and B, A writes ops up to seq=50, A restarts, A writes a new op which should be seq=51, broadcast to B, assert B applies it in the correct causal position
- [ ] If the fix requires changes to bft-json-crdt itself (to expose a way to seed the local seq), the upstream change is documented in the bug body and either landed or vendored
- [ ] After this fix is in place, replay-on-restart for the existing data (8 stories in pipeline_items at seqs 485-492 with lifecycle moves at seqs 1-15) is verified to produce the correct in-memory state — OR the existing broken-seq data is migrated as part of the fix
