---
name: "apply_and_persist should log when persist_tx send fails instead of silently dropping the op"
---

# Story 518: apply_and_persist should log when persist_tx send fails instead of silently dropping the op

## User Story

**Note:** content stuffed into user_story per bug 509 workaround.

---

## Context

`server/src/crdt_state.rs::apply_and_persist` updates the in-memory CRDT and then sends the signed op to the persistence task via a channel:

```rust
fn apply_and_persist<F>(state: &mut CrdtState, op_fn: F) {
    let raw_op = op_fn(state);
    let signed = raw_op.sign(&state.keypair);
    state.crdt.apply(signed.clone());                  // in-memory update
    let _ = state.persist_tx.send(signed.clone());     // ← fire-and-forget, error dropped
    ...
}
```

The `let _ = ...` discards the return value of `send()`. If the channel is closed (because the persistence task panicked, was shut down, or has dropped its receiver), the op is silently dropped from persistence — but the in-memory CRDT is already updated. The next restart will replay only the persisted ops, and the in-memory state will quietly diverge from the persisted state.

This is also one of the candidate causes for some of the state drift we've been chasing. It's hard to rule out because there's no log line to confirm whether the persist task is still alive or whether sends are succeeding.

## Real user story

As a developer or operator, I want any failure of `persist_tx.send()` to be logged immediately at WARN or ERROR level, so silent persistence loss is detectable instead of invisible.

## Observed 2026-04-09

Spent significant time investigating whether persist sends were silently failing. Eventually ruled it out empirically (we found that ops WERE being persisted, just with reset seq counters per bug 511). But the diagnosis would have been minutes instead of an hour if there was a log line to check.

## Fix (small)

```rust
if let Err(e) = state.persist_tx.send(signed.clone()) {
    crate::slog_error!(
        "[crdt] Failed to send op to persist task: {e}; persist task may be dead. \
         In-memory state is now ahead of persisted state."
    );
}
```

Apply the same fix at every `let _ = state.persist_tx.send(...)` site in crdt_state.rs (there are at least 2 — one in apply_and_persist, one in apply_remote_op).

## Acceptance Criteria

- [ ] Every call site of `state.persist_tx.send(...)` in crdt_state.rs logs at ERROR level on send failure
- [ ] The error message includes the channel error and a clear note that 'in-memory and persisted state may have diverged'
- [ ] Unit test: shut down the persist receiver (drop the rx end), call write_item, assert an error is logged
- [ ] No regression in the happy path (no extra log lines on success)
- [ ] Consider: also expose a counter / metric for persist send failures so it can be monitored without grepping logs

## Out of Scope

- TBD
