Code review of 1185 (merge 0f4b0c95) found the retry subscriber's central
invariant did not hold: the MergeFailure->Merge bounce caused by its own
retry reset both its attempt counter and the block subscriber's counter,
so the shared merge_failure_block_threshold budget was unreachable and a
deterministic gates failure retried forever.
One subscriber now owns one counter driving both policies:
- Counter survives PipelineEvent::MergeRetryStarted bounces (finding 1);
a third consecutive failure blocks even with retries in between.
- Mixed failure kinds share the single budget (finding 5).
- Retries respect recovery: no counting or scheduling while a mergemaster
is active, and perform_auto_retry re-checks before firing (finding 2).
- perform_auto_retry applies the same eligibility gates as
assign_merge_stage (review hold, frozen, blocked, unmet deps) so freeze
now stops a retry loop (finding 4).
- Per-story scheduling generations invalidate stale sleeping timers
(finding 6).
- One-shot startup scan schedules a catch-up retry for stories already
parked in GatesFailed, so restarts no longer strand them (finding 3);
kept out of the periodic reconciler to avoid re-retrying exhausted
stories every tick.
- Chat is notified only after the merge actually starts; a failed trigger
logs instead of claiming a retry ran (finding 7).
- Config reads moved onto spawn_blocking (finding 8, bug 1170 class).
Deletes merge_failure_retry_subscriber.rs; notification plumbing
(WatcherEvent::MergeAutoRetry et al) is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
read_llm_session acquired the CRDT_STATE mutex, then called
extract_llm_session_view while holding the guard — which called
our_node_id(), which locks the same non-reentrant std::sync::Mutex.
The thread deadlocks itself and parks forever HOLDING the lock; every
other CRDT user then queues behind it. With light traffic that's a
partial wedge (MCP `show`/content reads hang while /health stays
green); during a CRDT-write burst (unblock → auto-assign) enough
tasks pile up to pin every tokio worker: liveness heartbeat stops,
/health dies, full sled freeze. Root cause of bug 1170's repeated
sled freezes, confirmed by live gdb capture: thread parked in
lock_contended at presence::our_node_id ← read_llm_session ←
event_matches_persona, with all other threads queued on CRDT reads.
Fix: extract_llm_session_view now takes local_sled_id as a parameter;
read_llm_session computes it from the guard it already holds. The
trigger path (event_matches_persona on persona-subscribed WS events)
explains the raciness — it needs a chat/persona event racing a
pipeline transition.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
logging-list/logging-json were in bft-json-crdt's default feature set,
so every production build printed multi-KB debug dumps on every CRDT
op — executed INSIDE the global CRDT_STATE mutex. A stdout write that
stalls while holding that lock blocks every task touching the CRDT
(tick loop, watchers, MCP, RPC), each one pinning an OS worker thread
until the tokio pool is exhausted: liveness heartbeat stops, /health
dies, zero CPU. This is the mechanism behind bug 1170 (two full-sled
freezes on huskies-server, both seconds after a CRDT write burst, the
second insert's dump truncated mid-print in the log).
The features remain available for CRDT debugging via explicit opt-in.
Also fixes a latent race in persist_tx_send_success_emits_no_warn:
it counted [crdt_persist] warns in the process-global log buffer,
which parallel tests also write to; the debug prints had been acting
as an accidental serializer. Now filters for its own story id.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
/api/upgrade now verifies the target can be replaced (create + remove
the swap's tmp file) before returning 202. A sled that cannot write
its own binary — e.g. a container predating the /opt/huskies/bin
layout — fails phase 1 of `upgrade all` loudly instead of returning
202, staying healthy, and silently remaining on the old version, which
is exactly what happened on the first fleet deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
Since story 884 the Matrix permission listener (and the sled uplink,
when configured) hold services.perm_rx for the process lifetime. The
WS chat handler's blocking `lock().await` on that same mutex therefore
parked the entire WS connection loop forever on StartChat: chat_fut was
never polled, RPC frames on the socket were never answered, and
everything queued behind the dispatcher's serial session lock —
wedging /mcp and /rpc while /health stayed green.
Use try_lock instead: if another task already owns permission routing,
run the chat without the local permission-forwarding select arm (a
pending future keeps the select shape unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
Agents die all the time; the pipeline's retry machinery re-queues
their work. Skipping busy sleds just created version skew and manual
retries for no real protection. `upgrade all` now sweeps every sled
unconditionally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
Huskies Agent
TimmyandClaude Fable 5