read_llm_session acquired the CRDT_STATE mutex, then called
extract_llm_session_view while holding the guard — which called
our_node_id(), which locks the same non-reentrant std::sync::Mutex.
The thread deadlocks itself and parks forever HOLDING the lock; every
other CRDT user then queues behind it. With light traffic that's a
partial wedge (MCP `show`/content reads hang while /health stays
green); during a CRDT-write burst (unblock → auto-assign) enough
tasks pile up to pin every tokio worker: liveness heartbeat stops,
/health dies, full sled freeze. Root cause of bug 1170's repeated
sled freezes, confirmed by live gdb capture: thread parked in
lock_contended at presence::our_node_id ← read_llm_session ←
event_matches_persona, with all other threads queued on CRDT reads.
Fix: extract_llm_session_view now takes local_sled_id as a parameter;
read_llm_session computes it from the guard it already holds. The
trigger path (event_matches_persona on persona-subscribed WS events)
explains the raciness — it needs a chat/persona event racing a
pipeline transition.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
logging-list/logging-json were in bft-json-crdt's default feature set,
so every production build printed multi-KB debug dumps on every CRDT
op — executed INSIDE the global CRDT_STATE mutex. A stdout write that
stalls while holding that lock blocks every task touching the CRDT
(tick loop, watchers, MCP, RPC), each one pinning an OS worker thread
until the tokio pool is exhausted: liveness heartbeat stops, /health
dies, zero CPU. This is the mechanism behind bug 1170 (two full-sled
freezes on huskies-server, both seconds after a CRDT write burst, the
second insert's dump truncated mid-print in the log).
The features remain available for CRDT debugging via explicit opt-in.
Also fixes a latent race in persist_tx_send_success_emits_no_warn:
it counted [crdt_persist] warns in the process-global log buffer,
which parallel tests also write to; the debug prints had been acting
as an accidental serializer. Now filters for its own story id.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
/api/upgrade now verifies the target can be replaced (create + remove
the swap's tmp file) before returning 202. A sled that cannot write
its own binary — e.g. a container predating the /opt/huskies/bin
layout — fails phase 1 of `upgrade all` loudly instead of returning
202, staying healthy, and silently remaining on the old version, which
is exactly what happened on the first fleet deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
Since story 884 the Matrix permission listener (and the sled uplink,
when configured) hold services.perm_rx for the process lifetime. The
WS chat handler's blocking `lock().await` on that same mutex therefore
parked the entire WS connection loop forever on StartChat: chat_fut was
never polled, RPC frames on the socket were never answered, and
everything queued behind the dispatcher's serial session lock —
wedging /mcp and /rpc while /health stayed green.
Use try_lock instead: if another task already owns permission routing,
run the chat without the local permission-forwarding select arm (a
pending future keeps the select shape unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
Agents die all the time; the pipeline's retry machinery re-queues
their work. Skipping busy sleds just created version skew and manual
retries for no real protection. `upgrade all` now sweeps every sled
unconditionally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
Killed:
- rebuild_and_restart (in-container cargo self-compile): the MCP tool,
the `rebuild` chat command in all four transports, the web-ui bot
command, and the underlying function. This was the path that caused
the exec() deadlocks.
- upgrade_sled gateway MCP tool: second entry point to sled upgrades,
defaulted to serving the gateway's own macOS binary to Linux sleds.
- GET /api/huskies-binary (both sled and gateway route trees): served
current_exe(), wrong platform when the gateway is macOS. Superseded
by /api/artifacts/ which now also serves on the gateway route tree.
- `huskies upgrade` CLI subcommand and --source flag: third way of
doing the same download-and-replace. Escape hatch for a bricked sled
is `docker cp` + restart.
Kept, distinct jobs: `project-rebuild` (container/image updates),
`rebuild gateway` + script/local-release (gateway self-update).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
Finds the registered project carrying the huskies source tree, runs
`cargo build --release` inside its container (dedicated
CARGO_TARGET_DIR=target/sled-release so container builds stop
clobbering host target/release), then atomically publishes the binary
to ~/.huskies/artifacts/ with a .hash sidecar for convergence checks.
Full fleet redeploy is now chat-only: `release` then `upgrade all` —
no laptop access needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
- `upgrade all` sweeps every registered sled in sequence, streaming
per-sled phase markers and reporting a summary.
- Binary source is now the gateway's own artifact store via
host.docker.internal (was: unresolvable `gateway` hostname serving
the gateway's macOS binary to Linux sleds — would have bricked them).
- Sleds with active claude processes are skipped, never killed.
- After reconnect, /api/version git_hash is compared against the
published artifact's .hash sidecar; divergence is reported loudly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
GET /api/artifacts/:filename with filename validation (no path
components, no dotfiles). Sleds only ever download binaries from their
own gateway; this endpoint is where the gateway serves them from,
replacing the current_exe()-based /api/huskies-binary which serves the
gateway's own (macOS) binary — wrong platform for Linux sleds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
/opt/huskies/bin/huskies (chowned to the huskies user) with a symlink
from /usr/local/bin/huskies. Atomic replace needs write permission on
the directory for the tmp-write + rename, which root-owned
/usr/local/bin can't provide to the server process.
resolve_target_path() now prefers /opt/huskies/bin/huskies over
current_exe(), which can point at a stale location (e.g.
/workspace/target/release/huskies after a historical in-container
rebuild) that the entrypoint would never launch after a restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
CARGO_MANIFEST_DIR is baked at image build time as /app/server, but
project containers (Dockerfile.base) don't copy /app — the source is
bind-mounted at /workspace instead. Fall back to project_root when
the compile-time path doesn't exist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Huskies Agent
TimmyandClaude Fable 5