Agents die all the time; the pipeline's retry machinery re-queues
their work. Skipping busy sleds just created version skew and manual
retries for no real protection. `upgrade all` now sweeps every sled
unconditionally.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
Killed:
- rebuild_and_restart (in-container cargo self-compile): the MCP tool,
the `rebuild` chat command in all four transports, the web-ui bot
command, and the underlying function. This was the path that caused
the exec() deadlocks.
- upgrade_sled gateway MCP tool: second entry point to sled upgrades,
defaulted to serving the gateway's own macOS binary to Linux sleds.
- GET /api/huskies-binary (both sled and gateway route trees): served
current_exe(), wrong platform when the gateway is macOS. Superseded
by /api/artifacts/ which now also serves on the gateway route tree.
- `huskies upgrade` CLI subcommand and --source flag: third way of
doing the same download-and-replace. Escape hatch for a bricked sled
is `docker cp` + restart.
Kept, distinct jobs: `project-rebuild` (container/image updates),
`rebuild gateway` + script/local-release (gateway self-update).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
Superseded: the running fleet was created by `new project` (no /app
mount), and the one-way upgrade path now replaces the binary at its
canonical install location (/opt/huskies/bin/huskies) directly, so the
entrypoint needs no fallback logic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
Finds the registered project carrying the huskies source tree, runs
`cargo build --release` inside its container (dedicated
CARGO_TARGET_DIR=target/sled-release so container builds stop
clobbering host target/release), then atomically publishes the binary
to ~/.huskies/artifacts/ with a .hash sidecar for convergence checks.
Full fleet redeploy is now chat-only: `release` then `upgrade all` —
no laptop access needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
- `upgrade all` sweeps every registered sled in sequence, streaming
per-sled phase markers and reporting a summary.
- Binary source is now the gateway's own artifact store via
host.docker.internal (was: unresolvable `gateway` hostname serving
the gateway's macOS binary to Linux sleds — would have bricked them).
- Sleds with active claude processes are skipped, never killed.
- After reconnect, /api/version git_hash is compared against the
published artifact's .hash sidecar; divergence is reported loudly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
GET /api/artifacts/:filename with filename validation (no path
components, no dotfiles). Sleds only ever download binaries from their
own gateway; this endpoint is where the gateway serves them from,
replacing the current_exe()-based /api/huskies-binary which serves the
gateway's own (macOS) binary — wrong platform for Linux sleds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
/opt/huskies/bin/huskies (chowned to the huskies user) with a symlink
from /usr/local/bin/huskies. Atomic replace needs write permission on
the directory for the tmp-write + rename, which root-owned
/usr/local/bin can't provide to the server process.
resolve_target_path() now prefers /opt/huskies/bin/huskies over
current_exe(), which can point at a stale location (e.g.
/workspace/target/release/huskies after a historical in-container
rebuild) that the entrypoint would never launch after a restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
CARGO_MANIFEST_DIR is baked at image build time as /app/server, but
project containers (Dockerfile.base) don't copy /app — the source is
bind-mounted at /workspace instead. Fall back to project_root when
the compile-time path doesn't exist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1145 narrowed local-release to install-only (binary + codesign-heal
wrapper) and removed the cargo build + gateway hot-restart steps that
the script used to do. That broke the "rebuild the gateway" muscle
memory: running script/local-release no longer rebuilt or restarted
anything, just re-installed the same binary.
Restore the build + restart logic while keeping 1145's wrapper:
- `cargo build --release --bin huskies` before install
- Snapshot the prior binary to ~/bin/huskies-bin.prev for rollback
- Print PREV → NEW version delta after install
- Detect a running `huskies .*--gateway` process and SSH-safe-restart
it (kill descendants depth-first, then nohup the wrapper from the
detached subshell)
- Wait up to 10s for the new gateway PID to appear; on timeout, roll
back to the previous binary and try to relaunch it
- Refuse to restart when more than one --gateway process matches, so
we don't kill the wrong tree
- `--skip-check` bypasses script/check for already-verified changes
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TimmyandClaude Fable 5
Huskies Agent