Drop the drain check from upgrade — agent death is routine

Agents die all the time; the pipeline's retry machinery re-queues
their work. Skipping busy sleds just created version skew and manual
retries for no real protection. `upgrade all` now sweeps every sled
unconditionally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
This commit is contained in:
Timmy
2026-07-15 16:54:55 +01:00
co-authored by Claude Fable 5
parent f39c4b7c4b
commit 01b24ff2ae
2 changed files with 6 additions and 17 deletions
@@ -343,7 +343,7 @@ async fn wait_for_drain(container_name: &str, timeout_secs: u64) -> Option<Strin
///
/// Uses `docker exec <name> pgrep -f claude` — exits 0 with PID list when found,
/// exits 1 when no matches (treated as 0 active processes).
pub(crate) async fn count_active_claude_processes(container_name: &str) -> Result<usize, String> {
async fn count_active_claude_processes(container_name: &str) -> Result<usize, String> {
let out = tokio::process::Command::new("docker")
.args(["exec", container_name, "pgrep", "-f", "claude"])
.output()
@@ -7,8 +7,9 @@
//!
//! The binary comes from the gateway's own artifact store
//! (`~/.huskies/artifacts/`, published by the `release` command) — sleds never
//! download from anywhere but their gateway. Sleds with active agent
//! processes are skipped so an upgrade never kills in-flight work.
//! download from anywhere but their gateway. Agents running in a sled are
//! killed by the restart; the pipeline's retry machinery picks the work up
//! again, same as any other agent death.
//!
//! The gateway orchestrates each upgrade in four phases, streaming a marker to
//! the chat room at each step:
@@ -152,7 +153,6 @@ fn resolve_artifact_source(gateway_port: Option<u16>) -> Result<(String, Option<
/// Upgrade every registered sled in sequence, streaming per-sled phase markers.
///
/// Sleds with active agent processes are skipped (reported, not failed).
/// Returns a summary listing the outcome for each sled.
pub async fn handle_upgrade_all<F, Fut>(
projects_store: &Arc<RwLock<BTreeMap<String, ProjectEntry>>>,
@@ -199,8 +199,8 @@ where
/// begins. On any failure, an error message is returned and the previous
/// binary remains active on the sled.
///
/// Sleds with active agent processes are skipped so an upgrade never kills
/// in-flight work.
/// Agents running in the sled are killed by the restart; the pipeline's
/// retry machinery re-queues their work.
pub async fn handle_sled_upgrade<F, Fut>(
project: &str,
projects_store: &Arc<RwLock<BTreeMap<String, ProjectEntry>>>,
@@ -236,17 +236,6 @@ where
Err(e) => return e,
};
// ── Drain check ──────────────────────────────────────────────────────────
// Never kill in-flight agent work; the caller can retry once idle.
let container_name = format!("huskies-{project}");
if let Ok(n) = super::project_rebuild::count_active_claude_processes(&container_name).await
&& n > 0
{
return format!(
"skipped — {n} active agent process(es) in `{container_name}`. Retry when idle."
);
}
run_sled_upgrade(project, &sled_url, &source_url, expected_hash, send_phase).await
}