Drop the drain check from upgrade — agent death is routine
Agents die all the time; the pipeline's retry machinery re-queues their work. Skipping busy sleds just created version skew and manual retries for no real protection. `upgrade all` now sweeps every sled unconditionally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019fHdm92yjvguPi2LiXfLB9
This commit is contained in:
@@ -343,7 +343,7 @@ async fn wait_for_drain(container_name: &str, timeout_secs: u64) -> Option<Strin
|
||||
///
|
||||
/// Uses `docker exec <name> pgrep -f claude` — exits 0 with PID list when found,
|
||||
/// exits 1 when no matches (treated as 0 active processes).
|
||||
pub(crate) async fn count_active_claude_processes(container_name: &str) -> Result<usize, String> {
|
||||
async fn count_active_claude_processes(container_name: &str) -> Result<usize, String> {
|
||||
let out = tokio::process::Command::new("docker")
|
||||
.args(["exec", container_name, "pgrep", "-f", "claude"])
|
||||
.output()
|
||||
|
||||
@@ -7,8 +7,9 @@
|
||||
//!
|
||||
//! The binary comes from the gateway's own artifact store
|
||||
//! (`~/.huskies/artifacts/`, published by the `release` command) — sleds never
|
||||
//! download from anywhere but their gateway. Sleds with active agent
|
||||
//! processes are skipped so an upgrade never kills in-flight work.
|
||||
//! download from anywhere but their gateway. Agents running in a sled are
|
||||
//! killed by the restart; the pipeline's retry machinery picks the work up
|
||||
//! again, same as any other agent death.
|
||||
//!
|
||||
//! The gateway orchestrates each upgrade in four phases, streaming a marker to
|
||||
//! the chat room at each step:
|
||||
@@ -152,7 +153,6 @@ fn resolve_artifact_source(gateway_port: Option<u16>) -> Result<(String, Option<
|
||||
|
||||
/// Upgrade every registered sled in sequence, streaming per-sled phase markers.
|
||||
///
|
||||
/// Sleds with active agent processes are skipped (reported, not failed).
|
||||
/// Returns a summary listing the outcome for each sled.
|
||||
pub async fn handle_upgrade_all<F, Fut>(
|
||||
projects_store: &Arc<RwLock<BTreeMap<String, ProjectEntry>>>,
|
||||
@@ -199,8 +199,8 @@ where
|
||||
/// begins. On any failure, an error message is returned and the previous
|
||||
/// binary remains active on the sled.
|
||||
///
|
||||
/// Sleds with active agent processes are skipped so an upgrade never kills
|
||||
/// in-flight work.
|
||||
/// Agents running in the sled are killed by the restart; the pipeline's
|
||||
/// retry machinery re-queues their work.
|
||||
pub async fn handle_sled_upgrade<F, Fut>(
|
||||
project: &str,
|
||||
projects_store: &Arc<RwLock<BTreeMap<String, ProjectEntry>>>,
|
||||
@@ -236,17 +236,6 @@ where
|
||||
Err(e) => return e,
|
||||
};
|
||||
|
||||
// ── Drain check ──────────────────────────────────────────────────────────
|
||||
// Never kill in-flight agent work; the caller can retry once idle.
|
||||
let container_name = format!("huskies-{project}");
|
||||
if let Ok(n) = super::project_rebuild::count_active_claude_processes(&container_name).await
|
||||
&& n > 0
|
||||
{
|
||||
return format!(
|
||||
"skipped — {n} active agent process(es) in `{container_name}`. Retry when idle."
|
||||
);
|
||||
}
|
||||
|
||||
run_sled_upgrade(project, &sled_url, &source_url, expected_hash, send_phase).await
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user