huskies/.huskies/specs/tech/SPIKE_FLY_MULTITENANT.md

# Spike 811: Fly.io Machines API Integration for Multi-Tenant Huskies SaaS

## Goal

Investigate how to operate huskies as a hosted multi-tenant SaaS on
[Fly.io Machines](https://fly.io/docs/machines/). Each tenant owns one or
more huskies *project* containers; a fronting gateway routes traffic by
tenant and provisions/destroys backing machines on demand. This document
captures the architecture, the API surface we need, and the operational
concerns that need answers before we start writing production code.

## Architecture at a Glance

```
┌──────────────────────┐        ┌───────────────────────────────────────────┐
│ Browser / CLI / Bot  │───────▶│ huskies-gateway  (Fly app: huskies-gw)    │
└──────────────────────┘  HTTPS │   * authenticates tenant                  │
                                │   * picks active project for tenant       │
                                │   * proxies /mcp /ws /api to machine      │
                                │   * provisions machines via Machines API  │
                                └──────────────────┬────────────────────────┘
                                                   │ .flycast (Wireguard)
                                                   ▼
                          ┌────────────────────────────────────────────────┐
                          │ huskies-project-{tenant}-{project}             │
                          │   (Fly app: huskies-projects, machine per tier)│
                          │   * runs `huskies --port 3001 /data/project`   │
                          │   * persistent volume mounted at /data         │
                          │   * .huskies/ + sled CRDT live on volume       │
                          └────────────────────────────────────────────────┘
```

Two Fly apps:

* `huskies-gw` — small, always-on, replicated across regions; runs the
  existing `huskies --gateway` binary plus a thin **Fly orchestrator**
  layer that calls the Machines API.
* `huskies-projects` — single Fly app holding *one machine per tenant
  project*. Using one app (rather than one app per tenant) keeps quota
  management, IAM, and image distribution simple while still giving us
  per-machine networking (`{machine_id}.vm.huskies-projects.internal`)
  and per-tenant Fly volumes.

## Listed Concerns

The story brief flags the following concerns. Each is addressed below.

1. Machine lifecycle & API surface
2. Tenant isolation
3. Persistence and volumes
4. Networking & routing
5. Secrets and tenant credentials
6. Cost model and idle-shutdown
7. Wake-on-request / cold-start latency
8. Observability and logs
9. Disaster recovery and backups
10. Quotas and abuse limits

---

### 1. Machine Lifecycle & API Surface

Fly Machines is a REST API at `https://api.machines.dev/v1`. Auth is a
single bearer token per Fly organization (`FLY_API_TOKEN`).

Endpoints we will call:

| Verb | Path | Use |
|------|------|-----|
| `POST` | `/apps/{app}/machines` | Create a new project machine |
| `GET`  | `/apps/{app}/machines/{id}` | Poll status |
| `GET`  | `/apps/{app}/machines/{id}/wait?state=started&timeout=30` | Block until state |
| `POST` | `/apps/{app}/machines/{id}/start` | Wake a stopped machine |
| `POST` | `/apps/{app}/machines/{id}/stop` | Graceful stop (idle scale-to-zero) |
| `POST` | `/apps/{app}/machines/{id}/suspend` | Suspend RAM-to-disk (fast wake) |
| `DELETE` | `/apps/{app}/machines/{id}?force=true` | Destroy permanently |
| `GET`  | `/apps/{app}/machines` | Enumerate during reconcile |
| `POST` | `/apps/{app}/volumes` | Create persistent volume for tenant |
| `DELETE` | `/apps/{app}/volumes/{id}` | Reclaim volume when tenant deletes project |

States the orchestrator observes: `created → starting → started → stopping
→ stopped → destroying → destroyed` (`replacing` and `suspending` are
transient).

A successful provisioning sequence is:

1. `POST /volumes` (one-time per tenant project, 1 GiB default).
2. `POST /machines` with `config = { image, env, mounts: [{volume, path:"/data"}], guest, services }`.
3. `GET /machines/{id}/wait?state=started` (~10–20 s on cold start).
4. Cache `{tenant, project} → machine_id` in the gateway CRDT
   (`gateway_projects` LWW-map already exists — extend the value with
   `machine_id`, `volume_id`, `last_used_at`).

Destruction:

1. `POST /machines/{id}/stop` (graceful, lets sled flush).
2. `DELETE /machines/{id}?force=true`.
3. Optionally `DELETE /volumes/{id}` (only when tenant explicitly deletes
   the project; idle stop must **never** delete volumes).

### 2. Tenant Isolation

* **Filesystem:** each machine has its own ephemeral root and its own
  Fly volume mounted at `/data`. Volumes are not shareable across
  machines, so tenants cannot read each other's CRDT.
* **Network:** machines on the same Fly app can reach each other via
  6PN private networking. We must explicitly *not* expose the project
  server externally; only the gateway holds a public IP. Project
  machines bind to `[::]:3001` and rely on `.flycast` private routing.
* **Credentials:** project machines never see the gateway's
  `FLY_API_TOKEN`. Tenant-supplied secrets (Anthropic key, Matrix
  password, etc.) are stored as Fly secrets *scoped to the machine* via
  the `secrets` field at create time, encrypted at rest by Fly.
* **CPU/RAM:** `guest = { cpu_kind: "shared", cpus: 2, memory_mb: 2048 }`
  is a sensible default; larger tenants get `performance` cpus. Hard
  caps prevent a runaway agent from eating a neighbour's quota.

### 3. Persistence and Volumes

* Fly volumes are zone-pinned. We pick the volume region from the
  tenant's primary region (`PRIMARY_REGION` env on the gateway), with
  fallback to `iad`.
* The volume holds:
  * `/data/project/.huskies/` — pipeline.db (sled), bot.toml, project.toml
  * `/data/project/.git` — repository (initially cloned at first run)
  * `/data/project/` — working tree
* Sled needs a clean shutdown. The orchestrator must always `stop`
  before `destroy`. We rely on Fly's `kill_signal = "SIGTERM"` + the
  existing huskies shutdown path in `rebuild.rs`.
* **Snapshots:** Fly snapshots volumes daily by default (5-day
  retention). For paid tiers we extend retention via `snapshot_retention`
  on the volume.

### 4. Networking & Routing

The gateway already proxies MCP/WS/REST by active project. For SaaS we
add tenant resolution **before** the project lookup:

```
Host: alice.huskies.app   →  tenant = alice
  ↓
GET /tenants/alice/projects/foo → project_id, machine_id
  ↓
proxy to fdaa:0:abcd:a7b:e2:1::3:3001  (or {machine_id}.vm.huskies-projects.internal:3001)
```

* Tenant resolution lives in a new `tenants` CRDT LWW-map keyed by
  subdomain → tenant_id; reuses the existing CRDT bus.
* Internal DNS: `<machine_id>.vm.huskies-projects.internal` resolves on
  the private network. `<app>.flycast` is the load-balanced anycast
  name; we prefer the explicit machine address since each tenant has
  exactly one project machine at a time.
* TLS terminates at the Fly edge for `*.huskies.app`. The gateway
  receives plain HTTP/2 inside 6PN.

### 5. Secrets and Tenant Credentials

* `FLY_API_TOKEN` lives only on the gateway (`fly secrets set
  FLY_API_TOKEN=… -a huskies-gw`).
* Per-tenant `ANTHROPIC_API_KEY`, `MATRIX_PASSWORD`, etc. are POSTed by
  the tenant in the SaaS UI, encrypted with the gateway's KMS key, and
  passed to the machine at create time via the Machines API
  `config.env` (Fly stores env values encrypted).
* Rotation: changing a tenant secret means `POST /machines/{id}/update`
  with the new env, which triggers a rolling replace. The orchestrator
  schedules this during the tenant's idle window when possible.

### 6. Cost Model and Idle-Shutdown

Indicative pricing (us-east, 2026):

| Machine | Hourly | Notes |
|---------|--------|-------|
| `shared-cpu-2x@2048` always-on | ~$0.027 | $19/mo if 24×7 |
| `shared-cpu-2x@2048` suspended | ~$0.0009 | $0.65/mo idle |
| Volume 1 GiB | ~$0.0002 | $0.15/mo |

Multi-tenant pricing requires **suspend on idle**:

* Auto-stop: in the machine config, set `services[].auto_stop_machines
  = "suspend"` and `services[].auto_start_machines = true`. Fly's
  internal proxy stops the machine after the configured `min_machines`
  count is zero and there is no incoming traffic for ~5 min.
* On the next request, the proxy auto-wakes the machine. Suspend resume
  is ~300 ms (RAM snapshot from disk); a full `stopped → started` is
  10–20 s. We prefer `suspend` for SaaS.
* For long-lived agents (a coder agent running on the machine), the
  gateway sends keepalive pings so Fly does not idle-stop while work is
  in progress. Implementation: gateway tracks `active_agents` count for
  each machine in CRDT; if `>0`, hit `/api/agents` once per minute.

### 7. Wake-on-Request / Cold-Start Latency

Three latency tiers:

| Tier | Wake | When |
|------|------|------|
| Suspended | ~300 ms | Default for active tenants |
| Stopped | 10–20 s | Tenants idle > 7 days |
| Destroyed | 60–90 s (clone + boot) | Free tier reaped after 30 d |

The gateway returns a `202 Accepted` with a `Retry-After: 1` header
while wake is in progress and surfaces a "warming up" splash. The
existing `huskies-gw` MCP code path needs an explicit wake call for
in-flight requests because Fly's automatic wake only triggers on TCP
SYN to a registered service port.

### 8. Observability and Logs

* `fly logs -a huskies-projects -i <machine_id>` streams stdout/stderr.
  We expose this through the gateway as `GET /api/admin/tenants/{id}/logs`.
* Each machine ships logs to the gateway via a sidecar `vector`
  process? Decision: **no** — Fly's built-in NATS log shipper is enough
  for v1; revisit if log volume grows.
* Metrics: Fly auto-exports per-machine CPU/RAM/network as Prometheus
  series scrapeable from a `huskies-metrics` machine in the same 6PN.
  We hook into Grafana Cloud's free tier for the dashboard.

### 9. Disaster Recovery and Backups

* Volume snapshots (daily) cover hardware failure.
* The CRDT replicates to the gateway over the existing `/crdt-sync`
  WebSocket. The gateway keeps a 30-day rolling backup of each tenant's
  CRDT in S3 (`s3://huskies-backups/{tenant}/{date}.ops`). This lets us
  reconstruct the project tree even if a Fly volume is unrecoverable.
* Restore flow: provision a fresh machine + volume, replay the latest
  snapshot, then replay incremental ops from S3. Documented in a
  follow-up runbook story.

### 10. Quotas and Abuse Limits

* Per-tenant: max 2 concurrent agents, max 8 GiB volume, max 4 CPU,
  max 200 OAuth-paid model dollars per month. Enforced in the gateway
  before calling the Machines API. Over-quota → `429 Too Many Requests`
  with a Stripe upsell page.
* Per-Fly-app: Fly soft-limits 1000 machines per app. At scale we
  shard tenants across `huskies-projects-{0..9}` apps using
  `consistent_hash(tenant_id)`.
* Abuse: every tenant signs up with a verified email + Stripe card.
  Free tier capped at 1 project, suspended after 7 days idle, destroyed
  after 30 days idle.

---

## Decisions

| Decision | Choice | Rejected alternative |
|----------|--------|----------------------|
| Apps topology | **Single `huskies-projects` app, one machine per tenant** | One app per tenant: clean isolation, but blows out Fly app quotas and complicates IAM |
| Idle strategy | **Suspend, not stop** | Stop: cheaper but 20 s cold start is poor UX for chat |
| Secrets path | **Machine env via Machines API at create time** | Fly app-level secrets: shared across all tenant machines, leaks across tenants |
| State storage | **Per-tenant Fly volume holding sled + git** | Object storage only: would require rewriting sled backend |
| Tenant resolution | **Subdomain → CRDT `tenants` LWW-map** | Path prefix routing: harder to issue per-tenant TLS, breaks browser cookies |
| Volume retention | **Never delete on idle stop; only on explicit project deletion** | Auto-delete after N days idle: too easy to lose user data |

## Open Questions

1. How do we hand off long-running coder agents during a Fly host
   evacuation (machine replace event)? Suspend won't survive a host
   reboot; we may need a "draining" hook that finishes the current AC
   and commits before allowing replacement.
2. Should the gateway also live as Fly machines (auto-scale) or stay
   as Fly app v1 with replicas? Probably the former for global routing,
   but that's a separate spike.
3. Billing surfaces: do we pass through Fly's per-machine cost to the
   tenant, or amortize it into a flat per-project price? Product call.
4. Outbound network egress (model API calls, git pushes) is metered by
   Fly. At Claude Opus rates, model API egress dwarfs everything else,
   so this is a rounding error — confirm at 100-tenant scale.

## Proof-of-Concept Script

A working sketch lives at
[`fly_multitenant_poc.sh`](./fly_multitenant_poc.sh). It demonstrates
end-to-end: read `FLY_API_TOKEN`, create a volume, create a machine
attached to it, wait until started, stop, and destroy. The script is
runnable but is **not** what production code looks like — production
will translate these calls into Rust against a typed `flyio_machines`
client crate, called from a new `server::service::cloud::fly`
module that the gateway invokes on tenant signup.