huskies/README.md

# Huskies

A story-driven development server that manages work items, spawns coding agents, and runs them through a pipeline from backlog to done. Ships as a single Rust binary with an embedded React frontend.

## Getting started with Claude Code

1. Download the huskies binary (or build from source — see below).

2. From your project directory, scaffold and start the server:

```bash
huskies init --port 3000
```

This creates a `.huskies/` directory with the pipeline structure, `project.toml`, and `.mcp.json`. The `.mcp.json` file lets Claude Code discover huskies' MCP tools automatically.

Huskies also ships an embedded React frontend. Once the server is running, open `http://localhost:3000` to see the pipeline board, agent status, and chat interface.

3. Open a Claude Code session in the same project directory, or visit http://localhost:3000/. 

4. Tell Claude: "help me set up this project with huskies." Claude will walk you through the setup wizard — generating project context, tech stack docs, and test/release scripts. Review each step and confirm or ask to retry.

Once setup is complete, Claude can create stories, start agents, check status, and manage the full pipeline via MCP tools — no commands to memorize.

## Chat transports

Huskies can be controlled via bot commands in **Matrix**, **WhatsApp**, and **Slack**. Configure a transport in `.huskies/bot.toml` — see the example files:

- `.huskies/bot.toml.matrix.example`
- `.huskies/bot.toml.whatsapp-meta.example`
- `.huskies/bot.toml.whatsapp-twilio.example`
- `.huskies/bot.toml.slack.example`

## Prerequisites for building

- Rust (2024 edition)
- Node.js and npm
- Docker (for Linux cross-compilation and container deployment)
- `cross` (`cargo install cross`) optional, for Linux static builds. Only needed if you are building for a different architecture, e.g. if you want to build a Linux binary from a Mac.

You only need these installed if you want to build Huskies yourself. Alternately, you can just download and run the `huskies` binary for your system from https://code.crashlabs.io/crashlabs/huskies/releases

## Building for production

```bash
cargo build --release
```

The release binary embeds the frontend via `rust-embed`. Output: `target/release/huskies`.

For a static Linux binary (musl, zero dynamic deps):

```bash
cross build --release --target x86_64-unknown-linux-musl
```

Docker:

```bash
script/docker_rebuild

# or 

script/docker_restart
```

## Running in development

```bash
# Run tests
script/test

# Run the server
cargo run -- --port 3000

# In another terminal, run the frontend dev server
cd frontend && npm install && npm run dev
```

Configuration lives in `.huskies/project.toml`. See `.huskies/bot.toml.*.example` for transport setup.

## Releasing

Requires a Gitea API token in `.env` (`GITEA_TOKEN=your_token`).

```bash
script/release 0.7.1
```

This bumps version in `Cargo.toml` and `package.json`, builds macOS arm64 and Linux amd64 binaries, tags the repo, and publishes a Gitea release with changelog and binaries attached.

## Multi-node CRDT sync (rendezvous)

Huskies nodes can replicate pipeline state in real-time over WebSocket.  Add a
`rendezvous` field to `.huskies/project.toml` to configure a peer:

```toml
rendezvous = "ws://other-host:3001/crdt-sync"
```

On startup, this node opens an outbound WebSocket connection to the configured
URL and exchanges CRDT ops bidirectionally.  The connection is fully symmetric:
both sides send a bulk state dump on connect, then stream individual ops as they
are applied locally.

### Reconnect behaviour

If the peer is unreachable on startup (or the connection drops mid-session), the
client retries with exponential backoff starting at 1 s and capping at 30 s.
Failures are logged at **WARN**; after 10 consecutive failures the level escalates
to **ERROR** to surface persistent connectivity problems.

### Deployment topologies

**Peer-to-peer (two nodes pointing at each other):**

```
Node A  ←→  Node B
```

Configure each node with the other's `/crdt-sync` URL.  Both nodes exchange ops
directly.  Supported by this story — ops propagate in both directions and both
nodes converge to the same state.  Works well for two machines collaborating on
the same project.

**Hub-and-spoke (many clients → one central rendezvous):**

```
Client 1 ──┐
Client 2 ──┤── Hub node
Client 3 ──┘
```

Point multiple client nodes at a single "hub" node.  The hub receives ops from
all clients and re-broadcasts them.  Clients do *not* connect to each other —
convergence is mediated through the hub.  The hub itself runs a normal huskies
instance with `rendezvous` unset (it only accepts inbound connections).

> **Caveat:** Hub-to-client relay depends on the hub's `/crdt-sync` inbound
> WebSocket handler re-broadcasting every received op to all other connected
> peers.  That broadcast happens automatically via the shared `SYNC_TX` channel
> (each locally-applied remote op is re-emitted), so hub-and-spoke works today
> but has not been load-tested.  Follow-up work may be needed for large fan-out
> (many spoke clients) to avoid broadcast-channel lag.

## Startup reconcile pass

On startup, after CRDT replay and database initialisation, huskies runs a
**reconcile pass** that compares pipeline state across three sources:

1. **In-memory CRDT** — the primary source of truth, reconstructed from
   `crdt_ops` on startup.
2. **`pipeline_items` table** — a shadow/materialised view written alongside
   CRDT updates, used for fast DB queries.
3. **Filesystem shadows** (`.huskies/work/N_stage/*.md`) — legacy rendering
   still written by some paths and read by agent worktrees.

Any disagreement between these sources is **drift**. The reconcile pass logs a
structured line for each drifted item:

```
[reconcile] DRIFT story=X crdt_stage=Y db_stage=Z fs_stage=W
```

(`MISSING` is used where a source has no record for that story.)

### Drift types

| Type | Meaning |
|------|---------|
| `CRDT-only` | Story present in CRDT but absent from `pipeline_items` |
| `DB-only` | Story present in `pipeline_items` but absent from CRDT |
| `FS-only` | Story on the filesystem but absent from both CRDT and DB |
| `stage-mismatch` | Story present in both CRDT and DB but with different stage values |

Note: a filesystem shadow that lags behind the CRDT/DB stage (both of which
agree) is **not** treated as drift — the FS is a best-effort rendering and is
allowed to lag.

If any drift is detected, the Matrix/Slack/WhatsApp bot startup announcement
includes a count and a suggestion to check the server logs.

### Opt-out

Set `reconcile_on_startup = false` in `.huskies/project.toml` to disable the
pass during the migration window if it produces noise.
## Debugging

### Inspecting the in-memory CRDT state

When diagnosing state issues, use the `dump_crdt` MCP tool or the `/debug/crdt` HTTP endpoint to inspect the raw in-memory CRDT state directly. These surfaces show the ground truth that the running server holds — not a summarised pipeline view and not the persisted SQLite ops.

**MCP tool** (from Claude Code or any MCP client):

```
mcp__huskies__dump_crdt
# dump everything
{}

# restrict to a single item
{"story_id": "42_story_my_feature"}
```

**HTTP endpoint** (browser or curl):

```bash
# dump everything
curl http://localhost:3001/debug/crdt

# restrict to a single item
curl "http://localhost:3001/debug/crdt?story_id=42_story_my_feature"
```

Both return a JSON document with:

- **`metadata`** — `in_memory_state_loaded`, `total_items`, `total_ops_in_list`, `max_seq_in_list`, `persisted_ops_count`, `pending_persist_ops_count`
- **`items`** — one entry per CRDT list item (including tombstoned/deleted entries), each with `story_id`, `stage`, `name`, `agent`, `retry_count`, `blocked`, `depends_on`, `content_index` (hex OpId for cross-referencing with `crdt_ops`), and `is_deleted`

> **This is a debug tool.** For normal pipeline introspection use `get_pipeline_status` or `GET /api/pipeline` instead.

## Source Map

### Core

| File | Description |
|------|-------------|
| `server/src/main.rs` | Entry point, CLI argument parsing, and server startup |
| `server/src/config.rs` | Parses `project.toml` for agents, components, and server settings |
| `server/src/state.rs` | Global mutable session state (project root, cancellation) |
| `server/src/store.rs` | JSON-backed persistent key-value store for settings |

### Agents

| File | Description |
|------|-------------|
| `server/src/agents/mod.rs` | Types, configuration, and orchestration for coding agents |
| `server/src/agents/gates.rs` | Runs test suites and validation scripts in agent worktrees |
| `server/src/agents/lifecycle.rs` | File creation, archival, and stage transitions for pipeline items |
| `server/src/agents/merge.rs` | Rebases agent work onto master and runs post-merge validation |
| `server/src/agents/pty.rs` | Spawns agent processes in pseudo-terminals and streams output |
| `server/src/agents/token_usage.rs` | Persists per-agent token consumption records to disk |
| `server/src/agent_log.rs` | Reads and writes JSONL agent event logs to disk |
| `server/src/agent_mode.rs` | Headless build-agent mode for distributed story processing |

### Agent Pool

| File | Description |
|------|-------------|
| `server/src/agents/pool/mod.rs` | Manages the set of active agents across all pipeline stages |
| `server/src/agents/pool/types.rs` | `AgentPool`, `StoryAgent`, and related data structures |
| `server/src/agents/pool/start.rs` | Spawns a new agent process in a worktree for a story |
| `server/src/agents/pool/stop.rs` | Terminates a running agent while preserving its worktree |
| `server/src/agents/pool/wait.rs` | Blocks until an agent reaches a terminal state |
| `server/src/agents/pool/query.rs` | Lists available/active agents and info lookups |
| `server/src/agents/pool/process.rs` | Kills orphaned PTY child processes on shutdown |
| `server/src/agents/pool/worktree.rs` | Creates and configures git worktrees for agents |
| `server/src/agents/pool/test_helpers.rs` | In-memory pool construction and test assertions |

### Agent Pool — Auto-assign

| File | Description |
|------|-------------|
| `server/src/agents/pool/auto_assign/mod.rs` | Wires sub-files and re-exports public items |
| `server/src/agents/pool/auto_assign/auto_assign.rs` | Scans pipeline stages and dispatches agents to unassigned stories |
| `server/src/agents/pool/auto_assign/reconcile.rs` | Startup reconciliation: detects committed work and advances pipeline |
| `server/src/agents/pool/auto_assign/scan.rs` | Scans pipeline stages for work items and queries pool state |
| `server/src/agents/pool/auto_assign/story_checks.rs` | Front-matter checks: review holds, blocked state, merge failures |
| `server/src/agents/pool/auto_assign/watchdog.rs` | Detects orphaned agents and triggers auto-assign |

### Agent Pool — Pipeline

| File | Description |
|------|-------------|
| `server/src/agents/pool/pipeline/mod.rs` | Stage advancement, completion handling, and merge orchestration |
| `server/src/agents/pool/pipeline/advance.rs` | Moves stories forward through pipeline stages |
| `server/src/agents/pool/pipeline/completion.rs` | Processes exit results and triggers pipeline advancement |
| `server/src/agents/pool/pipeline/merge.rs` | Orchestrates the merge-to-master flow for completed stories |

### Agent Runtimes

| File | Description |
|------|-------------|
| `server/src/agents/runtime/mod.rs` | Pluggable backends (Claude Code, Gemini, OpenAI) for running agents |
| `server/src/agents/runtime/claude_code.rs` | Launches Claude Code CLI sessions as agent backends |
| `server/src/agents/runtime/gemini.rs` | Drives Google Gemini API sessions as agent backends |
| `server/src/agents/runtime/openai.rs` | Drives OpenAI API sessions as agent backends |

### CRDT

| File | Description |
|------|-------------|
| `server/src/crdt_state.rs` | Pipeline state as a conflict-free replicated document backed by SQLite |
| `server/src/crdt_sync.rs` | WebSocket-based replication of pipeline state between nodes |
| `server/src/crdt_wire.rs` | Serialization format for `SignedOp` sync messages |
| `server/src/pipeline_state.rs` | Typed pipeline state machine |

### Database

| File | Description |
|------|-------------|
| `server/src/db/mod.rs` | Content store, shadow writes, and CRDT op persistence |

### HTTP Server

| File | Description |
|------|-------------|
| `server/src/http/mod.rs` | Module declarations for all REST, MCP, WebSocket, and SSE endpoints |
| `server/src/http/context.rs` | Shared `AppContext` threaded through all HTTP handlers |
| `server/src/http/agents.rs` | REST API for listing, starting, stopping, and inspecting agents |
| `server/src/http/agents_sse.rs` | Server-Sent Events endpoint for real-time agent output |
| `server/src/http/anthropic.rs` | Proxy for model listing and key-validation to Anthropic |
| `server/src/http/assets.rs` | Serves the embedded React frontend via `rust-embed` |
| `server/src/http/bot_command.rs` | Bot command HTTP endpoint |
| `server/src/http/chat.rs` | REST API for the LLM-powered chat interface |
| `server/src/http/health.rs` | Returns a static "ok" response |
| `server/src/http/io.rs` | REST API for file and directory operations |
| `server/src/http/model.rs` | REST API for model selection and LLM provider management |
| `server/src/http/oauth.rs` | Anthropic OAuth callback and token exchange flow |
| `server/src/http/project.rs` | REST API for project initialization and context management |
| `server/src/http/settings.rs` | REST API for user preferences and editor configuration |
| `server/src/http/wizard.rs` | REST API for the project setup wizard |
| `server/src/http/ws.rs` | Real-time pipeline updates, chat, and permission prompts |
| `server/src/http/test_helpers.rs` | Shared test utilities for HTTP handler tests |

### HTTP — MCP Tools

| File | Description |
|------|-------------|
| `server/src/http/mcp/mod.rs` | Model Context Protocol endpoint dispatching tool calls |
| `server/src/http/mcp/agent_tools.rs` | Start, stop, wait, list, and inspect agents via MCP |
| `server/src/http/mcp/diagnostics.rs` | Server logs, CRDT dump, and story movement helpers |
| `server/src/http/mcp/git_tools.rs` | Status, diff, add, commit, and log on agent worktrees |
| `server/src/http/mcp/merge_tools.rs` | Merge agent work to master and report failures |
| `server/src/http/mcp/qa_tools.rs` | Request, approve, and reject QA reviews |
| `server/src/http/mcp/shell_tools.rs` | Run commands, execute tests, and stream output |
| `server/src/http/mcp/status_tools.rs` | Pipeline status, story triage, and AC inspection |
| `server/src/http/mcp/story_tools.rs` | Create, update, move, and manage stories/bugs/refactors |
| `server/src/http/mcp/wizard_tools.rs` | Interactive setup wizard tool implementations |

### HTTP — Workflow

| File | Description |
|------|-------------|
| `server/src/http/workflow/mod.rs` | Shared story/bug file operations for HTTP and MCP handlers |
| `server/src/http/workflow/bug_ops.rs` | Creates bug, refactor, and spike files in the pipeline |
| `server/src/http/workflow/story_ops.rs` | Creates, updates, and manages acceptance criteria in stories |
| `server/src/http/workflow/test_results.rs` | Writes structured test results into story markdown |

### I/O

| File | Description |
|------|-------------|
| `server/src/io/mod.rs` | Filesystem, shell, search, onboarding, and story metadata operations |
| `server/src/io/fs/mod.rs` | Module declarations and re-exports for file operations |
| `server/src/io/fs/files.rs` | Read, write, list, and create files and directories |
| `server/src/io/fs/paths.rs` | Resolves CLI and session-relative paths to absolute paths |
| `server/src/io/fs/preferences.rs` | Reads and writes model selection and user settings |
| `server/src/io/fs/project.rs` | Tracks known projects and resolves the active project root |
| `server/src/io/fs/scaffold.rs` | Creates the `.huskies/` directory structure and default files |
| `server/src/io/onboarding.rs` | Checks whether scaffold templates have been customized |
| `server/src/io/search.rs` | Full-text search across project files |
| `server/src/io/shell.rs` | Runs commands in the project directory and captures output |
| `server/src/io/story_metadata.rs` | Parses and modifies YAML front matter in story markdown |
| `server/src/io/watcher.rs` | Filesystem watcher for `.huskies/work/` and `project.toml` |
| `server/src/io/wizard.rs` | Multi-step project onboarding flow with per-step status |
| `server/src/io/test_helpers.rs` | Shared test utilities for I/O module tests |

### Chat

| File | Description |
|------|-------------|
| `server/src/chat/mod.rs` | Transport abstraction for chat platforms |
| `server/src/chat/lookup.rs` | Shared story-lookup helper for chat commands |
| `server/src/chat/timer.rs` | Deferred agent start via one-shot timers |
| `server/src/chat/util.rs` | Shared text utilities used by all transports |
| `server/src/chat/test_helpers.rs` | Shared test utilities for chat handler tests |

### Chat — Commands

| File | Description |
|------|-------------|
| `server/src/chat/commands/mod.rs` | Bot-level command registry shared by all transports |
| `server/src/chat/commands/ambient.rs` | `ambient` command handler |
| `server/src/chat/commands/assign.rs` | `assign` command handler |
| `server/src/chat/commands/cost.rs` | `cost` command handler |
| `server/src/chat/commands/coverage.rs` | `coverage` command — show or refresh test coverage |
| `server/src/chat/commands/depends.rs` | `depends` command handler |
| `server/src/chat/commands/git.rs` | `git` command handler |
| `server/src/chat/commands/help.rs` | `help` command handler |
| `server/src/chat/commands/loc.rs` | `loc` command — top source files by line count |
| `server/src/chat/commands/move_story.rs` | `move` command handler |
| `server/src/chat/commands/overview.rs` | `overview` command handler |
| `server/src/chat/commands/run_tests.rs` | `test` command — run the project's test suite |
| `server/src/chat/commands/setup.rs` | `setup` command handler |
| `server/src/chat/commands/show.rs` | `show` command handler |
| `server/src/chat/commands/status.rs` | `status` command and pipeline status helpers |
| `server/src/chat/commands/timer.rs` | `timer` command handler |
| `server/src/chat/commands/triage.rs` | Story triage dump subcommand of `status` |
| `server/src/chat/commands/unblock.rs` | `unblock` command handler |
| `server/src/chat/commands/unreleased.rs` | `unreleased` command handler |

### Chat — Matrix Transport

| File | Description |
|------|-------------|
| `server/src/chat/transport/matrix/mod.rs` | Matrix bot integration |
| `server/src/chat/transport/matrix/config.rs` | Deserialization of `bot.toml` Matrix settings |
| `server/src/chat/transport/matrix/commands.rs` | Re-exports from `crate::chat::commands` |
| `server/src/chat/transport/matrix/transport_impl.rs` | Matrix `ChatTransport` implementation |
| `server/src/chat/transport/matrix/assign.rs` | Assign/re-assign a coder model to a story |
| `server/src/chat/transport/matrix/delete.rs` | Delete a story/bug/spike from the pipeline |
| `server/src/chat/transport/matrix/htop.rs` | Live-updating system and agent process dashboard |
| `server/src/chat/transport/matrix/notifications.rs` | Stage transition notifications for Matrix rooms |
| `server/src/chat/transport/matrix/rebuild.rs` | Trigger a server rebuild and restart |
| `server/src/chat/transport/matrix/reset.rs` | Clear the current Claude Code session for a room |
| `server/src/chat/transport/matrix/rmtree.rs` | Delete the worktree for a story |
| `server/src/chat/transport/matrix/start.rs` | Start a coder agent on a story |

### Chat — Matrix Bot

| File | Description |
|------|-------------|
| `server/src/chat/transport/matrix/bot/mod.rs` | Sub-modules for the Matrix chat bot |
| `server/src/chat/transport/matrix/bot/context.rs` | Shared state (rooms, history, permissions) |
| `server/src/chat/transport/matrix/bot/format.rs` | Markdown-to-HTML conversion and startup announcements |
| `server/src/chat/transport/matrix/bot/history.rs` | Per-room message history for LLM context |
| `server/src/chat/transport/matrix/bot/mentions.rs` | Checks whether a message mentions the bot |
| `server/src/chat/transport/matrix/bot/messages.rs` | Processes incoming messages and dispatches commands |
| `server/src/chat/transport/matrix/bot/run.rs` | Connects to homeserver and processes sync events |
| `server/src/chat/transport/matrix/bot/verification.rs` | Interactive emoji verification flow for E2EE |

### Chat — Slack Transport

| File | Description |
|------|-------------|
| `server/src/chat/transport/slack/mod.rs` | Slack Bot API integration |
| `server/src/chat/transport/slack/commands.rs` | Incoming message dispatch and slash command handling |
| `server/src/chat/transport/slack/format.rs` | Markdown to Slack mrkdwn conversion |
| `server/src/chat/transport/slack/history.rs` | Conversation history persistence |
| `server/src/chat/transport/slack/meta.rs` | `ChatTransport` implementation for Slack |
| `server/src/chat/transport/slack/verify.rs` | Request signature verification |

### Chat — Discord Transport

| File | Description |
|------|-------------|
| `server/src/chat/transport/discord/mod.rs` | Discord Bot integration |
| `server/src/chat/transport/discord/commands.rs` | Incoming message dispatch and command handling |
| `server/src/chat/transport/discord/format.rs` | Markdown to Discord format conversion |
| `server/src/chat/transport/discord/gateway.rs` | Minimal Discord Gateway WebSocket client |
| `server/src/chat/transport/discord/history.rs` | Conversation history persistence |
| `server/src/chat/transport/discord/meta.rs` | `ChatTransport` implementation for Discord |

### Chat — WhatsApp Transport

| File | Description |
|------|-------------|
| `server/src/chat/transport/whatsapp/mod.rs` | WhatsApp Business API integration |
| `server/src/chat/transport/whatsapp/commands.rs` | Processes incoming messages as bot commands |
| `server/src/chat/transport/whatsapp/format.rs` | Markdown-to-WhatsApp conversion and message chunking |
| `server/src/chat/transport/whatsapp/history.rs` | Per-number history and messaging window tracking |
| `server/src/chat/transport/whatsapp/meta.rs` | Meta Cloud API transport via Graph API |
| `server/src/chat/transport/whatsapp/twilio.rs` | Twilio transport for sending/receiving messages |

### Chat — Transport Abstraction

| File | Description |
|------|-------------|
| `server/src/chat/transport/mod.rs` | Pluggable backends (Matrix, Slack, WhatsApp, Discord) |

### LLM

| File | Description |
|------|-------------|
| `server/src/llm/mod.rs` | Chat orchestration, prompts, OAuth, and provider integrations |
| `server/src/llm/chat.rs` | Multi-turn conversations with tool-calling LLM providers |
| `server/src/llm/oauth.rs` | Token refresh and credential management for Claude API |
| `server/src/llm/prompts.rs` | Static prompt templates for chat and onboarding |
| `server/src/llm/types.rs` | `Message`, `Role`, `ToolCall`, `ModelProvider` types |

### LLM — Providers

| File | Description |
|------|-------------|
| `server/src/llm/providers/mod.rs` | Module declarations for Anthropic, Claude Code, and Ollama |
| `server/src/llm/providers/anthropic.rs` | Streaming completion client for Claude Messages API |
| `server/src/llm/providers/claude_code.rs` | Runs Claude Code CLI in a PTY and parses output |
| `server/src/llm/providers/ollama.rs` | Streaming completion client for Ollama models |

### Utilities

| File | Description |
|------|-------------|
| `server/src/log_buffer.rs` | Bounded in-memory ring buffer for server log output |
| `server/src/rebuild.rs` | Server rebuild and restart logic |
| `server/src/workflow.rs` | Test result tracking and acceptance evaluation |
| `server/src/worktree.rs` | Creates, lists, and removes git worktrees for agent isolation |

## License

GPL-3.0. See [LICENSE](LICENSE).