# LLM Context From Events Design overview for making any LLM-driven chat persona (Timmy at the gateway, Sally at a single sled, future personas) aware of huskies events without the user having to re-narrate them. ## Goal **Update the LLM's context non-intrusively when a state transition happens.** No new LLM turn is fired; events are simply visible to the LLM the next time the user (or anything else) does cause it to run. The LLM should never need to be told what already happened inside huskies. ## Guiding Principle **Transports have nothing to do with LLMs.** A transport (Matrix bot, web UI, CLI, future TUIs) is a pure courier — it relays user text in, LLM text out, and never owns LLM-facing state. Anything the LLM needs to know lives in huskies, behind a single `assemble_prompt_context` helper that the transport calls. Adding a new transport must require zero changes to the event-awareness path. ## Three things this doc is NOT 1. **Triggers** — `on StoryMerged{1122} do Rebuild`. These are deterministic subscribers; they should never invoke the LLM. Covered in a separate design. 2. **Proactive wake** — running an LLM turn *because* an event fired, without the user typing. Costs tokens, risks ramble. Explicitly out of scope here; a separate decision to make later. 3. **A transport feature** — this design assumes any transport that invokes the LLM uses the same context-assembly helper. Matrix bot, web UI, CLI all funnel through it. ## Why Past Attempts Have Failed - **Buffer lived on the transport**, not on huskies. The current `BotContext.pending_pipeline_events` (`server/src/chat/transport/matrix/bot/context.rs:103-116`) is Matrix-only; web UI users see nothing of the kind, and the buffer dies with the bot process. - **Process-local, RAM-only**. Server rebuild → buffer empty. Any events between the old binary's last user turn and the new binary's first are silently lost. - **Unbounded `mpsc` channels drop under lag.** The server logs routinely show `[xxx-sub] Subscriber lagged, skipped N event(s)`. When the subscriber feeding the buffer falls behind, events vanish without being recorded. - **No end-to-end test.** Nothing asserts "fire event E, send user message M, the LLM's prompt contains E." - **No cross-process aggregation.** Events in a sled have no path to the gateway-side LLM context without bespoke plumbing per event type. ## Architecture at a Glance ``` ┌────────────┐ ┌────────────┐ ┌────────────┐ │ Sled A │ │ Sled B │ │ Sled C │ │ event_log/ │ │ event_log/ │ │ event_log/ │ ◄── source of truth └─────┬──────┘ └─────┬──────┘ └─────┬──────┘ (CRDT-backed) │ │ │ └───────────────┼───────────────┘ ▼ ┌────────────────────┐ │ Gateway aggregator │ ◄── tail-merges all sled logs │ event_view/ │ into a single ordered stream └─────────┬──────────┘ │ ▼ ┌────────────────────────┐ │ Per-LLM-session state │ ◄── scope filter + │ sessions// │ high-water mark per stream └─────────┬──────────────┘ │ ▼ ┌────────────────────────┐ │ assemble_prompt_context│ ◄── single helper used by │ (session_id) -> Str │ every transport before └─────────┬──────────────┘ each LLM turn │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ Matrix bot Web UI CLI / TUI ``` ## Event Model — Reuse What Already Exists There is no need to invent a parallel event taxonomy. Huskies already has a complete typed enum and a single broadcast bus: - `server/src/pipeline_state/transition.rs` defines `PipelineEvent` with **30 variants** covering every state-machine transition (`DepsMet`, `GatesStarted/Passed/Failed`, `QaSkipped`, `MergeSucceeded/Failed/FailedFinal`, `Accepted`, `Block/Unblock`, `Abandon`, `Supersede`, `ReviewHold/Cleared`, `Reject`, `Triage`, `Close`, `Demote`, `Freeze/Unfreeze`, `MergemasterAttempted`, `FixupRequested`, `ReQueuedForQa`, `MergeAborted`, `HotfixRequested`, `MergeRetryStarted`). - The same module defines `ExecutionEvent` with 7 variants for agent lifecycle (`SpawnRequested`, `SpawnedSuccessfully`, `Heartbeat`, `HitRateLimit`, `Exited`, `Stopped`, `Reset`). - Every transition fires a `TransitionFired` event on a single internal bus. Ten subscribers already consume it (audit-log, worktree-create-sub, worktree-cleanup-sub, merge-failure-sub, merge-block-sub, done-archive-sub, content-gc, cost-rollup-sub, stage-notification-sub, event-triggers). **The LLM context injector is just the 11th subscriber on the same bus.** It writes typed events into the per-sled CRDT event log described below; everything downstream reuses the existing taxonomy. Each persisted entry carries: ``` struct LoggedEvent { id: EventId, // monotonic per sled sled_id: SledId, timestamp: UnixSeconds, transition: TransitionFired, // story_id + from + to + PipelineEvent // (or ExecutionEvent — see open question) } ``` The few events that genuinely don't fit the pipeline state machine (e.g. `ProjectAdopted`, `Rebuilt`, `GatewayHealthChanged`) live in a small, separately-enumerated `InfraEvent` enum, but the same log and the same subscriber pattern still apply. ## Session Model An LLM session is a first-class CRDT entity: ``` struct LlmSession { id: SessionId, persona: Persona, // "Timmy", "Sally", ... scope: ScopeFilter, // { sleds: All } | { sleds: Set } high_water: BTreeMap, // per-stream created: UnixSeconds, } ``` The session id is what the transport carries; it's not the Matrix room, not the web socket id. A given Matrix room may map to one session; a web UI tab may map to another. Multiple transports for the same human can share a session if you want — that's a separate UX call. ## Prompt Assembly Contract Every transport calls one helper before invoking the LLM: ``` fn assemble_prompt_context(session_id: SessionId) -> String ``` Behavior: 1. Read the session's scope filter and high-water marks. 2. Fetch events from the gateway aggregator that match the scope and are newer than the high-water marks. 3. Render them as a single `` block, ordered by sled then timestamp. 4. Advance the high-water marks to the latest event seen, atomically with the LLM-turn-start CRDT op (so a crash mid-turn doesn't double- inject). 5. Return the rendered block (empty string if no new events). The transport prepends the result to the user's prompt and invokes the LLM as usual. ## Persistence & Reliability Rules - **Event log is CRDT-backed.** Survives sled restart. - **High-water marks are CRDT-backed.** Survives gateway restart. - **Aggregator uses bounded queues with drop-oldest semantics**, and every drop logs `[event-agg] dropped N events for session ; client must re-fetch from `. The aggregator never silently swallows events — if the queue is full, the session gets a sentinel event `EventStreamGap { from, to }` so the LLM can see it missed context. - **End-to-end test required**: `fire(Event::StoryMerged{1122}) → user sends "what's going on?" → assembled prompt contains "1122 merged"`. ## Multi-Persona Scoping The same machinery serves both Timmy and Sally: | Persona | Scope filter | Notes | |---------|------------------------------------|--------------------------------| | Timmy | `{ sleds: All }` | Gateway-wide; aware of every sled | | Sally | `{ sleds: { huskies-server } }` | Single-sled; sled-local events only | | Manny | `{ sleds: { huskies, ketflix } }` | Hypothetical; subset | Sally never has to know Timmy exists, and vice versa. Their sessions advance their own high-water marks against the same underlying log. ## Decisions | Decision | Choice | Alternative | |------------------------|-------------------------------------|----------------------------------------------| | Event publication | Each sled owns its log | Single global log: cross-sled bottleneck | | Aggregation | Gateway tail-merges | Each session pulls from each sled directly: N×M fanout | | Buffer location | CRDT-persisted | In-process: lost on rebuild (current bug) | | Event identity | Typed enum | Strings: structured-log creep, no compile-time safety | | Drop semantics | Drop-oldest + `EventStreamGap` | Silent drop (current bug): LLM lies confidently | | Session ↔ transport | Session is separate from transport | One per transport: web tab + Matrix get different views | | Proactive LLM wake | OUT OF SCOPE | Wake on every event: cost + ramble | ## Open Questions 1. **Session lifecycle**. How are sessions created and garbage- collected? Created on first transport message? GC'd after N days idle? 2. **Event retention**. How long are events kept in the log? Forever feels wrong; "since last terminal session turn" feels right but needs care for multi-session readers. 3. **Multi-transport same session**. Should one human's Matrix and web UI share a session by default, or always be separate? 4. **Render budget**. If 500 events accumulated between turns, do we render all 500 or summarize? A `summarize_events` fallback path is probably worth designing in from the start. 5. **Aggregator placement when there is no gateway**. A standalone single-sled install has no gateway — does the sled itself host the aggregator? (Probably yes; trivially "aggregates" its own log.) ## Phasing - **Phase 0 (now):** this design doc. - **Phase 1:** typed `Event` enum + per-sled CRDT-backed event log; one publisher subscribes to existing pipeline transitions and writes `StoryStaged` / `StoryMerged` / `StoryMergeFailed`. - **Phase 2:** `LlmSession` CRDT entity + `assemble_prompt_context` helper, wired into the Matrix bot's `handle_message` (replaces the existing `pending_pipeline_events` Vec). End-to-end test covering the fire-event → user-turn → prompt-contains-event contract. - **Phase 3:** Gateway aggregator over multiple sleds; Timmy's session scoped to `All`. Sally's session scoped to a single sled. - **Phase 4:** Web UI and any other transports migrated onto `assemble_prompt_context`; the Matrix-specific Vec deleted. - **Phase 5:** Bounded queues + `EventStreamGap` sentinel; observability for `assemble_prompt_context` runs (events injected, gaps observed). Each phase ships independently. Phase 2 alone delivers the user-facing fix: Timmy sees what merged when you next say anything, without you needing to re-narrate.