246 lines
12 KiB
Markdown
246 lines
12 KiB
Markdown
# LLM Context From Events
|
||
|
||
Design overview for making any LLM-driven chat persona (Timmy at the
|
||
gateway, Sally at a single sled, future personas) aware of huskies
|
||
events without the user having to re-narrate them.
|
||
|
||
## Goal
|
||
|
||
**Update the LLM's context non-intrusively when a state transition
|
||
happens.** No new LLM turn is fired; events are simply visible to the
|
||
LLM the next time the user (or anything else) does cause it to run.
|
||
The LLM should never need to be told what already happened inside
|
||
huskies.
|
||
|
||
## Guiding Principle
|
||
|
||
**Transports have nothing to do with LLMs.** A transport (Matrix bot,
|
||
web UI, CLI, future TUIs) is a pure courier — it relays user text in,
|
||
LLM text out, and never owns LLM-facing state. Anything the LLM needs
|
||
to know lives in huskies, behind a single `assemble_prompt_context`
|
||
helper that the transport calls. Adding a new transport must require
|
||
zero changes to the event-awareness path.
|
||
|
||
## Three things this doc is NOT
|
||
|
||
1. **Triggers** — `on StoryMerged{1122} do Rebuild`. These are
|
||
deterministic subscribers; they should never invoke the LLM. Covered
|
||
in a separate design.
|
||
2. **Proactive wake** — running an LLM turn *because* an event fired,
|
||
without the user typing. Costs tokens, risks ramble. Explicitly out
|
||
of scope here; a separate decision to make later.
|
||
3. **A transport feature** — this design assumes any transport that
|
||
invokes the LLM uses the same context-assembly helper. Matrix bot,
|
||
web UI, CLI all funnel through it.
|
||
|
||
## Why Past Attempts Have Failed
|
||
|
||
- **Buffer lived on the transport**, not on huskies. The current
|
||
`BotContext.pending_pipeline_events` (`server/src/chat/transport/matrix/bot/context.rs:103-116`)
|
||
is Matrix-only; web UI users see nothing of the kind, and the buffer
|
||
dies with the bot process.
|
||
- **Process-local, RAM-only**. Server rebuild → buffer empty. Any
|
||
events between the old binary's last user turn and the new binary's
|
||
first are silently lost.
|
||
- **Unbounded `mpsc` channels drop under lag.** The server logs
|
||
routinely show `[xxx-sub] Subscriber lagged, skipped N event(s)`.
|
||
When the subscriber feeding the buffer falls behind, events vanish
|
||
without being recorded.
|
||
- **No end-to-end test.** Nothing asserts "fire event E, send user
|
||
message M, the LLM's prompt contains E."
|
||
- **No cross-process aggregation.** Events in a sled have no path to
|
||
the gateway-side LLM context without bespoke plumbing per event type.
|
||
|
||
## Architecture at a Glance
|
||
|
||
```
|
||
┌────────────┐ ┌────────────┐ ┌────────────┐
|
||
│ Sled A │ │ Sled B │ │ Sled C │
|
||
│ event_log/ │ │ event_log/ │ │ event_log/ │ ◄── source of truth
|
||
└─────┬──────┘ └─────┬──────┘ └─────┬──────┘ (CRDT-backed)
|
||
│ │ │
|
||
└───────────────┼───────────────┘
|
||
▼
|
||
┌────────────────────┐
|
||
│ Gateway aggregator │ ◄── tail-merges all sled logs
|
||
│ event_view/ │ into a single ordered stream
|
||
└─────────┬──────────┘
|
||
│
|
||
▼
|
||
┌────────────────────────┐
|
||
│ Per-LLM-session state │ ◄── scope filter +
|
||
│ sessions/<id>/ │ high-water mark per stream
|
||
└─────────┬──────────────┘
|
||
│
|
||
▼
|
||
┌────────────────────────┐
|
||
│ assemble_prompt_context│ ◄── single helper used by
|
||
│ (session_id) -> Str │ every transport before
|
||
└─────────┬──────────────┘ each LLM turn
|
||
│
|
||
┌──────────────┼──────────────┐
|
||
▼ ▼ ▼
|
||
Matrix bot Web UI CLI / TUI
|
||
```
|
||
|
||
## Event Model — Reuse What Already Exists
|
||
|
||
There is no need to invent a parallel event taxonomy. Huskies already
|
||
has a complete typed enum and a single broadcast bus:
|
||
|
||
- `server/src/pipeline_state/transition.rs` defines `PipelineEvent`
|
||
with **30 variants** covering every state-machine transition
|
||
(`DepsMet`, `GatesStarted/Passed/Failed`, `QaSkipped`,
|
||
`MergeSucceeded/Failed/FailedFinal`, `Accepted`, `Block/Unblock`,
|
||
`Abandon`, `Supersede`, `ReviewHold/Cleared`, `Reject`, `Triage`,
|
||
`Close`, `Demote`, `Freeze/Unfreeze`, `MergemasterAttempted`,
|
||
`FixupRequested`, `ReQueuedForQa`, `MergeAborted`,
|
||
`HotfixRequested`, `MergeRetryStarted`).
|
||
- The same module defines `ExecutionEvent` with 7 variants for agent
|
||
lifecycle (`SpawnRequested`, `SpawnedSuccessfully`, `Heartbeat`,
|
||
`HitRateLimit`, `Exited`, `Stopped`, `Reset`).
|
||
- Every transition fires a `TransitionFired` event on a single internal
|
||
bus. Ten subscribers already consume it (audit-log,
|
||
worktree-create-sub, worktree-cleanup-sub, merge-failure-sub,
|
||
merge-block-sub, done-archive-sub, content-gc, cost-rollup-sub,
|
||
stage-notification-sub, event-triggers).
|
||
|
||
**The LLM context injector is just the 11th subscriber on the same
|
||
bus.** It writes typed events into the per-sled CRDT event log
|
||
described below; everything downstream reuses the existing taxonomy.
|
||
|
||
Each persisted entry carries:
|
||
|
||
```
|
||
struct LoggedEvent {
|
||
id: EventId, // monotonic per sled
|
||
sled_id: SledId,
|
||
timestamp: UnixSeconds,
|
||
transition: TransitionFired, // story_id + from + to + PipelineEvent
|
||
// (or ExecutionEvent — see open question)
|
||
}
|
||
```
|
||
|
||
The few events that genuinely don't fit the pipeline state machine
|
||
(e.g. `ProjectAdopted`, `Rebuilt`, `GatewayHealthChanged`) live in a
|
||
small, separately-enumerated `InfraEvent` enum, but the same log and
|
||
the same subscriber pattern still apply.
|
||
|
||
## Session Model
|
||
|
||
An LLM session is a first-class CRDT entity:
|
||
|
||
```
|
||
struct LlmSession {
|
||
id: SessionId,
|
||
persona: Persona, // "Timmy", "Sally", ...
|
||
scope: ScopeFilter, // { sleds: All } | { sleds: Set<SledId> }
|
||
high_water: BTreeMap<SledId, EventId>, // per-stream
|
||
created: UnixSeconds,
|
||
}
|
||
```
|
||
|
||
The session id is what the transport carries; it's not the Matrix room,
|
||
not the web socket id. A given Matrix room may map to one session; a
|
||
web UI tab may map to another. Multiple transports for the same human
|
||
can share a session if you want — that's a separate UX call.
|
||
|
||
## Prompt Assembly Contract
|
||
|
||
Every transport calls one helper before invoking the LLM:
|
||
|
||
```
|
||
fn assemble_prompt_context(session_id: SessionId) -> String
|
||
```
|
||
|
||
Behavior:
|
||
|
||
1. Read the session's scope filter and high-water marks.
|
||
2. Fetch events from the gateway aggregator that match the scope and
|
||
are newer than the high-water marks.
|
||
3. Render them as a single `<system-reminder>` block, ordered by sled
|
||
then timestamp.
|
||
4. Advance the high-water marks to the latest event seen, atomically
|
||
with the LLM-turn-start CRDT op (so a crash mid-turn doesn't double-
|
||
inject).
|
||
5. Return the rendered block (empty string if no new events).
|
||
|
||
The transport prepends the result to the user's prompt and invokes the
|
||
LLM as usual.
|
||
|
||
## Persistence & Reliability Rules
|
||
|
||
- **Event log is CRDT-backed.** Survives sled restart.
|
||
- **High-water marks are CRDT-backed.** Survives gateway restart.
|
||
- **Aggregator uses bounded queues with drop-oldest semantics**, and
|
||
every drop logs `[event-agg] dropped N events for session <id>; client
|
||
must re-fetch from <high-water>`. The aggregator never silently
|
||
swallows events — if the queue is full, the session gets a sentinel
|
||
event `EventStreamGap { from, to }` so the LLM can see it missed
|
||
context.
|
||
- **End-to-end test required**: `fire(Event::StoryMerged{1122}) → user
|
||
sends "what's going on?" → assembled prompt contains "1122 merged"`.
|
||
|
||
## Multi-Persona Scoping
|
||
|
||
The same machinery serves both Timmy and Sally:
|
||
|
||
| Persona | Scope filter | Notes |
|
||
|---------|------------------------------------|--------------------------------|
|
||
| Timmy | `{ sleds: All }` | Gateway-wide; aware of every sled |
|
||
| Sally | `{ sleds: { huskies-server } }` | Single-sled; sled-local events only |
|
||
| Manny | `{ sleds: { huskies, ketflix } }` | Hypothetical; subset |
|
||
|
||
Sally never has to know Timmy exists, and vice versa. Their sessions
|
||
advance their own high-water marks against the same underlying log.
|
||
|
||
## Decisions
|
||
|
||
| Decision | Choice | Alternative |
|
||
|------------------------|-------------------------------------|----------------------------------------------|
|
||
| Event publication | Each sled owns its log | Single global log: cross-sled bottleneck |
|
||
| Aggregation | Gateway tail-merges | Each session pulls from each sled directly: N×M fanout |
|
||
| Buffer location | CRDT-persisted | In-process: lost on rebuild (current bug) |
|
||
| Event identity | Typed enum | Strings: structured-log creep, no compile-time safety |
|
||
| Drop semantics | Drop-oldest + `EventStreamGap` | Silent drop (current bug): LLM lies confidently |
|
||
| Session ↔ transport | Session is separate from transport | One per transport: web tab + Matrix get different views |
|
||
| Proactive LLM wake | OUT OF SCOPE | Wake on every event: cost + ramble |
|
||
|
||
## Open Questions
|
||
|
||
1. **Session lifecycle**. How are sessions created and garbage-
|
||
collected? Created on first transport message? GC'd after N days
|
||
idle?
|
||
2. **Event retention**. How long are events kept in the log? Forever
|
||
feels wrong; "since last terminal session turn" feels right but
|
||
needs care for multi-session readers.
|
||
3. **Multi-transport same session**. Should one human's Matrix and web
|
||
UI share a session by default, or always be separate?
|
||
4. **Render budget**. If 500 events accumulated between turns, do we
|
||
render all 500 or summarize? A `summarize_events` fallback path is
|
||
probably worth designing in from the start.
|
||
5. **Aggregator placement when there is no gateway**. A standalone
|
||
single-sled install has no gateway — does the sled itself host the
|
||
aggregator? (Probably yes; trivially "aggregates" its own log.)
|
||
|
||
## Phasing
|
||
|
||
- **Phase 0 (now):** this design doc.
|
||
- **Phase 1:** typed `Event` enum + per-sled CRDT-backed event log;
|
||
one publisher subscribes to existing pipeline transitions and writes
|
||
`StoryStaged` / `StoryMerged` / `StoryMergeFailed`.
|
||
- **Phase 2:** `LlmSession` CRDT entity + `assemble_prompt_context`
|
||
helper, wired into the Matrix bot's `handle_message` (replaces the
|
||
existing `pending_pipeline_events` Vec). End-to-end test covering the
|
||
fire-event → user-turn → prompt-contains-event contract.
|
||
- **Phase 3:** Gateway aggregator over multiple sleds; Timmy's session
|
||
scoped to `All`. Sally's session scoped to a single sled.
|
||
- **Phase 4:** Web UI and any other transports migrated onto
|
||
`assemble_prompt_context`; the Matrix-specific Vec deleted.
|
||
- **Phase 5:** Bounded queues + `EventStreamGap` sentinel; observability
|
||
for `assemble_prompt_context` runs (events injected, gaps observed).
|
||
|
||
Each phase ships independently. Phase 2 alone delivers the user-facing
|
||
fix: Timmy sees what merged when you next say anything, without you
|
||
needing to re-narrate.
|