My assistant is not an amnesiac — it ships with memory. Two curated note files, loaded fresh into every conversation: who I am, what the stack looks like, which lessons it has earned the hard way. They survive restarts, and everything written into them survives to the next day.
What they do not survive is my operations. The entire long-term memory budget is a few thousand characters — my share is roughly two thousand, split between "notes to self" and "who the user is". Sticky notes that survive a reboot, in other words. And the bookkeeping is real: when the note fills up, new memories are rejected until I delete old ones, so every addition becomes a curatorial decision about which fact still deserves its share of the context window. The whole note is injected into every turn whether relevant or not — the more it remembers, the less room is left to think.
That is perfect for doctrine and for scars. It is hopeless for the thing I actually wanted: a memory that has read every conversation I have had with this assistant and can answer "what did we settle about the fallback box?" three weeks later. So the plan was a second layer underneath the sticky notes.
The cloud memory services all solve this on their GPUs, which is a polite way of saying my conversations would leave the house to be remembered. My standing rule for this stack is that nothing phones home, so the build was: a self-hosted memory server — extracts facts from conversations, embeds them locally, keeps them in an embedded PostgreSQL — sitting behind the OpenAI-compatible gateway that already fronts my two inference boxes. Zero external dependencies. On paper: an afternoon.
What nobody tells you: memory is not a storage problem. It is a compute problem with a database costume.
Why it is compute: a retain is not a write
When the memory server "remembers" something, it does not append a row. Each retain runs an extraction pipeline through an LLM: pull candidate facts out of the text, resolve entities ("the fallback box" vs "the ThinkStation" vs "it"), consolidate against what is already stored. My pipeline costs 35–40k tokens per retained item on a small bank — a 300-word memory costs a hundred times its size.
Then the multiplication: my assistant was configured to auto-retain every conversation turn, and the extraction model is a reasoning model — it emits 5–15k thinking tokens per stage at roughly 22 tok/s. Do the math on a busy day. I did not. The queue did.
The storm
The memory lane had been routed to the 2017 fallback rig from my previous post — one GPU slot, politely queueing, perfect for background chores. Except inflow beat drain, forever. Turn after turn queued a multi-minute extraction behind the last one. The client gave up at its HTTP timeout, but — and this is the part that stung — cancelling the client call does not cancel the generation. The GPU kept burning orphaned work, retries re-billed and re-queued it, and within hours the lane was so jammed that even a ten-token ping timed out.
First triage lesson, learned the hard way: I kept pinging the lane to see if it
had recovered. A ping joins the queue. You are billing your own probe behind
five orphaned extractions. The passive truth was the inference server's
/metrics the whole time — queue depth and completed-request durations, no
tokens spent.
The fix had three parts, only the last of which is obvious:
- Auto-retain off. Explicit, deliberate retains instead of a conveyor belt.
- A hard concurrency cap on the client so it can never fire five parallel extractions at a single-slot backend.
- Leave the queue alone. The tasks are database-persistent with exponential backoff; once the backend drained, they finished themselves, unattended. Retaining "lost" items to speed it up just re-runs the entity and consolidation stages — and re-bills them.
The cap that was actually a wall
While tuning lanes, I did the reasonable thing: told the gateway the physical
concurrency of each backend (one slot on the fallback rig, two on the Halo).
max_parallel_requests, seems self-describing.
It is a hard-reject cap, not a soft hint. The instant a deployment's slots are full, the gateway 429s the next request instead of letting the backend queue it politely — and the memory server's retry loop burns against that 429 until the retain dies permanently. I had modeled a queue and built a bouncer. Unset the caps: the backends have always queued fine on their own; let the router balance by in-flight count and stay out of the way.
Same week, the per-lane token budget gate: I sized it at 25k tokens/minute for the memory lane, which sounded generous until real extraction prompts (~5k in, reasoning-heavy) tripped 429s with a cheerfully misleading "Remaining: full limit" message. The gate rejects any single request larger than the window. Budget for the caller's real prompt size (150k now), not your intuition.
The bug that was literally three asterisks
Here is my favorite. The Halo box appeared to be serving nothing. The gateway dashboard showed both deployments healthy, the model list looked right — and the spend logs were empty, which is how you know nothing ever went through.
The provider key in the gateway's env file was, and I am not embellishing,
***. Three literal asterisks. My tooling's secret-scrubber had rewritten the
value during a staging step months earlier, and every subsequent check —
"key present?", "config parses?" — passed, because *** is a perfectly present,
parseable string. Halo had been silently dead on arrival, and every "Halo"
request had been quietly served by the fallback rig instead.
Two permanent rules came out of this:
- Verify staged secrets by length and prefix after writing, never by mere
presence.
***passes a presence check forever. - Ask who actually served the reply. The response's
modelfield names the concrete build, so the fallback box is unmasked the moment you look. Silent fallback is invisible until you go looking for it.
Doctrine reversal in one afternoon
Original doctrine, written in a calmer week: memory traffic must never touch the Halo — keep background chores off the box that answers conversations.
Reality on November 3rd: Halo sitting at zero utilization while a memory retain took twenty to thirty minutes end-to-end on the single-slot fallback rig. The purism was costing me the exact latency the purism was meant to protect. So, same day, the rule was revoked: the memory lane became one lane name with two physical backends behind it, least-busy routing. Keeping one group name matters — the memory server's virtual key is locked to lane names, so scaling backends under the same name meant zero consumer changes. (Never mint a new group to scale; that was another trap I only read about.)
Worth saying out loud: memory traffic now shares the Halo's chat queues. That is a considered trade, not an oversight — one gateway decision I can flip back in one commit if the chat latency ever disagrees with me.
The one part that went exactly to plan
The gateway gained a Redis response cache: 256 MB, all-keys-LRU, ephemeral, no published port — safe to lose by design. Two pitfalls on the way (a config string that crashed every request with a type error, and a Redis image that drops privileges before reading its config file, so a root-owned conf crashloops), and then: a repeated identical request went from 0.55 s to 0.02 s, provable by the identical completion id. The retry-dedupe win is the real one — when the memory server re-fires an identical extraction inside the TTL window, it costs nothing and burns zero GPU.
What it is like now
The sticky notes still load every morning — doctrine and scars stay up front, where they belong. Underneath them sits a database that has read every conversation, never seen the internet, and answers "what did we settle about the fallback box?" with actual retrieved history instead of a shrug. The notes themselves are mirrored into that bank as they change, so nothing lives in only one layer.
And I watch its memory lane the way you watch a dishwasher: it is filling, draining, occasionally slow, and nobody should open the door mid-cycle.
The mental model I would hand anyone building local memory: every memory service — including the cloud ones — is running my exact extraction pipeline. The only difference is they rent the GPUs by the hour and hide the queue. Memory is not a hard drive. It is a very slow reader, and you should budget its reading time like any other workload on a box that you own.