Last time I ended with a mental model: a memory service is a very slow reader, and you should budget its reading time like any other workload on a box you own. The next week proved the model right in the way that hurts — by showing me that "a workload" was actually two workloads wearing one coat, and that serving both from one lane meant serving neither.
This is the story of splitting them, and of the four traps I fell into on the way. Numbers are from my own dashboards and logs; some are rounded for publication.
One lane, the wrong shape
Recap of the machine room: my assistant's memory server sits behind an OpenAI-compatible gateway that fronts my inference boxes. The memory pipeline has (at least) two distinct stages:
- Extraction — every retained item gets read by an LLM to pull out facts and resolve entities. Chatty, frequent, arrives the moment a conversation happens. Tolerates modest quality; demands modest latency.
- Consolidation — the periodic merge of raw facts into deduplicated, contradiction-resolved observations. Batch, quality-hungry, and with a reasoning model on an 8,000-token batch prompt it runs ten to fifteen minutes per call.
For a few days both stages shared one lane name, routed least-busy across two backends. Elegant. Then the nightly consolidation batch would occupy the only slot on the fallback rig — the one-GPU workhorse from earlier posts — and a fresh retain, fired seconds earlier by an ordinary conversation, would sit queued behind fifteen minutes of merge work.
I watched a second-scale operation wait on an hour-scale one and concluded the obvious: you cannot route a conversation-speed chore and a batch monster to the same slot and expect either SLA to hold. The pipeline needed two sizes of machine.
Two workloads, two machines
The split, now live:
| stage | box | model | character |
|---|---|---|---|
| extraction | the Halo's integrated graphics | ~4B, non-reasoning | seconds per item, always on |
| consolidation | the 2017 fallback rig | big reasoning model | nightly batch, no hurry |
The extraction box is the interesting one because it is free capacity I already owned: the Halo — my Strix Halo chat box — has a serious integrated GPU, and the quantized 4B extraction model now shares that silicon with the big chat model, living alongside it in the same unified RAM. Measurements from the swap, prefill at 4k context: 2,577 tok/s on the new quant versus 2,029 on the old one — with decode twelve percent slower, which matters little for a job that reads far more than it writes.
Two details worth more than their trouble:
The 4B thinks by default. The model family I moved to reasons unless told otherwise, and there is no per-request off-switch I could reach from the gateway — the flag has to live on the server itself. One launch argument, or every extraction quietly pays thinking-token tax. Check what your "small fast model" actually does when nobody is watching.
KV cache precision was costing me 37% of prefill. The box ran its cache in q8_0 on the theory that unified RAM was tight. It isn't. Switching to f16 lifted short prefill from 1,722 to 2,727 tok/s and left decode untouched. A tuning knob I never turned because the reason for turning it had gone stale months earlier.
The gateway side stayed honest to the lesson from last time: lane names are the contract, physical backends are an implementation detail. The memory server's virtual key is still locked to lane names; the split added a lane and changed zero consumers. The gateway also gained a fallback rule — if the iGPU engine fails or overflows, the request lands on the big lane rather than dying. Memory work may be slow; it may not be lost.
The env variable that did nothing
Now the confession corner, first entry.
Pointing the consolidation stage at the big lane was supposed to be one env
line. I wrote CONSOLIDATION_MODEL=…, restarted, saw no errors, and moved on
with my evening. Nothing was configured. The app's per-stage settings live in
an env family with an infix — CONSOLIDATION_LLM_MODEL — and the name without
it is simply unknown, silently ignored, consolidation falling back to the fast
lane.
No crash. No warning. A config system that shrugs is worse than one that refuses to boot. Two rules, permanent:
- Never conclude a knob's name from memory — grep the shipped package for its env constants. The strings are right there in the code; my recollection is not.
- Verify a routing change by which counters moved, not by the absence of errors. My acceptance test became: retain one item into a throwaway bank, snapshot the request counters on every engine, trigger a consolidation — extraction counters must move on the Halo's little engine only, the chat engine's must stay flat even though they share a chip, and consolidation lands on the fallback rig only. Counters cannot shrug.
The timeout that was actually a bouncer's cousin
Second entry. The consolidation calls run long — that's the reasoning model doing its job. The client timeout was set at 240 seconds, calls needed 3.5 to 6.5 minutes, so every call died at exactly four minutes with a TLS-read error, the retry loop fired up to four times, and each retry re-billed and re-generated the full batch. Ops "completed" only via a higher-level retry — correct results, paid for four times over.
Symptom worth memorizing: connection errors spaced at almost exactly the timeout interval are your timeout, not the server's. And the UI's error rows are a historical attempt log — check the retry counter before panicking; "completed after retries" means nothing was lost, just expensively rehearsed.
The 504 storm
Third entry, and the best one. Even after sane client timeouts, one bad day produced 36 gateway 504s on consolidation calls. The culprit was the reverse proxy sitting in front of the gateway: its read timeout was 900 seconds, shorter than a worst-case reasoning call. The proxy gave up, returned an HTML error page, the memory server dutifully retried the entire batch — and here is the part that should be on a plaque in every self-hosting museum: the proxy tore down only its own upstream connection. The inference engine kept generating.
One orphan, caught in the engine's own metrics: 180,729 tokens generated over 5,282 seconds, finishing an hour after the last client had walked away. Nobody paid that bill's attention to it except the electricity.
The attempted fixes, and why each failed, is a masterclass in reading timers before tuning them:
- TCP keepalives: cannot reset an application read timer — the proxy counts response bytes, not connection idleness.
- Gateway-level SSE pings: in this build they only exist when streaming is on, and the memory server parses non-streaming JSON, so streaming it is not.
- Shorter reasoning: available, tempting — and I deliberately left it alone. The thinking budget on this lane is what buys the consolidation quality; capping it to make a proxy comfortable is optimizing the wrong layer.
The real fix was to stop letting the proxy own a timer it was never informed about: internal callers now hit the gateway's plain HTTP port directly, bypassing the proxy leg entirely, with timeouts raised past the worst legitimate call I have actually observed — which was 806 seconds. TLS and the proxy stay exactly where trust boundaries require them; inside one trusted network, hop count is not a virtue.
Residual, stated honestly: the engine still honors client disconnect only during prefill, so a genuinely runaway call can only be killed by restarting the engine. Some orphans are a reboot, not a retry.
Batch what can batch, but batch all of it
Consolidation quality work does not need to happen at 3pm. Auto-consolidation is now off fleet-wide; a nightly job queues the batches at 23:30, off-peak rates, one slot, no ambitions. Two scars from the scheduling work:
The job originally named one bank — my main memory bank — hardcoded. When the memory got split into per-project banks, the new ones quietly accumulated 250 to 800 unconsolidated facts while everything looked green. Recall kept working (raw facts serve until merged), which is exactly why nothing screamed. Any job that says "for each bank" must enumerate banks, never remember them.
And pausing is not the same as cancelling. Deleting pending operations leaves the nightly enqueuer alive, and it will happily re-queue everything at 23:30. The pause is two switches: cancel the rows and disable the cron.
Unlocking the one slot that mattered none
The final revision was arithmetic. Back in the storm days I had set a global concurrency cap of one — the correct medicine then. But after the split, extraction and consolidation run on different physical machines, and a global cap of one meant a second-scale retain still queued behind a fifteen-minute merge. Per-stage caps turned out to compose with the global one rather than replace it, so capping a stage alone did nothing at all — I know, because I did that first.
Final shape: global two, consolidation one (it shares the fallback rig's single slot with everything else there), retain deliberately single-flight — the storm guard stays. Verified the same night: the Halo's iGPU generating extracts while the fallback rig merged, both clocks clean.
Where it is honest and where it is not
Honest: extraction and consolidation now have the machines their clocks deserve, and every routing claim above has a counter snapshot behind it. One correction to the pitch: the chat SLA is protected by separate engines and queues, not separate hardware — memory extracts share the Halo's silicon with the chat model, which is exactly why they get the small, non-reasoning model. Same chip, different appetite.
Not honest, in the sense of "not safe": consolidation now lives on exactly one box, with the reasoning that if that box dies, memory merges can wait a day — true — and that extraction's fallback lane is the big lane, which is the same box. One power supply stands between me and a stalled memory pipeline. I made that trade on purpose and I am writing it down so I cannot pretend later that it was an oversight.
The bank itself hums along: five hundred-odd memory nodes over forty-odd documents, most of them consolidated observations, all of it read by machines I own, on schedules I set, at prices I can compute.
Last post's model said: budget the reader's time. This post's amendment: there is no single reader. There is the one who answers conversations' worth of trivia in seconds, and the one who rewrites the notebook every night while the house sleeps. Give each a machine and a timer of their own — and for the love of uptime, grep the code for the env variable name before you trust your memory. Which, in a blog about memory, I admit is the thesis statement I kept dodging.