FOU//CAL
IT & SystemsOct 28, 20264 min read

Squeezing the Halo: how the inference box went from default to production

The exact serving setup behind my local AI assistant — quantization, context budget, MTP, and concurrency — plus the measured numbers that justified each knob.

Previous post showed the dashboard. This one is about what the dashboard is watching: a Qwen3.8-Flash-Next model handling my real daily assistant traffic on a single Strix Halo mini-PC, and every non-default decision that got it there.

The starting point

A Strix Halo box — 128 GB unified memory, iGPU with a visible pool of 62 GiB of it GPU-mapped (GTT). No discrete VRAM, which is the whole point and the whole problem: memory bandwidth is shared, and everyone (model weights, KV cache, the OS) queues for the same pool.

The naive baseline is what anyone gets on day one: default quantization, default context, one session at a time, speculative decoding off. It works — and for a conversational agent it is unusable: a 60k-token conversation turn means a cold prefill every time and a serial queue behind every other channel.

The knobs, in the order they paid off

1. Quantization: Q4_K_XL. The weights live in the same memory as the KV cache, so bytes saved on weights are context you can actually afford. Q4_K_XL keeps the important tensors at higher precision and shaves the rest; the quality loss on my eval tasks was invisible, the memory loss was very visible on the RAM tile: container sits at 85.8 / 116 GiB with model + cache + headroom, which was not possible at higher precision with this context budget.

2. Context: 262,144, and a hard conversation budget. Big context is table stakes when a "conversation" means weeks of ongoing history plus a fat system prompt. But context is bought with KV-cache memory, so it is a budget, not a flex. The monitor's max ctx seen peaked at 200,115 / 262,144 — 76 %. That is the number I tune against: if live sessions approach ~90 %, I prune history, not context size.

3. Prefix caching: the single biggest win. A conversational agent re-sends the same system prompt and history every turn. With prefix caching, the engine keeps that KV cache warm, and the cache-hit tile lives at 98.8–99.3 %. The measured effect on what users feel:

scenario prompt cached TTFT
warm turn 61.3k tok 100 % 720 ms
cold prefill 60.9k tok 15 % 33.7 s

A ~52k-token cold prefill runs at roughly 1.5k tok/s on this silicon — which is fine, until you realize the entire product experience is "keep users out of the right-hand row." Cache eviction policy and history pruning matters more than any speedup trick after that.

4. MTP (multi-token prediction). The model ships with MTP heads; the verifier accepts 65–75 % of draft tokens here, which turns decode into a net win without touching the weights. On this platform — compute is plenty, bandwidth is the bottleneck — speculation is exactly the right kind of free lunch: it trades the cheap thing for the expensive one.

5. Concurrency: -j 2. Two sessions. Not more, not one. Measured decode in the requests table lands around 45–50 tok/s per request; a third slot doesn't add throughput when bandwidth-bound, it just splits TTFT headroom and lets one chatty session starve the other. Two is the sweet spot for "one urgent conversation + one background job."

Where "better than baseline" needs receipts

The per-knob effects above are real but measured in production, mixed conditions. Before I claim a headline speedup number over the naive default, I owe a controlled run. TODO, on this machine, one sitting:

# TODO: fixed prompt set (5 short, 5 long), measure with:
#   A) defaults: no cache, no MTP, -j 1
#   B) final config: cache + MTP + -j 2
# report: cold TTFT, warm TTFT, decode tok/s, RAM floor/ceiling

What I can already say without the A/B: the cold-prefill-to-warm-turn gap (33.7 s → 720 ms) is a ~45x difference in what you wait for, and prefix caching is the whole reason the warm row exists.

The boring parts that mattered

  • Channel-only design. The box exposes nothing inbound; inference is reached through the gateway via outbound connections. Attack surface for the monitor page: LAN only, read-only, scrape-only.
  • Thermals. CPU peaked at 63 °C under sustained load, idle 48 °C — no throttling observed, which on a dense little box I consider part of the tuning result.
  • Power. ~15–30 W while idle. A cloud GPU bill at 2 AM answers the same questions this box answers, at roughly infinite multiples of this number.

The config, as one block

model   = Qwen3.8-Flash-Next (Q4_K_XL + MTP)
engine  = gufo 0.25.0
n_ctx   = 262144
sessions = 2   # -j 2
spec    = MTP, on
cache   = prefix, on   # the one weird trick

Six lines, one machine, and a dashboard that keeps me honest about every one of them.

#local-llm#amd#strix-halo#tuning