Previous post showed the dashboard. This one is about what the dashboard is watching: a Qwen3.8-Flash-Next model handling my real daily assistant traffic on a single Strix Halo mini-PC, and every non-default decision that got it there.
The starting point
A Strix Halo box — 128 GB unified memory, iGPU with a visible pool of 62 GiB of it GPU-mapped (GTT). No discrete VRAM, which is the whole point and the whole problem: memory bandwidth is shared, and everyone (model weights, KV cache, the OS) queues for the same pool.
The naive baseline is what anyone gets on day one: default quantization, default context, one session at a time, speculative decoding off. It works — and for a conversational agent it is unusable: a 60k-token conversation turn means a cold prefill every time and a serial queue behind every other channel.
The knobs, in the order they paid off
1. Quantization: Q4_K_XL. The weights live in the same memory as the KV cache, so bytes saved on weights are context you can actually afford. Q4_K_XL keeps the important tensors at higher precision and shaves the rest; the quality loss on my eval tasks was invisible, the memory loss was very visible on the RAM tile: container sits at 85.8 / 116 GiB with model + cache + headroom, which was not possible at higher precision with this context budget.
2. Context: 262,144, and a hard conversation budget. Big context is
table stakes when a "conversation" means weeks of ongoing history plus a
fat system prompt. But context is bought with KV-cache memory, so it is a
budget, not a flex. The monitor's max ctx seen peaked at 200,115 /
262,144 — 76 %. That is the number I tune against: if live sessions
approach ~90 %, I prune history, not context size.
3. Prefix caching: the single biggest win. A conversational agent re-sends the same system prompt and history every turn. With prefix caching, the engine keeps that KV cache warm, and the cache-hit tile lives at 98.8–99.3 %. The measured effect on what users feel:
| scenario | prompt | cached | TTFT |
|---|---|---|---|
| warm turn | 61.3k tok | 100 % | 720 ms |
| cold prefill | 60.9k tok | 15 % | 33.7 s |
A ~52k-token cold prefill runs at roughly 1.5k tok/s on this silicon — which is fine, until you realize the entire product experience is "keep users out of the right-hand row." Cache eviction policy and history pruning matters more than any speedup trick after that.
4. MTP (multi-token prediction). The model ships with MTP heads; the verifier accepts 65–75 % of draft tokens here, which turns decode into a net win without touching the weights. On this platform — compute is plenty, bandwidth is the bottleneck — speculation is exactly the right kind of free lunch: it trades the cheap thing for the expensive one.
5. Concurrency: -j 2. Two sessions. Not more, not one. Measured
decode in the requests table lands around 45–50 tok/s per request; a
third slot doesn't add throughput when bandwidth-bound, it just splits TTFT
headroom and lets one chatty session starve the other. Two is the sweet
spot for "one urgent conversation + one background job."
Where "better than baseline" needs receipts
The per-knob effects above are real but measured in production, mixed conditions. Before I claim a headline speedup number over the naive default, I owe a controlled run. TODO, on this machine, one sitting:
# TODO: fixed prompt set (5 short, 5 long), measure with:
# A) defaults: no cache, no MTP, -j 1
# B) final config: cache + MTP + -j 2
# report: cold TTFT, warm TTFT, decode tok/s, RAM floor/ceiling
What I can already say without the A/B: the cold-prefill-to-warm-turn gap (33.7 s → 720 ms) is a ~45x difference in what you wait for, and prefix caching is the whole reason the warm row exists.
The boring parts that mattered
- Channel-only design. The box exposes nothing inbound; inference is reached through the gateway via outbound connections. Attack surface for the monitor page: LAN only, read-only, scrape-only.
- Thermals. CPU peaked at 63 °C under sustained load, idle 48 °C — no throttling observed, which on a dense little box I consider part of the tuning result.
- Power. ~15–30 W while idle. A cloud GPU bill at 2 AM answers the same questions this box answers, at roughly infinite multiples of this number.
The config, as one block
model = Qwen3.8-Flash-Next (Q4_K_XL + MTP)
engine = gufo 0.25.0
n_ctx = 262144
sessions = 2 # -j 2
spec = MTP, on
cache = prefix, on # the one weird trick
Six lines, one machine, and a dashboard that keeps me honest about every one of them.