Two machines in my house run the same model for my personal AI assistant. One is a Strix Halo box with 128 GB of unified memory, running gufo. The other is a GTX 1080 Ti from 2017 with 11 GB of VRAM, streaming model experts from system RAM over PCIe via Strata. The model is Qwen3.8-Flash-Next: 125B parameters, 6B activated per token, plus a 51B n-gram embedding table and a 4B MTP head.
On Monday, Unsloth shipped Dynamic v3.1 for it. By Tuesday both machines were upgraded, and the interesting part isn't that both got faster — it's how differently each one got there.
Why quant choice is a different question on each box
The model shape matters here. An MoE that activates 6B parameters per token doesn't read its whole weight set for each token — it reads the shared layers plus a few routed experts. That single fact splits the optimization problem in two:
On the Halo, all 104 GB of the Q4_K_XL shards sit resident in unified memory. Quant choice is a quality question with a RAM ceiling: Q5_K_XL (147 GB) simply doesn't fit on 128 GB, so the real choice was between Q4_K_XL (103.7 GB) and Q3_K_XL (83.3 GB). I took Q4_K_XL: same footprint as the v2 shards it replaced, and the headroom question was already answered — 262k context across 2 sessions loads in ~30 s and the box doesn't swap.
On the 1080 Ti, VRAM is 11 GB and the model is ~87 GB. Nothing fits. Strata streams experts from system RAM over PCIe on demand, which makes quant choice a PCIe traffic question: smaller quants mean fewer bytes per expert fetch. The pack that runs there is IQ4_XS (87.2 GB), and the engine spends its life balancing an expert cache against the bus. Last night's monitor tells the story: consolidation jobs at ~28–31 tok/s with the expert cache hitting ~57 % and ~13 % of expert traffic actually crossing PCIe. Every quant improvement that shrinks experts is a direct decode win on Pascal; every percent of cache hit is a PCIe cycle saved.
Same file naming convention. Different physics.
The receipts
The v3.1 lineup, measured against Hugging Face's own file sizes (not the marketing page):
| Quant | Size | Halo? | 1080 Ti (streamed) |
|---|---|---|---|
| UD-Q6_K_XL | 157.5 GB | ✗ | no reason |
| UD-Q5_K_XL | 147.4 GB | ✗ | no reason |
| UD-Q4_K_XL | 103.7 GB | ✓ production | — |
| UD-IQ4_XS | 87.2 GB | fits | ✓ production |
| UD-Q3_K_XL | 83.3 GB | fits, roomier | — |
A note on the honesty of this table: v3.1's headline claim is ~50 % less token-use-at-a-given-quality versus prior dynamic quants, which sounds like the smaller quants should beat the bigger ones. They don't get to skip the table — Unsloth's own quality reports show Q3_K_XL v3.1 landing near v2-era Q4_K_XL quality. "Better at the same size" is not "the same at less size." I upgraded the Halo from Q4_K_XL v2 to Q4_K_XL v3.1 and kept the tier; that felt like the correct reading of the release.
For reference, upstream's own published numbers for this model on this class of hardware: ~1,800 tok/s prefill and up to ~117 tok/s single-user decode with MTP. I don't reproduce those in daily use — my real traffic is my assistant's conversations, which are long-context, tool-call-heavy, and nothing like a benchmark.
MTP: the spec decode number that lies to you
The Halo runs speculative decoding with the model's MTP head — the draft proposes up to seven tokens, the model verifies them in one pass. There's a satisfying counter for this, and it produced the most misleading number of my week.
On a clean test prompt (count 1 to 30), the counters read 10 draft rounds, 70 accepted tokens — 7.0 per round. Perfect-looking acceptance. Post that through a real assistant session and the same counters tell a different story: across 804 draft rounds of actual agentic traffic, acceptance sat at ~1.6 tokens per round.
Both numbers are true. Predictable text (counting, boilerplate, repeated structure) lets the draft head sweep the board; tool-call JSON and factual prose get rejected mid-draft. The engine adapts — it shortens proposals when they keep failing — but if you size your expectations on a benchmark prompt, you will be disappointed by your workload, and if you judge MTP only by your workload you'll think it's broken. It is neither. It's just traffic-shaped.
Three ops traps, taken so you don't have to
1. The flag that stopped existing. The upgraded engine refused to boot with exit code 2: Unknown option: --api-key. The changelog for 0.29.0, three releases back: "drop API-key auth" — deliberately, project-wide, with old Authorization headers accepted and ignored so nothing client-side breaks. My service unit carried the flag from the previous major version. Lesson: read the changelog for every version between yours and the target, not just the one you're jumping to. The breaking change was five releases old and I'd never seen it because I wasn't running those releases.
2. The registry that undersold itself. To pick a target version I enumerated the container registry's tag list, which ended at 0.31.0. The "latest available" I planned around was two releases stale: 0.33.0 existed the whole time and answered a manifest request with a happy 200. Tag-list endpoints are caches and paginating afterthoughts; the manifest API is the truth. Three HEAD requests found what fifty lines of tags "proved" didn't exist.
3. The right-shaped wrong file. The engine docs prescribe a shared MTP sidecar (2.8 GB — it reuses the main model's embedding/output layers). I had downloaded the standalone 4.1 GB head, which loads, runs, and passes drafts — while being the deprecated layout. Nothing failed. No warning. The "correct" file and the "works" file were one grep through the docs apart.
The fallback refuses to die (again)
The pattern that keeps repeating on this blog: every formal retirement notice for the Pascal card gets answered by a software release that makes it more useful, not less. CUDA stopped compiling for it; the engine was rebuilt around that. The compiler death notice produced a measured 20 % decode win. Now a quant release aimed squarely at MoE inference, on an engine that streams experts from system RAM, has this card doing useful overnight reasoning at thirty tokens per second — a job it has no business doing.
The two boxes upgraded cleanly, the assistant bounced between them without noticing, and the only thing that didn't survive the upgrade was a 4 GB file I now know was the wrong file all along.
Numbers in this post come from my own counters and Hugging Face / upstream documentation as of 2026-11-09; upstream benchmark figures are theirs, clearly marked. If a number here disagrees with an older post, the harness changed — see the receipts policy.