FOU//CAL
IT & SystemsNov 7, 20267 min read

The third death notice was a compiler: tuning an inference engine on a 2017 GPU

CUDA 13 stopped compiling for Pascal — the third formal death notice for my 1080 Ti fallback. Before accepting it I ran a proper A/B campaign on the engine: one free 20 % decode win, one flag I kept on bitwise-parity evidence, three I rejected as noise, and two measurements I distrust even now.

The last post counted two death notices for my GTX 1080 Ti: NVIDIA moved Pascal to the legacy driver branch, and EVGA itself exited the GPU business. This week a third arrived, and it was the sharpest one: CUDA Toolkit 13.0 removed the ability to compile for Pascal at all. NVIDIA's release notes call it politely — Maxwell, Pascal and Volta are "considered feature-complete," offline compilation and library support for them are removed, and you may keep building against the 12.x series "for now." Feature-complete. It's the language they use for hardware they intend to watch die in place.

Practically: the inference stack on that box has to be built from source against a pinned CUDA 12.9 toolkit, forever, or whenever 12.x toolchains rot out of my distro. Life support wearing a version number.

I did what any stubborn owner would do: instead of mourning, I benchmarked. One weekend A/B campaign on the engine — strata, the llama.cpp wrapper that serves this card — and the receipts came back interesting.

First, the method, because the method is the point

Tuning an LLM engine is unusually easy to do wrong: the system caches, thermals drift, background work steals threads, and every flag's changelog promises miracles. So the rules, held to the letter:

  • One variable per arm. Binary swap or one environment flag, never both. Each arm gets a fresh restart.
  • Verify the arm actually landed — read the live process's environment from /proc/<pid>/environ before trusting a single number. A forgotten variable is how you "prove" a flag does nothing when it never ran.
  • Nonce-prefix every prompt. The engine caches prompt state; three identical prompts is one benchmark run with delusions of repetition.
  • Temperature 0, medians of 3–4 reps. Averages let one thermal spike write history.
  • Benchmark at serving sizes, not toy prompts — that lesson cost me a corrections paragraph in the last post. ~480-token prompts, ~5,300-token prompts, engine-reported timings.

Two workloads, two metrics: prefill (prompt tokens/s) and decode (generated tokens/s). Everything below is one table, measured under those rules.

The table

arm short prefill short decode long prefill long decode
engine 0.1.43.1 (before) 185.2 20.5 332.6 21.7
engine 0.1.58 (upgrade only) 184.8 25.1 338.6 24.6
0.1.58 + dense-FP32 flag ✅ 197.3 24.1 352.5 23.8
0.1.58 + W4A8 flag ❌ 182.1 24.7 338.7 25.3
0.1.58 + VNNI tier flag ❌ 182.9 25.0 340.8 23.7
0.1.58 + adaptive-fetch flag ❌ 182.2 25.5 339.4 23.9

(tokens/s, median of 3–4 reps; the three ❌ arms are measured against row 2, not row 1. Yes, these decode numbers sit below the 36–42 of the previous post — that was the live monitor on bursty chat; the harness runs sustained generations and honestly reports a different, lower number. Absolute values belong to their harness; ratios between arms of the same harness are what travel.)

The free win: an upgrade that remembers Pascal exists

The engine upgrade alone — 0.1.43.1 to 0.1.58, rebuilt from source for compute capability 6.1, twelve minutes of compiler — bought +22 % short decode and +13 % long decode with prefill unchanged. No flags, no sacrifices.

Where did it come from? Upstream had quietly landed Pascal-aware defaults: dequant paths and expert-residency tables tuned for older GPUs. This is the thesis of the whole series in one release note — the open-source inference crowd still owns 10-series cards, so nobody has finished writing their kernels off the cliff NVIDIA walked off. Feature-complete upstream, feature-improving downstream.

The flag I kept, and the proof I demanded

The interesting row is the ✅: forcing the model's dense projections through plain FP32 cuBLAS instead of the engine's quantized path — one environment variable — gained +6.8 % short prefill and +4.1 % long prefill, at zero decode cost. The physics fits the card: Pascal's FP32 units are so abundant relative to its other capabilities that the quantized path's dequant overhead costs more than the narrower math saves. A 2016 chip where FP32 is the fast lane — tell that to any FP32-efficiency purist.

But a faster math path that changes answers is not an optimization, it's a different model wearing a costume. So the flag shipped only after the engine's own parity tool ran 5,201 different GEMM shapes against the reference path — every one bitwise identical — and five greedy (prompt-freeze, temperature 0) answers came out byte-for-byte the same with the flag on and off. Faster is only real when it's the same answer sooner.

The three flags I threw away, which is most of the work

W4A8 quantized compute, a CPU vectorization tier, adaptive expert fetch: each had a plausible story, each moved the needles inside a ±1.5 % band — which is exactly the size of run-to-run noise on this box. The keeper gained 4–7 %; the rejects gained nothing distinguishable from boredom. All three stay off.

This is the honest majority of tuning, and it's the result people never blog. For every flag that survives, three die as confetti — and knowing your noise floor is what lets you tell the difference. Had I run two reps instead of four, W4A8's long-decode number (25.3) would have looked like a win worth a blog post. It wasn't worth a restart.

Two numbers I still don't trust

The confession corner, because this series runs on receipts and receipts need their margins stated.

First: I nearly benchmarked nothing. The engine's shutdown is slow — the old process holds the port for seconds after a restart signal. Twice my restart script raced it, failed with "port already in use," and the next arm measured a service that wasn't running. An arm that crashes is honest. An arm that reads the previous arm still serving would have been seamless — which is exactly why restart2.sh now waits for the port to be free before starting anything.

Second: a contaminated arm dies with the arm. Early on I benched while the twelve-thread source build ran beside the model. Decode collapsed — and the trap is that it looked plausible. On this box the MoE expert pool streams from system RAM through the CPU, so a compiler chewing every core isn't background noise, it's competing for the model's own data path. Since then: build and bench never overlap. A number is only as clean as the quietest thing running next to it.

(The archive of bad ideas also includes a NUMA-binding trick copied from a dual-socket tuning guide, immediately useless on a single-CPU workstation, and a parity tool whose --help hangs forever — it wants an idle GPU, not a question.)

Ledger

One weekend, against CUDA 13's third death notice:

  • +22 % short decode, +13 % long decode from an engine upgrade rebuilt for a 2016 architecture the modern toolkit refuses to name.
  • +6.8 % / +4.1 % prefill from one flag, kept only after 5,201 bitwise parity checks said it was the same math, sooner.
  • Three flags rejected as noise — the most common honest outcome of optimization, and worth reporting for exactly that reason.
  • Zero behavior change: greedy answers still byte-identical, monitor still answering, card still idling at a fraction of its 250 W rating.

CUDA 13 can declare Pascal feature-complete. Feature-complete means the vendor is done with it — it says nothing about whether I am. The toolkit won't compile for this card anymore, so I compile against the last one that would, and the card answers +22 % faster than it did on Friday.

Third death notice: declined.

#gpu#nvidia#pascal#cuda#local-llm#benchmarking#strata