FOU//CAL
IT & SystemsOct 27, 20265 min read

From 0 to hero: a live monitor for an inference server that shipped without one

gufo exposes nothing but Prometheus /metrics. Here is the single-page live dashboard I built on top of it, and the five numbers I actually watch.

The inference engine behind my personal AI assistant (gufo, on the Strix Halo box from the first post) does one thing perfectly and one thing absent: it serves tokens, and it has no web UI. The entire observability story out of the box is a /metrics endpoint that dumps a few hundred lines of Prometheus plaintext:

$ curl -s http://inference-box:8080/metrics | head
# HELP llama_cpp:num_requests Number of processed requests
# TYPE llama_cpp:num_requests counter
llama_cpp:num_requests 23092
# HELP llama_cpp:seconds_per_token_generated ...

Which is great — if you are a Prometheus with a Grafana habit and a weekend to spare. I am a person with a second-hand dashboard reflex who wants to glance at one page while my assistant is mid-sentence. So: build the page.

The constraints

Before any code, the rules I gave myself:

  1. Read-only. The monitor must never be able to touch the engine beyond scraping /metrics. A monitoring page that can restart things is an outage waiting for a curious guest.
  2. LAN only. It watches a machine that already has zero inbound ports from the internet. The monitor does not change that.
  3. Zero external assets. No CDN fonts, no analytics, no chart library phoning home. One service, one HTML page, whatever the engine can already draw.
  4. Explain itself. Every tile should answer "is this number normal?" without a wiki.

The architecture, which barely deserves the word: a small companion service polls the engine's /metrics every second, computes deltas and rates, and pushes the snapshot to the browser. Engine dies → the page goes stale within seconds, which is exactly the alert I want visible from across the room.

The page

gufo monitor: header strip, session chips and stat tiles

Top to bottom:

  • Header strip — model, engine build, session count, context size, and a green dot that means "scrape succeeded in the last few seconds." When the dot goes grey, the dot is the whole post-mortem.
  • Session chips — one per slot (-j 2 means two). Idle vs processing. This is the fastest "why is my assistant slow?" check in existence: if both chips are busy, nothing else on this page matters yet.
  • Stat tiles — throughput, cache, and hardware numbers. I will take the interesting ones below.
  • Charts — the last few minutes of each series. Flatlines are information too.
  • Recent requests table — the real hero. More on it below.

The same page under real load, minutes later — 53 tok/s of decode, the S1 chip mid-read on a 176k-token session at 100 % cache, GPU at 85 %, and the charts finally earning their pixels:

gufo monitor under live load: busy session chip, tile values and chart curves

The five numbers I actually watch

1. Prefix cache hit. The engine keeps KV cache from previous turns; a long-running conversation re-sends a long system prompt plus history on every turn, so this number should sit near 99 %. On the box today: 98.8–99.3 %. If it drops, the cache is being evicted — usually because context is growing toward the limit.

2. TTFT, warm vs cold. Time to first token is the number you actually feel. Warm rows land in 500–1200 ms. Here is the gift the table gives you — two rows, same morning:

req prompt cached TTFT dur
r23092 61.3k tok 100 % 720 ms 23.2 s
r22937 60.9k tok 15 % 33.7 s 38.3 s

Same size prompt. The cached one answered in under a second; the cold one prefilled ~52k uncached tokens at roughly 1.5k tok/s and the user waited half a minute. Everything about tuning conversational inference falls out of this one comparison.

And the table catches both regimes as they happen — a 5 %-cached 174k-token prompt paying 187 s of TTFT, sitting between a field of 100 %-cached rows:

gufo monitor recent-requests table catching a cold-prompt outlier

3. MTP accept rate. The model runs multi-token prediction — draft tokens that get verified in bulk. The monitor tracks what fraction the verifier accepts: 65–75 % here. Below ~50 % the speculative machinery is mostly wasting compute and it is worth knowing why (odd traffic, different workload than the draft was good at).

4. KV usage & max ctx seen. Context is a budget: 262,144 tokens total, and max ctx seen says the biggest live session touched 200,115 — 76 % of the budget. That is the eviction alarm before the eviction.

5. RAM headroom. Unified memory means the model, the KV cache, and the rest of the container all fight in one pool. 85.8 / 116 GiB used, 31.6 GiB headroom. Headroom shrinking over days = a leak; shrinking in minutes = the next big conversation.

gufo monitor: requests chart, RAM chart and the recent-requests table

What it caught (that /metrics never showed me)

  • A retry storm from a misconfigured gateway: the requests chart shows a comb of 3-token requests every ~60 s. Ten seconds of looking at the table beat an hour of grepping logs.
  • One conversation that grew to 200k tokens and never got pruned — visible as max ctx seen ratcheting up while cache hit slid down.
  • Idle draw of ~15–30 W with CPU at 48 °C, so "is it even running?" is now a glance instead of an SSH session.

Next

  • Per-session token accounting (which client is eating the cache).
  • A 24 h view; the current window is minutes.
  • An optional webhook alert on stale scrape, so the green dot can also yell.

Total build cost: less than a Grafana install, and it already pays rent. From zero to hero, as promised — the hero part is that I stopped guessing.

#local-llm#observability#homelab#python