Chasing Flash-Next: bringing a 95 GB Qwen model to a 96 GB homelab

A few days ago I documented how Qwen3.6-35B-A3B found a home on my Unraid box — a decade-old Xeon, 64 GiB of ECC DDR4 and an RTX 3060 with 12 GB of VRAM. This post is the sequel: the move up to Qwen3.8-Flash-Next, the 125B-parameter mixture-of-experts preview of the Qwen4 architecture. Where the first post was a triumph of fitting a small model into a small box, this one is about the arithmetic of a big one — how a 94.5 GB file loads onto a 96 GB machine, why it crawled at one token per second the week before, and exactly what I measured to prove the answer.

All measurements are done: the 96 GiB is installed, three full sweeps (A, B and final production) plus a thread-count probe matrix sit in the tables below. My bandwidth model predicted 10–18 t/s; the machine delivers 9,2 — the scorecard section explains exactly where my predictions went wrong, because that part is the useful one. And the final sections hand the pen to the server itself: days of real traffic, measured in its own cumulative counters — and, at the very end, priced in joules at the wall socket.

The hardware#

Xeon E5-2630L v4 — ten Broadwell cores (20 threads) from 2016, on four-channel DDR4 ECC. And that's the first thing the spec sheets got wrong for me: the modules are 2666-MT/s parts (two Hynix, two Atermiter, two more added later), but the L SKU's integrated memory controller tops out at DDR4-2133 — a hard 68,3 GB/s ceiling set by the CPU. Any faster RAM you buy for this chip runs at exactly this speed. RTX 3060 with 12 GiB of GDDR6 on a PCIe 3.0 x8 slot. A 256 GiB NVMe for scratch. Unraid on a Beikong BKHD-2011MATX industrial board, serving llama.cpp containers.

The board came with its own curiosity: with all six 16 GiB modules installed, the BIOS reports only four in SMBIOS (dmidecode shows 4×16 GB = 64 — while the kernel's memory map says 94 GiB and is right). Six modules across four channels is unavoidably a 2+2+1+1 population, so the box runs in Intel's flex mode: quad-channel interleaving for the first rank of every channel, dual for the overflow. Nothing a manual told me; everything free -h, dmidecode and a benchmark loop together did.

One detour worth pre-empting on the GPU: people panic when nvidia-smi shows the 3060 on a x8 slot and imagine half the card is offline. It isn't — only activations and the KV cache cross PCIe; the weights that live in VRAM never re-travel the bus. The link sat under 1% utilization in my runs. The GPU slot is a non-issue for this class of model; system RAM is where everything is decided.

The model and the trick that makes it look possible#

Flash-Next is 177B parameters at rest, 6B active per token. The naive math said "impossible below ~128 GB" — and that's exactly what I believed until I found AtomicChat's architecture-aware dynamic GGUFs. Their AD-4.27bpw-Q4_K_M-M64 build is a 94.5 GB file that claims to need only 54.5 GB resident.

The reason is one specific design choice worth understanding, because it decides what the SSD can and cannot do for you:

  • 51B of the parameters are an n-gram embedding table — a lookup the model hits once per token, reading ~2.7 KB at a hash-derived address. That is 3 MB/s of random reads at 36 tokens/s. A consumer NVMe answers one of those in under 100 µs against a ~30 ms token budget. This part of the file can live on disk and be paged in as needed.
  • The routed experts are different physics: every token touches ~6B active parameters, gigabytes of random reads per token. If those ever hit the SSD, you're not running an LLM, you're running a swap demo. They must be RAM-resident, no exceptions.

AtomicChat's quants exploit exactly that split: the n-gram table sits in its own shard, the OS memory-maps it, and only the 54.5 GB of "hot" weights are pinned. Which is how a 177B model runs on a 64 GB MacBook — the file is larger than the machine's entire RAM, and 39 GB of it never enters memory at all.

The corollary, and the whole thesis of this post: the SSD trick is a one-table trick. It buys you room to keep the experts in RAM by moving the one component that's cheap to page to disk. It does not make the experts pageable. Get those two numbers mixed up and the model runs at a single token per second — which is precisely what happened to me first.

flowchart LR
  F["94.5 GB merged GGUF (mmap)"] -->|"39 GB, paged on demand"| SSD["NVMe: 51B n-gram table (~2.7 KB random read per token)"]
  F -->|"54.5 GB, pinned"| RAM["96 GB RAM: routed experts (6B active per token) — never pageable"]
  F -->|"remainder"| GPU["RTX 3060: attention, KV, shared experts"]

So here's the thing: 54.5 GB + OS > 64 GB#

Before the RAM arrived, I loaded the model on the 64 GiB configuration anyway, to watch it fail. The model did load. It answered. And it measured like this:

Test Prompt tokens Prefill Decode
512-window request 861 10.4 t/s 1.1 t/s
2k-window request 3,451 16.5 t/s 1.4 t/s

One token per second on hardware that had run Qwen3.6 at 35 t/s. Nothing was broken — the arithmetic was simply against me: 54.5 GB of hot weights + ~8 GB for Unraid and containers leaves the page cache ~2 GB of headroom for everything else the experts need to touch. The kernel constantly evicted expert pages, and every token read some of them back from disk. The mechanism deserves its exact name, because it matters later: those 54.5 "resident" GB are mmap'd file pages — reclaimable cache, not pinned memory. free -h shows them under "buff/cache", not "used", and under pressure the kernel hands them back without asking. "Fits in RAM" means fits with enough slack that nobody ever wants them back — a very different bar.

Configuration Resident set Page-cache headroom Decode t/s (short ctx)
64 GiB RAM (measured) 54.5 GB + ~8 OS ~2 GB — evicting 1,1
96 GiB, --fit on (Run A) " ~33 GB — stable 8,8
96 GiB, --fit off --cpu-moe (Run B = production) " ~33 GB — stable 9,2

An early, expensive lesson: never benchmark during a parity sync#

The very first 96 GiB sweep happened to run while Unraid was mid-parity check — and it read "honestly": 5–7 t/s decode, and a 60k-token row that took 542 seconds. The numbers were internally consistent, the model answered fine, and every one of them was fiction. Parity work saturates exactly the two resources this model class lives on: disk I/O for the mmap'd n-gram table, and the page cache for the expert set — the kernel happily evicts "cold" model pages to make room for stripe traffic. Pausing the sync and re-running gave the numbers in this post: the same row improved by 40% on decode, with no config change at all.

If your box is a NAS, the load you're benchmarking against is often your own array. Check cat /proc/mdstat before you trust a single t/s.

How I actually measure it (and why the loop looks the way it does)#

The 1.1 t/s disaster and every number below come from the same ~50-line script, because "how fast is the model" is a meaningless question without "doing what, and reading which number off what." Three rules, baked into the loop:

  1. Drive prefill and decode apart. A model has two speeds — reading your prompt (compute-bound, parallel, hundreds of t/s) and writing the answer (bandwidth-bound, serial). Lumping them into one "tokens/s" hides which one you're fixing. The script hits llama-server's raw /completion endpoint and reports prompt_n / prompt_ms and predicted_n / predicted_ms separately, straight from the server's own timings block, so both columns are the engine's real measurement, not my stopwatch.
  2. Measure cold, on a controlled corpus. cache_prompt: False forces the server to re-read the whole prompt every time instead of serving it from KV cache — otherwise prefill speed is fiction. The prompt is one fixed sentence repeated N times to hit a target token count, so the sweep walks context depth cleanly from "single question" to "whole codebase in one go". (The table lists the server's own prompt_n per row, not my nominal target — a tokenizer calibration bug in my first run is precisely how the parity-sync problem above was caught twice.)
  3. Pin the sampling so decode is comparable. temperature: 0 makes generation deterministic — the model writes the same number of tokens each run, so decode t/s differences across context sizes mean bandwidth, not a different-length answer.
import json, urllib.request, time

BASE, MODEL = "http://localhost:8081", "qwen38-flash-next"
# one fixed sentence (~33 tokens); repeated to reach each target
SEED = "The suspension of the ignis8 has been upgraded with 1.8 springs and the shock absorbers were lowered to 220mm. "

def completion(prompt, **kw):
    payload = {"prompt": prompt, "temperature": 0, "max_tokens": 128,
               "cache_prompt": False, "stream": False, **kw}
    req = urllib.request.Request(BASE + "/completion",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=1800) as f:   # long timeout: cold 60k runs take minutes
        return json.load(f)

def tps(d):                                                 # read the server's OWN timings, not wall clock
    t = d.get("timings", {})
    pre = t["prompt_n"]   / (t["prompt_ms"]   / 1000)
    dec = t["predicted_n"]/ (t["predicted_ms"]/ 1000)
    return t["prompt_n"], pre, dec

for target in (256, 512, 1024, 2048, 4096, 8192,
               12288, 16384, 24576, 32768, 49152, 65536):
    reps = max(1, int(target / 33))                         # ~33 tokens per SEED line
    d = completion(SEED * reps + " What is 12*12?")
    pn, pre, dec = tps(d)
    print(f"ctx {target:5d}  prompt {pn:5d} tok  "
          f"prefill {pre:6.1f} t/s   decode {dec:6.1f} t/s")

The loop writes each row to a JSONL file so a run survives a dropped SSH session, and a second half fires one image through /v1/chat/completions to confirm the vision projector works end-to-end. The raw sweep deliberately uses /completion (no chat template, no thinking) — mixing a reasoning model's variable-length monologue into a "decode speed" number is how you end up comparing apples to weather. The fine steps are deliberate too: bandwidth effects don't announce themselves at round numbers, and a coarse sweep averages away the exact depth where decode starts to sag.

Three sweeps: A (fit on), B (fit off), Final (production)#

Config A is ryan4yin's tuned flag set — including his headline finding that llama.cpp's auto-offload (--fit on) beats manual expert placement. Config B flips exactly that one decision (plus the thread count stayed at 8). Final is B deployed through the production compose at ctx 65536. All runs: 96 GiB, parity idle, same machine, same day.

Prompt tokens A pre / dec B pre / dec Final pre / dec
220 36,8 / 8,8 32,9 / 8,8 38,3 / 8,9
460 68,6 / 8,7 68,2 / 9,1 68,0 / 9,2
940 121,0 / 8,8 120,2 / 9,2 120,1 / 9,1
1.870 196,1 / —¹ 194,8 / 9,3 194,8 / 9,2
3.730 209,9 / 8,5 208,6 / 9,2 208,3 / 9,0
7.450 213,2 / 8,5 212,0 / 8,8 211,5 / 8,6
11.170 211,8 / 8,0 211,6 / 8,7 210,2 / 8,6
14.890 211,0 / 8,1 210,0 / 8,4 209,8 / 8,4
22.330 217,9 / 7,8 217,2 / 8,1 217,5 / 8,1
29.770 211,4 / 7,5 211,0 / 7,6 210,7 / 7,9
44.680 206,9 / 6,9 206,3 / 7,0 206,5 / 7,2
59.560 198,4 / 6,4 196,7 / 6,7 198,0 / 6,8
vision (1.053)² 2.128 / 8,2 82,9 / 8,7 89,0 / 9,2

¹ Run A produced zero output tokens at this depth — a degenerate completion on the repetition corpus, reproduced three times. In Run B the same prompt generates normally. Whether an offload strategy can change that is above my pay grade; that it did is in the record. ² A's 2.128 prefill is a KV prefix-cache hit from an earlier identical request, not a GPU miracle — B and Final show what cold image reading actually costs (~85 t/s). Decode improved 8,2 → 9,2 across the trilogy: the production config is the best of the three.

xychart-beta
  title "Decode t/s across the context ladder (Final run)"
  x-axis [220, 460, 940, "1.9k", "3.7k", "7.5k", "11.2k", "14.9k", "22.3k", "29.8k", "44.7k", "59.6k"]
  y-axis "tokens/s" 6 --> 10
  bar [8.9, 9.2, 9.1, 9.2, 9.0, 8.6, 8.6, 8.4, 8.1, 7.9, 7.2, 6.8]
  line [8.8, 9.1, 9.2, 9.3, 9.2, 8.8, 8.7, 8.4, 8.1, 7.6, 7.0, 6.7]

Bars: production (Final) decode; line: Run B for reference. The slope is the bandwidth ceiling tilting as KV grows — 26 % end to end, no knee.

xychart-beta
  title "Prefill t/s: the plateau at ~210 (Final run)"
  x-axis [220, 460, 940, "1.9k", "3.7k", "7.5k", "11.2k", "14.9k", "22.3k", "29.8k", "44.7k", "59.6k"]
  y-axis "tokens/s" 0 --> 230
  line [38.3, 68.0, 120.1, 194.8, 208.3, 211.5, 210.2, 209.8, 217.5, 210.7, 206.5, 198.0]

Prefill pays fixed startup cost on small prompts, then flatlines — more threads don't move this plateau, it's the memory channel, not the CPU.

Verdict: --fit off --cpu-moe -ngl 99 wins every meaningful row (+5–8 %, never worse), and that is the opposite of what the gist's 4090 measurements said. The reconciliation: with 24 GiB of VRAM, auto-fit has room to offload experts and keep everything else warm; with 12 GiB it has to choose, and manual --cpu-moe (all experts in RAM, the entire attention stack in VRAM) is the better deal. His numbers were real — they just weren't for my box. Which is the same lesson the parity sync taught, one layer up.

The thread-count trap: same knob, opposite directions#

With the strategy decided, the remaining free variable was CPU threads (-t) — and the probe matrix says "it depends on what you do with the other knobs". A three-point mini-sweep (940 / 7.450 / 29.770 prompt tokens), run separately after each container change:

Config t=8 t=12 t=16
--fit on 8,3 / 8,2 / 7,4 — 8,9 / 8,7 / 7,8
--fit off --cpu-moe 9,2 / 8,8 / 7,6 8,2 / 8,0 / 7,1 (−10 %) not run

Fit-on gains ~6 % from doubling threads; fit-off loses 10 % going from 8 to 12. The explanation that fits: with experts in RAM and attention on GPU (fit-off), eight threads already saturate the channel's random-access throughput — more threads just add contention and latency to a memory-bound queue. Under fit-on, part of the expert work sits elsewhere and threads have real slack. The gist's "leave some cores for the OS" advice reads as wisdom again at these thread counts. Production answer: fit-off with -t 8, the fastest config this box has ever produced.

The serving stack, as measured#

The blunt docker run that first got the model answering on 64 GB (experts by hand, --fit off, -c 32768) remains in the git history; the tuned container that survived all three sweeps is this:

# Qwen3.8-Flash-Next llama-server — adapted from ryan4yin's RTX 4090 gist,
# retuned and A/B-verified on: Xeon E5-2630L v4 (10C/20T, quad DDR4-2133 = 68,3 GB/s ceiling),
# RTX 3060 12GB, 96GB RAM, Unraid.
#
# Model: AtomicChat AD-4.27bpw-Q4_K_M-M64 merge (~94,5 GB single file, 89,5% top-1)
# instead of the gist's unsloth UD-IQ3_XXS (~82 GB, 83,9%). The "M64" build keeps the
# 51B n-gram table mmap-able (~38 GB on SSD, ~2,7 KB random read/token) — so NEVER add
# --no-mmap, and budget: 55 GB experts + OS must fit into 96 GB with cache slack.
services:
  llama-flash-next:
    image: ghcr.io/ggml-org/llama.cpp:full-cuda # gist used full-cuda13; CUDA13 needs very new drivers — Ampere stays on 12
    container_name: llama-flash-next
    ports:
      - "8081:8001"
    volumes:
      - /mnt/llm:/models
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
    command:
      - --server # full-cuda entrypoint is a tool dispatcher, must specify explicitly
      - --model
      - /models/flashnext/flashnext-merged.gguf # AD-4.27bpw merge (89,5%); gist: UD-IQ3_XXS (83,9%)
      - --mmproj
      - /models/flashnext/mmproj-Qwen3.8-Flash-Next-F16.gguf # vision on; ~900MB VRAM -> with mmproj realistic ctx ceiling ~96K on the 3060
      - --alias
      - qwen38-flash-next # clean model ID; without it the literal file path collides with every client's provider/model addressing
      - --ctx-size
      - "65536" # smaller KV alloc -> more weight offload (gist: +8% decode). Request > ctx -> HTTP 400; clients keep their limit <= this. Stepwise 98K -> 131K planned, ~2,4 GiB KV q8 at 98K.
      # offload strategy — MEASURED (A/B sweep, 96GB, parity off):
      #   A --fit on:                  decode 8,8 -> 6,4 t/s (short -> 60k ctx)
      #   B --fit off --cpu-moe -ngl 99:  8,8 -> 6,7 t/s, wins every row (+5-8%)
      #   => B, and it is what the M64 build was designed for. The gist's "fit on wins"
      #      holds on 24GB cards; on 12GB the GPU budget is better spent keeping the
      #      entire attention stack in VRAM with experts in RAM.
      - --fit
      - "off"
      - --cpu-moe
      - -ngl
      - "99"
      - -t
      - "8" # thread matrix: fit-off t8 (9,2) beats fit-off t12 (8,2, -10%); fit-on t16 reaches only 8,9. fit-off + t8 = fastest found
      - -tb
      - "12" # prefill plateaus at ~212 t/s regardless (CPU compute ceiling) — kept as is
      - -b
      - "2048" # gist sweet spot was 6144 on the 4090; compute buffer grows with ub^2 -> 6144 would eat the 3060's VRAM. Keep -b == -ub or big batches split into a slow tail.
      - -ub
      - "2048"
      - --reasoning-budget
      - "4000" # hard cap on thinking tokens per turn — see "the thinking-budget trap"
      - --chat-template-kwargs
      - '{"reasoning_effort": "medium"}' # defaults to xhigh (wastes tokens); "high" effort measured to degrade quality (self-doubt loops)
      - --reasoning-preserve # pass reasoning history back to model across turns
      - --reasoning-budget-message
      - "... reasoning budget exceeded, need to answer.\n"
      # Sampling (unsloth recommended for thinking mode): temp=1.0, top_p=0.95, top_k=20, min_p=0.0, presence=0.0
      - --temp
      - "1.0"
      - --top-p
      - "0.95"
      - --top-k
      - "20"
      - --min-p
      - "0.00"
      - --presence-penalty
      - "0.0"
      - --flash-attn
      - "on" # required for quantized KV; saves VRAM (gist: ~1,5 GB vs F16)
      - --cache-type-k
      - q8_0 # gist measured q4_0 KV: dequant overhead cancels the VRAM gain (28,9 vs 30,1 t/s @32K) -> keep q8_0
      - --cache-type-v
      - q8_0
      - --image-max-tokens
      - "4000"
      - --image-min-tokens
      - "1024" # Qwen-VL requires at least 1024 image tokens
      - --cont-batching
      - --host
      - "0.0.0.0"
      - --port
      - "8001"
      - --api-key
      - "<set-your-own>" # a server bound to 0.0.0.0 WITHOUT --api-key now self-generates one and 401s everything — set yours deliberately, publish only through a TLS reverse proxy, /health may stay open for monitoring
      - --metrics
      - --log-verbosity
      - "3" # print_timing (prefill/decode split per request) — what the benchmark loop reads
      - -np
      - "1" # single slot -> max per-slot context; every parallel slot reserves its own KV on top
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    shm_size: "1gb" # /dev/shm defaults to 64 MB in Docker; llama.cpp's CUDA path wants larger shared buffers and fails quietly. tmpfs is usage-based, so the reservation is nominal.
    restart: unless-stopped

The two decisions the gist supplied that my measurements confirmed rather than overturned: q8_0 KV with flash-attn (the q4_0 dequant tax is real, and 24 KB/token hybrid KV is small enough not to fight over), and -b == -ub 2048 as the right batch shape for a 12 GB card — the prefill plateau at 212 t/s was completely insensitive to everything else, so batch tuning ends where the Xeon's AVX2 starts.

The thinking-budget trap (and the sampling that feeds it)#

Flash-Next is a reasoning model: it thinks out loud before answering, and that monologue is billed against your output budget. My first programmatic call asked for max_tokens: 80 and got back an empty string — all 80 tokens went into hidden deliberation and none survived for the reply. The thinking arrives in a separate reasoning_content field, so nothing is technically lost, but a client that only reads content sees silence.

Three compose settings work together against this. --reasoning-budget 4000 caps deliberation per turn, and --reasoning-budget-message is what gets injected when the cap is hit — a forced "you must answer now." reasoning_effort: medium (over the xhigh default) trims the monologue; the counterintuitive gist finding — which our daily use has since confirmed — is that raising effort beyond medium actively hurts answers, because the model falls into self-doubt loops. And the thinking-mode sampling set (temp 1.0, top_p 0.95, top_k 20, min_p 0.0) is Unsloth's recommendation; the familiar instruct defaults (0.7, presence 1.5) measurably degrade a reasoning model. --reasoning-preserve feeds earlier turns' thinking back in, which matters once you're running an agent across a whole codebase.

Context, and why this model is nearly free to keep open#

A classic attention-heavy model spends VRAM on the KV cache proportional to context — roughly 100–300 KB per token, so a 128k window costs 4–8 GiB of memory you didn't plan for. My old Qwen3.6 run pinned its GPU at ~10.2 GiB from 8k to 128k just to hold that reservation.

Flash-Next uses hybrid attention: of its 48 layers, only 12 keep a per-token KV cache; the other 36 are recurrent Gated-DeltaNet states that cost the same whether you're at 4k or 256k tokens. Result: ~24 KB of KV per 1k tokens — a 65k window needs about 1.6 GiB. The sweeps confirm the economics from the inside: decode falls a gentle, knee-less 26 % across the entire 60k range (9,2 → 6,8), exactly the profile hybrid attention should produce — the old dense models sagged 30-40 % within a tenth of that depth.

Practical: you can afford a large window, but the ~212 t/s prefill plateau prices "read everything" at five minutes per 60k tokens — once per session fine, per turn never. Let the agent grep and read the 5–15k tokens it needs, keep 60k+ for the occasional whole-project read, and keep the server warm: the vision row above shows the prefix cache turning a 60-second prompt re-read into a ~0,5-second one.

Predicted vs. measured: the scorecard#

This is the section I wish every hardware blog had, so here's mine. Before installing the RAM, every number in this post was a prediction from three inputs: the bandwidth model (t/s = effective GB/s ÷ GB per token), the gist's 4090 measurements, and free -h.

Question Prediction (before) Measurement (after) Grade
Decode on 96 GiB 10–18 t/s 9,2 peak, 6,8 @60k wrong by ~30 %, right order of magnitude
Why assumed ~45 GB/s effective; 3,2 GB/token ~31 GB/s achieved; ~3,4 GB/token incl. KV+router the model is honest, my inputs weren't
Decode knee "somewhere above 32k" none — smooth 26 % decline predicted wrong shape, right story (hybrid attention)
Prefill plateau ~80–150 t/s 212 t/s pleasantly wrong: better
--fit on wins (per gist) yes, expected no — opposite on 12 GB the transfer that failed; A/B saved it
-t 16 > -t 8 yes, +6 % (measured under fit-on) fit-off: t16 untested, t12 loses 10 % same knob, sign flip; strategies interact
64 GB thrashes ~1–2 t/s expected 1,1 measured nailed it
Parity sync effect unknown −40 % decode learned the hard way, twice

The biggest miss — 9,2 instead of ~15 — decomposes cleanly: the chip's 68,3 GB/s at 2133 (not the 85 I'd assumed for "DDR4-2666 quad"), times ~65 % real-world gather throughput (RDIMM bank churn, 4 KB expert reads, flex-mode imbalance on a 2+2+1+1 population), is 31 GB/s — which divided by 3,4 GB per token is exactly the 9,2 we measure. The bandwidth model didn't fail. My three coefficients in it did. Every one of them is the kind of number no spec sheet prints and only measurement prints — which is the entire argument for owning the benchmark loop, not just the model.

Everything in this table, though, was measured in a lab: same box, but controlled prompts, forced-cold cache, benchmark loop, parity sync paused. What the machine does when it is not being benchmarked is no longer anecdote either — the server's own cumulative counters close this post, and they are the half of the evidence I'd trust more.

What's next#

  1. Context ladder: -c 98304 is live now (the compose above shows 65536 — that was sweep day). The rule from the compose comment held: every client keeps its own limit below the server's number — opencode caps this model at 90k context + 8k output, so the sum fits inside 98304 instead of risking an HTTP 400 at the wall. Next rung is 131072, watching VRAM: KV costs ~2,4 GiB at 98K q8 with mmproj on board.
  2. MTP watch item: the model ships a multi-token-prediction head that could multiply decode speed, but llama.cpp still OOMs on --spec-type draft-mtp (/slots still reports speculative: false). When upstream supports it, re-measure — potentially the biggest single win left on the table. No benchmark run needed to prove it, either: /metrics will show the new lifetime average the next morning.

The short version: a frontier-class 177B model now fits this machine's budget for the price of two used DIMM kits, and the distance between 1 t/s and 9,2 t/s was measured — not guessed — with a parity sync paused, an A/B plan, and a fifty-line loop that always read the server's own numbers. The bandwidth model is real; its coefficients are borrowed until you measure them. Mine say: 2133, flex, thirty-one gigs a second, and a very patient Xeon.

The everyday, in the server's own numbers#

No sweep can capture what the box does all day — but it doesn't have to ask. The compose already passes --metrics, which exposes a Prometheus endpoint at the server root (not under /v1, easy to miss if you only know the OpenAI surface), and a second endpoint, /slots, shows what each context slot is doing right now. The counters accumulate since start-up:

curl -s http://localhost:8081/metrics | grep '^llamacpp:'
llamacpp:prompt_tokens_total              758054   # prompt tokens actually computed
llamacpp:prompt_tokens_cached_total     4366410   # prompt tokens served from prefix cache
llamacpp:prompt_seconds_total              5729
llamacpp:tokens_predicted_total            183669  # generated tokens (answer + thinking)
llamacpp:tokens_predicted_seconds_total   24824
llamacpp:predicted_tokens_seconds             7.4  # live rate of in-flight requests, not cumulative

A little division turns them into lifetime averages — and by now there is a real population behind them: opencode and the 5-o'clock journal agent came back from the cloud fallback to this box, and their traffic is what these counters count.

Cumulative since start Lifetime average What it says about the everyday
758k computed prompt tok / 5.729 s prefill 132 t/s Real agent prompts are 500–3k tokens — small prompts still paying their fixed start-up cost, so the everyday average sits far below the 212 t/s plateau. Prefill taxes context switching, not context size
184k generated tok / 6,9 h of decode time decode 7,4 t/s Mid-corridor of the sweep's 6,8–9,2 — not below it. Fresh-turn samples on the same machine run ~9 t/s on fluent prose and ~8,5 t/s on source code; the 6 % between them is token mix (code activates different expert subsets and longer generations), not configuration
4,37M cached vs 758k computed 85 % of prompt traffic served from prefix cache The "nearly free to keep the context open" claim verified on production traffic: six out of seven prompt tokens in a warm agent loop are never re-prefilled at all

That last row deserves a face: the request that was in flight while I took this snapshot held 65k prompt tokens in its slot — 55k of them served from the prefix cache, 85 % mid-sentence. And 184k generated tokens is roughly 135k words: the box has quietly written a novel's worth of text since boot, at the rate of someone typing about as fast as I read.

The benchmark loop deliberately measures the worst case (cache_prompt: False); daily workloads ride the server's best case. When the question is "is a box fast enough to live on," the cumulative counters — not the controlled sweep — are the honest number. Mine say the corridor between 6,8 and 9,2 t/s isn't just the model's weather; it's the whole forecast — and for a machine that once thrashed at 1,2 t/s on this very hardware, the forecast is mild.

What a token costs: the power at the wall#

Speed answers "can I live on this box." Energy answers "what does living on it cost" — and that one I metered, with the server hanging off a Zigbee measurement plug. The first attempt had to be thrown away, because the array was spinning and parity-clean work was still running: the lesson of section four, this time in watts. So: disks in standby, model loaded, page cache warm — 87,5 W idle baseline. Then the whole sweep, once more, but this time each request logs absolute timestamps for its prefill and decode window, and the plug is polled once a second, so every watt-sample lands on a phase.

Phase Rate Net draw above idle Energy per token ×0,40 €/kWh
prefill, plateau ~212 t/s ~105 W ~0,5 J 5,6 €-ct / Mtok
decode, fresh turns 9,2–9,4 t/s ~77 W 8,3 J 0,92 € / Mtok
decode, 33k context 7,9 t/s ~100 W 12,8 J 1,42 € / Mtok
decode, 66k context 6,8 t/s ~122 W ~18 J¹ ~2 € / Mtok
prefix-cache read — ≈ idle ≈ 0 J free

¹ The plug reports with ~6 s lag; on the deepest row the tail of a 283-second prefill smears into the decode window. Upper bound, not gospel.

Three stories in that table. The first is ratio, not absolute: prefill runs at ~0,5 J/token, decode at 8–18 — generation is ~20× the energy of reading, same split as the speed difference, but now in joules. It makes the warm-agent economics concrete: the whole 198k-token sweep (almost all prompt, by construction) burned 32,5 Wh ≈ 1,3 €-cent; a blended million tokens through this kind of traffic is worth less than two cents. The prefix-cache path — six out of seven tokens of everyday traffic — is energetically a rounding error.

The second story is the uncomfortable one for the localist camp: household electricity doesn't beat the API. Cloud pricing for the hosted equivalent of this model sits around 47 US-cents per million generated tokens; my wall meter asks 0,9–1,6 € for the same joules, before any of the amortization the datacenter also pays. Tariff is not a hardware property: whatever efficiency advantage local weights might have is smaller than the gap between a home rate and a colocation one. What the local box actually buys is the cache path that bills ~zero, plus privacy, no rate limits and a server that never changes its mind about you mid-project. Worth knowing that marginal energy isn't on the list.

The third story is the one that moved my billing config: the hardware itself. The dedicated parts — the 3060, the 96 GiB of DIMMs, the CPU, the scratch NVMe — cost about 300 €; spread over two years that is 0,41 € a day, and since decode owns roughly four-fifths of the busy clock, the generation pays ~0,33 € of it. At the volume I expect once opencode runs against this box as its default (call it 175k generated + 600k computed prompt tokens per day), that adds ~2,2 US-dollar per million generated tokens on top of the electricity — and my opencode cost block now honestly reads output 3,60 USD, input 0,22 USD, cache read 0,02 USD per Mtok. Which means the honest per-token price here is not a hardware constant but a utilization constant: right now, with only background agents generating ~40k tokens a day, depreciation alone is ~10 USD/Mtok — absurd. At 175k it is 2,2. And if the box served a single saturated 24 h slot — 9,2 t/s, 795k tokens a day — it collapses to ~0,5 USD/Mtok, which is almost exactly the API price. Self-hosting loses on marginal dollars at every realistic utilization; it wins on the cache path, privacy and availability, and it is those wins, dressed up as an energy question, that this machine was really bought for.

For scale, all-in: at the planned daily traffic the box's true cost is roughly 0,6 € a day — twenty cents of electricity, forty cents of depreciation — the price of one coffee a week, no longer a refill. For the traffic this box has actually served since boot, the novel's worth plus everything computed to read it cost twenty to thirty euro cents in joules and about half a coffee in depreciation. Mild weather, hardware rain — but I'd still check the forecast from this machine every day.

Update, 2026-10-10. Two threads from this post have since resolved. The MTP watch item: the reason --spec-type draft-mtp never loaded is that the upstream GGUF converter drops the model's MTP head — no flash-next quantization on disk carries it. And the speed question moved on: the same IQ3_XXS file now runs on Strata, a different engine, at four times the generation speed on this exact box — the follow-up post has the numbers, the dead ends, and the corrected plateau diagnosis. Everything measured in this post stays valid for the engine it was measured on.

Last updated: