Chasing Flash-Next: bringing a 95 GB Qwen model to a 96 GB homelab

A few days ago I documented how Qwen3.6-35B-A3B found a home on my Unraid box — a decade-old Xeon, 64 GiB of ECC DDR4 and an RTX 3060 with 12 GB of VRAM. This post is the sequel: the move up to Qwen3.8-Flash-Next, the 125B-parameter mixture-of-experts preview of the Qwen4 architecture. Where the first post was a triumph of fitting a small model into a small box, this one is about the arithmetic of a big one — how a 94.5 GB file loads onto a 96 GB machine, why it crawled at one token per second the week before, and exactly what I measured to prove the answer.

All measurements are done: the 96 GiB is installed, three full sweeps (A, B and final production) plus a thread-count probe matrix sit in the tables below. My bandwidth model predicted 10–18 t/s; the machine delivers 9,2 — and the last section explains exactly where my predictions went wrong, because that part is the useful one.

The hardware#

Xeon E5-2630L v4 — ten Broadwell cores (20 threads) from 2016, on four-channel DDR4 ECC. And that's the first thing the spec sheets got wrong for me: the modules are 2666-MT/s parts (two Hynix, two Atermiter, two more added later), but the L SKU's integrated memory controller tops out at DDR4-2133 — a hard 68,3 GB/s ceiling set by the CPU. Any faster RAM you buy for this chip runs at exactly this speed. RTX 3060 with 12 GiB of GDDR6 on a PCIe 3.0 x8 slot. A 256 GiB NVMe for scratch. Unraid on a Beikong BKHD-2011MATX industrial board, serving llama.cpp containers.

The board came with its own curiosity: with all six 16 GiB modules installed, the BIOS reports only four in SMBIOS (dmidecode shows 4×16 GB = 64 — while the kernel's memory map says 94 GiB and is right). Six modules across four channels is unavoidably a 2+2+1+1 population, so the box runs in Intel's flex mode: quad-channel interleaving for the first rank of every channel, dual for the overflow. Nothing a manual told me; everything free -h, dmidecode and a benchmark loop together did.

One detour worth pre-empting on the GPU: people panic when nvidia-smi shows the 3060 on a x8 slot and imagine half the card is offline. It isn't — only activations and the KV cache cross PCIe; the weights that live in VRAM never re-travel the bus. The link sat under 1% utilization in my runs. The GPU slot is a non-issue for this class of model; system RAM is where everything is decided.

The model and the trick that makes it look possible#

Flash-Next is 177B parameters at rest, 6B active per token. The naive math said "impossible below ~128 GB" — and that's exactly what I believed until I found AtomicChat's architecture-aware dynamic GGUFs. Their AD-4.27bpw-Q4_K_M-M64 build is a 94.5 GB file that claims to need only 54.5 GB resident.

The reason is one specific design choice worth understanding, because it decides what the SSD can and cannot do for you:

  • 51B of the parameters are an n-gram embedding table — a lookup the model hits once per token, reading ~2.7 KB at a hash-derived address. That is 3 MB/s of random reads at 36 tokens/s. A consumer NVMe answers one of those in under 100 µs against a ~30 ms token budget. This part of the file can live on disk and be paged in as needed.
  • The routed experts are different physics: every token touches ~6B active parameters, gigabytes of random reads per token. If those ever hit the SSD, you're not running an LLM, you're running a swap demo. They must be RAM-resident, no exceptions.

AtomicChat's quants exploit exactly that split: the n-gram table sits in its own shard, the OS memory-maps it, and only the 54.5 GB of "hot" weights are pinned. Which is how a 177B model runs on a 64 GB MacBook — the file is larger than the machine's entire RAM, and 39 GB of it never enters memory at all.

The corollary, and the whole thesis of this post: the SSD trick is a one-table trick. It buys you room to keep the experts in RAM by moving the one component that's cheap to page to disk. It does not make the experts pageable. Get those two numbers mixed up and the model runs at a single token per second — which is precisely what happened to me first.

So here's the thing: 54.5 GB + OS > 64 GB#

Before the RAM arrived, I loaded the model on the 64 GiB configuration anyway, to watch it fail. The model did load. It answered. And it measured like this:

Test Prompt tokens Prefill Decode
512-window request 861 10.4 t/s 1.1 t/s
2k-window request 3,451 16.5 t/s 1.4 t/s

One token per second on hardware that had run Qwen3.6 at 35 t/s. Nothing was broken — the arithmetic was simply against me: 54.5 GB of hot weights + ~8 GB for Unraid and containers leaves the page cache ~2 GB of headroom for everything else the experts need to touch. The kernel constantly evicted expert pages, and every token read some of them back from disk. The mechanism deserves its exact name, because it matters later: those 54.5 "resident" GB are mmap'd file pages — reclaimable cache, not pinned memory. free -h shows them under "buff/cache", not "used", and under pressure the kernel hands them back without asking. "Fits in RAM" means fits with enough slack that nobody ever wants them back — a very different bar.

Configuration Resident set Page-cache headroom Decode t/s (short ctx)
64 GiB RAM (measured) 54.5 GB + ~8 OS ~2 GB — evicting 1,1
96 GiB, --fit on (Run A) " ~33 GB — stable 8,8
96 GiB, --fit off --cpu-moe (Run B = production) " ~33 GB — stable 9,2

An early, expensive lesson: never benchmark during a parity sync#

The very first 96 GiB sweep happened to run while Unraid was mid-parity check — and it read "honestly": 5–7 t/s decode, and a 60k-token row that took 542 seconds. The numbers were internally consistent, the model answered fine, and every one of them was fiction. Parity work saturates exactly the two resources this model class lives on: disk I/O for the mmap'd n-gram table, and the page cache for the expert set — the kernel happily evicts "cold" model pages to make room for stripe traffic. Pausing the sync and re-running gave the numbers in this post: the same row improved by 40% on decode, with no config change at all.

If your box is a NAS, the load you're benchmarking against is often your own array. Check cat /proc/mdstat before you trust a single t/s.

How I actually measure it (and why the loop looks the way it does)#

The 1.1 t/s disaster and every number below come from the same ~50-line script, because "how fast is the model" is a meaningless question without "doing what, and reading which number off what." Three rules, baked into the loop:

  1. Drive prefill and decode apart. A model has two speeds — reading your prompt (compute-bound, parallel, hundreds of t/s) and writing the answer (bandwidth-bound, serial). Lumping them into one "tokens/s" hides which one you're fixing. The script hits llama-server's raw /completion endpoint and reports prompt_n / prompt_ms and predicted_n / predicted_ms separately, straight from the server's own timings block, so both columns are the engine's real measurement, not my stopwatch.
  2. Measure cold, on a controlled corpus. cache_prompt: False forces the server to re-read the whole prompt every time instead of serving it from KV cache — otherwise prefill speed is fiction. The prompt is one fixed sentence repeated N times to hit a target token count, so the sweep walks context depth cleanly from "single question" to "whole codebase in one go". (The table lists the server's own prompt_n per row, not my nominal target — a tokenizer calibration bug in my first run is precisely how the parity-sync problem above was caught twice.)
  3. Pin the sampling so decode is comparable. temperature: 0 makes generation deterministic — the model writes the same number of tokens each run, so decode t/s differences across context sizes mean bandwidth, not a different-length answer.
import json, urllib.request, time

BASE, MODEL = "http://localhost:8081", "qwen38-flash-next"
# one fixed sentence (~33 tokens); repeated to reach each target
SEED = "The suspension of the ignis8 has been upgraded with 1.8 springs and the shock absorbers were lowered to 220mm. "

def completion(prompt, **kw):
    payload = {"prompt": prompt, "temperature": 0, "max_tokens": 128,
               "cache_prompt": False, "stream": False, **kw}
    req = urllib.request.Request(BASE + "/completion",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=1800) as f:   # long timeout: cold 60k runs take minutes
        return json.load(f)

def tps(d):                                                 # read the server's OWN timings, not wall clock
    t = d.get("timings", {})
    pre = t["prompt_n"]   / (t["prompt_ms"]   / 1000)
    dec = t["predicted_n"]/ (t["predicted_ms"]/ 1000)
    return t["prompt_n"], pre, dec

for target in (256, 512, 1024, 2048, 4096, 8192,
               12288, 16384, 24576, 32768, 49152, 65536):
    reps = max(1, int(target / 33))                         # ~33 tokens per SEED line
    d = completion(SEED * reps + " What is 12*12?")
    pn, pre, dec = tps(d)
    print(f"ctx {target:5d}  prompt {pn:5d} tok  "
          f"prefill {pre:6.1f} t/s   decode {dec:6.1f} t/s")

The loop writes each row to a JSONL file so a run survives a dropped SSH session, and a second half fires one image through /v1/chat/completions to confirm the vision projector works end-to-end. The raw sweep deliberately uses /completion (no chat template, no thinking) — mixing a reasoning model's variable-length monologue into a "decode speed" number is how you end up comparing apples to weather. The fine steps are deliberate too: bandwidth effects don't announce themselves at round numbers, and a coarse sweep averages away the exact depth where decode starts to sag.

Three sweeps: A (fit on), B (fit off), Final (production)#

Config A is ryan4yin's tuned flag set — including his headline finding that llama.cpp's auto-offload (--fit on) beats manual expert placement. Config B flips exactly that one decision (plus the thread count stayed at 8). Final is B deployed through the production compose at ctx 65536. All runs: 96 GiB, parity idle, same machine, same day.

Prompt tokens A pre / dec B pre / dec Final pre / dec
220 36,8 / 8,8 32,9 / 8,8 38,3 / 8,9
460 68,6 / 8,7 68,2 / 9,1 68,0 / 9,2
940 121,0 / 8,8 120,2 / 9,2 120,1 / 9,1
1.870 196,1 / —¹ 194,8 / 9,3 194,8 / 9,2
3.730 209,9 / 8,5 208,6 / 9,2 208,3 / 9,0
7.450 213,2 / 8,5 212,0 / 8,8 211,5 / 8,6
11.170 211,8 / 8,0 211,6 / 8,7 210,2 / 8,6
14.890 211,0 / 8,1 210,0 / 8,4 209,8 / 8,4
22.330 217,9 / 7,8 217,2 / 8,1 217,5 / 8,1
29.770 211,4 / 7,5 211,0 / 7,6 210,7 / 7,9
44.680 206,9 / 6,9 206,3 / 7,0 206,5 / 7,2
59.560 198,4 / 6,4 196,7 / 6,7 198,0 / 6,8
vision (1.053)² 2.128 / 8,2 82,9 / 8,7 89,0 / 9,2

¹ Run A produced zero output tokens at this depth — a degenerate completion on the repetition corpus, reproduced three times. In Run B the same prompt generates normally. Whether an offload strategy can change that is above my pay grade; that it did is in the record. ² A's 2.128 prefill is a KV prefix-cache hit from an earlier identical request, not a GPU miracle — B and Final show what cold image reading actually costs (~85 t/s). Decode improved 8,2 → 9,2 across the trilogy: the production config is the best of the three.

Verdict: --fit off --cpu-moe -ngl 99 wins every meaningful row (+5–8 %, never worse), and that is the opposite of what the gist's 4090 measurements said. The reconciliation: with 24 GiB of VRAM, auto-fit has room to offload experts and keep everything else warm; with 12 GiB it has to choose, and manual --cpu-moe (all experts in RAM, the entire attention stack in VRAM) is the better deal. His numbers were real — they just weren't for my box. Which is the same lesson the parity sync taught, one layer up.

The thread-count trap: same knob, opposite directions#

With the strategy decided, the remaining free variable was CPU threads (-t) — and the probe matrix says "it depends on what you do with the other knobs". A three-point mini-sweep (940 / 7.450 / 29.770 prompt tokens), run separately after each container change:

Config t=8 t=12 t=16
--fit on 8,3 / 8,2 / 7,4 — 8,9 / 8,7 / 7,8
--fit off --cpu-moe 9,2 / 8,8 / 7,6 8,2 / 8,0 / 7,1 (−10 %) not run

Fit-on gains ~6 % from doubling threads; fit-off loses 10 % going from 8 to 12. The explanation that fits: with experts in RAM and attention on GPU (fit-off), eight threads already saturate the channel's random-access throughput — more threads just add contention and latency to a memory-bound queue. Under fit-on, part of the expert work sits elsewhere and threads have real slack. The gist's "leave some cores for the OS" advice reads as wisdom again at these thread counts. Production answer: fit-off with -t 8, the fastest config this box has ever produced.

The serving stack, as measured#

The blunt docker run that first got the model answering on 64 GB (experts by hand, --fit off, -c 32768) remains in the git history; the tuned container that survived all three sweeps is this:

# Qwen3.8-Flash-Next llama-server — adapted from ryan4yin's RTX 4090 gist,
# retuned and A/B-verified on: Xeon E5-2630L v4 (10C/20T, quad DDR4-2133 = 68,3 GB/s ceiling),
# RTX 3060 12GB, 96GB RAM, Unraid.
#
# Model: AtomicChat AD-4.27bpw-Q4_K_M-M64 merge (~94,5 GB single file, 89,5% top-1)
# instead of the gist's unsloth UD-IQ3_XXS (~82 GB, 83,9%). The "M64" build keeps the
# 51B n-gram table mmap-able (~38 GB on SSD, ~2,7 KB random read/token) — so NEVER add
# --no-mmap, and budget: 55 GB experts + OS must fit into 96 GB with cache slack.
services:
  llama-flash-next:
    image: ghcr.io/ggml-org/llama.cpp:full-cuda # gist used full-cuda13; CUDA13 needs very new drivers — Ampere stays on 12
    container_name: llama-flash-next
    ports:
      - "8081:8001"
    volumes:
      - /mnt/llm:/models
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=all
    command:
      - --server # full-cuda entrypoint is a tool dispatcher, must specify explicitly
      - --model
      - /models/flashnext/flashnext-merged.gguf # AD-4.27bpw merge (89,5%); gist: UD-IQ3_XXS (83,9%)
      - --mmproj
      - /models/flashnext/mmproj-Qwen3.8-Flash-Next-F16.gguf # vision on; ~900MB VRAM -> with mmproj realistic ctx ceiling ~96K on the 3060
      - --alias
      - qwen38-flash-next # clean model ID; without it the literal file path collides with every client's provider/model addressing
      - --ctx-size
      - "65536" # smaller KV alloc -> more weight offload (gist: +8% decode). Request > ctx -> HTTP 400; clients keep their limit <= this. Stepwise 98K -> 131K planned, ~2,4 GiB KV q8 at 98K.
      # offload strategy — MEASURED (A/B sweep, 96GB, parity off):
      #   A --fit on:                  decode 8,8 -> 6,4 t/s (short -> 60k ctx)
      #   B --fit off --cpu-moe -ngl 99:  8,8 -> 6,7 t/s, wins every row (+5-8%)
      #   => B, and it is what the M64 build was designed for. The gist's "fit on wins"
      #      holds on 24GB cards; on 12GB the GPU budget is better spent keeping the
      #      entire attention stack in VRAM with experts in RAM.
      - --fit
      - "off"
      - --cpu-moe
      - -ngl
      - "99"
      - -t
      - "8" # thread matrix: fit-off t8 (9,2) beats fit-off t12 (8,2, -10%); fit-on t16 reaches only 8,9. fit-off + t8 = fastest found
      - -tb
      - "12" # prefill plateaus at ~212 t/s regardless (CPU compute ceiling) — kept as is
      - -b
      - "2048" # gist sweet spot was 6144 on the 4090; compute buffer grows with ub^2 -> 6144 would eat the 3060's VRAM. Keep -b == -ub or big batches split into a slow tail.
      - -ub
      - "2048"
      - --reasoning-budget
      - "4000" # hard cap on thinking tokens per turn — see "the thinking-budget trap"
      - --chat-template-kwargs
      - '{"reasoning_effort": "medium"}' # defaults to xhigh (wastes tokens); "high" effort measured to degrade quality (self-doubt loops)
      - --reasoning-preserve # pass reasoning history back to model across turns
      - --reasoning-budget-message
      - "... reasoning budget exceeded, need to answer.\n"
      # Sampling (unsloth recommended for thinking mode): temp=1.0, top_p=0.95, top_k=20, min_p=0.0, presence=0.0
      - --temp
      - "1.0"
      - --top-p
      - "0.95"
      - --top-k
      - "20"
      - --min-p
      - "0.00"
      - --presence-penalty
      - "0.0"
      - --flash-attn
      - "on" # required for quantized KV; saves VRAM (gist: ~1,5 GB vs F16)
      - --cache-type-k
      - q8_0 # gist measured q4_0 KV: dequant overhead cancels the VRAM gain (28,9 vs 30,1 t/s @32K) -> keep q8_0
      - --cache-type-v
      - q8_0
      - --image-max-tokens
      - "4000"
      - --image-min-tokens
      - "1024" # Qwen-VL requires at least 1024 image tokens
      - --cont-batching
      - --host
      - "0.0.0.0"
      - --port
      - "8001"
      - --api-key
      - "<set-your-own>" # a server bound to 0.0.0.0 WITHOUT --api-key now self-generates one and 401s everything — set yours deliberately, publish only through a TLS reverse proxy, /health may stay open for monitoring
      - --metrics
      - --log-verbosity
      - "3" # print_timing (prefill/decode split per request) — what the benchmark loop reads
      - -np
      - "1" # single slot -> max per-slot context; every parallel slot reserves its own KV on top
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    shm_size: "1gb" # /dev/shm defaults to 64 MB in Docker; llama.cpp's CUDA path wants larger shared buffers and fails quietly. tmpfs is usage-based, so the reservation is nominal.
    restart: unless-stopped

The two decisions the gist supplied that my measurements confirmed rather than overturned: q8_0 KV with flash-attn (the q4_0 dequant tax is real, and 24 KB/token hybrid KV is small enough not to fight over), and -b == -ub 2048 as the right batch shape for a 12 GB card — the prefill plateau at 212 t/s was completely insensitive to everything else, so batch tuning ends where the Xeon's AVX2 starts.

The thinking-budget trap (and the sampling that feeds it)#

Flash-Next is a reasoning model: it thinks out loud before answering, and that monologue is billed against your output budget. My first programmatic call asked for max_tokens: 80 and got back an empty string — all 80 tokens went into hidden deliberation and none survived for the reply. The thinking arrives in a separate reasoning_content field, so nothing is technically lost, but a client that only reads content sees silence.

Three compose settings work together against this. --reasoning-budget 4000 caps deliberation per turn, and --reasoning-budget-message is what gets injected when the cap is hit — a forced "you must answer now." reasoning_effort: medium (over the xhigh default) trims the monologue; the counterintuitive gist finding — which our daily use has since confirmed — is that raising effort beyond medium actively hurts answers, because the model falls into self-doubt loops. And the thinking-mode sampling set (temp 1.0, top_p 0.95, top_k 20, min_p 0.0) is Unsloth's recommendation; the familiar instruct defaults (0.7, presence 1.5) measurably degrade a reasoning model. --reasoning-preserve feeds earlier turns' thinking back in, which matters once you're running an agent across a whole codebase.

Context, and why this model is nearly free to keep open#

A classic attention-heavy model spends VRAM on the KV cache proportional to context — roughly 100–300 KB per token, so a 128k window costs 4–8 GiB of memory you didn't plan for. My old Qwen3.6 run pinned its GPU at ~10.2 GiB from 8k to 128k just to hold that reservation.

Flash-Next uses hybrid attention: of its 48 layers, only 12 keep a per-token KV cache; the other 36 are recurrent Gated-DeltaNet states that cost the same whether you're at 4k or 256k tokens. Result: ~24 KB of KV per 1k tokens — a 65k window needs about 1.6 GiB. The sweeps confirm the economics from the inside: decode falls a gentle, knee-less 26 % across the entire 60k range (9,2 → 6,8), exactly the profile hybrid attention should produce — the old dense models sagged 30-40 % within a tenth of that depth.

Practical: you can afford a large window, but the ~212 t/s prefill plateau prices "read everything" at five minutes per 60k tokens — once per session fine, per turn never. Let the agent grep and read the 5–15k tokens it needs, keep 60k+ for the occasional whole-project read, and keep the server warm: the vision row above shows the prefix cache turning a 60-second prompt re-read into a ~0,5-second one.

Predicted vs. measured: the scorecard#

This is the section I wish every hardware blog had, so here's mine. Before installing the RAM, every number in this post was a prediction from three inputs: the bandwidth model (t/s = effective GB/s ÷ GB per token), the gist's 4090 measurements, and free -h.

Question Prediction (before) Measurement (after) Grade
Decode on 96 GiB 10–18 t/s 9,2 peak, 6,8 @60k wrong by ~30 %, right order of magnitude
Why assumed ~45 GB/s effective; 3,2 GB/token ~31 GB/s achieved; ~3,4 GB/token incl. KV+router the model is honest, my inputs weren't
Decode knee "somewhere above 32k" none — smooth 26 % decline predicted wrong shape, right story (hybrid attention)
Prefill plateau ~80–150 t/s 212 t/s pleasantly wrong: better
--fit on wins (per gist) yes, expected no — opposite on 12 GB the transfer that failed; A/B saved it
-t 16 > -t 8 yes, +6 % (measured under fit-on) fit-off: t16 untested, t12 loses 10 % same knob, sign flip; strategies interact
64 GB thrashes ~1–2 t/s expected 1,1 measured nailed it
Parity sync effect unknown −40 % decode learned the hard way, twice

The biggest miss — 9,2 instead of ~15 — decomposes cleanly: the chip's 68,3 GB/s at 2133 (not the 85 I'd assumed for "DDR4-2666 quad"), times ~65 % real-world gather throughput (RDIMM bank churn, 4 KB expert reads, flex-mode imbalance on a 2+2+1+1 population), is 31 GB/s — which divided by 3,4 GB per token is exactly the 9,2 we measure. The bandwidth model didn't fail. My three coefficients in it did. Every one of them is the kind of number no spec sheet prints and only measurement prints — which is the entire argument for owning the benchmark loop, not just the model.

And the model is now serving me for real: as I write this follow-up, day-to-day traffic runs at ~9 t/s on fluent prose and ~8,5 t/s on source code, i.e. the sustained rate sits right on the sweep's short-context peak, not below it. The last 6 % of the gap between those two numbers is token mix, not configuration: code activates different expert subsets and longer generations, and it costs bandwidth the prose doesn't. For a machine that thrashed at 1,2 t/s ten days ago, the corridor between 6,8 (deep context) and 9,2 (fresh turns) is simply its weather.

What's next#

  1. Context scaling: -c 65536 → 98304 → 131072 in steps, watching VRAM (KV costs ~2,4 GiB at 98K with mmproj) and keeping every client's limit under the server's.
  2. MTP watch item: the model ships a multi-token-prediction head that could multiply decode speed, but llama.cpp still OOMs on --spec-type draft-mtp. When upstream supports it, re-measure — potentially the biggest single win left on the table.
  3. Daily duty: opencode and the 5-o'clock journal agent switch from cloud fallback back to this box — the provider entries have been waiting since the setup week, and the reasoning-budget plumbing was built for exactly those runs.

The short version: a frontier-class 177B model now fits this machine's budget for the price of two used DIMM kits, and the distance between 1 t/s and 9,2 t/s was measured — not guessed — with a parity sync paused, an A/B plan, and a fifty-line loop that always read the server's own numbers. The bandwidth model is real; its coefficients are borrowed until you measure them. Mine say: 2133, flex, thirty-one gigs a second, and a very patient Xeon.