Why Local AI Feels Slow: A Beginner's Guide to Tokens, Prefill and the Memory Wall

TL;DR — A language model generates text one token at a time, and every single token requires reading huge chunks of the model through the chip. That makes memory bandwidth, not raw compute power, the speed limit of local AI. This post explains prefill vs. decode, why "27B parameters" can be fast but "125B" can be slower, what quantization does, how the context window eats VRAM, and how to predict before you download anything whether a model will give you 40 tokens per second – or 1.1. The numbers are all real measurements from my own box: a 12 GB RTX 3060 and (soon) 96 GB of RAM.

1. One request, two completely different lives#

When you hit "send", the model does two fundamentally different jobs back to back. Beginners mix up the two numbers, which leads to a lot of confusion when comparing machines – so let's get this straight first.

Prefill is the model reading your prompt. All prompt tokens are processed in parallel, in big matrix batches. It's compute-heavy and your GPU (or CPU with enough cores) can chew through it quickly – hundreds of tokens per second. You notice it as the pause between sending and the first word appearing.

Decode (or generation) is the model writing its answer. This is the part that crawls along one token at a time: the model must finish token 58 before token 59 even exists to be computed. There is no parallelism to exploit inside a single answer. Decode speed is the "38 t/s" you see in benchmarks, and it's the number that makes local AI feel fast or sluggish.

flowchart LR
  A[Your prompt e.g. 12000 tokens] --> B[Prefill: read all in parallel - fast but scales with prompt length]
  B --> C[First token appears]
  C --> D{Decode loop}
  D -->|one token at a time| D
  D -->|answer finished| E[Response]

My own measurements make the difference vivid. With a 35-billion-parameter model on this machine, prefill ran at 244–271 tokens/s while decode managed 33–40 tokens/s. A long 12,000-token prompt took about a minute to read – and then answered at a readable ~35 words per second. Same model, same hardware, two totally different beasts.

The practical consequences:

If your pain is waiting for the first word, you have a prefill problem (long prompts, cold caches, big codebases being re-read).
If your pain is slow streaming text, you have a decode problem – and decode has one master: bandwidth. (More below.)

2. What a token even is (30 seconds)#

Models don't read words, they read tokens – fragments of words in a vocabulary of ~150k entries. Rough conversions to keep in your head:

1 English word ≈ 1.3–1.5 tokens; 1 character of code ≈ 0.3–0.4 tokens
A 500-word blog post ≈ 700 tokens
A 100k-token context window ≈ 750 pages of text
A model answering in 400 words spends ~600 tokens of work

So when I write "my answer is 2000 characters long", the model actually produced ~600 tokens, which at 15 t/s is ~40 seconds of typing. The token meter is the speedometer of everything below.

3. The memory wall: why decode speed is bandwidth, not horsepower#

Here is the single most important idea in local AI, and almost nobody explains it up front:

Generating one token means reading (a large part of) the entire model through the chip. The question is only: how fast can the bytes arrive?

A model is a giant file of numbers (weights). To predict the next token, the chip must multiply the current activation vector against the weights – which means every relevant weight must physically travel from memory into the processor. The arithmetic is tiny compared to the transport. Experts call this memory-bandwidth-bound, and it gives you a formula that predicts reality shockingly well:

decode tokens/s ≈ (effective bandwidth of the memory the model lives in) ÷ (model bytes touched per token)

Let me plug in my own hardware. A consumer GPU like my RTX 3060 moves ~360 GB/s internally, while a quad-channel DDR4 workstation memory tops out around ~85 GB/s theoretical, realistically ~45–55. That's the difference between on-chip memory and system memory, and it's exactly why running a model partly on the GPU can feel like a hardware upgrade even when it "doesn't fit".

A sober example from my first week: when I finally watched the numbers closely, the 35B model pulling ~36 GB/s effective out of my dual-channel DDR4 told me everything – the CPU memory was saturated while the GPU sat idle at ~250 MHz. The GPU was waiting for deliveries.

4. "But 125B should be impossible, then!" — MoE and quantization#

Two tricks let huge models run at human speeds, and you need both concepts to read modern spec sheets.

4.1 MoE: giant file, small per-token footprint#

Mixture-of-Experts (MoE) models don't use all their parameters per token. A 125-billion-parameter MoE might activate only ~6 billion parameters for each token it writes: a small router looks at the token and wakes up only a handful of the 512 expert sub-networks – the rest stays asleep in memory.

flowchart TB
  T[token] --> R[Router - decides per token]
  R --> E1[Expert 7]
  R --> E2[Expert 190]
  R --> E3[Expert 411]
  R --> X1[Expert 1 - asleep]
  R --> X2[Expert 2 - asleep]
  R --> X3[Expert 3 - asleep]

This is why my 177B-parameter Flash-Next model writes ~15 words per second on hardware that would choke on a "mere" 35B dense model… except it doesn't. What matters for decode speed is activated parameters, and what matters for fitting into memory is the total file size. Two completely different numbers that marketing rarely separates. (The price: each expert is still a chunk of memory that has to reside somewhere, even if it's used only once per thousand tokens. That's where our RAM budget conversation comes in.)

4.2 Quantization: buying back size and bandwidth#

Full precision uses 16 bits per weight. A quantization of 4.27 bits per weight (bpw, the "bpw" in my model's name) shrinks the file by ~3.7× – and since bandwidth is the bottleneck, the same ~3.7× speed gain on the memory the weights live in. That's why you'll see people proudly report "Q4" files.

The catch is quality. Quantizing an entire model too hard makes it measurably worse, so modern community quants ( Unsloth "UD", AtomicChat "AD") spend bits unevenly: critical layers and the attention network get 5–8 bits, the sea of redundant expert weights can take 3–4. Which variants lose what is usually published as benchmark tables like this (from AtomicChat's own evals, top-1 accuracy on a 7,500-question suite):

Variant	Size (disk / resident)	Quality
UD-Q8_K_XL	188 GB	91.3% (reference)
AD-4.27bpw	94.5 / ~55 GB	89.5%
AD-4.57bpw	100 / ~60 GB	88.5%
UD-Q2_K_XL	58.4 GB	83.9%
AD-3.84bpw	80.7 / ~46 GB	82.7%
UD-Q2_K_XL	52.5 GB	74.2%

Look at rows 2 and 5: my 4.27bpw variant scores higher (89.5 vs 88.5) despite fewer bits than the 4.57 variant. Distribution beats volume. Lesson for beginners: in the MoE world, bpw is not the quality axis everyone advertises – read eval tables, not model names.

5. Where the model lives: VRAM, RAM, SSD – and a case study in falling off the cliff#

Weights must live in a memory tier, and there's a steep waterfall of bandwidth:

flowchart TB
  G[VRAM GPU - 12 GB at ~360 GB/s] --- R[RAM - 64 to 96 GB at ~45 to 55 GB/s]
  R --- S[SSD - terabytes at ~0.05 GB/s for random reads]

Anything in VRAM is untouchable speed – that's where attention layers, the router, and a slice of experts belong if you're lucky.
RAM is the big workhorse for MoE experts: slower, but the bandwidth is sequential-ish per active expert and the OS keeps it pinned ("resident" in the tables above).
SSD sounds like just another slow tier. It is not – for models it's death by randomness. A model's access pattern is random micro-reads; on an SSD that's a round-trip per lookup, and your effective bandwidth collapses ~1000×.

My own cliff: the Flash-Next quant needs ~55 GB resident. On 64 GB of RAM that looks like it fits. It doesn't: model + OS + services pushed past the limit, and the kernel began evicting and re-fetching model pages faster than I could generate – 1.1 tokens per second, far slower than the expected ~15. The model wasn't crashing, it was drowning: every token needed ~3.2 GB of data, and most of those bytes were being page-faulted off SSD instead of read from RAM. The fix wasn't software. The fix was $320 of DIMMs on order.

Rule of thumb I now live by: a model is only as fast as the slowest tier its active weights fall into. Budget resident size first, quality second.

One honorable exception proves how deep the rabbit hole goes: the Flash-Next architecture keeps a 51-billion-entry lookup table (its N-gram memory) that the CPU touches only 2.7 KB per token – so that one table can live on an SSD without breaking a sweat (38 GB mmap'd on disk, still ~10–17 t/s prefill measured). Nothing about local AI is universally true.

6. The context window: VRAM's hungry cousin#

The context window is how many tokens (prompt + answer) the model can juggle at once. It's a memory question, not a quality question – and the memory it costs is the KV cache: the model stores, for every token it has processed, some attention bookkeeping (K and V vectors) so later tokens can look back at earlier ones. Cost grows linearly with context length, and it lands in VRAM by default.

For classic attention-heavy models, a 35B at 8-bit KV wants ~1–2 GB per 32k context, and a 128k window quickly costs 4–8 GB – which is why many "12 GB GPU" guides quietly assume you won't use the full window. My benchmark sweep proved the squeeze: the dense 35B model's VRAM use stayed pinned at ~10.2 GB from 8k all the way to 128k context, because only active slots pay for KV – but the reservation had to fit.

Newer architectures break the scaling law. The Flash-Next family I'm running uses hybrid attention: of its 48 layers, only 12 do full attention with a KV cache; the other 36 are recurrent "Gated DeltaNet" states that cost the same whether the context is 4k or 256k. Result: ~24 KB of KV per 1k tokens instead of a hundred-plus. That's the quiet reason a 177B model with a 262k window is loadable on a box like mine at all.

There's also free speed hiding here. Inference servers (llama.cpp, Ollama, vLLM) reuse the KV of an unchanged prompt prefix between requests – "prompt caching". Every agent framework (opencode included) resends its whole conversation history on each turn; with prefix caching, the server only prefills the new tokens. Which is how a 12k-token prompt costing 60 seconds once costs seconds on turn 10. The flag to know: --cache-reuse / keep_alive=-1 / a warmed first turn; the enemy is restarting the server or swapping models mid-session.

7. The knobs that actually matter (and what each one really does)#

A short dictionary of the settings that populate config files and will explain themselves now:

Knob	What it really does	Beginner advice
`-c / num_ctx`	Size of the working context window	Bigger is never faster; pay what you use
`-ngl N` (GPU layers)	Push N transformer layers into VRAM	Give VRAM the attention first, then experts
`--n-cpu-moe` / cpu-moe	Keep MoE experts in RAM, rest on GPU	The core trick of my whole setup
`-b / -ub` (batch size)	Tokens prefill processes per pass	Big helps long prompts, costs VRAM
`-ctk/-ctv q8_0`	Quantize the KV cache	Buys you context cheaply
`OLLAMA_KEEP_ALIVE` / `-cka`	How long model+KV stay resident	Keep warm, or suffer reloads
`--parallel`	Concurrent conversation slots	One agent = one slot; each costs KV
`--jinja`, `--chat-template`	How the raw text is framed as a dialog	Mismatched = polite gibberish; check it

And the one meta-hint: --info (llama-server) prints exactly which layers landed where, and tools like nvtop, nvidia-smi plus free -h let you watch the memory wall in real time. Watching your GPU idle at 250 MHz while RAM saturates converts you into an expert faster than any article can.

8. Estimating before you download: the cheat sheet#

Assemble the four numbers, run the two formulas, get an honest prediction. It's not exact engineering – it's order-of-magnitude truth that saves you weekend-long detours.

Decode speed estimate: tokens/s ≈ bandwidth ÷ (active params × bytes per param + KV/misc overhead)

Memory budget: file size = total params × bytes per weight; then check whether the resident portion (minus what you mmap deliberately) fits into RAM with ~6–8 GB to spare for the OS.

Applied to my box with the model I'm running (6B active, 4.27 bpw ≈ 3.2 GB per token, quad DDR4 effective ~50 GB/s): 50 ÷ 3.2 ≈ 15 t/s – matching what the machine will be asked to prove after the RAM upgrade. Same formula, three configurations, all measured, one truth: RAM is the speed governor, not the GPU.

And a worked example for the impatient: a "70B" dense model at Q4 is ~40 GB of traffic per token → on 50 GB/s RAM that's ~1.2 t/s. Yes, one point two. Same model as a 70B MoE with 10B active: ~6 GB per token → ~8 t/s. If the spec sheet doesn't tell you "activated parameters", you cannot predict your user experience – you can only gamble.

9. What I wish someone had told me#

t/s is two numbers: prefill (reading, parallel, compute-bound) and decode (writing, serial, bandwidth-bound). Which one hurts tells you which one to optimize.
Total parameters ≠ speed; activated parameters ≈ speed; file size = the address it must live at. MoE decouples them – for the better and for the confusion.
Bandwidth is the currency. The memory tier the active weights fall into decides everything. "Fits" in RAM means resident plus working set plus OS, not "smaller than free -h says".
The context window is a VRAM budget item, and modern hybrid architectures are the quiet revolution making big windows affordable on old boxes.
Keep the model warm. Reloads and cache eviction are the difference between fluid and painful in day-to-day use; keep_alive=-1 is one of the highest-leverage settings per keystroke.
Measure, then believe: nvtop + free -h + one API round-trip tell you more than a hundred Reddit threads – and occasionally make your own blog post about your 1.1 t/s drowning accident.

If all of this felt like half the terms were new to you: that's the actual onboarding curve of local AI. It's a plumbing problem disguised as a research field – and once you can see where the bytes flow, every model card, every spec sheet, every "can I run it?" answer falls out of two formulas and one honest memory map. Mine now lives by them, at ~15 words per second, on a six-year-old Xeon typing next to my robot car.