A 35-billion-parameter model on a 12 GB GPU: measuring real-world speed for coding and note-taking

I wanted a private, always-on AI assistant for my home projects: an agent that can read a documentation tree, answer questions about a build, sketch out code changes and tidy up my notes — without shipping any of it to a cloud API. The catch: the only spare hardware I had was my homelab NAS, an Unraid box that is a decade old on the CPU side and carries a modest RTX 3060 with 12 GB of VRAM.

This post documents what actually runs on that machine, and — more importantly — what it feels like day-to-day. I benchmarked the model at context sizes from ~600 tokens up to 65k prompts against a 131k window and broke the timing down into the three phases that decide whether local AI is pleasant or painful: loading, reading your prompt (prefill), and writing the answer (decode). The verdict up front: coding agents work at a patient, usable pace; note-taking chores feel almost interactive; and the architecture of modern MoE models did most of the heavy lifting.

The machine#

Everything below ran on hardware that was never marketed as an AI workstation:

• CPU: Intel Xeon E5-2630L v4 — 10 Broadwell-EP cores (20 threads) clocked at 1.8 GHz base. From 2016.
• RAM: 64 GiB DDR4-2666 ECC, driven by a four-channel memory controller
• GPU: NVIDIA GeForce RTX 3060, 12 GiB VRAM, connected over PCIe 3.0 x8
• Storage: 256 GiB SSD holding the model files and the Docker runtime
• OS: Unraid, running the ghcr.io/chrizzo84/ollamaui container, which bundles the Ollama engine together with a web UI

The RTX 3060 is the machine's only real AI credential, and 12 GB of VRAM is the number that decides everything. It is less than the 16 GB of the current budget-card darling and a fraction of what the "run LLMs locally" marketing images imply you need. As it turns out, that is fine — for a specific class of models.

The model that makes this work: Qwen3.6-35B-A3B#

I originally set out to run Qwen3.8-Flash-Next, the 125-billion-parameter mixture-of-experts (MoE) preview of the Qwen4 architecture. Reality check: its smallest usable quantization is ~75 GB, and its 4-bit build is ~111 GB. With 64 GiB of system RAM, 12 GiB of VRAM and the OS and containers wanting their share, that model is simply out of reach — a hard fact no amount of tuning fixes. (That's a separate story about the model's N-gram embedding table; the short version is: some models have a memory floor you cannot quantize below.)

What does fit is its little sibling: Qwen3.6-35B-A3B. The naming tells you the trick — 35 billion parameters in total, but only about 3 billion are active for any given token. The model body is a mixture of experts across 48 layers, and for each token a router picks a small subset of those experts to actually run. Add a hybrid attention design where only a minority of layers carry a growing key/value cache, and the model behaves very differently from a dense 35B under memory pressure.

I ran the Unsloth Dynamic quantization UD-Q4_K_XL — a 23.3 GB GGUF file pulled straight from Hugging Face through Ollama:

hf.co/unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL

For comparison: a dense 35B model at 4-bit would need to stream roughly 17–18 GB of weights through the memory system for every single token it writes. The MoE model streams only its ~3B active parameters — around 1.2–1.5 GB of reads per token at this quantization level. That factor of ten is the difference between a slideshow and something you can work with.

Where the weights live, and why the PCIe slot does not matter#

Ollama's default placement strategy for models that don't fit into VRAM is to offload whole layers to the CPU until the remainder squeezes onto the GPU. For MoE models that is exactly the wrong decision: it drags the hot, bandwidth-hungry attention and router weights across the slow PCIe bus while leaving bulky, rarely-used experts on the GPU.

The fix is a single environment variable that keeps routed expert weights in system RAM while attention, routing, the KV cache and the vision encoder stay on the GPU:

LLAMA_ARG_CPU_MOE=1

My full container configuration, all of it Unraid environment variables on the OllamaUI container:

Variable	Value	Why
OLLAMA_CONTEXT_LENGTH	131072	server-wide default context
OLLAMA_KEEP_ALIVE	-1	pin the model in memory, never time it out
LLAMA_ARG_CPU_MOE	1	experts to RAM, everything else to GPU
GGML_CUDA_NO_PINNED	1	avoids pinned-memory OOM during large offloads
OLLAMA_NUM_PARALLEL	1	serialize requests instead of splitting throughput

With that, steady state looks like this: ~10.2 GB of the weights and all of the live attention/KV state sit in the 3060, and ~13 GB of expert tables sit in the Xeon's DDR4, computed on the fly by AVX2 kernels on the CPU cores. ollama ps shows roughly 45 % of the model resident in VRAM — and that split is correct for once, rather than a symptom of failure.

One worry I had before measuring: the 3060 only runs at PCIe 3.0 x8 in this box, half the lanes and an older generation — 8 GB/s. For the split above it is irrelevant. The GPU pulls its resident weights once, at load time; per generated token, only small activation tensors (single-digit MB) cross the bus, and the expert streaming happens between CPU cores and their own RAM at full bandwidth. The PCIe lane is a spectator here, not a bottleneck. The same x8 link would genuinely hurt a dense model offloaded layer-by-layer, though.

The three speeds that matter#

People quote "tokens per second" as if it were one number. It is three, and they differ by an order of magnitude:

1. Load — reading a ~23 GB file from SSD into RAM/VRAM after the engine (re)starts. Happens once per container restart thanks to keep_alive=-1, but it will bite the moment you change a setting and restart, so it is worth knowing.
2. Prefill — the model reads everything you gave it (prompt, conversation history, attached files) before it can answer. You pay for this once per request, and agentic tools pay it again every single turn, because every tool call replays the whole conversation.
3. Decode — writing the answer, token by token. This is the part that streams into the chat window and the one everyone measures.

Everything below is measured on the machine above, with a synthetic-filler harness against the local API. One pass per data point, warm model unless noted.

Prefill: about 250–270 tokens/second, almost context-independent#

The headline result: prefill speed barely depends on how full the context window already is. Prompt sweeps measured 244 t/s at 1k tokens, rising to a peak of 271 t/s at 16k, and only declining to 253 t/s at 65k-token prompts — a seven percent drop across a fourfold growth in context. Only the very smallest requests dip below (177 t/s at ~600 tokens, where fixed per-request overhead dominates the math).

What does depend on context size is the absolute time cost, because you are multiplying a roughly constant rate by a growing pile of tokens:

Prompt you attach	Approx. tokens	Prefill wait
A chat message plus one short note	500	~2 s
One source file / documentation page	2,000	~8 s
A handful of files, chat history	8,000	~30 s
The agent's accumulated work so far	16,000	~60 s
Several whole documents	64,000	~4 min
Everything, up to the window's edge	131,000	~9 min

Two facts derived from this are worth tattooing onto any local-AI user's forehead. First: the window size and the per-request cost are different things. Pinning a 131k context on this model costs about 1.4 GB of RAM and nothing measurable in VRAM (10.2 → 10.5 GB across the whole sweep) because only a few layers maintain a real KV cache. Large windows are cheap to have. Second: a full window is expensive to read. If your agent's session history has grown to 60k tokens, every tool call starts with four minutes of prefill — the model is silently re-reading everything you've ever sent it, over and over.

Practical consequence: keep agentic sessions trimmed. Summarize and restart. A 32k window used ruthlessly beats a 131k window used sloppily by a factor of ten in per-turn latency.

Decode: 26–38 tokens/second, and where the ceiling comes from#

Warm decode on short contexts measured 33–38 tokens/s, easing to ~31 at 32k-token prompts and 24–26 tokens/s at the fully-loaded 128k end — the mild decline coming from attention state and CPU-side bookkeeping growing with context, not from weights being re-read. A pure long-generation run (256 tokens) measured 34.7 t/s.

For calibration: ~35 tok/s in English prose is roughly 25–27 words per second — three times the reading speed of a fast human reader. A 300-token answer (~220 words) takes about 10 seconds; a 1,500-token answer (a solid page of text) takes about 45. Code streams slightly slower in wall-clock terms because a code-heavy tokenizer produces fewer words per token, so an 80-line function is a ~1–2 minute watch — acceptable for a review-and-edit workflow, too slow to enjoy as live autocomplete.

The ceiling is memory bandwidth, not the GPU. At this quantization, every token pulls roughly 1.2–1.5 GB of expert data out of DDR4. Multiply by 30 tokens/s and you need ~40 GB/s of sustained reads. A dual-channel DDR4-2666 board tops out around 40 GB/s theoretical / ~30 GB/s real, and my earlier planning estimate — built on the dual-channel assumption — was 5–12 tok/s. The machine does 30+. The reason: the E5-2630L v4 is a server chip with a four-channel memory controller. If you are picking hardware for large-model offload, channel count outranks almost every other RAM spec — more than capacity, more than speed grade. 128 GiB across six channels beats 64 GiB across two for this workload even though it's "only" older DDR4.

Cold load: two minutes once, fifteen seconds after#

The first cold start after a container restart — reading 23.3 GB off the SSD, placing it across VRAM and system RAM, allocating a 131k KV window — took about two minutes. But here is the detail nobody mentions: on a 64 GiB machine, the model file fits entirely in the OS page cache. Every reload after that measured just 14–17 seconds, because the "disk read" is really a RAM copy. The two-minute club only greets you after an actual reboot or a cache purge. With keep_alive=-1 it happens once per session anyway.

The daily feel: coding with a local agent#

Here is how the measured numbers translate into an actual working rhythm when an agent (opencode in my case) drives the model across tool loops. A turn is roughly prefill + decode + tool execution, and the prefill term explains why agents feel different from chat.

A typical sequence on a documentation/code task:

1. Kickoff (5–20 s). System prompt plus project instructions, ~1–2k tokens. Fast, because the window is empty.
2. Exploration turns (30–90 s each). The agent reads files, and every tool result grows the context; prefill scales with it, decode only produces the next call.
3. Drafting the change (1–3 min). Actual code output — a few hundred tokens plus a thinking trace, at decode speed.
4. Verify turns (30–90 s each). More prefill over the now-bigger history.

Realistic expectation for a small change across two or three files: five to fifteen minutes of wall clock. For bigger multi-step refactors: half an hour, often in parallel with you doing something else, which is the entire point of the agent model. If you expect interactive pair-programming tempo from a decade-old Xeon, you will be disappointed by the same physics that disappoint a 4090 owner — just less extremely. But for overnight-able or attend-from-another-room work, the throughput is genuinely useful.

What helps more than any flag: smaller context discipline. With sessions kept under ~8–16k tokens, a large share of those five minutes moves back toward two. Also note the thinking mode: this model writes its reasoning out by default, and on simple questions the thought trace can exceed the answer (~200–300 hidden tokens, six to ten seconds). For agentic use it is quality insurance; for quick lookups, disabling thinking is free speed.

The honest downsides, in order of how much they annoy me: prefill dominates agent loops more than I expected; long answers mean waiting on a progress bar of prose; and a single pinned model means switching to another model pays the reload. What doesn't annoy: nothing in this workflow ever hit an OOM on the RAM side (peak resident was ~24.7 GB of 64), the GPU stays cool under decode since the CPU does the expert math, and with OLLAMA_NUM_PARALLEL=1 two concurrent agents never corrupt each other's speed — the second just waits its turn.

The daily feel: note-taking in Trilium#

For note work I put the same model on the other end of a Trilium MCP connection — reading and editing my personal knowledge base through the same tool-loop mechanics. Here the workload profile inverts: prompts are short (one or two notes instead of a whole codebase), and the value is in writing: summarize, restructure, merge, extract, rewrite. That makes it decode-dominated — the good phase.

Concrete rhythms that became normal:

• "Summarize this note": ~1–2k tokens in, ~200 tokens out. The answer begins in about four seconds and is done in twelve. Feels interactive.
• Rewriting/redacting a section: in and out both mid-sized; 30–60 seconds per pass. Faster than doing it manually, with a taste for over-compressing that needs supervision.
• Weekly journal digestion: the one I actually like — feed a week's daily notes (~4–8k tokens), ask for a list of open threads and decisions. About a minute of prefill, then it streams for a few minutes while you make coffee.
• Extraction chores: pulling numbers from long notes into tables, converting prose to checklists — the boring stuff where cloud models were always a privacy net loss.
• Multi-note merges: 20–40k-token sessions appear naturally and still behave well thanks to the cheap-KV architecture, though each follow-up turn then starts with the ~1–2 minute prefill tax. Repeating "and now add…" five times is slower than one better prompt.

The qualitative difference from chat-with-cloud: latency stops being an insult the moment the model is allowed to work asynchronously. Note tasks are things I do in the background by design, and a steady 26–35 tok/s that never rate-limits, never changes terms and never sees the cloud turns out to be a better daily companion than a faster model I can't fully trust. The one thing I miss is vision (photos into notes) — the projector files for exactly this model exist; I just haven't wired them up yet.

Gotchas the benchmarks don't show#

• The quantization was the whole ballgame. UD-Q4_K_XL is a ~23 GB file, and 23 GB on disk becomes ~23 GB across RAM+VRAM no matter how you offload it. Read the file size, not the parameter count.
• Ollama's default context (4k) silently breaks agentic tool loops — tool definitions plus history exceed it and you get incoherent agent behavior instead of an error. Set OLLAMA_CONTEXT_LENGTH high.
• ollama ps never lies. It reports the actual VRAM/RAM residency split and "expires: forever" — the fastest way to confirm the offload and keep-alive settings are active.
• Pinned models are territorial. One ~25 GB resident model on a 64 GiB box is comfortable; two large ones is a knife fight. Stop the old before you start the new.
• The "model not found" classic for HF pulls: the local ID includes the hf.co/ prefix; OpenAI-compatible clients need it exact.
• API testing gotcha: /api/generate streams NDJSON unless you send "stream": false — looks like a broken parser, is a broken request.

Verdict and next steps#

The short answer to "can a decade-old Xeon, 64 GiB of ECC DDR4 and a 12 GB RTX 3060 run a useful local AI for coding and knowledge work?" is yes — at ~35 tok/s out, ~250 tok/s in, with a 131k context pinned, for a few cents of electricity and zero data leaving the house. The long answer: it is only possible because MoE plus CPU-expert offload turned "fits in VRAM" into "fits in RAM bandwidth times four channels," and because hybrid attention forgave an old CPU's prefill weakness.

And the upgrade path is no longer hypothetical: another 32 GiB of matching ECC is already ordered, which will take the box to its maximum population — six DIMMs, 96 GiB. At that point the machine that motivated this whole experiment becomes reachable: Qwen3.8-Flash-Next at its smallest usable quantization (~79 GB, UD-Q2_K_XL, roughly 83 % quality-retained). Worth being honest about the limits that remain: the full 4-bit build (~111 GB) still won't fit even at 96 GiB — the 128 GB class needs a bigger GPU to hold attention and KV, too. What stays identical is the interesting part: the four-channel bandwidth does not grow with capacity, so decode speed will not improve — only the size and quality of what fits into it. The plan is to re-run this exact benchmark suite on the upgraded box, and if the numbers move, this post gets an update.

All measurements from 2026-09-10 on the machine above; harness talking to a stock Ollama 0.33.3 API, single pass per data point.

Benchmark appendix#

Full sweep, 128-token generations except where noted. "Prompt" is what the model actually read; the window was 131072 throughout except where noted.

Test	Prompt tokens	Prefill t/s	Decode t/s	VRAM (GB)	RAM (GB)
Cold reload (page-cached)	14	–	34.7	10.3	14.5
512 window	585	176.8	35.8	10.2	12.9
2k window	1,026	244.2	37.8	10.2	12.9
4k window	2,050	251.3	33.3	10.2	12.9
8k window	4,098	263.3	32.7	10.3	12.9
16k window	8,194	266.4	36.6	10.2	13.1
32k window	16,386	270.9	34.4	10.5	12.9
64k window	32,770	264.7	30.7	10.5	13.4
131k window, 65k prompt	65,538	253.2	24.5	10.3	14.5
Pure decode (256 tok)	20	–	34.7	10.3	14.5

Plus the independent context-scaling run on 2026-09-10 morning (16k→128k filled windows): decode 34.5 → 26.3 t/s, VRAM constant at 10.2–10.3 GB, cold disk load 124 s.