Speaker A
The model file is 22 GB. The graphics card has six, and it still answers at 17 tokens a second on a card that shipped in 2016. That works because of one detail in the model's name, 35B, a 3B,
Guide to running a 22GB 35B parameter AI model on a 6GB GPU using llama.cpp flags for speed and memory optimization.
Key Takeaways
- Efficiently running large AI models on limited VRAM GPUs is possible by offloading expert weights to CPU RAM.
- Proper flag configuration in llama.cpp can multiply token generation speed without new hardware.
- Memory management strategies like MMAP and mlock are essential for stable and fast inference.
- Batch size tuning is a powerful but often overlooked lever for improving prompt processing speed.
- Turbo Quant offers promising memory savings for key-value cache but is not yet integrated into mainstream tools.
Summary
- The video explains how to run a 22GB QN3.6 35B AI model on a 6GB GPU efficiently using llama.cpp.
- It highlights the model's mixture of experts architecture, where only a subset of experts activate per token, reducing memory traffic.
- Five key llama.cpp flags are introduced to improve token generation speed from 3 to 17 tokens per second on older GPUs.
- The N-CPU-MOA flag offloads expert weights of certain layers to CPU RAM, significantly saving VRAM usage.
- Memory mapping strategies (MMAP, No-MMAP, MLOCK) affect performance depending on system RAM and model size.
- Adjusting n_gpu_layers helps fill VRAM and boosts speed further, while mlock prevents kernel paging for stable performance.
- Token generation speed is limited by memory bandwidth when experts reside in system RAM rather than GPU VRAM.
- Batch size settings influence whether prompt processing runs on CPU or GPU, with larger micro batches greatly accelerating prompt reading.
- Turbo Quant is a quantization technique that drastically reduces key-value cache memory but is not yet merged into llama.cpp.
- Context length management is critical, with this model using linear attention in most layers to reduce cache size and enable very long contexts.
Chapters
- 00:00Introduction and Model Overview
- 00:49Mixture of Experts Architecture Explained
- 01:31Offloading Experts to CPU RAM
- 02:09Flag One: N-CPU-MOA and Speed Gains
- 02:43Tooling Limitations and Ollama
- 03:18Flag Two: Memory Mapping Strategies
- 03:59Flag Three: Adjusting GPU Layers for VRAM
- 04:29Flag Four: Preventing Kernel Paging with mlock
- 05:04Token Generation Speed and Hardware Limits
- 05:21Batch Size Impact on Prompt Processing
Full Transcript — Download SRT & Markdown
Speaker A
35 billion parameters total. 3 billion of them fire per token. So, here is the whole video. Five llama.cpp flags that take this from three tokens a second to 17, and then the three places where the trick stops working. The model is QN3.6
Speaker A
35B A3B. Alibaba shipped it on April 15th, 2026, Apache 2.0, and it scores 73.4 on SWE bench verified, which is a strange thing to have sitting on your desk. The four-bit build from Unsloth is 22.36 GB. A 6 gig card holds about a
Speaker A
quarter of it. Open that file up, and it is 256 experts. For any one token, the router wakes eight of them, plus one shared. The other 248 sit still. Hold that next to a dense model for a second.
Speaker A
A dense 35B reads every one of its 35 billion weights to produce a single token. At four bits, that is roughly 20 GB of memory traffic per token. QN3.6 reads under one. Same knowledge on paper, a 20th of the movement. So, the
Speaker A
question stops being, does the model fit, and becomes, which parts of it have to be fast? Attention runs on every single token, so attention goes on the GPU. Embeddings, the shared expert, the key value cache, all GPU. The routed
Speaker A
experts are idle most of the time. Idle weights can live in system RAM. Until the middle of last year, you did that by hand with a regular expression, override tensor, backslash.ffn_.*_exps = CPU.
Speaker A
It worked. It was also the kind of flag you copy off a forum post at midnight and hope you paste it correctly. Then, on the 31st of July, 2025, a llama.cpp maintainer merged a flag called CPU-MOA. Four days later came
Speaker A
N-CPU-MOA. Keep the expert weights of the first N layers on the CPU. That pull request was open for 80 minutes before it went in.
Speaker A
That is flag one, and it is most of the win. On a 6 gig GTX 1060, QN3.6 starts out around three tokens a second. Add N-CPU-MOA-30, and it lands on 10. You have not bought hardware. You have not quantized down
Speaker A
further. You told the scheduler which weights were worth the fast memory. Picking N is a dial, not a formula.
Speaker A
Start with every layer on the CPU, then walk the number down. 28, 26, 24. Keep going until the loader tells you it ran out of VRAM. Then step back one. The right value is the last one that fits.
Speaker A
And this is where the tooling gets awkward because you cannot do any of this in Ollama. The feature request went up in August 2025, and it is still open.
Speaker A
One tester watched Ollama split a model 17% CPU, 83% GPU on its own, with no way to argue. LM Studio at least gives you a checkbox. All experts on the CPU, or none. The ticket asking for an actual
Speaker A
number has been open just as long. Flag two is smaller and stranger. No-MMAP. By default, llama.cpp memory maps the weights straight off disk, and lets the kernel decide what stays resident.
Speaker A
Switch that off, load the whole thing into RAM up front, and 10 becomes 13 and a half. Except build llama.cpp and that flag prints a deprecation warning.
Speaker A
Back in March, MMAP, No-MMAP, and MLOCK all collapsed into one option called load mode, with four settings. None, MMAP, MLOCK, DEO. Same behavior, one place, and a lot of guides online are now wrong. There is a second catch
Speaker A
underneath that one. If the model is larger than your RAM, No-MMAP is the wrong answer. The maintainers found that mixture of experts models sitting at 100, even 150% of system memory, run faster with memory mapping left on because the page cache
Speaker A
is doing real work. Flag three is arithmetic. Once the experts move out, most of your 6 gigs is empty again. So, push layers back with n_gpu_layers until VRAM fills up. 13 and a half turns into 17.
Speaker A
Three to 17 on a card from 2016 and 32 gigs of ordinary desktop RAM in three flags.
Speaker A
Flag four only matters if you leave the server running. Over a few hours, the kernel decides those expert pages are cold and pages them out to make room for something else. mlock tells it not to.
Speaker A
Your first token after lunch stops taking 4 seconds, but there is a ceiling here that no flag will move. The moment those experts live in system RAM, your tokens per second stops being a GPU number and becomes a memory bandwidth
Speaker A
number. Dual channel DDR4 at 3200 gives you about 51 GB/s. That 1060's own VRAM does 192.
Speaker A
You move the work into the slow lane on purpose, and the slow lane is nearly four times slower, which is why the numbers scale the way they do. A GTX 1070 with 8 gigs runs the same model near 18 tokens a second at 132,000
Speaker A
tokens of context. Fit the whole thing inside a 3090 and you get 105. A 5090, 183.
Speaker A
Now, if you have actually tried this, you have probably hit the wall I did.
Speaker A
Generation feels fine, then you paste in a file, and the thing just sits there.
Speaker A
Reading your prompt and writing its answer are not the same job, and llama.cpp does not run them the same way. Down in the CUDA back end, there is a single constant, op_offload_min_batch_size.
Speaker A
Its default value is 32, and you can override it with an environment variable. Every expert matrix multiply gets measured against that one number, and whichever side of it you land on decides who does the work. Writing an answer happens one token at a time, so
Speaker A
the batch size is one. One is under 32, so the CPU does the multiply out of system RAM at 51 GB a second. That is your 17 tokens a second. Reading a prompt happens hundreds of tokens at once.
Speaker A
That is over 32, so llama.cpp copies the expert weights across PCI Express into VRAM and hands the job to the GPU instead. Same model, same flags, a completely different machine underneath, which hands you a lever most tutorials skip. The default micro batch is 512
Speaker A
tokens. Raise it. In the maintainer's own benchmark on a similar offloaded model, going from 128 up to 2048 took prompt processing from 22 tokens a second to 345.
Speaker A
Fifteen times from one flag, and it got better again after that. In August 2025, a patch landed with a title that says exactly what it fixes. Copy only the used experts when offloading prompt processing.
Speaker A
Before that, every expert made the trip, all 256 of them. So, that is the speed half. Context is a separate fight. QN 3.6 ships with 262,000 tokens of context, and if you are pointing a coding agent at a repository,
Speaker A
you will use every one of them. Here's what that costs on this specific model.
Speaker A
20,480 bytes of key value cache per token per sequence, 20 KB. So, the full 256,000 tokens works out around 5 GB, which is small, and the reason is the shape of the model. 40 layers, but only 10 of them run real attention. The other
Speaker A
30 are gated DeltaNet, a linear attention that carries a fixed-size state instead of a cache that grows with every token. Three-quarters of this model has no key value cache at all. Put a normal full attention mixture of experts model
Speaker A
of the same class beside it, and that one wants 98,000 bytes per token. 4.8 times more memory for the identical conversation. The cheapest context you will ever get is the context that architecture never allocates. Five gigs of cache still will not sit next to 5 gigs
Speaker A
of hot weights on a 6 gig card, so the guide reaches for Turbo Quant.
Speaker A
Google Research and NYU put the paper on arXiv in April 2025, where it sat for 11 months until a Google blog post in March lit the whole community on fire. It is heading to ICLR 2026. The trick is a
Speaker A
rotation. Spin every key and value vector into a basis where the coordinates behave, then quantize against one fixed codebook.
Speaker A
3-bit, no calibration, cache type K Turbo 3, cache type V Turbo 3, flash attention on.
Speaker A
Google measured at least a six times cut in key value memory with the benchmark scores intact. In the guide, that same 6 gig card goes from 64,000 tokens of context to 256,000.
Speaker A
And here's the part the write-ups leave out. Turbo Quant is not in llama.cpp. 75 pull requests and issues in that repository mention it. Every one of the pull requests is closed and unmerged, two of them tagged as an AI policy
Speaker A
violation. To type Turbo 3, you build a fo...
Speaker A
It also stopped being a speed win somewhere along the way. When upstream rewrote the mixture of experts attention kernels, plain 16-bit got four times faster and the turbo path did not follow. On one MoE model, Turbo 2 now
Speaker A
decodes at 45% of F16. The Threads own verdict: this is memory, not speed, which brings me to the flag that did not make the list, speculative decoding, where a small model guesses several tokens ahead and the big one
Speaker A
checks them in a single pass on a dense model that is close to free.
Speaker A
[snorts] Here it fell over. The reason is elegant in a slightly infuriating way. One token wakes eight experts. Guess four tokens ahead and you are not waking eight anymore, you are waking most of them. A 2025 paper measured it on
Speaker A
Mixtral. One token pulled two experts. Eight tokens pulled more than seven. Three and a half times the data movement and your experts are sitting on the CPU.
Speaker A
So, every extra expert is another walk across the memory bus. That paper measured verification running two to three times slower and speculation making the whole system up to one and a half times slower than not speculating at all.
Speaker A
The workaround is multi-token prediction, a draft head trained into the model itself sharing its context and its cache instead of running a second one. It helps. Just less than you would like. Roughly 1.4 to two times on dense
Speaker A
models and 1.15 to 1.25 on a mixture of experts. Sparsity gives with one hand and takes with the other. One question left and it decides whether any of this was worth doing.
Speaker A
Is the 4-bit copy still the model that scored 73.4? Unsloth published numbers on the previous generation put 4-bit KL divergence at 0.0137 against 0.0026 for 8-bit. Five times a drift.
Speaker A
Perplexity, meanwhile, moves less than 1%. Their own warning is the useful part. Perplexity and KL divergence both get pushed around by calibration data and neither one reliably predicts whether the model can finish your task.
Speaker A
Run it on your own repository for a day. That is the benchmark that applies to you. So, set the bar where it belongs.
Speaker A
17 tokens a second is faster than you read and far too slow for an agent grinding through a code base unattended.
Speaker A
This is a very good local assistant on hardware you already own. It is not a drop-in for the API. But, the thing worth keeping is not the flag list. For two years, the rule was that VRAM is a
Speaker A
wall. The model fits or it does not and the only fix is a bigger card. Sparse models turned that wall into a budget.
Speaker A
You have stopped buying capacity and started deciding which weights deserve the fast memory. 6 GB, a card from 2016, 35 billion parameters answering in real time. Go run it, then tell me your tokens per second and your RAM speed.
Speaker A
Those two numbers together are the whole story. See you in the next one.
Topics:llama.cppAI model optimizationGPU memory managementmixture of expertsQN3.6 35B modelTurbo Quanttoken generation speedGPU offloadinglarge language modelsmemory mapping
