Speaker A
That's a 35 billion parameter model doing 70 tokens a second on an RTX 3060.
Exploring how an RTX 3060 can run a 35B parameter LLM at 70 tokens/sec using llama.cpp enhancements and expert caching techniques.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- Expert caching can significantly reduce PCIe transfer overhead if designed to avoid runtime swapping.
- A flat expert usage distribution means no consistently hot experts, challenging naive caching strategies.
- Scheduler behavior critically impacts performance by influencing data transfer patterns between GPU and CPU.
- Running missing experts on CPU and combining results allows for flexible memory management without output loss.
- Periodic cache refreshes may offer a practical compromise between static caching and dynamic expert usage.
What the video covers
- A 35 billion parameter model runs at 70 tokens per second on an RTX 3060 without aggressive quantization.
- The speed improvement comes from a novel caching mechanism for experts, contrary to expectations.
- Traditional layer-based offloading to CPU RAM is inefficient because expert usage is flat and no experts are consistently hot.
- A fixed slot cache for experts on the GPU was proposed to reduce costly PCIe transfers but initially performed worse due to frequent swapping.
- The key insight was to load experts once before running and never swap during inference, running missing experts on CPU instead.
- A technique was implemented where missing experts return zero results, allowing two passes (GPU and CPU) and summing outputs for correctness.
- Initial benchmarks showed slower performance due to scheduler inefficiencies that caused unnecessary data transfers.
- Reordering GPU and CPU work into contiguous blocks eliminated redundant PCIe traffic, restoring performance to baseline levels.
- Further profiling and oracle testing showed fixed expert sets can achieve near-optimal performance with minimal transfer overhead.
- Future improvements could include periodic cache updates every 500 tokens to balance transfer costs and performance gains.
Chapters
- 00:00Introduction and performance overview
- 00:59Problem with layer-based expert offloading
- 02:01Inefficiency of current token generation process
- 03:29Initial LRU cache implementation and failure
- 05:29Analyzing expert usage and transfer costs
- 09:58Designing a fixed expert cache with zero-return for missing experts
- 12:14Channel promotion and call to subscribe
- 14:44Scheduler inefficiency and data transfer issues
- 16:45Fixing scheduler order to improve performance
- 18:28Summary of results and future directions
Full Transcript — Download SRT & Markdown
Speaker A
And no, I didn't quantize it harder. Same Q4 model, same outputs token for token. But the speed is not the crazy part. The crazy part is what's making it fast. A cache that the stats say shouldn't help at all. So why does it
Speaker A
work? Why did the same idea fail months ago? And why did my first benchmark come out six times slower? Let's find out.
Speaker A
Because by the end your GPU might be doing the same thing. So couple of months back I'm sitting at my desk thinking about how much more I can squeeze out of these models and there's this one thing that keeps
Speaker A
bugging me. Right now when a model doesn't fit in your VRAM, you offload experts by layer. You tell it how many layers to push out to system RAM and every expert in those layers goes all of them together as a block. But experts
Speaker A
aren't equal. Some get picked constantly. Some barely come up at all. And the flag can't tell the difference.
Speaker A
It just takes the whole layer. So I kept coming back to the same thing. Why are we doing this blindly? Why layers? Why not slots? Give the GPU a fixed number of slots. An expert gets picked for work. It crosses over. It fills a slot.
Speaker A
If it keeps getting picked, it stays there, and you never pay to move it again. If it goes quiet long enough, something busier takes its place. That was the idea. Simple enough that I was a little annoyed nobody had already done
Speaker A
it. And here's why I wanted it. Without it, this is what every token has to do.
Speaker A
Attention runs on the GPU, then it goes down to the CPU because that's where the experts are sitting. Gets worked on down there, comes back up, and then it does the whole thing again for the next layer. Not once, every single layer.
Speaker A
That's 40 round trips to generate one token. And for most of that, your GPU is just sitting there waiting. Most of your token's life is spent traveling, not being worked on. I built it. A proper LRU cache for experts. Hot ones on the
Speaker A
GPU, cold ones in RAM, evict whatever goes stale. And it didn't work. Not slower than I hoped. It was slower than using no cache at all. So, I dropped the idea and I didn't look at it again for
Speaker A
months. Now, after I posted the fable video, I thought, let's give it another shot. But I wanted to know why it died first. So, I wrote a tracer. It hooks into the scheduler and writes down which experts the router actually picks every
Speaker A
token, every layer, which sounds obvious, but you can just ask the model what it's doing. Often we forget the obvious. And the first thing the data told me was don't build this. Expert usage is flat. The top 10% of experts
Speaker A
only carry about 18% of the work. That's nothing. There's no hot set. Nothing in there gets used enough to deserve a permanent slot on your GPU, which is a pretty clear verdict. The whole idea depends on some experts being hot, and
Speaker A
they're not. But that still doesn't tell me why the old one lost. Bringing no advantage and being slower are two different things. So, I added the transfer cost into the simulator. What does it actually cost to pull an expert
Speaker A
in? And there it was. The version that found its experts on the GPU most often was the slowest one I tested. Think about that for a second. The best hit rate lost because of how it got those hits. It was swapping experts in and out
Speaker A
constantly, always chasing whichever ones were hot right at that moment. And all that swapping adds up. 127 megabytes crossing PCIe for every single token.
Speaker A
The version that won barely swapped at all. 9 megabytes. So on a PCIe bus, missing isn't what hurts you. Loading is. Every time you pull an expert in, you pay for it. And my old cache was paying that on almost every token. The
Speaker A
idea was fine. It just cost more than it saved. And that tells you exactly what the design has to be. You never load anything while it's running. You choose what goes in once before you start. It goes up when the model loads and then
Speaker A
nothing moves again. Whatever you miss just runs on the CPU exactly like it would have anyway, which means the loading cost isn't small. It's gone.
Speaker A
There's no code left that can pay it. But there was one more number in that trace. And I walked straight past it.
Speaker A
The same data that told me the distribution was flat also said that a cache of 64 experts would hit nearly 80% of the time. Nothing is hot and you'd still find what you need eight times out of 10. Both of those are true and I
Speaker A
could not tell you why. That one sat in the back of my head for weeks. Turns out it's the reason any of this works at all.
Speaker A
Okay, so the core of it is one small change made in a lot of places. The part of the model that runs the experts had to learn to skip. If a token asks for an expert that isn't on the GPU, that step
Speaker A
returns zeros instead of a result. Once it can do that, you can run the whole thing twice. One pass over the experts on the GPU, one pass over the experts still in RAM. The missing ones come back as zeros. So you add the two results
Speaker A
together and get exactly what you would have got anyway. That's the design. The rest is plumbing. And the plumbing took a while. Memory corruption, a lookup table counting things that weren't there, a crash that only appeared when a
Speaker A
different test ran first, and about an hour spent looking at the wrong function entirely. Then it was working. Output coherent, every check passing the same tokens in the same order as before. So I ran the first real benchmark baseline
Speaker A
with no cache, 42 tokens a second. Then the new build, six tokens a second. And that's a strange place to land. If the cache just wasn't helping, I should have come out around 42 again. And I just remember thinking, "This can't be
Speaker A
happening. There has to be something weird going on." Okay, so quick pause. If you like running LLMs locally and figuring out how to squeeze the most out of the hardware you've already got, finding the unconventional ways to run
Speaker A
bigger models and run them faster, then this is the channel for that. So, subscribe. There's a lot more coming.
Speaker A
Now, back to the video. So, I had to fiddle around a bit more to find what's going on. And I want to be clear about how easy it would have been to just stop there. The trace data already said there was no hot set. Now
Speaker A
the benchmark said the cache made things worse. Two separate measurements, both pointing the same way. Caching doesn't help this kind of model. Write it up and move on. That conclusion was completely wrong. But it was reasonable. And that's
Speaker A
the part that makes it dangerous. But instead of taking the number at face value, I traced it. I wanted to see what actually ran and where.
Speaker A
And the cache had never run at all. Okay, so here's what was going on. The scheduler tries to keep the work in as few pieces as possible because every time it switches between the GPU and the CPU, it has to stop and sync up. And I
Speaker A
had handed it a graph that alternated a bit of GPU work, then a bit of CPU work over and over all the way down. So, it did the obvious thing. It grouped them together. It took my GPU work and moved
Speaker A
it in with the CPU work, which means it took the expert weights I had carefully parked in VRAM and carried them back across PCIe to use them on the CPU on every layer of every token.
Speaker A
And that's about 80 megabytes a layer going the wrong way. So the thing I built to reduce traffic over that bus had become the biggest cause of it. Now the fix. See I didn't change any of the maths. I just changed the order the work
Speaker A
is written down in. Instead of alternating, each side gets one unbroken run. All the GPU work in one run,
Speaker A
all the CPU work in another, then one add at the end to join them. There's nothing alternating anymore. So, there's nothing
Speaker A
left for the scheduler to tidy up. And none of the maths changed. It went from six tokens a second to 44, which sounds like a win, but the baseline was 42.
Speaker A
And there are two things I took from that, and neither of them is really about llama.cpp.
Speaker A
The first one is that being correct isn't en
Speaker A
But this one had two measurements agreeing with it and a story that made sense. That's the kind of wrong answer you accept and walk away from. And I almost did.
Speaker A
Right? So the graph is fixed. Now the only question left is how many experts you can actually afford to keep on the card. So I ran a sweep to find out. And the more slots you hand it, the faster
Speaker A
it goes. Right up until the pack stops fitting in VRAM. And the whole way up, the output stays identical to running without any of this. And there was something I had wrong the whole time.
Speaker A
I'd assumed this model had 128 experts in each layer. It has 256. So 124 slots isn't most of them. It's less than half. Then I tried a completely different model, GLM 4.7 Flash. 44% faster generating and 64% faster on the prompt. So it isn't a Quen
Speaker A
thing. Now the part I wasn't expecting. You already know MTP, speculative decoding. The model guesses ahead, then checks its own work. And on this setup, it already pulls its weight. 45 tokens a second becomes 55. But then you put the
Speaker A
cash underneath it and that 55 goes to 70 which is more than either of them is worth on its own. And the reason is that they are solving each other's problem.
Speaker A
Speculation only pays off if checking those guesses is cheap. And to check a batch of guesses you need every expert that any of them asked for which is a much bigger pile than a single token would ever touch. Now most of that pile
Speaker A
is already sitting on the card. So the checking got cheap and the guessing got worth doing. And then one more change.
Speaker A
The two chains the GPU 1 and the CPU 1 were still running one after the other which they never had to. They're independent by design. That's the whole reason the zero trick works. So I made them run at the same time. The CPU side
Speaker A
gets its own thread and just goes and the GPU doesn't stand around waiting for it to finish. It starts on its own half straight away. Which means both sides are actually working concurrently instead of running sequentially. And that GPU that spent this whole CPU time
Speaker A
sitting there waiting, it waits a lot less. And that takes 70 to 75, which doesn't sound like a lot, but it's five tokens a second from work that was already sitting there idle. One thing I did have to be careful with, this only
Speaker A
pays off on the small batches. On a big prompt processing batch, the merging costs more than the overlap saves. So, the system just turns it off there. When the batch is big, it goes back to the old way of doing it and prompt
Speaker A
processing runs exactly as it did before. You don't have to think about it and you don't lose anything. So where that leaves us, 80 tokens a second at peak and still over 70 once there's real context in there from 45. And I still
Speaker A
couldn't tell you why any of it works. Okay, so the thing I said I couldn't explain. Nothing is hot and you'd still find what you need eight times out of 10. And the answer turned out to be simpler than I expected. The
Speaker A
distribution is flat over a long run across thousands of tokens. Every expert gets used about the same amount. That part is true and that's what the average was telling me. But at any given moment, the model isn't using all of them. It's
Speaker A
using a handful. And that handful keeps changing. So over a whole run it looks like everything gets used equally because everything does eventually just not at the same time. And a cache doesn't care about eventually. A cache only cares about right now. Now the
Speaker A
result that actually convinced me. So I tested a version that just keeps whatever was used most recently. No profile, no knowledge of the model at all. and it hit about 45% of the time.
Speaker A
Then I tested an oracle, perfect information, allowed to see the entire run in advance and picked the best possible fixed set of experts before starting and that one hit 41.
Speaker A
The blind one won and that only happens for one reason. Which experts are popular overall turns out to be the wrong question. The right question is which ones were used a moment ago? And that's the thing I'd want you to take
Speaker A
from this even if you never touch any of it. See, averages hide timing. The average told me there was nothing here worth caching. And it was right about the run and completely wrong about the moment, which isn't really a llama. CPP
Speaker A
lesson. Anytime you look at aggregated data and decide there's no pattern in it, it's worth asking whether there's a pattern that just doesn't survive being averaged. And one more quickly because I got this wrong for months as well. Code
Speaker A
and chat use different experts. Their top sets only overlap about 40%. Which sounds like the cache would have to adapt as you switch between them. And that assumption cost me a lot of time.
Speaker A
So I actually tested it. A profile taken from completely the wrong workload still keeps most of the benefit. And if you do care about both, you don't have to choose. You profile them separately. One run while you're coding, one while
Speaker A
you're chatting, and then merge the two into a single profile. That one covers both domains, and it comes out about level with a profile built specially for either one of them. So you end up with one file that just handles whatever you
Speaker A
throw at it. Same mistake as before, an average telling me a scary story about a moment that doesn't happen. But there is one version of that idea I do still want to try. Not switching profiles when the workload changes. Repicking the set as
Speaker A
you go. Because right now the profile is completely static. I choose the experts once before anything starts. And that's what sits on the card until you shut it down. And everything I just showed you says the useful set drifts. So what
Speaker A
happens if I repick it? Not on every token. That's the old design. And that's exactly what the transfer cost killed.
Speaker A
But every 500 tokens, say, have another look. Swap out whatever's gone cold and then leave it alone again. So you'd pay for the transfers once every 500 tokens instead of constantly. And that might just be cheap enough to be worth it.
Speaker A
That's the next thing I want to find out. So how can you actually run this? It's merged into my fork now and it's two steps. First you capture a profile.
Speaker A
There's a separate tool for it. You run your model through it. Give it a prompt.
Speaker A
Let it generate a few hundred tokens and it writes down which experts the router picked. Do that twice with two different kinds of prompt. Something codeish and something chatty. Then just stick the two files together. 500 tokens are run
Speaker A
is plenty. And you only do this once per model. Then you serve with it. Exactly the command you already use plus two flags. Point it at that profile and tell it how many slots per layer you want.
Speaker A
And when it loads, watch for a line telling you how many layers and slots it set up and how much it uploaded. If that line isn't there, it quietly fell back to baseline, which almost always means you asked for more slots than actually
Speaker A
fit. But before you do that, I want to be straight about what you should expect because it isn't the same for everybody.
Speaker A
What matters is how much of the expert set you can fit in your VRAM. On my card running the 35B model, about half of them fit, and that's where all the numbers in this video come from. Now, same card, bigger model. Laguna is 118
Speaker A
billion parameters. Only 36 of its 256 experts fit and the gain drops to 5%.
Speaker A
That's five, not 66. Nothing changed except how much of the model I could hold. So if you're on a smaller card, work that out first. How much of the expert set fits. The smaller that fraction, the smaller the win. And below
Speaker A
about 15 or 20%, picking one fixed set stops being the right approach at all.
Speaker A
Which is exactly the case that repicking as you go might rescue. So that's one more reason I want to try it. I also want to make the slots cheaper, compress the experts harder so more of them fit in the same space and get this working
Speaker A
for the 30B A3B model because I know that's the one most of you are running.
Speaker A
So, back to where we started a cache that the stats said shouldn't help at all. It works because the stats were answering a different question. Nothing is hot over an hour. Something is always hot right now.
Topics:RTX 306035B parameter modelllama.cppexpert cachingGPU inferenceLLM optimizationPCIe transferLRU cachemachine learninglocal LLM
