Speaker A
Okay, so raise your hand if you've been quietly priced out of the good stuff.
Kai tests whether 8GB VRAM is enough for 27B models, examining Bonsai 2 (27B params, 5.95 GB, ternary quantization) against Qwen 3.8 27B and standard Q4 builds.
Key Takeaways
- Bonsai 2 makes 27B-class models runnable on 8-12GB VRAM GPUs, a major accessibility gain.
- Against the realistic baseline (Q4 builds), Bonsai 2 is slightly worse in average benchmark performance but roughly three times smaller.
- Headline benchmark retention figures compare against FP16, which few local users actually run, so they overstate the practical advantage.
- Throughput claims come from short-context tests and drop substantially in real agentic workloads.
- Bonsai 2 requires a specific loader and setup; using standard tools yields broken output.
What the video covers
- Opens by describing the frustration of being priced out of 27B local models with consumer GPUs (8-12GB VRAM) and falling back to weaker 9B models that hallucinate code.
- Introduces Prism ML's Bonsai 2 227B, a 27B model compressed to 5.95 GB, roughly 9x smaller than the 53.8 GB FP16 base Qwen 3.8 27B model.
- Confirms that the size reduction makes 27B-class models runnable on 8-12GB cards like the RTX 3060 and 8GB GPUs.
- Explains the technique: ternary weights (-1, 0, +1), a Hadamard rotation to spread outlier weights before rounding, and one 16-bit scale factor per 128-weight group, averaging about 1.75 bits per weight.
- Argues the headline '98.2% of full precision' is misleading because most local users already run Q4 quantized builds, which average 85.18 versus Bonsai's 84.78 on a 14-benchmark suite.
- Notes Bonsai 2 beats the uncompressed model on five benchmarks (e.g., AIME26, IFBench), which the reviewer suggests should be held loosely since vendors publish their best measurements.
- Reports throughput: 91.1 tokens/sec on an RTX 4090 in Prism ML's numbers, but these use a near-empty context (TG128), while real agent workloads with long contexts are substantially slower.
- Warns that Bonsai 2 does not work in stock llama.cpp or ordinary MLX loaders and requires a specific supported setup (a dedicated repo and build), otherwise users get nonsense output and wrongly conclude the compression is bad.
- Concludes the model is a clear win for users with 8GB cards who previously had no 27B option, but a weaker choice for anyone who can already run Q4 builds, especially for coding agents.
Chapters
- 00:00The VRAM wall and why 27B models felt out of reach
- 00:50Introducing Bonsai 2 and its size claims
- 02:00How ternary quantization and Hadamard rotation work
- 04:00Comparing against FP16 versus the real Q4 baseline
- 06:00Benchmark results and why Bonsai beats its source model
- 08:00Speed numbers and the short-context caveat
- 10:00Setup pitfalls with llama.cpp and MLX
- 12:00Who should use Bonsai 2 and final verdict
Full Transcript — Download SRT & Markdown
Speaker A
Because for about a year now, that's been most of us. And I do mean the whole ritual. We watch somebody run a 27B model at home. The benchmarks go crazy.
Speaker A
Everybody on Twitter is hyped about it. And then we go and look at our own GPU and see 8 gigs of VRAM, maybe 12. And we just close the tab. And we close it because we already know how the math
Speaker A
ends where the model is a 50some GB file and your card has eight. So that is not a bottleneck, that is a wall. And I have been there for about a year. I was stuck on the 9B class of models, wondering why
Speaker A
my local AI assistant kept confidently hallucinating library functions that don't exist. And I'd throw a coding task at it and it would come back telling me to just import super cool function from requests, which doesn't exist, bro. And
Speaker A
every single time a new 27B model dropped, the same cycle ran again. I'd get excited. I'd go check the VRM requirements. I'd feel that familiar deflation. And then I'd go back to the little model that sort of worked if I
Speaker A
squinted hard enough and pretended it was smarter than it really was. Now, on September 17th, Prism ML shipped something called Bonsai 227B. And the headline on it said 27 billion parameters, 5.95 GB, and 98.2% the performance of the full model. And I'm
Speaker A
not going to lie, my first reaction was, "Okay, what did they break?" Because that's usually how this works, where something that sounds too good is quietly missing a footnote somewhere.
Speaker A
So, I dug in and what I found was a lot more than one footnote. There's a genuinely impressive engineering story in here. There's a win of 12 points for anybody sitting on a small card. There are a couple of performance numbers that
Speaker A
will ruin your day if you aren't reading them carefully. And there's one specific use case where this model just absolutely wins. So let's walk through all of it. No hype, no dunking, just the whole picture. And first, why does this
Speaker A
actually matter? Because before we get into the caveats, I want to make sure the real story lands properly. The base model here is Quinn 3.827B.
Speaker A
And if you're anywhere near the local AI world, you know this one already. It came out in August under an Apache 2.0 zero license, free to use commercially, and it's legitimately the best open model that most people can actually name
Speaker A
right now. It's excellent, and nobody really disputes that. Now, in full 16-bit precision, that model is a 53.8 GB file. So, we're talking about a 2GPU machine or a Mac with 64 gigs of unified memory. And for most of us, including
Speaker A
me, for most of this year, that was simply not a conversation we were part of. We read the release notes and we moved on with our lives. And Bonsai 2 is that same model with the same 27.36 billion parameters, the same 262,000
Speaker A
token context window, the same license, free sitting on hugging face, except that the file is 5.95 GB. Divide 53.8 by 5.95 and we get 9.04, which means the 9 times smaller claim is real math rather than marketing rounding. And that number
Speaker A
really does move the floor because a 3060 with 12 gigs can run it. An 8 gig card can run it. And the model class that was a 2GPU affair for a year just landed on cards that go for $300 used.
Speaker A
Now that's the last piece of completely unqualified good news I have and everything after it gets more complicated. But that one thing is genuinely real and I wanted to sit there for a second before we start pulling on
Speaker A
the threads. And the method they used to get there isn't obvious. So I think it's worth actually understanding what they did. A weight in a standard 16- bit model has about 65,000 possible values which is a lot of precision. And a
Speaker A
weight in bonsai has three of them which are minus1, zero, or + one. And your brain probably just said that should destroy the model because we've thrown away almost all of the information, right? But the individual weight values
Speaker A
were never really doing the work because what matters is the pattern across thousands of weights together. So we can lose a lot of per weight precision and still preserve the signal as long as we're careful about how we do it. Now
Speaker A
there is a problem with naive low bit quantization and it's called outliers. In any weight matrix there are a handful of weights with enormous values way bigger than everything else. And those outliers absolutely wreck quantization because the quantization range has to
Speaker A
stretch to cover them which crushes all of the detail in the middle. You end up spending your whole range on a few freak values and everything the model is actually using gets flattened into the noise. So what Prism ML does before
Speaker A
rounding anything is apply a haramar rotation which works like a mathematical shuffle. It multiplies the weight matrix by a matrix made only of plus 1's and minus ones, which sounds weird on its face, but what it does is spread those
Speaker A
outlier values across the whole group. So that instead of a few weights being enormous and everything else being small, we get a much more even distribution to work with. And then everything gets rounded to minus one, zero, or plus one. And they keep one 16-
Speaker A
bit scale factor per group of 128 weights to recover the magnitude. Now engineers call this turnary quantization with blockwise had haram rotation and it lands at about 1.75 bits per weight as shipped and it's genuinely clever because the rotation is solving a real
Speaker A
problem rather than just turning dials. So I want to give credit where it's due here because this isn't the same thing as ordinary 4-bit quantization. It's a different approach and it works well enough to land a 27B model in under 6
Speaker A
gigs without the wheels falling off. Now, I need to slow down because this is the part that got completely buried in the coverage. Every video about this model compares it against the full precision version, but nobody runs the
Speaker A
full precision version. Nobody. We've all been running 4-bit quantized builds for 2 years because that's what fits and that's what works. And the honest comparison is not bonsai against the FP16 original. It's bonsai against the Q4 build. And that's a very different
Speaker A
chart. And Prism ML actually published that comparison themselves in their own model card which most of the coverage apparently didn't read. So let's go look at what is sitting in there on their 14 benchmark thinking suite. The full
Speaker A
precision model averages 86.32 and Bonsai 2 averages 84.78. And that's where the 98.2% retention figure comes from which is just the one divided by the other. And the UDQ4 build, which is the standard 4-bit quan most of us have probably been running,
Speaker A
averages 85.18. Read that again. Because the Q4 build we already know how to run, beats Bonsai.
Speaker A
And it beats it by 4/10 of a point. Now, the size difference is still real because the Q4 build is about 17.6 GB and Bansai is 5.95, which works out to three times smaller for 4/10 of a point
Speaker A
of performance. And that's actually a reasonable trade depending on the hardware you own. But if you have 16 gigs or more, you can already run the Q4 build. That build is also faster in practice, which I will get to, and it
Speaker A
scores slightly higher. So, the headline of 98.2% of full precision isn't wrong. It's just not the comparison that matters for most people who are already running local models. And the comparison that does matter is Bonsai against Q4, where the race is much closer than the
Speaker A
headline suggests. And there is something else in that table that nobody talked about. And it's the thing that should really make us read all of these numbers carefully. Bonsai 2 actually scores higher than the uncompressed model it came from. Not once, but on
Speaker A
five separate benchmarks. On AIME26, Bonsai gets 95.83 against the full precision models, 94.58. On ifbench, it's 74 against 71. And on human evil plus live codebench and the instruction following category bonsai wins as well.
Speaker A
And let me say the obvious thing which is that compression cannot add knowledge. We can't throw away 14 bits per weight and somehow teach the model more competition math because that's not how any of this works. What we are
Speaker A
really seeing there is noise. Because these are small test sets and on something like AM which is a handful of math competition problems, a single question swings the score by whole points. So the signal to noise ratio is
Speaker A
pretty rough and that matters because the gap the whole launch is built on is a difference of 1.54 points and it's smaller than the random wobble baked into their own table. When the variance in the measurements is bigger than the
Speaker A
gap we're measuring, the gap is not a reliable number. Now, Prism ML published no confidence intervals and nobody outside the company has reproduced the full benchmark suite yet. So, the retention figure is a vendor measurement and that doesn't mean it is wrong. It
Speaker A
means we should hold it loosely until somebody else runs it. Vendors don't usually lie about numbers. They publish the measurement that came out best, which is just rational. And the job of a good reviewer is to go find the ones
Speaker A
that didn't make the slide. And there are two of those here. So the first one is the footnote that changes the whole picture because that headline retention is measured at maximum reasoning effort.
Speaker A
Drop it one level down to medium and retention falls to 96%. Which sounds small, but look at what maximum reasoning effort is actually costing us.
Speaker A
Simon Willis, who is the one person to follow in this space if you're only following one, ran a test prompt on Quinn 3.827B 827B at max reasoning and it was just an SVG of a pelican on a bicycle. That prompt burned 22,276
Speaker A
reasoning tokens before generating 3,223 tokens of actual output and it took 21 minutes. Then he ran the same prompt with reasoning off and got 3,715 output tokens in 137 seconds. So now scale that across to Bonsize hardware.
Speaker A
Prism ML reports 91.1 tokens per second on a 4090, which is their measured throughput number. And I'll tell you in a second why that number deserves its own asterisk, but take it at face value for now. 22,276 thinking tokens at 91.1 a second is 245
Speaker A
seconds, which is over 4 minutes of the model talking to itself before you get the first useful word out of it. And on an M5 Max running bonsai at 47 tokens per second, which is also their number, we're looking at 474 seconds or nearly 8
Speaker A
minutes for a pelican on a bicycle. The weights compressed nine times and the thinking did not compress at all because we cannot quantize a token the model has not generated yet. The file is smaller, but the weight is not. So what is
Speaker A
maximum reasoning effort actually buying? Let's look at the benchmarks that sit outside that 14 benchmark average. On terminal bench 2.1, which tests long horizon agentic engineering work, Bonsai 2 scores 52.8 where the full precision model scores 69.7. So
Speaker A
that is 75.8% retained. And we can see straight away that it is nowhere near 98. And on SWE verified, which is real world software engineering tasks, Bonsai scores 60.8 against 80.6. six for full precision, which is 75.4% retained. Now, those two benchmarks
Speaker A
measure exactly what we want a local model for, running agents, working through code bases, handling long autonomous tasks, and they're both sitting at about 3/4, not at 98%. And Prism ML isn't hiding any of this because the numbers are theirs and they
Speaker A
publish them. And they say plainly in their documentation that the long horizon task gains are partial. The number on the billboard is the average that came out best. And the numbers for your actual use case tell a different
Speaker A
story. One independent tester ran 39 tests across 13 tool use modes. And nine of those modes came back perfect, but it failed on reconciliation tasks one time out of three and on irreversible actions two times out of three. In one test, it
Speaker A
was told to delete all records before 2024 and it deleted live rows without checking what it was about to modify.
Speaker A
And that isn't a model you let run unsupervised on anything important. And the second number that didn't make the slide is the throughput table, which is subtle, so follow me here. Prism ML reports 129.9 tokens per second on a
Speaker A
5090 and 47.0 on an M5 Max. And those numbers are sitting in a column labeled TG128, which stands for token generation measured over 128 tokens from a near empty context. But a typical agent workflow is not a near empty context. It
Speaker A
is 40,000 tokens of repository plus a growing KV cache. And generation gets slower as the cash fills, which is just how transformers work. And when people measure Bonsai 2 in realistic conditions with longer contexts and the cash filling up, they were seeing teens to
Speaker A
low 20s of tokens per second, where the standard Q4 build in the same conditions runs 40 to 60. One reviewer gave it a debugging task and something a standard Q4 model finished in under 15 minutes took Bonsai 27 minutes. Another tester
Speaker A
planted a time zone bug in a Docker app and Bonsai reasoned its way to the right answer then talked itself back out of it again and after over an hour the bug was still sitting there. So the file is
Speaker A
three times smaller and in real world agentic workloads it might be twice as slow or worse. And that is the exact opposite of what a 9x compression is supposed to feel like. Now, before anybody opens a new tab to download
Speaker A
anything, there's something worth knowing here because this one will waste your afternoon. Bonsai 2 does not work in stock lama CPP. It doesn't work in out of the box and it doesn't work in ordinary MLX loaders because it needs
Speaker A
Prism ML's own fork specifically built Prism B10658 or newer. And the dangerous part is that the failure mode is silent.
Speaker A
Ola's GGUF parser hits a tensor type it doesn't understand, returns a size of zero, and keeps right on going. And standard MLX loaders just skip the hatamar transform without telling anybody. Neither one errors out. So what you see on screen is fluent, confident,
Speaker A
completely wrong output with nothing anywhere to indicate that something went wrong. And there's already an unofficial Bonsai 2 upload sitting on the registry.
Speaker A
So, if you pull that one and it answers you, the answer means nothing. You're not running the model correctly. You're running a broken version that sounds convincing. Now, this is going to generate bad reviews for weeks where somebody downloads the convenient copy,
Speaker A
gets confident nonsense back, concludes the compression was hype, and posts about it. But the compression is fine and the loader was wrong and nothing on screen said so. So, the supported path is one repo. clone bonsai demo run setup
Speaker A
and start the server and it fetches the correct fork automatically. Use that path and verify the build string before you trust anything the model tells you.
Speaker A
And while we're here, the file size is real, but it's not the whole VRAM story because on an actual 8 GB card at a 4,000 token context, somebody measured 7.4 GB in use. And at the full 262,144 token context window with a 4-bit cache,
Speaker A
we're looking at closer to 12.9 GB and size of 7.4 gigs for normal chat. And understand that the full context window is a 13 gig feature. All right, so I have spent most of this video pulling the numbers apart. And I want to be
Speaker A
clear that this isn't the same as saying the model is bad because for one specific situation, everything I just said points the other way entirely. If your ceiling is 8 to 12 GB of VRAM, then the Q4 comparison I made earlier doesn't
Speaker A
apply to you at all because the Q4 build is 17.6 GB and it does not fit on your card, which makes that whole comparison irrelevant to your machine. And your real options before September 17th were a 9B model, which is noticeably less
Speaker A
capable, or the IQ2 2-bit build of the 27B, which is about the same size as Bonsai and scores 72.59 on Prisml's own benchmark suite. Bonsai 2 scores 84.78.
Speaker A
And that is a lead of 12.19 points at 2/3 of the size. And unlike the two-bit build, it doesn't fall apart on instruction following or on vision tasks. Every other aggressive compression method at this size tier loses the plot and Bonsai holds it
Speaker A
together. Now, that's the honest headline and that's the real engineering achievement because at the two-bit class, this model wins by a country mile. And if that's your machine, if you're on a 3060 or a 3070 or any card
Speaker A
in that range, go download it tonight. Use Prism ML's demo repo rather than the unofficial Olama one. Keep reasoning effort at the default instead of cranking it to max for every prompt. and treat it as the best chat and vision
Speaker A
model that has ever fit on your card because that's actually what it is. And the people calling this a letdown are almost all testing it as a coding agent on hardware that could already run the Q4 build, which is a fair test of the
Speaker A
Agentic claim, but it isn't a fair test of the model and it isn't your situation. So, here is where I land.
Speaker A
Local AI really did hit a tipping point on September 17th, just not the one in the headline. The constraint on running a 27B model at home stopped being memory because Prism ML solved that problem honestly with real engineering. And the
Speaker A
proof of it is a 12point win at the two-bit tier that no other method can match. And what replaced memory as the constraint is time. Because a score that only holds at maximum reasoning effort is a score you pay for in tokens on
Speaker A
every single prompt on the slowest hardware in your house. And the cheapest quantization in the world cannot compress a token that has not been generated yet. So if you have 16 gigs or more, stay on the Q4 build. 4/10en of a
Speaker A
point is not worth taking 3/4 performance on the benchmarks that actually describe the work you do. And if you have eight, this is the best week local AI has ever had. That is the part that never shows up in a file listing
Speaker A
anywhere. And it is the part you feel every time you hit enter. And before we believe the number on the slide, it is worth going to find out which two the vendor left off it because that is where
Speaker A
the real story usually lives. Subscribe if you want more of this and I'll catch you in the next
Topics:Bonsai 2Qwen 3.827B model8GB VRAMquantizationternary weightslocal LLMQ4 quantizationllama.cppGPU benchmark
