Speaker A
Two numbers, same graphics card, same model file, same prompt, 60 tokens a second and 167. Nothing changed between them. No new hardware, no smaller quant, no quality given up anywhere. One flag did all of it, and it already shipped
Qwen 3.8 27B model runs 3X faster using multi-token prediction (MTP) with no quality loss, enabled by a single flag already in the model.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- MTP is a free, lossless speedup feature already included in Qwen 3.8 27B model files.
- Many users miss out on speed gains due to disabled flags or outdated flag names in runtimes.
- Speculative decoding guesses multiple tokens at once, verified to maintain output quality.
- Speed gains depend on token guess depth and temperature settings, requiring tuning for best results.
- Hardware differences affect achievable throughput; 4-bit quantization benefits Nvidia Blackwell GPUs specifically.
What the video covers
- Qwen 3.8 27B model achieves up to 3X speed improvement on the same hardware using a single flag called multi-token prediction (MTP).
- MTP is integrated into the model weights and requires no additional downloads or hardware changes.
- Speculative decoding guesses multiple tokens ahead, verified by the full model to ensure no quality loss.
- Many tools have MTP disabled by default or use outdated flag names, causing users to miss the speed gains.
- The speed depends on how many tokens are guessed ahead and the sampling temperature setting.
- Different runtimes implement MTP with different flag names or methods, causing confusion and underuse.
- Performance varies by hardware, with Nvidia, AMD, and Apple devices showing different speed gains and trade-offs.
- 4-bit quantization and other formats can improve speed but are hardware-dependent and separate from MTP.
- Memory usage increases slightly with MTP due to the draft head, but the download size remains unchanged.
- Properly enabling MTP and tuning parameters can unlock significant throughput improvements without sacrificing output quality.
Chapters
- 00:00Introduction to speed difference and MTP feature
- 00:45Variability in reported speeds and usage context
- 01:23Technical explanation of model memory bottleneck
- 01:45How speculative decoding works with MTP
- 02:23Model integration and flag usage across runtimes
- 03:04Common issues with flag naming and throughput impact
- 04:10Tuning MTP parameters and temperature effects
- 05:27Hardware-specific performance and quantization formats
- 06:58AMD and Apple hardware performance insights
- 08:12Summary of throughput gains and trade-offs
Full Transcript — Download SRT & Markdown
Speaker A
inside the model file sitting on your desk. It is called MTP, multi-token prediction. And in almost every tool that runs this model, it is off. QN put these weights out on the 14th of August 2026 with an MTP head trained in. So,
Speaker A
the speed is free, nothing extra to download, nothing traded away. The output comes back identical, which is where it gets awkward because the number in your terminal is probably nowhere near it. Read the measurements posted on release day and that three times traces
Speaker A
back to one row of one reporter's table. The reporter who measured 167 on agent turns measured 93 on pros. Same card, same file, same flag, nearly double decided only by what you asked it to do.
Speaker A
There is also a dial on this thing, and most published recipes set it short of where it pays best. So, what is this flag actually doing? Who turned it off, and what does it buy on your hardware?
Speaker A
Start with what it fixes. Generating text is a queue with one server, one token, one full pass through 27 billion parameters, then start over for the next word. And that pass is not limited by arithmetic. It is limited by memory.
Speaker A
Your card spends most of every pass dragging weights across a bus so it can multiply each one exactly once. So, a card doing 80 tokens a second is not thinking 80 times.
Speaker A
It is reading the same 18 GB off memory 80 times for 80 single words.
Speaker A
Speculative decoding goes after the waste instead of the work. Something cheap guesses the next few tokens ahead.
Speaker A
Then the real model checks all of them in one pass for roughly what a single token used to cost on its own. Guess right and four tokens arrive for the price of one. Guess wrong and the bad ones get thrown out and you land exactly
Speaker A
where you started. A wrong guess costs you time. It cannot cost you anything else. That last part is the one to hold on to. The full model verifies every drafted token before it ships. So, speculation is incapable of changing the
Speaker A
answer, which means switching this on costs you nothing in quality. Usually, the guesser is a second model you have to find, download, and keep resident in memory. Qwen trained one into the file instead. Ollama has 12 tags live for
Speaker A
this model. Three of them marked MTP, and the 4-bit tag weighs 18 GB with it and 18 without. One gotcha before any of it works. This is a dense model, so the flag half the internet reaches for, the
Speaker A
one that pushes experts onto your processor, does nothing here at all. There are no experts to push. It is plain layer offload plus the speculative flag. So, it is free, it is lossless, and it is already sitting on your disk.
Speaker A
Which makes the real question, why so few terminals are showing the fast number. Part of the answer is dull. Four runtimes support this. Four of them spell it differently, and not one ships it switched on. In llama.cpp, it is a spec type. In vllm, it hides
Speaker A
inside a JSON blob nested in a serving config. In SG Lang, it is a speculative algorithm. In Ollama, you do not set a flag at all. You pull a different tag.
Speaker A
Four names for one capability, but the sharper answer is a rename. On the 13th of May, 2026, 3 months before this model existed, the llama.cpp project changed how that flag is spelled. Multi-token prediction itself landed in the main branch 3 days after
Speaker A
that. The old spelling was not removed with an error. It is still accepted and quietly ignored. So, generation keeps running and speculation simply stops.
Speaker A
Your throughput collapses, your command line still reads correct, and the law gives you no reason to look. One write-up measured the damage on a 4090, about 140 tokens a second falling back to about 70. Half the throughput gone
Speaker A
from a setting you are certain you configured because one word inside it went stale. And every guide written before that Wednesday still carries the dead spelling. So, the reason this feels like a secret is not that the capability
Speaker A
is obscure. The capability shipped. The instructions rotted. Fix the word, switch it on, and you meet the dial.
Speaker A
The flag takes a number. How many tokens to guess ahead? On the day this model shipped, a developer swept that number from two up to eight on a workstation Blackwell card and published the whole curve. At two guesses ahead, 81% of
Speaker A
drafts survived the check. At five, 56%. At eight, 40. Guess deeper and you are wrong more often, which is exactly what anyone would predict. But, throughput refuses to follow acceptance down. It climbs to 115 tokens a second at five
Speaker A
steps, then falls back. There is a peak in the middle, and most recipes online ship two or three, which leaves as much as 27% on the table. The number the community copies is not the number the hardware wants. Then, a second reporter
Speaker A
found something stranger. His draft acceptance had collapsed from around 85% on the previous release to around 65 on this one. And the cause turned out to have nothing to do with the model. It was temperature.
Speaker A
He was running the sampler at 1.0 because that is what this model's own authors recommend. A hotter sampler flattens the next token distribution, which makes the draft head guess wrong more often. He swept it downward and watched acceptance climb.
Speaker A
So, the setting that decides your speed is not the speed setting. It is the quality setting, three lines earlier in the config, which puts a real choice in front of you before any of the rest of this matters. Would you turn that
Speaker A
temperature down against the recommendation to buy the throughput because the ceiling on Nvidia is real, and it sits well above anything a flag can reach? SG Lang published 206.1 tokens a second on 15090 on release day, and a reader replied underneath claiming
Speaker A
216 on Windows. For the small desktop machine, they quoted 38.28. An independent operator benchmarked that same box for a full day and got 34 in real use, 38 averaged, 46 at peak. About as cleanly as a vendor number ever
Speaker A
reconciles. But, read what produced it. That headline is not multi-token prediction. The credit belongs to a 4-bit format called NVFP4, running alongside a completely separate drafter, and the 4-bit format only exists on Blackwell silicon. So, on a 3090 or a 4090, that number is not
Speaker A
slower. It is unreachable. The tensor cores that run 4-bit natively are simply not on the die. Somebody asked exactly that under the announcement, and nobody there contradicted them. On AMD, the interesting part is not a number, it is
Speaker A
a choice. AMD's own release day post quotes up to 51.8 tokens a second on a workstation Radeon, and every figure in it was measured on Vulkan, not on ROCm, not on the stack AMD builds, staffs, and markets. Independent sweeps put Vulkan
Speaker A
20 to 30% ahead on generation for these cards, with ROCm ahead on prompt processing. But, a vendor publishing its own launch numbers on somebody else's back end says more than any blog post about driver maturity ever could. And a
Speaker A
16 GB Radeon owner posted both halves of the real trade. At 128,000 tokens of context, 32 a second. Half the context, switch the flag on, 51. That is nearly 60% more speed bought with half his context window. The download is free.
Speaker A
The memory is not. The people who merged this measured about 2 and 1/2 extra GB held resident for the draft head. On Apple, the port is real, and somebody checked it properly.
Speaker A
A 48 GB laptop went from 8.3 tokens a second at 8-bit to 20.3 with a ported drafter. 2.45 times on average, three times on max, and under two on chat.
Speaker A
That same run turned up something better than the multiplier. 8-bit with drafting beat 4-bit without it, 20 against 14.6.
Speaker A
Better answers and more speed in one run, which is a trade quantization does not usually offer anybody. A desktop Apple owner reported the biggest jump in any of these threads, 10 tokens a second on llama.cpp, P 40 to 70 on a metal
Speaker A
engine with handwritten.
Speaker A
He asked whether the stack does prefix caching for this hybrid architecture at all, because an agent session rereads its whole history on every single turn.
Speaker A
If it does not, he argued, three times becomes half a time the moment a session gets long, and his claim is that no server on the market yet has both prefix caching and speculation for this architecture.
Speaker A
That is an argument rather than a benchmark. It is also the right question. Two shiny things arrived this month that do not transfer either. 17 days before this model shipped, a second speculation method landed in llama.cpp with strong numbers on smaller models.
Speaker A
A developer converted the official version for this one, ran it on a 5090, and reported no gain at all, sometimes a loss. Draft acceptance, 29%.
Speaker A
And a fork advertising 30 to 50% more throughput turns out, in its own published table, to mean a different model family entirely. For a dense 27B like this one, that same table claims 5 to 7%. This model is not on its
Speaker A
supported list, so strip it back. Where does a free, lossless, already downloaded speedup buy you nothing? On the machine that needs it most. A 12 GB laptop owner published both runs. 4-bit with the flag on gave 4 and 1/2 tokens a
Speaker A
second. 2-bit with the flag off gave 12.8. 2.8 times faster from the crude thing the clever thing was meant to replace. His model did not fit on the card, so his real bottleneck was the cable out to system memory. Speculation
Speaker A
hides memory latency. It cannot hide a transfer that has not started yet. So, here is the call. Turn it on today on every machine where the model already fits because it is lossless and it is sitting in the file you downloaded. But,
Speaker A
buy your video memory for the quant, not for the flag. Multi-token prediction multiplies the speed you already have.
Speaker A
It cannot create one. The receipts are three curves. Acceptance falling from 81% to 40. One card getting 60, 93, and 167 off the same file depending only on the job. And a laptop where switching the flag off one, 12.8 against four and a half. What
Speaker A
I am arguing with is not a company. It is the single number. Every figure here is true and every one of them is a ceiling somebody measured with the conditions cropped off the end.
Speaker A
So, if a flag this free has been shipping since May and still arrives switched off everywhere, how much of the compute being bought right now is paying for a default that shipped in the off position?
Topics:Qwen 3.827B modelmulti-token predictionMTPspeculative decodinglanguage model speedupNvidia GPUsAMD GPUsApple Metalmodel optimization
