Speaker A
141 tokens a second on a single consumer GPU. Same model file you already downloaded. I saw that number in a Reddit post last week and did exactly what it wanted me to do. I got excited.
Analysis of Qwen 3.8 27B model speed claims with DFlash2, clarifying real performance gains and speculative decoding mechanisms.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- DFlash2 provides a modest but real speed improvement (~23%) over Qwen 3.8's existing multi-token prediction.
- Speculative decoding is a key technique to improve token generation speed by guessing multiple tokens ahead.
- Comparisons should be made against realistic baselines, not worst-case configurations, to avoid misleading claims.
- DFlash2's engineering innovations focus on better token selection rather than just better guessing.
- Community benchmarks and vendor data suggest DFlash2's benefits vary with hardware and use case.
What the video covers
- Claims of 141 tokens/sec on a single consumer GPU are misleading when compared to realistic baselines.
- The real speed improvement of DFlash2 over the existing multi-token prediction (MTP) feature is about 23%, not 3x.
- Speculative decoding uses a small fast guesser to predict multiple tokens ahead, verified by the large model to save compute.
- Qwen 3.8 27B already includes MTP, a built-in drafter predicting several tokens ahead, achieving 114.7 tokens/sec.
- The 3x speedup claim compares DFlash2 to a baseline configuration rarely used in practice (47.4 tokens/sec).
- Community testing on various GPUs shows DFlash2 offers minor speed gains at the cost of increased VRAM and context window.
- DFlash2 improves on DFlash1 by scoring candidate token pairs to choose coherent token sequences, reducing gibberish output.
- DFlash2 adds only 2 million parameters and 0.6% latency for a modest increase in accepted tokens per pass, outperforming competitors.
- Engineering innovations include addressing self-attention collapse in the drafter with depthwise convolutions.
- Real-world usage shows DFlash2 benefits depend on workload concurrency and prompt type, with some tasks seeing higher gains.
Chapters
- 00:00Introduction and initial speed claim excitement
- 00:52Clarifying the real speed improvement
- 01:46Speculative decoding explained
- 02:40Multi-token prediction in Qwen 3.8
- 03:40Analysis of Reddit benchmark and hardware context
- 04:24Community feedback on real-world performance
- 05:24Vendor benchmarks and speed comparisons
- 06:17DFlash1 drafter overview and limitations
- 07:18DFlash2 engineering innovations
- 08:11Self-attention collapse fix and drafter efficiency
Full Transcript — Download SRT & Markdown
Speaker A
I opened a new tab. I started reading the comments to find out which flag to flip and how quickly I could get that number on my own machine. And then I actually read the table. Not the headline number, the whole table. Three
Speaker A
rows up from 140.66 in the same spreadsheet from the same person on the same card the same afternoon sits the number 114.7.
Speaker A
And that 114.7 is what the model already does when you've correctly turned on the speed feature it already ships with. So the three times faster claim that's been going around, that's 140.6 measured against 47.4. And 47.44 is a
Speaker A
configuration nobody who knows what they're doing actually runs. Against what your machine would do today properly configured, the new thing is worth about 23%. Now 23% is still real.
Speaker A
And the engineering behind Dlash 2 is clever enough that I want to explain it properly rather than just debunk the headline. But the headline is the headline. And the reason it's misleading isn't dishonesty. It's a table that has
Speaker A
both numbers in it and everyone's screenshotting the one on the right. Here's what's actually going on. Dlash 2 is a drafter. And to understand why the baseline matters, I need to make sure we're on the same page about what
Speaker A
speculative decoding actually does. Because this is the mechanism underneath the number, and the mechanism is the whole story. Generating text is a queue, one token at a time, one full pass through the model per word, and the bottleneck is memory. Your GPU reading
Speaker A
18 gigabytes of weights for every single token it produces. Speculative decoding goes after that waste by using a small fast guesser to predict the next few tokens ahead. Then the big model checks all of those guesses in one pass for
Speaker A
roughly what a single token used to cost. If the guesser was right, four tokens arrive for the price of one. If it was wrong, the bad tokens get thrown out and you land exactly where you started. Wrong guesses cost time. They
Speaker A
cannot cost quality. The big model verifies everything. Think of it like a commute. The big model is the road, same length no matter what. The drafter is the thing predicting which turns you're about to take so the junctions can be
Speaker A
cleared ahead of you. Get the turns right and you never stop. Get one wrong and you rejoin at the last correct junction, having lost nothing but the guess. Now, here's the part that makes the 141 token headline complicated. Quen
Speaker A
3.827b 827B already comes with a drafter inside it. It's called multi-token prediction, MTP, and I already created a video on this topic. A second small head trained directly into the model weights that predict several tokens ahead as part of the file you already downloaded.
Speaker A
It is not a plugin. It is not a branch. It is in the box. And in the Reddit table that everyone is citing, MTP gives you 114.7 tokens per second. The plain autoregressive baseline, no speculation at all, gives you 47.4.
Speaker A
Dlash 2 gives you 140.6. So the honest comparison is 140.6 against 114.7, not 140.6 against 47.4. The three times headline compares the best new thing against the worst possible starting point, which is a configuration that exists only as a benchmark anchor, not
Speaker A
as something anyone runs in practice. I want to be careful about whose numbers these are because this matters a lot.
Speaker A
The Reddit post is from Local Llama, 262 points. The author rented an RTX 6000 for the afternoon, pulled a specific Llama CPP pull request, and ran four prompts through four configurations, self-reported, one person, rented hardware, and they also disclose
Speaker A
unprompted, which is credit to them, that they're from a team that sells local model quants and an app. There is an incentive attached to these numbers.
Speaker A
Both of those facts deserve to be in your head when you read the table. The benchmark table is one person's afternoon on rented hardware. The comment thread is what happens when five different GPU owners try it that same
Speaker A
night and report back. And the comment thread is the more useful document. 25 points from someone with real context.
Speaker A
It's only on low context. On realistic use cases, it's slower than MTP. 26 points. Apple silicon. I couldn't beat my MTP config I already run. Tried multiple options and quants. Someone on a 5090 reported acceptance around 2.5, quite similar to MTP. A 3090 owner gave
Speaker A
the most useful single sentence in the entire thread. Maybe some 10%ish uplift over MTP, but at the cost of the 1.1 GB download. And for the 24 gig card, the context loss just wasn't worth the minor speed bump. A seen XTX owner. Same seems
Speaker A
to be more VRAM for similar performance. Three vendors of silicon, four different cards, one verdict. Roughly MTP parity paid for in VRAM and context window.
Speaker A
That is the answer to the question in my title. Not from people who rented hardware for an afternoon, but from the only people whose opinion costs them something. They own the cards. They spent the time. They tried it and
Speaker A
reported what they got. The vendor's own model card does the same thing in public, just with bigger numbers. Single H200, one concurrent user, autoregressive, 68.9 tokens per second. MTP 178.5, Dlash 2, 236.1.
Speaker A
Their speed up column says 3.43 times and it is measuring against the 68.9 against the 178.5 that the model already does with MTP. Dlash 2 is 1.32 times.
Speaker A
Both numbers are in the same table on the same row. And the one that describes your actual upgrade is the one that never makes it into any headline. I want to spend real time on the mechanism because the engineering behind Dlash 2
Speaker A
is interesting. And this is the part that nobody has actually explained. Dlash one was a block diffusion drafter.
Speaker A
Instead of guessing one token at a time, it filled in a whole block of eight at once in a single forward pass using an attention mask that let every position see every other position rather than only what came before. Fast, parallel,
Speaker A
and it had one obvious failure. Every position picked its own most likely word independently. So you'd get eight words that were each individually plausible but jointly gibberish. On the commute analogy, that's choosing the best looking turn at every junction
Speaker A
separately and ending up in a cul-de-sac. Dlash 2's fix is elegant. Instead of trying to guess better, it focuses on choosing better. It keeps the top 16 candidate tokens at every position, scores every neighboring pair of candidates against each other using what
Speaker A
the authors call a low rank bilinear attention over adjacent candidates, and then walks one coherent route through that lattice. The whole addition costs 2 million extra parameters and 0.6% extra latency. It buys a third of a token more
Speaker A
accepted per verification pass. And the comparison they print right next to it tells the whole story. A competing approach, a correction module from a drafter called DSpark that got hyped heavily a couple of months ago buys 2/3 as much improvement for 77.8 million
Speaker A
parameters and 9.6% more latency. That's roughly 40 times the parameters and 16 times the latency cost for less. The author's own summary line is the best sentence written about speculative decoding this year. Choosing is cheaper than predicting. The second innovation
Speaker A
is smaller and stranger. They measured that inside the drafter, attention paid to the block it's currently drafting collapses with depth. 30% in layer 1, 8% by layer 5. The late layers stopped looking at their own work. So they added
Speaker A
a two-tap depthwise convolution before and after every sublayer, letting each position mix with the one before it.
Speaker A
After that, layers four and five dropped from 9.4% to 0.5% self-attention collapse. Their conclusion, suffix decay is mostly a local problem. A five-layer drafter with those convolutions nearly matches a 15-layer one at a third of the
Speaker A
parameters. That's real engineering and it's the reason I don't want
Speaker A
concurrency from one request to 32, which is what a hosting company does, what any shared inference server does, and the whole idea evaporates. MTBench at 32 concurrent users, auto reggressive, 157 tokens per second, Dlash 2,525.
Speaker A
That's 1.01 times, essentially nothing. And MTP and DSpark don't just match auto reggressive at that scale. They go below it. 0.77 and 0.74 times. Actively slower than doing nothing clever at all. The reason is mechanical. Speculative decoding spends spare compute to cut
Speaker A
your waiting. At one concurrent user, there's lots of spare compute and the technique makes sense. At 32 concurrent users, there's no spare compute to spend. Every GPU cycle is already assigned. Drafting just adds overhead with nothing to fund it. Which means
Speaker A
this is a technique whose best case is precisely one person, one GPU, one request at a time. It is nearly worthless to a cloud. It is aimed exactly at us solo developers running local inference on consumer hardware.
Speaker A
That inversion is in nobody's title, but it's the thing I found most interesting in the whole release because it describes the techniques actual terrain more honestly than any benchmark number.
Speaker A
I've been careful about the numbers. So, let me be equally careful about where the praise is actually earned because there is a workload where this technique does what the headline promises and it's a workload a lot of us run regularly.
Speaker A
Someone on a single RTX3090 spent a week tuning Dlash 2 in public and got to 381 tokens per second on one particular task. Read what that actually is. About 133 on ordinary chat prompts. 382 when the model is reproducing its own
Speaker A
context, quoting a document back at you, or applying edits to code it just read.
Speaker A
In that mode, 15 of every 16 drafted tokens got accepted. Their verdict in their own words plus 53% where the model reproduces its context about 2:1 behind plain MTP everywhere else with twice the time to first token. So it's a mode for
Speaker A
rag frontends and coding assistants applying edits not a default. That is the real answer written by the person with the most to gain from overselling it. If your workload is retrieval, dumping a document into context and asking questions about it, or a coding
Speaker A
agent that reads a file and then applies changes to it, the drafter guesses right most of the time because the output heavily mirrors the input. That's where 381 comes from. That's where Dlash 2 earns every word. There's also something
Speaker A
worth noting that the vendor actually admits. They publish their own ceiling with a perfect selector. Acceptance length would be 6.79 instead of 4.2. 27 per pass. They call their version the simplest selector we could think of.
Speaker A
Vendors don't usually print how much of their own headroom they left on the floor. That's a genuinely honest disclosure. Before you go and set this up, I want to be honest about what the install path actually looks like right
Speaker A
now because it's not a checkbox. VLM wants a pull request branch that hasn't merged to main. Lama.cpp is pull request 27342, also not in a tagged release. OLAM is issue 17865.
Speaker A
SGLANG is a git subdirectory. The Apple path needs a pre-built fork that caps you at block size 5 instead of the 8 the benchmarks use because the quantized maple kernel gets less efficient at larger verify widths. That's very likely
Speaker A
why the Apple silicon owner in the comments couldn't beat their existing MTP config. Nothing that runs Dlash 2 is in a release build of anything. You're pulling branches. And there's one more thing. During the same week all these
Speaker A
benchmarks were being posted, a developer had to retract a benchmark table. Someone filed an issue saying the long context numbers didn't reproduce, they were right. Because the tables had been measured in a batch configuration that runs no speculative decoding at
Speaker A
all. So the numbers weren't measuring Dlash 2 in the first place. The developer held up, corrected it publicly, and gets credit for that. But that's the state of the evidence on this topic. 3 days old at time of writing.
Speaker A
Every table self-reported, one already withdrawn. I also want to be transparent about my own gap here. I haven't run this myself. Every figure in this video belongs to someone else on their hardware. And I've told you who's each
Speaker A
time. Here's the practical call. If you're on a 24 GB card, don't do anything this week. The people who tried it on your card got 10% over MTP and paid for it in reduced context window.
Speaker A
That's not a trade worth making for most workloads. Wait for the pull requests to land in tagged releases. If your work is retrieval, rag pipelines, a coding agent that reads files and applies edits, this is worth the branch install right now.
Speaker A
That's the one workload where the drafter guesses right 15 times out of 16 and 141 tokens per second turns into 381. The technique earns its marketing there. And whatever you decide about Dlash 2, go and check whether MTP is
Speaker A
already on. That number 114.7 versus 47.4 or 178.5 versus 68.9 is the real upgrade hiding in most people's configs right now. The drafter is already in the weights you downloaded. Half the people posting Dlash 2 comparisons this week are
Speaker A
quietly discovering they had never enabled MTP in the first place. Fix that before you evaluate anything on top of it. The one signal worth watching is those three pull requests merging. The day Llama CPP27342 lands in a tagged release is the day
Speaker A
this stops being a benchmark argument and becomes a checkbox. And then the only honest comparison left is DFl2 against MTP. The one that was in the table all along, three rows up from the number everyone screenshotted. The road
Speaker A
is the same length either way. All speculative decoding does is guess your turns. And a shortcut is only a shortcut against the route you were actually driving. 141 tokens per second is real.
Speaker A
It was measured on real hardware by a real person. The question is what it's a shortcut from. And the answer is a baseline that nobody in practice runs.
Speaker A
Sitting three rows up in the same table that produced the headline. The 23% over MTP is also real. And for retrieval and edit application workloads, something much bigger is real. That's the honest shape of this. If this helped you read
Speaker A
the next benchmark table a little more carefully, or at least go and check whether MTP is already on before you pull a development branch, that's everything I was going for. Hit the like button if it was useful. Subscribe for
Speaker A
more honest breakdowns like this, and drop a comment with what you're getting on your card with MTP switched on because I suspect a lot of people in the comments section are about to have the pleasant surprise of discovering they'd
Speaker A
never turned it on. See you in the next one.
Topics:Qwen 3.8DFlash2multi-token predictionspeculative decodingtoken generation speedGPU benchmarkslanguage model optimizationAI model inferencemachine learning engineeringmodel performance
