Speaker A
On this channel, I've been testing pretty much every speculative decoding method I can get my hands on. And this week, we got Quen 3.827 billion, which is one of the most interesting new local coding models. It beats everything else
Exploring Qwen 3.8 27B and DFlash 2 speculative decoding methods in llama.cpp with benchmarks and detailed explanations.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- Qwen 3.8 27B sets a new benchmark for local coding models in its size class.
- DFlash 2 significantly improves token acceptance by better path selection without increasing draft complexity.
- Combining multiple speculative decoding techniques in llama.cpp can yield faster and more accurate inference.
- DFlash 2 achieves over 20% more accepted tokens per verification pass with only about 1% added latency.
- Better token selection is more efficient than predicting new tokens, reducing wasted computation.
What the video covers
- Introduction to Qwen 3.8 27B, a new local coding model outperforming others in its size category.
- Overview of DFlash 2 update, which improves token acceptance with minimal latency overhead.
- Explanation of combining speculative decoding methods like DFlash and n-gram in llama.cpp for better token generation.
- Detailed mechanics of how DFlash 2 scores neighboring token pairs to improve path selection.
- Comparison of token acceptance rates and latency between DFlash, DFlash 2, and other methods.
- Discussion on how DFlash 2 maintains lossless rejection sampling while increasing accepted tokens.
- Presentation of benchmark results showing improved performance on limited hardware using llama.cpp.
- Insights into the trade-offs between parameter count, latency, and token acceptance in speculative decoding.
- Explanation of dynamic convolution and sequential Markov head techniques used in DFlash 2.
- Practical advice on choosing decoding methods and configurations for AI inference on constrained devices.
Chapters
- 00:00Introduction to Qwen 3.8 27B and speculative decoding methods
- 02:40Parallel token production and challenges in DFlash
- 05:33DFlash 2 improvements in token path selection
- 08:02Scoring mechanism and selector operation in DFlash 2
- 10:41Benchmarking token acceptance and latency claims
- 13:04Layer analysis and dynamic convolution explanation
- 20:13Comparisons of DFlash 2 with other methods and models
- 27:46Practical advice and concluding remarks on decoding methods
Full Transcript — Download SRT & Markdown
Speaker A
in its size benchmarks. Then just a few days later, Dlash got a major update, the Dlash 2, promising better acceptance than this park with much less overhead.
Speaker A
But here's something a lot of people don't know. In Lamascp, you don't necessarily have to choose just one speculative decoding method. You can combine a drafter like the flash with engram speculation and let the LMAC CP combine those results for you to get as
Speaker A
many tokens as possible. So today we are going to test normal decoding, engram one, two, and combinations of them to see how fast we can push co 3.827 billion on lamas because this is the inference engine most people use on limited
Speaker A
hardware. And if you are new here, I'm Bukash. I'm a senior inference engineer. And on this channel, I explain new AI inference techniques in simple intuitive terms and then benchmark them properly on real hardware. First, let's jump to the video. Here we can see Dlash 2
Speaker A
pushing parallel drafting one step further. Incoai claims that we have more than 20% more accepted tokens for every verification pass for around 1% added cycle latency. So we'll test that and as we can see we have the support for main
Speaker A
inference engines like VLM LCP or Alex. Now let's start the quick explanation of why it go now so much faster. So, Dlash claimed 20% more tokens out of every verification pass for 1% of extra latency with the output probably
Speaker A
unchanged. So, we'll check that for LSP with concurrency one because that's what most local users will use. So, the headline demo runs at 70 tokens per second on their laptop. So, it's a claim. We don't know what's the
Speaker A
measurements exactly, benchmarks and so on. And if you know from our other tests, you can influence the score quite a bit to make it look good. So the fix one that they have is the right token was already in the drafter list. So it
Speaker A
was just not picked well enough. So now it will pick it better. Then every drafter can finally see the slots before it because in Dflash, you can check my other videos. We produce all tokens immediately in parallel. So if you
Speaker A
produce token one and two, they appear at the same time. They don't know anything about each other. This will be important, but we'll explain it in much more detail soon. So, two new parts and one unchanged contract. So, as I said,
Speaker A
we'll now keep the top 16 candidates per position and score the nearby pairs and one coherent path is walked out of scores that were all computed in parallel. So, we add a bit of parameters, but it's only 0.6%. 6% per cycle. So we will use
Speaker A
the convolution as the backbone. So a two tap dynamic convolution will let every draft token position read its predecessor. So the accuracy will stop sliding at the end of the block. Then we still have one pass and it's still
Speaker A
lossless. So it's supported everywhere. We choose this stem basically because the verification comes from the big model then it's by theory will not be lost because the big model makes decisions.
Speaker A
So basically Dlash 2 will not draft more, it will keep more of what is already drafted. Now let's look how the selection will work. So we have a lightweight path selector. We have the speculative decoding and then we decode
Speaker A
is and this is already verified. So it will go here. Then we have mask which is three draft tokens that we want to get and the suggestion of these tokens are at position one, two, and three. Then we
Speaker A
have for example the accepted output that was accepted before in the turn before. Yeah. So the D flash predicts all three predictions in one pass, which means that position one doesn't know what's on position two. Position two doesn't know what's position one and
Speaker A
three. And why it's important? Well, in this case, we have the same tokens in positions one and two, which is not right for any type of sentence that we'll have. So this will mean that if we pick the same word, the verifier will
Speaker A
reject the block at position two because we can't have this. So we'll lose then all of these draft tokens and compute for nothing.
Speaker A
So the interesting part is that the D flash already keeps the 16 candidates in position one, two, and three already. And in these 16 candidates, there is 99.5% of correct tokens most of the time. But because we didn't know basically what's
Speaker A
position one and two are between each other, then we couldn't choose the best ones. So we only get like 85% most of the time. So now let's see what Dlash 2 will do. So it doesn't make a better
Speaker A
prediction. It makes better choices from predictions it already had. So it will score every neighbor pair as we can see here all of them in the same pass. So 7 times multiply by 16 times which gives us around 1,700
Speaker A
scores actually 792. And it will do it all in one shot. And now what will it give us?
Speaker A
Well, we can then walk and we can see what is in front of us. So, we can make better choices.
Speaker A
So, one score is two numbers added together. How much the drafter likes the word on its own which is already which it already had plus how well it follows the word before it. So, in this case, how well it likes the word here a and
Speaker A
then the score of itself. And that's how it will work in selector. So we see now that the block survives the selector only adds the 2 million parameters.
Speaker A
Whoops. So as we can see here, we then choose Shopkins and we can see that on the output they will get accepted. So we get selected the path. So the block survives and the selector adds only 2 million parameters and 0.66 cycle
Speaker A
latency and this increased the accepted length from 4.27 to 4.71 and for this this park needs around 40 times more parameters for much less acceptance.
Speaker A
Now this is the formula that we can see here. So we have the purple one. Uh so this means how much the drafter already liked B on its own. So it had already had this number. So for example for this
Speaker A
the B will be good and then we have the full formula and A as we can see the A is the position before. So what will be the rest of the formula? How well B follows A in this context basically. So
Speaker A
every line in the grid carried one of these scores. So it's not like token yet. Yeah, it's just scores and then tokens.
Speaker A
So by doing this the block survives and same drafter, same single pass, same lossless rejection sampling get to 4.71 from 4.27.
Speaker A
Now let's look how they measure this claim. So they use something that they named Rico and the blue one is how often the token that we choose to draft is actually right. Then we have the green which is how often the right token was
Speaker A
somewhere in its 16 candidates that we described before. So as we see we start from 85% in the blue case and we go to 72. But if we consider the 16 tokens we go from 99 to 87. So as we see we are
Speaker A
better in position six than we are in position zero which is typically the highest because it's straight after decode. So it's easy to know what word would be after.
Speaker A
Now there is something they name oracle. So basically if we use the green method we go from acceptance length of 4.27 to 6.79.
Speaker A
And now what it tells us is that choosing is cheaper than predicting. Here we have D flashpark and then D flash 2 with path selection.
Speaker A
So we add compared to D flash to DSpark 77.8 million parameters and we add around 9.6% latency for this at position zero we get 4.49 tokens from the base which is 4.27. But if we do path selection so
Speaker A
we add only 2 million parameters and 0.66 latency we get 4.61 which is much higher than this park in this case. So if you watch the other video you know how it works there is sequential Markov head. So basically with 40 fewer
Speaker A
parameters and 16 and 16 less latency. So if you watch the other video you can see that the sequential Markov head which we explained before is with 40 times fewer parameters and 60 times less latency lose with the path selection
Speaker A
and we still get the ceiling which I
Speaker A
suffix decay. Before I start explaining this graph, two quick definitions because it will be unreadable without them. So we have file here. This is number of layers and layers are def how much computation happens in one pass.
Speaker A
There are also positions which are with so how many tokens come came out of it.
Speaker A
So separate axis a five layer drafter and 15 layer drafter can both draft seven positions. Yeah. So we have recall one here by draft position. So recall one is how often the drafter's top guess for a position is literally the same
Speaker A
token the target model would have written. So it's basically the correct one. And one more thing before the numbers because it kills the obvious objection.
Speaker A
This uh measurement already removes compounding. So position six uh number only comes runs where position zero through five all came outright. So a weak tail is not errors piling up. It's drafter running out of capacity which is something architecture can fix. So here
Speaker A
we have the reference drafter which is in blue. It have five layers like we said and we see that we start with 85.4% of the time at first position. Yeah, this is the draft positions and as we can see we go lower to around 73%.
Speaker A
So quick also note we start from 16 not from zero here. Yeah. Uh and then let's see what will happen if we drop just to three layers.
Speaker A
So a three layer draft starts in almost the same place but as we see at the positions will have less accepted tokens basically.
Speaker A
Now let's look at another one. So now we can fix it with a depth. So we add 15 layers instead of five and three. Yeah.
Speaker A
And as we can see the 15 layer drafter holds the tail much better. Yeah.
Speaker A
Because as we see it ends later and the deep is not as steep but at the first position we almost gains nothing. So we draw like so much more layers and as we can see there is like a bit of increase
Speaker A
but nothing crazy. So let's look at the cheap version. Now we have two convolutions per sub layer on the five layer drafter. I will show diagramraph uh later. So as we see only three layers and we add 3% of parameters
Speaker A
with convolution and we basically have nearly the same score of the 15 as the 15 layer. So 3% more parameters and 0.7% more latency. So as we see dev pays for capacity at every position and convolution solves it where the problem
Speaker A
counts because we don't want to have like such dramatic dip here in the later draft positions.
Speaker A
We want to have as long draft as possible. So basically the convolution attack the exact problem.
Speaker A
Now let's talk about attention. So one cell here is one attention head and the drafter have five layers and each layer have 32 heads. So the grid holds 160 heads and this is like one head of attention.
Speaker A
Yeah, attention heads. And we have the layers I described before. And the color is the percent of attention spent inside this block.
Speaker A
Now brightness, yeah, it's not accuracy. It's each head splits each so each head splits its attention between the context before the block and the position inside the block. So the brightness is the share of that stay inside. So the market
Speaker A
one here sends 50.8% of its attention to the draft block and the rest to the context. Yeah. Now let's look at another one. So in first layer we can see that many heads are bright. So the early layers do the work of linking the draft
Speaker A
positions to each other. So the layer one average is 30% inside the block. Now layer five average is only 8%. Why?
Speaker A
Because we add a lot of context here. So by layer five almost the full row is dark. These heads now spend their attention on the context instead of.
Speaker A
So what remains sits in few heads. So a few handful of cells carry early all the within block work in the last two layers. So layer five and six sits at 76% and most of its neighbors are under 1%.
Speaker A
So nobody designed it. Its training produced it because the context is where attention gains most and the whiffing block job was left behind.
Speaker A
So we give the whiffing block job a separate model and let attention keep the context chop. Now how we do it that's where the convolution sits. So we have the attention MLP this is one draft layer repeated five times. Yeah we add
Speaker A
convolution between them. So Dlash 2 inserts these small convolutions around the sub layers. So as we see sub layer is attention or MLP and the post places one before and after each attention and MLP sub layer and the
Speaker A
convolution don't replace attention just since sits around them. Now let's look what inside one convolution these boxes are not tokens each box is the hidden state of one position a long vector of numbers basically. So a real block holds
Speaker A
four to 16 positions and they are shown here. One, two, three. So now we'll look what's inside one of these boxes. So take one draft position. Will the drafter works? It's not a word. We have here the hidden state. It's a list of
Speaker A
numbers, one per channel. So we have a few thousand numbers for this one slot.
Speaker A
Those numbers then pass through the drafter layers that are purple. Every layer changes the number. There are still numbers at the end of the stack.
Speaker A
So nothing here has committed to a word yet. Now the LM head is the part that leaves the number behind. It turns them into a score for every token in the vocabulary.
Speaker A
So we have a 100,000s or so scores for one position. As we have can see here, we have good simple clean and the scores for it. The scores are illustrative of course. And then now we choose award and
Speaker A
that's when we accept it. Yeah. So the drafter takes the top scoring token and this possession finally will be accepted. So we put good into it and everything before this point was reversible. So it's not like a commitment hit. Yeah.
Speaker A
And this convolution sits in the middle on the numbers. So blending numbers is a multiply and an add. So blending words would mean picking a word first and that forces the position into a queue. So the conversion works here on numbers and
Speaker A
then it become word only here. So each output position start from its own input vector scaled by k0. K0 it's like itself and nothing is lo so nothing is lost from the position itself. We can see the formula here. So now then it adds a
Speaker A
vector of the position before it scaled by K1. K1 is the information from predecessor and position two here now carries a trace of position one. Yeah, this is the link that attention gave up in the later layers. So position one here has no
Speaker A
predecessor inside the block. So it reads the last verified token that's a real token not a guess. So this block is like anchored to the text that the target model that the target model already produced. Uh before in other
Speaker A
explanations people also use anchor. Yeah. The first decode token from where the draft starts basically.
Speaker A
And every arrow starts in the input road. So no position reads another position output. So all of them compute at the same time. Yeah. That's why we have this position one multiple times because we don't want to do the
Speaker A
connection like between them because they all are computed in the same time like all of this poof is computed once h but because we use the tree and we do then this then it's not sequential like in this spark so it's one fuse operation
Speaker A
now the two weights are not fixed each one is learned based on kernel plus a smart correction computed from the current hidden state. This is what dynamic means here. The mix changes with the content instead of applying one
Speaker A
fixed rule and then the convolution keeps no state between draft cycles and never reads outside the block. So attention the LLM head and the verifier stays unchanged.
Speaker A
It drops into an existing D flash drafter without touching verification. So the output stays probably the same.
Speaker A
A kernel that reaches one position back recovers most of what 10 extra layer buys.
Speaker A
So we added 16.5 million added parameters which is 3%. We added 0.7 added cycle latency and with it within block attention layer four and fives it will fall from 9.4 4 to 0.5. So by doing these two fixes together, we add
Speaker A
basically 1.05 tokens over the Dlash one. So here we have various data sets GSM 8K, MAF 500, human evil, MVP, MTB badge and we can see here mean. So the MTP was the beginning of the speculative decoding with Eagle 3 and so on. The
Speaker A
mean was around 4.54. So this was the drafter tokens. Yeah. Acceptance. And then with D flash we have 492 with the spark 499. And with the flash 2 we have 597. So it's like evolution. So we have plus 21% mean
Speaker A
acceptance length over D flash and 0.48 over DSpark. We only added 1 dot% of the latency as I said many times before. and the acceptance held for the last position up to 86% on my 500. Yeah. So because yeah it depends on the task. Now
Speaker A
there are two drafters out today with this method. Decoin 3.8 27 billion and Muse Gleimmer 30 billion. So you can see the scores here and as we see the Muslim performs better with this method which is quite interesting
Speaker A
and more than a full token ahead of a community spark there on both models.
Speaker A
Yeah. So this is the flash one DSpark D flash 2. Here we have FTP compared to the flash 2.
Speaker A
Now you can of course run it tonight as we said. So this is the copy of the runs but you can check the basically once on the website it will be linked down below. And now let's go to the test.
Speaker A
Okay. So now we'll go to the benchmarks that I run nearly all day for 3 4 days.
Speaker A
And first thing first is that the flash 2 allow only up to seven draft tokens at the same time and the flash one in LAP only 15. But this could change because this is prepr version. Now regarding the
Speaker A
bench, I use uh 6,000 pro blackwell and 96 GB of RAM. This is the LA CP version all in concurrency one without reasonings because some benchmarks basically don't work with it. And also it make it super long because Quen like
Speaker A
to overthink and then we need to go into reasoning and then there is like a lot of talks which reasoning is best and it seems that most people use medium. Uh but it change also depending on which quant you use. So yeah basically we turn
Speaker A
it off. Okay. So when I was finishing editing this video at 2 a.m. there is new update on artificial analysis for intelligence of this model. So actually with the XH high you can run this model locally and it will perform as good as
Speaker A
D64 Pro nearly and GPT 5.6 six which is crazy for local model you can run it on one GPU like 590 [sighs and gasps] and also medium and low so probably you would want to go with medium or x high
Speaker A
but this is for the artificial analysis but uh genic index is actually better why because you do coding this is like typical model for coding I mean it's also good for all other stuff because thanks to distillation But
Speaker A
what we can see here is that this local model that you run at your PC is actually seventh from the top from all the models measured here, which is crazy. So yeah, so it's high is 51 and medium is 50. So
Speaker A
honestly looking at these two and knowing quen that it likes to think a lot I would use probably medium and start with medium and then go higher. So I know I saw there is like 20 videos about which thing you should use but yeah
Speaker A
answer probably here and I don't watch these videos but probably medium is the answer looking at this course and knowing the architecture of these models. So yeah, now back to video and we use greedy sampling to have as
Speaker A
reproducible result. The only difference is if we do like a lot of coding agentic task then I prefer to use the parameters that are suggested for coding because that makes sense. Yeah, the quantizations for quant.8 and the one I run before are not the
Speaker A
same. Why? because we don't we are not allowed to have the exact same version there for everything but if whatever it was possible we use it. So for example for the drafter we have Q4KM target model also Q4 KM and so on. The
Speaker A
MTP model is Q8 GGF. Why? Because in drafter one in D flash we use also this even though uh there was Q4KM because at that point there was not this exact version of the drafter and uh usually you would not want drafter in lower
Speaker A
version because you know as accurate it is you want it to be as accurate basically to get as many draft tokens so your target model will accept it. But yeah, that's a small degression. Now what benchmarks we will run? So the
Speaker A
first one was live codebench because it's competitive programming problems. All of them are like in the coding space but they are also like separate problems. So we will not have a context a lot. So this is like if you will ask a
Speaker A
lot of questions of different topics basically and it's blind to but there is issue with it. Yeah because like if you work on some code you do multi-turn coding so this will not be good measure for it. So I make multi-coding benchmark
Speaker A
myself and basically it create gradio app that wants to kind of imitate the LACP front end. You can check it in the repo if you want. There are all you can run it yourself and everything like always everything will be in GitHub.
Speaker A
But basically I wanted to like simulate specific multi-coding behavior because that would be most closest to like coding on one repo or aentic coding and this is what these methods benefits also most nearly okay maybe for math it's a
Speaker A
bit more but yeah you understand the topic then the synthetic benchmark just to trust it for shape because you can't control the exact output input on each of these benchmarks basically that were before. So this is just to check and
Speaker A
then one request to send every sweep is just to write a fresh pros. So write a very long story for the model to basically generate a lot of code. So this is the opposite from codes. Yeah, we want to invent fiction and so on and
Speaker A
it we use it to check the accepted p accepted percentage in this report. So this is example what the synthetic harness will actually send and there is also issue sometimes that happens here that inflates the benchmark that basically it use
Speaker A
fictional stories but if they end the package goes into kind of ber mode and just multiply the sentence multiple times. that's happened but the best this is the best we have also to keep the numbers honest I was monitoring the GPU
Speaker A
temperature if it's not thermal throttling and so on we do one server at a time nothing was running on the PC at this point we have fresh container per each test we cool the GPU between measurements because this also influence
Speaker A
everything and we try to measure the noise basically to not trust like like the scores. Exactly. So whenever it was possible we test it or we run two or three test of the similar runs to check the spread basically. But yeah, I I
Speaker A
needed to be uh how you say selective about it because otherwise it will take weeks. And I measure it so can see that the heat is real but it wasn't as harmless. Now the first thing we look is
Speaker A
the life code bench. So I use 100 problems. We replay it in the same order. each allowed to stop naturally because yeah it decide how many tokens it will basically output.
Speaker A
So the number is what matters. We can see all the parameters and now let's look. So the baseline coin 3.8 without speculation 67 tokens. We can see inter token latency the wall clock and the output the output will be a bit
Speaker A
different here because it depends on the model and so on. Uh then the deflash so as we see it's 2 * 26 times more and the interesting part so this is all the quen model because when the flash one was
Speaker A
there was no quen uh 3.8 eight basically and we use the best number of heads and it still is slower. We see it's just two times more. So this is uh basically proving that D flash 2 is uh upgrade.
Speaker A
Yeah. And then I also use engram. So engram it's a method that I explained in my other video that you can check which is other speculative method and it's the interesting part is that if you use lamasc you can also turn on these
Speaker A
methods and then whenever possible or efficient it will be used. So we can see that the speed line against the model was there and also and then actually we do a bit of twist. So I explain all the benchmarks we run. So I
Speaker A
run them with just D flash 2 and then I appended engram mode engram map K4V because there are different methods.
Speaker A
So to do same weight same GPUs and just tune these things and now we can look here. So we have first the blue is the flash alone and then I don't know what is this color is uh both
Speaker A
drafters because if you use both then it will use the one that is better basically. So we get 182 tokens on editing code with a model. So there is 18 turn session but for this I only used turn from one to 9. Why? Because after
Speaker A
nine turn I asked it to like check the full database and so on which inflate the score. So I wanted to see it in the not best case but the worst case basically. And as we see we get 35%
Speaker A
increase in tokens per seconds after adding the lookup drafters. Then we have some forced synthetic just who runs to see and actually we get 52%. So this I will not trust honestly because this scientific gener generators typically do
Speaker A
something um dump like multiplying the same sentences and so on and then it don't work. And that's probably what happens because we can see that in life codebench which is the real benchmark of multiple runs it's only 1%.
Speaker A
And then when we do writing a fresh story we get minus 30 which is interesting because technically engram don't use that much resources but because we were doing something novel then I guess that the drafters are not benefiting so much from it and then
Speaker A
because we have these two modes maybe they were being used when the D flash would be more beneficial so on because there's algorithm to choose it but yeah it's new thing it's PR so it could not work now the interesting part is that
Speaker A
Dlash is hard captain send tokens and as I said and D flash one is at 15 so comparing them at their own maximum measures the cap not the drafter so D flash one was rerun at seven on the same
Speaker A
binary and both were put through the same 100 real prompts again and as we can see here then the D flash to win which is good.
Speaker A
So it's 1.14% ahead and as we see the accepted tokens are much higher. So this is something we would expect and the also interesting part is that the VM cost is much lower than D flash one but remember I use the
Speaker A
high quant there also. Yeah and I think it's mentioned here. Yeah it's mentioned here. So yeah this is something interesting that I put in my other benchmarks but this is like perfect case. So it show us good numbers
Speaker A
I explained before because the synthetic drafters like to break. So I run multiple runs at different context size just to see if how the drafters will behave as the context grows.
Speaker A
And here we have it. So we can start from the base. So no speculation as we can see at maximum context pretty straight line. The median decode in seconds is 59.
Speaker A
Then we have MTP we increase to 150. Then the blue is the flash two with N7.
Speaker A
Um so this is maximum. So this is actually hard to it's uh ah yeah so it's 211 and then D flash one with S7 is actually more which is 219 but these scores are like yeah perfect example which will probably not happen. So trust
Speaker A
more the argentic scores of multi-turn coding constantly. And then we have the engram only. And as you see we have a big boost with engram which is basically free which means that the synthetic benchmarks are still not working even though I updated package
Speaker A
probably and so on. And this we have then the flash one with 15. So it was the max but if we use D flash 2 with engram we go over to 500 tokens and this context. Yeah. And we can see these
Speaker A
scores here. So with D flash 2 with look up of engrams we get 8x increase on these contexts which will probably increase because the synthetic benchmark can generate as many tokens that are unique in a sense and this is mentioned here also yeah in
Speaker A
the report and there will be more details probably in readme if you're curious. So then we would like to find the best amount of draft tokens.
Speaker A
Technically yeah the maximum is seven. We can take it but there is a bit of cost always involved in it. So what I do is I cut the life code bench problems to exact number of tokens and then a task
Speaker A
instructions appended it and we do 16 prompts per whiff and we run the flash 2 and MTP with it just for comparison. And as we can see we achieve peak for MTP around four which actually three four which this correlates with typical
Speaker A
setting to all SG langm cookbooks that you can look for big models. Yeah if you look for high through output. So this is quite okay.
Speaker A
And here we get five which was also mentioned um by the authors of the paper. They suggest sometimes seven or five because yeah it depends as always on what you do. Uh but if I would start I would start round five. Yeah. And we
Speaker A
can see also there is point when it goes quite down. Yeah. Because this is all because of the context and so on. In my last test there was common complaint about speculative decoding and that it fall apart at log context. Yeah. And
Speaker A
there are a few costs related to it. Yeah, because there's prefill so everything that will fit will fit before we see the token and then decode. So the output. So what I do is the live codebench problems were concatenated and
Speaker A
then tranked to exact token count. So we know that the this is exactly 16,000 tokens of coding not something else. And then we set up it to different context lengths to basically check it.
Speaker A
And we can see here where was the point that we start getting the decrease. So this is the log scale of prong length and share of baseline prefill speed that we keep. Yeah. So look up. Yeah. the typically the cost is minimal but
Speaker A
actually we notice something yeah I think this is at one so this is the same and yeah I was surprised uh and as we see MTP and D flash then also with context actually it's getting there so I
Speaker A
make also this table which basically tell us about prompt depth the prefill token per seconds with note after with D flash then the speed that we kept compared to base and then the prefill time with non drafters and then prefill time with D flash and
Speaker A
as we see it takes more with D flash. Yeah. So we then have the extra time but then this extra time could be basically compensated because of decode that takes much more. So then if we calculate it then to break even we just need to get
Speaker A
output length of this amount of tokens. So basically it looks like it mitigate each other. But this is not the best test you could probably do, but this was the best I came up with the time to basically run it. So, as we can see, we
Speaker A
have the drafter and the tax that we need to pay at different context lengths and so on. And as we see the lookup, so the engram is cheapest. This is probably some rounding error at this point. So then basically I run again my agent
Speaker A
encoding benchmark and you can see here we are like setting sliders server health probe and so on and then from turn 10 we will basically ask it to remit file it already wrote then do like full code investigation adding dog
Speaker A
strings poke reports and so on which basically involved feeding into memory all of uh code that we already generated and I run it then on all 18 turns which inflated and as we can see yeah they inflated actually here not that much
Speaker A
uh which which makes sense because the engram give the biggest boost. Yeah. When we feed the full uh context like a code base that was before. Yeah. So this is like quite clearly visible here.
Speaker A
That's how I designed this test. So basically we can see that the engram will allow you for example if you rebase full coding database or something that you have. It's worth to turn it on because it will be faster. Actually the
Speaker A
models cinemas CP turn so fast that you could uh create FL or something and just rerun it or script and so on which could be interesting and then we can see that the lookup only also give quite a boost. So yeah but you can use
Speaker A
all of them together and then you get basically this which is like 344 tokens per second and here it's all of this in table. So as we see six times more the acceptance is oillating because this I actually don't check. Um, I think I
Speaker A
check it on the official version, but I don't know if this new one how it works because the acceptance length could be only for D flash 2 or for all speculative methods, but it's kind of close and it
Speaker A
also is multi-turn. So, there's many things that could go wrong there. So, thisation is kind of okay. Yeah, the maybe good thing to mention is that you have many parameters that you can specify here, but I use defaults because
Speaker A
it's like making grid search of them and benchmarks will take probably month. So there is one parameter that I wanted to mention which is uh draft perm. So this parameter stop drafter add block early when it's unsure to save some resources
Speaker A
basically. Yeah, of course in autogressive it will make uh probably more savings but I wanted to test it because I saw that like even in the VLM sg lang um in this part there is specific parameter that for example
Speaker A
deepc use. So basically when you have high to output high concurrency requests you want to modulate the number of draft tokens. Yeah because it still use utilization of server. So for example, if you have not many requests at high
Speaker A
concurrency, at some point you can allow more draft heads because you have more resources to spare. But if you will keep the same number of the heads when you will start getting four times more requests then this drafter will be
Speaker A
actually not helping you but making all the requests thrower for the users. So what they have is they have thing that basically measure all the GPUization resources and decide which number you should get. But yeah, I think it was
Speaker A
just this we added to VLM. So yeah, I wanted to check if it works here and there are some issues basically and we do some control runs and so on and it seems like for D flash the change is not
Speaker A
even there. It could be working but we could not see it because the server did basically identical work for us. And on MTP the flag works which could explain because yeah it's longer. So probably I don't depending on the time you watch
Speaker A
this video you can check this parameter because it could save you some resources or maybe there will be already algorithm that will tune your number of draft tokens where you're best based on your resources and I do quick tests with the draft if
Speaker A
it gets shorter and if it survives better but the true output was like yeah I don't see big change but This will be hard to observe because I have like basically too big GPU to observe the big saving here. I will need to run
Speaker A
like I said high concurrency probably workload to see it better. So yeah, there are some limitations. Yeah. So I want to be clear about it. There is no deep context decode measurements anywhere in this study. Why? Because I
Speaker A
don't have time. And the multi-turn harness can of course measure wrong things because if you use you can use your repetitive things there. Yeah. So it's always best to check. Uh and yeah for good measurements you should rerun
Speaker A
everything and so on but it's not always possible. I rerun it few times. It's sometimes mentioned sometimes not. But basically you should run the benchmark again the next day to prove all the scores if there is no errors and so on.
Speaker A
And the no quality measures were taken because we tested it till other videos and it always match. I think at the beginning I do like smoke test with um life codebench to basically see if it's passing the problems. It was passing
Speaker A
them. So yeah I don't go further just didn't have time. So this video is already long but if you want to run it there will be repository down below docker compose file you can run all tests you can sync the file and so on
Speaker A
and in last video I have issues with audio so this time I wish it's better because I get new microphone so if you like this video and you would like to hear more about this kind of stuff then
Speaker A
like and subscribe I have also all the speculative me decoding methods explained some videos about explaining the quantization in detail and other stuff. So also let me know what you would like to see below and let me know
Speaker A
which engine you actually run because I'm curious because I see more people using VLM sg because of the new features even on single concurrency and I use them for high concurrency. So we could do some runs with them. So that would be
Speaker A
it. Nice to see you guys and see you in the next one. Bye.
Topics:Qwen 3.8 27BDFlash 2llama.cppspeculative decodingAI inferencetoken acceptancelocal coding modelsdynamic convolutionMarkov headbenchmarking
