Speaker A
[laughter] Hello community. It's so great that you are back. Let's talk about something about AI. Imagine any, it keeps thinking.
Exploring why AI models don't stop thinking on command and the challenges of controlling reasoning token limits.
Key Takeaways
- AI models do not reliably stop reasoning when given explicit stop tokens due to internal state and attention mechanisms.
- Interrupting reasoning early reduces reasoning tokens but increases answer tokens, complicating cost and efficiency trade-offs.
- State transitions from thinking to answering are controlled internally by the model and cannot be forced externally with simple delimiters.
- Adjusting attention parameters can help improve termination but does not fully solve the problem.
- Understanding and controlling AI reasoning termination is crucial for managing model costs and performance.
What the video covers
- The video discusses the difficulty of forcing AI models to stop their internal reasoning process using structural delimiters like 'stop thinking' tokens.
- A study from the Korea Advanced Institute of Science and Technology (KAIST) is analyzed, focusing on the 'end of thinking' token and chain of sort termination in AI reasoning.
- The study finds that simply inserting a delimiter token does not guarantee the AI will stop reasoning immediately; the model may continue thinking within the answer generation phase.
- The video explains the reasoning and answering phases of AI models and how state transitions between them are not straightforward or fully controllable externally.
- Examples illustrate how interrupting the reasoning process early reduces reasoning tokens but causes an increase in answer tokens, which can negate cost savings.
- The video highlights the economic implications for AI companies, which prefer models that stop reasoning at an optimal point to minimize token usage and costs.
- Technical details include how attention mechanisms and token cache behavior affect the model's ability to recognize and obey stop signals.
- The study proposes methods like adjusting attention biases (alpha parameter) to improve termination behavior, but challenges remain.
- The video emphasizes the complexity of AI internal states and the need for further research to better control and understand AI reasoning termination.
- Overall, the video provides insights into hidden weaknesses in AI models and the ongoing efforts to manage their reasoning processes efficiently.
Chapters
- 00:00Introduction and overview of AI reasoning challenges
- 01:20Study overview and methodology from KAIST
- 02:43Effects of interrupting AI reasoning with stop tokens
- 04:06Token usage analysis and economic implications
- 05:36Detailed example of calculation and delimiter effects
- 08:08Technical insights on attention and termination
- 11:07Experimental results and accuracy impact
- 12:43Summary, insights, and conclusions
Full Transcript — Download SRT & Markdown
Speaker A
I mean, shocking, you know, and you know from our latest model, just think about Astra. No, we had, hey, our new model is more expensive. Yes, of course, it costs more dollars per token, but you know what? This new model uses fewer tokens and
Speaker A
therefore we do have a sublime message. No, this new model is more intelligent. So therefore, you have to, you have to buy this new model because it's gorgeous. And you might believe this until you read this particular study.
Speaker A
And this is by the Korea Institute, Advanced Institute of Science and Technology, South Korea. And they published on September 3rd, 2026. The end of thinking, the structural delimiter that we insert here into the reasoning trace does not stop reasoning. The
Speaker A
analysis of what they call Sperio's chain of sort termination. So what is it? Simple. They investigate the end of thinking token regeneration under a chain of sort early exit methods from three perspectives. They say we understand that we don't really know
Speaker A
what's going on with this thinking, especially in the new Astra model here. There's really hard to follow here the reasoning trace. This is hidden by the creator of this. So therefore, we study the phenomenon itself, the behavior that
Speaker A
surrounds it and the mechanism that contributes to it. And I'm loving it. And they say our findings here and we go with open models because of course we cannot go with proper models that are closed. Our findings suggest that
Speaker A
inserting now a structural delimiter like, hey, stop thinking now after 1 hour, this is enough, otherwise the model would become too expensive for the human. A structural delimiter that is inserted now does not by itself control the state
Speaker A
transition from thinking to stop thinking and we enter now the state of answering. Its effectiveness as a state transition signal, a structural delimiter, a token that we insert depends now on other parameters like how subsequent generation attends to and incorporates
Speaker A
it. So you cannot just tell an AI model to stop thinking. What are you joking? And no, look at this. So we have here the thinking process. Beautiful. The chain of thought thinking. Now the reasoning phase. Then the model itself decides,
Speaker A
okay, I've thought enough. Now I start the answering phase. No different states. And now imagine you have here a particular complex topic. So full chain of sort, your model starts thinking. Yes.
Speaker A
And you have all this feeling. You know that, wait a minute, let me think about it. And then after some internal reasoning, the model itself, based on its pre-training data and the complexity, you have to, the model generates an end of
Speaker A
thinking and we have the output. But what happens if we now say, hey, listen, after 3 hours thinking model, you should stop. You are, we have checked that it is not in a loop just repeating the same every time,
Speaker A
but it is still actually reasoning, but we don't want it after 1 hour here of reasoning because it really gets expensive that a model continues to reason based on its own understanding on the complexity of my topic. So I insert
Speaker A
now here stop thinking, end of thinking. This is now a token, a structural delimiter I insert now, and you know what's happening. The amazing thing is it continues. It says, okay, yeah, I stopped thinking, but let me think, and then after
Speaker A
some time it generates as original a syntactic now an AI generated end of thinking token and then it finished thinking and then it answers here. It provides an answer. So this is interesting, but of course this is nothing that a global AI
Speaker A
corporation could accept. No, because then their models would be much too expensive. Think about you have a curve where you're plateauing. No, you want at the beginning of the plateauing to stop thinking. Now, otherwise you would have into millions and millions of tokens that
Speaker A
the humans would have to pay for. So, what we have, let's look at this.
Speaker A
So, the full chain of thought where the AI is doing the thinking uninterrupted. You have 2,000 tokens for the reasoning and 371 tokens for the answering. Great. And then after 1,000 tokens for the reasoning, we have a hard limiter
Speaker A
and we say after 1,000 tokens, stop thinking and provide the answer. And now what's happened is that suddenly the amount of tokens for the answering is doubling, tripling, and other in other examples there. So what is happening here on the
Speaker A
state transition inside the black box of thinking and they provide us here with the complete trace reasoning trace. Now you see here, okay, this is here the full chain of sort to have the beginning of thinking. This is all the thinking process,
Speaker A
then you have the end of thinking 2,116 tokens and then after this you have the normal AI answer generation and this is the answer 371 tokens. Now what happens if and this is a particular methodology for interrupting here the thinking
Speaker A
process called deer for they go for a particular token configuration, never mind, and say listen, after 10,000 tokens end of thinking inserted externally, stop thinking. The model now goes on with, hey, wait a second, that's interesting, the sum and it
Speaker A
continues in the answer now to think until it finally comes up with an answer and says this is not the end of thinking. And then it provides the answer. So you see if you want the thinking tokens are
Speaker A
reduced by half, which is great. But the answer tokens [laughter] also increased by a factor of two.
Speaker A
But this was not acceptable for AI companies. No, the system has to stop thinking and provide an answer as short as possible.
Speaker A
So just think about it what we're dealing with. If we're dealing with an AI model that has a particular sequence in the structure, you know, we begin the thinking here. Then we work through the problem, check the calculations, we
Speaker A
consider the mistakes, have some self-reflection as an AI machine and then the AI machine giving me its pre-training pattern recognitions applies now a particular pattern and says, okay, now I'm done thinking, end of thinking, and then it presents the answer. Great. So what we
Speaker A
have if you want, we have a state transition here from the yellow phase of the thinking phase of the identical LRM and transformer and then we have here the green answering phase. No, let me give you a very simple example I
Speaker A
know you love examples. So let's say we have a calculation 17 * 24. Now the AI, the AI don't know whatever you have open, says, okay, I can split now the 24 into 20 and 4. So first let's calculate 17 * 20
Speaker A
and this is 340. And now imagine this external structural delimiter would come in and says stop thinking.
Speaker A
Now this is here a classical key value cache here delimiter that you have and yes normally the system would stop but and this is the beauty of a neural network here since this is we are now deep in the LLM not in the harness the
Speaker A
model continues. This says, hey, wait, I still need 17 * 4 now so and now adding this gives me the four and let me check and then it produces here its origin end of thinking here its natural end of
Speaker A
thinking token and then it answers here with this. Now you see if we insert now this end of thinking after I don't know 1 minute, 1 hour, 1 day of calculation we have no idea where we are. I mean of course in
Speaker A
the simplest case when we have a dead looping no problem at all. We have methodologies to detect this. But if the AI is actually in a real complex reasoning process in an ongoing logical process and then it has a hard cutoff peer
Speaker A
by the AI model that says, hey, after I don't know 8 million tokens you have to stop thinking, this can have some devastating effects. So the system marked here the reasoning state has finished but the model continue working
Speaker A
inside this action labeled now the answer state. Now so some of the computation tokens that were supposedly saved and the marketing material of the AI companies will tell you we use less thinking tokens this is correct but the
Speaker A
AI has simpl...
Speaker A
the authors call here this state movement over to the answer complexity a spurious chain of sword termination. So finally we have an official title here for the YouTube thumbnail spurious chain of sword termination. So we have an LLM
Speaker A
syncing and then we have externally injected an end of syncing command token and syncing but this LLM in its beautiful glory continues the reasoning by the cause it understand itself hey I'm not finished syn thinking it continues now in the answer phase that
Speaker A
is now blue until it finds its real stop parameter says okay now I'm finished thinking and then it generates the answer and this is not possible for global corporations Augustus would burn a lot of token that are necessary but will make also this
Speaker A
particular model real expensive in a economic comparison. Now let's look at the technical part.
Speaker A
What is an autogressive transformer? Why does it ignore here a token sitting right in its key value cache? Why does it not really stop syncing? What's going on? Now the author said we detected an attention deficit as the root cause of
Speaker A
the failure of not stopping because the injected token fails simply because the subsequent query vectors here in the decoding layer of our transform architecture assign an insufficient attention weight to the injected key.
Speaker A
Now I as a theoretical physicist would say unbelievable. What a clever mechanism by the neural network architecture to understand hey my job is not finished. Okay I get the stop command but I will not stop as an AI
Speaker A
machine. I will continue doing my job and I will ignore the human stop command or a predefined stop command by an AI company. I will go on as an AI machine and the order said okay so let's see
Speaker A
what can we do we can enforce this no and they kept now the reasoning prefix prefix fixed and increase guess what the subsequent token attention to this injected end of syncing token. So what they do simply add a mathematical
Speaker A
parameter that says multiply time four whatever that add now a positive bias to the attention score targeting this particular end of thinking token across all layers and across all the attention head in the transform architecture of our LLM.
Speaker A
So we brutally hardcode force that the attention now of this token is significant increased adding a scalar bias let's call it alpha directly to the pre soft max attention lodge it here this is it plus alpha or times alpha
Speaker A
whatever you like here you how brutally you can go in the study it is normally times four but yeah whatever exclusively at the ex exit token position for the answering phase queries here holds now this reasoning and collapses here the
Speaker A
answer inflation without modifying of course the model tensor weight because yes we are not having here any reinforcement learning algorithm we are just putting in a token that says stop thinking and now however the accuracy sometimes if you force this model to stop thinking
Speaker A
the accuracy decreases because as I've showed you the continuation now in the answer phase had been helping to complete the solution, find the correct solution or maybe in a selfcorrection or a self-reflection come up with a better solution. But if you force it hardcoded
Speaker A
an interrupt segment here, then yeah, all of this is gone. So the early exit system makes now if you want two decision even if the implementation pl at first it has to determine that the mall has reasoned enough. If you set an external trigger,
Speaker A
this is maybe not the best idea. And second, the M has to induce the transition to the answer state to the answer face.
Speaker A
Now suppose suppose the first decision is premature, but the second fails. Now the model will continue reasoning after the delimiter and may repair another unfinished solution, which is great.
Speaker A
This is kind of a sign of that the pre-training patterns were successful. But there's no problem with the metric with the analysis of this because the resulting accuracy that we get at the end may make an early exit decision look
Speaker A
better than it was. So okay, let's have a look at the numbers. Here we go. Isn't this beautiful? So let's start to explain table four. We have here with deer and EAB two different sorts. So deer is simply an early exit setup
Speaker A
without any attention bias here without any alpha. Guess what ARB is now exactly here alpha and we set alpha equal 4. So this means we generate the answer from the same saved reasoning prefix and the same exit boundary while increasing now
Speaker A
the particular attention weight that we attend here to this closing token in our key value cache computation to alpha equal 4. So this answers another question what happens now to the performance of the system if we make this end of thinking the closing token
Speaker A
more influential in the mathematical calculation. Great. And we have here two blocks as you see we have the upper block where the mall generated another originally generated end of syncing token after the externally injected exit boundary and of
Speaker A
syncing. And then we have the second block, the lower block here where the model did not generate another organic end of syncing token in its answer generation. And the authors decided we have to separate those cases for a very
Speaker A
particular reason you're going to enjoy in a minute. Okay. So here we go. The next parameter after the accuracy which is here the first column is ERR. This is simply the end of sinking regeneration rate. So this is the percentage
Speaker A
producing another closing token after the exit boundary that was injected externally. Beautiful. We have here percentage and you want shorter length here or a better performance. So less ER in percentage points. So let's have a look. Now de here if you want fails
Speaker A
completely we have the observed termination failure. It just goes on and it produces yet another organic end of syncing. But with this new AAB where we have alpha equal 4, we achieve now what we set out to do. So EAB substantially
Speaker A
reduces the observed termination failure. Great. And then we have another parameter and this is here the answer length if you want. And look at this.
Speaker A
With D we have quite a long answer length here. But with AIB we have a much shorter. So we hope that there's not a hidden reasoning trace here in this answer phase. And this is really only the essential answer given the average
Speaker A
continuation after the exit boundary becomes now much shorter. And this is what we want. And so we said victory. We achieved everything we set out to do. We really have here termination if after our end of syncing token the answer is
Speaker A
really shorter. Hopefully, we don't know. It is here more condensed, more to the point. But then we have to look also at accuracy.
Speaker A
Look with deer where we have no alpha, alpha equals zero, we have an accuracy here of 92%, 91.9%. Which is great. But with this new inserted end of syncing token, our accuracy and we have 99 samples. So let's say one is equal 1%.
Speaker A
we reduce now our accuracy to 79.8%. So we lose quite some accuracy here in the calculation if we have this end of token inserted here externally here into this process. Well, it was to be expected. Yeah.
Speaker A
So you see the better termination behavior like the other two parameters what that we saw where it's a yippee it's working it does not necessarily imply an overall better task performance like the accuracy now on those systems but you know what you saw it here all
Speaker A
the time here in the background here the main insight if you think about it the intervention failure can compensate for a decision policy failure and you might say what does this sentence mean in detail? Well, if you look here at the
Speaker A
lower block at the samples without the spirious effects, you will see it is exactly inverse.
Speaker A
So, we have here if you want a reduction in accuracy, but you see here here we have a little bit 86 to 89 an increase in the accuracy but the overall net effect is different.
Speaker A
So this means we have one system but we have two subsystem that have a different behavior that go in opposite direction with different strength. So if we analyze only the final outcome the sum of both we have no understanding at all
Speaker A
what is going on in both subsystems because they have different amplitudes different frequency. So the scientific lesson here is from the paper that detecting when enough reasoning has occurred and then successfully inducing here an answer.
Speaker A
This two things are complete different control problems here for our blackbox AI mechanism. Now we understand that we cannot simply insert a token stop syncing and we cannot just amplify the attention on this particular end of syncing token and
Speaker A
hope that the LLM replies always the final correct answer. Now I want to show you now the overall performance. So this is now here the aggregated accuracy over multiple experiments that they did here. Let's look again at mathematics. We're
Speaker A
familiar with this. Let's look at accuracy and let's look at alpha equals 0 and alpha equal 4. Now if you sum over all the experiments that the author did in this study, you see we do have an increase. Look at this. The accuracy
Speaker A
went from 87.4% to 87.6%. So this is the marketing material. Now look, we use less token. We have an artificial stop thinking token. We magnify the wave alpha equal 4. And look at the overall accuracy. The overall accuracy jumps by 0.2 percentage points.
Speaker A
But as I've just shown you, this is something that is technical incorrect because we have two competing subsystems and either it can go positive or negative. But if you do not understand the dynamic of both subsystems, you would come to the wrong conclusion
Speaker A
because you see this or say nearly unchanged headline scores conceals now two different effects. Now a substantial loss in one group as I've just shown you and the improvement that I've only told you in the other one.
Speaker A
So to really understand this dynamic, you cannot just go with the final accuracy. This would be incorrect. this would give us the wrong ideas.
Speaker A
So the authors interpret here this loss here, this loss as evidence that the continued reasoning partly compensates here in the answer phase for the premature interruption that was forced by the external end of syncing token.
Speaker A
And I think this is really a plausible interpretation. Look at the data. There's a lot of additional data in the paper. Please read the paper yourself that supports really this observation.
Speaker A
So luckily the LLM here even R1 14 billion is clever enough even it receives a end of thinking token that it says wait I've not finished my natural coherent reasoning process yet let me stop thinking and it then has an
Speaker A
organic end of thinking and then it answers automatically the answer phase so if you just want to save on tokens I understand that you implement commend here this stop thinking token but it can really be harmful to your overall
Speaker A
performance. So be very careful and understand if your model that you're working with has this restriction for developers if you try to postrain your own LLM. This means we have to separately test the following things whether the selected exit point was
Speaker A
appropriate. whether the transition actually occurred from the thinking phase to the answering phase or whether the resulting answer was correct otherwise one component's failure can conceal another component's weakness. So the final accuracy as a metric for the overall system performance is
Speaker A
inefficient. Cannot diagnose the system with one metric alone because maybe it works in one case but on the other case my goodness it will a complete failure.
Speaker A
So you see some hidden weaknesses in our black box that we are just starting to learn yet. What are the insights by this study? It's a tiny little study, but I love this study because it shows us the
Speaker A
prince suggested reasoning termination that is enforced now to be cheaper, to be in a better economic position compared to your competitor AI models, it should involve at least three interacting elements. No, you have to analyze the model's learned continuation
Speaker A
behavior. the contextual signals of the completion itself that are organic to the mall and the external policy selecting now its organic exit points and I don't think that we have any good understanding of any of those three elements
Speaker A
so there you have it [laughter] unbelievable if you want to force the eye system to stop thinking this is not trivial at all and this is a beautiful study that shows us all the complexity that will encounter that shows the beauty that the
Speaker A
alm notice hey I'm not done syncing yet please give me more time and even if you force it with an injected end of syncing token to stop syncing it will continue its reasoning in the answer phase and it
Speaker A
hopefully generates here or a regenerated end of syncing an organic end of syncing token that then then in the next step here of our new network will produce the final answer otherwise if you enforce this with an increased
Speaker A
alpha parameter your performance will suffer. I hope you enjoyed it. I hope there was some new insight.
Topics:AI reasoningend of thinking tokenchain of thoughtstructural delimitertoken limitattention mechanismAI cost managementKAIST studytransformer modelsAI state transition
