Can 3 "Soft Tokens" Replace 1,000 Reasoning Tokens in AI?

Explores SHIFT, a novel AI method using soft tokens to replace extensive reasoning tokens, enhancing efficiency and mathematical smoothness.

Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.

Generated from the transcript and can be wrong — check the timestamp.

Key Takeaways

Soft tokens enable a continuous, differentiable representation crucial for effective AI training.
SHIFT method significantly reduces token count needed for reasoning without losing information.
Maintaining all vocabulary tokens in softmax preserves smooth mathematical manifolds for backpropagation.
Mathematical superposition of tokens allows richer, more fluid concept representation inside AI models.
Contrastive loss and embedding space alignment are key to SHIFT's retrieval and reasoning performance.

What the video covers

Introduction of SHIFT: Self-Construction Harness with Implicit Fine-Grain Sinking for Retrieval, combining RAG and long sinking concepts.
Explanation of transformer architecture focusing on token embedding, positional encoding, and multi-head attention.
Definition and geometric interpretation of soft tokens as weighted averages of entire vocabulary vectors.
Discussion on softmax function's role and why cutting off low probability tokens harms differentiability and training smoothness.
Soft tokens represent fluid superpositions of concepts, maintaining continuous manifolds essential for backpropagation.
SHIFT leverages soft tokens to create a mathematical superposition of reasoning chains rather than discrete hard tokens.
Comparison of reasoning complexity: 3 soft tokens can replace nearly 1,000 traditional reasoning tokens.
Mathematical spaces and transition between them are introduced for advanced reasoning and retrieval.
Use of contrastive loss to optimize query-document alignment in the embedding space.
Demonstration of how SHIFT achieves efficient reasoning with fewer tokens while preserving complexity and accuracy.

Chapters

What is a soft token in the context of AI?

A soft token is a vector representation created by calculating a weighted average over the entire vocabulary, representing a fluid superposition of concepts rather than a single discrete token.

Why does SHIFT use all vocabulary tokens instead of cutting off low probability ones?

Keeping all tokens ensures the mathematical manifold remains smooth and differentiable, which is essential for effective backpropagation and training stability.

How does SHIFT improve reasoning efficiency in AI models?

SHIFT replaces thousands of traditional hard tokens with just a few soft tokens that encode complex reasoning in a continuous vector space, reducing computational cost while preserving reasoning quality.

Full Transcript — Download SRT & Markdown

Speaker A

Hello community. So great that you are back. Let's talk about a brand new scientific idea called SHIFT. SHIFT stands for Self-Construction Harness with Implicit Fine-Grain Sinking for Retrieval. And you might say, what? We combine something from RAG with a long sinking, and we have a self-construction of an AI shell that we call an AI harness, like a cloaked coat. And we bring all of this together into one paper.

Speaker A

sinking and we have a self construction of an EI shell that we call an EI harness like clawed coat. And we bring all of this together into one paper.

Speaker A

Absolutely. And if you ask, hey, where are we? Yeah, of course, we are in artificial intelligence. So, let's have some fun. And if you think AI is just here with Fable 5 or Opus 5 and some context engineering or some prompt engineering or some harness engineering or some blooper engineering, yes, this was here yesterday. This was the video yesterday. But today, today we want to go a step further. Today, we want to go to the leading and bleeding edge of AI.

Speaker A

engineering or some harness engineering or some blooper engineering, yes, this was here yesterday. This was the video yesterday. But today, today we want to go a step further. Today, we want to go to the leading and bleeding edge of AI.

Speaker A

So let's start. Now you remember if you look at the transformer architecture here, and this is the complete complexity that we need for our mathematical understanding of what is happening inside the black box of AI. We have the input tokens. No, the cat sat on the mat, beautiful. We have the embedding. We have the positional encoding. Then we have your transformer blocks, the multi-head attention. And we will talk about here internally what is happening here in the head. Because what we had in the past is that here for the logit, no, we then had a simple probability distribution where you saw, okay, CAT had 91%.

Speaker A

tokens. No the cat set on the mat beautiful. We have the embedding. We have the positional encoding. Then we have your transformer blocks the multi head attention. And we will talk about here internally what is happening here in the head. Because what we had in the

Speaker A

This 87% is not possible but just ignore it. Now this is here an AI-generated graph. So 91% CAT, and we know CAT was the result for the next token prediction. And you might say, great, and now we do something much more interesting. And now we go from a hard token to a soft token. So what is it exactly defined? A soft token is a single vector representation created now by calculating a weighted average of the entire dictionary that I use. For example, if I do theoretical physics, I have about 42,000 words that have here a token representation. If you go token equals word, but of course you understand I have a lot of technical terms in science and yeah, 42,000 is about fine for me.

Speaker A

This 87% is not possible but just ignore it. Now this is here an AI generated graph. So 91% CAT and we know CAD was the result for the next token prediction. And you might say great and now we do something much more

Speaker A

Now you know a neural network in a classical know we use here to generate the percentages our softmax function no but you know here and in this long running out of the softmax function they never really go to zero know in those trailing and if you ask I to complete the sentence hey the boat sailed across there and now you might say hm not the land maybe it's the ocean or maybe it's the water you know the softmax function actually calculates the percentage of every single word of my 42,000 words equal token. Great. But because those probability never hit true here when the R calculates the soft token, it is now forced to do a matrix multiplication against the entire vocabulary matrix. And you might say,

Speaker A

example, if I do theoretical physics, I have about 42,000 uh words that are have here a token representation. If you go token equal word, but of course you understand I have a lot of technical terms in science and yeah 42 is about fine for me.

Speaker A

okay, but what does it mean geometrically? So when I build here the next soft token vector, you can be sure that over 99% the shape is m to a vector that goes with ocean or sea or water. you know,

Speaker A

trailing and if you ask I to complete the sentence hey the boat sailed across there and now you might say hm not the land maybe it's the ocean or maybe it's the water you know the softmax function actually calculates the percentage of

Speaker A

give him our sentence. But it is also microscopically pulled by the vectors of banana microscope or every other word in theoretical physics here if you have a 42,000word dictionary.

Speaker A

okay, but what does it mean geometrically? So when I build here the next soft token vector, you can be sure that over 99% the shape is m to a vector that goes with ocean or sea or water. you know,

Speaker A

And of course, you might say, "Hey, yeah, let's make it easy." No, the computation reduced the cost. Why not just cut off the bottom 41.9K words to save time? And we do this. This is our softmax. But we lose something. And we

Speaker A

And of course, you might say, "Hey, yeah, let's make it easy." No, the computation reduced the cost. Why not just cut off the bottom 41.9K words to save time? And we do this. This is our softmax. But we lose something. And we

Speaker A

lose something today we cannot afford to lose. And this is here if we do some calculation the differentiability of our system. So you know if we have here the training algorithm we will train a system and we have the back propagation

Speaker A

everything. It is essential that everything is here on a manifold a smooth manifold and we have some continuous manifolds where we can do some differentiation for the back propagation and even if I have a word somewhere now to force it to be zero now

Speaker A

everything. It is essential that everything is here on a manifold a smooth manifold and we have some continuous manifolds where we can do some differentiation for the back propagation and even if I have a word somewhere now to force it to be zero now

Speaker A

contribute a billionth of a percent, the equation remains perfectly smooth. We can now do our training and our calculation. This means the soft token is just a blend of a few highly probable concepts, but mathematically speaking, it is a fluid superposition of the

Speaker A

hardcode it to be zero this can create a microscopic mathematical cliff where our ER calculation become chaotic and we don't want this today so by keeping now all 42,000 words in the weighted average, even the one that only

Speaker A

all 42,000 words in our dictionary this is extreme crazy and said why should I do this well let's have a look now if we go with that the soft token is now a mathematical superposition of our token and let's say token equal words or

Speaker A

contribute a billionth of a percent, the equation remains perfectly smooth. We can now do our training and our calculation. This means the soft token is just a blend of a few highly probable concepts, but mathematically speaking, it is a fluid superposition of the

Speaker A

reasoning chain. But now we stay inside the black box. And inside the black box we have a mathematical superposition of some fluid continuous concept and we do not rely here is this a human word or not? Who cares? It's an alien

Speaker A

entire dictionary at once. Do you feel what this means? Instead of having here from our probability distribution just the arc max here the top no like the cat here at the top of our gshian probability distribution now we keep the complete superposition of

Speaker A

mechanics and quantum field theory. Yes, superposition of words. So this is not a quantum object. Of course we have some ideas but more about this in a different video. So, so this means really be sure the resulting vector and think about

Speaker A

all 42,000 words in our dictionary this is extreme crazy and said why should I do this well let's have a look now if we go with that the soft token is now a mathematical superposition of our token and let's say token equal words or

Speaker A

It's complete some crazy vector representation. A continuous mathematical concept. It essentially means hey the result is mostly ocean but with some flavors of sea and water but also all the rest of the 42,000 elements. No.

Speaker A

of words this allows not to sync using fluid continuous concept instead of human words. So this is what we call a latent reasoning. We don't go and translate this into heart tokens that we have human English word in the

Speaker A

Piging University say oh wow okay let's go for it. Institute of computing technology Chinese academy of sciences and beyond university. So you say okay this is a highlevel paper published July 23 2026 and it is shift. So let's have a look at

Speaker A

reasoning chain. But now we stay inside the black box. And inside the black box we have a mathematical superposition of some fluid continuous concept and we do not rely here is this a human word or not? Who cares? It's an alien

Speaker A

of rack system and we will integrate this with some nonhuman thinking because come on why we should stick here to some human language. We are talking about artificial intelligence. So let's go.

Speaker A

mathematical trace. Full stop. We can calculate with this. We know the rules of mathematics and we will apply the rules of mathematics to this object. Who needs human language and I know superposition here [laughter] as a theoretical physicist you know quantum

Speaker A

documents here or retrieval information from a database we have some candidate documents now D1 to DN and now the retrieval process aims to learn a particular encoder function that projects now the textual inputs into an M dimensional embedding space. This is

Speaker A

mechanics and quantum field theory. Yes, superposition of words. So this is not a quantum object. Of course we have some ideas but more about this in a different video. So, so this means really be sure the resulting vector and think about

Speaker A

documents that we have in this vector representation and say okay boring old stuff. Absolutely. So to solve now the reasoning intensive queries and now we go for long reasoning for long jobs that have not one question and one answer but

Speaker A

this soft token is a vector representation in a particular mathematical space does not exist in a human dictionary. It's not a human word.

Speaker A

have a complete salad here of whatever we have to do. What we how we solve this? Well we have a rewrite then retrieve. So we have an AI system that says hey listen human whatever you give me here for a stupid prompt I have to

Speaker A

It's complete some crazy vector representation. A continuous mathematical concept. It essentially means hey the result is mostly ocean but with some flavors of sea and water but also all the rest of the 42,000 elements. No.

Speaker A

all the documents that have a high co similarity. Now show you this paper. Yeah. And they introduced an external alignment reasoner to first produce an explicit reasoning trace. Remember we in long complex reasoning. So we don't have an

Speaker A

And you might say, okay, so we went now from hard tokens to soft tokens. And why is this now the secret weapon of our new preprint of today? And here we have it.

Speaker A

So somehow we have to put this into code. So our reasoning trace we have to modify this. Now whenever you read a paper you know there was a paper before this and this is the paper before our actual paper today. This was published

Speaker A

Peking University say oh wow okay let's go for it. Institute of Computing Technology Chinese Academy of Sciences and Beyond University. So you say okay this is a high-level paper published July 23, 2026 and it is SHIFT. So let's have a look at

Speaker A

have some brilliant paper and the state key laboratory of cognitive intelligence and the Hong Kong Polytechnical University and they already talked about reason embedding enhancing the text embedding for reasoning intensive long complex reasoning chains. And we have a

Speaker A

it. So what we're talking about today is the implementation of an AI harness, some long reasoning traces. Of course we want to have the system self-learning. We are talking about retrieval optimization. So this means we are the latest generation

Speaker A

simple and I want that everybody understand it. So what do you have? We have an input query Q. Now this is my stupid question to the machine. We directly get the input embedding X. This is here simply E where E denotes the

Speaker A

of RAG system and we will integrate this with some nonhuman thinking because come on why should we stick here to some human language. We are talking about artificial intelligence. So let's go.

Speaker A

mathematical space and of course we are depending here on the vocabulary size let's go with 42k now unlike standard decoding which selects discrete vocabulary indexes this is now our latent tokens are represented as a continuous embedding and you know

Speaker A

Now at first we have to formulate our simple task. No. And this is easy. We have a user query Q. This is my task that I give to the machine and let's say since we are talking about retrieval

Speaker A

So now a tiny tiny little bit it's not even mathematics it's just some formulas no so the soft latent token remember that we're now talking about a latent token not about a human word token tj is defined as the expectation of the

Speaker A

documents here or retrieval information from a database we have some candidate documents now D1 to DN and now the retrieval process aims to learn a particular encoder function that projects now the textual inputs into an M dimensional embedding space. This is

Speaker A

embedding matrix no and then we get the soft latent token DJ and you see we can abbreviate this term here in the parenthesis simply with P.

Speaker A

the classical way we do here retrieval. Yeah. Because we put here all the information in a particular vector embedding space and then we just go with a cosine similarity and we compare our query to the mathematical presentation of all the documents or the set of

Speaker A

course to be one. This means that all vocabulary embeddings are points in the mall embedding space that we constructed. So E1 to EV 42,000 and this means that consequently the soft token now that we built lies in the

Speaker A

documents that we have in this vector representation and say okay boring old stuff. Absolutely. So to solve now the reasoning intensive queries and now we go for long reasoning for long jobs tha

Speaker A

scalar variable to a vector variable and then to a tenza variable. The expectation is the probability weighted average of all possible values X. So you see we have here our e our embedding here and this is simply here the sum

Speaker A

over all x and p * x and then you say wait a minute wait a minute and careful e with a parenthesis is not the embedding matrix it is the expectation operator. This is an operator algebra that we are dealing with mathematically

Speaker A

and E is the embedding matrix and there is if you think about it at least in [laughter] in my understanding of a transformer here this simple uh formula that is valid. So we do have an expectation operation like here probability weighted

Speaker A

average over all possible values and with the definition of PJ here transposed and we apply to the embedding matrix. Now you see exactly the operator notation and the pure embedding matrix connected over PJT.

Speaker A

So use here the probabilities in PJ as the weight. Take the corresponding vectors of the embedding matrix E and average those vector all possible solution and the result is the soft token TJ T subj sorry typo. Okay.

Speaker A

Now I ask you [laughter] JPT to show me an image of a convex hull of a vector representation. Okay, it's not really perfect but I think you get what I mean.

Speaker A

So we have here a probability our new probability vector P lies now in the vocabulary simplex. If you don't know what a simplex is mathematically, go to my channel, look for the search result input simplex or simplex manifold and

Speaker A

you will get here beautiful videos explaining everything. Now embedding matrix E not the operator just the matrix maps this highdimensional simplex into our EI models M dimensional embedding space as we just have saw. So because the coefficient now a little bit

Speaker A

of mathematics I just tell you you might believe it or not because the coefficients are non- negative and sum to one there is a theorem that says okay now our soft token lie in the convex hull of the vocabulary embeddings so we

Speaker A

are inside of this which makes sense no it's a fluid structure so we have a convex hull so you see we started here with such an easy representation of the internal working here of a blackbox transformer and our W's our language which modeling

Speaker A

hat here contributed here to our mapping to our transformation. So now we have to go and take here the step of here a long reasoning trace. How we do this? How we can code this? How we can implement it? What is the

Speaker A

mathematical theory? I'm so glad you asked. So let's say at a particular step J the mall the imol takes now the input of the original query embeddings. This is your my question now in this mathematical vector space together with

Speaker A

all previously generated latent tokens. So you have here the embedding matrix and then T1 to TJ step J and this is it.

Speaker A

We have our FTA ta here and this is here in our new hidden state and through causal attention you know have multi head self attention and cross attention everything each latent token TJ is able to attend now both to the query context

Speaker A

Q that we have here and earlier reasoning step T1 to TJ J minus one thereby resembling here the sequential dependency pattern of an explicit chain of sort reasoning trace. Yep. We achieved what we set out to do. But you

Speaker A

notice we have not gone with a hard um coded no with hard um vector representation, hard token elements, but we have gone with some soft tokens and some soft latent tokens because we wanted to have not a result of a word

Speaker A

but of a complete distribution probability distribution. So this is now if you want to end this and you understand. So we had a high complexity of human language and blah blah blah and after we put it through a

Speaker A

transformer what we get out in the next token prediction we got out this beautiful orange heart token we have one single word this is the next token prediction and now we understand well okay we miss out on all the other

Speaker A

elements in our probability distribution. So we do not go with a hard token. So only this orange element here in this image but we take now the soft token and the soft token would be this area here with all the other

Speaker A

probability distribution to be correct about all 42,000 uh words if we have a vocabulary with 42k.

Speaker A

If you go for social media plain English you can come down to 18k. Great.

Speaker A

And then I looked at figure one. So remember we are just starting to read to you this paper. And from looking at figure one I said hey wait a minute.

Speaker A

What do you mean you have two loss function? Because I'm a little bit lost here. Yeah as it turns out they talk about the retrieval space and a latent reasoning space.

Speaker A

And I thought oh wow so we are not working in the same mathematical space but now we build two different mathematical spaces. So this means we have transition between those spaces. So we have we have to build mathematical

Speaker A

bridges that cross over from one space to the other and we have some optimization for the retrieval. Of course this is like rack and then we have some optimization of a mathematical space for some latent reasoning of course since we're not working here in a

Speaker A

classical predict the next heart token element. and I said, "Oh, wow. This is now this is now a little bit challenged." So, let's just jump right into it. So, our soft tokens have two massive advantages.

Speaker A

I mean, I would say so. No, we have no lost sort since it is not just the final single word cat as a result for the next token prediction, but we have a probability distribution over all 42,000 tokens in the vocabulary. So, we have an

Speaker A

extreme information density. And hard tokens force the eye to prematurely collapse that fuzzy sword process having here all the information into a single boring specific word like cat destroying here all the other reasoning complexity or what it could

Speaker A

also have been a a valid solution. But now if we switch over from hard tokens to soft tokens this will allow the eye to hold something much more a higher complexity in its memory. And now we have a fluid ambitious sort process

Speaker A

as it continues to process now this search query. And you say oh yeah wait a minute we are back at retrieval. No because I'm just now uh having here an optimization of a mathematical representation of the sort process

Speaker A

itself. But now I have to convert this that we can search a database. What a coincidence that B here the second advantage is we have a differentiability. So this means to train now any on this particular function that we want and the behavior

Speaker A

that we want. Developers like you my beautiful viewers use now gradients and derivatives to track how much it stapes contributes to the final answer. But you cannot mathematically calculate the derivative of a hard token choice because picking now word is a sudden

Speaker A

sharp cliff in the mathematic and we have a non-ifferiable step. But guess what? With a soft token, this is just a smooth continuous weighted average. So now we can do it. So now we have this AI's invisible sorts in the latent

Speaker A

mathematical representation all the way back to the beginning. So this means that the reasoning process is not just one word and another word and another word, but now we have the complete thinking complexity.

Speaker A

Maybe it was a cat. Maybe it was a chicken. Maybe it was a human. Maybe it was an electron. Maybe we were talking about transportation theory. We have now a if you want priority top-down listed way of a fluid sort complexity not of a

Speaker A

single word. So geometrically as we know the soft token is a point inside the convex hollow of the vocabulary encodings. Or if you want to see it a little bit more from a critical physics point of view, it is a superposition of semantic

Speaker A

objects. But these semantic objects can either be word or maybe they are higher level abstraction of nonhuman words.

Speaker A

Superposition you remember no quantum field theory no quantum mechanics nothing we here in computer science and there's nothing like Heisenberg entre or any other like this. So this is just classical computer science.

Speaker A

This allows the eye to sync using fluid continuous concept instead of rigid human words keeping now our internal reasoning representation much richer much faster and mathematical trainable because we have differentiability after we cleared up here the introduction now we understand I mean I

Speaker A

understand what is the topic of this new paper let us now start now we can go and say listen let's start with a geometric and scientific analysis of the soft tokens and thinking processes, the retrieval oriented deformation that we

Speaker A

have to do to this mathematical space and then a fine grade self reconstruction of the reasoning complexities crossing here two different manifolds.

Speaker A

Now what a coincidence and as a subscriber of my channel which I hope you are you know 2 three days ago I had here this EI my very first EI generated images here and what a coincidence I was

Speaker A

talking here no the EI was talking 2 minutes about what is a mathematical manifold the geometry of manifolds here from Bardman mid 19th century and I show you here the reman manifold and locally flat ukidian space and the tangen space and

Speaker A

Interesting. So I introduced the main concept to you already days ago. Yes, I know. Absolutely brilliant. And now we can talk why the generation and the retrieval of new information need different manifolds because up until now we smashed it all together in

Speaker A

one. And now those researcher from our colleagues on China tell us you know what much more intelligent is to have optimized mathematical spaces or manifolds that are optimized at first for the generation of the next token prediction and an additional

Speaker A

mathematical new space that is optimized for the retrieval of long reasoning traces. And what a coincidence also an introduction to mathematical manifolds.

Speaker A

I have in this paper where I showed you here if we have eight slices and the context of each of those slices explained exactly what is a mathematical manifold. So here this is the second source. Yes, I know.

Speaker A

Now you know that the transformer hidden state H is a point in a very large vector space classical. But real M states probably do not occupy every possible point in this new mathematical space because this would make our

Speaker A

computation go to infinity. And now we hope we have the fingers crossed that all our data tend to lie in a structured lowdimensional subspace. And you know what we call this? It's a manifold. And then if this is in a region we call it a

Speaker A

representational manifold and we have no idea if this is correct but otherwise it would be not computable. So we are absolutely flying blind.

Speaker A

Now in this new paper they tell us hm it is maybe useful to imagine two task condition geometries different mathematical geometries. At first we have a generation oriented manifold that is shaped and oriented here by its task for the next token prediction. This is

Speaker A

the classical LLM. And then we have another manifold. Let's call it the retrieval drag orientated manifold shaped by the need to distinguish relevant documents from irrelevant one given here a particular long reasoning trace.

Speaker A

So now I want to introduce you for to my understanding. So I give it now here call it a solution or I call it now a bridge. A bridge is simply for me a mathematical object that helps me to go

Speaker A

from one place to another place if there's a river between those objects. No. So a bridge is a function or a learned mechanism now that connects two represent two mathematical representation that cannot be used together directly. Now we have a river

Speaker A

separating this complexity because they do have different mathematical geometries or levels of detail. So therefore to cross over I need to build a bridge. This is my simplified version because otherwise we will jump now in differential geometry topology. But I

Speaker A

want that everybody can understand this paper here and my video hopefully explaining it. Yeah. So we begin with latent hidden states H1 H3 and these states come from a generative language model a classical LLM your Opus 5. Now

Speaker A

we hope that they lie in a region of representation space that is shaped primarily for predicting what token comes next. If not we have a problem and AI will fail. But to a retrieval is a complete different story is the

Speaker A

retrieval needs now a vector who geometry is shaped for a different task to distinguish the relevant document from irrelevant documents. If I have in my database I didn't [clears throat] anat or if I have a mathematical paper on

Speaker A

quantum field theory it's a complete different paper. Now if I have a question about physics, I'm not interested in the part.

Speaker A

So the crossing that we have now here at first we have a generation oriented manifold of our classical LLM and we have somehow to go to a retrieval oriented geometry. We have to bridge over and shift now build this bridge.

Speaker A

Bridge is my term careful just a simplification with the learned residual transformation. So we do have whenever we map something whenever we go over we do have a transformation and this is well think about in a transformer residual stream data stream we have a

Speaker A

residual transformation and this is our thigh great and then followed by birectional attention and the pooling and everything that you know from the classical transform architecture. So this means the mall internal sword is expressed in the geometry language of generating the

Speaker A

next token and bridge one translates it into the geomet geometric language of search this particular space. Look careful we have some complete different methodology and I would say why now there's a [laughter] problem because the normal classical dense retrieval that we have

Speaker A

here for your graph rag system or whatever you have converts my question into a one vector representation and searches for documents with a similar vector where your um database has been transformed and mapped into a vector space. So that works when the question

Speaker A

directly describes required information. But if we have difficult question with long reasoning traces, we have a problem. So the generative model has been trained to predict the next token.

Speaker A

Its hidden state therefore preserve the information useful for continuing text. The retrieval requires something completely different. It must create a clear separation between the cosine similarity of some vector representation given the query where we have a document D plus that is interesting. So this

Speaker A

similarity must be higher. Then we have a similarity of our query vector with a vector representation of a document that is not related to my top.

Speaker A

So you see we go now if you look at this system we use a little bit of theoretical physics and we say hm wait a minute. So generative state can contain large shared components. No and many vectors in this representation may

Speaker A

partly point or have the direction that there is a dominant direction. Now whenever we have in a space in a gravity space or wherever some dominant direction oriented tensor structure we know that the space itself is not isotropic. So we do have preferred

Speaker A

direction and therefore we call the property that we encounter here anisotropy and to show you the mismatch we have here [laughter] the left chew and the right chew and they're completely not [clears throat] matching at all. So for

Speaker A

you to understand the latent trajectory lies in a geometry shaped for language generation designed for the particular task of predict the next token. But now it must eventually operate in a geometry that is only job is be the perfect

Speaker A

mathematical space for to decide between different documents given my particular query and I wanted you return the document that is best suited domain complexity whatever for the query.

Speaker A

So therefore we just have to reshape the mathematical trajectory for the retrieval job. And how do we reshape trajectories? Well, we have as I showed you a residual transformation of phi in the original state remains available through the hidden layers while the

Speaker A

learn correction term. So this is simply a mathematical formula where we say you know what we add a correction term to it. This is our phi and this modifies here our h gilder here. So the hidden um composition is

Speaker A

now modified a little bit. So in manifold language this can be understood as a local deformation.

Speaker A

And now you know perturbation theory in mathematical and theoretical physics we can use here as already shown here I think three videos ago. I talked in detail about the Jacobian function and the jacobian matrix and this will describe the mathematics for describing

Speaker A

a local deformation in the space and now we talk about the jacobian. Yes, exactly. It was the video where I showed you that entropic discovers the Jsp space and I talked J stands for Jacobian and there I explained the Jacobian. So

Speaker A

if you want to go back here entropic Jspace and the Jacobian tells us here in perturbation theory how small directions around the hidden layers are expanded contracted or even rotated in our mathematical space. So this deformation that we have may now rotate useful

Speaker A

directions. Remember about the isotropic amplify distinction relevant to the retrieval suppress irrelevant shared components and change the distances and the angles for a coen similarity between nearby states. So we modify the mathematical space for retrieval with a simple correction term and this is in

Speaker A

mathematics a jacobian matrix. This is a learned feature transformation, not a guaranteed projection or what we call a defomorphism. Careful. No, it is not. It is a a very particular mathematical object. And now this new paper shift allows all three transformed states and

Speaker A

you might say why? I'll explain this in a second to communicate through birectional attention mechanism that we have here in the transformer heads. No.

Speaker A

And then it combines them into the final retrieval vector. the retrieval vector not the generative vector.

Speaker A

So the arts of the prepar frame their solution as fixing two mismatches and say hey wait a minute we just talked about the representation mismatch here between the retrieval and the generation and now we have a second problem.

Speaker A

Absolutely. What a surprise. I was waiting for this. We have now a second problem in the order called this the super vision supervision mismatch.

Speaker A

I know it's a great so this was just half of the paper I know. Isn't it beautiful? So the second problem is now completely different but they have a solution. So they say the retrieval loss evaluates mainly whether the final

Speaker A

vector finds the correct document. It does not really strongly specify what each intermediate soft token, remember the complexity of a soft token, a fluid differential mathematical object should contain. What is the content of this nonhuman verbal word sequence?

Speaker A

Now theoretically our EI mal opus or whatever you have could produce an effective retrieval vector while its intermediate sort complexity non-human can remain vague or redundant or just pure nonsense. No. So h somehow we want that those intermediate sort to this

Speaker A

intermediate soft token representation really are focusing on the essence of a long reasoning trace. Let's say we have three soft tokens.

Speaker A

So how we do this? Well, takes a little bit of mathematics. What a surprise. So let's start. The author solve this explicitly this here by training here the reasoning trajectory of this LLM for this particular task. And they say okay

Speaker A

we do have to train an AI system to do this crazy So they say okay let's think about it. Yeah. Suppose the explicit reasoning contains 1,000 tokens. No, these are the human words.

Speaker A

Let's make it easy. Token word. But shift has only three soft token. So you see the problem. So each soft token as a new mathematical construct in a higher dimensional space is now assigned to let's say in the simplest way a section

Speaker A

of the explicit human verted trajectory. Yeah. Let's chop it up here in three equal length pieces and let's say this is yet the soft token. Now it turns out no because the essence even in human language is not three equidistance

Speaker A

chunks but you have to find have to really understand what is here the sentence all about. So how we do this transformation from three soft tokens to 1,000 verbal human tokens? Well, we make the mathematical space explode. No. So,

Speaker A

the orders introduced two new coordinate systems. Alpha K and beta I. Alpha K identifies the broad latent reasoning step. And beta I is now the fine positioning here inside this reasoning step or this reasoning segment here. No, you say, "Hey, that's a simple idea."

Speaker A

Okay. So the model then creates a position condition particular state P KI Gilder this is our classical TK state plus here our new five position where we have now the alpha K and the beta I as parameter of this new space

Speaker A

now in mathematics and let's say in manifold language if you study mathematics at the university after three years you would understand now what I tell you otherwise you just have to believe Each course soft token becomes now a

Speaker A

very particular anchor and the coordinates alpha and beta generate a sampled family of underlying nearby states around this anchor. Think about our classical epsilon environment about a particular point in space and we build now here new families of nearby possible

Speaker A

states around that single point here. This is the single soft token. Now we are operating here in several thousand dimensional vector spaces with higher complexity. So it might not be that simple but if you study a little bit you can understand it or maybe ask

Speaker A

your system if you have a weekend coming up. Now I give you a hint if you really want to try to understand it on a different level. This of course you might say hey this resembles a local cart. If you go

Speaker A

about topology, differential topology and mathematics. No. Yeah. But it is only careful. It is only chart like it does not prove smoothness, invertability or a formal two-dimensional manifold itself. It is a different mathematical beast. Okay, back to the nova. So these

Speaker A

expanded states are used to reconstruct the long explicit reasoning trajectory. So you see the next token lost therefore pressures the three soft tokens to retain some fine grained recoverable information and hopefully with the right pre-training data and with the right

Speaker A

alignment of our large language model this system now really learned to do this with three soft token to extract the main information the main complexity in the data and therefore it can answer hey what information must each latent

Speaker A

state retain that with only three latent states elements. We have here the complete latent state information.

Speaker A

And you might say why with three? Well, it turns out that the auto shift feeds the token this soft token back into the model and they repeat here this operation here. They create a short latent trajectory and they repeat it

Speaker A

three times. So at the inference run, the order of shift uses only three such latent steps. So we run through the complete complexity, don't get me wrong, but we just only do three steps. And you might say maybe we have a convergence at

Speaker A

seven steps or 10 or 12 steps. Yeah, but I mean this is just a research prototype. So they just do three steps.

Speaker A

I just noticed that I think I forgot to show you the result that it is working.

Speaker A

So please have a look at the paper. But I was so fascinated here by this idea by the central idea of the paper that I forgot to show you here the result presentation.

Speaker A

What it means from the idea that we have now three latent swords in a nonhuman presentation and hopefully this is now here becoming one effective direction for the reasoning complexity.

Speaker A

Fingers crossed. So therefore if you see my bridges you know we have two bridges because we have two problems that we encounter and bridge one solves the first one and bridge two solves the second one. Guess what? So bridge one is a structural

Speaker A

change kept for interference to fix the anisotropic generation geometry for the retrieval space. Bridge 2 is a high resolution chain used only during the training of the AI to force now the core soft token representation to memorize the right and correct and optimized fine

Speaker A

grained logic steps. Fingers crossed we don't know if it worked. It depends all on your training data.

Speaker A

So again, shift only uses here three soft latent tokens that are constructed specifically for the query while its training target might contain 985 explicit human reasoning tokens. This is exactly what the authors show you in the paper.

Speaker A

And you might say okay but how how is this possible? Now as I told you we do have a explosion of the mathematical space for a particular reason. So let's have a look at this. Now shift cannot directly

Speaker A

compare the three higher complexity vectors with the 985 token reasoning trajectory. I mean come on 3 to 985. No it therefore expands each latent state into a many position conditioned version using here our coordinates that we just went through alpha and beta. And we have

Speaker A

here this TKI gilder. Remember alpha says which course latent step we are associated with and beta tells us where we are inside this particular step detailed reasoning segment the fine graining positioning.

Speaker A

So this means in a [clears throat] manifold mathematical language, one course anchor TK is now quotation mark expanded in pure mathematical terms into a coordinate dependent family of nearby representation and therefore we hope to have the right representation in this

Speaker A

specific mathematical space. So you see a bridge is therefore in my interpretation in my gadang experiment a learned translation across this mismatch across the river. So the first bridge crosses between two tasks condition geometries mathematical spaces and the

Speaker A

second bridge crosses between two levels of representational resolution and you just chosen this to have really a complexity that is just amazing. I really if you start to read the paper you say my goodness this is really something special highly

Speaker A

recommend this paper. So another way to present you the main idea of the paper we have the representation bridge this is reshaping the latent geometry of the space. It creates a new retrieval vector not an X token creation

Speaker A

vector and this is used primarily during the inference run. And then we have the second bridge, the supervision bridge as I call it. It expands the course state into position condition families around this anchor. [clears throat] Mathematically not trivial. It

Speaker A

reconstructs explicit reasoning during the training because we have to train this uh machine to do exactly this what we want. So this means we have to find here the right training data and then start the training process and

Speaker A

everything that you know and it does not it's not needed as an explicit reconstruction process during the retrieval itself. This is training specific.

Speaker A

Okay. Again shift is not crossing only a single river single complexity. It crosses two. Yeah, it crosses here the pure geometric gap that we have from constructing different mathematical spaces for different tasks and also a granularity gap because remember we have

Speaker A

three soft tokens and we have 1,000 tokens of human word reasoning trace. So this paper is gorgeous, beautiful. Have a look at it. So now we have it here.

Speaker A

Now we understand what's going on and then as I told you we have the training process. So how do we do the training?

Speaker A

Well, it's just an EI system. So, we define a loss function. But since we have two mathematical spaces, we define two loss function. And you say, why is it so easy? Well, I don't know. It's just simple AI. Now, so shift combines

Speaker A

two objective here and the loss function. At first, we have the contrastive loss CL. And it teaches here what to place the query near the correct document and away from the incorrect documents. It's so easy if you understand it. If you know exactly what

Speaker A

it is doing and you explain it then to your students, you say, "My goodness, it's so simple. Why couldn't they present the paper in this simple way?" And then we have the second loss function. This is the reconstruction

Speaker A

loss function. And what does this function teach us? To preserve enough fine grained information in the latent three-step trajectory to reconstruct the explicit human English wordbased reasoning complexity.

Speaker A

Therefore, the contrastive learning organizes the retrieval manifold and the reconstruction discourages here the latent path from becoming semantically empty almost co and similarity almost identical and this means well you don't provide any new information and therefore your value goes to zero. No,

Speaker A

you want to have here a semantically different manifold. So if you want to have it here in my Einstein like here famous single sentence explanation what is shift what should you remember shift teaches any model to think briefly in a completely

Speaker A

different continuous geometry remember 42k distributions reshape that geometry now for search modify it add mathematical correction terms and retain enough structure remember that these hidden steps do not become meaningless copies of itself so that our three latent tokens are just

Speaker A

identical twins. So again careful we have different mathematical spaces but also during different training or inference run. So during the training shift uses here the three soft latent steps explicit reasoning trajectory averaging here about our 985 tokens retrieval supervision and next token

Speaker A

reconstruction supervision and during inference shift uses here the original question my question three soft latent steps of course the retrieval bridge the solution and no generated 985 token reasoning trace anyway We say, "Hey, wow, that was quite a ride. Amazing." Here, what a brilliant

Speaker A

idea. Now, I have a thought. How can I make how can I communicate this to people who have no idea what I'm talking about in AI? So, let's come up with an example.

Speaker A

Imagine shift. You need a little bit of creativity now here in this gadunking experiment. Imagine shift as learning a short path through a highdimensional landscape.

Speaker A

Now those three soft tokens are in the simplest way just three points that build here a a path forward. Yeah. And we have two bridges that act now on the same mathematical path. Bridge one bends a little bit the space and integrates

Speaker A

now a path making its new direction useful for the document retrieval rag. and bridge two attaches your defined grain coordinates to the point on the path allowing explicit reasoning to supervise them during the training process.

Speaker A

I hope maybe this mental image helps you to understand what we're doing. We are banding manifolds. We are banning mathematical spaces here for our particular task.

Speaker A

So, wow, here we have it. Three soft tokens here. [laughter] And my goodness, their complexity in their reasoning interaction in new adapted modified mathematical spaces.

Speaker A

You remember Jacobian matrix. And so my goodness, they can have really a new predictive power to understand how we have here an optimized reasoning process, especially for long context reasoning dependencies. And you cannot have a rack system where you just have

Speaker A

one query vector and one solution vector from all the content of a database. It is not possible anymore. We moved on to a higher complexity.

Speaker A

And since it is impossible to end this video without giving you here the result, here is it. Table two. This is the main retrieval performance here on a particular benchmark. You have here different benchmark here for stack exchange, for coding, for theorem based

Speaker A

and here in the last column the average performance indicator. You have it here for different models. You have it here for the standard dense retriever models themselves for some basic contrastive learning from a Q13 model and is llama

Speaker A

model for some explicit reasoning where I showed you rewrite then retrieve and then we have latent reasoning model.

Speaker A

These are the direct competitors here. Laser or gear CSA and down here in orange in the orange box or in the whatever Lila box they have is here shift and shift on three different malls a Q13 0.6 billion and 8 billion and a

Speaker A

llama 3.18 billion. And you see just look at the average okay in two average performance data it outperformed everything else just in one here it did not really outperform because it has 26.3 and here another competitor has 26.9.

Speaker A

So interesting. It is not therefore the overall best model. But the idea to optimize the mathematical spaces for each single mathematical operation for a rack retrieval and reasoning alm is just breathtaking.

Speaker A

So just to be clear now shift does not simply transfer vectors from one manifold to another. No, it creates a short latent trajectory, deforms that particular trajectory towards the necessary retrieval geometry. We bend space and anchors it to a detailed explicit

Speaker A

reasoning through the reconstruction. This is here the correct way. I mean, I hope that I understood reading this paper. Now, I hope you had a little bit of fun. It would be great to see you in my next

Topics:SHIFTsoft tokensAI reasoningtransformer architectureself-construction harnesstoken embeddingsoftmax functiondifferentiabilitycontrastive lossretrieval augmented generation

Get More with the SozAI App

Transcribe recordings, audio files, and YouTube videos — with AI summaries, speaker detection, and unlimited transcriptions.

App Store Google Play

Or transcribe another YouTube video here →

Free tools: TXT to SRT · SRT Validator · Merge SRT · Subtitle to Text · All tools