Speaker A
Hello community. So great that you are back. Honestly, I just wanted to tell you in a very short video, hey, we have a new transformer. Yes, an absolute new transformer architecture. Not the good old transformer from Google that was
Discover the new METIS transformer architecture integrating native memory for AI, surpassing classical models with innovative memory management.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- METIS is a pioneering memory foundation model integrating memory directly into transformer layers.
- It solves key challenges of external memory systems by enabling continuous, differentiable memory access.
- The architecture supports dynamic forgetting of information, enhancing AI adaptability and privacy.
- METIS is open-source and accessible for the community via Hugging Face and GitHub.
- This model marks a significant evolution in transformer design, potentially shaping the future of LLMs.
What the video covers
- Introduction of METIS, a new transformer architecture with integrated memory, distinct from the original Google transformer.
- METIS models (4B and 9B) are available on Hugging Face with full code, papers, and training resources.
- The architecture originates from a university spin-off involving Shanghai Technology Corporation and top Chinese and Singaporean universities.
- METIS introduces a novel forget mechanism allowing AI to delete specific memories, a complex task in current AI systems.
- The model replaces external memory modules with a continuous function space integrated inside the transformer layers.
- It features a dual memory block system: a local memory block with fast runtime weights and a hypermemory block with slow trainable weights.
- This design overcomes limitations of external memory such as optimization bottlenecks, target mismatch, and non-differentiability.
- Memory is treated as a predictive compression problem, with learned compression policies jointly optimized with memory access.
- The architecture hypothesizes memory as a native transformer capability, similar to how reasoning evolved via chain-of-thought.
- The video explains theoretical foundations, mathematical formulations, and practical implications of METIS for future AI development.
Chapters
- 00:00Introduction to the new METIS transformer
- 02:23Why METIS paper is overlooked and its significance
- 04:50Challenges with external memory in AI systems
- 07:21Classical vs new METIS architecture overview
- 10:16Memory blocks: local and hypermemory explained
- 12:55Memory scoring and importance selection mechanism
- 18:04Memory injection and mathematical foundations
- 22:55Role and advantages of METIS memory integration
- 28:44Memory decay, stability, and performance insights
- 31:00Startup context and future outlook of METIS
Full Transcript — Download SRT & Markdown
Speaker A
invented by Google and patented by Google. But now we have Mus with memory and it is available for you. Look, Hugging Face. You have it here updated two days ago. You have a 4B model. You have a 9B model. It is there for you and
Speaker A
it is compatible here with the Hugging Face implementation. So you just go there and this is the code. It is there available for you using the classical transformer code absolutely compatible.
Speaker A
You have a GitHub repo for you, an English Chinese paper overview, quick start training evaluation citation everything for you. And you have a beautiful paper, a mathematical, I mean a computer science paper here on the new ME tensor algebra. And you might
Speaker A
say, why is nobody talking about it? Look, it was published on July 29, 2026.
Speaker A
And I have to say, this is a new transformer model. This is a new memory foundation model that is absolutely interesting because it stands close to the classical transformer model because you know we only have a transformer model and then we have some diffusion
Speaker A
models and then we have some MUMBA models but more or less all our important models are transformer-based models and now we have a new one. So gee this is interesting but look at this. Okay, the main author is here from a Shanghai
Speaker A
Technology Corporation, a spin-off here. But look, he's also here a member of the Renmin University of China and we have from the National University of Singapore, the Shanghai Jiao Tong University, the Tongji University. So, it seems to me this is here a classical
Speaker A
university spin-off. So, I'm absolutely pro startup companies that take some insights from the university studies and they start a classical startup.
Speaker A
Now, so let's have a look. You know what's nice? You can say forget something. You can say, "Hey, please forget that Alice likes to eat beef burger." And then when you come back and say, "Hey, what does Alice like to
Speaker A
eat?" And the AI comes back and says, "Hey, I don't have that information." Do you know how complex our current procedure is to delete memories from my AI system?
Speaker A
Now I looked at the paper and I said what is here the cause that people are not reading the paper and are not fascinated here by the inside of this paper. I mean just look at it. So I had
Speaker A
three questions. Why is everybody ignoring this beautiful new preprint? I mean just think about this new forget technology that is suddenly available for everybody. You can download from Hugging Face. And gee, we have a new transformer architecture with a new
Speaker A
complexity regarding memory that we had to put in a harness or into a rack system. I mean, come on.
Speaker A
They have such a beautiful additional explanation about everything. They even have a theoretical error analysis. They have here a classical Taylor expansion.
Speaker A
They really want to know, hey, how sensible is our result here? How far can we go and tell here our community? Yeah, this is actually working. So, let's have a look at this. And I tried to make it
Speaker A
within 5 minutes, but maybe I need a little bit more. So, I think Matis is a first prototype. It's a prototype of a new what they call a memory foundation model. So, it is an LLM, but it builds
Speaker A
upon the Google Transformer architecture and it integrates memory in new unseen ways. Remember all other people on this planet use external memory modules. No classical agent may appear to have memory but the backbone itself does not. The database
Speaker A
stores the past event and embedding model indexes them. The retriever selects no particular passages. The reranker filters them again. The selected text is concatenated to a particular prompt structure and then the LM processes this particular prompt structure again. So this means the agent
Speaker A
is stateful only because the surrounding harness repeatedly reconstructs the state for the LLM of a core of an agent.
Speaker A
So we do have if you want an architectural decoupling from the core LLM and intelligence of the harness where the harness is actually preparing here the state for the LLM and preparing here the prompt structure and we do have
Speaker A
problems with this external memory. We have a target mismatch. External rack system fetched the text based on surface level embeddings while the LLM backbone might actually need deeper latent representation from its own core that are not identical with a vector database
Speaker A
representation. We do have an optimization bottleneck. You cannot backpropagate gradients efficiently through a discrete non-differentiable external database operation. And of course, our key value cache will just explode. And yeah, and there's also a deeper memory paradox. Now at storage time the system
Speaker A
does not yet know what particular future question the memory or the information or the facts or the knowledge, the procedural knowledge should be encoded.
Speaker A
So therefore memory is fundamentally if you think about it from an abstract point of view a predictive compression problem. Think about a vector space.
Speaker A
This is just a compressed representation of some data. So we do have a present information level but we need a compressed state that anticipates here more or less the future use. So we can encode it in an optimal
Speaker A
representation that whenever the question comes in let's say about physics or medicine or finance that we do have an optimal presentation for this particular use case.
Speaker A
And this new preprint the central proposal is more or less that a compression policy should be learned jointly with the way the new transformer will later read and have access to the memory. A carrier infrastructure.
Speaker A
So let's take a step back. Matis now as we call this new methodology replaces what we have as a discrete rack workflow with a continuous function space that is natively integrated into the transformer.
Speaker A
Finally after almost 10 years the transformer architecture patented by Google is now further developed. Yes, beautifully. So they introduce here what they call here not a transformer block but a Matis block in the transformer block and they have two different block
Speaker A
structures. So they have a local memory block that is representing the current memory state and this block is updated by a static trainable hypermemory block that is based on the current context complexity.
Speaker A
Or if you want to have it simplified and if you have seen this video where I show this low and fast training of the AI, slow for the AI weights and fast for the AI harness, you know this level one
Speaker A
and level two syncing. You can see it also in this framing. The hyperblock contains the trained slow weights and the local block contains the runtime fast weights.
Speaker A
So this is our classical architecture here. External decoupling with the information, the external blocked gradient and we have some sequential output structure. And now comes the new idea. The authors now hypothesize that just as reasoning here in the last two
Speaker A
years became a native capability of our models. Remember it all started with chain of thought. Now memory is the next object we should bring into the transformer and not leave in the harness. So memory should also be internalized directly into the forward
Speaker A
pulse computation in the layers of a transformer architecture as a set of dynamic persistent parameters and the authors did a beautiful job. I just want to show you here their explanation of our good old transformer our friend for the last 10 years. The
Speaker A
causal language model we denote a sequence of input tokens. Causal language model first maps the discrete tokens into a continuous vector representation. Let E denote the token embedding matrix where V is the vocabulary size. The initial hidden state is obtained extracting the
Speaker A
corresponding embedding and combining them with the positional rope embeddings. And after that the initial representation is processed sequentially through the stack of end
Speaker A
model head maps this final state back to the vocabulary space to predict the pro the probability distribution for the next token. This process is commonly mold by linear projection of the normalization followed by a classical softmax function and here h is the final
Speaker A
hidden vector the position and w represents the projection matrix down. After sampling, this new token is appended to the sequence and the model repeats the process until it decodes an end of sequence token reaches its maximum length. Beautiful to the point
Speaker A
absolute gorgeous description of the transformer in a functional density that is just yeah that's it. And they explain pre-training. Yes, of course. Standard auto reggressive next token prediction objective. This is it. So now let's have a look understanding now of the
Speaker A
classical architecture. They now alter the causal self attention mechanism defining now some specific native memory storage and some memory utilization procedures and they are more or less three steps in my wording and I make it a little bit simpler because if
Speaker A
you want to read the paper go it's a little bit on the mathematical side. I just want to give you the main idea. So we do have a native memory state. So instead of a growing key value cache in
Speaker A
a classical self attention historical knowledge now after interaction with the CI system is compressed compactified into and now this is here just an approximation but the order a fixed size dense memory matrix M and a key normalization vector at each layer time
Speaker A
step T. Okay. So the native storage procedure so we write now this now to decide what to remember the hypermemory block learns an importance vector W to score the current hidden state. So this is here a self-learned mechanism where this CI
Speaker A
system is now trained to identify here or assign an importance factor given a particular piece of information of data streams here and you can score this. So you have then a top down top 20 down most important vector states and hidden
Speaker A
states and then you have the read procedure know you have the native utilization procedure you read out this memory and the beauty is now you don't have to go with some complex mathematical you just do this during the
Speaker A
forward pass metas queries here this internal memory state using here learned memory query projection so we're real close to the original self attention mechanism and the standard attention output is simply now mixed mixed I show you the formula in a minute with the
Speaker A
memory attention readout using here a kind of a balancing parameter gamma this is it so instead of a growing key value cache we have here if you want the right and the read procedure and all of this is done here coh in a coherent
Speaker A
mathematical apparatus so that we can mix it with the standard self attention mechanisms Great. So yeah, this is our attention mechanism. So we have the original attention and then we have here our new terms and of course we have to train
Speaker A
this now. So they call it here they call this here a mid training because here they want to train it here on some synthetic data to explicitly learn the operation via some loss objective. And [snorts] they came up with the idea okay
Speaker A
we do have multiple elements of this loss function. We have a reconstruction loss. We have exactly this update for get to reflect loss function and then kind of a regularization prevent the hallucination and the pollution loss function.
Speaker A
So now you see see our architecture changed our optimization procedure changed and the efficiency increased significantly.
Speaker A
Great. Now that we have a feeling what is expected here and what the orers had in mind. Let's have now a closer look.
Speaker A
As I told you, they say in our transformer layer, we add now a mus block and a mus block inside the transformer block here of the Google transformer has a local memory block and a hypermemory block. Beautiful.
Speaker A
The local memory block as you see is adapted at the testing phase and the hypermemory block is optimized at the training phase. So you understand immediately thinking back to the rag optimized procedures. Okay, I understand what the local memory block are
Speaker A
responsible for maintaining the dense representation of the prior information while the hyper memory block constructs of course a parametric function space for native memory procedures to transform here the memory state. This is also what we have done here with hyperx
Speaker A
systems. Now if you say hm why do we have a local memory block and a hyper memory block just look here in one of my last videos we had the same no we were talking here about a multi- aent topology
Speaker A
optimization during the inference run and I told you here if you take this controller step here for the repair of the optimized if you want trace back topology then I just said hey look this short-term playbook just understand it
Speaker A
as a short-term memory memory optimization and here the reflector after I think 24 steps or 12 steps here now rewrites now what have I learned this is just here if you want a long-term memory or a particular skill
Speaker A
update that you transfer into a new skill and you can update this skill so the idea of a short-term memory of a long-term memory just in we encountered here in one of my last videos so this idea is nothing new it is just how they
Speaker A
implemented it not in a playbook like here for the topology in a verbal description but now we go deep into the transformer architecture and we build it here in the mathematical understanding of the transformer.
Speaker A
So the local memory block maintains a memory state at the current step we define a dense memory network M. This is our matrix representation inside the ELF local memory block. So at a particular step T, we denote it as M depending on T
Speaker A
and the model also maintains a query key normalization vector. This is important. This is S. So we have M and S. And both M and S are dynamic parameters updated across different steps. At the first when we start, we have everything down
Speaker A
at zero. And then we go. Now the hypermemory block are responsible for updating here the dynamic parameters in our local memory block based here on the intermediate activation absolutely of the current input X and the output Y. So
Speaker A
each of them consists of a static parameter obtained to mid training which remain unchanged during the interaction parametric foundation. Yes blah blah blah you're familiar with this. So this means we have well we start here with a learned importance vector that scores
Speaker A
every single token and then the system selects the smallest number of the highest scoring token. So what is the most important content of my data stream whose accumulative probability reaches here or crosses over a particular threshold. So the system identifies hey
Speaker A
this is a new information. This is an information I need here given my parametric knowledge and my learned capabilities. This is something I have to remember. And this selective aggregation here of probabilities of course is if you want experimentally the
Speaker A
most important architectural component here in this hypermemory block and what we do we go the classical way of a transformer. We build here construct memory keys and memory values.
Speaker A
And remember in the classical self attention and cross attention we had the same we had here our textual keys and our textual value composition. And now we do this for the memory structures. So the selected hidden states are projected
Speaker A
into a learned memory space. So we just build if you want in the simplest case we just add a new mathematical space where we put the most important memory elements of our discussion that I had with my EI of the last two months. So
Speaker A
what really happened in this two months? And then of course we have to find a mathematical representation. So we compress this into a tensor structure into a matrix structure. So we convert here a variable number of tokens and
Speaker A
with the probability and the relation and whatever into now they or go interestingly with a fixed size associative memory matrix.
Speaker A
Now this is interesting. This has some implications, no because we do have temporal decay because I'm constantly talking to my eye. So the memory matrix will fill up with newer, more important information. So this means we do have a
Speaker A
temporal decay since it's fixed size. So something has to fall out at the other end of the matrix this old unimportant stuff that has already been updated.
Speaker A
Yeah. And then what we just have to do then we just have to fuse it here with the ordinary intent attention. So matis now retains the standard causal attention mechanism that you know and combines it simply with the memory attention
Speaker A
mechanism that looks quite similar. Now this is our current attention. This is our new memory attention. And then the transformer receives two information streams.
Speaker A
the current interaction and the memory if you want the compressed history data and the memory is injected as a residual source at multiple transformer depths rather than a text here at the input where we start with our embedding and
Speaker A
rope and position encoding. So this is now a deep layer mathematical operation in the transformer where we just combine or simply add here the memory attention in a very particular way and gamma is our factor here.
Speaker A
If you want to see this here in a flow diagram, beautiful. This is exactly what we just went through. This is our native storage procedure with our misus block that is part here of our transformer block. Now you see we have here in green
Speaker A
the original attention. You see this here I follow here the shade and then then comes the interesting step. Now we have here the [clears throat] memory attention that is calculated. We just went through this and you might say,
Speaker A
"Hey, do we have a little bit more of a mathematical details?" Oh, I'm so glad you asked because this is it here. Now, we already went through this. This was the native memory utilization procedure.
Speaker A
Huh? But now we go a little bit deeper into the mathematics. So, we define now the memory attention in a complete way.
Speaker A
This is it a gilder. This is for matrix. Yeah, this is a diagonal element. have the details in the paper where our Q denotes the memory query states with optimizable parameter. We add an identity vector to the normalization
Speaker A
denominator to prevent the numerical overflow improve numerical stability. And then the memory attention is integrated into the main branch of the tension and replaces our first equation with this particular equation where the norm is applied to the memory readout
Speaker A
also to align the scale with the original attention branches and everything is according to what we already know from a classical transform architecture.
Speaker A
After completing this we denote that the input hidden state at the out layer is here our hidden layer L minus one. Then the hypermemory block aggregates this into a compact representation. So we learned adaptive aggregation. Here's the softmax formula that you are familiar
Speaker A
with at a particular uh temperature here. And then everything is expressed with L that we know and denotes here the minimum number and the dense memory network is updated here. This is our M and this is our S parameter that we
Speaker A
talked already at the beginning of this video. And here you have it now in mathematical beauty.
Speaker A
So what does it mean after all of this? Let's take a step back. So up until last week before this paper was published two days ago, what was memory? Memory was simple. No, memory in our models was either a textual memory or it was a
Speaker A
latent memory in a mathematical vector space for example or it was part here of the learned experiences of the LLM and it was part of the parametric memory of the core of the agent the LLM or VLM itself.
Speaker A
Now we have a new mechanism number four arrived. Just as a sum up textual memory typically represents information as text relying here on rag retrieval augmented generation for storage and retrieval.
Speaker A
The methods provide information for backbone to support inference by in context learning our good old friend ICL. You have an example here memory tree or memory bank whatever you build it. Latent memory captures the memory through intermediate activations of the
Speaker A
model. Next mem genen our typical models and then we had parametric knowledge memory and parametric memory injects here the knowledge into the internal model parameters. This is here really our tensor weights. No. Yeah. Rome for example. Beautiful. So you see the
Speaker A
storage mechanism was either a classical rack or the LM weight modification with complete fine-tuning with rack we had what it is stored words and records how it is written the harness for rack writes here the text and then the harness decide how to bring
Speaker A
it together and present it then to the LLM how it is retrieved retrieve text and insert into the prompt for the core LLM with the weight it was different yeah we had a gradient based training and was automatically activated during
Speaker A
the inference run and now we have mis state. So what is now stored a compressed latent association from recent interaction what we call memory.
Speaker A
How is it written? It is written as a forward pass fast weight update. So completely different to the harness or the gradientbased training. This is not just a forward pass update. And how is it retrieved? It is an internal memory
Speaker A
attention branch of the complete self attention mechanism, cross attention mechanism, multi head self attention mechanism of a classical Google transform architecture.
Speaker A
What is now why why we need it? What is different? So mi's intended role is now to be rapidly writable, internally accessible within the transformer and a latent experience. So we don't have to transform it back to English words and we can stay within the
Speaker A
mathematical optimized spaces. Yeah. Then you auto show us hey we we learned this. No we trained mus the training included 357,000 primary samples 46 million primary data tokens 27 sources and benchmarks 600,000 samples and beautiful great they did all
Speaker A
of this and we absolutely trust them. Gorgeous. So we have a Chinese startup or Singapore startup insights. Let's look at the result. The overall performance are memory based Q&R tasks. No. And they go here with two different benchmarks. Locomo and here
Speaker A
next ma'am. Beautiful. Now let's look at here what I showed you the QN 3.5 27 billion model. This is if you want the pure QN3. And then based on the Q1 3.5 they added this particular training for Amitis and we go here also for the 27
Speaker A
billion free trainable parameter model. Now just look at here the performance average here we have 0.07 from a Q1 on this particular benchmark.
Speaker A
So this is almost nothing but look at Mis 27B. Here we have 26.74 compared to 0.07.
Speaker A
The other benchmark here locom or next mam here also average we have here 17 for the pure cuban 3.5 27b and for our matis 27b we have now 50.82.
Speaker A
So you might say okay this is absolutely impressive. So for this particular test for this particular complexity this new memory inner transformer is really a performance jump. But wait a minute, wait a minute. We have to read the paper
Speaker A
real carefully and maybe activate a little bit our human brain to understand what's happening because look at this.
Speaker A
This is now here the Cuban 3.527B with the full context given. So if mis does not have to do anything with its memory you provide everything at the data the information the factual data the procedural data full context given
Speaker A
to the Q13.5 27B and now this mall has a performance here instead of 26 now we have 65 and go to the next M benchmark instead of here average 50.82 we have now a performance of 78.8 date. So this
Speaker A
means okay so whatever mat is decided to encode as important memory knowledge that it should keep given or expecting a particular complexity level as a query my human query it is not there yet if you give the system absolute continuous
Speaker A
full access to all the necessary data but compared to an average of 0.07% 07% I would say okay it is a significant jump forward but I would argue here that the mis the mis native memory selection there's space that we can further
Speaker A
optimize it no because it is really not as good as you provide the cub with the complete textual evidence.
Speaker A
Yeah, but compared here to no context evaluation, it just performs here all the other ones.
Speaker A
What is really interesting, but also we will see later limitation. Mus completely avoids a delinear grow of the key value caching and our classical transformer at they have here context window of 128k. matrix requires practically constant storage only 2 MB
Speaker A
of lowering approximation and say hey wait a minute lowerang approximation yeah this is a trick that they used compared to the full context standard here okay so but this has implication now look at this here we have here let's
Speaker A
go for this solid line m's first step is here in this indicator then m's middle step is here with this triangle indicator and m's last step is here with this cross out indicator here. Solid line you see the crossed out the solid line
Speaker A
here crossed out is really here going down and this is more or less here the worst performance.
Speaker A
What it mean? When more information is available, you have longer conversation, more complexity, more data stream, everything has to be compressed into a fixed single matrix.
Speaker A
You can't compress it. You only can compress it to a certain level and then it's just overflowing. And then the performance drops sharply.
Speaker A
So after several hundred words for a really tiny little model recall of first, the middle and the final facts become low in the performance or maybe even unstable. If you continue with this, what is also interesting, we have
Speaker A
these temporal decay that we expect. Now as successful updates accumulate more and more, the earlier memories of course are overwritten or decay. The middle memories start really to fluctuate as the author shows in this paper. And even the recent memories can become unstable
Speaker A
if we really open a floodgate and we flood here our fixed size matrix here with incredible amount of new data. I mean this was to be expected honestly because look at this the effect is now remember we mix here the classical self
Speaker A
attention with the memory attention. So this will now have effect on the overall performance of the architecture. So this means even or once the irrelevant information is stored the overall performer can decline because the mixture of the attention mechanism in
Speaker A
this new transform architecture means that the native memory that is simply now overflowing can damage now the ordinary reasoning process of the classical self attention process.
Speaker A
So this means the native memory can also have a negative effect. And look here at the gap compared to a cuban 3.54b.
Speaker A
Now matis is destroying performance because its memory mechanism is overflowing. So we have to choose the right size for this particular matrix given your particular job complexity in your particular domain. So if you work in mathematics or theoretical physics or
Speaker A
finance or whatever in medicine don't choose the small too the space too small because otherwise yeah so this means native memory is not just a passive storage it has an influence here on the classical reasoning model. It modifies
Speaker A
the hidden computation at every equipped layer in the transformer. And this means if we store some irrelevant information or if we have some state information this can corrupt also the instruction processing and therefore also the reasoning processes. So careful this has
Speaker A
now an interconnected complexity that has a massive effect and can reduce as you see here in this gap here in the very last column our performance of the QN 3.54 billion model.
Speaker A
You remember that I told you that the the authors of this paper use here not here the full complexity but they go here with a low rank decomposition for the ranks of the memory state. Look at this. They say okay let's go with a full
Speaker A
memory state. Let's go. This is 124 dimensional. This is the rank. And now we reduce the rank. And as I showed you in one of my last videos, I have a rank normally if I do physics here between I
Speaker A
don't know. Yeah, let's go with 64 128. Now, as it turns out, if you look here at the scoring, okay, if you start only with a rank one, yeah, you have no performance. Then you go to rank four
Speaker A
for the low rank decomposition, lower, you get better. 16 rank, you improve. But at rank 64, you have already reached the plateau.
Speaker A
So this is now interesting because you see the numerical values here plateau out. So this means that the encoding also you have a mathematical space of 124 rank is encoded only at a rank 64. So we only use a tiny subspace of the available
Speaker A
space. And they show this the full memory state dimension is 1K but a ring 64 SVD reconstruction preserved above 99.9% of the full state average performance.
Speaker A
So this is I would say definitely a new optimization if you understand and beautiful study by the orus they show hey wait a minute our encoding our compactification if we go with here a low rank adaptation like in Lara we
Speaker A
encode it here too restrictive we don't use the whole space no because they show that a useful short-term memory occupies here a comparatively low dimensional subspace which is good if you have only a little bit of data to be learned in
Speaker A
your conversation. [snorts] But which is bad if you have a real high complexity, let's say in physics or you have to calculate something extreme complicated.
Speaker A
If it then also goes with a low dimensional subspace, you simply lose here the information.
Speaker A
Behavioral generalization, we talked about it. You can forget, it can update. But of course, we have to ask, hey, to what lay adapt is this true? Is this just an answer about the system? because the information is somewhere
Speaker A
but at least it is not hopefully available for an answer. So you might say okay but wait a minute what was wrong with the original attention. Now the normal self attention only assesses here tokens that are currently inside your context window.
Speaker A
Now I have context window here. 250k, 1 million, 2 million tokens. So beautiful. But we have other bitter effects that override this. No. And the solution that we found is hey we build a harness around our LLM. And this harness can
Speaker A
compensate now by retrieving now a markdown entry. No. Yeah. But this means that the harness must recognize that a particular data record is relevant for my particular query. Then the harness must and the harness is outside of our
Speaker A
LMM remember. Then the harness must decide and make the active decision to retrieve it then insert it into the prompt after some reanking and then make the LLM process the sentence of the prompt content here again with this
Speaker A
particular information. So this was our workaround no with this ordinary self attention where we had no memory element in the classical transformer.
Speaker A
What misus now adds is it says listen I give you a second information highway within your transformer layer. So what do you have? We have the ordinary attention mechanism over the classical text tokens the current token dynamics.
Speaker A
But I give you something in addition. I give you a memory directly in your layers and you can have the memory attention mechanism that will select now over the history that you have that the AI learned either the AI learned itself
Speaker A
or the AI learned with the interaction with a human the history the last two months we can compress it into a matrix and this matrix will be here our knowledge pool from which we can extract at particular layer depths It's in the
Speaker A
transformer. We don't need rack. We don't need any API call. We don't need any other protocols. It is into our core LLM deep in the layer. We have memory attention.
Speaker A
It's a beautiful idea. I love this idea. Finally, we go the next step. But I told you at the beginning of this video, it is a prototype. So therefore, we do have limitation. And let's be clear and again
Speaker A
I receive always these mails but why you are so anti-AII why you show us the limitation of current AI systems why do just say hey this is perfect and this is gold standard and because we want to continue to learn we want to continue to
Speaker A
optimize so therefore we have to understand what is working what is not working and what is not working we optimize so here of the limitations there is a fundamental failure mechanisms no all historic target key value association are superposed into
Speaker A
the same fixed memory matrix now with misus but this means remember the geometric subspace and autoonality of subspaces and and for transform and everything and value and you got it yeah this means if unrelated memories that are not stored in this fixed size memory
Speaker A
matrix are not geometrically separable they will interfere with each other so if you don't have orthogonal subspaces or we take care that some subspaces do not intervene in the reasoning or the memory process of the attention mechanism with other subspaces. We get a
Speaker A
solid out of it. Matis adds here a learned recurrent and as I told you a fast weight memory state inside a frozen transformer. So the historical interactions are compressed into dense matrix representation during the ordinary forward pass. We don't have
Speaker A
to do the complete training and are later access through a second attention pathway without replaying the original text and without interference time gradient descent. It is only a forward pass and this is what we do naturally in our transform architecture.
Speaker A
But unlike rack which pro rides you a lossless external textual database. Now your external database can be as almost unlimited in size. As I told you, Matis relies here on a fixed size memory matrix structure. We only have limited
Speaker A
space and therefore we have a compression mechanism. We have a parametric compression and yeah we have the compression itself is not lossless but we have a lossy parametric compression mechanisms. And the authors really explicitly show that as the
Speaker A
number of sequential updates increases. So if the capacity now increases or the amount of text in a single step grows over the capacity, matis begins to override its matrix representation. This means more or less it forgets here or it
Speaker A
maybe even scramble earlier facts we have then not a complete coherent logic but only fragments left in our matrix representation. So of course choose the right size of this matrix. No. So this leaves me here with the last piece of
Speaker A
information I would like to show you here from this beautiful study. Have a look at it. You learn so much. It was so interesting for me to read this. Mis is a prototype. But Mis is not yet a
Speaker A
replacement for all of the textual memory that we use with the classical Google transformer. Now this native memory has some beautiful future possibilities and it may absolutely complement rag where we have databases that are huge. But let's be clear in the
Speaker A
current implementation it will not eliminate here our external memory modules like rag or this methodologies because in the classical implementation it is limited to its memory matrix structure and the feed process of the structure.
Speaker A
Yeah, there's a technical detail I have to mention because as I've showed you here, some of the formulas here in this preprint, beautiful, the error analysis, the equation, they're all about a linear update process of this memory matrix.
Speaker A
Whereas in the final benchmark that they published and that I showed you here, they use here a nonlinear update function. Now, you have to understand this is a startup, a Chinese startup company. Of course they not publishing everything in their latest details in
Speaker A
their latest development steps. So yeah they have something here that where they say hey we can go above linear we have a better optimization. So just be aware yes of course and I absolutely understand this here intellectual property rights from a startup you
Speaker A
cannot give everything away but the amount of insight they provide is just beautiful. Mis really demonstrates here a meaningful new engineering and learning paradigm for our transformer architecture for our LLMs, VLMs, VLAS.
Speaker A
This means for the complete AI system that we have. So what we have is our Lauren semantic compression plus a persistent fast weight state a new memory attention mechanism and this mechanism is a gradientfree online update just in a feed forward pass. I
Speaker A
mean come on it couldn't be simpler more elegant and more precise. Do I have some ideas how to further develop this? Some ideas how we can improve this? Absolutely. Hey, I mean three, four or five ideas, but I just
Speaker A
wanted to show you absolutely amazing. Go and read some papers that are not maybe mainstream papers that are not on Twitter or not on Substack, but have a look at papers that are hidden. They are absolutely amazing.
Speaker A
What is the strongest contribution of this matis paper? It is showing that the memory write and read policies that we introduced here can be trained as an internal transformer computation compatible with the classical transformer and then executed with a
Speaker A
forward pass alone. So we have an absolute compatible version how we do this and this is exactly at the beginning of the video I showed you in hugging face you can download it you can operate it with the classical hugging
Speaker A
face environment. This is it. Now the other side the most important unresolved problem of matrix is equally fundamental.
Speaker A
We have a fixed latent compression that produces here this interference produces here the temporal decay and we hope here that we have here a controllable or maybe an uncontrollable pollution if this particular matrix size is too small and we simply have override a temporal
Speaker A
decay and therefore we have to come up with a better solution here for a dynamic allocation of our resources but more about this in a later video. What I wanted to show you they also provide as a future road map here how this memory
Speaker A
foundation malls will evolve. They start with a level one stateful capability. Level two self-managing capabilities.
Speaker A
Level three experience selfarning capability. Level four a persistent capability with enduring internal Mars. and level five here. The absolute crown here, the self evolving capability with an open-ended capability evolution. So, this is absolutely a fascinating paper.
Speaker A
I love it. It's a little bit on the mathematical side, but if you're familiar with it, you don't have to follow your every mathematical detail, every mathematical index. Just try to understand here the main idea. And I hope with this video I provided some
Speaker A
start, some help here. If you are if you want to read this paper now you know more or less what it is the main ideas in this paper. So why not try it out yourself? I hope to see you in my next
Speaker A
video.
Topics:METIStransformer architecturememory foundation modellarge language modelHugging Facememory integrationAI forget mechanismneural compressiondynamic memorymachine learning
