**RAG Just Became a Trainable Neural Graph (WikiFM) โ€” Transcript & Summary | SozAI** Source: https://sozai.app/transcript/rag-trainable-neural-graph-wikifm/ Explores a novel trainable neural graph approach for RAG using a wiki foundational model integrating graph and semantic spaces. ## Key Takeaways - Traditional RAG and graph RAG models struggle with capturing dense semantic details in knowledge retrieval. - Graph foundational models enable cross-domain, multi-task learning with transferable knowledge. - The new wiki foundational model integrates graph and semantic spaces into a unified mathematical framework. - Relation-aware graph attention mechanisms improve message passing and reasoning over complex heterogeneous data. - This approach outperforms previous RAG methods and demonstrates foundational model characteristics. ## What the video covers - Introduction to a novel wiki foundational model combining structural graph and semantic textual spaces for RAG. - Discussion of limitations in traditional RAG and graph RAG approaches, highlighting loss of semantic detail in sparse graph structures. - Overview of graph foundational models (GFMs) and their evolution from task-specific graph neural networks to large-scale pre-trained models. - Explanation of the message passing paradigm in graph neural networks and modifications needed for advanced attention mechanisms. - Presentation of a hybrid mathematical space merging graph and semantic representations for improved knowledge retrieval. - Detailed explanation of passage nodes and entity nodes as vector representations within the graph structure. - Description of the new relation-aware graph attention mechanism enabling multihop message passing across heterogeneous graph and textual nodes. - Comparison of the new wiki foundational model with previous RAG variants, showing superior performance on benchmarks. - Discussion on challenges like parameter sensitivity and distributed training inefficiencies addressed by the new model. - Final remarks on the foundational nature of the model, its adaptability, and potential applications across domains. ## Chapters 1. 00:00 Introduction to the new topic and wiki foundational model 2. 01:31 Limitations of traditional RAG and graph RAG models 3. 03:18 Evolution from graph neural networks to graph foundational models 4. 05:10 Mathematics of message passing in graph neural networks 5. 07:09 Non-mathematical explanation of the new mechanism 6. 08:47 Graph and semantic representation integration 7. 12:23 Relation-aware graph attention mechanism 8. 19:27 Challenges in attention mechanisms and training 9. 22:57 Performance benchmarks and methodology comparison 10. 26:21 Conclusions on foundational model properties and future directions Answers ## Questions about this video What is a graph foundational model? A graph foundational model is a large-scale pre-trained model trained on diverse graph datasets across domains and tasks, enabling transferable knowledge and adaptability for various graph-based applications. How does the new wiki foundational model improve over traditional RAG? It integrates both structural graph representations and dense semantic textual information into a unified mathematical space, enabling better multihop reasoning and preserving semantic details lost in previous sparse graph approaches. What role does the relation-aware graph attention mechanism play? It parameterizes message passing over both relational and textual nodes, allowing selective weighting and normalization of attention coefficients across heterogeneous neighborhoods to enhance reasoning and knowledge retrieval. ## Full Transcript โ€” Download SRT & Markdown 00:01 Speaker A Hello community. Hello. So great to see you. So today we have a brand new topic. 00:07 Speaker A We go about wiki foundational model and we build something we have never built before. A common mathematical space where we have a structural space from the graph representation and the semantic space here from our textual representation and we will go the next 00:23 Speaker A step after a graph neural network and we will build a new dimensional space for RAG. So let's have a look. We have now a wiki foundation model for complex agentic reasoning by Tencent Mash University, Hong Kong Baptist University 00:40 Speaker A and Chen University published September 16, 2026 and they have a simple topic and they say look whenever we have to retrieve additional information like in RAG, now here you see this on the left-hand side what you have you have your 00:54 Speaker A text and then you have some text chunking and then you have a user query and you have retrieved fragments that may or may not contain the complete information and you have several problems. No, you have you miss here the 01:07 Speaker A cross-document reasoning and everything and this was really bad and this is the reason why we invented graph RAG here in the middle of your image. No, but this was over sparse. No, it was here advanced for structured reasoning while 01:21 Speaker A oversparse structures strip away here the dense textual semantic. So [snorts] this means we did not get here really the semantic detail of the text. 01:31 Speaker A Everything was just barebone network structure. So this means we had simple some stereotypic triplets like company is based in a particular city. So if you want in a real scientific analysis here, graph RAG here, it really lost the soul 01:50 Speaker A and the context here, the details of the text here also here a little bit here of all the particular of the feeling of the text, how it feels for a human being. 01:59 Speaker A Yeah. And you know we have beautiful neural networks like a graph neural network that we trained here on one particular graph on one particular task here. So great task-specific calibrated graph neural network but today we will use something much more advanced. We 02:16 Speaker A will go with a graph foundational model and we will use this here for molecules, social network dynamics, code graph and of course wiki graphs and you might say okay so we do have a shared graph embedding space that we're going to 02:31 Speaker A build a new mathematical space for many real-world graphic complexity and this will help us to analyze a lot of application from drug discovery to link prediction, social media to code analysis to you're not going to believe it to 02:46 Speaker A RAG knowledge retrieval and unseen graph elements to discover new patterns. Now you are familiar if you are here a member you would you're a subscriber of my channel but if you're new to AI you say hey wait a minute what is a graph 03:02 Speaker A foundation model? Well I would recommend you this paper from 2025 here you have from University of Notam, University of Connecticut and University of Virginia here a beautiful paper that explains what it is so we started here years ago here with traditional graph 03:18 Speaker A neural networks you had a simple domain data set let's say a molecule combination. Then you had some end to end training and then this graph neural network could perform a single task on the graph topology. But then we moved 03:32 Speaker A on. We said okay so we do have some cross domain database about social academic rap or molecular data and then we do a lot of pre-training and adaptation. We fine-tune it. We distill at a learning process prompt optimization zero shot and then we build 03:47 Speaker A something that we called graph foundational models. Now the beauty was we could use it for different downstream task. Now node identification, missing link identification, complete subgraph identification or the generation here of subgraph text commonologies. 04:04 Speaker A Now in this paper you also see here a graph foundation model definition and definition will be really important to understand why we need this. So a graph foundation model or a class of large scale models pre-trained on extensive 04:17 Speaker A cross domain and cross task graph data sets and through a particular pre-training those GFMs graph foundational models acquire some transferable knowledge and general purpose capability demonstrating some emergent properties and adaptability across diverse graph-based application as I told you molecular property 04:39 Speaker A prediction, recommendation system sync. So our GFMs effectively leverage now the structural and the relational information in order to enhance the predictive performance of this AI system, a graph foundation model. 04:55 Speaker A Now we talked about two years ago about the mathematics of graph neural network here and you know at the core of a graph-based learning lies here what we called the message passing paradigm where the node representation are iteratively updated by aggregating here 05:10 Speaker A the information from the local neighborhood this is nothing else than here if you want a context aggregation here on a graph topology so formally given here a graph with a node set V and edge set E the updated rule here of a 05:26 Speaker A graph neural network was defined here with this update and aggregate process and now you remember when we had a graph convolutional network a GCN aggregation is performed via the normalized summarization where when we went on here to a graph attention network we had this 05:43 Speaker A beautiful attention mechanism now as will turn out in today's new paper again we will have to modify the attention mechanism itself for a particular reason. Yeah. And in the good old times 2025, we had the supervised pre-training, the generative 06:00 Speaker A pre-training, and the contrastive pre-training here with our graph foundation models. And this was the way we trained those particular models here for our complexity. So now you understand if we move on now to graph foundation model, this will be here the 06:17 Speaker A main instrument we're going to use in our AI models today. Now this new paper today this wiki foundation wall is now a very specific configuration because it goes now said okay this was RAG and this was graph RAG and now we go to an LLM wiki this 06:37 Speaker A needs a particular representation that builds now on a graph representation and a semantic representation and you're not going to believe it we're going to build since we have two different mathematical spaces with two different mathematical representation of different data, we 06:55 Speaker A will morph them into one mathematical space. This is not as easy as you might imagine. So therefore, we need a little bit of mathematics. So in the first part of the video, I will show you the precise mathematics. In the second part 07:09 Speaker A of the video, I will explain this to you without any mathematical knowledge. Just I'm going to present you some ideas how I see this and use just here simple examples to explain the complexity of this new mechanism. 07:26 Speaker A Okay, let's start. So the first element here is a little bit more the mathematical side and you might say hey what's going on here? Now we start here with a text passage. Now this is here some paragraph or some important 07:38 Speaker A sentence here from your medical textbook or whatever you have. No. And now we do not use RAG or vector a graph or whatever because now we need something very specific. We build now an encoding and we have a language encoder. 07:52 Speaker A Beautiful. So we get here I don't know a d-dimensional vector. Let's say 756 dimensional 768 dimensional. 08:00 Speaker A But we do not chunk it up. We have now what they call a passage node. So this particular sentence that you see here on your left-hand side is now the vector representation and it is called a passage node. Now a node is of course an 08:16 Speaker A element of a graph representation and the beauty is you do not only have passage nodes but you also have entity nodes. Now this entity nodes here from this headline you see Jack one. So this is a particular molecular whatever and 08:33 Speaker A [snorts] this has here an is an entity node and is extracted here from the passage node. Okay, we are talking here about a particular domain knowledge. We are talking about diseases. We are talking about inhabitation. 08:47 Speaker A So this builds up now our graph representation of the textual complexity that we have. But you see a graph representation has particular strength given its mathematical representation. A textual representation has also some strength that we as a human immediately 09:05 Speaker A understand. Is this a positive sentence, a negative sentence? Are there some connotation? What are the fine details that are expressed in this text? If I just put this in a graph and don't care about it like in graph rack, I miss out 09:20 Speaker A on a lot of the little detailed information. So what this is what we want to have this conserve now because we want to retrieve here this evidence and maybe we want to intrig here the real sentence here from our body of 09:34 Speaker A knowledge here that is really available to us. We want to find this sentence here for this particular information that we're looking for. 09:44 Speaker A Okay. Again in orange entity nodes entity nodes provides an explicit topological knowledge on this particular domain that we're going to map. So when message passes through the entity node they provide no dem with rigid distinct relational path structure. The blue one 10:01 Speaker A are the passage nodes. The passage nodes provide some dense continuous textual evidence. They preserve what I call this nuanced context. No this environmental context. What are what is the environment we're working in? What are further details? The tone some macro 10:19 Speaker A document flow that get destroyed when the text is forcible summarized into some rigid entity triplets for the classical graph representation. We want the details. And now this method uses now a dual space projection forcing both note types into the same dimensionality 10:39 Speaker A in a new mathematical space. And if you want, this is the beauty of the paper. 10:45 Speaker A There's a mechanism that does this for us that fuses those spaces together into one new mathematical space. And you know it, it's graph propagation. What is it? 10:56 Speaker A Graph propagation is simply the mechanism that fuses here two information spaces that have different mathematical representation like the explicit entity topology. Just show you this here, the orange one. and then the dense passage semantic document passage semantic in blue. So let's do this. A 11:16 Speaker A graph propagation transforms if you want an isolated document passage representation into a context enriched representation. And you see we operating now with new message passing algorithms to get here the context of one element on our graph. So it is almost identical 11:36 Speaker A like two years ago when we did context optimization of our prompt structure. We provided additional environment specific graph specific nop specific information to a particular representation. 11:52 Speaker A So therefore a passage can become retrievable suddenly in our rag system in our highdimensional rag complexity because it is structurally now connected to relevant evidence on the graph even if it's vering even if it's semantic complexity is not strongly similar to 12:10 Speaker A the query embedding we have now in this particular mathematical space. So you see with different weight structures you are familiar with this we aggregate it we update it and you might say hey I know this no this is mathematically real 12:23 Speaker A similar here to what we already did in the good old times with a GNN. So this means the graph representation lets now each passage and the entity nodes revise its vector representation using now the selectively weighted information from 12:39 Speaker A the connected nodes nearby. So here we have again what is nearby what is similar and can we have additional context here. So in our WFM this allows now a multihop graph structure to reshape here the passage retrieval without compressing or doing anything 12:58 Speaker A else to our data. We want to have the full-fledged data. So if you want in this new WFM methodology, the graph propagation for UF and neutrii is kind of similar to the message parsing that I explained to you in my videos two years 13:13 Speaker A ago on a graph neural network base and we had three different levels of message passing. 13:20 Speaker A If you want to be a little bit more precise on a mathematical level, you see our WFM graph propagation is a relationware attention weighted GNN message passing methodology and we will even have a correction term regarding here the 13:35 Speaker A particular calculation of the attention mechanism itself. So this special contribution here of WFM graph propagation is not a fundamental new propagation mechanism but it is if you want a kind of a stabilization of a GNN message passing but now over a 13:52 Speaker A hybrid wiki containing both the explicit entity topology and the dense message or document passage embeddings. 14:03 Speaker A This is now our fused mathematical space. So let's have a closer look. Since we have to be absolutely precise, we need definitions. 14:13 Speaker A So what is a graph foundation model paradigm? Let G be our graph denote a large scale graph structure where V are vertices is a node set and E is the edge set and R represents the relational type. A graph foundation model is here 14:29 Speaker A projects now the discrete graph structure into a continuous D-dimensional embedding space. And unlike task specific graph neural networks that GFM learns, some generalizable topological representation across diverse domain topologies yielding a note representation matrix H that supports your zero shell domain 14:49 Speaker A transfer and continuous downstream reasoning. Great. And now for our particular paper an LLM wiki knowledge representation is now what it is a hybrid text augmented knowledge repository denoted as W. So E represent the fine grained entity concept R 15:10 Speaker A denotes a structural interconnection between the entities and D remember this is a collection of dense structured text documents associated with those entities. So each entity E is mapped to one or more passage context weaving here some sparse topological 15:28 Speaker A path with a dense passage semantics T. Now there is another chapter in this paper where they go also for a GPU node optimization. So a a GPU CPU acceleration mechanism for have a better performance if you operate on eight 15:47 Speaker A Nvidia GPUs. I will ignore this part. I will just go for the pure theory and for the understanding here of the wiki graph construction and the mathematical optimization itself. Let's talk about the wiki graph construction and the dual 16:02 Speaker A space initialization. So to bridge now the gap between the discrete relational tribles that we have and the dense documents context that we also have we represent not the order and lm wiki as a hybrid topology W. Its note 16:18 Speaker A set is the union of E and D. The first edge family retains each typed entity relation where the second connects the entity to the passages in which it is describing it is described or mentioned and the cross layer links that make it 16:34 Speaker A special make a passage reachable from the entity topology without reducing its text to an additional triple. So if you want this sentence here carries here the weight of the methodology. So this means we built now a continuous dual space 16:49 Speaker A embedding initialization. Remember in the good old space a time of rag we built a vector space representation and embedding. Now we have a dual space embedding. 16:59 Speaker A So we do have a topological entity embedding and a dense document embedding. You know both as a subscriber of my channel. So got it. And then we have a linear projection layer W maps each textual state to the structural 17:13 Speaker A width. We use here as a common propagation dimension D and initialize an entity by H. And consequently the entity and the passage states have the same width before attention is applied. 17:25 Speaker A Also they originate from different mathematical structural and semantical spaces. Now as I told you we have to have another element in our new methodology and this is what we call an attentative graph foundation model of course. So this means the pure attention mechanism 17:43 Speaker A of a classical transformer GNN is not enough anymore because we will encounter a problem. So our wiki foundation model parameterizes now the message passing via a relation area graph attention mechanism over both the relational topologies and the textual nodes. So 18:03 Speaker A this mean given a particular query my human query the retrieval first identify its seed entities and its passages and induces here the local computation graph on which the following propagation is then performed. However, query conditioning therefore determines which 18:19 Speaker A wiki neighborhood participates here in this aggregation process that I already showed you. While the attention equation determine how the messages within that neighborhood are weighted and I showed you here 0.1 you got the idea. 18:37 Speaker A Now if you want to be a little bit precise on a mathematical level we have to talk now about relation aware attention waiting. 18:47 Speaker A So for a particular node V either an entity node or a document node D it and its connected neighborhood selected specific neighborhood under a relation R. We calculate now a relation dependent propagation score. Each incident message is identified by this pair entity entity 19:06 Speaker A message uses the original Vicki relation whereas the entity passage messages use the corresponding cross layer link type. 19:14 Speaker A So that means the relation attention score pi nothing to do with our llm pi measures how much information propagates now in this particular environment from U to V conditioned on R. So you see this is a tangent superbolicus function. 19:28 Speaker A Beautiful. And then you take attention coefficient comparable across heterogeneous neighborhoods. We normalize this and you have now this particular function alpha. 19:40 Speaker A If you want to code this, this is exactly what you code. If you want to understand the paper, we now start this section without mathematics and we just want to have a little bit fun. So we relax now completely and let me tell you 19:55 Speaker A a story about a beautiful new paper. So you know in the good old times rag retrieved fragments from text here from a database without understanding here the relationship of those fragments. 20:05 Speaker A Graph rack two years later restored now the connection but compresses away the meaning that we as humans can read out or here of some paragraph of text. 20:18 Speaker A Now today's approach here the LLM wiki approach attempts now to preserve both. So this means the LLM wiki has to contain the entities the concepts the typed relation the markdown pages the source passages and indexes and summaries and everything that goes with 20:35 Speaker A it. But as the authors show us the problem is that this richer structure is difficult to encode with the classical graph neural network or GNN. Why? 20:47 Speaker A Because these dense text note create huge heterogeneous neighborhoods. And if you calculate here on those extreme complex neighborhood or classical attention, the attention becomes nearly uniform in this particular sub neighborhood. Let's say every node has suddenly the parameter 0.2.2.2 21:09 Speaker A 20 nodes have the same parameter. So this means we are not sensitive enough in our attention mechanism anymore. So the multihop selectivity collapses and the distributed training becomes really expensive and just yeah this is pure nonsense as they found out. So they had 21:26 Speaker A to invent the attention mechanism for this particular problem. So what they did turn here the wiki if you want in a learned geometric memory structure in this simplification. So how does it work? The system represents the wiki here as W. So we have E of W contains 21:45 Speaker A here the entities and the concept of this particular domain knowledge. R contains here the relationships and guess what decondense now the dense source passages. 21:57 Speaker A The system simply now connects two different layers. We connect now the entity topology with the full document passages. 22:05 Speaker A So this has now a beautiful advantage because this lets now explicit graph paths guide here the navigation while the retrieve passages retain here the evidence required by the LLM for a fine grain detailed analysis. So this means retrieval becomes an iterative reasoning 22:24 Speaker A process. Ah finally no because the agent retrieves your initial evidence set generates your provisional answer because we're just starting to loop no and ask hey is this evidence sufficient for the particular complex query that my crazy human asked me and if not the 22:42 Speaker A machine formulates now a follow-up query targeting now especially the missing evidence. Was there some parameter in the answer that I have no idea what this um character represents? Why do I care about temperature? Why do I care about 22:57 Speaker A some quantum fluctuation? Why is there suddenly something about here emotion in the answer? I have to find more evidence. So the retrieve passages accumulate across the iterations. 23:11 Speaker A Yeah. And then we have a final answer flags here. the stop of the process and the resulting insight is we have now a structured navigation for our REC structure some dense textual evidence where we have really the nuances there's 23:24 Speaker A no chopping up of this textual elements and we have an iterative retrieval there's looping and you got it and this finally will become our new agentic wiki reasoning if we have to train this system of course it's an AI machine it's an AI 23:42 Speaker A system that we have To train for a particular task for complexity we use now a loss function. Now the loss function has to combine multiple training objectives. No because we have here the topological training objective where we have to preserve the graph 23:56 Speaker A relation. We have the alignment that places here the linked entities and the passages near one another. And then we have a variation that prevents here the attention logit from becoming as I told you this is the main mistake in the old 24:10 Speaker A system completely uniform. So a little bit of a complex loss function and the training begins by aligning the entities and the documents while the graph encoder is still frozen and the complete network is then unfrozen for a joint mathematical 24:26 Speaker A optimization exercise that you know if you want to have the particular loss function here in mathematical terms this is it. So what are the results? I just show you one table and you see here if you have here this comparison with light 24:40 Speaker A rag or graph rag here or hippo rag 2 or gfm rag here in the very last line you see our actual new methodology wiki foundational model WFM and you see it outperforms here in all these three benchmarks here everything else 24:58 Speaker A so you may say great what about the insights just to tell you this wiki foundation m changes the rule of external memory. 25:07 Speaker A Remember a traditional rack system treats memory as kind of a database. Graph rack treats memory as a navigationable symbolic structure, a graph puff. 25:18 Speaker A But now our wick foundation model treats memory as a learned geometric medium that has here this particular sequence. 25:27 Speaker A So therefore we can say our WFM a novel agent native foundation model paradigm that bridges now for the first time the fundamental gap between the structural knowledge retrieval here via graph structure and a dynamic atantic reasoning that is based on the details 25:45 Speaker A here of human readable text and then we just build a unified topology mapping combining now two different worlds. On the one hand we have the topological space discrete entities and relational the crow f rag elements are projected via lookup tables 26:03 Speaker A into some continuous embeddings. On the other hand, we have the semantic space. Dense document passages are mapped via a pre-trained LLM backbone into the same dimensional space. And then we introduce explicit cross layer edges connecting the entities to full document nodes. 26:21 Speaker A And this finally allows here continuous textual semantic to participate directly in a multihop message passing on our graph neural network. So you see from transitioning from the traditional sparse triplebased graph like graph rag in the good old times we are now moving 26:39 Speaker A towards some high density LLM wiki and the aba wiki foundation model establishes a continuous geometric space capable of joint learning of a the dense document semantic and b the complex relational topology of this space and what I really like is not just that it's 26:58 Speaker A a reflection and again a rag or whatever It is simply a kind of to preserve the full document passages as first class graph note. So we really still have access here to the original documentation of a physical effect for 27:13 Speaker A example and we can jointly learn how evidence moves between the semantic context and the explicit topology of our graph representation. 27:24 Speaker A Are there limitations? Well, of course you know me there are limitations. So let's have a look that it is a query conditioned attention. Yeah, it's a little bit overstated because the query selects the active graph neighborhood but the formal attention score itself 27:40 Speaker A contains here no explicit query vector. The self reflection kind of belongs to the surrounding agent loop. No, so the answer generating LLM decides whether to search again or not. So we are limited here to the capacity of this particular 27:55 Speaker A LLM and this is not which would be beautiful. It is not an emergent property of the wiki encoder itself. And whenever you see foundation malls no that popped up three years ago here the foundation mall claim I think it's not 28:13 Speaker A really there the complete set of evidence to claim that this is really a foundation model. Yeah. So does not clearly report the pre-training scale, the optimizer configuration, the release checkpoint or control serial transfer experiment across unseen Viki domains 28:30 Speaker A that we really can say that it is a foundational model. But yeah, I like the idea. I like the paper. Have a look at the paper. It is really something new. I kind of have here this presentation that 28:42 Speaker A I showed you today. a little bit of mathematics, a little bit of just describing you here the main idea, the main inside of this paper. But of course, you have the maximum benefits. 28:53 Speaker A If you now start to read the paper yourself after watching this video, you know exactly what we are talking about. 29:00 Speaker A You can now focus on all the mathematical details. Enjoy it. I see you in my next Topics: RAG Graph Neural Networks Graph Foundation Model Knowledge Retrieval Semantic Representation Message Passing Graph Attention Machine Learning AI Natural Language Processing --- This is the markdown twin of https://sozai.app/transcript/rag-trainable-neural-graph-wikifm/ โ€” the same content, without the markup. Published by SozAI (https://sozai.app). Reuse and quotation are allowed with attribution and a link back. Machine-readable index: https://sozai.app/llms.txt ยท data API: https://sozai.app/api/