Speaker A
Hello, community. So great that you are back. Yes, you look at this image and you say, "My goodness, this is just AI craziness." Absolutely not. This is really the scientific content of the two papers we're going to talk about today.
Explores Agentic Lightning V1, a new RL math framework for harnessed LLMs, addressing complexities in multi-agent reinforcement learning.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- Harnesses introduce significant complexity by filtering and reformulating information between LLM and environment.
- Agentic Lightning V1 mathematically formalizes RL training under harness constraints with multi-agent orchestration.
- Traditional continuous token history assumptions no longer hold; new prompt-action sequence structures are needed.
- Re-tokenization and prompt reformulation by the harness require novel loss function designs and gradient averaging.
- This framework sets the stage for future RL training on complex models like mixtures of experts.
What the video covers
- Introduces the challenge of reinforcement learning (RL) with harnessed large language models (LLMs) where a harness sits between the LLM and environment.
- Explains how traditional RL assumes direct LLM-environment interaction with continuous token history, which breaks down with harness interference.
- Presents Agentic Lightning V1, a new mathematical framework developed by Microsoft and University of Edinburgh to handle harnessed agentic RL.
- Highlights the shift from continuous token histories to pair API call prompt structures and multi-agent sub-agent handoffs.
- Discusses the harness’s role in context construction, control flow, tool execution, and prompt generation, complicating RL training.
- Describes how the harness reformulates prompts and tool calls, causing re-tokenization challenges previously unaddressed.
- Details the new RL loss functions and training batch handling to accommodate harness complexity and token sequence discontinuities.
- Emphasizes the importance of averaging gradients over entire rollout tasks due to non-deterministic outcomes in harnessed RL.
- Mentions the extension of this methodology to mixture of expert models, increasing complexity further.
- Calls this work a new blueprint for large-scale harnessed RL, marking a significant evolution from classical RL approaches.
Chapters
- 00:00Introduction and Context of Agentic Lightning
- 01:41Overview of Agentic Reinforcement Learning
- 03:48Review of Previous Work and State Space Chaos
- 05:31Traditional RL Without Harness
- 07:29Challenges with Harnessed RL and Prompt Reformulation
- 09:33Mathematical Innovations and Loss Function Redesign
- 11:37Training Batch Handling and Non-Determinism
- 13:49Summary and Future Directions
Full Transcript — Download SRT & Markdown
Speaker A
And you might say, "Why? Why is this of interest to me?" Well, look at this.
Speaker A
Just yesterday, August 18, we have here a post by Sam Altman that says, "Yeah, we have post some frontier reinforcement learning training." And whatever you have read on the internet here on some crazy theories, we look into the technology of this, and I
Speaker A
want to show you this is not simple at all. So, let's have a technology view on this particular thing. Now, you might find that the first paper about Agent Lightning here, and this is here the next version, is about the precise
Speaker A
mathematical reinforcement learning methodologies on the exact surface of the LLM to the harness. Because remember in my last video I was talking about exactly this, how can we train your LLM if suddenly we have a harness here in
Speaker A
the middle. It completely disturbs you the mathematic. It breaks the mathematic of reinforcement learning as you know it from a classical LLM.
Speaker A
[snorts] And then the second paper will be that we put or we drive this idea into the industrial environment.
Speaker A
And we will use here the simple model and put it into and you know the next models will be mixture of expert models.
Speaker A
We put it now into a mixture of expert model because my goodness, now the reinforcement learning complexity here with different expert systems here within one shell will become so much more difficult to understand. So, let's go to the leading and bleeding edge of
Speaker A
technology, and if you think reinforcement learning is simple and is solved, welcome to this video because I'm going to show you you have no idea what's happening right now in the environment. So, just to make sure we're on the same page, agentic reinforcement
Speaker A
learning. Now, you have your policy model from your LLM, you have your action tokens, you have the tokenizer and detokenizer, you have the action and you have the interaction, the direct interaction with the environment. The environment gives you a feedback and
Speaker A
observation, you have again the tokenizer and detokenizer, and your observation tokens come back to the LLM and the tokens are here beautifully direct here responding here the contact with the environment. Now if you have this harness in between like your Claude
Speaker A
or your Hermes or whatever you have, you have now at least some strange elements sitting here in between your LLM and the environment. So you do have an information filter that cripples here the beautiful pure information stream that rewrites something, that optimizes
Speaker A
something, that compactifies something, and it just presents you a lot of problems. And in my last video I showed you one of the solutions was to have here an OpenAI-like API call here. And I showed you in my last video why this
Speaker A
presents here the solution if we have a harness present and our goal is reinforcement learning training for our policy model, for our LLM.
Speaker A
Beautiful. So let's talk about this new complete new methodology for a harnessed agentic reinforcement learning. So agentic we have multiple agents here and this is we have to take care about this contradictory sphere around the core of our LLM.
Speaker A
This is our harness. So let's talk about it. What is it? It's a new kind of paradigm in which the deploy time harness is not directly involved in the LLM post training.
Speaker A
So this is what we want because this is the real system complexity we're going to deal with later. We want here that in the post training we have the same harness complexity like in the deploy time complexity. Great.
Speaker A
Now in my last video here this was part one. If you have a look at this Maxwell's of EI, how the API proxies tame here, the reinforcement learning state space chaos, and we talked about this, and now we go a step further.
Speaker A
Now, this is here 18th of August here, new study by Microsoft with a university, it's a young university, University of Edinburgh, about the next iteration of Agent Lightning. Now, we go from a from version zero to version one.
Speaker A
So, this is now exactly our harnessed agentic reinforcement learning. So, you is the leading and bleeding edge that Microsoft publishes and the young university publishes here. Great.
Speaker A
Now, you see, we have now not only that the state is defined by the environment, but we have now for the LLM, the harness in the environment here defines now the external state. The model input is not anymore a continuous token history, but
Speaker A
we have now pair API call prompt structures here with a prompt and an action sequence. And our agents are not anymore angle any single react agents, but we have multi-agent sub-agent handoffs, and all the complexity of the real world. I'm not talking even about
Speaker A
the mixture of expert, how we have to train four mixture of expert data sets, because this will be a mathematical wonderland. So, okay.
Speaker A
Again, if you're new to AI, you know, we have this traditional agentic reinforcement learning. The latent state is primarily the environment state, no?
Speaker A
The policy model LLM interacts mostly directly with the environment through a transparent layer, and the model predicts uses here the action token, the environment returns an observation.
Speaker A
And the tokenized observation extends existing history. The policy observe one continuously extended token history.
Speaker A
Beautiful, linear, non-disturbed by any harness at all. The rollout naturally forms a one a linear token trajectory.
Speaker A
This was the good old times. And now the new idea, forget the good old times, because we are now in the brand new chaotic times of harnessed agentic reinforcement learning. Because now the policy model no longer interacts directly with the environment, yeah? We
Speaker A
have a latent [snorts] state that contains now both, as I already showed you, the harness state as itself in its own complexity and the environment state.
Speaker A
So, we have chaos plus chaos. So, you say, "Great. Now, let's find here the reinforcement learning mathematics for this because otherwise we cannot code any training algorithms." So, you remember the harness owns the context construction for the LLM,
Speaker A
owns the control flow, the tool execution, the agent orchestration, and independently constructs you the request prompt for each single model call in the interaction between harness and LLM. And the policy, our LLM, observes only the exact prompt delivered through an LLM
Speaker A
API, and only on this information, not the direct information, the feedback from the environment, generates a response conditioned on this single condition prompt, like, "Now, generate this action." It is so important that you really get a feeling for this. So, let's do an
Speaker A
example. The agent is now fixing a bug. Beautiful. So, we have now, remember my last video, we talked about API call one. The harness sends, "Hey, find the cause of the bug." And the model the LLM answers, "I will inspect now the first
Speaker A
file, parts of Python." So, the harness executes now this particular action and receives now a tool result. And you say, "Beautiful." So, with this tool result, we have now API call two.
Speaker A
The harness constructs now a new prompt, and this is important. It constructs a new prompt containing now the earlier conversation, what we already know from call one, plus the tool result.
Speaker A
And you say, "Hey, great. Yeah, of course. What could be the problem with this?" And now the response of the model proposes now, yeah, change the code in line 17 and do blah blah blah.
Speaker A
So, if the call two contains now the call one exactly, you say, "Okay, to train in order our RL trainer can combine them now into one continuous training sequence like we had in the traditional enchanting reinforcement learning." Now, even with the harness, but
Speaker A
you know, the harness is now a living thing. It does something crazy. Maybe it summarizes the earlier conversation. It reforms the tool call. It removes simply del
Speaker A
And then, of course, call two is not exactly a continuation of call one. And you say, "Who cares? This is an AI.
Speaker A
It must be intelligent, no? It must immediately understand from its emergent intelligence superintelligence that this is somehow call one connected to call two." Turns out, no, the AI has no idea about this. This is just a machine, and you have not instructed the
Speaker A
machine exactly what to do. So, the trainer RL now starts a new segment. So, we lost here that this is here a con- ongoing conversation.
Speaker A
So, once the real harness controls the work, the training system no longer sees here the complete process. Now, it sees only the selected conversation prepared for by the harness. The harness is filtering the environment, and this is where the current mathematic for
Speaker A
reinforcement learning breaks. Simple as that. Showed it in my last video. So, the central methodological insight is therefore the harness rollout is not necessarily one continuous token trajectory. It is broken. It is segments. It is fragmented. It is pure chaos, and now we
Speaker A
have from the mathematical point of view find what segments belong in a linear chain to what other segments because we have API calls.
Speaker A
So, this is now collection of policy decisions generated under separated constructed context for the LLMs.
Speaker A
And you may say, "What can go wrong? Now we have an intelligent AI. Yes, beautiful." So, let us show Well, let us go through each step now and let's encounter the first problem that you ought to show us here from Microsoft and
Speaker A
all the universities. Immediately, we encounter what they call a retokenization problem. Because we have here two different systems. The agent harness communicates with the model APIs through, guess what?
Speaker A
English words, English sentences. This means text messages. But you know exactly that the mathematics for reinforcement learning is based on token sequences.
Speaker A
And you might say, "Yeah, okay, so no problem. I know the mapping function. This is a tokenizer or a detokenizer.
Speaker A
So, I can transfer here for a particular tokenizer that I select." On my channel, I have 15 videos about how to build different tokenizers here 3 years ago.
Speaker A
So, you say, "Yeah, no problem. Now we build a tokenizer and then we have it.
Speaker A
We do have coherence." Turns out, no, we don't. Because what the harness summarizes here, what is happening, what what is not compactified, it chunks the text differently.
Speaker A
Every API call has a different token sequence. And why do we have to tokenize again?
Speaker A
Why we have these tokenize and detokenization? Well, guess what? Because the agent API communicates using the text matches and not the token IDs. So, what is happening? Think about it. What What loop do we have? The model LLM generates
Speaker A
tokens. The server converts them into readable text, detokenization. The harness stores that particular text in the history. The harness adds another tool or salt or a new instruction here for the LLM in the context window, beautiful. And it sends the complete
Speaker A
conversation back here as text to the context window of the LLM. And now the models are must tokenize it again since this is now a new additional prompt.
Speaker A
And this is now something worth think about it. Each call is therefore effectively stateless at the API boundary.
Speaker A
And this means since we're talking here about not a deterministic system, but a statistically probabilistic system that even if you have the identical LLM and the identical harness, the tokenization, the retokenization will be different.
Speaker A
And you see this is not possible. This is This is really happening. This is really happening. So, how we deal with this? How does is not a new idea, the new method. We have now to preserve what the engineer, the RL engineer
Speaker A
actually really saw. So, now you know computers do not read whole words, no? They divide text into pieces, the tokens. Just think about the word having as I showed you in my last video, no? We have a token combination of H and
Speaker A
aving. This is here. But when you do the retokenization, you might have here a different token sequence like have and ing, no?
Speaker A
Different token combination. So, for us the word the English word is identical, but inside the reinforcement learning training for the transformer architecture, the underlying pieces are absolutely different. Because here I have H and here I have have, and here I
Speaker A
have aving and here I have ing. So, these are different tokens. So, this means my sequence of token will have a different token sequence.
Speaker A
And therefore, or here you have it in a screenshot from the study here, no?
Speaker A
Having and having here the retokenization that is happening in a rollout that records now since we no contact with the real environment, we have now a sequence row.
Speaker A
And here [snorts] we have it, prompt one, action one, prompt two, action two. You see, we're cut off from the environment. We're only given by the harness a particular prompt one and the LLM decides on an action one.
Speaker A
And then we have prompt two based on this. And then the LLM decides again on the action two.
Speaker A
But we lost contact to the environment. And now we have these re-tokenizations since we go over API calls.
Speaker A
So, the real AI agent does not produce a clean conversation, it harness may summarize context, restarts call, invoke sub agents, reformat messages, delete messages, delete context, re-compactify context.
Speaker A
Every single call is different and the LLM has no idea for the reinforcement learning mathematics what goes together.
Speaker A
So, one real attempt can become now several separate training fragments because guess what? This a harness is not operating on a file system basis, no?
Speaker A
So, what is now the main innovation, you might say, by Microsoft? Except, okay, we understand that we have a problem, a massive problem with the re-tokenization that nobody thought about before, but what we do with it?
Speaker A
Now, they introduce now three correction, three corrective terms, no? So, they say, "Listen what? The joint fragments, whatever fragment belongs together here in a sequence, must be joined together, no?
Speaker A
But only if the underlying tokens match exactly. Otherwise, if we have a different token sequence, we have to keep them separate and therefore we have to have a new sequence.
Speaker A
Then we have to calculate the learning signal. Remember, we are here in reinforcement learning, so we have a signal that goes back. And you might say, "Yeah, based on what sequence?" And they say, "Based only on the complete
Speaker A
attempt, not per fragment." And then, only one attempt gets one vote. And finally, they tell us here, "You have to give every attempt equal total training weight, regardless of how many fragments or tokens it produced." And you might say, "Okay, but hey, my
Speaker A
loss function is not at all built like this. My mathematics is not working for this." So, let's have a closer look. So, what is happening is here dynamic sample count now.
Speaker A
So, think about it. Sometimes an agent finishes here my job here in two calls, now.
Speaker A
But sometimes, the agent decide, "Hey, I need at least three sub-agents that do here some sub-jobs for me, and it takes maybe 50 different calls to do the same job, came up with the same result, but now, instead of two calls, 50 calls
Speaker A
and three sub-agents. So, you see, all the communication between LLM and harness will be now much more complex for 50 calls, will have now, I don't know, whatever redundant information, compactification, we walked around here some random pathways that are just crazy, instead
Speaker A
that we solved it here elegantly with just two calls to the right mathematical libraries.
Speaker A
So, you might say, "Great, but remember in the traditional reinforcement learning mathematics, when an AI succeed, the reward, your reward is sequence of action. You have the final reward system, now.
Speaker A
But if any AI succeeds by making 50 small calls, and another AI succeed by making two brilliant mathematical calls, the traditional mathematical reinforcement learning algorithm now says "Hey we will now start waiting the 50 call run as more important, now. Look at what
Speaker A
beautiful 50 step, no? This must be important, no? This is a long sequence of reasoning, no? Because you generated more training samples, and this was the old classical reinforcement learning methodology no?
Speaker A
But now, if it does not understand that the two brilliant calls are much better, much more precise, less token, we have now the if you use the same RL mathematics on this particular problem, it makes you the RL training unstable
Speaker A
and pushes the eye to be widely inefficient. Now I must say, okay, maybe some AI companies have no problem with this, that you use 50 times more token, but hey, we are here in a perfect world, of course, no? So, what is the mathematical
Speaker A
fix is to ensure the method divides here the loss by the rollout level, no? So, meaning that an attempt that took 50 calls gets the exact same optimization weight structure as an attempt that just took two calls. So, careful, we have now
Speaker A
modified the statistical numerator and denominator, not according to the different other amount of calls, but here to the rollout level.
Speaker A
And you know me, and I say, hey, wait a minute, the dynamic sample count also makes now that loss normalization a non-trivial choice. And you're absolutely right, and you have here different ideas by the authors here how to do this, no?
Speaker A
So, consider a training batch of our rollouts, and our rows, where the rollout row is producing now n samples, and a sample J of rollout row has L response tokens per token loss.
Speaker A
Yes, yes, yes, and you use now Dapo and whatever, and it turns out no, you have to use now a very specific uh response function here, and this is here equation number 16. If you want to go through the mathematics, have a look,
Speaker A
you see also here much more detail in the appendix A of this paper, where you have the complete uh explanation here of each and every single term.
Speaker A
So, this means now kind of summarizing the paper. So, we do have three major innovation here that solve now kind of the problem. Now, but we have to preserve the real token history, not the rewritten history. We compute now the
Speaker A
rewards, the feedback here for the RL learning here per a completed attempt as the authors define, and give them each attempt equal total influence for the training.
Speaker A
And when I read this, I said, "Great. So, what is not the mathematical definition of an incomplete attempt?" Because, hey, look, we have to do the mathematics, and we have to do the coding, and then we can execute the
Speaker A
program. So, an attempt is a complete run of the agent on a single task.
Speaker A
This preprint also calls it a rollout. So, attempt equal rollout. Let me give you a simple example. What is an attempt? Suppose the task is repair the bug in a particular Python file yeah?
Speaker A
So, the attempt begins when the agent receives this task, and it ends either when the agent submits a solution, when the tests are evaluated, the time action limit is reached, or the execution fails.
Speaker A
So, whatever happens to this, this is a complete attempt. And during this attempt, the agent may make many more calls, and may use many tools and tool calls and API calls and whatever. All of these becomes one attempt.
Speaker A
Now, of course, for the training system, it has to assign a unique rollout ID, yeah? A new rollout ID for particular thing is number 17. Beautiful. So, we know Now, you know, we have a complete bookkeeping, every more call, every
Speaker A
tool-related event, and any final reward. And you say, "Ah, now I get it." is associated now with the rollout ID 17, and you got it here, yeah?
Speaker A
So, remember, we do all of this because we want to have a new mathematical reinforcement learning here, especially for the feedback function for, you got it here for the reward that comes back here the final task for the end of the
Speaker A
task. So, for the coding task, guess what we have if it passes here a one, just to reward success, or if the test fails, a zero non-success.
Speaker A
Now, the system can also record its length, the number of calls, the number of tokens, but you understand the important measure for learning is the final reward function as it has been in the good old times.
Speaker A
Now, the authors tell us, "Hey, about fragments." What is a fragment? A fragment is only a training data segment extracted now from that single attempt. So, what it does contain? One or more consecutive model calls that can be joined using now the exact token
Speaker A
continue function. This is a fragment. So, one complete attempt may therefore produce three fragments, F1, F2, F3.
Speaker A
Those are the fragments, and they are not different attempts. Careful when you read the paper in the notation.
Speaker A
Sometimes you have to be really be careful about the mathematical notation here, how they are addressed now. So, they are three pieces of the same attempt that go together and can be joined using exact token continue functions.
Speaker A
Beautiful. Give an example. Repair a Python file, yeah? Call one, agent says, "Hey, I will inspect this this file." And the harness executes now this particular command. Call two, previous conversation plus now the tool result, yeah?
Speaker A
Harness comes back and says, "Yes, I run the tool in my Python sandbox. The error is at line 42." And communicates this back to the core, the LLM, and the LLM here tells us, "Oh, okay, I will change now the condition at
Speaker A
line 42, yeah?" So, if the call two contains now the call one, exactly, you see this orange text here, they can be joined because here a beautiful call two contains call one exactly the exact same token sequence and we can join them.
Speaker A
So the harness then compresses the conversation. And call three is now summary the parser condition at line 42 was changed and the response by the LLM here is okay, I will now run the tests.
Speaker A
Because this summary is not an exact continuation of call two because suddenly we have a summary you see call three becomes a new fragment.
Speaker A
So this is now fragment two is now exactly the content of call three. So therefore you have it now. If we have a particular rollout, let's say this is rollout number 17, we have this are now contains since we have call one, two,
Speaker A
three F1 and F2 and they F1 join call one and call two and F2 has only call three therefore and the reward is of course once it has passed.
Speaker A
So you might say, "My goodness, really necessary here?" Yeah, absolutely because two rollouts are independent execution of the same task and they can differ because remember the LLM samples here its next response probabilistically.
Speaker A
So this means identical system but it is statistics. It is not deterministic now. So one rollout might pass and one rollout might seem to fail.
Speaker A
Great. So what this preprint solves with its three mathematical hm how to call it real beautiful definitions here on the surface of LLM two agent and a communication protocol do not let fragmentation distort you the terminal reward function. But what it
Speaker A
does not solve and this is the limitation of the paper, which fragment actually causes not a success, which fragment is doing the real work to understand and to correct it so that we have a correct result at the end.
Speaker A
So, therefore, you have it, no? The paper is rather simple. The core ideas are conceptually simple.
Speaker A
It is, number one, preserve the exact tokens that the model actually experienced. Second, calculate the reward statistics, the mathematics here, per real execution, not per data fragment.
Speaker A
Leave the data fragment alone, build this new statistical parameter, and give each execution equal total training weight, not depending on the fragment like in the old model.
Speaker A
If you've done this, you're on the right track. So, you see, this comes here from preventing subtle but serious statistical errors.
Speaker A
This is statistic pure, which parameter depends on what other parameters, what are the consequences if you choose one parameter over the other parameter, and got it.
Speaker A
Beautiful. Yes, here you have it, no? You have your LLM core produces here your beautiful token sequence. You have the recording proxy here that sends it over to the harness, and the harness gateway produces now something here because this is the harness defining now
Speaker A
for the LLM core the feedback from the environment. And either you have a beautiful something that really has here call one and call two really contains call one completely, and there's no summarization, no compactification ring, or you have a break in the sequence. You
Speaker A
have summarization, you have compactification, you have whatever. So therefore you have to make sure that we are not going here for the fragments, but one reward is given per attempt. And you see, in the first case, we have here
Speaker A
multiple fragments here, but it is only given one reward or maybe you just have one run and one segment here fragment here and then of course you also get here the reward for this particular element yeah?
Speaker A
And third, you have equal weight per attempt statistically calculating here RL. So, if you think this preprint is not really a new idea, something new spectacular, but it is rather here about the measurement and the statistical correctness at the boundary between the
Speaker A
LLM and its harness for our particular harness agentic reinforcement learning mathematical definitions. Beautiful. So, they implemented this new agentic light here in just 3,500 lines of code.
Speaker A
So, you see this statistical, let's say, optimization focusing on the right parameters, doing here the right renormalization of our loss function and and and can be done here with little bit of help of AI within 3,500 lines of code, which is
Speaker A
rather nice. Have a look at the appendix A for all the details. So, what is now the new mathematical result? What we have to change?
Speaker A
We just learned from this preprint that you must rewrite your loss function to average the gradients over the entire task rollout.
Speaker A
Rather than a per token or a per API call or else the cherry eye behavior will poison your training data. And this is exactly what happens up until now.
Speaker A
So therefore rewrite your loss function to average the gradients over the entire task rollout.
Speaker A
So, you see this is simple to do. This is just a little bit of an adjustment, but we have to really understand what happened to the system, why we have to modify the statistical parameter and how we have to modify the parameter.
Speaker A
Now, let's talk here about the implementation on a multi-GPU configuration. Is it synchronous, asynchronous, whatever? Now, you know in the agenda our RL the synchronous RL setup is slow, no? All rollouts in a batch must finish before the training
Speaker A
step can update the model, so the training step must wait for the slowest rollout in the batch, leaving many GPUs idle. And then we had asynchronous RL, beautiful. You see this especially in my last video when we talked about a real
Speaker A
asynchronous real solves this problems, but they found here even a better collocated asynchronous RL methodology for rollout and weight updates share the same pool of GPUs, and they found a new optimization method for this.
Speaker A
Now, you see it here. This is here on top, you see synchronous. On the right-hand side you have here the classical asynchronous, no? More required GPUs, and but you see this is here from the time optimization, not great from the full rollout, the partial
Speaker A
rollout, and the update functions. And their idea is here the collocated uh asynchronous methodology. But, you know what? This is just too simple for us because you say, "Hey, we're experts." So, surprise, let's go to the second paper. And the second paper is by Huawei
Speaker A
Technologies and the Chinese University of Hong Kong. They go a step further. They build on the ideas that we just walked through, but have a different solution than Microsoft. Huawei says, "Let's go further." So, they have LEGO reinforcement
Speaker A
learning. And you have immediately a feeling if you read LEGO what this RL methodology is all about. We are on the same topic, we have the same task, we have an agent with a harness system, and now we want
Speaker A
to do the reinforcement learning. Huawei is focusing here exactly the same like our American counterpart here from uh Now, this is here the leading and bleeding edge how we do the reinforcement learning for new models.
Speaker A
But the new models have a harness. So, we have now to talk about the harness native reinforcement learning.
Speaker A
Four, and now let's make it easy. Let's go to something that can be easily verified, coding agents.
Speaker A
Beautiful. It is published on the same date. So, the authors had no idea that somebody else was doing this. You have the GitHub here. You have the hugging face here after information. Great.
Speaker A
So, what we're doing now? We are going now here from the laboratory here on the left side with our beautiful harness native corner out in the industrial real complex world where it is real dirty, real chaotic, and we say, "Yeah, nice to have a single
Speaker A
model, but now we go for a mixture of expert model, and we do encounter another problem of routing to particular expert.
Speaker A
This will be also fully asynchronous here. This will be fun from a mathematical point of view. So, we will have also have the same problem here on number four with the reward integrity here.
Speaker A
Then we will have something that is real nice here, some task filter. And you see, we are working here also with open hands, with cloud code.
Speaker A
We have our coding harness structure and our harness ports for the communication. But now in this second paper, they go a different way, and I think they go a much I wouldn't say a much more intelligent way, but a much more advanced way.
Speaker A
What we are really talking about is think about if you read both papers. This is the new blueprint for the gigantic RL, but not the old one. This is now really the harness gigantic RL, now.
Speaker A
It is kind of the instruction manual for taking an LLM, putting it in a real software environment in your company, and allowing it with its harness to safely teach itself through trial and error. Going here through all the
Speaker A
communication with the environment through the harness filtering. And the deterministic harness optimization of the data communication to the context window of the LLM.
Speaker A
So, if the preprint from yesterday laid the mathematical groundwork for separated agent training, this now here, our legal RL, is the battle-tested industrial-scale engine built by Huawei to handle the what do you call horrific reality of the actual
Speaker A
construction site that it is really deployed here in an industrial scale. And you know what? They found, you know, it is not just enough fixing the tokenization mathematics. This is nice.
Speaker A
This is a next step, but we also know what is the step after that one.
Speaker A
Because when deploying complex coding agent that use here QN 3.5 mixture of expert model, or you can use any other huge model, they all more or less mixture of expert model, no?
Speaker A
To deal with real coding environments, here software repos, three new chaotic thing are going to happen, and Huawei inspected and reports now about the next problem we going to encounter, the next three chaotic things we going to encounter, and you are not going to
Speaker A
believe it. But this will be continued in my video tomorrow. I hope you enjoyed it. Hope you had a little bit of fun.
Speaker A
Go read both papers, come back, and we talk about it in detail tomorrow. And yes, there is also two additional papers waiting for you, because this is really the leading and bleeding edge.
Speaker A
And this is where research is happening right now.
Topics:Agentic Lightningreinforcement learninglarge language modelsharnessed RLmulti-agent systemsMicrosoft researchUniversity of Edinburghtokenizationprompt engineeringmixture of experts
