Speaker A
Hello, community. So great that you are back. Let's talk about this, and you might immediately say, "Hey, wait a minute. What's happening here with the title? There is something wrong with the tokenization of this text." And you are
Explores how API proxies solve reinforcement learning challenges in AI agents trapped in proprietary software harnesses.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- Proprietary harnesses block direct RL training access to LLMs, breaking classical RL assumptions.
- Harness-induced modifications corrupt the RL feedback loop, causing ineffective or hallucinated learning.
- API proxies can isolate RL trainers from harness complexity, enabling effective RL on LLM cores.
- Co-evolution of LLMs and harnesses is crucial for future agent development and training.
- Harnessed agentic reinforcement learning is a promising new approach to address these challenges.
What the video covers
- Reinforcement learning (RL) struggles when training large language models (LLMs) embedded in proprietary software harnesses due to limited access and tokenization issues.
- The harness acts as a filter between the RL engine and the LLM, disrupting the direct observation-action-reward loop essential for RL mathematics.
- Traditional RL was designed for clean, closed environments like games, not complex agent-harness interactions.
- The harness modifies, filters, and compacts communication, causing RL training to break down and leading to hallucinated or corrupted learning signals.
- A deterministic harness is beneficial for controlling LLM hallucinations but complicates classical RL training.
- The video discusses the concept of placing an API proxy between the RL trainer and the environment to restore a clean training signal.
- This proxy approach creates a physical wall that isolates the RL engine from harness-induced noise and preserves the integrity of the training data.
- The presenter references recent advances and open-source projects tackling these challenges with hybrid asynchronous RL systems.
- The video is an introduction to the emerging field of harnessed agentic reinforcement learning, aiming to solve these RL-harness conflicts.
- The content highlights the importance of co-evolving LLMs and their harnesses for effective agent training.
Chapters
- 00:00Introduction and tokenization issues
- 01:11LLM and harness architecture overview
- 02:12Challenges of training LLMs with proprietary harnesses
- 03:11Reinforcement learning basics and limitations
- 03:58Harness role in agent environment interaction
- 05:02How harness disrupts RL training mathematics
- 07:31Deterministic harness benefits and RL breakdown
- 08:37Classic RL approaches and their failure
- 09:35API proxy solution for RL-harness problem
- 11:32Emerging approaches and harnessed agentic RL introduction
Full Transcript — Download SRT & Markdown
Speaker A
absolutely right. And exactly this problem was holding us back for two years, but not any longer. Welcome to my new video. So, the question is simple. Hey, how do you use reinforcement learning or training here to train now here the
Speaker A
brain of an agent? This is, you know, our LLM. When this particular LLM is trapped inside a rigid external software harness that we do not have access to modify or we cannot control it. What you go here with a software harness that is
Speaker A
a proprietary harness, no? You could just kind of override the code or modify this code base. But you know that you have to work with an agent, not only with an LLM. So now you have a problem if you want to train this LLM at the
Speaker A
core of the agent, what you do? Because look at this. You have here, in the middle we have our beautiful structure here of our tensor layers here. This is here the architecture of a transform architecture. This is an LLM,
Speaker A
the blue in the middle. And then we have a huge sphere around this LLM and this is here our harness structure. And if you've seen my last video here, exactly this title here, the co-evolution of LLMs and harnesses, I talked and I
Speaker A
showed you here exactly how to do the co-evolution of both parts of the system.
Speaker A
But you know that agents are not anymore here standard LLMs. And if we tackle this here on a lower complexity level, no? We say, "Okay, so we run now this LLMs inside the agent harness." And we know that the harness is now the object
Speaker A
that manages here all the tools, the context, the control flow, which makes here the harness a critical new component. So, talking about reinforcement learning, we are not only anymore have free access to the LLMs, but suddenly we do have something in the
Speaker A
middle. We do have something a filter that blocks our access to the LLM, and therefore, our reinforcement learning mathematics will break down. And I know you are shocked now, but please don't be shocked because there's a way out.
Speaker A
So, the question is, hey, what? What if we only want to train here the LLM, the core of the agent, and what if we have not access to the harness since it's proprietary, or you don't want to spend
Speaker A
the money or whatever, and we don't want to train the harness. We just want to go here and train the LLM because maybe this LLM is here residing locally on our laptop.
Speaker A
Now, you know, reinforcement learning here was built more or less for video games, no? Not at all for anything in the time when there was a harness [snorts] or a software harness or cloud code or whatever you have harness, no?
Speaker A
So, reinforcement learning from its idea, from its build-up was perfected here in closed, clean environments like playing games like chess here, or some other Google games, Pac-Man in the easiest way, no? And the mathematics for Pac-Man is simple. Just think about it,
Speaker A
no? You have the state. This is more or less the board. Then you have the action complexity. This is if you move left, right, or whatever. And then you have a reward structure. It couldn't be easier.
Speaker A
State, action, reward, and you know exactly here how to do reinforcement learning by human feedback or with verified rewards or whatever.
Speaker A
But a modern agent uses here this agent harness. Like SWH or cloud code harness, whatever you have, you open hands, no? To navigate here on a real computer. And suddenly, our reinforcement learning training mathematic engine is blind. It
Speaker A
does not have access anymore to the environment. Suddenly, there's something in between the reinforcement learning training engine and the real environment.
Speaker A
And this is the harness. And it doesn't get to see here the board game anymore because the harness sits now in the middle.
Speaker A
So, what is the loop? And I think this is really time. Let's have a look. I know this will be an ultra-short video, so let's have a look.
Speaker A
The harness looks here at a computer screen, at the environment, reads some files, does some operation, and then writes a long prompt for the context window of the LLM.
Speaker A
And the harness sends this prompt to the LLM, and then the LLM spits out after understanding exactly what was here in this context window. The LLM spits out now an action. It has decided on something. You have to choose this. You
Speaker A
have to write this code segment. You have to overwrite this and this. And then the harness takes this code, tries to run it in its own not probabilistic but deterministic environment, gets maybe an error message, summarizes the error, deletes
Speaker A
old messages to save the memory, does some summarization or some compactification, and creates a brand new prompt. And this brand new prompt is now sent back again to the context window of the LLM. This is our loop.
Speaker A
This is the interplay we have currently with our agents, wherever you are. But do you see the problem that we have now? If you say, "Hey, we want to do a reinforcement learning. We want to have our RL engine for the core of the agent,
Speaker A
the LLM itself." Because we don't have access anymore, no? Because the RL mathematic needs to know exactly what the LLM saw in step two, so it can figure out what the action was in step three. Was it a positive reward, a
Speaker A
negative feedback loop, whatever it was, no? But we have a filter in between, and this is the harness. And the harness keeps changing here the absolute mathematical rules that we need from the reinforcement learning methodology, from the method that was implemented for a
Speaker A
non-harness structure. So, we are in deep trouble. Now, by the time the final reward now comes in in this agent structure, say the code was successfully passed, they hold the test, and we have the maximum reward achieved. The RL engine looks
Speaker A
back at the log files, looks at all the complexity that it went through in this dialogue harness, LLM harness, LLM. And now this RL engine says, "Oh, wow." Because now the history has been constantly tampered by the harness.
Speaker A
Because the LLM has not anymore any access to the environment to have a direct communication line. Everything is filtered and modified and optimized and compactified and structured different by the harness.
Speaker A
And therefore, reinforcement learning broke down. So, if the reinforcement learning engine now tries to run its calculus, its simple gradient descent mechanism, eh?
Speaker A
On a tampered history where it has been constantly overwritten and rewritten and modularized by the harness, the mathematic literally breaks down.
Speaker A
So, this means the LLM updates now its weight based on this spaghetti code of nonsense interaction between LLM and harness structure that went forth and back 50 times when there was a simple single command, a simple single library to implement on
Speaker A
a beautiful solution. So, now the LLM updates because of the harness complexity a complete hallucinated reality. Yes, this was the real harness LLM conversation, but this was just horror. This was not optimized. This was not in any way the
Speaker A
way we want the reasoning trace to behave. So, you see the harness determines now exactly how an agent, we're talking here about the core of the agent, the LLM, observes its environment, acts over long horizons, and recovers from failures.
Speaker A
And this is a good thing. And this is beautiful. We built the deterministic harness exactly for this to counterpart here the craziness here of the observation here of the LLM, and whatever hallucination we encounter with an LLM. And this is great, but now, if
Speaker A
you just want to have a classical reinforcement learning mathematics implied here on an agent with an active harness, the mathematics breaks down.
Speaker A
We cannot do it. And you may say, "Okay, but wait a minute. We had a way, you know, the old way, you know, the classic way in the Wild West, you know, when we all were cowboys, you know?"
Speaker A
So, you couldn't just plug in an agent into the RL training engine, yo
Speaker A
sequences, we had to implement here this particular loop here of the agent. So, this simply means we had to rewrite all the core logic here of our agent in PyTorch, like I did hundreds of years ago, forcing it simply to a abide here
Speaker A
to the strict format and everything that the trainer code demanded here because we wanted to have in reinforcement learning.
Speaker A
So, and you remember if you have been born at this time here, there was a not so old way, you know, and this was what we called the proxy-based approach.
Speaker A
So, usually a harness uses your agent, and this is programmed to send internet requests here, let's say to API.openai.com, you know, to get your LLM brain to answer back you on its particular context window, you know.
Speaker A
And in this new, if you want, proxy-based you change your one line of code in the harness, and you tell it instead of sending the request to OpenAI or whatever you have in Tropic, send your request here to my local proxy. And
Speaker A
this was something that we over the months understood the benefits of this, you know, because this proxy acting now as a fake, let's say, OpenAI server is secretly wired directly into the RL training engine.
Speaker A
And nothing beautiful happened, you know. So, what's what now the loop? The loop was simple. The unmodified harness sends out a prompt to the proxy. The proxy forwards the prompt to the RL trainer. The RL trainer passes it
Speaker A
through to the LLM wherever it is being trained now. It generates now the LLM a beautiful decision, a response, write this code, and sends it back through the proxy.
Speaker A
And then the proxy sends it to the harness. So, the harness has no idea it was talking to a training engine. It just thinks it was talking to ChatGPT or OpenAI or Anthropic or whatever you have. So, you see, this was what we
Speaker A
sought here 2 years ago. Hey, this is the latest. No, this is the best here that we have.
Speaker A
And we were so happy, no? Said, "Hey, this means you can grab any agent off GitHub, Cloud Code, OpenHands, Open Flow, whatever you want, train an LLM to be better at it without changing the agent's code. You just point the agent's
Speaker A
API URL to your trainer's proxy, sit back, and let the Mathematic handle the rest." And it was beautiful.
Speaker A
Until we found out that it is not working because the mathematical broke. Mathematical methodology broke down.
Speaker A
And we thought but we had found a solution no? We could decouple the messy agent software from the complex training mathematics here for reinforcement learning, no? Allowing them to finally train autonomous software engineers at scale no?
Speaker A
And there were such beautiful papers here. Remember here Rhol or a real asynchronous Rhol or Slim, no?
Speaker A
Generating here this methodology, no? And I don't know if you remember 2024, this was the paper, no? Hybrid Flow Rhol, a flexible and efficient reinforcement learning by human feedback framework no?
Speaker A
Was by ByteDance, no? Rhol Volcano Engine reinforcement learning for LLMs. Was the open-source version of this Hybrid Flow that I just showed you. Or we had a real asynchronous reinforcement learning system for for a language reasoning eh?
Speaker A
By Tsinghua University on group, and this was just great. I mean, just look at this. 7 hours ago, they optimized this for Q&A support using language, eh?
Speaker A
So beautiful eh? And they called it here the reinforcement learning bridge for LLM based agent applications. And updated several times here, beautiful.
Speaker A
With the optimized medical framework. And then SLIME, eh? SLIME, I don't know if you're familiar with it, the LLM post-training framework for reinforcement learning scaling, providing here two core capabilities, eh? The high-performance training and the flexible data generation, eh?
Speaker A
So, in general, we sought here for quite some time that this proxy solution was, let's call it the holy grail for AI researcher to really penetrate through here our harness configuration, this complexity, and really say, "Yes, we're tunneling through here and a quantum
Speaker A
tunneling effect here directly to the core of the agent, to the LLM, and we have access to this." Because the proxy created a clean physical wall, eh?
Speaker A
Our machine learning researcher colleagues here put their PyTorch code here on a secure cluster for the TPU for the trainer, and their let's call it so, not so perfect, messy, chaotic agent software goes on a separate cluster somewhere.
Speaker A
And you had your execution summary, eh? But, doing this and learning this and experimenting with this, we find out that there is a butt.
Speaker A
The reinforcement learning mathematics does not work on text messages. And you said, "Of course not. It works off exact sequence of tokens, eh? Because we do have a specific tokenizer algebra, and we choose a specific tokenizer to do
Speaker A
exactly this, given our complexity, given our domain specificity, and they given a lot of other things." And yeah, it was a sequence of token, and it relies on an unbroken chain of cause and effect.
Speaker A
And you know what? By putting now an API proxy between the trainer and the environment, the machine learner trainer was suddenly blind. And you said, "Yeah, of course, no?" It just say here this flourish independent text messages hitting here its API.
Speaker A
And we still thought, "Yeah, but it's it's a linear sequence, no?" Until we learned it is not.
Speaker A
But because it's an API, now the harness treats it like a normal internet communication, no? And the harness receives the API AI response, does some work, and then creates a new prompt, and sends it back to the AI.
Speaker A
And you might say, "What? Wait a minute. What? It creates a new prompt and sends it back to the AI?" But this harness modified something, no?
Speaker A
It altered here the linear sequence. It brought complete chaos here into our beautiful machine learning mathematics, no?
Speaker A
Because sometimes this harness decided to add some extra space or maybe a comma or maybe use a different word. Or think about it. Anthropic here is watermarking its text, no? And if they have no specific changed probability distribution for
Speaker A
certain tokens where their token is here their signal token that this is a watermark, this is now changed. So, we cannot use this Anthropic watermarking for reinforcement learning or any other additional space or any other token sequence, no? That is not according to
Speaker A
the original token, no? Or maybe the harness will delete old message just to save some memory or compactify it, no?
Speaker A
But then the real the real heavy thing, no is tokenization, like I showed you the very beginning of this video, no?
Speaker A
Imagine you have the word having, no? But because the API text is reformatted by the harness, the AI tokenizers are now acting on this thing again. And now they tokenize it maybe differently than they did just milliseconds ago. Because now having as
Speaker A
a as a word as a human English word is becoming now two different tokens like H and a token having. Or it becomes here two other tokens like have and another token ink.
Speaker A
Or you have another tokenizer that is not compatible and it changes here completely eh?
Speaker A
So, you have multiple effects that crash here your linear sequence of tokens that is so important for the classical reinforcement learning mathematics. It does not work anymore.
Speaker A
But, this means we're going [laughter] This here is is an image here of our reinforcement learning mathematics genius. This is here the algorithm here that you see here with the big eyes here. It looks at this and says, "Help,
Speaker A
I'm panicking. I don't understand the world. What is happening, eh?" Says, "Wait, these text tokens don't line up like I used to be in the classical reinforcement learning, eh?
Speaker A
Did you I take your one long continuous action or is this simply 15 separate unrelated task, eh?" It lost here the sequence.
Speaker A
So, this means all what we applied up until now from all these beautiful papers that I showed you at the beginning of this video, eh?
Speaker A
We have a massive massive problem. It's not really working. And I might say, "But, why? It it was such a beautiful idea." Yeah, but if you look closer now with the knowledge that we have from experimenting with this,
Speaker A
you can say now simply throwing a proxy between two things that need to communicate and have full access of the information eh?
Speaker A
So, simply throwing a proxy between two things is not the scientific breakthrough we thought it was because it turns out it's just software engineering and we forgot half of the data in this new software engineering.
Speaker A
So, this is the sad sad state that we are currently in, eh? And at exactly at this moment now now we have something that is absolutely fascinating that is happening in summer time and I have thought hey maybe today
Speaker A
I do not produce a video because maybe I'll go to the beach and I have a little bit fun here in summer time of the northern hemisphere. I hope you could have a beautiful skiing here in the southern hemisphere of our planet. But
Speaker A
now now we have this. We have four new preprints that were published today. Today is August 18th for me here in Europe.
Speaker A
Which combined now in a beautiful way. These four papers were published here without knowing from each other. But reading this paper I found four papers that that just beautiful.
Speaker A
That they build a new coherent framework exactly to answer this. So it seems that we have here a new approach.
Speaker A
And you might know it because it's around here the idea for quite some time some weeks now. Harnessed agentic reinforcement learning. This is now exactly tackling here the problem that I described in this video. And this is just the intro video. This is just to
Speaker A
bring you up to speed. This is just that we have here the same level of knowledge what we're talking about in the next two videos.
Speaker A
Because what we want is a harnessed agentic reinforcement learning methodology. A new mathematics that we can apply now. And the first paper in this will be tomorrow. You see this is here published today for me August 18th, 2026 by Microsoft and Fudan
Speaker A
University and Tsinghua University and University of Edinburgh. Agent lightning the next version. And this is here on the path toward this harnessed agentic reinforcement learning. But yeah, I would like to stop here with this introduction because tomorrow we can have a look at this
Speaker A
paper and the very next paper that I think goes with this paper, builds on this paper and goes even further than this Microsoft paper.
Speaker A
But yeah, I think have a look here. Whatever you know, just get you a little bit familiar, have a look at my last video, and then tomorrow we can really start. I hope you enjoyed it. I hope you
Speaker A
had a little bit of fun. Maybe there was some new information for you, and I hope you are interested in the next video.
Topics:reinforcement learninglarge language modelsLLMsoftware harnessAPI proxyagent trainingtokenizationmachine learningAI agentsharnessed agentic reinforcement learning
