Speaker A
Hello community. So great that you are back. Let's talk about training the harness, and we have a new methodology, a new technology. Let's have a look. So just to be absolutely clear, we are talking about a new method for
Explores a novel methodology for training runtime harnesses in AI agents, focusing on failure attribution without retraining LLMs.
Key Takeaways
- Training the runtime harness can improve AI agent reliability without modifying the LLM itself.
- Failure attribution is critical to avoid misdiagnosing LLM hallucinations as harness bugs.
- Batch-level cross-instance failure aggregation enables efficient and generalized harness improvements.
- Automated harness repair must be validated to prevent overly restrictive or harmful patches.
- The proposed methodology significantly improves harness performance compared to naive self-evolution.
What the video covers
- Introduces a new evidence-based runtime engineering method for control harnesses in AI agents.
- The method focuses on training the harness, not the underlying fixed LLM, avoiding reinforcement learning or gradient updates.
- Builds on prior 2025 research about multi-agent iterative refinement and failure aggregation.
- Defines the failure attribution problem to distinguish between harness errors and core LLM limitations.
- Highlights risks of naive self-correction that may introduce overly restrictive harness rules.
- Explains the harness role in managing context, planning, tool invocation, state sync, and governance around the LLM.
- Demonstrates the methodology’s efficiency gains via batch-level failure aggregation and cross-task diagnosis for generalization.
- Shows that automated harness evolution must be carefully validated to avoid degrading performance.
- Presents experimental results indicating improved accuracy with the new failure-driven cooperative refinement (FDCR) approach.
- Warns about potential pitfalls of reactive self-modification and emphasizes the need for systematic failure pattern analysis.
Chapters
- 00:00Introduction to harness training and new methodology
- 01:42Distinguishing harness training from LLM training
- 03:15Harness components and precise terminology
- 05:09Example: failure attribution in reservation cancellation
- 06:46Risks of naive self-correction in harness repair
- 08:40Experimental results and accuracy improvements
- 10:24Failure pattern detection and root cause analysis
- 12:22Automated harness evolution and validation
- 14:36Summary, warnings, and future directions
Full Transcript — Download SRT & Markdown
Speaker A
evidence-based runtime engineering of our control harness. New study, September 10, 2026. This is here by the Chinese Academy of Sciences, the University of Chinese Academy of Sciences, Beijing Institute of Technology, the Yang Delta Region Institute of Chinua University,
Speaker A
and my goodness, a lot of very, very intelligent people, and they give us a new paper: efficient and effective training of runtime harnesses for LLM agents, and they discover something, they discover something beautiful.
Speaker A
Now the core idea is to adopt the batch-level cross-instant failure aggregation paradigm to jointly analyze failure evidence for multiple task instances and further introduce a failure-driven cooperative refinement methodology to diagnose failure causes and iteratively refine the harness modification and the
Speaker A
specification. And if you say, "Hey, this is it, thank you." Well, this was a short video, enjoy it. If you say, "Hey, can I know a little bit more?" Well, of course.
Speaker A
Now we remember in 2025 we had a study already here by UNCC Chapel Hill on multi-agent iterative coarse-to-fine refinement for reasoning, and they had LLM reasoning can be improved by scaling test-time compute with aggregation. So
Speaker A
this means simply generating multiple samples from our LLM and then analyzing them and aggregating over those multiple samples, trying to find causal patterns.
Speaker A
Now this new paper has a beautiful new main solution that builds on this paper from 2025. So just to be clear, this new preprint does not train the LLM, and this is not reinforcement learning. We are here really in the harness. So it trains
Speaker A
the control harness that is wrapped around our fixed tensor LLM by deciding now which observed failures are genuine harness effect and which merely expose some limitation here by the current LLM because maybe you work with a 3 billion
Speaker A
or 7 billion free trainable parameter LLM locally. So what they came up with is that we have now a failure attribution problem, and if you want, this is the real scientific contribution of this new paper. So batch processing will
Speaker A
provide the efficiency gain while a cross-task diagnosis I'm going to show you in a minute is really intended to provide here the generalization gain of this new methodology. Now, if you're new to AI, welcome. You know, we have at the
Speaker A
core of an agent, our LLM or a large language model, vision language model, and then we have a harness. And the harness has everything from skill, memory, runtime state, governance, tool access. Great. We even have here an LLM,
Speaker A
then a coding harness like here or cloud code or whatever Codex from OpenAI. And then we even had a video where we talked about a harness of harness optimization.
Speaker A
But in general, we just have here an LLM core. Then we have here a runtime state here plus a skill memory here with retrieve and execute and an outer layer of our control layer for governance and the execution harness that will do the
Speaker A
authorization and the verification, of course, the interfaces to the external world for tool use in this environment and the feedback. So therefore, if we have to be precise in our terminology, in our notation, what does training a runtime harness as the author specified
Speaker A
actually mean in this particular paper? Training is potentially misleading here because we also train an LLM. But here this training has nothing to do with gradient descent. There's here no backpropagation, no reinforcement learning update, no modification to the LLM
Speaker A
tensor weights. We are training a runtime harness that is not a neural network. Now you know on my channel I have a simple definition: what is an agent? This is simply here a harness here, a runtime harness surrounded here,
Speaker A
and we have at the core our fixed task LLM or VLM, and we have here our policy pi data. So therefore TA is here simply my abbreviation for the mall or just put M in there instead of TA, and this is it.
Speaker A
Now the harness, what does a harness really control within an agent? It controls here the context and message management, the planning and the task decomposition for the complete agent, all the tool invocation, the propagation of tool results, the state
Speaker A
synchronization, the confirmation before any side effects, and stopping and recovering behavior. So therefore, harness training in this paper has to be redefined and our terminology of this YouTube channel as an automated program repair and optimization using agent
Speaker A
execution traces as evidence. So this preprint starts here from an already human-optimized harness configuration. So this is we are here typically in a post-training methodology for harness optimization, not in harness creation from scratch.
Speaker A
Simple example: suppose you have an L agent that cancels a reservation without obtaining here confirmation. So this means the horn is supplied to the correct instruction, current state, and confirmation request, but the core LLM of our agent, let's say it was a Q138
Speaker A
billion, made a mistake. Maybe it was a hallucination. Maybe it doesn't know what to do. Somehow it ignored them. So this means here the core of the agent, the LLM model, made the mistake. Or we have the other option. The harness
Speaker A
failed to preserve here the confirmation state of our complete system or allowed here the cancellation tool to execute without checking its precondition. No verification. So this means here the runtime harness made the mistake. So normally you, I just type in
Speaker A
"Hey, correct this code," or "Do this better," or "Integrate here another mathematical methods here to optimize the code."
Speaker A
But what is now optimized, the model or the harness? And now here Jingua and all the beautiful universities tell us here there is now a new danger that you have to be aware of: the danger of repairing every failure. The moment the failure
Speaker A
occurs and you have here you see an error on your screen and you say, "Correct it," or "Self-corrected," or "Self-code here a solution," because if you have this serial self-evolution system, this might respond now to a cancellation error by
Speaker A
introducing now a global rule as an agent. You have cloud code and the cloud code says no. Okay. So therefore the solution to the problem is never allow the agent to cancel a reservation, which is in itself nonsense because
Speaker A
there are cases that we will have to do this. So this means this particular patch that is now generated by cloud code fixes here this particular triggering case, but on the other hand it destroys here some real-time cancellation
Speaker A
that might be necessary behavior that is here essential in other cases. That means the harness has converted one model error into a permanent software restriction that is implemented in the harness. And yeah, the harness is our control runtime layer to reduce the
Speaker A
hallucination of the core of the LLM. But now it made a mistake. Now the preprint, the authors of the preprint call this a model-specific accommodation. So this means the harness changes itself to compensate for a particular LLM model weakness even when
Speaker A
the harness architecture was not fundamentally defective. There was no error in the harness. It was maybe a hallucination by the LLM, and now the self-coding harness, if you want, or an external coding engine corrects now here the runtime control cycle of the
Speaker A
harness. This might be helpful, but it might incur more problems, and I want to show this. So in short, this means we have an AI, beautiful here, cloud code, now an immediate solution to the problem that is now happening on this little LLM
Speaker A
and the current harness configuration on a single problem. But it turns out we were AI cloud code here, we are in the wrong dimension of our multi-dimensional solution space. We are in the wrong space, in the wrong mathematical space
Speaker A
because we do something here in not the neural network architecture, and we are at the wrong locati...
Speaker A
Yeah, there are a lot of tests. I just want to show you here one accuracy average. So if we go here for a certain configuration here the direct configuration we have an accuracy of below 40%. If we go now for a human
Speaker A
augmented error correction and a debugging and an optimization every single time we increase now our average accuracy to 51.67%.
Speaker A
Great. And if we say now to the cloud code, hey do a self evolution find now or debug it here and find new solution cloud code it immediately here in a serial sequence. Whenever this happens the self evolution aentic
Speaker A
self-evolution reduces now the overall system accuracy from 51% to 46%. So you see this is happening. We are correcting or claw code is correcting here for something on a level that is the incorrect level to correct this particular mistake. Yeah. And you see
Speaker A
here the new solution here with and without an ablation study here. So we will jump from 51 to 59%. So let's have a look what is happening. What is the idea? At trust we generate we we [snorts] see that a self-improving
Speaker A
harness that you might have with cloud code or whatever code configuration you like become worse inside the training case. So we decreased from 51% to 46%.
Speaker A
So you see great. So EI is analyzing here the a mistake. Yeah. And it goes here in a control runtime environment to correct here global parameters that might be local. Now the new methodology simple you know I like simple ideas. It
Speaker A
ask hey does the same underlying failure mechanism that we have that we see in the trace recur across different tasks.
Speaker A
So you want to allocate hey is this here a mistake by a lamp but a neural network configuration because we only have a three billion model or is this something we have to take care in the harness structure.
Speaker A
Now simple example consider surface level failures. No after changing a flight the agent reasons from an old itinerary array or after refunding an order it reasons from an old account balance or after replacing an item it reasons from an old order status. It
Speaker A
doesn't update or whatever there is. Right? You see semantically the task are different. But if you do here an error analysis here and you say hey analyze what happened you will see that structurally the failure is identical in
Speaker A
all three cases. Let's say the harness code does not refresh some environment state after a state changing tool call.
Speaker A
So where do you correct this now in the harness runtime or you really say hey I have to do this here in the reconfiguration here of the tensor weight structure of the LLM.
Speaker A
So if you want the scientific core of this new methodology is simply do not learn from the identity of the failed task but do learn from the mechanisms that is shared by several failures and learn collect several failures. Try to
Speaker A
detect pattern in this failure manifold and then analyze what the failure is before you try to come up with a code solution.
Speaker A
Yeah, this has a new methodology. I do not like the name of this methodology.
Speaker A
At first I cannot pronounce it. You know, English is my third language. But this has also some interpretation in biology. This is something I do not like. It's kind of a morphing here of a particular insect. And I say gross. No,
Speaker A
I don't use this particular name. But I want to show you train the harness.
Speaker A
Beautiful. So we have incoming failures that you have still loop or missing confirmation or premature stop or malfform tool calls. We have here a lot of code coming in. We analyze it here and we need a lot of data streams
Speaker A
because we need to find certain patterns in these data streams and then we say hey quite a lot of them is one of noise.
Speaker A
We can ignore it. we just throw it away. But then we analyze it. We find new patterns and we say okay now we understand here the exact nature what happened and now we understand do we have to control here a particular
Speaker A
element in the LLM or can we find here a particular solution for the harness. So you see we have here frozen LLM. So here we go only and do a harness modification. The harness is this beautiful structure around our blue LLM,
Speaker A
the neural network that you see here beautifully in 3D. Now, how you do this? Now, you are not going to believe it. We're going to use AI for understanding here or detecting here this particular pattern. We use here four agents, an analysis, a
Speaker A
critique, an engineering, and a moderator agent that will do the thinking and the pattern recognition for us.
Speaker A
This new methodology has four stages. So at first we have to execute a batch of training task. We have to generate the data. Now stage two having all this data stream we have to extract here the failure evidence. Whatever was not
Speaker A
working we collected. Stage three we now group failures by patterns by a particular mechanism. We want understand hey what error is happening and what is the root cause of this code error. and stage four. Now it gets interesting
Speaker A
because then we have this particular new diagnose here where we have four agents in a communication and in a code correction then four particular states.
Speaker A
And if you are a subscriber to my channel and you really enjoy this channel and say hey again the last three videos were about an abstraction mechanism to find patterns and now again we are talking more or less about an
Speaker A
abstraction that we have to detect failure patterns. Is it really so simple with the I machines? Absolutely. You can laugh about it. It is that simple. We always reduce the complexity to detect failure patterns, detect patterns, detect root causes. And yeah, so we are
Speaker A
back to the main topic. Yeah. Stage three, what is it? Failures are grouped into patterns. And you say, wow, amazing. So we go to stage four. So here we have this diagnosis of the batch and this is here the failurdriven
Speaker A
collaborative refinement and it uses here our four diagnostic rules and I already told you those will be for agents. What a surprise with AI. So you have it here we have the role of an analyst of a critique of an engineer and
Speaker A
a moderator. Now you will see that in the paper we have a backbone LLM that is doing here sharing here all four roles.
Speaker A
So maybe this is not the best configuration but we go with the paper. So what is now the responsibility of the first the analyst. It infers possible harness deficiencies and proposes minimal changes and then we have a critique agent that says hey wait a
Speaker A
minute wait a minute we [snorts] have to test to do some test whether the evidence supports those changes and identify overreach or regression. We do not accept the result of the output of the analyst. I will critique it and send
Speaker A
it back if I don't like. So we have a mini loop between analyst and critique.
Speaker A
But if it goes on, we go now to the engineering. So this tracks now the agreement, disagreement and unresolved implementation requirements and it builds something and says okay I have a solution how I can build this and then
Speaker A
we have a moderator and the moderator guess what does here the assignment the discussion leads the discussion. So we have one structured modification specification that we come up out in this particular phase. So just to be clear, the critique explicitly examines
Speaker A
the risk. Risk can be multiple triggers that are too broad or blocking some legimate behavior or violating the runtime contract or the regression on previous successful task or modifying more of the harness element that are absolutely necessary. And you got it.
Speaker A
And then a separate coding agent determines how to implement it. Guess what? Yeah, use whatever you like. Cloud code. So you see this is however let's do a little bit of a limitation analysis a rolebased prompting. Yeah, rather than
Speaker A
a newly trained diagnostic LLM model or a general AI agent model because here in this paper the orus use here deepseek version 4 pro as the backbone for all the different or here the coding agents and you got it and the roles here of our
Speaker A
new methodology. So they are not independent sources of evidence. So whatever your monetary options allow maybe go here not only with one version with one deepseek but yeah you know optimize this. Okay. So just to make sure the coding agent has now an input
Speaker A
and the input are the moderator specification. The current harness source and the relevant execution evidence are now provided to the coding agent and the coding agent produces now with its inherent intelligent quotation mark a new harness optimization. So we
Speaker A
go for a hornness at i minus one to a harness hi where we have simply an edit function that is based on the intelligence of our coding agent of cloud codeex whatever you prefer. Great.
Speaker A
So here we have now our four little fellows here. This is here the analyst agent in blue. This is the critique agent in red. This is the engineering.
Speaker A
He does something so he's green. and then check for the specification and do for all the logical uh constraints. Here we have the moderator that is also coordinating here the interaction of the agents and then we have something where
Speaker A
we say okay this is now a new harness patch and then we can feed it into the harness evolution and we have here a dynamically looping harness optimization. Great. Remember we do not modify the LLM itself. So you can work
Speaker A
with a cloud-based LLM, a propriatory LLM. And we already talked about if you want in another video, I showed you how to do then integration of this new knowledge into the parametric knowledge and therefore modifying then the LLM
Speaker A
weights. But in this paper, the LLM weights are locked. We have a cloud-based LLM and this is it. We optimize the harness object and the harness complexity only. and MSA beautiful. So the task LLM does not edit itself. You have in this paper a Q138B
Speaker A
this is the task model and then as I showed you we have a deepseek version for pro as an external diagnostic and coding model. So if you say self evolution careful with this particular technical term should be interpreted as
Speaker A
an automated harness evolution not literally a self modification by the same underlying LLM and then simply if we have now a new harness configuration hi this is accepted accepted only if we have some test we see hey this aggregate training
Speaker A
score is really better than before at hi minus one so therefore for this new harness candidate will become our actual harness. Beautiful. So we have an iterative improvement in a sequence and then yeah after three after at most
Speaker A
three candidate generation rounds the final harness is frozen and again evaluated on held out task and other LLMs just to make sure we are not locked into a particular LLM harness configuration.
Speaker A
Now for the task. For the task you have to find tasks that are valuable for this training process. Not simply because it is too difficult. It is valuable when it exposes the following elements. Some previously unseen interaction structures
Speaker A
or a complete new execution path or some hidden harness deficiency that were previously not discovered. So therefore, choose the quality of your harness training data in your particular domain with your particular complexity carefully.
Speaker A
Okay. Yeah. We the authors did this for a crossmodel transfer analysis. So they started with a Q138B, they went to a 14B and then to a 32B and even with an oldfashioned llama. So great. If you want to see this, I just want to show
Speaker A
you here the 14B from a Q13. And in this benchmark, we look at the accuracy, the average accuracy in percentage. So you see the direct configuration provides 16 percentage points. The human augmented jumps this to 38% and the pure self
Speaker A
evolution if we do not apply the new methodology, if we just let a self-evolution mechanism take place, falls from 38% to 23%. Great. And it is only with this new methodology here with FDCR that we jump from 38% human
Speaker A
augmented harness configuration to an optimized AI modified FDCR harness configuration at 43.3% accuracy average. Beautiful.
Speaker A
Here are the last words by the orders. The orers tell us the experiment across multiple LLMs and benchmarks showed that this new methodology achieves up to so this is the best case scenario less than two times each speed up in the harness
Speaker A
training compared with existing harness while also at the reasoning accuracy here is 18.5% not percentage points. So this means if we start here at I don't know 10% and we increase not by 18 percentage point but only by 18%.
Speaker A
You know we are not really making some significant jump. So careful and this is here all the best case scenario of all the tested methodology that we have presented to you. So you might say hm 18% is not really impressive but maybe
Speaker A
you accept it. Let's talk about the limitation. You know there are always limitation at the end of my videos.
Speaker A
It does not report or it does not show a beautiful correct detailed mathematical annotation protocol or some raw decision counts or some interrator agreements or it is more or less um an evidence shown.
Speaker A
There is some indicative evidence shown presented in a beautiful way but it is lacking here the overall deep mathematical understanding of why it is happening what it is happening in detail and present to us in a in a complete
Speaker A
mathematical way we rather have hey this is the evidence look at it so we have indicative evidence yeah okay self evolution I already explained a little bit relies here on If you want real external stronger optimizer, go with the best coding LLM
Speaker A
you can find, cloud code or codec or whatever you like. If you go here with less powerful models, those self evolution of the harness elements might not be as powerful as you would hope for.
Speaker A
Yeah, Qent task model is not modifying its own runtime. You have your Deepseek version 4 Pro that performs here the diagnostic and the editing through open code in the paper. What are the inside of the paper? I think what we or what I
Speaker A
see is that the harness evolution is now moving away from hey how do we patch immediately this code error this failure of our code execution. And we should first ask hey is this failure here really some concrete evidence that the
Speaker A
harness is wrong? Does it really happen? Often can we really say we have identified the failure abstraction, the failure category, what is happening and then only then we can move and say okay this is either an LLM based
Speaker A
hallucination or an error or whatever or this is something a particular pattern that we can detect and that we can correct in the harness configuration.
Speaker A
So careful what cloud code is proposing here on solution because you might insert code at a complete wrong location. So this means now in general this is a meaningful paper that shows a shift from some immediate reactive self
Speaker A
modification code optimization here in a linear sequence to hey wait a minute let's analyze more failure data. Let's see if we detect patterns in this failure data set. So we move to an evidence-based runtime engineering and not an immediate code optimization that
Speaker A
is blind to the systemic evolution here of this failure system. I hope you liked it. Therefore, evidence-based runtime engineering. We started the video with this. I hope I explained it in my understanding in very simple terms.
Speaker A
There's a lot of additional data that you find, numerical test evaluation data you find here in this paper. Read the paper yourself and I hope to see you in my next video.
Topics:runtime harnessfailure attributionLLM agentscontrol harness trainingfailure-driven refinementautomated program repairAI agent governancebatch failure aggregationlarge language modelsharness optimization
