Speaker A
A language model is writing a program. Three lines in, it's already named six calls to other models, six requests, each one taking seconds, and nothing happens. The harness just watches. It reads the tokens as they arrive. It can
Explores why agent harnesses wait to execute tool calls due to JSON design and how programmatic tool calling with code actions improves efficiency.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- JSON-style tool calling causes unnecessary waiting and token overhead.
- Programmatic tool calling with code actions significantly reduces token usage and improves efficiency.
- Speculative execution enables running tool calls early, speeding up agent responses.
- Leading AI labs have converged on code-based tool calling as the standard.
- Handling side effects carefully is crucial for safe speculative execution.
What the video covers
- Most agent harnesses wait for the entire model generation before executing tool calls, a behavior inherited from JSON-style tool calling.
- JSON tool calling was simple and easy to validate but caused inefficiencies by delaying execution until the full JSON object was generated.
- OpenAI introduced function calling in June 2023, which was widely adopted by other labs, establishing JSON as the universal tool call format.
- This design leads to large context windows filled with tool definitions and repeated copying of outputs through the model, increasing token costs.
- Cloudflare, Anthropic, and OpenAI have since shifted to programmatic tool calling where the model writes code to call tools, reducing token usage and improving speed.
- Code actions allow running tool calls as soon as they are fully parsed, enabling speculative execution and reducing wait times.
- Alex Jiang from MIT introduced recursive language models and speculative programmatic tool calling, using a shadow interpreter to safely pre-run calls.
- Speculative execution can speed up tool calls by 1 to 1.7 times, though it requires careful handling of side effects and external state.
- The transition from JSON to code-based tool calling is now widely adopted by leading AI labs, proving code as the superior action space.
- Despite improvements, speculative execution involves trade-offs around uncertainty and potential external side effects.
Chapters
- 00:00Introduction to agent harness waiting behavior
- 00:47Origin of waiting behavior in JSON-style tool calling
- 01:32OpenAI's function calling and widespread adoption
- 02:17Token costs of tool definitions and context window overhead
- 02:56Limitations of JSON tool calling and model inefficiencies
- 03:40Code actions and improvements by Cloudflare and others
- 04:36Programmatic tool calling by Anthropic and OpenAI
- 05:29Alex Jiang's recursive language models and speculative execution
- 06:26Mechanics and benefits of speculative programmatic tool calling
- 07:13Challenges and trade-offs of speculative execution
Full Transcript — Download SRT & Markdown
Speaker A
see all six calls sitting there fully specified, and it holds every one of them until the last character of the program lands. Only then does it run them, one after another, while you wait.
Speaker A
That isn't a bug in anyone's code. That's how almost every agent harness shipping today behaves. And this August, a PhD student at MIT wrote one sentence explaining where the behavior came from, which is why this video exists. His
Speaker A
name's Alex Jiang, and here's the line from his own blog, word for word. "Most harness designs will wait for the entire model generation to complete its generation before executing the tools.
Speaker A
This design is likely a consequence of JSON-style tool calling, where this was not really a bottleneck." Read that second sentence slowly, because it's the whole story. It wasn't a design decision for code agents at all. Waiting is a habit, inherited from
Speaker A
a format where waiting was free. Why was it free? When a tool call is a JSON object at the end of the model's turn, there's nothing to overlap it with. By the time the model types the closing brace, it's finished generating. There's
Speaker A
no meanwhile left to use. But a program isn't a blob at the end. A program names its first tool call almost immediately, and then keeps going for thousands more tokens. Every one of those tokens is time the harness could spend running a
Speaker A
call it's already read in full. Instead, it sits on its hands. So, where did that shape come from? The 13th of June, 2023.
Speaker A
OpenAI shipped function calling. You described your functions as JSON schemas, and the model replied with a JSON object naming one and filling in its arguments. That was simple, checkable, easy to validate, which is exactly why every other lab copied it
Speaker A
inside a year. And a year and a half later, when Anthropic open-sourced MCP, the standard for plugging tools into models, that shape became the universal plug. So, that is where it came from.
Speaker A
Two bills are attached to it and both come due before your model does any work. The first one is definitions.
Speaker A
Every tool you connect ships its name, its description, and its full schema into the context window before the user has said a word. Anthropic published the arithmetic. Five typical servers, 58 tools, about 55,000 tokens gone.
Speaker A
GitHub's official server alone is 35 tools and 26,000 tokens. You're paying rent on the toolbox before you open it.
Speaker A
The second bill is the one the story turns on. In JSON tool calling, every result comes back through the model.
Speaker A
Point an agent at a 10-MB log file and the whole file lands in the context window, even when all you wanted was a count. Chain two calls and the first one's output gets copied through the model's attention purely to become the
Speaker A
second one's input. You're using a language model as a very expensive pipe. Kenton Varda, who runs Cloudflare's developer platform, put the deeper problem better than I'm going to. He wrote that making a model work through tool calls is like putting Shakespeare
Speaker A
through a month-long class in Mandarin and then asking him to write a play in it. A model has read a planet's worth of real working code. It only ever met tool call syntax in synthetic training data.
Speaker A
So, if the wrapper's expensive and the model's better at code anyway, what happens when you make code the action?
Speaker A
That question got asked early. In 2024, Xingyu Wang and his co-authors published CodeAct and the argument was almost rude in its simplicity. A model shouldn't be filling in a form. It should be writing a Python program. One action space
Speaker A
instead of one per tool with loops and conditions and variables for free. Over 17 models, they measured up to 20% higher success than text or JSON and a library called Smol Agents found the same thing from another angle. About 30%
Speaker A
fewer steps, which is 30% fewer model calls you pay for. Cloudflare made the token argument impossible to argue with. In September 2025, Varda's team shipped Code Mode, which turns tool schemas into a TypeScript interface and lets the model
Speaker A
write TypeScript against it inside a sandboxed V8 isolate. This year they pointed it at their own platform, 2 and 1/2 thousand endpoints as tool definitions, that's 1.17 million tokens.
Speaker A
As a code mode server with two functions, search and execute, about 1,000. Anthropic ran the same play twice inside 3 weeks in November 2025. On the 4th, it took 150,000 tokens of tool definitions down to 2,000 by executing
Speaker A
code against MCP. On the 24th, it shipped programmatic tool calling in the API itself, where Claude writes Python that calls your tools in a sandbox and only the final answer comes back. On a 75-tool agent benchmark, that was 38%
Speaker A
fewer input tokens on the bill with no loss of accuracy. And on the 9th of July this year, OpenAI closed the loop, shipping programmatic tool calling in its main API alongside GPT-5.6.
Speaker A
Same idea, JavaScript instead of Python, its own V8 sandbox. That launch page carries a customer number I'll come back to at the end because it's the one that actually settles this. Step back for a second because a lot just happened. 3
Speaker A
years after JSON became the universal way to call a tool, OpenAI, Anthropic, and Cloudflare have all shipped the same replacement and they agree on the mechanism. A model writes a program. The program calls the tools. Only the answer
Speaker A
comes home, which raises a question worth sitting with. If code actions were measurably better, why did it take the frontier labs until the end of 2025 to ship them? And what else are you running today because it was the correct answer once?
Speaker A
Into that gap walks Alex Jiang, and he'd earned the right to notice it. In December, he published recursive language models with his advisor at MIT, an inference strategy where a model treats an enormous prompt as an environment and calls itself over pieces
Speaker A
of it from inside a Python REPL. His repo has passed 5,000 stars. And in a system like that, the expensive thing is precisely a submodel call written inside code. So, watch one of those programs while it's still being typed. Line one
Speaker A
queries a submodel with a string. Line two queries it again. Line three is a loop over document chunks, one subcall each. Your harness has read all of that inside the first second or two of a generation that'll run for another 30.
Speaker A
It knows the arguments. It refuses to act. Speculative programmatic tool calling is the instruction to act. As tokens arrive, parse the half-finished program. The moment a tool call is complete and its inputs are knowable, launch it now in the background and file
Speaker A
the result as a promise. When the real program finally runs, that call returns instantly because the answer's been sitting there for 20 seconds.
Speaker A
Speculative is borrowed on purpose. Your processor has done this for 30 years. Guess which way a branch goes, run ahead on the guess, throw the work away when the guess was wrong. Jiang moved the trick up a level to the tools. Knowing
Speaker A
what's safe to guess is the hard part. Jiang keeps a shadow Replit, a deep copy of the real interpreter that runs the partial program off to one side, so the real one stays untouched. If the model's code turns out to be broken, the shadow
Speaker A
gets discarded and as far as your actual program is concerned, nothing ever happened. Purity is the rule inside that shadow. A literal argument fires the instant it's parsed. An argument built from earlier variables fires, too, as long as everything feeding it is pure
Speaker A
with no side effects. But if anything in the chain touches the outside world, opening a file, writing to disk, that call gets blocked and the real interpreter does it the slow, careful way. A tool author's whole contract is
Speaker A
two flags on a decorator. Is this speculatable and is it pure? At this point, the mechanism's clear and the only question left is whether it pays.
Speaker A
Ho
Speaker A
billion parameters through a vLLM server, five runs per configuration at two temperatures to hold the variance down.
Speaker A
His result, in his own words, is speedups on the order of 1 to 1.2 times, 1 to 1.2. A newsletter that picked it up called that modest, and it is.
Speaker A
In the animated figure on his page, where six slow sub calls run inside one turn, the version without speculation collapses to 2.4 times slower. But that's a demonstration, not the benchmark, and he's careful about the difference.
Speaker A
He's equally clear about why a trustworthy number is hard to pin down. The speedup depends on the latency of your tools, the load on your serving engine, and the choices your harness happens to make on that run. This isn't
Speaker A
a property of the technique, it's a property of your workload. It's also arriving from several directions at once, which usually means something real is underneath. A 2024 system called Conveyor Cut requests latency by up to 38.8% by executing tools partway through
Speaker A
decoding, and a Berkeley group measured 1.7 times this spring by speculating tool calls for voice assistants. But Jeng's argument for why the programmatic version has the most room is the convincing one. With adjacent tool call, by the time the model has emitted enough
Speaker A
tokens to specify the call, its turn's essentially over. There's no overlap left to win. With a program, the runtime is unknown, the call patterns are far more complicated, and the gap between knowing the arguments and needing the answer can be enormous. Now the costs,
Speaker A
and there are two things worth knowing. The load is the first one. Speculating issues requests that may never be needed, and Jeng says so plainly. The worst case is a serving engine clogged with concurrent, possibly wasted, speculated requests.
Speaker A
If your sub model runs on the same GPU generating the main context, aggressive speculation competes with the thing you're trying to speed up. The second one's sharper and it isn't his. On the 1st of June this year, four researchers
Speaker A
published a paper called ghost tool calls. A speculated call reaches an external service before the agent has decided to make it and if the agent abandons that branch, the request is still out there. You can't unsend what somebody already received. Timing is the
Speaker A
issue, they argue, not authorization, so read only permissions don't save you. And guess a search query on someone's behalf and you've told the search engine what they were about to ask. And the steel man for Jason, which it's earned.
Speaker A
A Jason tool call is legible, one object you can log, diff, replay and hand to an auditor. Code mode gives you a program in a sandbox and every tool result comes back as a string that's about to meet an
Speaker A
interpreter, an injection surface Jason never had. Anthropic publishes the case where its own feature loses too. On a benchmark where each turn makes one or two sequential tool calls, programmatic tool calling left the scores unchanged and cost roughly 8% more. So here's
Speaker A
where I land. Code has won as the action space and that's settled, not by anybody's blog post, but by what the vendors shipped. Anthropic shipped it twice in three weeks. Open AI shipped it in its main API. Cloudflare rebuilt its
Speaker A
MCP server around it. When three companies with different business models reach the same design inside a year, you're not looking at a trend, you're looking at a correction.
Speaker A
But the action space isn't what this video's named after, scheduling is. Every harness that moved to code kept the Jason timing. Generate everything and then execute. And that piece was never justified for code. Somebody justified it for a format where there
Speaker A
was nothing to overlap. Three years later, it's still there holding up a system that stopped having that constraint, so delete it. And remember the customer number I promised from Open AI's launch page. On a finance research benchmark, programmatic tool calling
Speaker A
matched quality while using 24% fewer output tokens and finishing tasks 28% faster. 28% of the wall clock from scheduling alone. No weights were retrained. Be honest about which group you're in though. If every turn of your agent makes exactly one tool call, stay
Speaker A
where you are. Anthropic's own numbers say the code path cost you 8% for nothing. And if your tool calls have side effects a stranger can observe, read ghost tool calls before you turn speculation on near production.
Speaker A
There's a bigger point hiding in here and it's the one this channel keeps walking into. Your harness is the product.
Speaker A
Same weights, different loop, different result. Somebody read an assumption copied from harness to harness for 3 years, asked where it came from, and found it came from a format nobody uses for this anymore.
Speaker A
That leaves the question I can't answer for you. To speculate is to act on a guess before you've committed to it.
Speaker A
When the guess is a token, you throw it away and nothing happened. When the guess is a request to somebody else's server, something happened. So, who's responsible for the call your agent never meant to make?
Topics:agent harnesstool callingJSONprogrammatic tool callingcode actionsspeculative executionOpenAIAnthropicCloudflarelanguage models
