Speaker A
Okay, so there are so many models out there nowadays, and if you're trying to build an agent, you're constantly trying to decide which model do I use for which parts of the agent.
Overview of NVIDIA's Switchyard, an open source model router improving agent efficiency by dynamically selecting models per task step.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- Dynamic model routing significantly improves efficiency and cost-effectiveness in multi-step AI agents.
- Switchyard supports diverse model APIs and formats, easing multi-provider integration.
- Built-in observability helps developers understand routing decisions and optimize performance.
- Both tuning-free and tunable routing algorithms offer flexible approaches to model selection.
- Switchyard is open source and designed for easy local or cloud deployment without heavy resource demands.
What the video covers
- NVIDIA released Switchyard, an open source library that routes calls to the most appropriate model dynamically during agent execution.
- Switchyard addresses the inefficiency of static model choice in multi-step agents by adapting model selection to task complexity.
- The library supports multiple model APIs and formats, enabling seamless integration with providers like OpenAI, Anthropic, and OpenRouter.
- Switchyard offers built-in observability features including logs, decision rationale, token usage, latency, and outcomes.
- Routing algorithms are categorized into tuning-free (e.g., LLM classifier, stage router, escalation router) and tunable (e.g., prefill router) types.
- Tuning-free routers either assign a model per session or escalate to stronger models based on runtime signals like errors or complexity.
- The tunable prefill router uses learned policies to predict model success balancing accuracy, cost, and latency.
- Switchyard claims up to 50% faster responses and 25% better token efficiency by combining open and proprietary models.
- The library is lightweight, requires no GPU RAM, and allows users to define custom routing policies and swap cloud providers easily.
- Switchyard is positioned as core infrastructure for orchestration in AI agents, with potential for future integrations and custom policy development.
Chapters
- 00:00Introduction to model selection challenges in AI agents
- 00:39Introducing NVIDIA's Switchyard library
- 01:10Problem of static model choice in agents
- 01:32Varied task complexity in agent workflows
- 02:02Benefits claimed by Switchyard
- 02:57Switchyard overview and open source nature
- 03:21Support for multiple API formats and providers
- 04:19Observability and logging features
- 05:22Tuning-free routing algorithms explained
- 06:27Escalation and stage routers in detail
Full Transcript — Download SRT & Markdown
Speaker A
So it's become quite a common thing now where people will have sub-agents that use a quicker, faster model than the orchestrator agent, which uses a bigger model. But taking that to the whole next step is having a router that actually sits
Speaker A
there and decides on each call which model to use and takes care of making sure that the calls are in the right format, that they're using the right kind of API, that you're using the right kind of tool request, et cetera.
Speaker A
So this brings me to the topic of today's video. Today, NVIDIA is releasing an open source library that decides per step which model handles a task.
Speaker A
So along with them releasing a model today, which I'll cover in another video, in this video, I want to talk about that router.
Speaker A
I want to talk about what the problem is, and I want to talk about one of the things that I've been seeing that a lot of the top companies have been doing over the past six months, but it's been much harder for many people to run locally or even
Speaker A
for their cloud environments, et cetera. Alright, so the problem here is all about static model choice, right?
Speaker A
You've got long-running agents. They're often gonna take hundreds of steps to basically get from the sort of initial task through planning, through to the final results.
Speaker A
And those steps are gonna vary widely in difficulty, right? You've got things like retrieval in there.
Speaker A
You've got summarization. You've got things like simple entity extraction in there. You've got classification.
Speaker A
And those things don't need a frontier model. You're kind of crazy if you're wasting tokens on something like Fable or even GPT-5.6 Sol with a model that's costing you a lot when you just want it to do a very simple task.
Speaker A
And on the other hand, you've got things that are complex. You've got things like planning, orchestration which take a lot to actually do.
Speaker A
So as the agent state keeps changing, you've got tool results coming back, errors appear, an orchestrator suddenly spawns a whole bunch of subtasks that need to be done.
Speaker A
Really what you want is a model or a choice of models that actually adapt to that.
Speaker A
So the whole challenge here is about routing each step in an agent trajectory to the right model at the right time.
Speaker A
And what NVIDIA is actually releasing here, they're claiming can get you 50% faster responses and 25% better token efficiency.
Speaker A
So when they reached out to sponsor the compute for this, I was totally open to see what they're actually doing. Alright, so that brings us to what they're actually releasing, and this is Switchyard.
Speaker A
And they're claiming that this basically, with combining open and proprietary models, can get you 50% faster responses and 25% better token efficiency.
Speaker A
So this is an open source model routing library here, and what it actually lets you do is sit between your agent and the models, and then it decides which model gets used for which call from your agent.
Speaker A
So it's important to understand that Switchyard itself is not a model. This is a library, right?
Speaker A
This is not NVIDIA trying to force you to use a particular model for this.
Speaker A
This is just a regular library that you can download from GitHub and you can use yourself.
Speaker A
Now, at its core, it's got an SDK inside there that allows you to basically swap cloud providers very quickly.
Speaker A
One of the cool things that I like about this a lot is that we know now there are lots of different formats for sending data to the various providers and getting data back from the various providers.
Speaker A
You've got things like the traditional OpenAI endpoint, which pretty much everyone supports, you know, on things like OpenRouter and stuff like that.
Speaker A
But you've also got the Anthropic style endpoints, which a number of companies like Kimi and I think some others started to support as well.
Speaker A
And then you've got things like the newer OpenAI responses API endpoints, and this can automatically out of the box translate what's going into an endpoint like that and what's coming back from an endpoint like that.
Speaker A
So if you wanted to use the responses API format and you're formatting all your code for that, perhaps because you've been using OpenAI, but now you want to try using OpenRouter, this will actually be able to convert the responses API into a format that OpenRouter can then use for all the
Speaker A
open models that it's got access to. So that alone is a huge win here. On top of this, obviously you can use the models that they've got, or you can use your own models and rules to decide how that routing is gonna be done on a per request basis.
Speaker A
Another cool thing that this has got is built-in observability. So you can get your logs from a selected model.
Speaker A
You can get the decision rationale of why it decided to route to a particular model.
Speaker A
You can get token usage, latency, outcomes, all from being able to inspect what's actually going through this router here.
Speaker A
Alright, so let's look at the routing algorithms themselves. There are sort of two families here, the tuning-free and the tunable ones.
Speaker A
So the tuning-free ones are things like LLM classifier, LLM as a judge. It basically picks the model, and then you stay with that model for the rest of that session.
Speaker A
So in a case like this, you're not sort of reclassifying for every single turn.
Speaker A
And that allows you to take advantage of things like caching and other things that get you perhaps price discounts, et cetera.
Speaker A
So this is really good for domain routing. If you know, for example, like, okay, I've got my Hermes agent or I've got my particular agent and some of the calls that are going to make are actually coding-related things, some of them are going to be looking up news kind of
Speaker A
things or general writing kind of things. They're very different tasks and it would actually make sense for having them to be different models, et cetera.
Speaker A
The second tuning-free way is a stage router. So this is built more for sort of coding agents, and this estimates sort of what stage the run is from the tool results and stuff like that.
Speaker A
So if you've got errors, if you've got things going into loops or something, it can then start to spot some of these things and escalate them to basically pushing it to perhaps a better model for doing this kind of thing.
Speaker A
And if it sees that if you're grinding on a particular kind of call, it can then push you towards a more efficient model as well.
Speaker A
The third tuning-free way is the escalation router itself. So this basically starts every conversation on a cheap model, and then an LLM judge watches how the run is going and once it sees problems, it then basically connects to a much stronger model.
Speaker A
So one of the key distinctions is that the LLM classifier that I talked about before predicts the difficulty before running the actual prompt.
Speaker A
The escalation is used later on to judge whether the run is actually going well.
Speaker A
Alright. The second family of routing algorithms are the tunable ones. So an example of this is the prefill router.
Speaker A
So this is a learned model that reads the prefill stage signals, and then its output is not actually answering any of those things.
Speaker A
Its output is just predicting the model success. So it's not actually generating answers. It's basically got a policy that blends that predicted accuracy with things like cost and latency.
Speaker A
Now NVIDIA's actually got a whole paper around this particular approach. Alright. So at this point, probably the main question you're gonna have is, "Hey, is this actually gonna save me money? Is it actually gonna give me better outputs?"
Speaker A
And I should point out here that this is actually an open version of what we've seen other companies do already.
Speaker A
So you've got things like Model Fusion from OpenRouter, where it basically has its own way to decide what models.
Speaker A
We've had things like FUGU from Sakana, which is really combining a bunch of different LLMs in a pool, and then it's basic...
Speaker A
So we do know these things kind of work. it can just often be a pain to actually set them up if you want to use it locally and if you want to pick what models it's basically gonna have access to and what
Speaker A
models it's not gonna have access to. All right, so some of the key takeaways here.
Speaker A
I think, this is really sort of showing that the system of models pattern, is where production agents are heading here, right?
Speaker A
This is sort of core infra for actually doing this. And NVIDIA's framing themself is that, you know, orchestration is here to stay in here So the repo itself has some introductory routing examples at launch.
Speaker A
You can create, test, and contribute your own algorithms in there. It will be great to sort of see if people start sharing, what are their best ways to actually do a lot of these things.
Speaker A
for things like the harnesses and the agent frameworks that NVIDIA's actually partnering with, it seems like for most of those, this is all just gonna run on the CPU alongside your harness.
Speaker A
You're not even gonna need any GPU RAM for the library itself. So I've got to say that this is, a really cool step from NVIDIA.
Speaker A
Routing here can really be a hard engineering problem. You've got lots of decisions to make.
Speaker A
You've got state management. You've got the whole thing about when do you use it, when do you not use it?
Speaker A
This library basically gives you the infrastructure to get started on doing this. The actual policies for what's gonna be routed to what is still gonna be up to you, and I do think over time, as some of these patterns become
Speaker A
clear, this is something that I may even look at doing more videos about All right, so if you wanna get it, just go and check out GitHub now.
Speaker A
There are some links in the description of where you can find out more, their blog posts, et cetera.
Speaker A
I think they've got integrations are rolling out through their partners And like I said, perhaps in the future we can look at a follow-up video on building a custom routing policy hands-on, and using the different staged routers in there as well.
Speaker A
So let me know in the comments, if you've tried something like this before and if you're excited about it.
Speaker A
This is definitely something I think is a step forward and something that is just gonna become standard, six months, a year from now.
Speaker A
As we're seeing so many good open models, come out, it just doesn't make sense to pay top dollar for every token that you're actually generating to build something.
Speaker A
Anyway, as always, if you found the video useful, please click like and subscribe, and I will talk to you in the next video.
Speaker A
Bye for now.
Topics:NVIDIASwitchyardmodel routingAI agentsLLM orchestrationopen sourcemachine learningmodel selectiontoken efficiencyagent routing
