Speaker A
Okay, last talk to bring us home, we have Arjun and Ronak, co-founders of Trajectory. They're doing a bunch of interesting research around what I think is one of the hotter topics, continual learning.
Arjun Karanam of Trajectory discusses continual learning in AI, focusing on closing the experience gap to improve AI agents with use.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- Continual learning addresses the experience gap in AI, enabling models to improve through real interactions.
- Capturing detailed interaction data is essential for effective continual learning.
- Differentiating between model and harness updates optimizes learning and system performance.
- Evaluation environments should closely match real user environments for meaningful feedback.
- AI systems that learn continually can become exponentially better over time.
What the video covers
- Trajectory is building a platform for continual learning to help AI agents improve through experience, not just IQ.
- Current AI models improve in intelligence but lack experience, making them less effective initially.
- The 'experience gap' is the difference between raw intelligence and practical, learned performance over time.
- Trajectory captures interaction data (traceability) to learn from real-world usage that is often discarded.
- They develop model specifications to define agent goals and use reinforcement learning algorithms like SDPO to improve models.
- The system distinguishes between model updates and harness updates, depending on the type of feedback.
- Four key wishes to improve continual learning include better traceability, evaluation, harnessing, and model improvement.
- Traceability involves capturing full interaction trees, including sub-agents and corrective user feedback like undos and retries.
- Evaluation should closely mirror the user environment to ensure relevant training and feedback.
- The ultimate goal is AI systems that compound improvements with use, becoming faster, cheaper, and more accurate.
Chapters
- 00:00Introduction to Trajectory and Continual Learning
- 01:11The Experience Gap in AI Models
- 02:10Importance of Capturing Interaction Data
- 03:14Model Specifications and Reinforcement Learning
- 05:25Four Wishes for Improving AI Agents
- 06:33Wish 1: Enhanced Traceability
- 07:49Wish 2: Evaluation Alignment
- 08:59Additional Considerations and Closing Remarks
Full Transcript — Download SRT & Markdown
Speaker A
Please welcome Arjun to the stage. Cool. What's up? How's it going, everyone? Thank you so much. I'm really excited. I know I'm close to the last one, so hopefully there's some remaining attention span.
Speaker A
I'm here. Ronak is also back there. So we're together. But let's get started. We're Trajectory, and we're building the platform for continual learning.
Speaker A
Who are we? I think we're working on a really, really cool mission, and that's the most fun part. The second most fun part is I get to do this with my two best friends. Ronak over here worked at One Surf and Train Speed 1
Speaker A
before this, and Michael, who worked on really cool robotics stuff at DeepMind. And what I wanted to start off with was the worldview that we have and how that shapes why we think this problem is important.
Speaker A
And so the worldview that we have is that, and it's not a hot take, we are living in an incredible time in human history here. Every single week it seems like another model is coming out that is
Speaker A
leapfrogging the last one. It's undeniable that models are getting better and better. But we think that they're getting better on one axis, and that is IQ. These models are smarter and smarter, but it always feels like when you're talking to them, it's their first day on
Speaker A
the job. And so you take this analogy: if you have a Terence Tao in your pocket, that's great.
Speaker A
But Terence Tao day one at an accounting firm is probably not the best accountant there.
Speaker A
But give him a few years or honestly maybe even like a few days, he'd probably be really good.
Speaker A
And so we have this orthogonal axis of experience that is much more important than we think in conjunction with IQ. And that's what we're calling the experience gap, and that is what we want to close as a company.
Speaker A
Um, and then the question is, like, how do you close that? Where does experience come from? Well, it's already out there right now. There are so many—hundreds of millions is a number—but so many tokens out there that are being
Speaker A
generated by these agents. You view them or sometimes you don't even view them, and they get thrown away. But this is all real work that they're doing, and then people are acting upon them, and then it's being thrown away. And so our
Speaker A
take as a company, as Gabe mentioned, like our opinionated take is that this is a signal that we should be learning from.
Speaker A
And this is also how humans get better. And in classic AI fashion, like if humans do it, probably a good way to mental model off of. Uh, and so today's agents, they're still slow, expensive, error-prone. Over time, you implement
Speaker A
them, they probably don't get better. We want to imagine what learning agents look like, where as they are used by people, they get better and better over time.
Speaker A
Uh, and tangibly this has two good benefits. Right away it gets you much faster, better, cheaper models. Uh, but more excitingly, it gets you to this goal of systems that compound with use.
Speaker A
And so I want to start off by talking about how we're approaching this problem, uh, and then some interesting stuff for you guys at the end.
Speaker A
And so where it starts off, and this kind of dovetails off of what Harrison was talking about before, is it starts off with traceability. We need to first capture these interactions, capture this experience that's being thrown away.
Speaker A
So that's step number one. Uh, and then step number two, and this is where we start getting to our research. The way we view the company we're building is we're building a product to allow for continual learning,
Speaker A
but we're doing cutting-edge research under every single one of them to make this possible.
Speaker A
And so you have all these interactions, and the next thing we're building is this idea of a model spec. This idea that you need to have a way to define what do I want my agent to do and have
Speaker A
the agent learn against that. So we're doing really cool research here on how to extract user interactions, turn that into reward, extract traces, and turn that into exact specs of what you want your agent to do.
Speaker A
And then, okay, cool. You have what you want your agent to do. What do you do with that? Well, there's two surfaces.
Speaker A
One is the models, right? You want your models to learn from actual interactions here. And so we're doing really cool research on algorithms like SDPO, and things along those lines where we're using RL to
Speaker A
take these full long traces and improve models over time. But models are not the only thing. You have harnesses as well. And we're doing a lot of really cool research on when you have good feedback from people, does that go to a harness, or does that
Speaker A
go to a model? An example here is if it's like a fact, like this person, this company has been delisted, right? You probably don't want to train that knowledge into the model.
Speaker A
It's probably context that should be available to the harness. And so that is how kind of how we're splitting things, and that is the product that we're building. Allowing you to go from real interactions all the way to specs of what you want your agent
Speaker A
to get better at, to better models, and then better harnesses, and then being able to deploy those models right away, and then start using our product.
Speaker A
And so that's the world we want to get to. If you have built agents, which I'm sure most of you guys have, you probably know that there are so many things that go wrong with every single part of these, and then how every single
Speaker A
one of these is an unsolved problem. And so the way I wanted to frame the rest of this is I'm going to imagine that I am like straight from Aladdin, and I have like four wishes.
Speaker A
And what are those wishes that I would ask to make the agent ecosystem better? And so like we have agents today, we have this gap, and there's a level where agents get to, and after that we can really see the exponential
Speaker A
effects of continual learning. And what are those? I'll walk through them step by step with the same four categories I talked about.
Speaker A
What we would like to see on the traceability side. What we would like to see on the eval side, what we'd like to see on the harness side, what we'd like to see on the model side. So let's start with wish
Speaker A
number one. Uh, and how can companies get there faster? Uh, traceability. So two sub-wishes here. Number one is around tracing the entire tree of what happens, uh, including sub-agents.
Speaker A
Uh, what a lot of companies do is they trace the main action that is happening, but they throw away tool calls or sub-agents that are made. Um, and that makes it very hard to learn from the whole
Speaker A
thing that you've done. So that's wish number one. Uh, sub-wish number two, uh, is probably the most important in here, and that you should build your product around how you can both capture this interaction data, but also elicit the right amount
Speaker A
of feedback, uh, so that you can then capture it. Um, and a really interesting thing here is I think the level one way of thinking about this is, uh, okay, let's have like a thumbs up, thumbs down, and
Speaker A
like let's capture that. And that sounds amazing in theory, uh, but it's incredibly noisy, and if you've used any coding agent, you know you kind of just like accept everything that the agent does.
Speaker A
And it's only like five commits later that you're like, oh crap, like this broke everything. Let me go and undo that. And so it's the corrective behavior, the edits, the undos, and the retries that both need to be
Speaker A
like elicited from the user, but also captured. So that's my wish on traceability. The second wish is around evals. And the mental model to use here is your product that your user uses should be as similar as
Speaker A
as close to what the eval is done in, in, for wise, should be as close to where the training is done in.
Speaker A
In an ideal world, these are all the same. There's no difference between th
Speaker A
that they're requesting on the frontier that might not be possible, all super helpful. Second is making every task roll outable. This is a pretty big infra challenge, uh but it's this idea that hey, if a user has done one thing, can I
Speaker A
somehow replay what they what the user has done? And that is an incredibly helpful boon to to moving forward in this agentic era. And then lastly, this goes with number one, grading through like the real harness, uh not making a
Speaker A
variation of the harness, but like grading the real harnesses that people use in production.
Speaker A
So that is wish number two. Uh wish number three, getting into the harness, uh pretty important. Uh number one is and and I think the mental model to use here is I think a lot of products have built their harnesses around models
Speaker A
that existed a year a year and a half ago. Where the primary function of the harness was to prevent the agent from doing bad things.
Speaker A
Which was really good when agents would randomly break or have misformatted outputs, but now we're very much in a let the agents cook world.
Speaker A
Uh and so I think I would view building the harness as not enforcing specific flows that need to be done, but more what are the primitives that your product has, whether that's search tools or private information, and then viewing the agent
Speaker A
as orchestrating those primitives and then letting it do that. So that's sub wish one. Uh sub wish two, and this is how we're building our product as well, is making the agent interface as close to the user interface
Speaker A
as possible. Uh in an ideal world, every single thing that I can do on your UI, your agent can do via tool call as well.
Speaker A
Uh I think that is a world that makes training and continual learning so so much easier.
Speaker A
Uh and then lastly, this is a small one, uh making tool responses informative. I think it's easy to be like, "Okay, I just want to know the state." Like let's say I'm like calling a search tool. I want to know the state. I want to see if
Speaker A
it worked. Uh and if it's writing to a a DB or something, it just writes to it.
Speaker A
And what we see often is like the response from that tool call will be uh done. Finished.
Speaker A
That sounds like good in theory, but from an agent's perspective it's like incredibly confusing. And if you're trying to train or learn off of that, there's like no signal to learn off of.
Speaker A
Right? You have no idea what actually was written. You have no idea what was actually was read. And so, that's the last sub wish under the harness is to make your tool responses really informative.
Speaker A
So, that's wish number three. I think like canonically I only get three wishes, but let's imagine I have four.
Speaker A
Uh and that's around the models. Uh and we've talked about that a lot today. Uh and that is uh I wish it was as easy as like just switching to an open weight model, but if you've tried that, you've probably
Speaker A
seen there's 50 other considerations from security to safety to uh you know, different access provisioning stuff. So, start getting comfortable running on open weights cuz that is what unlocks the door to owning owning your weights and then continually
Speaker A
improving on top of them. Um and then Gabe also talked about this this idea that like model routers are uh going to probably play a really large role in routing intelligence to the exact capability of the task they're at.
Speaker A
And so, experimenting with with routers is another wish I would wish cast into the world.
Speaker A
And so, that is the high level here. And so, I think like when we work with companies, the first thing we often do is do an audit of of these and like where things are. And for the most part there's a lot of work
Speaker A
to be done to get here and so we work with them to build it out and it would be my dream if we go to a company and they have a lot of this built out.
Speaker A
Granted a lot of the work we're doing is also knowing that a lot of this we can't expect and so we're doing a lot of cool research on, you know, assuming that nobody has Evals, somebody doesn't have Evals, how can we overcome that? Or
Speaker A
assuming that the traceability is off, like how can we learn from the signals that are there? And so that is that that is my my goals for the for the universe here.
Speaker A
Uh just want to quickly go over uh I think this is like a very very core research and product problem and so we're really excited with the team we've put together but I think more importantly and then two of the folks
Speaker A
who are here today uh we're working with extremely frontier customers. The folks who are really really pushing the boundaries of what agents can do in production both today and tomorrow.
Speaker A
Uh and so really excited to be working with them and another thing I want to show is we have Oh, that's my email.
Speaker A
Uh we have a beta and so a really really core belief we have is that this capability, this ability to own your own intelligence, should not be something that you need to consult away a lot of the times. It should be
Speaker A
expertise you build on your own because of how important it is for your product.
Speaker A
Um and what that means for us is we want to build a product that empowers any company to own their own models and mainly get to continual learning from real interactions and so here's some some cool screenshots from our product. I'm
Speaker A
I don't know if Nico is here but I'm supposed to show this Nico later today so this is like a sneak peek. Uh but here's Harvey. Here's the lab benchmark that that Gabe was talking about. Really really easy to import. Um
Speaker A
training a model is incredibly easy. We're extremely proud of the interface we've put together here cuz if you've post trained, you know, there's like 50 million things that go wrong, 50 million knobs to turn, and it's all researcher
Speaker A
intuition, trademark asterisk, whatever that means. And what we've tried to do is take all of how our researchers post train their models and use an agent behind the scenes to take care of most of those knobs and just
Speaker A
only expose the ones you need to know to make post training as easy as possible.
Speaker A
So, really easy to train a model, see how the model's doing, evaluate, compare it, uh see if it's better than the model you were using before, and then deploy it.
Speaker A
And so, in terms of actual work you put in, and that was not waiting for the model to train, probably like 15 minutes. And so, that is the world we're we're trying to get to to empower all these companies and
Speaker A
hopefully more folks in this room. Uh So, yeah, that's our goal. We want to every company to own its own experience layer, and we're excited to build that alongside of some of y'all here, hopefully.
Speaker A
[applause] Thank you. It's a great talk. You know, I wonder like, you know, uh when you talk about continual learning, like uh what do you think about their trainable object um between model weights, harness, tools, application layers?
Speaker A
Yeah. Uh how how do you think of them? How do how do you prioritize?
Speaker A
Yeah, I think I like a fun fun phrase is that if you ask like six researchers what continual learning is, you're going to get like seven answers, probably. Uh you can be very pure about this and be like, "Oh, it needs to be like a human,
Speaker A
and it is just the weights that need to be adopting in real time, one shot." But, the way we view it is that the intelligence that your product is run off of is a system.
Speaker A
It's a system that has many components, and true continual learning is something that optimizes across that system and optimizes based on what parts of the system needs to be updated uh for the information that you're learning. Like a
Speaker A
mental analogy that I have is like when you are you know, saving things, right?
Speaker A
We no longer think of like where in your RAM or where in your hard disk to save.
Speaker A
That's the level that's abstracted based on what makes the most sense. We think about models versus harnesses versus context in the same way. It feels wrong that we're having to make the decision off of very little priors what to update based on what.
Speaker A
This is a scientific problem that can be solved. Let's do that and then let's abstract that away.
Speaker A
Cool. Oh yes. Yes. Um, some of the speakers today talked about, you know, the importance of not training on your customer data.
Speaker A
Yeah. And I imagine like a big part of continual learning is you are learning from all these interactions. Um, how are you thinking about things like differential privacy? Like how do you actually make your system better with I
Speaker A
imagine most of these application companies have customer specific data arrangements that make that very hard.
Speaker A
100%. I think that's an incredibly important problem. Uh, actually before this I was at Apple and this is a problem we worked on quite a bit as well.
Speaker A
Um, and there's a lot of clever things you can do where you're not actually training on customer data, you're instead sampling distributions from your customer data and then synthetically generating your own and doing kind of like a mental models maybe like a cryptographic
Speaker A
thing where you're like comparing those distributions to see is my data actually on distribution, but you're not training directly off of the customer data. Um, and so that that's totally right.
Speaker A
There's a lot of interesting work there and it's like very central and already with our customers we're doing some interesting and and fun things in order to uh, to get over that.
Speaker A
Yes. Hey, I'm Sid. I'm from Finch. Um, thank you. Um, I think episodic memory plays uh, a role in continual learning to some extent.
Speaker A
For example, like if a user does something on our platform that corrects what an agent did before, we might want to update an agent's behavior. Does Trajectory have like an opinionated take about that in the platform?
Speaker A
Yeah, I think Okay, on episodic memory and I'm just going to repeat here. On episodic memory and like what if I'm thinking about this correctly, how different types of feedback impact what is learned?
Speaker A
Yes. Yes, okay. So many thoughts. One way to look at this is there are two different types of signals. There are signals that are just something went wrong. Like for example, if somebody like flames the agent says like, "You're really bad." But he
Speaker A
doesn't follow anything up with that or just thumbs down or like drops off the session. Those are cases where you know something went wrong, but not what right looks like. Then you have things like the agent retried and then it got to the
Speaker A
correct solution or you corrected it by saying this is a difference. And so that is like a difference we make. In the latter case, we're very, very confident about the reward associated with the with the output and and what we're assigning.
Speaker A
With the former, we know to penalize that behavior, but we don't necessarily say like there's like a correct right answer. That's one way of looking at it.
Speaker A
Another way of looking at it is there are different like there's like a hierarchy of like how pertinent the information is people-wise. There's some information that's probably globally accurate, globally true.
Speaker A
Things like a tool call repeatedly failing when trying to call a certain tool. That's probably relevant to everybody.
Speaker A
And so this should be trained into the model and the model should get better at learning this tool.
Speaker A
A certain user is like, "I never want to use the sub agent. Please, please, please don't do it." That's probably not something you should train into the model and leave to the context and what we were really excited about is
Speaker A
this is the stuff Harvey was talking about where this is actually something that will probably happen on a per org basis, or per customer even further, a per customer basis. And so, that's another way to think about it as well.
Speaker A
So, the hierarchy of like what the feedback is pertinent to. Got it. Thank you.
Speaker A
Yeah. Yes. Thank you. I think one of the themes of today has kind of been this idea that there are some, you know, tasks or part of your product where you kind of want to maybe like experiment more, go cheaper, add in
Speaker A
more open source stuff, right? And then other areas where kind of being at the frontier and using a closed model or a closed framework is more useful. I guess like extending that analogy to continual learning, are there tasks or workflows,
Speaker A
or like what is the shape of problem where something like this you've kind of seen really matters a lot, right? Versus where like just using, you know, statically trained frontier model or even a statically trained open source model, and then like you know, using the
Speaker A
harness to inject relevant information into the context is sufficient to kind of achieve what the user usually wants to achieve.
Speaker A
Yeah. This is a great question. I think the level one answer is I think for the most part most tasks work for continual like, but the ones I'm most excited about are tasks that are on the frontier. The mental
Speaker A
model I have that that we have for AI progress in in in a way is people will ask models, people will ask products with the expectation of what they think the product can do.
Speaker A
Uh and maybe they'll ask at the edge of what the product can't do, and then they'll see it fail, or they'll see it do something wrong, and they'll retreat back and be like, oh, it's it wasn't good enough. I can't use it for this.
Speaker A
I think like a great example is like, would I even dream of typing in some of the like the whack query that give the cursor two years ago, like definitely not. I was like way lower than this, but
Speaker A
then over time I started querying it where the models were getting better and I was like, "Okay, maybe we can do bigger and bigger things." What we're most excited about is the way that has happened is like models
Speaker A
have generally gotten better and we've been targeted pushing in certain directions. Uh but what we're seeing with some of our customers are cases where users ask for things, the model can barely do it, but then during training it learns how to do it and then the user
Speaker A
can now do this thing they couldn't do before. That chain is what's really exciting about continual learning of like really pushing that frontier of what's possible based on what the user tries and cannot do.
Speaker A
Awesome. Thank you, Arjun. I think we're out of time. Uh really appreciate you doing this. It's awesome. Thank you.
Speaker A
[applause]
Topics:continual learningAI agentsexperience gapreinforcement learningtrajectorymodel improvementtraceabilityevaluationmachine learningartificial intelligence
![[Full Episode]Nothing Else Compares（English-dubbed）#cdr… — Transcript](https://i.ytimg.com/vi/YYJruShKWz8/maxresdefault.jpg)
![Like My Coffee | Game Changer [Full Episode] — Transcript](https://i.ytimg.com/vi/9EbvU8OZtqI/maxresdefault.jpg)
![The Drinking Game | Game Changer [Full Episode] — Transcript](https://i.ytimg.com/vi/HqYTmgzZEiA/maxresdefault.jpg)
