Speaker A
So, the first thing I want to talk about, it's pretty serious, okay? I need all of you to take this seriously. Put your phones down, put your laptops down, put your water bottles down. We're done hydrating. Are you all ready?
A Cursor engineer explains how he built trust in AI coding agents, moving from heavy micromanagement to letting agents auto-merge PRs, centered on verification, evals, and turning review feedback into
Key Takeaways
- Trust in AI coding agents is earned through verification: letting agents run and test code for real closes the loop and makes correctness achievable.
- Effective agent use mirrors good engineering management, with feedback encoded into systems rather than repeated manually.
- Recurring review feedback should be converted into automated hard rules, evals, or lint constraints.
- Parallelizing many agents only becomes viable once you trust the output of a single agent.
What the video covers
- The speaker, an engineer at Cursor, describes his journey of learning to trust coding agents, comparing it to an engineering manager moving from micromanaging to trusting reports.
- A year-long trust curve: early on he was constantly in the loop, watching every agent output, and couldn't parallelize because he didn't trust results.
- Over five months he reached the point where agents auto-merge PRs; he reviewed about 20 already-landed, good PRs on main one morning.
- He reports high productivity: roughly 1,000 PRs shipped last month and nearly 800 landed so far in the first 12 days of the current month.
- Verification is identified as the most important skill: agents must be able to run the code, take CPU traces or heap snapshots, and test the app as users would, closing the feedback loop.
- A case study from his first weeks at Cursor: the agents window performance problem, where the agent confidently misdiagnosed flame graphs until he gave it proper tooling.
- Early attempts at verification skills were often useless because agents couldn't navigate the UI, which pushed him to build better tooling.
- Evals are described as unit tests for agents, and reading tool calls and thinking blocks is a way to find where agents fail.
- Start verification locally, where agent behavior is observable, before expanding to more complex setups.
- Rewriting code is argued as sometimes worthwhile with agents, and review comments should be turned into hard rules and lint-like constraints rather than repeated nitpicks.
Chapters
- 00:00Opening and introduction
- 03:00The trust problem with coding agents
- 07:30The trust curve and productivity journey
- 20:00Verification as the key skill
- 28:20Case study: the Cursor agents window performance work
- 43:20Evals and reviewing agent behavior
- 48:20Rewrites, code smells, and hard rules
- 51:40Q&A and closing remarks
Full Transcript — Download SRT & Markdown
Speaker A
ready? Very excited to announce. [applause] Uh, you can't end PM insult yet. We'll we'll release that later today. Okay. So don't don't take your laptops out. Um but yes very exciting milestone and after I do the release you can get
Speaker A
Very excited to announce.
Speaker A
Yes. Yeah we're good. So yeah today yeah I think I think the big theme for me as I've been using agents to write code and I'm sure a lot of you have had the same experience as well is how do you trust it? You know,
Speaker A
[applause]
Speaker A
know guessing hallucinating uh, you know, confidently stating that they found the smoking gun, uh, for the hundth time, uh, but it's actually not the real problem. You lose a lot of trust. And when you lose when you don't
Speaker A
Uh, you can't end PM insult yet. We'll release that later today. Okay. So don't take your laptops out. Um, but yes, very exciting milestone, and after I do the release you can get started on your applications by installing babel plugin react as well as the new eslint plugin. Uh, okay, so I assume you can see my screen.
Speaker A
team and I have a bunch of, you know, I have a team of engineers uh on my team and I don't trust them, then the mode of operation I'm going to be in is going to be like micromanagement, right? I'll
Speaker A
Yes. Yeah, we're good. So yeah, today, I think the big theme for me, as I've been using agents to write code, and I'm sure a lot of you have had the same experience as well, is how do you trust it? You know, especially if you are an engineer that's been writing code for a very long time, you have a lot of opinions and lessons that you've learned about doing good engineering. And when you see agents just, you know, winging it and guessing, hallucinating, confidently stating that they found the smoking gun for the hundredth time, but it's actually not the real problem, you lose a lot of trust. And when you don't have much trust in your agents, I feel like you really can't get the most out of them.
Speaker A
And so I drew this chart because uh it's not it's not a very scientific chart, but like this is how I imagine myself and my journey through using agents. So you know like fast forward or back forward uh or fast back uh fast
Speaker A
For me, the parallel is management. If I'm an engineering manager of a team and I have a bunch of engineers on my team and I don't trust them, then the mode of operation I'm going to be in is micromanagement, right? I'll have to spend a lot of time looking over my reports' shoulders and checking that they're doing their work well, like they're not shipping bugs to production.
Speaker A
several like a handful of agents and you find yourself just constantly figure uh you know trying to understand what your agents are doing uh and you're very very in loop. You're watching every single output. you are sitting there prompting
Speaker A
I drew this chart because it's not a very scientific chart, but this is how I imagine myself and my journey through using agents. Fast forward, or fast backwards, about a year or so, when nobody, or not many people, were using agents to code, I think you got into this mode where you were very heavily in the loop with one or several, a handful, of agents, and you found yourself constantly trying to understand what your agents were doing. You were very, very in the loop. You were watching every single output, sitting there prompting, and you really couldn't parallelize beyond that, because again, you don't have that trust. You can't go to a hundred agents, spawn a hundred agents, when you don't even trust the output of one agent.
Speaker A
so over the past 5 months I feel like I've really been able to uh like ascend this trust curve and now I'm at the point where uh I actually have this sounds kind kind of scary to say this
Speaker A
Over the past five months, I feel like I've really been able to ascend this trust curve, and now I'm at the point where, and this sounds kind of scary to say, and it makes me sound like a slop artist, but I promise I'm not, I actually have my agents now automerging PRs for me. Which is a wild thing to say, but I woke up today and there were about 20 PRs landed, and I just reviewed them on main. They were already landed, and they were good. So how I got to that point is basically what I wanted to talk about today. Feel free to jump in if you have questions, Colin.
Speaker A
up today and there were like 20 PRs landed and I just reviewed them on main like they were already landed and they were good. Uh so how did I get to that point is basically what I wanted to talk
Speaker A
Oh, yeah, of course, I've got to show this chart. Oh, do not trust someone who requested to control my computer. I probably won't do that. But I'm sharing this chart not to flex, but to show the journey, so you can see the curve. It sort of inversely matches the contributions I've been able to land at Cursor. I joined five months ago, and in my first month I wasn't very productive, because I was learning the codebase and didn't know what the heck was going on. As I got more confident in my agents, I've really been able to ramp up my productivity. Last month I shipped a thousand PRs, which is ridiculous, and this month we're only on the 12th and I'm already at almost 800 PRs landed. So the velocity is definitely high, and I'm sure a lot of you will be questioning how much of this code is actually good. That's fair to question. But I think if you set up your agents well, you can get to a very similar level.
Speaker A
to control my computer. Uh probably won't do that. Uh but yeah, so this chart I think I I'm I'm sharing this chart not to kind of like flex but to kind of show like the journey like so you can see like the curve like it sort
Speaker A
So I'm going to talk about how we do that. I'm curious, call in with your experience as well, but for me, the most important skill you should have in your toolbox when you work with agents is verification. By verification I mean the ability for an agent to actually run the code, take CPU traces or heap snapshots, or open an iOS simulator, whatever the way your application is exposed to your users, it can do the same thing and run it for real and actually test and verify it. That's the thing that really closes the loop. It doesn't guarantee your agent writes good code, but it allows them to at least write correct code, which is a really big step forward for being able to trust your agents.
Speaker A
the codebase didn't know what the heck was going on and as I got more confident in in my agents uh I've really been able to kind of ramp up my productivity. Uh and again like yeah like last month I
Speaker A
I can share one example we have within Cursor. Let me open this and make it full screen. There you go. For Cursor's agent window, this is actually an interesting story. When I joined Cursor five months ago, I was supposed to join a different team, the cloud agents team, but since I have a lot of experience working on React, and the agents window is a React application, I was asked to help out with the agent window work. There weren't really a lot of skills to help me. So I found myself thinking, okay, the agents window is going to launch in about a week, we have a really tight deadline, and I was just sitting there like, okay, I'm going to open up the performance tab in Chrome dev tools, take a trace, and try to make sense of this flame graph myself. And keep in mind, this was my first week. I had some idea what I was looking at, but the codebase was completely fresh to me.
Speaker A
questioning like how how much of this code is actually good. Um, and I think yeah, like that's definitely fair to question.
Speaker A
I realized my agent had no idea either. I would take a screenshot of the tree, download the trace, send it over, and it would say, yeah, it kind of looks like this, and confidently state what the problem was. Then I'd try to fix that, and it turned out that wasn't the actual problem. This was a very slow process. If you've ever done performance work yourself, or even just development with an agent where you don't have a verification skill, you are the verifier. You're the bottleneck. You tell your agent to do something, it goes off and writes some code, you open your local dev build, and then you say, oh, it doesn't work. Then you copy-paste screenshots or console errors, your agent slowly works through that, tries to understand it, and tries to fix it, but you're constantly in the loop, being the bottleneck. There's really no way to parallelize.
Speaker A
Um, and so I'm going to talk about how we do that. Uh so for me I think I'm curious like I guess call in your experience as well but uh for me I think the most important skill that you should have in your
Speaker A
So the Chrome DevTools skill was one of the first skills I built for Cursor. Glass, by the way, is the code name for the agents window that we use internally. The code itself isn't super interesting, and your agent can very easily make one for you. If you're building...
Speaker A
however your application is exposed to your users it can do the same thing and uh run it for real and actually test and verify it don't work because that's the thing that really closes the loop.
Speaker A
uh it doesn't guarantee your agent writes good code uh but it allows them to at least write correct code uh which is a big a really big step forward for being able to trust your agents. Um I will I can share one example that we
Speaker A
have uh within cursor uh oops where let me open this make this uh make me full screen. There you go. Uh so for the for cursor's agent window uh so this is actually an interesting story but uh when I joined
Speaker A
cursor 5 months ago uh they're actually uh well I was supposed to join a different team I was supposed to join like the cloud agents team uh but then since I have a lot of experience working on react and agents window is a react
Speaker A
application uh I was uh I was asked to basically help out with the agent window work. Uh but um there wasn't really a lot of like skills to help me. So I just found myself like okay uh agents window is going to launch
Speaker A
in like a week, right? We have a really tight deadline and um there was uh you know I was just sitting there like okay I'm going to open up the performant the the Chrome dev tools and just like take
Speaker A
a trace look at it myself and try to make sense of this flame graph. And keep in mind, I was just like in my first week, so I had no idea what I was looking at, no idea where, you know, I
Speaker A
mean, I had some idea, but you know, the codebase was completely fresh to me. Uh, and I realized like my agent had no idea either, you know, like I would take a screenshot of the tree, I would download
Speaker A
trace, I would send it to it, and it be like, yeah, it kind of looks like this, you know, uh, and it would like confidently state like it's this thing.
Speaker A
And then I try to fix that and turns out that's not the actual thing.
Speaker A
So this was a very very slow process and if you've ever done any like performance work yourself or you know just even development with an agent where you don't have a verification skill you are the verifier right you you're the
Speaker A
bottleneck you you you tell your agent to do something and then it goes off and write some code then you open up your you know local dev build and then you start to say oh you know doesn't work
Speaker A
then you got to copy paste screen uh you know screenshots or console errors or whatever uh and then your agent like slowly kind of like uh you know works with that and then tries to understand it and um fix the thing but then you
Speaker A
were constantly just in the loop and and being a bottleneck. So there's really no way to parallelize. So the control glass skills are like one of the first skills I built uh for cursor uh and glass by the way is the code name for agents
Speaker A
window that we use internally but it's just cursor I guess. Um, and so this skill uh is I guess the the the code itself is not super interesting. Your agent can very easily make one for you.
Speaker A
uh where if if you're building an Electron app or a web app or even iOS uh applications uh you can teach your agent how to use like the ChromeDev tools protocol or through uh Apple has some utilities as well for running the
Speaker A
simulator and taking traces and controlling programmatic control as well. Uh so that's really useful. Uh but one thing I want to talk about is uh the this thing uh where is the read me? Uh so this skill comes with this very unique
Speaker A
feature called or not feature uh unique file called a feature map. And so the story then is like I built this skill and so now the agent was able to uh actually run the agent window and take traces and whatnot. Uh but it had no
Speaker A
idea what what the agents window was. So um you know like someone would say like oh the the left sidebar is like laggy or something like that or you know the right side the the PR tab is not working
Speaker A
and the agent would just be like kind of flailing around. It would spend a lot of time trying to like look up the code and you know where is this feature? How do I actually get to it on the UI which made
Speaker A
it basically completely useless. Uh you know like we would I would run the skill locally and you know it would spawn a dev build uh but then it just be turnurning like I just try to click here it wouldn't know how to get to things um
Speaker A
and it was just an awful experience uh so was putting arrows on my screen um so uh yeah this this feature map has been really useful uh because it teaches the agent how to get to all of the features
Speaker A
that you have um and in PAC the plugin that I I've made uh if you search for PAC cursor on Google you you'll find it uh but there is a create verification skill in that plugin where it actually
Speaker A
helps you set up something like this for yourself um including the feature map so it will actually explore the code and build up this initial feature map that tells your agent how to get to all of the different features that you have uh
Speaker A
and this is extremely powerful because now like you have these user reports that come in uh you you can actually map even like a vague report or even a screenshot. So we have this uh internally at cursor where uh we have a
Speaker A
slack channel with you know lots of people giving us feedback on the agents window and rockbot and whatnot. Uh, and often times the report is very bad like very low quality like someone will just put very often we get like a screenshot
Speaker A
like and then someone just says question mark question mark question mark like what is this and you know like the without this your agent like I have no clue right but with a feature map like this it has a lot more context and
Speaker A
understanding of how to actually navigate how to get to all the different features. Uh so like you know example like I guess like the sidebar like what is the sidebar uh you know like all the different sub features that are present
Speaker A
in it um like from the user point of view here's where how do I get to it all the different keyboard shortcuts uh even like the the what do you call it the DOM elements or yeah like the
Speaker A
attributes that you use for selecting things through the CDP uh are all there. So uh again yeah this is like really really powerful uh for for agents uh and pack ships uh that create verification skill but also a maintain
Speaker A
verification skill uh so you can keep this up to date. Cool. Yeah, I was just going to ask how you created that. So do you mind sharing a little bit more about um that that process in the context of Pstack and
Speaker A
maybe just what Pstack is for the folks who aren't familiar? Yeah. So, P stack is pretty interesting because uh well, first of all, the name is kind of goofy. like the P the P in P stack is like potato potato stack
Speaker A
because I um so uh uh there is a pretty uh famous person Gary Tan who is the CEO of Y Combinator and he's come up with this plugin called Gstack uh Gary Stack and uh funnily enough we share the last
Speaker A
name we have no relations uh but I thought it would be funny to kind of you know poke fun at Gary and make Pstack my version of of of of his plugin. Uh but kind of tailor it to my own set of p
Speaker A
engineering practices. Uh but I honestly actually never set out to build PAC. Uh it just started with a bunch of skills, right? Like I started with that control glass skill and then I started with another skill like called
Speaker A
how which I also noticed through like observing agents. Um, so like you know in the early days of me, you know, trying to climb this ladder, I was like super in the loop and I was basically nitpicking my agents to an extreme
Speaker A
degree, I was like I would tell it um, you know, this feature has stopped working. Here's a bug report. Like why isn't it working?
Speaker A
And very often the agent would just like confidently state like, "Oh, it has to be this, right? It has to be this thing." And I noticed like when I looked at the actual tool calls, I noticed it wasn't actually reading the code that I
Speaker A
thought should be affected. And that made me just extremely suspicious. And at that point, I was like, I'm not going to I can't trust any this agent anymore cuz it's just it's just completely hallucinating. And I think I think it's very easy to just you know
Speaker A
like build up that distrust and not and kind of feel helpless like you know you don't know how to help your agents succeed. But like again I think the the the management analogy is super helpful because like imagine if you were a
Speaker A
manager of an engineering team and you had an engineer on your team who was a really good coder. No business context whatsoever. So you know they they just you just hired them and they they onboarded you know like 5 seconds ago.
Speaker A
Uh and so how do you actually teach that person to be effective? So how you do that is through a skill. uh skill being just you know it's just markdown right but you know it encodes a lot of
Speaker A
information instructions a lot of uh you can really draw out a lot of intelligence from an agent by well some people on Twitter call it like you know pull the agent to a different latent space which is kind of like a fancy way
Speaker A
of just saying like since uh you know LLMs are sort of like they predict the next token uh when you give it some high quality tokens uh to begin with then you know it it can kind of pattern match on
Speaker A
like a higher space that's you know smarter. Um so that's like a very interesting model there but yeah I built Pstack very very incrementally. Uh so uh started with just really observing how agents you know all the fail different failure
Speaker A
modes of of that agents were having and every time I saw that I just okay I'm just going to make that a skill right like stop hallucinating actually go and search up look up the code use a lot of
Speaker A
sub agents uh and yeah stop guessing yeah that makes sense one one kind of followup question here uh both from myself and from a bunch of people in the chat so I guess It's two two parts. So one is
Speaker A
like how do you maintain these skills? So like the product changes over time. Obviously there's a lot of people who are shipping against the codebase. So how do these skills get maintained? Uh and then second to that is like how do
Speaker A
you know when your verification is is good enough? Uh like and you know you can trust that the ver verification loops that you've built are going to I guess you trust that the outputs uh when they're done.
Speaker A
Uh yeah maybe I'll talk about um I think somehat related. Maybe I'll start with this one first. So like how do I maintain these skills?
Speaker A
So um if you're not familiar with this concept, [clears throat] an eval is essentially like a way to uh well I the mental model I have is like it's like a unit test for an agent. Um and uh you
Speaker A
can actually make your own eval. You don't need like a special framework for them. You can build you can you can build one depending on like you know how scientific and how rigorous you want to be. Uh my screen is red.
Speaker A
Yeah, there's a little button. Um sorry. Do you mind like disabling the drawing or something? I I can't see my screen.
Speaker A
Yeah, sorry if you guys could not draw on the screen that'd be great. But um there's a little button in the I got a troll.
Speaker A
Yeah, the little drop down. How do I clear? Yeah, you got it. Perfect. Um, good to know.
Speaker A
Yeah. So, eval unit test your skills basically. And actually in Pstack, we ship uh under potato mode, there's a playbook if you search for it called Eval playbook. Um, and it's uh uh it's like not it's actually pretty
Speaker A
pretty rigorous the way it's done. Uh, but um, essentially what I do is I spawn a lot of different sub agents. I have like my main coordinator agent uh come up with a rubric for uh what I want the
Speaker A
skill to do. Um and then it spawns all these sub aents and it's it creates individual directories for them uh which are cleverly named to not let the sub aent know that it's being evaluated because uh agents can actually tell and
Speaker A
when they do they change their behavior. Uh, but it does a bunch of stuff like that to um essentially yeah like test whether or not the skill I'm making or changing is actually doing what I think it does. Um, and one of the really nice
Speaker A
things about cursor is that we are we we support so many different models. So you can actually eval your skill across all sorts of different models um and you know get a sense of how well it performs across that different matrix. um
Speaker A
especially for the models that you use. Uh so I do this a lot. Every time I I modify a skill, I will run one of these uh like the EBAL playbook uh and make sure that you know it's actually leading
Speaker A
to a result I want. Uh but I will say like maintaining skills is actually pretty hard. Uh it requires I think a lot of taste and observation. So you kind of need to be very good at being a backseat
Speaker A
driver. You know what I mean? Like if you do pair if you ever done pair programming for example, uh and you watch a coworker code and you just like you you could probably do this better.
Speaker A
You know, you could do you know like why did you not do this, right? You you ask a lot of questions to your coworker. And it's kind of a similar thing here. You like you don't want to just be a passive
Speaker A
observer of your agent. You want to be very in the driver seat in the initial stages when you're building up your own set of skills. uh you know obviously you can use something like PAC but if you're building your own set of skills it's
Speaker A
very I think you know opening up the all the tool calls and like reading the code and reading all the uh the agent behavior and their thinking blocks is a really great way to see where they they fail right like what what you know where
Speaker A
are they being done and then you can go and build a skill for that and then with verification how you trust it is it's I think it's also very similar iteration loop uh where you know like I actually
Speaker A
did the same process for verifying the verification skill where I actually get um so one thing that's interesting about eval is that you can sort of hill climb them meaning that uh your eval can produce a score right uh a score that
Speaker A
you can get your coordinator to produce uh but also comp uh you can have a judge agent of a different model to uh kind of cross reference and make sure that the first model is not being biased, right?
Speaker A
The model that's judging all of the sub aents that are running the thing. Uh but you can also like hill climb. So meaning that you can you can use like /loop in cursor and you can say okay keep looping
Speaker A
on this eval right until everything is 10 out of 10 as an example. Uh and I did the same the basically the same approach with the control skill. And so I kind of it was very it was very hands-off
Speaker A
actually. Uh so you know I uh I kind of built I built that skill that way like the CLI and that skill. Um and over time it's gotten really good. Uh but yeah it was definitely not super smooth at the
Speaker A
beginning. It required a lot of iteration and I think there's an analogy here for me which is um well I make this analogy later in a different slide on my drawing here. Uh but I think of it like
Speaker A
uh you know as a as a engineer now you're sort of more like you uh like maybe a manager or the analogy I like is like you're like a a chef in a restaurant. Uh you you're the head chef.
Speaker A
Uh you're not cooking all the food yourself anymore. You have a team of cooks, right? You have line cooks, you have a sue chef, you have, you know, all these different stations.
Speaker A
Um and it's your job to really design the environment. you know, you you're in charge of setting up the kitchen. You're in charge of, you know, like giving tasks to different people. So, um yeah, it's a very interesting way of
Speaker A
working. Uh but yeah, that's that's how I've basically built uh these verification skills. Yeah, just just one thought there on like to go try to go one layer deeper.
Speaker A
So are you let's say we wanted to build um an eval or a skill for for something and we wanted to kind of get better on its own which is is what I think you're suggesting. Uh are you doing that in
Speaker A
like a work tree kind of isolated with like the sub agents and and then the reviewer agent and and all that? Is it happening like in some type of cloud hosted environment? Like what's the the more the practical steps if I wanted to
Speaker A
go do this uh and like set up a verification system for something? What would I what would I do or where would I start?
Speaker A
Um I think that uh the best place to start is local because you can observe you can definitely observe what your agents are doing. So uh if you're building a verification skill for yourself uh I would definitely start
Speaker A
local and just have your agent bring up the application whether it's like a CLI or a desktop app or whatever. And so you can actually observe right you can see how the agent is interacting with the the application. you can see it, you
Speaker A
know, how it calls like the different APIs that that allow it to interact with the uh the application.
Speaker A
Um, but uh for me personally uh I have basically been kind of all in mostly all in on cloud agents because they're extremely powerful. Uh and the really powerful thing about cursor is the the cloud agents actually where if you spend
Speaker A
a little bit of time setting up your environment these control skills, these verification skills pay a huge amount of dividends because it's not just something that makes you as a single engineer better.
Speaker A
It actually levels up your whole team uh and even your whole company because uh you can actually start thinking about cloud agents. start thinking about automations that automatically do things like uh I I get I I kind of talk about
Speaker A
this a bit later, but I'll just kind of get into it. Uh where where you know, for example, like I talk a lot about this agent we have called Benny, right?
Speaker A
who who uh you know takes all of the bug reports that we get and it automatically goes off in the cloud, opens up a cloud uh it's you know its desktop. It runs cursor in its own computer and it uses
Speaker A
the same control skills to interact with the application and try to reproduce the bug uh or the user report, right? And this is so so powerful because at once I can immediately I I get so much information from this automatically like
Speaker A
here in this example you can see that uh the Benny actually reproduced the bug uh but it's already fixed on main. So it actually confirms that we fixed this problem already and all I need to do is just release another build of of cursor.
Speaker A
Uh so that's like huge information there that I didn't have to go off and sit with an agent, you know, and spend an hour trying to figure like is this fixed? this is not fixed. So you you you
Speaker A
gain back so much time. Uh but you know everybody on my team benefits from this.
Speaker A
Everybody at the company benefits from this. Uh so definitely think that uh you know keeping these uh using cloud agents is super powerful. Uh but yeah it's like a journey. You have to trust it first right before you you get to this point.
Speaker A
And that's it goes back to what I was saying here where you know it's very hard. It's almost impossible. And I would definitely encourage you not to try to jump from, you know, like if you're still in this zone, you don't
Speaker A
want to jump to like I'm going to spawn hundred of thousand or thousands of cloud agents right now because you're just going to waste a lot of tokens. Um, and it's going to be extremely expensive.
Speaker A
Yeah. So just to kind of recap so far, basically the if we wanted to go on the journey that you've kind of gone on, it would be to start with verification, building some some skills and some some ways of determining that the agents are
Speaker A
producing at least like correct code. Whether like you said whether it's good code or not is maybe a separate question, but like it's it's technically solving the problem by looking at you know stack traces, looking at you know
Speaker A
the the actual behavior in the app and so on. Um, and then once we trust it locally, then we can start to think about scaling into the cloud and running more agents that are picking up signals, I guess, on their own, right? So whether
Speaker A
that's like a bug report that comes in or something, they can go and pick it up and solve the problem and and give us back a PR. And then maybe the last step is like automerging the PRs, which uh is
Speaker A
where you're at, maybe not where we want to um and then and reviewing the one main, but um is that is that about right?
Speaker A
Yeah, exactly. I think yeah, that's why I drew this this uh this this curve, right? because that this this basically describes my journey of you know when I started barely could use a couple agents and I was just observing every single
Speaker A
thing. I think there's really no shortcut for going from here to there because this is really about your personal level of trust in agents, right? Um obviously, you know, as a as engineer, you don't want to just slop
Speaker A
code into production. So, how do you actually build up that trust? Takes um a lot of uh I guess taste and judgment. Um but uh you know, like I think plugins like Pstack definitely kind of help you uh get up to speed much quicker. Uh and
Speaker A
so I guess it's like if you trust me and you trust PAC then in by extension you can maybe trust your agents but if you don't trust me and I I definitely would not encourage people to blindly trust me
Speaker A
uh uh you know if you build up your own set of skills that you can obviously you know take a look at PAC and kind of fork it make it your own improve the skills definitely encourage that uh but for me
Speaker A
it's really all about it just keeps coming back to trust you Every one of us here in this chat have a different standard for engineering. Uh and there are different things that are important for us in our codebase and uh when you
Speaker A
are able to encode all of that into skills and you can verify that your agent is actually doing them that allows you to really kind of ascend this curve and um uh you know start automating things. Uh there's another piece I
Speaker A
wanted to talk about. Um if there's more Yeah, go for it. I'll I'll pick up more questions as I go. But yeah, I think there's a third part to this which I haven't talked about yet, which is kind of an interesting one
Speaker A
which is like refactoring and rewriting. Like one of the uh I guess most controversial one of the most controversial topics in the industry I think is like should you rewrite your app or not? Um because I think engineers
Speaker A
are very prone to this where especially when you join a company you come in and you see like the codebase and you're like oh man this is like who wrote this code you know it's terrible I want to rewrite the whole thing there is a
Speaker A
very common inclination and I think a lot of you know before agents um and I guess arguably even now people will definitely discourage you from re rewriting stuff but I'm actually here to make a case for why you might want to
Speaker A
consider Um because I think it really depends. Uh you know uh brownfield applications I think are actually in a pretty good spot especially if they're set up well already. Uh and like recently I've been talking to some people but uh you know I
Speaker A
I was just observing I I just noticed this parallel which is that a lot of big tech company problems are now everybody's problems. Um, and the big tech company problem, you know, like when I was working at Meta, like we had
Speaker A
this giant monor repo, we had like, I don't know, tens of thousands of engineers just, you know, like banging on their keyboards and and shipping code and a lot of really great engineers at Meta.
Speaker A
Uh, but, uh, I'll say like, you know, you'll be surprised that the code quality is actually not that good.
Speaker A
[laughter] Um, and so I often joke that like, you know, before AI slop, we had human sloth. Um and so uh you know I think a lot of big tech infra like uh like what Meta has or Google you know you know
Speaker A
really big tech companies are actually designed for that where you you're sort of like you're catering to the the you know like uh this sounds so bad to say but like the the least capable engineer on your team right you build you build
Speaker A
frameworks you build conventions you build guard rails you know you restrict credentials so that you know your intern doesn't wipe your production database.
Speaker A
Um there's uh you know if you have that level of infra already I think your agents can actually already do a very solid job right because they have the the guard rails are already in place for agents to not cause havoc or not cause
Speaker A
too much havoc uh in your codebase um and you can always add more you know guard rails. Uh but I think like green field applications especially are you know like the brand new applications are like the biggest risk in my opinion. Uh
Speaker A
and also the greatest opportunity because you know if you vibe code a project uh a prototype um like we did for Grockbot you know Grockbot was spun up very very very quickly. Um and if you if you haven't heard of of Grockbot it's
Speaker A
like a a new application we just launched yesterday. Uh it's it's really cool. uh lets you orchestrate your create like individual agents that have their own identity and you can kind of orchestrate them. It's super cool.
Speaker A
Definitely check it out. Um but yeah, that was it's like a very it was a very green field application like most prototypes are so like vibe coded very quickly. Humans were not reading the code at all. And uh I had this tweet
Speaker A
recently uh where I said something about organic architecture. Um maybe I'll find it. Uh but the idea is that uh when you have a completely vibe coded application, you essentially have no guardrails whatsoever. So uh your agents when you give them a task, they will
Speaker A
just solve it in whatever method is the most convenient. And over time you get into this uh situation where you have a codebase that is spiraling out of control because you don't understand it.
Speaker A
Uh your agents understand it I guess in a way but like they've built something that is you know optimized for short for shortcuts. Uh and uh you know it will you will suffer you'll have a lot of of
Speaker A
issues with that application. Uh so I think starting your codebase with uh like very strong constraints is very much needed uh because like when you have a codebase that you can trust, right? when you have guardrails that actually help you uh uh help your agents
Speaker A
write good code, you can get into the, you know, like into this part of the curve where I I where like I I said, you know, I woke up today and I had like 20 PRs merged u by my agents and that's
Speaker A
because I invested a lot a lot of time uh over 600 PRs I I I calculated yesterday uh when I refactored all of Grockbot to this new architecture that I've been thing. Um, and yeah, I've gotten to a point where I
Speaker A
I don't really look I really don't look at the code anymore. And um, I say that not just, you know, to sell you tokens, but because I, you know, it it it took a lot of work to get to that point. I
Speaker A
spent a lot of tokens to get the codebase to this point where I no longer have to look at it. Uh, but I'm very excited because, you know, of the potential where, you know, it's not just this doesn't just benefit me. It
Speaker A
benefits everyone contributing to Grockbot and it also empowers you know designers and product managers and you know pe uh even GTM people to add features to grabbot and I don't have to worry you know I don't have to to wake
Speaker A
up at night in in the middle of the night and worry like oh someone's just merged a perf regression right I have a ton of constraints and CI it's like it's actually very annoying to write code in in graph web but like
Speaker A
agents absorb all of that annoyance Um but yeah, I'm happy to talk about what exactly that is. Um yeah, I think one question um before we get into the this part here is just around that element of like what
Speaker A
your your your CI looks like or maybe some of the constraints and then also like the average PR size. I saw a question about that earlier just to give people you know kind of a a glance. It doesn't have to be like mathematically
Speaker A
average but just a you know like what generally the size of the a PR is. um if it's only a couple lines of code or you know um yeah um I think it depends uh let me I'm trying to do this in a way where I'm
Speaker A
not going to like you yeah you don't have to share the actual number like an actual average I think this is fine benchmark but like we have so okay this is not that interesting but uh well fun fact is
Speaker A
that virtualization in grogbot and in uh cursor is actually powered by uh pretext uh which is a sort of new library that someone's built. Um that's really interesting. You should you should check it out. But that's not really that
Speaker A
important. Uh I think the average PR size I actually don't know. I I don't know if I want to click on these. Uh I probably can, but I would say [clears throat] like they can range anywhere from a few hundred lines or 50
Speaker A
lines to like a thousand depending on what the thing is doing. Uh, so like here I'm actually like deleting a bunch of files. So I expect that it's just this like mostly deletion. Uh, but yeah, it kind of varies.
Speaker A
There's no like Yeah, there's no like hard cap or hard limit. Are there all like 50 line PR?
Speaker A
There's no hard cap. Yeah, there's definitely no hard cap, but I I do encourage my agents to split up their work into multiple PRs. Uh I do that mostly because uh I like I like the idea of the I guess maybe this is much harder
Speaker A
to do now as in the world of agents and you have like so many commits but I like the idea that you know the git history is a very rich source of context. Uh, and I like the I like each PR to sort of
Speaker A
atomically describe what that small piece of thing is doing, which also makes it easier for me to revert changes and like figure out, you know, oh, I shipped a bug and it's just it's here, right? It's not in this 40,000 line PR
Speaker A
where you who knows what landed in there. Uh, but I don't have a hard cap on PR size.
Speaker A
Cool. And then um yeah, also quick question on like CI. So again, you don't have to go into like uh the screen share of like your CI does, but just generally would you describe what the CI kind of
Speaker A
looks like uh or how strict it is? Uh yeah. So uh well specifically for Grockbot. So Dune is the is the sort of cheeky code code name for the architecture that we've built for Grockbot. Um the CI looks pretty
Speaker A
annoying because there's checks for everything. So like literally I have um uh well if you've written any react for example you know you know that one of the biggest foot guns in react is use effect. Uh so in
Speaker A
uh Dune and in graphbot we've banned use effect. So Dune is just you can the the mental model of what Dune is uh you can kind of think of it as like Nex.js JS for uh electron apps and it's designed
Speaker A
for agents to write uh and it's like custom for you know our agent powered applications. Um so the CI checks are very like specific to that like you know don't use use effect it's it's it's banned like CI will fail uh and yell at
Speaker A
you. We have like some of the more interesting ones that people might raise eyebrows is like I actually ban code comments as well uh which is very interesting. Uh, but I've noticed that 99% of the time agents just write code
Speaker A
comments that kind of describe some historical thing that is actually totally irrelevant to the code. Um, like it will often say like, you know, oh, Lauren said you should never do this and it's now in in a code comment. Like
Speaker A
what? Like why what that was? I didn't say that as like a durable, you know, global rule. I just meant like your this PR sucks and you should change that part.
Speaker A
But agents don't really understand us that well surprisingly uh and or they kind of assume too much and they kind of do things in like very stupid ways. So like yeah we just ban everything.
Speaker A
Everything you can imagine like the agents are bad at we ban. Uh so one example that we actually suffer a lot in the agents window is we have uh you know you if you've used the agents window you've definitely seen performance
Speaker A
issues and you know we're constantly trying to fix them. Uh but it's like a it's a never- ending struggle because there's so many pull requests that get merged. Every any one of them could just regress performance or stability or
Speaker A
reliability. Uh you know the agents window doesn't have this architecture yet. I plan to do bring this learning back there and kind of refactor everything there. uh but uh it just regresses super often uh because uh there's just one example is like we have
Speaker A
very poor um isolation between processes. So like on you know on on Electron you have a renderer thread that renders your UI but you also have like a main thread that you can run other code that you know doesn't need to block the
Speaker A
renderer. Um, but we do a poor job of separating those things. And so oftentimes you just accidentally have code that gets pulled into running on the renderer thread and then all of a sudden you're competing with the the renderer that you know that
Speaker A
has a very if you want like 60 fps you have to every frame that gets drawn has to be done in 16 milliseconds. So very very small you know deadline per frame uh if you want you know a very smooth
Speaker A
product. Uh and when you start building bringing in accidentally bringing in you know things that are like very computationally heavy or they have a lot of IO uh then you just get into like a lot of jank right your your FPS really
Speaker A
drops you start uh you know losing frames you get long tasks that take more than 16 milliseconds and you just get this really choppy experience.
Speaker A
So all of those patterns that we've learned basically building electron apps we've encoded into this framework and it becomes like a hard failure. So I literally in in grabbot we literally have a directory called electron main electron renderer and we have uh import
Speaker A
uh CI I guess where we actually check the dependency graph to make sure you're not accidentally importing code from one directory to another. Uh so that's enforced by CI um as well as bug bots uh which is our which cursors um like code
Speaker A
review tool that runs on CI uh you know in our agents MD it's everywhere like so I I I I um I have this thing here where I I talk about like um you know like there are multiple layers I think for
Speaker A
building a good codebase. Uh obviously the codebase is one where uh if you have an architecture like this where it's extremely strict uh you know the the the way to build features is very conventional that's like the strongest
Speaker A
strongest level of enforcement because agents just love to copy existing patterns. So, uh, one example of this in Rockbot is like we have this these concepts called like a feature and we have entry points and transcript cards like oh you know the cards that you see
Speaker A
in the chat these are all like like nouns I guess in in the framework and so there's a very conventional way of creating them and so like a feature is all in in a single directory as an example and so all of the code that
Speaker A
contributes to that feature lives in one directory so it's all coll-located in one case makes it super easy. You know, agents don't have to like uh grap around and try to figure out like where all the things are. It just looks at the feature
Speaker A
and like, oh, okay, I'm working on the onboarding feature in Grockbot. Uh I'm just going to work in this directory.
Speaker A
And for 80% of the work, it's mostly just very encapsulated there. [clears throat] But uh like it's like very it's like designed again for you know like the dumbest agent like you don't have to think, right? the the the one of the key
Speaker A
principles I have for this framework is like the shortest the shortest path is the best path.
Speaker A
So uh because that plays exactly to how agents love to write code is like they like to take shortcuts really you know they they'll find the quickest way to solve the problem. So why not make that the best way to solve the problem? Uh so
Speaker A
I I I probably won't get into all the specific details. Um and uh the this framework is really more of a collection of ideas and principles rather than something that will open source. Uh you can you can you know screenshot this I
Speaker A
guess if you want and uh tell your agent to uh do some build build something like this for you too.
Speaker A
Um yeah, but it's really all about the layers uh you know like the the codebase is one part with features uh and directories and you know import or blocking import dependencies uh that shouldn't be imported uh but and and it
Speaker A
all enforces that and static analysis. So like uh there's CI checks. We have a lot of lints for bad patterns that we observe. Uh compiler diagnostics. Uh there's also rules and bugbot which are um I think like three, four, five are
Speaker A
more soft, right? These two actually make make CI red, right? So that you know there's a hard constraint where the agent can't just write crappy code for rules and skills and bogbot. Your agents can still forget, right? You can
Speaker A
still or it may not always consistently apply them. So I like to layer them, but I don't I don't like to rely on them as the only source of enforcement because it's very very soft, right? And if you
Speaker A
if you only have rules and bug bot and skills and a style guide for your code, you will it's only a matter of time before your codebase looks like complete trash. I'm sorry to say that but uh I
Speaker A
definitely recommend yeah like you know investing in you know things that can be hard and forced right and this is why you know maybe the choice of tech stack that you use is also very important. Um like I think for example Rust is sort of
Speaker A
making you know it's like getting super popular again uh because the compiler is so strict right the compiler enforces so many different things you know there's a borrow checker that you have to appease and if as long as you make sure your
Speaker A
agents don't write unsafe code blocks uh you can more or less feel somewhat confident that if the code compiles it probably works and it's good. Uh but you see it gives you that level of trust and confidence that you as a human engineer
Speaker A
no longer need to go and check it yourself. You know you you rely on code and static analysis to actually make that uh a lot smoother. Um, and I I guess the worst part, the worst place to be in is if you are stuck in code review
Speaker A
land where you actually enforce all of the constraints, the invariance in your codebase by literally the human person saying, you know, reading the code and like, okay, you should not do this, right?
Speaker A
Every time you have to do that, you should consider that as a code smell, like a anti- pattern. And you should say, okay, instead of me commenting on the PR, how do I turn this into a hard rule, right? How do I turn this into a
Speaker A
lint rule, how do I turn this into a CI failure, or how do I even categorically eliminate this problem uh entirely? Uh I I can talk about another migration I've done, but I'll probably pause here.
Speaker A
Sure. Yeah. I feel like that's that's where I am to be honest is is what you're describing right now which is that like I don't have all of these rules. So I have some things to go do after this
Speaker A
session in terms of being able to scale my agents. I'm I'm definitely on like the uh you know maybe a couple of parallel ones locally stage. So like two to three locally and I'm sure most people here are on the same. So uh yeah
Speaker A
I know we only have a couple minutes left. Lauren, was there anything else that you wanted to to highlight? I obviously there's lots of questions so I can grab more but I want to give you a few minutes if there's anything else you
Speaker A
want to talk about. I think I've been yapping for quite a lot so I'm maybe let's just do questions.
Speaker A
Okay cool. Uh one question that had uh a couple of uh came up a couple times was just around like token usage.
Speaker A
So the the question is like is is what you're describing a realistic thing for people who are on you know uh a normal set of token usage. They don't have you know basically unlimited tokens uh to work with.
Speaker A
I think that's a really good point. I mean, like obviously, you know, I work at a AI lab where we have unlimited tokens. So, uh I definitely cannot say that, you know, this is something everyone should do in the exact same way
Speaker A
that I did it. I think it's possible to get to this point without, you know, breaking the bank.
Speaker A
But, you know, if you're like an engineering leader or, you know, you're you you have a startup that you lead, um I think to me it's a question of ROI. Um and it's like uh yes you spend a lot of
Speaker A
money on tokens in the upfront stage you know like refactoring a codebase is going to take a lot of tokens uh adding all these things uh is going to take a bunch of tokens but if we're heading to
Speaker A
a world where agents are writing all the code and you know you want to be very lean right you don't want to have to hire you don't want to be you don't want to become like meta right like I mean
Speaker A
like in terms of you don't want to become a 10,000 person engineering org because I mean that's a cool problem to have but also you you have so much overhead there's like planning you know like you it's it's a personally I I
Speaker A
wouldn't uh it it's not super fun but um I think you want to stay very nimble right and you want to you want to be like agents are all about allowing you to do things that you couldn't do before
Speaker A
that's really to me like the value of agents you know it's not just storing tokens on every single little thing but um to me like the thing I couldn't do before is like enforce this level of constraints in a codebase by myself
Speaker A
right like I'm just a single person you know uh it would have taken me years to build this framework uh and do all the refactoring and test everything myself and verify you know like run imagine if there it was just me right now in in
Speaker A
pre- agent era just like running you know by it would take me so long, right?
Speaker A
And my salary is pretty high, right? Like, so, you know, the the question I think an engineering leader might have is just then, you know, like what is there's a trade-off of do you hire someone to do this or do you spend the
Speaker A
tokens to set up a code base so that even the the most naive, right, the dumbest agents can do a good job. And when you actually get to this point like even agents that are not you know fable
Speaker A
size do an excellent job of writing code and this pays a lot of dividends as well for me personally where I've empowered not just myself but again like PMs designers engineers who are not familiar with Grockbot to just contribute in a
Speaker A
way that is sustainable. So, I think yeah, it's definitely like a trade-off for sure. You know, like nothing is like free for sure. Uh, and tokens are pretty expensive. Uh, but oh, actually, uh, I I I don't know how many
Speaker A
of you have seen this, but we actually announced Grock 4.6 today. So, very exciting. Finally out. Um, so yeah, Groth 4.6 would be like a great it was very very smart. Uh, it's really good on the on the benchmarks. Uh, and it's the
Speaker A
same the tokens. Uh well uh I hopefully I'm not saying this incorrectly, but uh I believe the cost per token is the same as 4.5. So you're actually getting more intelligence for the same cost. Uh I think this is an area that cursor tries
Speaker A
to cursor and SpaceX AI try to really optimize for like that heredto frontier of you know cost versus intelligence. uh you know we don't necessarily want to build the biggest model ever because that is extremely expensive to run. It's
Speaker A
really about like how do you find that sweet spot right? You don't you don't need a giant model but it's just super smart right and it's not very expensive for inference.
Speaker A
Uh but um yeah, I think to kind of round it up, um I think it's like a it's it's there's a if you do your own analysis, I feel like it's pretty positive. It it'll be pretty positive that the ROI you get
Speaker A
from investing in stuff like this, uh just empowers not just yourself, but your whole team to be so much more productive, right? Like imagine if you have an army of engineers like me who are shipping so much improvements and
Speaker A
and bug fixes. uh you know every day, right? Like that is pretty exciting. Cool. Uh one last question before we wrap up. This one is for the people in product on the on the call.
Speaker A
So let's say we do have an army of engineers who are shipping like Lauren.
Speaker A
I'm just curious like how is the product team or other functions of your company keeping up given that like if you're shipping so quickly, have are they using AI more to do their jobs? like as much as you can speak to that obviously you
Speaker A
don't have like you're not in that role but just curious about how that works.
Speaker A
Um I think this is where Grothbot has been actually exceedingly powerful. Uh where so before grabbot like you know uh obviously cursor only had cursor like we only had agents window we had a CLI we had an IDE and these are really like
Speaker A
power user tools right like devel they're designed for developers so it's very very developerentric you can do knowledge work in them but it like the UI is not really optimized for that. So we actually didn't really have uh well I
Speaker A
think like a lot of people like you know GTM product like they might have used cursor uh to do their work but it definitely wasn't like a delightful experience for them. Um I think now with Grogbot uh it's become Grogbot is basically like
Speaker A
their kusher moment for people who are not in tech in my opinion like it's like it's like a very very accessible way to use agents in a very comfortable very familiar interface. It looks like iMessage. Um and it's very fun to you
Speaker A
know you can give your agent a fun name. uh you can have you can kind of do orchestration with in a very like natural way where you can sort of you know each agent is like a person right
Speaker A
now you got a team of agents like working on you have one one agent per account that you manage as an example or if you're a PM you have you know you can have an agent that summarizes all the
Speaker A
work that Lauren did last night and then now you know what I did right so I think our PMs are leveraging that a lot and they're shipping code too uh so you know like often times they will just say oh
Speaker A
here's a bug I fixed it can you look at it and then I'll go review it and actually it's just perfect. I'm like okay stamp. Uh so uh that I think that shows that you know the the Dune
Speaker A
architecture is holding up right the all the the really strict constraints allow people who are not experts in engineering to contribute at a high level. Uh so I'm I feel like I'm already seeing that pay off a lot where uh you
Speaker A
know designers and PMs are just able to to to ship features directly. Um, and that just makes the Grockbot team super fast, right? Where we can ship so quickly. Um, and we have a lot planned.
Speaker A
So, I'm very excited uh to, you know, uh to to ship more ship more
Topics:AI coding agentsagent verificationCursorautomerge PRsevalsReact performancetrust in AIdeveloper productivitycode reviewagentic software engineering
![[YBA] frosti — Transcript](https://i.ytimg.com/vi/KAH0o3xSL3w/maxresdefault.jpg)
