Speaker A
Hey everyone, welcome. I'm here to talk about everyone's favorite subject, vibe coding. And somewhat controversially, how to vibe code in prod responsibly. So let's talk about vibe coding and what this even is. So first of all,
Eric from Anthropic discusses vibe coding in production, its challenges, and how to responsibly leverage AI-generated code for real products.
Key Takeaways
- Vibe coding is more than just AI-assisted coding; it’s about fully leveraging AI to generate code while managing risks.
- Responsible vibe coding requires focusing on product functionality and safety rather than micromanaging every line of AI-generated code.
- Tech debt and security remain major challenges when using AI-generated code in production environments.
- The evolution of vibe coding parallels the historical adoption of compilers, suggesting eventual widespread acceptance.
- AI’s exponential improvement will transform software development, necessitating new workflows and trust models.
Summary
- Eric, a researcher at Anthropic, introduces the concept of vibe coding and its growing popularity beyond traditional engineering.
- Vibe coding involves extensive use of AI to generate code, sometimes without fully understanding the underlying implementation.
- The talk highlights the risks and downsides of vibe coding, including tech debt, security issues, and unpredictable behavior in production.
- Eric shares his personal experience using Claude AI to write code while recovering from a hand injury, emphasizing tight feedback loops.
- He references Andrej Karpathy’s definition of vibe coding as fully embracing AI-generated code and forgetting the code itself exists.
- The analogy to early compilers is used to illustrate how developers initially distrusted automated tools but eventually embraced them.
- The importance of balancing trust in AI with maintaining core architecture that engineers deeply understand is stressed.
- Eric discusses the exponential growth in AI’s capability to generate longer, more complex tasks and the implications for software development.
- He advocates for safe and responsible vibe coding in production by focusing on product outcomes rather than code details.
- The talk concludes with optimism about AI tools like Claude 4 improving trust and productivity in coding workflows.
Full Transcript — Download SRT & Markdown
Speaker A
I'm Eric. I'm a researcher at Enthropic uh focused on coding agents. Uh I was the author along with Barry Zang of building effective agents where we outlined uh for all of you our best science and best practices for creating agents no matter what the application is. Uh this is a subject
Speaker A
I'm Eric. I'm a researcher at Anthropic focused on coding agents. I was the author, along with Barry Zang, of Building Effective Agents, where we outlined for all of you our best science and best practices for creating agents no matter what the application is. This is a subject
Speaker A
out how to make this happen effectively uh was really important to me and I was luckily able to figure that out well and sort of help u bring that into a lot of anthropics other products and in our
Speaker A
that's near and dear to my heart. Last year, I actually broke my hand while biking to work and was in a cast for two months, and Claude wrote all of my code for those two months. And so figuring
Speaker A
it's a lot of AI and a lot of the code is coming from the AI rather than them writing itself. But I think when you are still in a tight feedback loop with the model like that, that isn't truly vibe
Speaker A
out how to make this happen effectively was really important to me, and I was luckily able to figure that out well and sort of help bring that into a lot of Anthropic's other products and in our
Speaker A
important is that vibe coding was when people outside of the engineering uh industry really started getting excited about code generation. Copilot and cursor were great but only sort of for uh engineers but someone that didn't know how to code uh suddenly with vibe coding they could find
Speaker A
models through my research. So let's first start talking about what is vibe coding. A lot of people really conflate vibe coding with just extensive use of AI to generate your code. But I think
Speaker A
and you said, "Hey, you know, random things are happening, max out usage on my API keys, people are bypassing the subscription, creating random [ __ ] on the DB." Uh, and so, you know, this this is kind of the downside of vibe coding of what started happening. And the positive sides
Speaker A
this isn't quite true. A lot of people, you know, they're using Cursor, they're using Copilot.
Speaker A
stakes are really high if you do it for a real product? And the most successful cases of it are kind of these toy examples or fun things where the stakes are very low. And my answer for why we should care about vibe coding is because of the exponential. The length of tasks that AI can do is
Speaker A
It's a lot of AI, and a lot of the code is coming from the AI rather than them writing it themselves. But I think when you are still in a tight feedback loop with the model like that, that isn't truly vibe
Speaker A
as the AI is writing a lot of your code. But what happens next year? What happens the year after that? When the AI is powerful enough that it can be generating an entire day's worth of work for you at a time or an entire week's worth of work, there is no way that we're going to be
Speaker A
coding. When I say vibe coding, I think we need to go to Andrej Karpathy's definition, where vibe coding is where you fully give into the vibes, embrace exponentials, and forget that the code even exists. I think the key part here is forget the code even exists. And now the reason this is
Speaker A
give into this and find some way to leverage this task. Um, I think my favorite analogy here is like compilers. I'm sure in the early day of compilers, a lot of developers, you know, really didn't trust them. They might use a compiler, but they'd still read the assembly that it would output to make
Speaker A
important is that vibe coding was when people outside of the engineering industry really started getting excited about code generation. Copilot and Cursor were great but only sort of for
Speaker A
my challenge to the whole software industry over the next few years is how will we vibe code in prod and do it safely? And my answer to that is that we will forget that the code exists but not that the product exists. Thinking again to that compiler analogy, you know, we all
Speaker A
engineers, but someone that didn't know how to code suddenly, with vibe coding, they could find themselves coding an entire app by themselves. And this was a really exciting thing and a big
Speaker A
level with software. And one thing I really want to emphasize is that this is not a new problem.
Speaker A
unlock to a lot of people. Now, of course, there were a lot of downsides of this, and you had people coding for the first time and really without knowing what they were doing at all.
Speaker A
in financial accounting? And these are all, you know, problems that have existed for hundreds or thousands of years and we have solutions to them. A CTO can still write acceptance tests uh for an expert uh that works for them even if they don't understand the implementation under
Speaker A
And you said, "Hey, you know, random things are happening, max out usage on my API keys, people are bypassing the subscription, creating random stuff on the DB." And so, you know,
Speaker A
that they do understand and slices of the data so that they can build confidence in the overall financial model even though they themselves might not be an expert in how the entire thing flows.
Speaker A
this is kind of the downside of vibe coding of what started happening. And the positive sides of vibe coding that you'd see were all things that were really kind of low stakes. It was
Speaker A
individual contributors where we understand the full depth down to the stack. But that's something that in order to become most productive, we are going to need to let go of in the way that every manager in order to be most productive is going to need to let go of some details. And just
Speaker A
people building video games, building sort of fun side projects, things where it's okay if there was a bug. So, you know, why do we even care about vibe coding if it seems like something where the
Speaker A
knowing the implementation underneath it. Now I have one caveat to that today which is tech debt.
Speaker A
stakes are really high if you do it for a real product? And the most successful cases of it are kind of these toy examples or fun things where the stakes are very low. And my answer for why
Speaker A
is one of those rare things where there really isn't a good way to validate it other than being an expert in the implementation itself. So that is the one thing that right now we do not have a good
Speaker A
we should care about vibe coding is because of the exponential. The length of tasks that AI can do is doubling every seven months. Right now, we're at about an hour. And that's fine. You don't need to
Speaker A
parts of our system that uh nothing depends on them. they are kind of the end feature. They're the end beller whistle. Um rather than things that are the branch or trunks beneath them like here in white. Uh here the orange dots are all these leaf nodes where honestly if you have a system
Speaker A
vibe code. You can have Cursor work for you. You can have Claude code write a feature that would take an hour.
Speaker A
core architecture that we as engineers still need to deeply understand because that's what's going to change. That's what other things are going to be built on and it's very important that we protect those and make sure that they stay extensible uh and understandable and flexible.
Speaker A
And you can review all that code, and you can still be intimately involved as the AI is writing a lot of your code. But what happens next year? What happens the year after that? When the AI is powerful enough that it can be generating an entire day's worth of
Speaker A
more and more to write code um that is extensible and doesn't have tech debt. Um using uh you know the clawed 4 models uh over the last week or two within anthropic has been a really exciting thing and I've I've given them much more trust uh than I did with uh 3.7. Um, so I think that this is going
Speaker A
work for you at a time or an entire week's worth of work, there is no way that we're going to be able to keep up with that if we still need to move in lockstep. And that means that if we
Speaker A
for you but what you can do for Claude. I think when you're vibe coding you are basically acting as a product manager for Claude. So you need to think like a product manager. What guidance or context would a new employee on your team need to succeed at this task? I think a lot of times we're
Speaker A
want to take advantage of this exponential, we are going to have to find a way to responsibly give into this and find some way to leverage this task. I think my favorite analogy here is like
Speaker A
"Hey, implement you know this feature," there's no way you'd expect them to actually succeed at that.
Speaker A
compilers. I'm sure in the early days of compilers, a lot of developers really didn't trust them. They might use a compiler, but they'd still read the assembly that it would output to make
Speaker A
all of that same context and is set up to succeed. When I'm working on features with Claude, I often spend 15 or 20 minutes collecting guidance into a single prompt and then let Claude cook after that.
Speaker A
sure it looks how they would write the assembly. But that just doesn't scale. You know, at a certain point, you start needing to work on systems that are big enough that you just have
Speaker A
It's looking for files. We're building a plan together that captures the essence of what I want, what files are going to need to be changed, what patterns in the codebase should it follow. And once I have that artifact, that all of that information, then I give it to Claude, either
Speaker A
to trust the system. The question, though, is how do you do that responsibly? And I think my challenge to the whole software industry over the next few years is how will we vibe
Speaker A
you need to be able to ask the right questions. And uh despite the title uh of my of my talk, I don't think that vibe coding and prod is for everybody. I don't think that people that are fully non-technical should go and try to build a business fully from scratch. I think that is
Speaker A
code in prod and do it safely? And my answer to that is that we will forget that the code exists but not that the product exists. Thinking again to that compiler analogy, you know, we all
Speaker A
We recently merged a 22,000line change to our production reinforcement learning codebase that was written heavily by Claude. So how on earth did we do this responsibly? Uh and yes, this is the actual screenshot of like the diff uh from GitHub for the PR. Um the first thing is we,
Speaker A
still know that there's assembly under the hood, but hopefully most of us don't need to really think about what the assembly actually is. But we still are able to build good software
Speaker A
There was still days of human work that went into this of coming up with the requirements, guiding Claude and figuring out what the system should be. And we really really embraced our roles as the product manager for Claude uh in this feature. The change was largely concentrated
Speaker A
without understanding that assembly under the hood. And I think that we will get to that same level with software. And one thing I really want to emphasize is that this is not a new problem.
Speaker A
of those parts. And lastly, we carefully designed stress tests for stability. Uh, and we designed the whole system so that it would have uh very easily human verifiable inputs and outputs. And what that let us do these last two pieces is it let us create these sort of verifiable checkpoints
Speaker A
How does a CTO manage an expert in a domain where the CTO is not themselves an expert? How does a PM review an engineering feature when they themselves can't read all the code that went into
Speaker A
Uh and we were able to verify correctness based on the input and outputs of the system that we designed it to have. So basically we designed this system to be understandable and verifiable even without without us reading all the code. And so ultimately by combining those things we were
Speaker A
it? Or how does a CEO check the accountant's work when they themselves are not an expert in financial accounting? And these are all problems that have existed for hundreds or
Speaker A
write this entire thing from hand uh by hand and review sort of every line of it. Um, and I think one of the really exciting things about this is is not just that this saved us, you know, a week,
Speaker A
thousands of years, and we have solutions to them. A CTO can still write acceptance tests for an expert that works for them even if they don't understand the implementation under
Speaker A
much bigger features and much bigger changes. uh sort of like the marginal cost of software is lower and it lets you consume and build more software. So I think that was the really exciting thing about this is not just saving the time but now kind of feeling like oh things that are going
Speaker A
the hood. They can see that these acceptance tests pass and that the work is high quality. A product
Speaker A
you can do for Claude. Focus your vibe coding on the leaf nodes, not the core architecture and underlying systems so that if there is tech debt, it's contained and it's not in important areas.
Speaker A
manager can use the product that their engineering team built and make sure that it works the way they expected, even if they're not writing the code. And a CEO can spot check key facts
Speaker A
know, demanding that you read every single line of code uh or write every single line of code.
Speaker A
that they do understand and slices of the data so that they can build confidence in the overall financial model even though they themselves might not be an expert in how the entire thing flows.
Speaker A
good at this. So overall that is uh vibe coding and prod responsibly. uh and I think this is going to become one of the biggest challenges for the software engineer for the software engineering industry over the next few years. Thank you. And I have uh plenty of time for questions. Yeah,
Speaker A
And so thinking about these examples, managing implementations that you yourself don't understand is actually a problem as old as civilization. And every manager in the world is actually already
Speaker A
of the agent KI? Yeah, so the uh I think this is a really interesting question and I think there are reasons to be very worried about this and also reasons to be very optimistic about this. I think the the reason to be worried like you mentioned is that you know we are not going to be there in
Speaker A
dealing with this. Just we as software engineers are not used to this. We are used to being purely
Speaker A
run really fast. Um I think the positive side of this is that I have found that I'm able to learn about things so much more quickly by using these AI tools. A lot of times when I am coding with
Speaker A
individual contributors where we understand the full depth down to the stack. But that's
Speaker A
Tell me about it. like what is it? Why did you choose it over another? And having sort of that always there pair programmer. Um like again I think what what's going to change is that people that are lazy are not going to learn. They're just going to glide by. But if you take the time
Speaker A
something that in order to become most productive, we are going to need to let go of in the way that every manager in order to be most productive is going to need to let go of some details. And just
Speaker A
market fit versus flops, we're going to be able to take so many more shots on goal. I feel like uh especially sort of like system engineers or architects over it takes, you know, oftentimes like two years to like make a big change in a codebase and really kind of come to terms with
Speaker A
like us as software engineers, you know, we let go of some of the details of understanding the assembly itself that's happening under the hood. And the way that you do this while still
Speaker A
long as they're putting in the effort to trying. Yeah. Going back to your pre-planning process, uh, what's the balance between giving it too much information and too little? Are you giving it a full product requirement document? Is there kind of a standardized template that you put together
Speaker A
being safe and being responsible is to find an abstraction layer that you can verify even without
Speaker A
are my requirements like this is what I want at the end. There's other times where I know the codebase well and I will go into much more depth of like, hey, these are the classes you should use to implement this logic. Look at this example of a similar feature. Um, I'd say it all
Speaker A
knowing the implementation underneath it. Now I have one caveat to that today, which is tech debt.
Speaker A
much effort into creating sort of a very rigorous uh, you know, format or anything. I would just, you know, think about it as like a junior engineer what you would give them in order to succeed.
Speaker A
So right now, there is not a good way to measure or validate tech debt without reading the code yourself. Most other systems in life, like the accountant example, the PM, you know,
Speaker A
Like there were reports a couple months back of like the top 10 vibecoded apps being super vulnerable and a lot of important information was released. Well, not released but proven to be released and the person who did it wasn't even like like a pro hacker and stuff and so
Speaker A
you have ways to verify the things you care about without knowing the implementation. Tech, I think,
Speaker A
being Claude's PM and understanding enough about the context to basically know what is dangerous, know what's safe, and know where you should be careful. And I think yeah, the the things that uh get a lot of press about vibe coding are people that have no business coding at all
Speaker A
is one of those rare things where there really isn't a good way to validate it other than being an expert in the implementation itself. So that is the one thing that right now we do not have a good
Speaker A
for our internal case of this this example, um it was something that's fully offline. And so we knew there weren't any like there were uh we were very very confident that there was like no security problems that could happen into this. Uh in our case it's like run in something that's
Speaker A
way to validate. However, that doesn't mean that we can't do this at all. It just means we need to
Speaker A
we Less than 0.5% of the world's population are software developers and software is an amazing way to scale ideas. So how do you think the products need to change to make it easier for people to v code and build software while also avoiding some of the things that we run into with people
Speaker A
be very smart and targeted, aware of where we can take advantage of coding. My answer to
Speaker A
the payment parts are built for you and all you have to do is sort of fill in the UI layer. Um, and you know, you can vibe code that and it basically gives you some nice fill-in-the-blank sandboxes where to put your code. Um, I feel like there's tons of things like that that could exist.
Speaker A
this is to focus on leaf nodes in our codebase. And what I mean
Speaker A
But uh maybe that's a good like product idea that someone should do here is is build some way to make like a provably correct hosting system that can have a backend that you know is safe no matter what shenanigans happens on the front end. But yeah, I hope people build good tools that are
Speaker A
complements to vibe coding. Hi. Um so for test driven development, do you have any tips because like I often see that cloud just splits out the entire implementation and then writes test cases.
Speaker A
um sometimes they don't they fail and then I just want you know I'm trying to prompt it to write the test cases first but I also don't want to like you know verify them by myself because I haven't seen
Speaker A
implementation yet so do you have an iteratable approach that you know have you ever tried it for yeah test driven development yeah yeah I I definitely uh test driven development is very very useful in vibe coding um as long as you can understand what the test cases are even without
Speaker A
that it helps claude sort of be a little bit more self consistent even if you yourself don't look at the tests. Um, but a lot of times, uh, I'd say it's easy for Claude to go down a rabbit hole of
Speaker A
writing tests that are like too implementation specific. Um, when I'm trying to do this, a lot of times I will encourage I will give Claude examples of like, hey, just write three endto-end tests and, you know, do the happy path, an error case, and this other error case. Um,
Speaker A
and I'm kind of like very prescriptive about that. I want the test to be like general and end to end.
Speaker A
And I think that helps make sure it's something that I can understand um, and it's something that, um, uh, that Claude can do without getting too in the weeds. I'll also say a lot of times uh when I'm vibe coding the only part of the code or at least the first part of the code that I'll read
Speaker A
is the tests to make sure that you know if I agree with the tests and the tests pass then I feel pretty good about the code. Um that works best if you can encourage Claude to write sort of
Speaker A
very minimalist endto-end tests. Thank you for the very fascinating talk. Um, I also appreciate that you've done what a lot of people haven't done and tried to interpret one of the more peculiar lines in Karpathy's original post, embrace exponentials. So, I wonder if I could pin you down a little more
Speaker A
and say, how would I know if I've embraced the exponentials? Like, what precisely means following that advice? And and to maybe put it down a little more in what I think it intends to mean, it sort of maybe alludes to this, the models will get better. Um, do you think there's
Speaker A
some legitimacy in saying just the fact that the models will get better doesn't mean they'll get better at every conceivable dimension we might be imagining we hope they'll they'll be in. Yeah. Uh, so yeah. So how do I embrace exponentials, sir? Yeah, absolutely. So the uh I think you
Speaker A
got close with sort of the the quote of uh keep assuming the models are going to get better, but it's a step beyond that. the the idea of the exponential is not just that they're going to keep getting better, but they're going to get better faster than we can possibly imagine. Um,
Speaker A
and that's kind of like when you you can kind of see the shape of the dots here. It's it's not just that it's getting steadily better, it's that it's getting better and then it's it goes wild. Um, I think the other funny quote I heard from this this was a I think in uh Daario and Mike Kger's
Speaker A
talk is uh machines of loving grace is not science fiction. It's a product roadmap. uh even though it sounds like something that's very far out like when you are on an exponential uh things get wild very very fast and faster than you expect. Um, and I think, you know, if you if you talk to someone
Speaker A
that was doing computers in the 90s, it's like, okay, great. We have a couple kilobytes of RAM.
Speaker A
We have a couple more kilobytes of RAM. Uh, but if you fast forward to where we are now, it's like we have terabytes. And it's like, it's not just that it got twice as good, it's that things got
Speaker A
millions of times better. And that's what happens with exponentials over a course of 20 years. So, we shouldn't think about 20 years from now is like what happens if these models are twice as good.
Speaker A
We should think about what happens if these models are a million times smarter and faster than they are today, which is wild. Like I we can't even think about what that means. In the same way that someone working on computers in the 90s, I don't think they could think about what would happen to
Speaker A
society if a computer was a million times faster than what they were working with. But that's what happened. And so that's what we mean by the exponential is it's going to go bonkers. All right. Yes. I got a couple well I got one question but it it's kind it's kind of two parts. The first
Speaker A
part when it comes to via coding I have like two different workflows. I have one where I'm in my terminal and then I have one when I'm in VS code or cursor. Um which which workflow do you use and
Speaker A
if you're using cloud code in the terminal how often do you compact? Because what I find is um my functions will get a new name as the longer I vibe code or you know just things kind of go
Speaker A
off the rails the longer I go and if I compact it still happens if I create like a document to kind of guide it I still have to you know get it back on track. Yeah. Yeah. Great question. Um I do
Speaker A
both. I often uh code with uh clawed code open in my terminal in VS Code. Um, and I'd say that like clawed code is doing most of the editing and I'm kind of reviewing the code um as I go in uh in VS
Speaker A
Code, which you know is not true vibe coding in the sense here. Uh, or maybe I'm reviewing just the tests uh from it. Um, I like to compact or just start a new session kind of whenever I get clawed to a good stopping point where it kind of feels like, okay, as a human programmer,
Speaker A
like when would I kind of stop and take a break and maybe like go get lunch and then come back.
Speaker A
If I feel like I'm at that kind of stage, that's like a good time to compact. So maybe I'll start off with having Claude find all the relevant files and and make a plan and then I'll say okay like
Speaker A
you know write all this into a document and then I'll compact and that gets rid of 100k tokens that it took to create that plan and find all these files and boils it down to a few thousand tokens.
Speaker A
Hey uh so one question is following up uh his previous question which is uh have you used other tools along with cloud code to like increase your speed a little bit more like running multiple cloud codes together using git work trees and then like sort of merging few things or stack PRs or
Speaker A
something like that. Is that something that you like personally follow or would advise to? Second question is um how do you like how do you very structurally and like in a very nice engineering like um like way approach a part of the codebase that you're not very familiar with but you want to
Speaker A
like ship a PR in it really fast and you want to do it in a really nice way and not wipe code it.
Speaker A
So yeah like what what are the what are your ways of like using cloud code to help do both these things? Yep. Uh, yeah. So, I definitely use clawed code as well as cursor. Um, and I'd say typically
Speaker A
I'll like start things with clawed code and then I'll use cursor to fix things up. Or if I was like if I have very specific changes, if I know exactly the change that I want to do to this file,
Speaker A
I'll just do it myself uh with cursor and sort of target the exact lines that I know need to change.
Speaker A
Um the second part of your question was um oh yeah like uh how to get spun up on a new part of the codebase. Um before I start trying to write the feature I use clawed code to help me explore
Speaker A
the codebase. So I might say like tell me where in this codebase off happens or you know where in this codebase something happens. Tell me similar features to this and like have it tell me the file names. Have it tell me the classes that I should look at. Um, and then kind of use that to try to
Speaker A
build up a a mental picture to make sure that I can do this and not vibe code. Make sure I can still get like a a good sense of what's happening. And then I go work on the feature with Claude. Uh,
Speaker A
thank you so much. I'll be still around and can Miller and answer other questions.
Topics:vibe codingAI code generationAnthropicClaude AIsoftware developmentcoding agentstech debtproduction codingAndrej Karpathycode safety
