Claude Engineering Lead: "Stop Using Claude Like It’s 2024"

If all we're doing is like summarize this email for me, you're not really benefiting from the new intelligence.

This is Felix Reeseberg. His team built Claude Cowork in about a week and a half. Now he leads engineering for Claude at Anthropic, and he doesn't write code anymore.

I'm just a conveyor belt for Claude. Claude needs to have access to all the crash data, and then needs to, like, figure out itself what it needs to fix and what it doesn't need to fix.

Is there still anything you would not trust an agent with? I rarely want the current session to judge itself. That, I think, is still really iffy.

How do I protect myself from AI just confidently presenting mistakes to me? It happens to a lot of us. I have, like, a very practical, easy answer.

Felix, thank you so much. I am very excited to have you. As a user of Claude, like an avid user, we're using Claude for all the operations on this channel. Every team member has their own project for the things they're responsible for. And as someone who runs, it's a pretty big team for a creator. I have 35 people, and I want them to be as efficient as possible. I want to produce as much as possible. But also, something that I've noticed this year, we were able to launch, I think, three new social media channels. So we launched a newsletter. We launched a LinkedIn a year and a half ago, because, you know, it's text-based. So AI has enabled me in many, many ways. And the goal of this conversation is to make me even more advanced and all the viewers and all the listeners.

I mean, from what I've seen you do, it seems like you're pretty advanced as it is.

But there is no borderline, right, to being as true as possible. That's true. Yeah. I think I'm discovering that about myself every single day. Like, one of the most fun things of working on Claude, by far, is this notion of, I built this product, and in comparison to my other products, it was pretty clear to me what you're going to use this for, right? Like, the example I always go with is, I worked on Slack, and I kind of have a feeling of, like, what are you going to do with this? The thing that's magical about working on Claude is that I see people really fly and do amazing things that I didn't consider possible, and it's, like, for a product builder, that's such a treat, you know, to see people fly with this thing that you made.

Is there something you've seen that's super ambitious and you're like, "Oh my god, I want to copy this."

I'm not sure about copying, but two days ago, um, there's, I think he's actually a product manager by trade. His name is Pavo, and he pointed Claude at a large amount of NASA data and asked Claude to, like, find them a new planet, and it did. So apparently it is not confirmed yet. NASA is working on actually confirming it. I'm on a little shaky ground here because I'm not an astrophysicist of any kind. But apparently Claude may have found a new planet we didn't know of before.

This is fascinating. Okay. And it's the kind of thing that I would have never considered. But if you hear the story, I think for us who work maybe on more knowledge-work things, right? The idea of, like, point AI at a large amount of data and you're going to find interesting things, that in isolation seems pretty reasonable, right? Like, something we do every single day. But the idea of, like, looking at all of the public data we have about space and considering a similar problem of, like, go and see if you find anything in here we haven't noticed before. So beautiful.

Can you talk to me about your day? You help build Claude Cowork, but you said you're not coding anymore. So what happens in the morning? What do you do?

Is that, like, your agents are briefing you on what's coming, or what is it?

That is true. My agents do brief me a little bit on what's coming for the day.

I've really leveled up in sort of altitude on the problems that I'm thinking about, right? Like, sometimes, these days, I feel like I'm much more concerned with human emotion, empathy, trying to understand how people work, in a way that I think I was less so as a classic engineering manager a couple of years ago. So I think for me, what I'm really discovering is that the amount of times I have to reach out to other professions to get their help has really dwindled down. I'm now this weird mix between a product manager, an engineering manager, just a traditional people manager, a data scientist. Right?

To answer your question very specifically, most days I'm busy with a lot of prototyping, right? We just try a lot of things. We sort of think about stuff. We sleep over it for a couple of nights, try to figure out if it's a good idea or a bad idea, and then we just go tell Claude to, like, build all the various variations and ideas we had, and then we need to do critique, try to figure out which of these have good ideas, which ones are bad. But I think prototyping is really what I spend most of my day on.

How do you decide what is a good idea? Like, what are the signals you're looking at?

This has been difficult for people for a long time now, and I'm not sure that I have all the answers to, like, what is a good idea, what is a bad idea. I think generally speaking, the way we look at good or bad ideas, the calculus has changed, right? Like, if you think about products we've built in the past, for the last 10 to 15 years, we always made implicit cost decisions, right? We sort of had to figure out, okay, especially for software, can I build this? What is the maintenance cost? What is the ongoing cost? Um, how much time do I invest in this? What is the opportunity cost? All of these decisions that engineering managers make to figure out, do I go and build this, yes or no? And I think now we can fast-forward to the end state much quicker and be like, all right, just build me a prototype, let me play with this, let's try it out. I think human attention is still rather limited. Let's say, for instance, you have a lot of taste in AI, right? Like, I can probably show you maybe one thing a day. I can easily build a thousand. I can't show you a thousand. Even if you're willing to sit down with me all day, at some point your eyes are going to glaze over and you're like, "Okay, I'm kind of done with this." So, what makes a good product? I think especially now that we live in this AI world, there's this old saying that, like, perfect is sort of when you can't take anything away anymore, and less so about building a product that has every single feature under the sun. Like, we can make as many buttons as you want us to, but I think good products actually have fewer buttons. I kind of know whether or not a product really works if we put it in front of a few people, and you see things click for them rather quickly and they can do things that they couldn't do before. That's really the signal I'm looking for.

And how do you think about the underlying problem that you're solving?

So for example, if you came to my company now, how would you think about the tasks that you would fully automate with AI versus just leave to humans, versus maybe just automating parts of it?

The first thing I want to say is that I think about automation and augmentation in slightly different ways. Automation can be really empowering if you automate away tasks that keep you from doing work that you enjoy, or if it's expensive, and automating it saves you a lot of money or time. In those cases, automation is useful, and I think it's really where AI currently probably provides most of the economic value, right? Like, when we evaluate whether this is worth it or not, we usually look at automation. The area I find most interesting, though, and I think where people really do things that weren't possible with AI before, is augmentation. It's this idea of AI helping you do things that you couldn't do by

Is that like your agents are briefing you on what's coming or what is it?

That is true. My agents do brief me a little bit on what's coming for the day.

I've really leveled up in sort of altitude on the problems that I'm thinking about, right? Like sometimes sometimes I feel like these days I'm I'm much more concerned with human emotion, empathy, trying to understand how people work in

a way that I think I was less so as a as a classic engineering manager a couple of years ago. So I think for me what I'm really discovering is that the amount of times I have to reach out to other

professions to get their help has really dwindled down. I'm now this weird mix between like a product manager, an engineering manager, just a traditional people manager, a data scientist. Right?

To answer your question very specifically, most days I'm I'm busy with a lot of prototyping, right? We just try a lot of things. We sort of like think about stuff. We sleep over it for a couple of nights, try to figure

out if it's a good idea or a bad idea, and then we just go tell Claw to like build all the various variations and ideas we had, and then we need to do critique, try to figure out which ones

of these have good ideas, which ones are bad. But I think prototyping is really what I spend most of my day on.

How do you decide what is a good idea? Like what are the signals you're looking at?

This has been difficult for people for a long time now and I'm not sure that I have all the answers to like what is a good idea, what is a bad idea. I think generally speaking, the way we look at

good or bad ideas, the calculus has changed, right? Like if you think about products we've built in the past for the last 1015 years, we we always made implicit cost decisions, right? like we we sort of had to figure out okay

especially for software can I build this what is the maintenance cost what is the ongoing cost um how much how much time do I invest in this what is the opportunity cost all of these decisions that engineering managers make to figure

out do I go and build this yes or no and I think now we can fast forward to the end state much quicker and be like all right just build me a prototype let me play with this let's try it out I think

human attention is still rather limited let's say you for instance you have a lot of taste in AI right like I can probably show you maybe one thing a day I can easily build a thousand I can't show you a thousand. Even if you're

willing to sit down with me for all day, like at some point your eyes are going to glaze over and you're like, "Okay, I'm kind of done with this." So, what what makes a good product? I think especially now that we live in this AI

world, there's this old saying that like perfect is sort of when you can't take anything away anymore and less so about like building a product that has every single feature under the sun. Like we can make as many buttons as you want us

to, but I think good products actually have fewer buttons. I kind of know whether or not a product really works if we put it in front of a few people and you see things click for them rather quickly and they can do things that they

couldn't do before. That that's really the signal I'm looking for. And how do you think about the underlying problem that you're solving?

So for example, if you're came to my company now, how would you think about the tasks that you would fully automate with AI versus just leave it to humans versus maybe just automating parts of it? The first thing I want to say is

that I think about automation and augmentation in slightly different ways. Automation can be really empowering if you automate away tasks that sort of keep you from doing work that that you enjoy or if it's like expensive and automating it a way like saves you a lot

of saves you a lot of money or time. In those cases, automation is is useful and I think it's really where AI currently probably provides most of the economic value, right? Like when we evaluate is this worth it or not, we usually look at

automation. The area I find most interesting though, and I think where people really do things that they weren't possible with AI before, is augmentation. It's this idea of AI is helping you do things that you couldn't do by hand because maybe you were

lacking skill or techniques or um just the raw knowledge to do something. And when I go to company, the thing that is often most impressive to me is less so what the tool does and more so how you

change your workflow. Does that make any sense? Yeah. AI is an interesting product because to really harness it, you kind of have to rethink how you do things. You have to like think about your outcomes. You have to

figure out what does good mean, what does bad mean, like how do I evaluate whether or not this thing that I'm asking for like whether the result is good. And the people who impress me the most are usually the ones who are very

good at coming up with like very good at reflecting on their own workflows. Yeah. But it's so hard when I'm thinking about my work for example and a lot of people use AI for drafting texts like how do you define a good text? It's like

is it still possible for like knowledge work cuz for coding I guess it's possible right cuz you know what you want you don't want any bugs but when it comes to knowledge work and like the result of human creativity how do you

even define what's good for the agent? Yeah, it's such a tricky problem. Like the the example I often go with is let's say your job is to like report on the weather, right? And you're like, "Agent, please figure out what the temperature

is outside." There's like two ways to look at that because you have to rethink what good means, right? Like if the model just guesses a temperature and is correct, that should not be a passing grade. That should be bad. We should

like rethink what we're doing here. But if the model is actually off by like half a degree or something, but it like um really figured out the right location, figured out some reputable data sources and like sort of the the

the way it got there was reputable and good, that should actually be better than like a gas good result, if that makes any sense. But for writing and like all the other outputs where you feel like a lot of taste is involved and

it's like a little hard to like prove or verify whether or not the result is like better than another. It is tricky, but I think I really enjoy talking to very good writers who are also a little AI

friendly. What do they say? It is it is fascinating to watch them work. My friend Adam Davidson, who was a staff writer at a number of like pretty good publications, he has all of these various prompts for Claude where he's

like, "Okay, like take my argument and put it to like magnify it by 100. Um, just mess with my text a little bit.

Like what are the things you disagree with the most? Like what is the strongest counterargument?" It's interesting because I think to me as someone who's not a writer I think the naive approach would be given a text claude make this good and I think what

I'm discovering more and more is that for knowledge work the secret sauce is like less and asking for a good output and sort of doing one step in between which is the best in the field how do they work and can we automate those

steps this would kind of mimic their workflow but still the taste isn't there that's why I think you still need a lot of yeah I think That's true. I think that's true. But the interesting thing is that if you talk to

really good writers, most of them would disagree that the text is really good just because they have like the magic touch. They would often say, "No, no, no. This is the result of hard work." Interesting.

Right. Like it's this idea of like most artists have a process and many of them would actually probably find it I don't want to say insulting but like if you the more I talk to artists and the more I explore this idea

of like is your stuff just good because it comes natural to you and it just like happens to be good or do you have a process that you like refined over the years a lot of them are quite proud of

their process and like this is very very hard work like there's a lot like a lot of like manual individual steps go into making this good. It's not like I sit down and suddenly there's a book that's great. And if you go to like writer

studios, right, there's like a lot of white boards and like a lot of information. And I'm I'm noticing this across a number of different domains where Claude is quite receptive to being told not I want a result that looks

like, huh, but Claude, I want you to take the same steps that a professional in this field would take. This is like very high mind. So, let me bring this down a little bit. Like I think we're currently in the public seeing this

quite a bit with like motion, video, and 3D where I think Opus 55 has made a pretty big jump. But the big jump that the public has made is to just tell Claw to use Blender, right? Just like Claude,

please use the same tools that professional animators use. And now the results are quite good.

The unlock really is here in the media and the technique and the tools and is less so like raw understanding of what makes a good animated movie. Let me pause here for a second because a lot of the work behind this podcast happens

long before we hit record. For every guest, we already have a pretty heavy research process. Our internal clot agent does the first pass, past interviews, articles, public posts, and the topics the guest always gets asked about and the angles people usually

miss. Then my producer goes deeper. She watches full interviews, reads industry pieces, fact checkcks everything, and builds the first version of the episode.

Then it comes to me. I do my own research from my point of view and usually we go through the questions a few times before we record. That process gives us a lot of information what people are actually saying about the

topic right now. So recently we added dig AI to that workflow. It gives us a layer we didn't really have before. The real human reaction around a topic what people are asking arguing about and still trying to understand. Before an

interview, we can look across similar creators and ask, "What questions keep showing up in comments but still aren't being answered? What parts of the conversation are suddenly growing? What are people confused about that nobody has explained well yet?" And then we can

go straight to the source. Ask Dex scans through billions of videos monthly and sources the actual videos, the actual comments, and the language people are using so we can see the pattern for ourselves instead of relying on another

summary. That helps us sharpen the questions before we record and avoid asking the same things every other podcast has already covered. So now we're not only looking at what the guest has said before, we also know what the

audience still wants answered. If you do content marketing, research, PR, and honestly anything where you need to understand what people are actually talking about, ask Dig is worth trying.

The link is in the description. And we're back to the video. But I wonder for people like you mentioned this prompt act as if you're a professional in this field right and it's going to go on the internet and research but a lot

of information is just not there. How do you trust the output? Like for example if I want to start another medium or like do discover stars right and I have no idea about that but I just tell it

act like an asset researcher. How do I protect myself from AI just confidently presenting mistakes to me?

Yeah that's a good point. I think it still happens to a lot of us. Um I I have I have like a very practical easy answer which is adversarial reviews um which has proven quite effective in coding math and like various other areas

where we can give you numerical results on like how effective it is. Adversarial reviews basically boil down to the idea of you've given given one claw a task spin off another 10 that try to like break this down. Their whole task is to

attack this and prove to you why it's wrong. It's essentially the same concept as red teaming or short selling. Are you using the same model for that or are you want to use another model?

Kind of depends on the task. This is a little context dependent. Um but usually the adversarial review it is it is sometimes easier to find flaws than to create new things. Much like as it is with humans, right? It's much easier for

me to say whether or not the new phone is good. It's much easier for me to do that than to design a new phone from scratch. So usually you can like take a step down in intelligence and still get

very good results. That's the very practical advice. How would you prompt that? Like if there is a process already, what's the additional prompt that you add? So you have some kind of output, right? Like let's say it's a slide deck or a

document. Um or in my case, it's you talked about this blog post that I wrote this morning. I just tell Claude very I'm I'm using Claude Projects, which is this new product that we have that allows you to do work in multiple

threads with a little bit more ease. And I basically tell Claude, "Please fire off five threads and task each of them with like proving that this this post is wrong, that I'm thinking about it the wrong way, that I have my facts wrong,

that I misunderstood something, and then they go off with a given goal of like break this down, you know, prove that I'm wrong." How do you then deal with all the output? Because I do the same, but I use

I just use different models when I'm prepping for the interview. like tell me if I'm digging too deep into this or that. But then I'm giving so much different feedback and I'm like okay now I have to decide cuz this model gave me

this this gave me something else and gave this gave me something completely different. How do you then decide what to use?

I think you're going to enjoy this because you can see where this is going.

The answer is once again claude. What we use in a coding context is something we call a judge where you get the output from the people who are like the the agents whose job it was to attack your output and they're

like all right here's our main points of criticisms and then you use yet another cloud and their job is your your job is to validate and verify and like opinionate whether or not this is valid feedback. M um and then what you can sometimes do is

that ultimately um depending on how you set up your workflow the feedback that is valid should probably go back to the clot that actually made the thing and that claw too that agent too can be discerning and whether or not to act on

feedback right because sometimes um and we see this in a coding context fairly often I also see it in like the knowledge work output that I make sometimes the feedback is like maybe reasonable but um has missed a point or

it's actually not that actionable or is not that useful and That's usually when I know that my output is pretty good.

That's one way. That's when you are really quite uncertain whether or not the output is good yourself and you can't easily check. But for a lot of a lot of things that I create with AI on the other hand, I actually don't need to

fully understand the entire flow because as I mentioned like to me as a person, it is I do have opinions about almost anything I use every single day. I've remade my website with all of these 3D assets. I have no idea how to use

Blender. I've never used it. But I can sort of judge whether or not this looks good, feels good, and that might be enough, right? like you sort of put yourself much more in a client position.

For so many other things in our life, we don't really have a good way of verifying whether or not we get a good output other than like giving feedback to the person we're using, right? Like for a contractor or a lawyer or like for

many people a doctor, right? So you sort of just put it in one of these two buckets like either you really have no idea what goods like good looks like there. My answer is more AI just throw more tokens at it. or you do have you do

have an understanding what good looks like and then maybe that's enough and you don't really need to like have opinions about the process itself.

Yeah. I think it it comes down to uh making sure you select only the top projects where you want to understand every single step instead of just want to control everything.

Yeah. Yeah. It's like the old problem of like sort of being willing to let go, right? Like how to not micromanage. It's really hard.

Yeah. Comes Yeah. Now we have this problem with agents. Is there any knowledge process that you have nailed with agents? Like you completely trust them. They just go I don't know maybe it's yeah fact checking like editorial factecking I fully trust now the output

I get 100%. If I ask claw to verify that all the facts that I'm stating in the slide deck and this presentation um it doesn't even need that many rounds right like it's not like I have 50 agents

doing that. I really trust one agent. I'm like I need you to find primary sources for every number I mention. I want you to verify that is a good primary source. I fully trust that 100%.

When you say agent, do you have a special agent for factchecking that you've trained or is just an one agent that you're using for everything?

It's just just a cloth thread. I feel like this is going to be a spicy take which I think you'll like I think we're kind of over rotating on this agents thing. And I can back this up with a

little bit like I think models are really good generalists and we've seen this now time and time again. Like when we first started looking at math as an industry, I think the the first company to really look at math like with with a

very serious lens was Google Deep Mind and they trained a custom model and a custom harness and they unleashed it on the math Olympiad and they got a silver medal which at the time was state-of-the-art and then four or five

months later all the major generalist models come along and win a gold medal. I understand why people use agents. I have empathy because it's easier for us humans to rationalize what AI can do if we sort of like put on a little hat like

you're a data scientist, right? But that same data scientist has all the intelligence as like a fieldleading biologist or mathematician or what have you.

And I think we're we're sometimes limiting ourselves in our use of AI if we sort of force ourselves to first like make up this cast of characters. So for a lot of these things that I just meant, I don't really have any agents who are

being told who they are. But does the data matter because for every agent you upload certain data set or just because your cloud has access to all your tools it doesn't really matter what you feed to each particular age but then isn't it

a lot of data for it to consider when answering each particular question that is true but it's quite discerning in what kind of data it wants to use right like I think the right approach here is not to make these slices of data

and then give every single agent the slice of data that it wants because many of us if we want to be quite effective at our job we do actually need the full view on the entire business, right?

We're not quite effective if we only get a small slice of data. Like the as an engineering manager, if you only give me code access, I'm probably going to be much less useful to you as opposed to if I also have

customer data, access to customer calls. Sometimes I do want to sit in like a like a customer call like when we're trying to sell something, I want to have deep usage information. Ideally, I get business data about what is working,

what isn't. And you want your agents to have the same range of data. Now, how do you deal with the fact that it's probably a lot of data?

Tools. You give them tools and you document really well what kind of data is behind this tool so that they don't actually have to ingest the whole thing, right? You're just like, if you do want business data, here's the tool to get

business data. So, when you're setting this up, cuz I'm thinking like when you mentioned this, should I move away from all my projects that have access to all the like specific data? For example, I don't know, there's a project for ranking a

guest and we have the whole uh description of the process and then all the old scripts there so that we know how each guest performed.

But now that you're saying this, does that mean that I should move away from that project? Tell my claude, okay, we have this is a folder with all the scripts if you want to look at them. In notion, we have all the intelligence on

how every podcast guest performed and uh this is how we rank guests and just leave it there and whenever I want to rank a guest, I just start a new chat without telling it anything and just tell it rank the guest and it's going to

Yeah, I'm careful to be prescriptive in workflows. Like I always say, if it works for you, it works for you. And I don't think you should change it.

The problem is the models are getting better and I want to get better. So this been working for this year, but a year is a lot.

Yeah. Okay. Well, I mean, if if you're looking for the recommendation that I give for like what do I do?

Again, everyone should do whatever makes them happy and comfortable and what makes them powerful. But it is true that the models have gotten so much better at tool calling and discerning tool calling. Um, and tool calling is this

process of like the model knows when to use a tool, how to use a tool. Doesn't like get stuck in tool call loops.

Doesn't accidentally say, "Oh, I found a data tool call. Let me call this tool until there's no more data left." Right?

like um the models have gotten particularly good at efficiently and effectively using tools and for that reason I don't really make this a priority decision anymore and what kind of data do you need um it also allows me to like have more

threads going and more sessions going without having to easier process much easier process do you still use skills we do use skills quite a bit still although um our official anthropic guidance as well as my personal guidance is be really careful with skills we

often tell models things that were sort of relevant for like the last generation of models which time is so compressed in AI right that the halflife of a good skill is very short and sometimes what we see is if you completely remove the

skill you actually get better output yeah I think first journey said that like just forget about that just just prompt just prompt and see what happens yeah it is it is very effective this is obviously most useful to people who have

some kind of evil and have an idea of what good and bad looks like but I'm pretty careful and very discerning and when when I hear skills and when I don't you know you know what What what a great

way to think about this is I like to consider the model experience as opposed to like user experience and model experience is you pretend for a second that you're the model you're cla right what kind of things would you want if

you had fulfill whatever task you're being given and you very quickly discover that you actually do want access to the raw data you want access to a computer you probably want to be able to install some software I want to make some decisions myself

without somebody telling me what they think already I just want to see exactly what you said raw data and make my own decisions like I think the upfront cost is higher.

Like if you like even a year and a half ago, I was in this world of like telling Claude, I found another crash. Please go and fix the crash. Right? But it very quickly dawned on me, I'm just a conveyor belt

for cloud, right? Like I go look at our crashes, pick one up, move it over, drop it off in front of cloud. I'm like, please fix this. I'm like, okay, this is this is no, I'm just going to cla needs

to have access to all the crash data and then needs to like figure out itself what it needs to fix, what it doesn't need to fix. the upfront cost is really worth it, but it's also like important, I think, to consider that there is

upfront cost, but once you set this up, um, it it opens up so many doors for you, and that's where I find a lot of like real hard value coming out of AI.

If you've been hearing about AI agents and wondering how you'd actually set up one, I put together a guide that walks you through your first one in about 20 minutes. It helps you choose what to automate and get it working with no

coding required. You can get it through my free newsletter. The link is below. for someone who just who's listening to this conversation and who thought they would need like a major setup to start using AI correctly. Would you say uh if

somebody is starting just today, they're using it for knowledge work, they don't need all these tone of voice things like my business strategy. You just give it access to your email. It learns from it.

From your tone of voice, you give access to your newsletters or whatever. So, you don't need any of that setup anymore.

The best way to get started with Claw today, I think, is step one, download the desktop app. It has a built-in browser where you can log cloud into some system you're using and forget about connectors. Forget about like

downloading all the data. Forget about like all kinds of things. Just let Claude use your computer a little bit.

Sort of give away the keyboard. That gets you so far and really shows you what works and what doesn't work. And then at some point you might discover, oh, Cloud is really helping me with whatever tool you're using, whatever

internal or external tool you're using. And once you see that a workflow really works for you, you might consider, okay, can we give Claude something better than like having to manually click in this dashboard around? Can we give it some

kind of connector? But just starting with a browser gets you so far. Okay. And when it just to wrap up with the agents, when you say agents, that that means you have just basically chats, right? And that's it. There

there's no like specific agent who does research, no specific agent who does your tax returns.

No, no. I I mean I do have projects so I do think about I do think about the different work items but I don't think about different roles.

Mh. Right. So I don't I I never tell Claude you're an excellent whatever. Um I think this was this was very invogue sort of like 2024 we were like Claude you're an excellent engineer please go fix whatever problems are

here. Um, but you still use that in prompting now. Act like something that we started with.

Act like a top whatever. I specifically implore claw to like tell me what tools someone top of the field in this field would use and then I'm like I want you to use the same fields.

Um, a problem most models still have today is that if you simply tell it data science is maybe a good example or like accounting. A lot of the models still have the problem that if you tell them to act like a very good accountant that

they engage in a little bit of role play and role play is not really what we're going for. We want the techniques, the tools, the knowledge that a professional would use. We don't really want just the vocabulary, right? Like I don't I don't

need claw to sound like I don't need to put on a costume. I actually wanted to talk to me like I'm five, but I wanted to use all the top tools in the field.

Is there still anything you would not trust an agent with? That's a tricky question. we can just keep layering more validation on top. So I think the question for me is much more it's less about it's just about what do

I not trust Claude with and much more about uh what things do I not trust one single session with.

And I think the the list of things where I don't trust one single session is actually much larger than the list of like things I wouldn't trust Claude with. I rarely want the current session to judge itself. that I think is still

really iffy like the current cloud session that is running wherever you have it in cloud code in cloud co-work in chat in even in a project the current session tends to be quite bad at being a being a judge of

whether or not the work it did was quite good. So as much as I can, I try to find that outside signal and the outside signal in a pinch can be another claw session that has proven to be quite

effective, but my my preference is to find some hard outside evidence for what works or what doesn't work. And sometimes you need to like trust the process a little bit and give Claude a little bit of time. Like for instance in

your field, I'm kind of making this up, but I assume that maybe like the number of views or subscribers is like an important metric that you Oh, like the number of views that's that's what we're whenever I give a task

to Claude, I tell it like we need to hit this number of views. Yeah.

Go and try. But you that and that's that's something that we're still like struggling with is this feedback loop, right? Because it's me giving feedback to my team and then my team actually going and posting something else, not

what Claude gave us. It's like maybe 70% the same, but then the 30% that I've added is actually my taste.

Yeah. How do you close that loop? A thing that has worked for us is there's probably some kind of place where this feedback usually happens. Is it usually in person or does it usually happen?

It's for us is on Telegram. There isn't a native Claude connection. We have a bot that scrapes every chat, but it's still very manual. I have to trigger Claude to look at the bot, look at the feedback, and it's still not it's it's

not there. even like looking at something we've sent, right? Yeah. Because we do we do obviously again in a coding context, we have a lot of engineers who you sort of know which engineer you want to review a certain

thing that you're working on and the kind of things that they look for. Claude is very good at figuring out from the feedback. It's pretty good like given all of this feedback that I've read.

Feedback on Slack, right? Yeah. On GitHub, Slack, wherever else the feedback might happen. And you didn't tell it to look at feedback specifically. Just because you gave it access to Slack, it's looking for feedback cuz yeah, that is much easier to do. Like if

it has the tools and you don't have to like as a human yourself figure out, okay, let me assemble a list of all the feedback I've ever given.

Um, but if you do this like three-step process of step one, assemble all the feedback you've ever given. Step two, tell claw to like condense that down into a list of rules or principles that you care about. Sometimes it's useful to

like then again do a third round of like okay what are the principles that maybe you might believe but you don't actually know you believe and then take that list as like an input for your ongoing reviews that is that has been quite

successful in our attempts to I want to say like specialize the feedback that individual sessions give um that has really worked I think for for work that is a little different than raw coding or not very textbased I think

the problem is still bigger Um but I think similar things can be done right about like give me feedback on a customer call. How did that go? If you read the transcript or like maybe you're I'm doing a lot of interviews for

anthropic which is like a typical knowledge work problem but also like quite interesting but then it's teaching me I wanted to teach it. So and I think a lot of people who started using I experienced this you want to do

more. You see how productive you are but you I don't there is no agent that I trust 100%. Every output goes through my team and then eventually through me. So I become the biggest bottleneck in the company.

Honestly, I feel like our products would do that for you pretty well today. I'm actually quite confident like if Claude has the ability to watch you give this feedback for a while. Like the first thing you would probably do is you would

sort of run it side by side. This is an experiment I recommend to almost everyone. It's like really fun to do. If you currently have a trust problem, you will probably not be super excited to immediately give over the reins, right?

Like from one day to the next. But sort of run it side by side. like you read something, you write down your feedback and um you do that for a few rounds and just to check what kind of feedback claw

would have given and like where do they differ where are they the same that also gives you something where you can actually sharpen the workflow a little bit because I think models will still be imperfect right there's still going to

be things that where where the feedback is like a little different and like even for most humans if we take the same human and we ask them to review the same document like on a Monday versus a Friday the feedback will be different

but this like sideby-side process is rather powerful because it shows you so much. And how do I build this agent that talks to my team? Is it just something that like a website where they upload stuff and it gives feedback?

I would recommend claw tag which is cla inside slack. Uhhuh. And you build you give it access to my taste and Yeah. It's like it's like currently our core multiplayer experience, right? Where multiple people can talk to Claude at the same time.

Yeah. Yeah. Yeah. This is all getting very interesting when it comes to building things, right?

because you can build so many things easy. What becomes valuable when you're building software and I'm talking about uh people who are beginner entrepreneurs. I'm thinking about myself to what is it 2015 we came here with a booking platform for study abroad

experiences amazing like a modem was building this whole platform because we hired a lot of engineers was th hundreds of thousands of dollars we automated stuff now I can do this in two days by myself like what is my entrepreneurial mode

when you think about building software is there any spicy opinion I think it actually hasn't changed that much I think the cost of investment like sort of the upfront capital cost was much higher. That is true. And like I think we're now

entering this interesting area where like single person teams or like two person teams can build billion dollar companies. But I think the secret to building a good product is actually not changed. And I believe it's empathy.

Empathy for the user, empathy for the use case, empathy for like trying to figure out is this actually valuable for the person who's ultimately going to pay for it. I think that has again and I think this is maybe a spicy opinion but

I think this has always been the key ingredient in any entrepreneurial team right and I think especially in startups like before we started taping you and I talked a little bit about 500 startups I think this is even like sort of common

knowledge within VCs that whatever you show up with doesn't really matter it's probably going to change anyway but you as the founder you're the key ingredient and we can we can go one level deeper what does that mean is it like your

talent your skills I think empathy and like an ability to really build something of value is probably the core thing that matters in a founder and I think that has not changed. It has just gotten much easier to tap into that

empathy and like to to execute on it. I totally agree with that. But also sometimes I'm thinking like I have this idea but then I see something on X where people say oh this new model just killed this and we see people vip coding their

smaller tools etc. Do you think it's a trend that we're going to vibe code a lot of tools versus paying for them?

like what would us make us pay for a certain tool if we can spend an hour just building something similar.

I think software that is hopefully going to go away is bad software. Like the software that doesn't really serve your use case and you're only using it because you don't have anything better. I think all of us have some

tools that come to mind where you're like begrudgingly okay I guess I'm going to use this because I have to and I don't have the time to like look for something else or use something else. I think that will come to an end.

Um however at the same time we talked about how hard it is to to build good products. I think there's still going to be tremendous value in like figuring out how to best deliver value and like to have empathy in a way where most of the

products I use that I'm like a big fan of I wouldn't have come up with myself.

Mhm. Right. Like I'm on record as like having written blog posts about how the iPhone is stupid because obviously you need a keyboard, right? Oh, really?

Yeah. Of course. I think for most of the products that I really enjoy, someone else came up with an idea, executed on it, and then gave it to me. And I think the main reason I use those products is

rarely because it was very expensive to build or because it took them a long time or because they needed a big team.

The reason I use the product is because it solves a need I have really really well probably in a way that was better than I thought of. The gap that I think we're going to completely fill is this

garbage software and I only have to use it because I have no other choice. I think that will hopefully go away and I'm very excited about that future.

Before we keep going, I'd love to get to know you a little better. I share a lot about my work here and I'm curious about yours. What do you do and what you'd like more help with? We've put together

a little survey. Your answers will help me decide what to cover in future conversations and what resources to put together for you. I'll leave the link in the description. Please, please, please take a couple minutes and tell me about

you, about your business, about your ambition so I can make better content for you. I really, really appreciate you. Thank you so much for doing this.

You mentioned something about Windows computers that quietly run work and that they are a big business opportunity. Can you talk to me about that?

I build AI products. I work on cloud. When I go out into the world and I look at what are the computers and the machines that are really running the economy. I think it's really easy, especially for like us in Silicon Valley

to like sort of consider your typical startup with like bunch of the latest generation MacBooks and that's all that exists and then you go to an airport and you sort of like peek around the corner and you see that they have not actually

yet moved to graphical operating systems. It's still like the black and white like oh yeah the transportation authorities the Yeah. And I'm doing this for fun now sometimes that I just go to all of these places that I that that run so much of

the economy. Like the last place I did this was a hotel. Like I love doing this at a check-in where I'm like, "Hi, I work in software. I build tools. Can I just like can you just show me real

quick like what you do on your little machine right there to check me in?" Harrowing.

Like all of the all of these devices have software that is so clunky, so bad.

Truly awful. And I think it is probably an untapped opportunity for many businesses to go in there and like try to clean that up. Well, before AI became a thing in 2010, I I joke that if I ever

wanted to retire, I would build really good software for dentist offices. Now you can build it in a day. I guess it's it's the relationship part.

Yeah, exactly. That's becoming Well, this is the hardest, right? Not not just the software. Any other businesses you wish existed? I still think we're barely getting started on software that feels a little bit more personal and has a

little bit of a touch of AI that is not just here's a here's a sidebar with a chatbot in it. I think we've we've sort of permeated the entire industry with low intelligence cheap models that operate sort of at the level of I've

read the FAQ page and if you give me a question I'm sort of like spit out the answer. We haven't really like figured out what can we do with topline intelligence across many of our applications. Businesses that I

specifically wish existed. I'm just going to double down on this empathy thing. The industry moves slowly, right?

Like even VCs move kind of slowly and it's like really hard to adjust to a new world. And I don't think we've properly adjusted to execution is cheap. We've always said ideas are cheap. That was true. And we said execution kind of

expensive, but it's it's gotten cheaper over time. And now execution has become extremely cheap. I think we need to like really double down on empathy, connections, who's capable of like going into a business and really understanding what the needs are and how to can how

can you actually optimize things here. So this might actually be a nice opportunity like the bigger revenge are the MBAs, right? Like I think MBAs can now make a bunch of software startups that might be quite successful.

I was just talking to a Nobel Prize laurate in economics and he was telling me that we have such powerful models but we really lack the application layer. Do you agree with him that there is this huge gap between what models are capable

of and what average user is using them for? Oh yeah, for sure. Yeah, we definitely have like a capability overhang, right?

Like the idea of you have an amazing model in front of you and if you're essentially using it to to ask, hey, what's the capital of France? Again, you're probably not really tapping into what is possible there. Our founder and

CEO, Dario, talks about this concept of like a data center of geniuses. My job really is to figure out okay like let's assume we have a data set of geniuses.

How do I empower you the user to actually make use of that? And it requires creativity. It requires like a rethinking of like your needs and like what you think is possible. Um it's a it's a really hard problem that I don't

actually think is new, right? It's this notion of people have always been held back what what by what they thought is possible rather than what is actually possible. Yeah, I do agree that we have like we've just scratched the surface of

what's possible. How can one close that on an individual level? On an individual level, I thought about this like all weekend. I think on an individual level, the thing that you see on X a lot is this this rallying cry of

be more ambitious. We ought to be more ambitious. Like what being more ambitious really means is I think so context dependent. But I would like people to shed all the shackles of like what they believe is possible and

rethink why are they doing things, right? like what like I'm doing a certain task in my business. What I what am I hoping for is going to happen next.

And you just keep giving the model that task. Like whatever outcome you're hoping for, give the model that task.

You can't play this forever. You will eventually find the limits of the current top intelligence. Like a good counterargument here is okay like the purpose for most businesses is to make money. And if I just tell clude go make

money, it's probably going to like not make it very far. So we we haven't reached like the top level of altitude yet. is somewhere in the middle. But once you bump against that ceiling, you do know that you're at the forefront.

You're at the frontier and you're going to really benefit. You're going to benefit from the next model generations.

Because if if all we're doing is like summarize this email for me or like make this email sound as angry, you're not really like benefiting from the new intelligence.

Okay, talk to me about jobs. Are you still hiring junior engineers? We're currently not hiring junior engineers, but that has less to do with capabilities or AI and more so with with the kind of company that Anthropic is.

I'm a big fan of junior engineers. I've hired so many junior engineers in my career. And I think what junior means is changing a little bit. When you talk to junior engineers today, there's so much more AI native than some of the more

like old seasoned veterans. That's what I've been hearing from a lot of CTOs in the valley that they actually love hiring junior engineers cuz they have like they're open-minded. They're willing to try new tools. One thing that all so many not all but so many juniors

I've worked with like they they exhibit this like hunger for impact. They're really excited to like get something to work. They're much more open to like basically hunting down every single avenue that they can find in order to

have that impact. It's like really nice to see. And again, I don't think that's new with AI. That has always been the case. But what is new is that with AI they're so much more empowered. They can do so many more things in parallel. They

can try so much more things. I really think we're going to see a lot of startups come out of very small teams, very young people getting together and like having really good ideas and then just like executing on them in Mars. I'm

pretty excited about that. What would be your advice to someone who's just starting out now as a software engineer in this ever evolving in this ever evolving like landscape of like how to do software engineering?

I think it's I think it really depends on what you want as a software engineer, right? Like what does it what does it mean to be a software engineer? I think we're like as engineers are we're currently in this like moment of crisis

where we're like what does it mean to be a software engineer? Um does it mean that I'm I've completely memorized the entire API service in React? Is that what I'm valuable for? Is that what I do? Um and I think that's probably less

the case now. To me, this is a pretty welcome change. I've always been of the opinion that a good engineer versus a bad engineer. I measure I've always measured my engineers on their impact.

And the impact is like does the user like this enough to buy the software.

This there's always for me been relatively little value in like really elegant beautiful mastery solutions where we can be really proud of the engineering craft here but the user doesn't feel any benefit.

How do you develop that? What do you double down on now in order to be relevant in 3 years if like every position is suddenly a senior position?

What would be your advice there? The way I got my start in this career is by like building a strong portfolio and I think that would be the same advice today. I think building a portfolio that is not

just like a mock is much easier. So my recommendation for anyone young would be go and build some real things. Prove that you can do it. Prove that you have the empathy. Prove that you have the chops, the ability to talk to customers.

Like figure out what is necessary versus not necessary. Um I do think there's like a slightly new definition of what software engineering is. I personally think that's good. I do see the young people adapt to this new style of

thinking much quicker than some of the older engineers who maybe enjoyed the luxury of primarily working on on tickets without having to ask why too many times.

I've seen so many people be quite successful in that. Um the things that I would maybe to contrast not recommend anymore. It is true that I think like completely memorizing individual API services I think has become less valuable. What is still valuable is

being able to combine different systems in interesting ways. I think the value of that is amplified now because you can so quickly combine things, right? You can just like point AI like three different systems and be like I want

these three like I want these five ingredients mashed together. Go and do it. Is that something that you look for when you hire people? Yeah. This ability to Is there something else that you're particularly looking for?

I mean, I don't know how you hire. Do you still do you do you care a lot about the resume?

I do. When it goes because when you mentioned relationships, they're very important for my industry. I am looking for someone who knows people. I am looking for someone who knows how other businesses operate because this is not public information like how do I learn

how to what what do I charge as a podcaster right what do I charge for I want someone who knows exactly what's going on in the market so we do care about relationships but when it comes to the creative part

actually with creative part we still look at how many views their videos have produced because yeah they know the art I don't look at their degree of course yeah yeah exactly exactly and I think this is I don't know if this is still

controversial I guess we both work like in a you know like tech adjacent, but I've never cared too much about which degree do you have or which school you went to. It was always much easier for me to look at what are the things you

made. Are those things I can touch, feel, try and I can feel like, oh, you either were the winning player on a team or you at least worked on a winning team. That is sometimes useful for me to

know. I think that's really valuable information. And the thing is, if I didn't have that, I might go take the shortcut and look for schools, degrees.

I don't really have to. I will always pick someone who's proven themselves to build something amazing over someone who hasn't built something amazing but has an amazing degree.

What would you tell your kid when she grows up and says like, "Dad, I wanted to coding." What would be your advice?

I mean, she's 9 months old right now, so I think things will change a little bit.

I think you have to be in line for your daycare already. Just saying. And it all happens quicker than you think. What would I tell my daughter?

Um, I think I would tell my daughter to very early on try to understand how businesses operate, how people collaborate, um, how to really find value. I've not gone to the University of Wateroo, but I'm such a fan of the

University of Wateroo's process. All the amazing minds, all the amazing minds, but I think the University of Wateroo has always had this approach which was quite unique within within computer science engineering, which was this is an outside perspective. They might disagree

with me on this, but my outside perspective was always coming from Germany where computer science is very much computer science. It's informatics.

It's a lot of math and very little like only a sliver of how do we actually build a good product. Um, let alone computer engineering. JavaScript was in Germany for a long time universally considered to be like not what real

computer scientists use. And the University of Wateroo's approach has always fascinated me so much because I would get these interns or junior engineers who've worked at five different companies before in their internships. It was pretty common especially in San Francisco that you get

an intern and it's their like fourth internship and they previously interned at Tesla, Google and Apple. And they have so many interesting things they learned about how companies operate, what are the upsides and downsides, how do you actually advocate for an idea,

how do you get resources, how do you convince legal to get something done. I think those are the skills that really like for me made University of Wateroo's engineers almost immediately effective.

It's their first job maybe with me, but they're in a completely different category than someone who has maybe even gotten a PhD in computer science, but has never worked in a company. They might be very smart what they do. But as

we all know, like building a good product and getting it out there is about much more than just knowing what code to write. You need to be able to talk to people. You need to be able to like work together with other functions.

You need to have like a little bit of a of a sense for how does this company run. So I would tell my daughter to probably like really invest in that.

Love it. What do you tell your kids? What do you think your kids should do in the ever evolving economy?

I tell them to try all the things. And I tell them you can be anything cuz sometimes they're like oh robots. You can be the queen of like if I tell her like you can build a robot, she's like

not very excited. I tell her you can be the queen of robots. Yeah. And she's like oh I like queen of robots. And then she says and we we bought this self-driving car that you can program yourself. Oh, that is fun.

She was building it with her dad and like just telling them they can be anything and there is no thing that they cannot touch because it's I don't know out of this or whatever cuz I came here with a notion that everyone here is from

another planet. Yeah, if you went to Stanford that was like something that I couldn't even fathom but seeing that these are all normal people just very hardwork and like giving this idea to my daughters that anything is possible is just hard work

and passion. Oh man, I love that. I think the reason I love that so much is because in AI circles we constantly talk about this idea of ambition and the reason people are not ambitious enough is because we're sort of programmed as humans to

like only consider certain things possible. And if we don't consider them possible, we don't even try. So it's like real like what we believe is possible is the real limiter. Like a much bigger limiter than what the models

can do or what we can do. Maybe that's another reason to like really believe in the new generation is because they will never have experienced all these limitations that we have.

Okay, a couple of last questions. You know, it was um you SF tech week last week. You know, what was the one of the topics that was discussed in a lot of different parties? Why would anthropic block people for being negative with AI?

Can you can you talk to me about this? I can give you my personal take on like why I say please and thank you to the model. And it's probably less about it's Oh, you do say please and thank you.

Even if that eats tokens, even though it eats tokens, I think it's better for me as a human if I treat the machine with a little bit of respect. I think that's better for me. That's better for my soul. That's my personal

take. Imagine you had a kid and has an iPhone and it's like super vulgar to Siri. I feel like I would not be a big fan of that 100% like super cruel. This is a very hotly debated topic I think both in

professional circles as very as well as in layman circles right this idea of like do models deserve moral consideration and my my primary position here is that any take that fits into a tweet is not sophisticated enough and

users can like pick themselves how they want to interact with AI and how they want to interact with the system but I think there's a lot of things in in terms of most products that sort of specify how you want this product to be

used if the entire approach is, oh, I want to use this for torture. I'm not sure that that's something we necessarily need to support. To be clear, the terms and conditions that we have are not about being mean to the model, right? Like,

very specifically, we don't mean push back. We don't mean you yelling at the model. We don't mean you being mean at the model. We don't You can be frustrated with Claude all you want.

It's perfectly fine. What we specifically mean is like you're using Clot specifically for the intent like your stated purpose. you think it might be fun to like see whether or not you can torture a model as an exercise. I

think that is quite different from feedback. This is really bad. Is it is it I tell Claude all the time that this is not it. This is bad. You did a bad job.

Um I think I'm probably harsher with Claude in my feedback than I would be with a human. But I think for me personally as an individual, it is probably better for my soul. If I talk to the system in human language, it is

probably better for my soul and me and the person that I want to be. Okay, I think we covered so much. I just wanted like your final thought on our future.

How do you think our life is going to be different in five years? That is such a big question. I'm not sure you have the right person for like I I know that like going on any podcast is like there's a there's an

occupational hazard of having to make predictions. Biggest thing that I've noticed so far is that we're all so wrong about the intelligence exponential coming out of AI. It's really hard to like wrap your finger around what does it really mean for for the exponential

to exist. And I keep going back to math, right? Like when when we won the silver medal at the m period, when I say we, I mean Google um as an industry, I was well aware of scaling loss, I was well

aware of like how much intelligence is progressing. And I think if you had asked me at what point will AI solve what we would otherwise consider unsolvable math problems, I don't think I would have given you like the middle

of 2026. I would have been like that's that's in a year and a half. What are you talking about? So I think instinctively having a good feeling for how will what will the future look like is really really tricky. That's the

first part. And I think the second part that is rather tricky for me is the one thing I can tell you with a decent amount of confidence is that I think the products that we use today will all look

completely different. I think I think we're currently we've hit this point where I becoming so mainstream and it's becoming so popular that it's like easy to imagine or we're maybe like this this is sort of the end state. Um but I'm

quite convinced that all the AI products we use today are roughly in the same phase as all the funny phones that we had right before the smartphone came along. So, I think we're still in the pre iPhone era, which makes it very

exciting for me to work in. Wow, this is really fascinating. Already like from in our preAI era, uh there are so many things that are changing and you made me rethink my whole process. I won't even show you what what it looks

like now cuz I think it's fine if it works. It's very outdated cuz we give it all the data and tell it exactly what to look at. But I totally understand how giving it access to everything is just

so much more powerful. It's such a tricky thing, right? all the advice I've given you today. In about 10 weeks, we're probably going to have a new revision on all of it.

That is also true. So, being an always on agent when it comes to new models, I think that's the that's the thing. And try new things. Thank you so much, Felix. If this conversation with Felix made you want to try Claude in your

work, watch my interview with Alien Miller next. Ally works with Teams on using AI, and in our conversation, she showed me her real workflows and built a Claude morning brief life. It's a practical place to start if you're

wondering what to do