Speaker A
Please welcome to the stage member of technical staff at Anthropic, Puneet Shah. Welcome to the last session of Code with Claude here in London.
Learn how to maximize the Claude Platform with prompt caching, agent building, and practical demos from Anthropic's Puneet Shah.
Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.
Generated from the transcript and can be wrong — check the timestamp.
Key Takeaways
- Prompt caching is critical for reducing costs, improving speed, and increasing effective rate limits on the Claude Platform.
- Understanding and monitoring your prompt cache hit rate is the first step to optimizing agent performance.
- Anthropic’s cloud platform offers built-in analytics and tools to diagnose and improve prompt caching.
- Building agents with multiple integrated tools and managing context effectively leads to better model performance.
- Practical demos like HeroCorp illustrate how to leverage the platform for real-world business applications.
What the video covers
- Puneet Shah from Anthropic presents the final session of Code with Claude in London, focusing on maximizing the Claude Platform.
- The session emphasizes the importance of prompt caching for cost savings, rate limit boosts, and latency improvements.
- Prompt caching can reduce costs by up to 90% and effectively increase rate limits by up to 5x with high cache hit rates.
- Anthropic provides analytics on prompt caching via their cloud platform console to help users optimize their cache hit rates.
- Common caching issues include dynamic prompt elements like timestamps that break cache consistency.
- A one-line code change can implement basic prompt caching, and pre-packaged Cloud API skills help improve cache performance.
- Puneet introduces HeroCorp, a fictional superheroes-for-hire company, to demonstrate practical agent use cases and dashboards.
- The demo showcases integration of multiple data sources like web, Slack, Gong, and Jira to provide a holistic company view.
- The session covers advanced agent topics including tool usage, context management, and cost optimization strategies.
- Overall, the video provides actionable insights for developers building and deploying agents on the Claude Platform.
Chapters
- 00:00Introduction and Welcome
- 02:07Agent Builders and Production Use
- 03:48Prompt Caching Explained
- 05:15Customer Examples and Cache Hit Rates
- 06:36Getting Started with Prompt Caching
- 08:31HeroCorp Demo Introduction
- 10:15Dashboard and Data Integration
- 11:59Advanced Agent Features and Cost Management
- 19:59Summary and Closing Remarks
Full Transcript — Download SRT & Markdown
Speaker A
Yeah, let's give it up for everyone who's come before.
Speaker A
So I'm a product manager on our team, on the platform team at Anthropic, and I've shipped some of the features like our 1 million context window, fast mode, and a number of our improvements on prompt caching, amongst others.
Speaker A
On prompt caching, amongst others, and that's what makes me especially excited about this session. You guys have made a very good decision to come here because this is about the cloud platform.
Speaker A
And if you think about what we have, we have these great models, but there's this whole layer on top of it, the platform, that's not about just getting you the intelligence but helping you build a real business, real products that really deliver for your users on top of those models.
Speaker A
And so, in the spirit of this being the last session, I want to like, let's get a little movement.
Speaker A
The spirit of this being the last session, I want to like, let's get a little movement. So if you have built an agent of any sort, big, small, I don't care what, go ahead and stand up.
Speaker A
Okay, good. Nice. A lot of agent builders here. This is nice. Now, if you have put one into production, go ahead and stay standing and everyone else just take a seat.
Speaker A
Okay. If you're really happy about the quality, about the cost, about the speed of those agents, stay standing and everyone else take a seat.
Speaker A
Okay, nice, nice.
Speaker A
Okay, look around, remember these folks, these are our true experts. Go find them at happy hour afterwards. You can all have a seat now. Thank you, thank you for humoring me.
Speaker A
What I want to do in this session is share some of what we've learned about what helps you get the most out of that cloud platform.
Speaker A
To get the most out of that cloud platform. And one of the first things that we have on the platform that I want to talk through is prompt caching.
Speaker A
If you remember nothing else from this session, think about prompt caching. And what caching is, is it's a way that we, if you're not familiar, where we take your input tokens, we process them, and then we cache that before we generate the output tokens.
Speaker A
And that cache, we then continue to reuse as the conversation moves on.
Speaker A
And so when you have a new message, moves on. And so when you have a new message in the conversation, we just process those additional new tokens, but the rest we just pull from the cache.
Speaker A
And, okay, why is this useful for you? Why should you care? Well, the first reason is you get a 90% discount because we're not reprocessing them. We pass those savings on to you. You get a 90% discount, so a huge cost savings to actually build your agent.
Speaker A
The second is that you also get a rate limit boost effect. Second is that you also get a rate limit boost effectively if our rate limits are not
Speaker A
they don't count your cached tokens and so if you have an 80% cache hit rate meaning 80% of your tokens are cached then you effectively have a five times larger rate limit in practice so that's great and then the last
Speaker A
benefit is latency if you are starting to cache a lot in that conversation's getting a lot longer because we're no longer processing all those tokens what ends up happening is Because we're no longer processing all those tokens, what ends up happening is your
Speaker A
time-to-first token goes down. And so these are great benefits. And if you're kind of looking for a target and you're building agentic applications, something in the kind of 80% above range is a good place to try to target. But if
Speaker A
you look at some of these customers here, we have Replit, Cursor, Perplexity, Cloud Code.
Speaker A
They're hitting 90-plus percent. These people have really, I've talked to all these customers, they've put a ton of effort into making their prompt caching work.
Speaker A
Because of all of those benefits I've described. And one of the things that they all start with is just understanding what is your prompt cache hit rate. What is the prompt cache hit rate you get? That's the first question you should understand. And
Speaker A
Thankfully, because we've learned how much work these people are putting in, we said, why don't we build it for you guys? And so now today, if you go to the console on the cloud platform, console.anthropic.com, you can actually see analytics.
Speaker A
Right next to your cost and usage pages, you can actually see analytics on prompt caching. And even just yesterday, we continue to improve this. Yesterday, we just launched ways to actually figure out why has your cache broken? Turns out the ordering of how
Speaker A
you make those prompts matters a lot. A common error I see is that people put a timestamp into the system prompt. You know, what day is it? Okay, that's useful, but then it breaks the system prompt because it changes as you go. And
Speaker A
that as you go, and that then breaks your cache. The tokens need to be exactly the same. And so you can actually see, how did it break? And you'll see that on the analytics page. And that's a great
Speaker A
place to get started. If you're seeing it at 0%, it's okay, that's why you're here. You can start with a one-line code change with autocaching. Implements kind of a basic prompt caching. Or better yet, go to the Cloud API skill. Go to
Speaker A
Cloud Code or many other coding agents, and we have a skill built in, prepackaged,
Speaker A
or many other coding agents and we have a skill built in, pre-packaged, that lets you ask it to improve your cache hit rate and it will help with how you manage and order that prompt to get the best performance. So prompt caching,
Speaker A
super important. We're gonna talk a little bit more about it, but I want to talk about something I've been working on, I've been really excited about, you guys are here for my startup pitch, thank you so much. I'm also in my free time the
Speaker A
CEO of HeroCorp. I'm the CEO of HeroCorp, and I have my CTO actually here, Ben, do you want to come out here?
Speaker A
He runs our technology team. Let's give it up for Ben. And can we switch over to the demo laptop? So we're, wait, Ben, what is, Ben, this is cloud with code. This is 2026. This looks like it's from the 90s. What's going on here? Guys, who wants a better theme with something
Speaker A
a little bit more of the times? Yeah, we got a few thumbs up. Okay, Ben, let's improve the theme. Yeah, we got a few thumbs up. Okay, Ben, let's improve the theme. Let's make it ready for Code with Claude.
Speaker A
Let's let that cook. Nice. So, we are HeroCorp. We are a superheroes-for-hire company. You know, we help fight crime in the city, keep the tube running on time, you know, staff your three-year-old's birthday party and make sure there's a sufficient number of balloons.
Speaker A
You know, staff your three-year-old's birthday party and make sure there's a sufficient number of balloons. We do it all. And some people have misconceptions, I think, about the superhero world. They think that we're like fly by the seat of your pants. We're
Speaker A
analytical. We use OKRs. We plan. And that's why we have a dashboard. And this dashboard is how we track our performance. You can see 90-day retention, 94%. We could do better. We got one flight risk there, probably because our compensation is a
Speaker A
little bit low. Okay, yeah, I probably should be paying the superheroes more. You can see some of our different superheroes in the Anthropic Cinematic Universe. Our lawyers required us to use these names instead of others.
Speaker A
Are listed here with their quotes about how things are going. So do you mind pulling open our, we built a little developer console. You'll hear me repeat this a number of times. You should always look at the transcript of your agents to really
Speaker A
understand what's going on. And so right here you can see we're pulling in a bunch of data and if maybe you want to pop open one of those you'll see here there's a bunch of data we're pulling in
Speaker A
from web, from Slack, from Gong, from Jira, all the different sources to aggregate it to take a big holistic view on what's going on at the company and, you know.
Speaker A
a good first step to start out with let's take a look at maybe the rest of this dashboard um wait Ben there's five OKRs this is I give you a million tokens of context and this is where we get okay so turns out we've filled up the context window and now we
Speaker A
got a little bit OK, so it turns out we've filled up the context window, and now we've got a little bit more work to do. So let's switch back to slides. What we need is context engineering. And what context engineering is, is the art and science of figuring
Speaker A
out what context you expose to Claude to give your agent the best performance. And again, I said this earlier. I'm going to repeat it many times. Look at your transcript to see it, to see what the Claude, the models are to see
Speaker A
it, to see what the models are actually seeing. And that will, I think, be incredibly illustrative to see is there a lot of stuff that you really don't need to be passing to Claude or is there a lot of relevant stuff that's keeping
Speaker A
it on track? And we'll go through today, there's many techniques, we'll go through three primary ones about how you can implement better context engineering on your models. And so to start, the first is around figuring out what tools we pass to the model
Speaker A
and kind of narrowing that down. The second is about narrowing... model and kind of narrowing that down. The second is about narrowing down what results from those tools we share to the model. And finally, as that conversation continues in the history, it's about
Speaker A
keeping that conversation going for kind of almost unlimited feeling context. So let's go through these one by one. The first is tool search.
Speaker A
Agents use a ton of tools. It's one of the magic of making great agents is tool calls. And cloud models And cloud models are especially capable of leveraging those tools. We see tens, even hundreds of tools used in agents today. Now, one of the problems, though, is when
Speaker A
you define all those tools and you pass them to the model, what ends up happening is they fill up a lot of your context, and that leaves a lot less room for the actual work that needs to happen in the model. And so
Speaker A
what you're looking at here is that, you know, that context fills up after just a couple turns. You know, that context fills up after just a couple turns. So instead, we have tool search tool. And what tool search tool does is it figures
Speaker A
out, you define all your tools up front, but we only pass the model a tool that tells it, okay, when you think you might need a tool, call this.
Speaker A
And it gives then a list of the tools, and then only then does it put into context the actual definition of the tool. And you can see here on the bottom row, on that middle row, rather, we're just passing in those orange tools
Speaker A
right when needed, leaving a lot more space for the actual stuff you need. right when needed, leaving a lot more space for the actual stuff you need. And Lovable tried this. You heard from Bobian earlier on this stage, and he was talking about
Speaker A
how he's told us with Tool Search that they actually reduced their overall token conception by 10 % with just this solution. And importantly, it actually improved the performance of their model as well. That they saw that because you're putting less gunk
Speaker A
into context, you're putting more relevant stuff there only. You're putting more relevant stuff there only. It turns out the models actually perform better. So they've rolled this out to all their users. Okay, so that's tool search. What happens about what
Speaker A
you do with those results from the tools? That's the next one, is programmatic tool calling. For programmatic tool calling, the idea here is how do we curate that content that's coming back from tools just to what's most relevant? And the
Speaker A
insight here is it turns out Claude's really good at writing code. The insight here is it turns out Cloud's really good at writing code. If you've used Cloud Code or any other coding tool that leverages the models, you have been able to
Speaker A
experience this, and we use that to your advantage here as well. We have Cloud just write a simple Python script that can actually call those same tools, get the same results, but instead do a little bit of work to curate that content
Speaker A
and then And folks at Quora - Send it to the model, just what's most relevant. And folks like Quora have used it. They've used it with HTML content, where they've been able to strip away all the stuff that's just irrelevant, keep the part that's relevant, and seeing the performance of their models
Speaker A
improve. Okay, so the conversation continues. You've probably heard from folks on this stage before about how, from like Lisa and Jeremy, about how we're seeing the models able to do even hours of autonomous work today. And so inevitably, you're
Speaker A
going to hit that million context threshold. and what with compaction is a way to allow you to continue that conversation instead of getting halted to a screeching stop when you hit that full context. What happens is it summarizes the context with your prompt, shifts it down to
Speaker A
lower context, removes the turns that are no longer relevant, and then the conversation continues.
Speaker A
Rinse, repeat. And from there you get this almost feeling of unlimited context. keeping the models on track through compaction. And Hex has used this, that since using this, they've been able to simplify down their code and seen that their
Speaker A
model is able to continue to perform nicely. Nice, so let's go back to the demo. I think, Ben, we've got a couple solutions of context engineering. Take us, take us forward. Let's see what we can do.
Speaker A
Let's see what we can do. So right here, you're seeing immediately that we've now seen that context window bar kind of in the left -hand panel. You see that's going up a lot slower because we're putting in less
Speaker A
into context in each turn. And what you'll notice that when it hits around 400k just right there, you notice how it went back down? What happened was that we hit a threshold. We've set it to 400k. I'll get into that in a bit.
Speaker A
And then it compressed it down with compaction. And then it compresses it down with compaction. So, hey, this is great. Let's take a look and see what we can do with, like, how is this working? Let's take a
Speaker A
look at tool search, maybe. Can we find one in there? Yeah, there we go.
Speaker A
So we want to get our hero retention metrics. Remember, this is an analytical business.
Speaker A
We need to understand retention. And so it calls the models and it asks them, okay, what are the tools that can help me The tools that can help me get my hero retention metrics, turns out there's a hero
Speaker A
retention metrics tool. And now note in each of these, you can notice that like that one is 14 ,000 tokens in the schema. The hero list is 6 ,100.
Speaker A
The next one's 9 ,300. Like these are large tools and that's totally normal for a lot of agents. And because we haven't put them into context, there's a lot less room that's been taken up by these tools that we won't need. And instead
Speaker A
we call, okay, hero retention metrics, need. And instead we call, okay, here are retention metrics, that's the correct one. If we go down to the transcript to find the, here we go, great. So what you see here now is these are the exact
Speaker A
definition of that tool, just that tool, none of the rest. And great, now the model can use the tool it needs and it leaves room for everything else. So hey, we've got the tool we need. Now what happens next when we get the
Speaker A
results? Let's take a look at, yeah, right here. This is, so Gong is, if you're not familiar, it's a tool that kind of records. So Gong is, if you're not familiar, it's a tool that kind of records your sales team's conversations so
Speaker A
you can look at the data, analyze it, see how the field is reacting to maybe the product you've launched. And these three -year -olds, I love them, but they really care about their green balloons a lot when we send superheroes to them. They
Speaker A
go on and on for 30 minutes, even 60 minutes sometimes, and it's just not that relevant, frankly, for when we're putting this dashboard together. I don't need all that data. I just kind of need the general sentiment, the vibes of how they feel
Speaker A
about. The general sentiment, the vibes of how they feel about the conversation. So with programmatic tool calling, the model has created a nice little script here. And it first looks at the first 2 ,500 tokens, characters. It's there,
Speaker A
just understands what's the structure. And then it realizes all we want is the aggregate sentiment. And so it then writes a simple way to loop through, get the aggregate sentiment from the variety of calls, and stream that in to the dashboard. And great,
Speaker A
what previously was this. into the dashboard and great what previously was this massive result block that you just saw actually is now down to just the parts of it that we really need for what we need to do right now again keeping
Speaker A
that context narrow and remember this session is not about a cool demo there's lots of cool demos in AI love them too but the the kind of core thing here is about actually putting into production and in that case you need to really
Speaker A
make sure that the context really matches what's actually needed to make your your product successful ...matches what's actually needed to make your product successful.
Speaker A
So, okay, that's the second one. Let's talk about the third one, which is compaction.
Speaker A
So we hit that 400K threshold, and now we need to take it down. We have chosen 400K. Now, I launched the Million Context. I'm a big fan of the Million Context window. Lots of scenarios where Million Context is great. It might be, for
Speaker A
your scenario, that the right combination of intelligence, cost, latency, is not a million. Start with, you know, 500K, 400K is often a good start. is not a million. Start with, you know, 500k, 400k is often a good starting point, but, you know, it
Speaker A
changes by model. And what we've done is we set that threshold and then we create the summary. And you get to create your own prompt to help guide it.
Speaker A
And then it just puts together the key facts that's needed to summarize where things are, keep the conversation on track without losing the wrong context, and adds that in here. And then great, moves on the conversation.
Speaker A
moves on the conversation. So, super, we've implemented context engineering. For those of you with your accounting eyes, you can see the cost is down to about 11 pounds. We've reduced it down about a third from where we were earlier. But I have to
Speaker A
confess one thing. This has been a tough business. This one superhero you'll hear about, Cryothane, he does a lot of stuff that just increases my insurance premiums, and it is like a tough business.
Speaker A
like margins are really really thin and um yeah we need to get the cost down every time i'm loading this right now it's costing 11 pounds that's pretty high and um what model are we using on this okay we're using for opus 47
Speaker A
good model but it's high intelligence and therefore also higher cost i wonder like could we maybe figure out how to do this with sonnet and haiku but do it with closer to opus intelligence let's Do it with closer to Opus intelligence?
Speaker A
Let's switch back to slides and see what we got. Okay, so we've got advisor strategy. Of course I had a solution to that problem.
Speaker A
So the idea behind advisor strategy is that you have the model, the agent is run with an executor that's Sonnet or Haiku. And the kind of insight here is that it can kind of figure out what to do with all these
Speaker A
different shapes, except when it's... is that it can kind of figure out what to do with all these different shapes, except when it kind of, you'll see in a second, it encounters this sort of weird oddball shape. And it doesn't know what to
Speaker A
do with this, and it just calls the advisor, asks what to do, and then it gives the advice on what to do. The insight here is if you've ever worked with development teams, you've probably noticed there are senior engineers paired with junior engineers
Speaker A
that make that junior engineer so much more powerful. That junior engineer is still the person hands on keyboard getting work done, On keyboard, getting work done, but with the coaching, the mentorship, the code reviews, the help on architecture from the senior engineer, they were able to actually achieve so much more than
Speaker A
they otherwise would have, sometimes even approaching what someone with that senior engineer skills could have done solo. And that's the kind of same idea here. It turns out that same logic works with models, that pairing the kind of sonnet haiku gets you kind
Speaker A
of the prices of those models approaching even sometimes opus intelligence, a big intelligence boost.
Speaker A
and folks like bolt have used this they've kind of gotten better architectural decisions where they can see on complex tasks it improves performance while on less complex tasks it has no extra overhead which doesn't call the tool and so it's a pretty nice trade -off it's like a Pareto optimal way to
Speaker A
and intelligence performance. So let's go back to the demo machine and let's take a look. Let's see, let's go ahead and implement it.
Speaker A
Okay, so we're starting at like just about 11 pounds and you'll see, yeah, that bar is going up a lot less fast. Of course, we are using Sonnet now. As you can see there's Sonnet 4 .6 with Opus 4 .7 Advisor. Okay,
Speaker A
so it kind of seems obvious of course the cost four seven advisor okay so it kind of seems obvious of course the cost is going to be lower we just switch to a more uh inexpensive i guess like a higher value model um
Speaker A
and the question is not is it better quality it's not is it lower cost it's is it also better quality is it is it also losing intelligence on this dashboard and so if we want to scroll through i think that there was a so that we have this this one
Speaker A
contract that we've been I think that there was a, so that we have this one contract that we've been struggling with. We really need to land it. I don't know if we're going to raise our next round if we don't get this. It's
Speaker A
the Metropolis renewal. Metropolis has been our big customer from the start. They've been big supporters of HeroCorp. I need to keep them on board.
Speaker A
And so Sonnet looked through the transcripts, was like, I think the renewal is on track. We're doing well, but this is important. So let's ask Opus. And this is one of those ways that the advice. It's important. So let's ask Opus. And this
Speaker A
is one of those ways that the advisor tool can be used. You can pass it a transcript. It'll take a look. Anything wrong? And it'll report back of, hey, you might have missed a thing or looks good. In this case, Sonnet said it
Speaker A
was green. But Opus looks through and it looks through it deeply, more fine eyed and is able to say, actually, the mayor specifically wants cryothene. That guy was increasing my insurance premiums. Yeah, he specifically wants him. He's too good to lose. I know
Speaker A
it. Anyway, he's great. It's too good to lose. I know it. Anyway, he's great, but he specifically wants him, but he's just unavailable that day.
Speaker A
And so actually, this is red. The renewal will not go through if he's not available on the day of the big event that they want at City Hall. And so this is a watermelon, if you've heard that term. It's green
Speaker A
on the outside, but deep, red on the inside. And so what happens is Opus overrides Sonnet and lets us...
Speaker A
know and catches this spark. So that's good. Okay, this is great. We were able to kind of recover that intelligence and have good performance. So, okay, to backtrack here, what happened in this dashboard? We started at over 10 times the cost, and we
Speaker A
brought that down through one prompt caching. We figured out what our prompt cache hit rate was, zero. We implemented it. And by the way, one great way to implement it again is the Cloud API skill. A great way to implement it again is
Speaker A
the Cloud API skill within Cloud Code. It helps figure out a lot of that logic for you. We looked at the transcript then to see exactly what's going on, and we then implemented context engineering. First, reducing what tools are sent to the model.
Speaker A
Two is then curating what data from those results gets sent back to the model.
Speaker A
And then finally, helping create almost unlimited context with compaction. And then the last part was then helping reduce that cost further.
Speaker A
while preserving intelligence through Advisor Tool. So we've done all this. The opus has given us a nice little button here in the era of AI agents, the CEO. I just click buttons nowadays. So should we save the Metropolis contract?
Speaker A
Yeah, let's do it. Go ahead and click the button, and Metropolis is saved. We're going to get our next round, I think, I hope. Saved. We're going to get our next round, I think, I hope. Let's see. If the VC's in the
Speaker A
house, we can talk afterwards. So let's switch back to slides. Okay, so key takeaways. Again, to review, what did we cover here today? First thing, prompt caching. If you do nothing else, prompt caching. Figure out what your prompt cache
Speaker A
hit rate is and implement it. Cloud API skill is a great way to get started. Then we did the context engineering pieces. We curated what's getting sent to the model, both from tools, from curated what's getting sent to the model, both from tools,
Speaker A
from results of those tools, and then compacting it as the conversation grows. And finally, advisor strategy, a Pareto optimal way in many cases to get better cost and intelligence. A great trade -off. Now, I've talked about things that
Speaker A
have launched literally in the last 24 hours here. So the cloud platform is evolving fast. I'm excited you've been here for this talk, but your learning journey is You've been here for this talk, but your learning journey is not over. We're going to
Speaker A
continue to try to do our best to make sure you can build not just great demos, but real production agents that work for you and your customers, that let you build a business on this platform. And we're going to do that continuously. That's
Speaker A
our commitment to you. And that means keeping abreast of all the things that we're launching. Frankly, I didn't even have enough room on this slide to put everything that's launched in 2026. This is just a subset of what we've already launched this
Speaker A
year. I'm particularly excited about automatic prompt caching, a one -line way to implement prompt caching if you've never implemented it, or the cloud platform on AWS. This one's very cool. I mean, it's this whole platform that we've talked about, available where I know a lot of folks in
Speaker A
this room use our models on AWS. It's all right there. So, yeah, this is a great way to get started. I'm excited to see what you guys build, and thank you for coming. I'm excited to see what you guys build and
Speaker A
thank you for coming.
Topics:Claude PlatformAnthropicprompt cachingagent buildingAI agentscost optimizationrate limitscontext windowcloud platformdeveloper tools
