Skip to content

Every Skill You Add Breaks Your Harness (Here's the Fix)

Expanding skill libraries in AI agents reduces performance mainly due to skill selection failure, not token overload. Fixes include trimming and search-based routing.

Ask about this video. Answers come from its transcript only — with the timestamp, so you can check them.

Generated from the transcript and can be wrong — check the timestamp.

Key Takeaways

  • Skill selection failure (shadowing) is the main cause of performance loss when skill libraries grow.
  • Context token overhead is less impactful than previously thought but still relevant at very large scales.
  • Proper skill descriptions and metadata are critical for effective routing and skill invocation.
  • Trimming skill/tool libraries and using search-based routing significantly improves agent success rates and latency.
  • Progressive disclosure design balances token cost but can limit routing accuracy if descriptions are truncated.

What the video covers

  • Anthropic reduced the context cost of loading a built-in Claude API skill from over 200,000 tokens to about 25,000 with a one-line fix.
  • Loading many skills causes performance degradation due to two main factors: context overhead and skill selection failure (shadowing).
  • Databricks research showed that skill selection failure, not context overhead, is the primary bottleneck when expanding skill libraries.
  • With larger skill libraries, agents often abandon invoking any skill rather than choosing the wrong one, leading to performance drops.
  • Skill routing relies heavily on the skill body (91.7% of attention), not just the name or description, but descriptions are often truncated or missing.
  • Many public skills lack proper routing descriptions, causing accidental skill selection or abandonment.
  • Anthropic’s progressive disclosure design loads metadata first, then bodies and references, to reduce token costs but limits routing signal quality.
  • GitHub engineers improved Copilot’s performance by trimming the default tool set from 40 to 13, increasing success rates and reducing latency.
  • Anthropic introduced tool search to reduce token costs and improve routing efficiency by loading only relevant skills.
  • The video concludes that managing skill libraries with search and selective loading is essential to maintain agent performance.

Chapters

  1. 00:00Anthropic's One-Line Fix to Claude Code
  2. 00:57Databricks Research on Skill Library Impact
  3. 01:33Effect of Increasing Skill Library Size
  4. 02:18Decomposing Performance Drops: Context vs Shadowing
  5. 03:05Abandonment and Wrong Skill Invocation Explained
  6. 03:48Progressive Disclosure Design in Skill Loading
  7. 05:02Skill Router and Attention Analysis Findings
  8. 06:25Skill Description Truncation and Its Impact
  9. 07:08Solutions: Trimming Skills and Tool Search
  10. 07:57Conclusions and Best Practices for Skill Management

Answers

Questions about this video

Why does adding more skills reduce an AI agent's performance?

Adding more skills reduces performance mainly due to skill selection failure, where the agent either picks the wrong skill or abandons skill invocation, rather than due to token context overhead.

What is progressive disclosure in skill loading?

Progressive disclosure is a design where skill metadata (name and description) is loaded first, with the skill body and reference material loaded only after selection, minimizing token costs but limiting routing signal quality.

How can developers fix performance issues caused by large skill libraries?

Developers can improve performance by trimming the skill library to essential skills, using search-based routing to load only relevant skills, and ensuring skill descriptions are complete and informative.

Full Transcript — Download SRT & Markdown

00:00
Speaker A
On the 17th of August, Anthropic shipped a one-line fix to Claude code. Read it slowly. Reduce the context cost of loading the built-in Claude API skill from over 200,000 tokens to about 25,000. One skill Anthropic's own, eating a context window before you typed anything. A developer had already
00:20
Speaker A
measured it in July. One migration document inside it was 36,000 tokens. An August report found the worst case. The skill could not detect the project language, so it loaded all of them. C#, curl, go, Java, PHP, Python, Ruby,
00:36
Speaker A
TypeScript, plus 26 shared markdown files. 812 kilobytes. The reproduction task needed 33 kilobytes of that. The other 96% was freight, so the diagnosis writes itself. Skills got fat, context filled, agents got worse. Delete tokens, fix the agent. That diagnosis is correct. It is also the smaller half of
00:57
Speaker A
the problem and the half you can see. In May, Hong Wen Song and Song Wei at Databricks measured the other half. It does not appear in your token counter.
01:06
Speaker A
They loaded 202 skills into an agent instead of only the ones it needed. Pass rate fell 21 points. Then they did the thing the field had skipped. They asked how much of that drop was actually the tokens. Here's their setup and the
01:19
Speaker A
control is what makes it worth your time. Start with what they call Oracle skills, only the ones already proven to help that exact task. That is the baseline. Then pad the library out to 52 skills, then 102, then 202 without
01:33
Speaker A
removing anything that works. The task does not change. The helpful skill is still sitting right there. The only thing that grows is the crowd it is standing in and the pass rate falls at each step, eight points, then 14, then
01:45
Speaker A
21 across two and a half thousand runs. Then the split and this is the contribution. There are only two ways extra skills can hurt you. Either the agent picks correctly and executes worse because its context got heavier, call that context overhead, or it picks the wrong skill in
02:01
Speaker A
the first place, call that shadowing. So, the Databricks pair read every run, sorted it by which skills the agent actually invoked, and measured the two effects separately. The paper says no prior work had decomposed it. Skills were being evaluated one at a time against a no skill baseline. The two
02:18
Speaker A
numbers behave nothing alike. Context overhead comes in at six points, six points, seven points. It barely moves while the library quadruples, and its confidence interval does not clear zero. The authors will not claim it is real.
02:32
Speaker A
Their word is suggestive. Shadowing goes three, then eight, then 14. It scales with the library. At the largest one, it is 2/3 of the whole drop, and across the bigger libraries, it is the only component the statistics can confirm. Their conclusion is one sentence, and I will read it exactly.
02:49
Speaker A
The skill selection failure, not the enlarged context, is the primary bottleneck when expanding the skill libraries. And the behavior underneath that number is stranger than a wrong pick. With only the correct skills loaded, the agent invoked the correct ones in 88% of runs. At the largest
03:05
Speaker A
library, that falls to 52.6. So, where did the other third go? Not mostly into wrong skills. Wrong skill runs went from zero to about 9%. The big migration was into runs where the agent invoked nothing at all, 12% up to 38.5.
03:20
Speaker A
Abandonment, not error. The agent is not making a mistake there. It is looking at a shelf of 200 jars, recognizing nothing on it, and doing the job from scratch the hard way. Your skill library did not get overruled. It got skipped, and your
03:35
Speaker A
logs will not say so. One row of their table makes it concrete. The task is called Mario coin counting. The correct answer is a bundle of three skills, FFmpeg, image editing, and an object counter, and all three are loaded and
03:48
Speaker A
available. Also loaded is a skill called video frame extraction, written for a completely different task, whose description happens to match the words of the query better. The agent picked it in 26 runs out of 26.
04:01
Speaker A
So, why does a system that can read your entire codebase fail to recognize a folder somebody handed it? The answer is not incompetence. It is the design, and the design is a good one. It is the reason skills are cheap at all.
04:13
Speaker A
Progressive disclosure. Three levels, and Anthropic's own documentation puts numbers on each. Level one is metadata, the name and the description out of the front matter. Loaded at startup, about 100 tokens per skill. Level two is the body. Under 5,000 tokens, and it loads only after the skill has already been
04:32
Speaker A
selected. Level three is bundled reference material, and it costs nothing until something reads it. The documentation summary of all this, you can install many skills without context penalty.
04:43
Speaker A
That sentence is accurate. It is also the whole problem, because it is a claim about tokens, and it gets read as a claim about safety. The description is not merely it is the entire ballot your router gets to read, which raises a
04:56
Speaker A
measurable question. How much of the signal needed to route a task actually lives in a one-line description?
05:02
Speaker A
In March, a team published Skill Router, a system that routes across roughly 80,000 skills, and they ran an attention analysis on their own re-ranker.
05:10
Speaker A
Across 75 queries, the attention split like this. The name, 7.3%. The description, 1.0%. The body, the part progressive disclosure has deliberately not loaded yet, 91.7%. Take the body away, and routing accuracy falls 37 to 44 points. They tried the obvious repair, distilling the body down
05:31
Speaker A
into better descriptions, and it still came in 7 to 21 points short. The signal does not compress into a label. One caveat, because it matters, that is their re-ranker on their benchmark, not Anthropic's model on your machine. What
05:45
Speaker A
it establishes is where the routing signal lives, not how any particular vendor routes. The mechanism generalizes. The exact numbers do not.
05:53
Speaker A
And out in the wild, that 1% is often not filled in at all. A study of 55,000 public skills found 26.4% shipping with no routing description whatsoever. A skill with no description cannot be chosen on purpose. It can only
06:09
Speaker A
be chosen by accident. In July, researchers collected 3.8 million skill files off public GitHub, spread across 282,000 repositories. Half of them were verbatim copies of each other. That is the shelf your agent is now expected to read, which brings us to the sentence that
06:25
Speaker A
earns this video its title. And Anthropic wrote it themselves in the Claude Code docs under a heading you would not go looking for. Skill descriptions are cut short. The listing has a character budget scaled at 1% of the context window. Names fit. Descriptions do not. When it overflows,
06:43
Speaker A
Claude Code shortens them, and I am quoting, which can strip the keywords Claude needs to match your request. It drops them starting with the skills you invoke least. So installing skill 51 can shorten skill 12 until it stops
06:55
Speaker A
matching. That is the title in shipped code. So what do you actually do about it? Three moves in rising order of effort, and not one of them is delete your skills. The first is the one people hate, and it has the best evidence
07:08
Speaker A
behind it. Cut the shelf. In November 2025, two engineers at GitHub, Anisha Agarwal and Connor Peat, trimmed Copilot's default tool set 40 built-in tools down to 13. Success rates went up two to five percentage points on SWE Lancer and SWE Bench verified. Those runs used GPT-5. The other model was
07:29
Speaker A
Sonnet 4.5. Both measured back in November 2025. Response latency fell 400 milliseconds. Their own line for it, "Too many tools doesn't always make it smarter. Sometimes it just makes it slower." The tools they cut are not gone. They are routed by embeddings, and Agarwal and Peat measured that, too.
07:48
Speaker A
94.5% tool use coverage from embedding routing against 69% from the static list it replaced. Move to search instead of load.
07:57
Speaker A
Five days after that GitHub post, Anthropic shipped tool search on its API. And the way the docs ordered the benefits is the giveaway. Bullet one is tokens, over 85% off a 55,000 token.
08:12
Speaker A
ability to pick the right tool degrades once you exceed 30 to 50 available tools. That is a vendor writing down the shadowing result in its own documentation months before anyone measured it in a paper. Their November 2025 evaluation moved accuracy on tool use on Opus 4.5 from 79.5 to 88.1. Same
08:32
Speaker A
tools, same model. The only change was that the model had to go and find them.
08:36
Speaker A
It runs today on Opus 5, Fable 5, and Mythos 5. Claude code switches it on for you once MCP tool descriptions get large, which tells you Anthropic has already conceded this argument for tools. Skills are the same shape, one
08:51
Speaker A
layer up. Move three, and hardly any harness does it yet. Route on the content, not on the label. Skill router is a small encoder and a small re-ranker, 1.2 billion parameters between them, that actually read the body. 74% top one routing across 80,000 skills.
09:09
Speaker A
The 16 billion parameter pipeline it beats manages 68. 13 times fewer parameters, 5.8 times lower serving latency. The winning move was not a bigger router. It was letting the router read the thing it is routing. So, here is where I land without hedging it. Retrieval beats installation, and it is
09:27
Speaker A
not close. For most people running an agent today, the right number of skills in front of it is small and chosen by a search step, not large and chosen by whatever you happen to install. Three receipts, 202 skills cost 21 points of
09:41
Speaker A
pass rate, and 2/3 of that was the agent picking wrong. 91.7% of the routing signal sits in a body the router is not shown. And a shipping harness truncates descriptions by design, starting with the ones you use least. Now, the case
09:55
Speaker A
against me, because it is a real one. The Data Brick study is two models from one provider on a library that tops out around 200, and they say so in their own limitations. Their contest overhead interval is wide
10:06
Speaker A
enough that they refuse to call it zero. At 10 times that library size, tokens could come back as the villain, and retrieval is not a cure, either. On the 20th of August, a benchmark landed that measures the failure which survives it,
10:19
Speaker A
and it is a specific and nasty one. Public skill routers find the helpful skill roughly 85 to 89% of the time in their top three.
10:28
Speaker A
In about a third of those same top three, they also surface a same capability sibling, right family, wrong execution contract. The fix moves you from wrong shelf to wrong jar. The villain was not a company, it is a
10:40
Speaker A
habit. Treating skills like apps, where one more install is free, and the pile is an asset. It is a search index that was never built, queried by a model that is not allowed to read it. So, which do you want your routing layer
10:53
Speaker A
to be? A vendor default you cannot inspect, or something you own and can measure?
Topics:AI agentsskill librariesClaude codeAnthropiccontext tokensskill routingskill selection failureprogressive disclosuretool searchagent performance

Get More with the SozAI App

Transcribe recordings, audio files, and YouTube videos — with AI summaries, speaker detection, and unlimited transcriptions.

App Store Google Play
Or transcribe another YouTube video here →

Free tools: TXT to SRT · SRT Validator · Merge SRT · Subtitle to Text · All tools

More from Cloud Codes

View All
Every Agent Harness Copied a Mistake From JSON — Transcript12:32
Every Agent Harness Copied a Mistake From JSON — Transcript
2,284 1 English
New Local AI Engine Everyone Will Be Using in 2027 ? (F… — Transcript11:48
New Local AI Engine Everyone Will Be Using in 2027 ? (F… — Transcript
3,465 1 English
Qwen 3.8 27B is 3X Faster With ONE Setting — Transcript10:37
Qwen 3.8 27B is 3X Faster With ONE Setting — Transcript
1,898 1 English
Nemotron 3.5 Lightning Is NOT a Transformer (Mamba + Mo… — Transcript15:13
Nemotron 3.5 Lightning Is NOT a Transformer (Mamba + Mo… — Transcript
4,065 1 English
NVIDIA Just Made AI Memory Transferable Between Models … — Transcript9:48
NVIDIA Just Made AI Memory Transferable Between Models … — Transcript
3,178 1 English
Context as a Variable: The Fix for Context Rot (RLMs) — Transcript11:49
Context as a Variable: The Fix for Context Rot (RLMs) — Transcript
3,220 1 English
Run a Kimi K3 on 8GB RAM (No GPU) — Transcript11:02
Run a Kimi K3 on 8GB RAM (No GPU) — Transcript
1,892 1 English
Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp G… — Transcript11:37
Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp G… — Transcript
2,023 1 English
This Boy Was FORCED To Join The School Fight Club | Man… — Transcript0:52
This Boy Was FORCED To Join The School Fight Club | Man… — Transcript
399 1 English
Temperatures Increases +100° EVERYONE is DEHYDRATED He … — Transcript41:59
Temperatures Increases +100° EVERYONE is DEHYDRATED He … — Transcript
10,208 1 English
#1 UFC FIGHTER Reincarnates As A SLAVE Into The STRONGE… — Transcript3:48:14
#1 UFC FIGHTER Reincarnates As A SLAVE Into The STRONGE… — Transcript
51,482 1 English
Barfi | Full Movie | 1080p 60fps | Ranbir Kapoor ,Ilean… — Transcript2:38:17
Barfi | Full Movie | 1080p 60fps | Ranbir Kapoor ,Ilean… — Transcript
278 1 English