Speaker A
[music] Yes, so hi everyone. My name is Philip. I'm based out of Germany. I'm part of the Google DeepMind team, mostly working on Gemini API and agents. And we are going to talk about why you should not ship skills without eval. And maybe
Philipp Schmid from Google DeepMind explains why AI skills need thorough evaluation before deployment to ensure reliability and performance.
Key Takeaways
- Always evaluate AI skills before shipping to avoid poor performance in production.
- Differentiate between capability and preference skills to manage skill lifecycle effectively.
- Human-written and well-tested skills outperform AI-generated ones on average.
- Skill descriptions are critical as they guide the model's decision to invoke a skill.
- Regularly run evals to monitor skill relevance and retire outdated skills promptly.
Summary
- Philipp Schmid introduces the importance of not shipping AI skills without proper evaluation (evals).
- Many users employ coding agents but lack evals to verify skill effectiveness.
- Skills are categorized into capability skills (teaching new abilities) and preference skills (customizing behavior).
- Skills typically consist of a title, description, and skill body, with descriptions guiding when models should use them.
- User-invoked skills are powerful for dev tasks, while model-invoked skills are more common in customer-facing agents.
- Skill Bench is a benchmark evaluating thousands of skills, showing that skills generally improve model performance.
- AI-generated skills can sometimes degrade performance, highlighting the need for human-written and tested skills.
- Eval processes involve multiple trials due to agent non-determinism and help decide when skills can be retired.
- Keeping skills concise is crucial to reduce context cost and maintain efficiency.
- Proper evals protect preference skills that encode domain-specific knowledge, ensuring consistent use and quality.
Chapters
- 00:00Introduction and Importance of Skill Evals
- 01:17Current Use of Skills and Lack of Evals
- 02:30Challenges with AI Skills and Task Failures
- 03:38User Interaction with Skills and Agent Behavior
- 05:00Structure and Definition of Skills
- 06:31Capability vs Preference Skills
- 07:53Durability and Use Cases of Preference Skills
- 08:57Skill Bench Evaluation Results
- 10:00Best Practices for Writing and Using Skills
- 15:04Evaluating Skill Performance and Retirement
Full Transcript — Download SRT & Markdown
Speaker A
before we start, I need a little bit of your help. So if you could raise your hands if you use coding agents to write code.
Speaker A
before we start, I need a little bit of your help. So if you could raise your hands if you use coding agents to write code.
Speaker A
Okay. Do you have evals for those skills? Okay, yeah, that's um not a lot of hands. Everyone uses skills, no one has evals. Hopefully we can fix that today.
Speaker A
So yeah, hopefully every hand goes up, right? And do you use skills with it?
Speaker A
none of those skills had evals. Most of them were AI written um not really tested and it's very hard to know if your skill is good or bad because like agents are really non-deterministic. So you might not know if your uh task fails
Speaker A
Okay. Do you have evals for those skills? Okay, yeah, that's, um, not a lot of hands. Everyone uses skills, no one has evals. Hopefully, we can fix that today.
Speaker A
the agents we use and the agents we build. Um most of us use agents for writing code, doing productivity work.
Speaker A
And like very important is like why checks fail in production. And, um, Skill Bench is a very popular and nice, like, eval or benchmark which, um, indexes like over 50,000 skills from, like, it up and, like, tries to look into them and almost
Speaker A
And if your agent does not invoke the skill on the first time, you will notice it very quickly. You stop your your task and reprompt it or like use slash commands for for triggering those skills. When you build an agent inside
Speaker A
none of those skills had evals. Most of them were AI written, um, not really tested and it's very hard to know if your skill is good or bad because, like, agents are really non-deterministic. So you might not know if your, uh, task fails
Speaker A
there's a big difference between the agents we use and how we use skills and the agents we build and how our customers might want to use skills in like the context there. And what is a skill? I mean, every one of us knows
Speaker A
because your skill is bad or if your task fails because it's way too challenging for the model. So, um, very important, um, before we go into it, I want to, like, really make sure that we know the difference between
Speaker A
disclosure. So, most of the skills start very small. So, you have the title and a description. The description is normally part of the model's context. So, the model knows when to use the skill. Second layer is we have a skills body with more
Speaker A
the agents we use and the agents we build. Um, most of us use agents for writing code, doing productivity work.
Speaker A
And I like to differentiate between two kinds of skills. So, they are capability skills and preference skills. Capability skills teach models something they cannot do consistently at the moment.
Speaker A
That's the agents we use. It's like Antigravity, Cursor, Claude Code. And there you are the engineer and you have context about skills, right? If you write some prompt to, I don't know, like help me build a new Gemini API feature.
Speaker A
us when we can retire skill and when not. And then we have preference skills.
Speaker A
And if your agent does not invoke the skill on the first time, you will notice it very quickly. You stop your, your task and reprompt it or, like, use slash commands for triggering those skills. When you build an agent inside
Speaker A
and those uh preference skills are then protected with e-walls where because most of like the foundation models might not uh integrate the knowledge which is very specific to your use case or your domain. And preference skills are very
Speaker A
your application for consumers or customers, they have no idea about what a skill is. They don't start their prompt with "Use customer support skill to, like, help me refund" or "Use refund skill to help me solve my problem." So,
Speaker A
So, do skills work? Yes, they do work and I going back to a skills bench which has an update of 1.1 which has evaluated all kinds of open and closed models in different harnesses showing that skills on average improve the performance by
Speaker A
there's a big difference between the agents we use and how we use skills and the agents we build and how our customers might want to use skills in, like, the context there. And what is a skill? I mean, every one of us knows,
Speaker A
contributions. And then they did a second analysis on self-generated or AI-generated skills, right? It's very easy if you are in a coding agent and you work on something, tell the model create a skill and then it writes a
Speaker A
hopefully by now, what a skill is. It's like basically really a folder with a skills.md file in it and then some additional assets to make that skill really work. And the big difference with skills is that they work on progressive
Speaker A
provide. Uh AI-generated skills can uh impact performance negatively. And that skills or skills.md files should be below 500 lines of words. So, if you have your laptop open and have a skill available, if you open that and if it's above 500
Speaker A
disclosure. So, most of the skills start very small. So, you have the title and a description. The description is normally part of the model's context. So, the model knows when to use the skill. Second layer is we have a skills body with more
Speaker A
uh based on the context and the description, the model decides to use or read a skill to, uh uh, get more context to solve a task. And then there are user-invoked skills. And I think people underestimate how powerful user-invoked skills are. Um,
Speaker A
instructions, more details, and hopefully more references to external files. And then you can really go deep in those reference files where there's all of the context the model needs to discover to solve the task.
Speaker A
the very uh, normal dev work which could be run in a script should most likely be a user-invoked, uh, skill. And when you build agents for customer, you don't have those user-invoked skills. We are only working in the model-invoked
Speaker A
And I like to differentiate between two kinds of skills. So, they are capability skills and preference skills. Capability skills teach models something they cannot do consistently at the moment.
Speaker A
write good skills. And most importantly, if you work with model-invoked skills is the description because the description are most of the time two sentences we provide to the system instruction to help the model know when it should use a skill or not.
Speaker A
Maybe it's like, I don't know, like tracing some logs, creating a new React app. And those capability skills are temporary. So, the better our model gets, the more likely it is that we can remove those skills. And evals will tell
Speaker A
that skill and then how it should use that skill. Um, very common is like use that skill if you are working on a React application, for example. And then, of course, the when.
Speaker A
us when we can retire a skill and when not. And then we have preference skills.
Speaker A
the Interactions API if you're working on like a chat application. So, you need to give the model like clear instructions and directives on when it should use the skill and how it should use the skill. And similar to what we have seen in the
Speaker A
Those are more durable, mostly encode some references. So, if you have a specific workflow in your team or a specific style language or other preferences which are very specific to your company, um, will have or create preference skills
Speaker A
always pay that 100 200 tokens cost and you don't want to have a super long description because then you always have to pay that. When you have a very long skill MD file, it will be always read into context when the model decides to
Speaker A
and those, uh, preference skills are then protected with evals because most of, like, the foundation models might not, uh, integrate the knowledge which is very specific to your use case or your domain. And preference skills are very
Speaker A
That's why we want to keep it as concise as possible, but still include all of the reference and details for the model to solve the task. And then of course, the layer three is like we can have those reference files where the model
Speaker A
valuable, so we really want to make sure that those are working and we don't, like, update our agents to, uh, degrade performance.
Speaker A
to deploying to AWS and deploying to Google Cloud. Those should not be part of your skill MD file. Those should be references. That you have a reference for AWS, reference for Google Cloud, maybe a reference for Azure so that the
Speaker A
So, do skills work? Yes, they do work and I going back to a Skill Bench which has an update of 1.1 which has evaluated all kinds of open and closed models in different harnesses showing that skills on average improve the performance by
Speaker A
Then we should set the right level of freedom. Um I see many people very clearly describing the exact workflow in a skill. Step one, go there.
Speaker A
roughly 15%. Skill Bench covers around 100 different tasks, uh, based on, like, coding and also productivity across different languages. It's, uh, openly available and they have a very nice website, a very nice leaderboard, are also very open for, uh, community
Speaker A
tokens for that exercise. You can create a script. You can tell the model use that script to run a specific workflow.
Speaker A
contributions. And then they did a second analysis on self-generated or AI-generated skills, right? It's very easy if you are in a coding agent and you work on something, tell the model create a skill and then it writes a
Speaker A
config, update the port, and then like deploy again. The model knows what to do. Just like hey, if we need to change the config, here's the file, make the change. Then uh don't skip negative cases. So, we always look at the when we
Speaker A
skill.md file. We maybe look at it very closely. It roughly covers what we want to do and then we just accept it and start using it. And what I found out is that human-written skills are the best we can
Speaker A
trigger. Maybe you work with React, maybe you also work with Angular, and the model always loads the skill if you are working in like a web development environment, but if you are very specific for like hey, only use that
Speaker A
provide. Uh, AI-generated skills can, uh, impact performance negatively. And that skills or skills.md files should be below 500 lines of words. So, if you have your laptop open and have a skill available, if you open that and if it's above 500
Speaker A
Um And then test early. So, that's what we are going to look at. We should really try to test when you create a new skill.
Speaker A
lines, you should definitely look at the skill after our session. And, um, the last topic about what is a skill and how a skill works, uh, we have different ways of triggering our skill, right? We can have a model-triggered skill meaning
Speaker A
is not over triggering the skill and confusing itself. And then if you have already some customer or production traces, try to include those as well because nothing is better than than real-world data.
Speaker A
uh, based on the context and the description, the model decides to use or read a skill to, uh, uh, get more context to solve a task. And then there are user-invoked skills. And I think people underestimate how powerful user-invoked skills are. Um,
Speaker A
no-ops. And what he found is that AI generated skills tend to include a lot of no-ops. And no-ops basically is an instruction which does nothing to change the agent's behavior. It's like before making an implementation easy to read.
Speaker A
and they most of the time just accept, uh, the overhead by pro- like adding it into the context. I have, like, many user-invoked skills for more workflow type of tasks like creating a pull request, uh, staging documentation, and all, like, of
Speaker A
So, um definitely look at those no-ops. He have has published a very good skill in in his like skills repository. Uh and then last but not least, uh know when you should retire skill. Um skills are not there to live forever. Models get
Speaker A
the very, uh, normal dev work which could be run in a script should most likely be a user-invoked, uh, skill. And when you build agents for customers, you don't have those user-invoked skills. We are only working in the model-invoked
Speaker A
uh for your tokens, and then also um don't keep like it redundant. So, save cost at the end and maintenance also as well. And to look at a little bit of a practical example and also how you can
Speaker A
skills, and that's where we are, like, focusing on for the small eval section we are going to look in a second. So, writing skills, um, is an important topic. Uh, we're going to look at eight, um, examples on how or tips on how you can
Speaker A
agents. And the Interactions API was released after the last training of Gemini. So, the model or Gemini 3 and like 3.1 or even 3.5 has no context about what is the the Gemini Interactions API. So, we decided, "Okay, let's look
Speaker A
write good skills. And most importantly, if you work with model-invoked skills, it is the description because the description is most of the time two sentences we provide to the system instruction to help the model know when it should use a skill or not.
Speaker A
Those are uh based on like data we see uh from real users trying to generate uh Gemini code from um synthetic generated uh test cases, and also from like feedback we see people like, "Hey, the model is like using Gemini 2.0 even if
Speaker A
And it's bad if your description is too weak because then it might trigger too often or it might not be triggered if you need it. So, very important is the why and the how for the model. So, why it should use
Speaker A
simple uh assets. So, one of that was a JSON file with all of our test cases.
Speaker A
that skill and then how it should use that skill. Um, very common is like use that skill if you are working on a React application, for example. And then, of course, the when.
Speaker A
Uh we have a should trigger. That's basically there to tell us if the agent should read the skill or not read the skill. And then we have different expected checks. Uh we look at them in a little bit. Those are basically uh very
Speaker A
And we should write directives instead of essays. So, we should not say something like, "Hey, the Interactions API is recommended for multi-chat, um, multi-chat because it handles, like, session state," and it's like where you should be way more directive like use
Speaker A
In this case, it was the Gemini CLI which uh passes the output and returns runs it so we can like take a look at the the outcome whether we have valid code for the interactions API or not.
Speaker A
the Interactions API if you're working on, like, a chat application. So, you need to give the model, like, clear instructions and directives on when it should use the skill and how it should use the skill. And similar to what we have seen in the
Speaker A
SDK? Do we use the correct um model? Do we use the correct methods? Do we use any old patterns?" And we created um very basic asserts for all of those cases, which are very cheap to run. So, we can run our skill against the evals
Speaker A
Skill Bench results, we should keep the skill lean and layer information. So, the description is the cost you always pay on every model invocation. So, on every model call, the description is part of the model context. So, you
Speaker A
a judge. But of course, you can use LLM as a judge if you have like more complex skills which need to look at the whole traces or the whole steps taken. And a very easy case is like you just create
Speaker A
always pay that 100-200 tokens cost and you don't want to have a super long description because then you always have to pay that. When you have a very long skill.md file, it will be al
Speaker A
and then like try to um improve your skill based on that. And that's also how we now uh eval skills at Google DeepMind. So, the we don't use YAML, but it's like just as like uh an example. Uh we have um
Speaker A
tests or evals alongside every skill we have internally at Google DeepMind. Um every test has multiple cases with like a prompt. Uh we all run them in like clear workspaces, so you can define your workspace or environment if it
Speaker A
should include additional files like your application environments. You have uh startup commands, which basically preloads or installs libraries into the environment. And then you have script evals or data. Those are those regex where we look at all of the traces to
Speaker A
see what the skill triggered, was a certain command run, was a certain CLI run. And then we also have LLM as a judge where we have some expectations, which are basically matched against like hey, did it trigger the skill, did it
Speaker A
run a certain bash command to like also evaluate it. And we run them on every change to the skill. So, if a change happens to or like a diff to the skill file, the eval will be run, and
Speaker A
there will also be a result, and the change will not be merged if it is not improving the test cases. So, we always have those regression tests for every change to the skill, and you can only change the skill if it improves the eval
Speaker A
or add new evals. And um yeah, that that's how we we we basically manage it. And then last but not least, uh 10 examples for best practices for skills.
Speaker A
You don't need to take photos. They are in the blog post I can share later. So, um the I mean, we had it many, many times.
Speaker A
The the skill skill description is very important. Uh we have seen 50% of the failures uh because the skill was not triggered correctly because the prompt of the user was not uh detailed enough for the model to understand, "Hey, I
Speaker A
need to use that skill to solve that task." And especially if you build agents for others, they are not aware of the skill descriptions you have for your model and for your skill. So, they might write something very um shallow and then
Speaker A
the model needs to know, "Okay, I need to trigger that skill." Um we should write directors over passive information. So, we should always think about it. You should tell the agent what to do or not what to do
Speaker A
and not just like, "Hey, if you feel happy today, please use the skill." Um include negative tests. Uh we always forget negative tests. Start small. Even like 10 to 20 skill eval samples are better than nothing. You will be
Speaker A
surprised on how much you will find even from like five to 10 examples. And then like definitely create outcomes, not paths. Uh we don't want to test if the model loads the skill on like the first turn. We really want to test if it can
Speaker A
achieve the task based on the prompt. And if it loads the skill, it loads the skill. If not, then not. If it um loads the skill after five turns, that's also okay. Then we want to have isolated runs
Speaker A
because coding agents are very good at finding or cheating. So, if you run inside uh your existing environment, it might look up previous chats or it might look up some other executions and then like try to cheat it and get the context from the
Speaker A
skill without even using the skill. Then definitely run more than one trial when running evals. Like agents, our models are non-deterministic.
Speaker A
Maybe the first one works, the second one doesn't. So, always run the to six uh trials per case and to measure reliability Uh um test across different harnesses if you work with like or if you have employees or
Speaker A
um people working with like different harnesses, not only just evaluate against Claude or anti-gravity. If you have people working with cursor, try to include them as well because agent harnesses behave differently and of course model behaves differently. So,
Speaker A
maybe your skill is very good with a Gemini but very bad with Codex and then you have uh customers, consumers using your harness with Codex and then it fails. And then um create your evals.
Speaker A
So, if your um model is good enough that it doesn't need the skill anymore, keep that eval. You don't need to throw that eval away because you throw the skill away. You can keep that eval to make sure that the model or the
Speaker A
agent keeps the performance and as soon as you start seeing some degradation, you can reintroduce the skill. You can maybe tweak some other tools or pieces to keep like the the performance up and then really detect when you can retire
Speaker A
skill and you will be very surprised with all of the model updates how fast you can retire skill which you might need it like six months ago but not today anymore. And I have some homework for you. So, if you are back from
Speaker A
holiday on Monday, um pick uh the most used skill and write five test prompts. Uh you can also use your coding agent and ask it to see look at your trajectories which are my most used skills and then try to create some some
Speaker A
skills. It's you have seen it's like very easy to write your eval harness. It's like a JSON or YAML file and then like some Python script which runs your coding agent or your agent harness and then like look at the outcome.
Speaker A
Definitely uh try to look at the removing no-ops. Maybe it does not change the eval performance but it helps you save cost because all of the tokens which are not helpful or not changing the agent behavior are money you will
Speaker A
like spend. So, look at um writing great skills uh from Matt. It's you can find it on on GitHub and then also run ablation test. So, run always evals with your skill loaded and without your skill loaded. Only that way you will know when
Speaker A
you can retire skill or if a skill is really helpful for your performance. So, don't ship skills without evals.
Speaker A
Thank you. [applause]
Topics:AI skillsskill evaluationGoogle DeepMindGemini APIcoding agentspreference skillscapability skillsSkill BenchAI-generated skillsagent non-determinism
