← All transcripts

5 Best Practices for Building AI Agent Skills Transcript, AI Summary & Key Points

IBM Technology · 5 days ago · Education · 13:22 · EN

🧠 AI Summary

AI agent skills are procedural knowledge given to an agent, typically in a skill.md Markdown file, to teach it a specific way of doing a job. Reliable skills use descriptions that clearly define when to trigger, content based on real expertise and corrections, lean bodies with progressive disclosure for extra references, deterministic scripts for fragile steps, and security vetting before execution. Skills can run code with access to local files and API keys, so they must be treated like untrusted dependencies.

🔑 Key Points

  • A skill is procedural knowledge that teaches an AI agent a particular way to perform a particular job, usually through a skill.md Markdown file.
  • A skill's name and description determine whether it triggers; the name can be up to 64 characters and the description up to 1,024 characters.
  • Skill content should come from hands-on expertise or existing artifacts such as reports, run books, review comments, and pull request feedback, with corrections recorded as environment-specific gotchas.
  • Only information the model would not know on its own should be included in the main skill body; the body is recommended to stay under about 500 lines or roughly 5,000 tokens.
  • Additional material can be placed in a references sub-folder and disclosed only when needed through progressive disclosure.
  • Fragile or exact steps should use deterministic scripts in a scripts directory instead of asking the model to improvise the logic.
  • Skills from strangers should be treated like untrusted dependencies because they can run code, access the local file system, and access available API keys.
  • An audit of almost 4,000 public skills found that over 35% had a security flaw and 13% had a critical issue such as prompt injection or malware.

✅ Actionable items

  • Write skill names and descriptions that explain both what the skill does and when the agent should use it.
  • Make descriptions slightly pushy because models tend to under-trigger skills.
  • Create skills from a task performed by hand or from existing reports, run books, review comments, and pull request feedback.
  • Record every manual correction as a gotcha in the skill body.
  • Keep the main skill body under about 500 lines or roughly 5,000 tokens.
  • Move material that is only sometimes needed into a references sub-folder.
  • Match the level of prescription to the fragility of each step: use instructions for loose steps and code for fragile ones.
  • Explicitly tell the agent whether to run a script or read it as reference material.
  • Read an unfamiliar skill's contents and check what systems, files, and credentials it reaches before running it.

🤖 AI in practice

Used for

Teach an AI agent a specific procedural way to perform a job by placing instructions, environment-specific gotchas, and task-specific expertise in a skill.md file. 00:21
Generate a monthly compliance report from internal data using a task-specific agent skill. 05:39
Provide an agent with additional skill instructions only when they are needed, instead of loading all detailed material into the main context. 07:30
Perform fragile calculations in a compliance report without relying on the model to regenerate the logic. 08:58
Run an agent skill in a coding-agent environment with deterministic scripts. 09:26

Advice

  • Write each skill's name and description so they independently explain what the skill does and when the agent should use it.
    At startup, the agent loads only each skill's name and description, and models tend to under-trigger skills.
  • Make the skill description slightly pushy rather than underselling when it should be used.
    A model may skip a skill it should have used, so a touch of overselling is safer.
  • Build skills from hands-on expertise or existing artifacts such as old reports, run books, review comments, and pull-request feedback.
    The value of a skill is the specific way of doing a job that the model cannot obtain on its own.
  • Record every manual correction as an environment-specific gotcha in the skill body.
    Otherwise the same correction will have to be made repeatedly.
  • Keep the main skill.md body under about 500 lines of text, or roughly 5,000 tokens, and move larger material into a references sub-folder.
    Every line in the selected skill body competes for the model's attention and context.
  • Match the prescriptiveness of a skill step to how fragile it is: use instructions for loose steps and code for steps that must be exactly right.
    The model can take different valid paths for loose steps, but regenerating fragile logic produces inconsistent results across runs.
  • Explicitly tell the agent whether to run a script or read it as reference material.
    Leaving the script's intended use implicit allows the model to guess.
  • Remove guesses from parts of a workflow that must be correct by hard-coding them as deterministic logic rather than relying only on more tests.
    A test catches only what was already considered, while deterministic code takes the guess out of the loop.
  • Vet any skill obtained from a stranger before running it, as if it were an external software dependency.
    A skill can execute scripts that access the local file system and available API keys, and an open standard does not establish that an individual skill is safe.
  • Read an external skill's code and check what it reaches out to before installing or running it.
    An audit of almost 4,000 public skills found security flaws in over 35%, with 13% containing something critical such as prompt injection or malware.

What it could not do

  • An AI agent may fail to trigger a skill it should have used. — The model sees only the skill name and description at startup, and vague descriptions can lead to under-triggering.
  • Having an LLM write a skill without supplying real expertise produces generic instructions such as handling errors appropriately and validating inputs. — The transcript describes this output as generic mush that mostly repeats what the model already knows.
  • A detailed skill body can become expensive by consuming context and competing with everything else already in the context window. — The selected skill's body is loaded into the shared context, so unnecessary detail uses the model's attention.
  • A model's probabilistic decisions are not always consistent across multiple runs. — This is especially problematic when the workflow contains steps that must be exactly right.
  • More testing alone cannot guarantee trust in an agent skill. — A test only catches conditions that someone already thought to check.
  • An agent skill can execute unsafe software and access the computer's local file system or available API keys. — This risk applies to executable scripts contained in a skill folder, including skills obtained from strangers.

🧰 Tools & AI usage

  • Claude Code — Runs agent skills and their executable scripts.09:26
  • Codex — Supports a similar workflow for running agent skills and scripts.09:29

AI is used for

  • Perform a specific job using procedural instructions — An agent skill supplies the particular process and expertise that the model does not already know.00:21
  • Select whether to run a skill — The agent uses each skill's name and description at startup to decide which skill to trigger.02:01
  • Execute routine steps and scripts — The agent handles routine work while deterministic scripts perform fragile logic that should not be improvised.08:09

🔗 Links mentioned

📄 Transcript

Searchable transcript of 5 Best Practices for Building AI Agent Skills — IBM Technology (13:22). Search for a phrase, then click its timestamp to jump straight to that moment in the video.

Captions sourced from the original video on YouTube, published by IBM Technology. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.

00:00 Agent skills are about the simplest way there is to make an AI agent better at a specific job. Because they're so simple, they're also really easy to get wrong. Yeah, so we are going to cover the five agent skills best practices for building these skills. And a quick reminder of what a skill even is. It's procedural knowledge handed to an AI agent. The model already knows plenty of facts, but what it doesn't know is your particular way of doing a particular job.

00:32 So a skill teaches it that. And the format for this, I mean, it's almost comically simple. It is basically just a skill dot MD file. So MD markdown in a folder. But Martin, here's the part that worries me. Agent skills are handing a probabilistic model of folder of text and trusting it to run a fragile multi-step job. Yeah, and a skill can also contain and run code.

01:04 So in sorting one off the internet means running a random person software on your machine. And while there is an open agent skills standard defined at agent skills dot I O. These are some of the things we need to consider when creating AI agent skills. So Martin, how about we get started with the best practice? Number one. Let's do it! Best practice number one.

01:29 The description is the trigger, by which I mean it's the agent description that decides whether a skill ever actually runs. And by description, you mean? Every skill.md file like this guy here opens with a bit of YAML so in here we define a name and we also define a description. And there on the short side, according to agentskills.io, a name can be a maximum of 64 characters, and a description, which describes what the skill actually does, is limited to 1,024 characters.

02:11 So I might have, let's say 100 of these skills installed. Now the agent can't read all of them at once without filling up its context window. So at startup, it just loads the name and the description of each skill. And here's where the best practice comes in, right? Yeah, yeah. So the name and the description need to contain enough information by themselves so that the agent knows when to use it.

02:36 So a compliance skill of generates reports might be just a little bit too vague. Right, right. The fix is to say what the skill does and when the agent should make use of it. Give me a good version of the description. Ah, this sounds like a bit of a test, but fortunately here is one I made earlier. So generates the monthly compliance report from internal data used when someone asks for the compliance report or the monthly filing, something like that, it says what the skill does.

03:10 And it also says when the skill should do it. I've also heard the description should lean a bit on the pushy side. Pushy? Models tend to under trigger. They might skip a skill they should have used. So it's safer to oversell the description a touch rather than undersell it. Oh, like LinkedIn posts, got it. So that's getting a skill to trigger. What goes inside it once it does?

03:38 Well, that part's all yours, brief. All right. Yeah, and this is where a lot of skills go sideways, because the temptation is to just have the LLM write the skill for you. Yeah, the first skills I created were all like this. I'd say like, hey, AI agent, write me a skill that does X and it generated stuff, which honestly, I only glanced at that then call the skill, build a complete success.

04:02 And what you get out of that is very generic mush. Handle errors appropriately, validate inputs, stuff the model already knew. Which brings us to the best practice number two, build from real expertise. Meaning what exactly? The whole point of a skill is your specific way of doing a specific job. So the content has to come from somewhere the model can't get to on its own.

04:28 And there are two ways to do that. One, you walk through the task by hand once and write down what actually worked, including the corrections you made along the way. Or two, you synthesize it from artifacts you already have, so things like old reports, run books, review comments, PR feedback. Yeah, and fellow Brit ,Simon Willison, has a line about this.

04:49 He says, keep the domain expertise and let the agent do the routine part. That's right. And Mr. Willison is absolutely right about this. You bring the expertise. The model brings the typing. And what does this mean for the skill.md body? I feel like I've drawn this box a few times already. Yes, you have. It means the highest value section you can put in the skill body is gotchas.

05:19 Environment-specific facts that defy reasonable assumptions. Every time you correct the agent by hand, that correction is a gotcha. Write it down. Otherwise, you'll be making the same correction next week. And the week after. Mm-hmm, you're not wrong. Now, even when you do all this right, it doesn't always work for the first time around. I built a skill for a monthly compliance report from an actual real report I had done, which is real expertise, exactly what we're talking about.

05:48 But on the first run, the row totals didn't add up to the common totals. So the math was right. Yeah, the math was wrong. On a compliance report. Yeah, but that's the good kind of wrong because it's the kind you can catch. We'll come back to this one. Point is, a good skilled body gets thorough. And thorough, that means it can get... really, really long.

06:11 Yeah, and long, as it turns out, gets expensive. Which brings us to best practice number three, which is spend context wisely. At startup, the agent only sees the name and the description, but when it selects a skill, that's when it actually reads the rest. So now we take a look at the body of the skill.md file and bring that into context. And that context is shared with everything else already in the context window.

06:40 Yeah, every line here in the skill body is now competing for the model's attention, which means the goal is to write left. Which feels a little bit in opposition to our last best practice, because the more detailed and thorough a skill body is, the better. You'd think so, but the model is already smart. It knows stuff. It knows what a PDF is. It knows, what a database migration does.

07:05 So only write down what the agent wouldn't know on its own. Stuff that's not already part of the AI model's training data. And we can put this into numbers. It's recommended to keep this body here of skill.md to under about 500 lines of text, or we could say that's roughly about five thousand tokens and just keep it at that. And when it's bigger than that?

07:30 Well, when it's bigger than that, just split it out. The skill folder can actually hold a sub-folder called references, and this is a directory, and the agent will only open the files in the references when it actually needs them. It's part of a pattern called progressive disclosure, which is to say disclose additional information like this only when it is needed.

07:54 So staying lean is one thing, but some steps you really don't want the agent guessing out at all. So guessing at what? Every time the model runs your skill, it reads the instructions and improvises through them. For loose steps, that's fine. A lot of paths get you to the right answer. But for a step that has to be exactly right every single time, you don't want the model regenerating the logic on the fly.

08:19 Hmm, this sounds to me like a... Best practice number four? It is. Make use of deterministic scripts. Match how prescriptive you are to how fragile the step is. So, loose step, write instructions. Fragile step, right code. Okay, and code, that code is where? The skills folder that holds a scripts directory. So scripts directory, same idea as references.

08:50 You drop a script in there, say script one. And the skill body just tells the agent, run this. The script doesn't get loaded into context, so you'd save tokens as well. And it's more reliable than having the model improvise from scratch every time. Now you do have to be explicit about intent though right, so the model doesn't just kind of read the script as reference material.

09:16 Yeah, exactly. Say, run this script or read this as reference. Don't leave that to a guess. Yeah, and that works in Claude Code, but it isn't just an Anthropic thing. Opening a codex works roughly the same way as well. So back to my compliance report, the one where the row totaled and reconciled. Yeah, the math one. Yeah, that one. The masked up is now a deterministic math script.

09:45 The model doesn't add the numbers anymore. It calls a script that adds the numbers. And that whole class of bug just goes away. Yeah, because the script doesn't guess. This script, math script here. It's running code. Right. And the answer isn't write more test. You can test your way to trust. A test only catches what you already thought to check. The answer is, for the parts that have to be right, take the guess out of the loop.

10:13 Yeah, right. So this is moving away from probabilistic, which is what an agent does by itself into being more of a deterministic model instead. And I've been doing this a lot in the skills that I build. I don't want to make the agent make guesses if something can be hard coded as logic deterministic, because if the agent is having to make probabilistic decisions, those decisions will not always be consistent across multiple runs.

10:41 So it was skill you built yourself and that you've read and that your hardened the fragile steps on, that's a skill you can trust. But not all skills you'll run, you build yourself, right? Right? What about the skills you'll run that came from a stranger? Which brings us to best practice number five, vet a skill before you run it. So I said a skill can run code.

11:07 The stranger software on your machine thing. So a skill folder can contain, as we said, executable scripts that we can run, scripts that can access stuff, access stuff like, for example, the local file system on your computer, or in fact, it could access any API keys that you happen to have lying around. Which is exactly what makes skills so powerful.

11:36 But I saw an audit from earlier this year that scanned almost 4,000 public skills, and over 35% of them had a security flaw of some kind, and 13% had something critical going on like a prompt injection or straight up malware. Which means we have to treat an agent skill like, well, like any other dependency, just the same way we'd check a random package before pulling it into a project, which is to say, read what it does and check what it reaches out to.

12:13 So just because an agent's skills are an open standard, that doesn't say anything about whether a given skill is safe. Okay, so that's our five best practices. A good skill is... A skill the agent will actually trigger. Built with real hands-on expertise. Kept lean so it doesn't fill up the context window. With a deterministic script whenever a guess would be dangerous or just inconsistent.

12:45 And vetted before it ever runs. And look, this is all moving fast. It's an open standard and more agentic platforms are adopting it. So this list is going to grow. Which is really us asking what did we miss? If you've built agent skills and maybe you've got a sick best practice, well drop it in the comments. Preferably one that isn't malware. Yeah, preferably.