Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel Transcript, AI Summary & Key Points

00:00 Hello. My name is Nikolai Petrov and we will be talking about skills and agents knowledge of those skills. We'll start it with the simple prompts where we've been playing with the new stuff, new work being done by the agents, repeating those prompts over and over again, and then we start packing those into the skills, creating repeatable, reproducible, portable type of their functionality or capabilities that we can teach agents to work with.

00:33 And some of those skills are very important, hundreds of stars on GitHub. There are different types of their skills. Some teach models something new, bringing new knowledge or capabilities to the model, but they're also type of their just steering or process skills where we do not bring any additional information but just getting repeat table flow of the actions that we want agents to perform.

01:04 Both are useful. In this session we will be talking about new knowledge type of the skills and we'll see what we can get from the model's understanding of those skills. So we get our first skills packed, start using them basically starting with their the feel of does it work or not. But there is also their more technical way and more deterministic way for the checking if skills are working or not and what's the impacts of those skills are evolves.

01:43 There are different formats where evolves can be defined, but at the sense this is task definition that you want your agent to be able to solve. You put it in the place, ideally more deterministic, so we can validate it. And we can two arms. One where no skill loaded and their second one where we're loading the skill and providing this additional information to the context.

02:09 And we compare how they runs. There are multiple things. One is how many evolves been solved. In this case with no skill we're solving three out of five. With skill we're solving all five, meaning that we get two more tasks. There also would be implications on the tokens usage or time that it took those tasks to be completed and other factors. But at end we are comparing those and seeing what makes sense what not.

02:39 And this is one of the main tools for evaluating the skills and saying that they not just nice ideas but actually working pieces of instructions and context for the agents. But what next? We get to the stage there we have their working skill. It solved some evolves. But we know that there should be a better way. We have models evolving. We have some maybe cases there we want to get skill more efficient, meaning consuming less tokens.

03:16 So what's their their options there? One is just change line by line and try brute force all the different combinations. Another would be to think through what makes sense so maybe use something as the user feedback on the skills to see where skills perform well there it's not. And with new models just running skill and seeing how it works, what's the options there?

03:49 And how large those modification should be? Just lines, sentences, ideas. So what we found to be useful is instead of relying on their luck, we can put more deterministic process here using idea of their items from their fact score. We can split their skill and instructions into those items and for each of those have pair of question and answer that we expect.

04:29 This could be either general ideas like proposition in this case or contracts there we put more definitive expectation on their action. And in general, we will be getting their some predefined amount of the items per skill and we will have let say 25 or 30 items per skill. Then for each of the items, we will again run two arms, one without skill and one with skill.

05:10 And we will have judge or panel of judges that will be evaluating model answers. So we asking those questions to the model that we interested in. We evaluating those questions with their judge, ideally different family of the models or panel of judges. We will ensure that the question do not leak answers. We depending on the case can compare with no tools or additional information to check their internal model understanding or provided with their some tools like web access.

05:47 So it would be more practical. And we can get their simple forward counts matrix. Basically where we are checking that the model uh knows or do not know something and how this maps to their skill impact on this. So the simplest option is where model was able to answer this particular questions even without skill or with skill. This means that skill have no impact on this and this was already part of the model's knowledge.

06:29 The second one is where without skill actually model was able to answer this correctly. And by adding skill we saw degradation in performance and model capability to talk or explain particular aspects. The ideal case is where without skill model wasn't able to answer something and with skill we start seeing that it can answer those new type of the questions and scenarios and this is what we were aiming for teach model something new.

07:02 But there also can be there fourth option where skills were written not ideally or in some might be not clean way where model wasn't able to answer it either without skill or with skill and while this topic might be relevant and important for the model it still sort of misses some of those concepts. The simplest thing to fix are instructions or items that actually hurts performance and for popular skills those are pretty rare.

07:41 So in their uh box below you see that it's just one case out of the mostly 400 items in entropic skills and others don't have those. But for their smaller skills that are less popular you probably having more of those. This is what we are seeing with some of our um experiments. And this is just straightforward the simplest thing to fix to ensure that your skill do not reduce model capabilities and performance and bring only prevalent context there.

08:22 And we also can actually color this in this scenario this is done by lines but actually going with items you might have there different coloring for a single line or a single sentence. And this just gives your example there visualization where this would be helpful details and instructions in terms what new stuff you bring to the model and what is something that model already knows.

08:55 Cost wise we are running all those questions and asking model but still relatively this is cheap for 29 skills 14 bucks of there sonnet and haiku calls with their sonnet used as a judge. And there couple ways how we also can check their strictness of their some knowledge. We have lenient and strict ways of the checking this where first checks the general idea while the second one is checking for exact details and you can think about the difference between general knowledge and understanding of the topic to the

09:43 understanding of the exact keys parameters and exact values or steps. And what we see across the models is the difference is large especially with their larger frontier models where we see opus dropping from 70% understanding of their skill in case of the general understanding to just 22% of the specific details. And this is much our expectations. We see that new models are pretty well generalized and understand concepts.

10:19 But it's still helpful to provide their with exact details and data that will be useful for their solving those problems. Here is outlook of their comparing those three popular rappers with the skills across different models. So one of their interesting observations here is that smaller model have in general lower understanding of their skill with their larger sonnet having better understanding, which is why we have bigger models.

10:59 But also we see that overall their models already understand a lot of their details and context of those skills. And you would expect that with the newer models and more advances in their better training capabilities, we will have more and more in terms of the model's understanding. But also we see that this pretty significantly differs skill to skill.

11:29 And it's not bad. Some skills are intended to be directional. But it's also useful piece of information to see how those skills bring new knowledge to the different model and different families and this should be run ideally in the broader set of the models. But just having their this novelty piece of information is just one of the signals. It's basically initial vector for optimization.

12:06 At the end, we still have to run evolves and validate that all all changes that we try to make with their skill actually works. So the one of their simpler scenarios would be we can look on their skill and try to remove some or all of their already known pieces of information from the skill and run evaluation. It might come up with their same scoring.

12:34 Basically we will be getting their same results model still capable of solving those problems. Then great we can reduce skill size, save tokens, save context. And instead of their brute forcing and guessing what we can or should remove or add or how to fix their uh degradations we can have a guidance. So all of those measurements experiments been done with the docume which is a framework for their comparing everything related to the agents and context where it's is skill, agents MD, MCP tools, whatever.

13:17 And novelty is one of their pieces of the functionality where you can get this initial vector for their skill optimization and evaluation but there are also uh evals. There is assessment of the context impact, routing, comparing their uh descriptions, naming, bunch of the things that related to their skills where it would be helpful to actually get their more deterministic evaluations to make conscious decisions in terms of the what should be done with the skills.

13:54 So thank you for your time and please go ahead and write better skills.