Context Engineering for Coding Agents Across 1,000 Repos Transcript

00:05 My name is Ajay Prakash and I'm a software engineer at LinkedIn and uh so I'll be talking about context engineering at LinkedIn. So um imagine you are a engineer in a software company a big big software company and you are happen to be uh on call for your team and your team owns set of very critical services which has which is used by uh like millions of users and all of a sudden uh so you get a pager alert about a latency spike in one of your service and it is causing disruption for a lot of your users.

00:49 While you're trying to figure out how to deal with this issue and how to mitigate it faster, you take the link to that alert and give it to a coding agent uh like GitHub copilot. the coding agent. It fetches the instructions on how to figure out how to uh debug this issue or debug alerts in your company and it identifies that uh so this issue is coming from uh one of your service and uh then it also fetches instruction on how to debug that service and based on the instructions it fetches the logs metrics and recent

01:24 changes and deployments that went in and it identifies that the issue is coming because of different service which is a downstream service which is throwing a lot of errors. Then the coding agent also goes and fetches the instruction on how to fetch how to uh debug the issue in the downstream service and it does the same thing based on the instructions on how to fetch the logs metrics and it identifies that there was a recent uh PR code change that went in which was buggy and it got deployed and it is causing all of

01:59 the issues. But the impressive thing is the agent was figure out able to figure out all these things and uh while you were uh figuring out how to deal with the tissue and it will also once it finds the information it also creates a detailed summary for you on what it found and what was the root cause and what are the steps uh actions need to be taken to mitigate the issue.

02:25 And it uh once you confirm that and verify it will also go ahead and mitigate the issue and take the action to mitigate the issue. And this happens in matter of uh few minutes. Right? And uh the coding agent doesn't stop there. It will also go and creates a detailed report and updates the incident management system for you. And also it goes beyond where it fetches instruction on how to make the coding on the affected service where the bug was happening and it also creates a PR for you and as I said it all happens in

03:03 matter of few minutes. This would have easily taken u without coding agents few hours of time and this is not a fiction. So this is how teams at LinkedIn are using coding agents as a very productive co-workers with a deep knowledge of LinkedIn systems to uh automate their workflows and be very productive. And this is just one example. We have over 600 uh such workflows and thousands of tools helping teams automate their workflows.

03:36 And uh this is possible because of a system that we built called as uh we are calling it contextual agent playbooks and tools. Uh and I'll be talking about today I'll be uh going over why we built it and how we built the system and what did we learn from the by building the system and from its success. To understand why we built this system, we have to go back in time of the early days of AI coding assistants.

04:09 Uh the early coding assistants were mainly uh LLM based smart autocomplete where you can it can figure out how to uh it will help you autocomplete a sentence or a line or a code block based on the context it has. And then um the agent mode was introduced in all of the coding tools uh across different coding uh tools. The agent mode was really powerful because the agent now with AI powered agents now have access to tools to edit the files and also run terminal commands.

04:44 Now they could figure out what how to make the changes and not only suggest those changes, they can also go ahead and make those changes and they can edit files across multiple files within your uh workspace. So this was really powerful and uh in early 2025 uh Andre Karpati famously termed this u uh term called as wipe coding which the idea is uh you don't have to look at the code.

05:17 You can just keep prompting the agent on what you want and the agent will figure out what changes to make and it will make the changes and it will also run the uh because it has access to the terminal it will also run some commands to verify the changes it makes. So you just have to I I mean just watch the output. So this idea was really powerful and it created a lot of hype around the industry right and just like any other company even at LinkedIn there was a lot of excitement to use this coding agents and we had

05:54 access to all of the we provided access to the coding uh tools early on to all the engineers and we u also wanted everyone to use it and be really productive. And lot of us tried the coding agents to do wipe coding or to become um automated coding in our LinkedIn codebase but it was not working. The wipe coding was not working. Uh can you guess why it was not working?

06:30 Okay. So uh the main issue was uh the LinkedIn's codebase is so mature and uh there is lot of internal context the agents didn't had the context of how to make the changes properly. Even though idea is really powerful but when you try it in your uh company the agents miss lot of the context and either produce subpar results or it will completely hallucinate and create wrong code.

06:58 The engineers had to constantly give the right prompts and give the right context to the agents to guide them in the right direction. And the this was opposite to what you hope for, right? Because you want to be productive, but now you're babysitting this agent to do the right thing and understand your company's uh code base. So, a lot of engineers just went back to regular manual coding because everyone has deadlines to it.

07:30 So uh yeah the uh it was not working as we expected to understand this problem more concretely. So if you take a look at the LinkedIn stack it's massive. We have thousands of repos thousands of microservices and apps which depend on each other and they're all built using a lot of internal frameworks at every layer. And also we have a lot of custom infra built.

08:01 For example, we have our custom databases built. We have a custom track tracking and obser uh experimentation system observability and configuration management which are really mature and these are built to handle the unique challenges of LinkedIn scale and engineers whoever joins LinkedIn they spend over one week just uh through a rigorous boot camp just to try to understand the internal systems get familiar with them and it also takes few weeks for them to be really productive because there is so much to uh learn and

08:40 do things right. The same problem is with the coding agents as well. You just cannot throw a coding agent at such a massive system and expect them to be productive. So we asked this question to ourself. How do we make these coding agents understand our LinkedIn systems, internal systems so well that they can ship the code our engineers can trust. And one of the problem with coding agents also was the engineers couldn't really trust the code that was getting produced because we have a really high quality bar for code

09:20 and also the reliability of systems. And since the agents didn't had the right context, we cannot really trust the co uh trust the code that it was generating. So we asked ourself, how do we make the code that is generated by these agents really trustworthy and around the same time anthropic released and open-sourced model context protocol or MCP, right?

09:50 and it become really popular and also got adopted as a open standard and it become a open standard for connecting the tools to the agents and all of the coding agents started supporting MCP. So this was a really good moment for us because now we can connect the coding agents and extend their capab uh capability by adding the tools so that they can access our internal system and understand our uh internal system.

10:21 So the first tools set of tools that we added was uh code search. So we had a very sophisticated code search um system already existing which engineers were using. So the it it uh this code search engine it basically ingest all of the code within LinkedIn across thousand repos and allows the engineers to search for code using query uh keywords or reax and advanced filters like file types and language etc.

10:57 And now we using MCP we wrapped these uh code search tool to query the code for relevant code snippets and also get the file content of the file and connected it to the coding agents. This was a really huge unlock because now the coding agents not just have to rely on their pre u pre-tained data because uh remember all of these LLMs are just trained on open-source code.

11:25 They're really good at generating the code but they don't know what the code in the LinkedIn looks like. So with by adding uh code search capability now the coding agents were able to use the tools to understand how the code looks like in LinkedIn by performing a query and getting the relevant examples and also if they need they can also go and like read the entire file and the LLMs um or the coding agents generally so they are really good at uh tool calling so the users Even if you give a natural language query like

12:04 hey show me how to do like a lang chain uh example. So even if the query is like really vague and in natural language the coding agents can repeatedly use the code search tool to figure out what right uh what queries to use and they can repeatedly call the code search with different kinds of things and improvise and find the relevant context and then summarize that.

12:32 So this was really powerful and it unlocked a lot of possibilities. So with that we also started adding more and more tools. For example, we added a fe um made it possible for the coding agents to search for the docs and wikis and also read the uh you you can give any uh link to the doc or the wiki. They can read that and get that into the context. Similarly uh they we added tools to read the feature flags task management and data platform.

13:09 So what this unlocked every tool that we added what it unlocked was now the agents can get better and richer context to do their work. For example, using docs, you can now give product requirement uh product requirement documents and also architecture uh documents and u uh some wiki pages um which were created by the team which has a lot of internal context.

13:38 Now the coding agents have access to it. So this is all good. Uh now the agents are able to get all the context but still there is a it was not quite working especially if the task becomes even slightly complex the agents were not able to do it end to end you could not you couldn't rely on agents to figure out everything and do it uh do a task a slightly complex task end to end so the main problems were the first is um the tribal knowledge to do any task task coding task.

14:14 It's not just about figuring out how to um find the relevant code examples and just doing the coding. It also involves figuring out how to install the dependencies and uh run the com what right commands to run to build the uh com compile the code or test the code right and these are all the nuanced knowledge it was only these were scattered in across lot of docs and wikis and slack threads or it's just a tribal knowledge which lives in some senior engineer's head right the even with all these tools the coding agents

14:56 were not able to find u the relevant context it's like finding the needle in the haststack. So this is one problem. The second problem is context overload. So whenever a agent uses a specific tool, the tool output also takes space in the cont LLM's context and all of the LLMs have a limited context window even they're getting bigger and bigger but it is still limited.

15:28 So the what happens when the coding agents they try to look for the relevant context to do a certain task more tools they use they get overloaded with the context and finally they uh they decides to at certain point they summarize all of that information and uh using compaction. So with once the context window is compacted it loses lot of the uh information right and then it has to uh do all of the tool calls again to find that relevant context.

16:03 So this is one of the reason sometimes coding agents get into a weird loop where they do a bunch of tool calls figure out how to do things but they all of a sudden they get summarized and uh they will forget what they were doing and they start to do that all over again. So that's the contrast overload problem. The another big problem is the agents doesn't have a long-term memory or the coding agents.

16:31 Um now we are seeing some form of memory. Uh but these were the early days and the there was a no long-term uh durable long-term memory. The issue with this is even if the agent figures out by all of the tool calls and all of the context you give u by hand the next time you want to do the same task the agent has to figure out all of these things all over again.

16:57 This is really time consuming because if you're an engineer and you are doing a task, you want to get it really fast. But if the agent is always trying starting from scratch and trying to figure out all of these things over again, it is not really uh good experience and also this is very expensive because it takes up lot of uh tokens. So what is the solution to all of this?

17:29 Of course the as I said the solution is to provide the memory. How do we make the agents remember a context on how to do things for u so we have to understand procedural memory. Procedural memory is just the context on how to do things. If the agent for example whenever you give a task it had figured out uh all of the context it needs to perform a job and it was able to get to the final state.

17:58 You store that context, store that all of the uh relevant context as a procedural memory and when the agent wants to do the same task next time you serve that procedural memory on all the steps and all the background information and nuance knowledge to the agents to do the same task again. So by giving this procedural memory if you are able to do this efficiently then you can make the agents remember things and do the entire task without taking too much of time and also without uh burning a lot of tokens.

18:35 That's the whole concept of procedural memory and this is procedural memory is also the concept behind agent skills uh which is um open sourced by anthropic at LinkedIn. we had a similar implementation. Um, so we didn't use agent skill because it was not even a thing uh back then. So we developed our own concept of procedural memory. We called it playbooks.

19:03 So what are playbooks? Remember we built a MCP and connected the tools. We use the same MCP to serve the procedural memory. So uh we call it playbooks. So mainly like whenever you want to do a certain task we have a playbook for that which consists of name, description and instructions and the name and description give the hint to the coding agent on what that specific task is and the playbooks it gets now the coding agents can invoke the playbooks as a regular tools and when it invokes all of the instructions are

19:43 returned as a tool output. Back to the coding agent. For example, there is a playbook for writing offline pipelines using airflow. If a engineer asks, hey uh can you set up an airflow pipeline for me? Then it the coding agent realizes that it has access to the airflow playbook via MCP and it invokes that to get the relevant context and it can also get the context from the user on what specific uh changes it needs to do and you it uses all of this context to perform the task and anyone at LinkedIn can create a playbook

20:26 and make it uh check it into a central repos. Now it is available for everyone else at LinkedIn. So this is a really powerful concept and it unlocked um it made it possible for to share the prompts and share the context. So what uh what does a good playbook look like and also when whenever we are creating a playbooks we have to keep these two things in mind.

20:58 Uh so these are the design philosophies we try to enforce. One is a playbook it should do the first philosophy is the playbook should be self-contained which means a playbook should do the exactly one specific thing. uh for example uh if it is for uh creating a airflow it should have all of the concepts and uh instructions and everything and if it makes it if there is a playbook which is doing multiple different things it is better to break it up into multiple playbooks but and the agent can always decide to choose

21:32 which playbook to use. And the second philosophy design philosophy is uh composibility which means if you have a long very long playbook and there are some tasks which are which can be extracted out then it is better to create a smaller playbooks which can be referenced by the bigger playbooks. So this is similar to how you do code uh reusability right where you break the code into multiple places.

21:58 And the first advantage uh there are two main advantages of this composibility. The first one is u reuse. As I said just like code the playbooks can also be reused where multiple playbooks can reference the smaller playbooks to do a certain action. And the second biggest advantage is uh the progressive disclosure of uh context. So what I mean by that is when the coding agents want to perform a task want to get a context if they load all of the context um up in one go it overloads the context right and it's not also

22:40 good for the performance of the agent. So with progressive disclosure of uh agents progressive disclosure of context when a big playbook references a small playbooks and so on the agents can choose to read the smaller playbooks or playbooks only if they need to. If they don't need to then they can skip those. So this is a better efficient management of context.

23:11 So with these two philosophies we can now create a organizational context within LinkedIn where you create a graph of you can almost capture all of the context within LinkedIn as a graph of playbooks which the agent can now navigate to figure out how to do things um at LinkedIn and do longer and longer running um uh tasks. And as we are creating more and more playbooks, there is a one common problem right with any knowledge base, it gets outdated because these playbooks are created at some point in time and at that

23:56 time you captured certain instructions but as the systems evolve the the instructions in those playbooks get outdated. But great thing about coding agents is they can improvise which means when they use a playbook and if they detect that there is a outdated information they can figure they don't just stop there. They can improvise and figure out what is the right solution using the tools that we have.

24:25 And once they find the uh right solution and also they can find that the information is outdated and they can try to find what is the latest information or they can always ask the user of the coding agent to provide the relevant information and this is also works for any edge cases. You the playbook doesn't have to be really perfect from the day one.

24:49 So you can start with a basic set of instructions and there are always edge cases and the agents they are really good at find uh finding some edge cases and we have in our system prompt which encourages the coding agents to always capture these learnings outdated information or any edge cases and summarize that at the end of the session and then we also encourage the coding agents to check out the code for the uh playbooks and also capture all of their information to update the playbook which can be now checked in.

25:29 And this creates a really good flywheel where more and more the playbooks are used they agents they can update it with the new learnings. So the memory the procedural memory it as in more it is used it becomes better and better. So what does a how does it all look like uh in a from an architecture point of view. So we have one local MCP server and all of the tools and playbooks are served via this uh MCP server and so we have two kinds of uh playbooks central playbooks and local playbooks.

26:11 So this is just to another context engineering um aspect where certain playbooks are only used or only needed in a specific uh repository, right? And so only when the user is working in that repository you would need that playbook. So because we are we are using a local MCP server we have access to the file system and whenever the we can also the MCP server also has the awareness of which directory it is working in and when the MCP server starts and it identifies that uh which workspace it is working in and the if it

26:55 has the playbooks directory then it can load all of the playbooks in that directory. So all of the teams playbooks and uh specific repository specific playbooks can go here and they also get checked in with the repository where they exist in so that the teams can create all of their playbooks without having to uh worry about overloading the system. And then we have the central playbooks which is these are the playbooks which are applicable across multiple repositories.

27:24 And um so these are uh we want to make sure that only uh playbooks which are applicable across the repositories or even without repositories they go in here and then we have uh tools and all of the tools they require authentication right so we have a seamless authentication um uh library where every tool they connect to either our internal system or our external internal system and they would need some kind of authentication.

28:00 So this happens using a seamless o where whenever the user is using the tool for the first time and it identifies that it requires authentication it launches the authentication flow generally oath and the user can authenticate and the o layer it captures the tokens and stores it in a secure keychain and the next and then the tool execution continues and the next time the tool is called it identifies it can fetch the same tokens uh refresh the tokens and perform the tool execution and this MCP server it is also

28:41 pre-installed on all LinkedIn laptops and it is also auto updated every hour whenever there's an update so this means that anyone can create a playbook or a tool and it will be avail once they once it gets deployed it is available for everyone else within one hour. And also having a central platform allows us to every tool that we create has to go through a infosc review which um which is because the MCP is fairly new and it can open up a can of worms for a lot of security issues and having uh everything in one place

29:21 helps our team to make sure every every tool is audited and also every tool is secure. So, how many of you are wondering how is how many tools can this pos uh this MCP server possibly fit before it degrades the performance? Anyone wondering about how many tools? Yeah. So, uh I think this is a general problem with MCP itself, right? uh where I'm talking about creating a lot of thousands of tools but uh if you are familiar with the MCP or in general tools beyond a every tool takes up space in the context and it also uh

30:12 as more and more tools are added to the uh given to the coding agent it takes up a lot of space in their context window and it does it degrades the performance of the uh agent itself. For example, if you add a lot of tools and even if you don't need them, it causes the agent to perform compactions more often. So, generally around 30 tools u beyond 30 tools, it makes the uh agents slow down and also uh degrades their performance as I said.

30:50 So, how do we scale this beyond 30 tools? Of course, we want to create a lot of playbooks and lot of tools and this is a central system. So what we do is instead of exposing all of these tools and playbooks as directly to the MCP, we replace them with three tools. The first one is search u search for the tools and playbooks. The agent can use this tool to whenever it needs to perform an action or find a playbook, it can use this tool to first do the search based on keywords and also uh tags like domain and what action

31:34 uh the tool uh does and once it u this search tool will surface relevant results of the tools and playbooks and then it um the agent can decide based on the name and description which tool to use and then it can get the schema of the tool. Uh uh this is to get the input arguments uh how to construct the uh arguments for that tool and then it can execute.

32:02 So this way we are able to scale the tool catalog tools and playbook cat logs to thousands of um tools and playbooks. So who maintains all these tools and playbooks and server right? Uh because you keep uh everyone keeps adding the tools and everyone keeps adding the playbooks they if it um if everyone just keeps adding tools there should be someone to maintain them.

32:32 So we follow this ownership model where there is a central team which is responsible for maintaining the uh MCP server itself and uh any and they also maintain a set of core tools. So this ensures that if there is any issue with the MCP server or um any of the core tools the central platform uh we have on call they will help uh and fix any issues and then we have a open contribution model where as I said anyone can go and contribute um of course with the infosc review but whoever adds the uh whichever team adds the

33:12 tools or playbooks and makes it available for everyone they also should be maintaining those tools and playbooks and any uh playbooks or tools which are not used or not maintained will actively and aggressively deprecate them. The whole point is we want the every tool and everything to be accounted for to maintain the health of the entire system. So uh we have over 8K u users who use the uh use the MCP server and any of the tools on a daily basis and this is not just engineers right so we gave access to coding tools or

33:55 coding agents to all uh other functions as well like product managers designers TPMS and they use all of these um all or some of the tools uh for their workflows and uh again the coding agents are not just for coding anymore. you can do a lot more things for example debugging uh debugging also a lot of people use the coding tools to um because it has access to uh docs and uh wiki they can go and create the uh docs or write the docs um using the uh using the tools from the MCP and um so we also looked at what are these

34:45 playbooks primarily used for, right? Um so we have over 600 playbooks which are automating the workflows and uh we realized that these are the top five broad themes which are which the teams are creating for the to uh playbooks and using them. The first one is interestingly debugging and investigation of issues. So uh not this is not rel it is related to coding but this is not exactly coding.

35:17 Uh so what we realized is uh all of the teams now they are converting their runbooks into playbooks and the agents are when they have when you give them good set of instruction on how to where to look for the data and uh uh go and uh investigate. They are really good at ingesting all of this data and figuring that needle in the haststack. And this is where uh this is one of the area where humans also hate doing this.

35:48 So this is really interesting um uh aspect of the coding agents. And the second one is uh of course which was intended purpose of this uh whole playbooks was the coding. So if you look at coding in um big companies like LinkedIn, it is a lot of boilerplate code because we have um lot of internal frameworks and setting up the databases etc. It takes up a lot of boilerplate code which is very manual and given the right set of instructions the agents can do a really good job of creating those boilerplate code for you and

36:28 also running the commands and verifying that it is set up properly and you can do a longer and longer running task but when you have a lot of boiler plate to uh set up. So this makes it really uh easy for engineers to cut out all of the boring stuff and manual stuff of uh setting up the boilerplate code and focus instead on really interesting things which is how to uh what products to build and how what changes to make and how to iterate faster.

37:06 Then the third uh topmost use case is also code cleanup and migrations. And if anyone is working in big company, they will know that this is one of the pain which is there is constant migrations and these are not fun but you have to pay the tax and this is where coding agents can really help where you have a lot of migration uh that you want your uh the dependent teams to do.

37:31 So you can create a playbook and you can ask the coding agents to go and do the uh migrations and create a PR and because the uh migration playbooks are really good, they can actually do a really good job of and we can actually rely on them to do a uh the coding agents to do the right job. And uh all of the playbooks also need to have a good verification steps where the agent can go and verify its own work.

38:04 So then um we also have lot of as I said there is a lot of custom infra and lot of um it requires a lot of management and also we have a lot of um pipelines for example offline pipelines and AI um training and a lot the playbooks um really help the coding agents. So these are the task which the coding agents can help with where instead of engineer going and managing these pipelines and fetching the logs and uh trying to debug them or trying to do some um manage some of the longunning tasks that you can delegate that to

38:45 coding agents and give them instructions via playbooks and they they can um do that job. And the fifth one is uh environment and repo setup. So as I said u any new engineer joins now they can use this playbooks to create set up their uh local development environment and uh get productive sooner instead of figuring out all of the things in the readme and going through this process.

39:18 So what next right? So we have a good system which is working. So what are we working on next? So so far uh as I mentioned the engineers were responsible for creating the playbooks. Of course we have playbooks to write playbooks which means the engineers are not manually writing the playbooks. They are using the coding agents to write the figure out the right context and capturing that as a playbook.

39:44 But we want to automate that where there is a we can have a background agent which background coding agent which looks at all of the uh information from the PRs and also the agent sessions and the telemetry to understand what are the common workflows that can be automated and they can write the playbooks themselves. And this allows us to scale the entire organizational context of um at LinkedIn much faster.

40:17 And another thing we are working on is also we have a lot of uh playbooks. Now of course we have a way to uh update them and keep them up to date but it is still a manual process where somebody has to remember or direct the agent to go and update the playbooks with the new learnings. We want to automate that where the there is a background coding agent again which um like an offline job which looks at all of the agent sessions and the um any new learnings from the telemetry uh etc and keeps the and takes a look at the

40:55 playbooks themsel and see if there is any gaps and it can go and make updates to that playbook. So I'll leave you all with uh two key takeaways. The first one is um it's not enough to give AI coding tools to the engineers in your team and in your company and expect them to be productive. The first one is um you need to have uh you need to think from the day one how to put enough guard rails around the quality how to ensure that the coding agents are creating a quality uh output.

41:38 So this is really important in your uh uh company. And the second one the downstream effect of it is you need to make sure whatever the code your code um coding agents are generating is having the same uh it's not degrading the reliability of your systems. This is a very common issue where as the code velocity goes up uh it will cause some downstream uh effects.

42:03 So you need to think from the day one on how to maintain the quality and the reliability. The second one is uh the coding agents and the harnesses models everything keeps getting updated but one thing uh doesn't change is the coding agents still need access to your internal context the your tribal knowledge that doesn't change right so you need to have some kind of infra where you you make it easy for the coding agents to access your internal um tribal knowledge via it is via skills or playbooks uh like we did you need

42:44 to have some kind of um uh infra for the agents to navigate your company only then they will be successful uh and the third one is um I didn't mention it in the slide but uh the developer experience um to get the adoption of the coding agents you need to make the experience of using the coding agents really seamless and then the productivity and it it is really hard because um the problem is these coding tools are really new and people may not um right away like the code that code uh the coding agents generate or they

43:23 don't uh want to use that. So you need to put constant effort as a company to make that experience better for them to co generate the code. So that was my presentation. Thanks for all um for giving me the opportunity to present it here today. And uh you can connect me connect with me on LinkedIn and also feel free to catch me later uh today or tomorrow to um discuss anything.

43:55 >> All right, let's thank you. >> Great presentation. Thank you. Uh quick question. So you got the data point for a lot of user using the playbook over the past few months right now 8,000 people. Can you provide some insight about the productive and quality over the time? Yeah. Yeah. Uh this is a good question. So um yeah uh so a lot of people are using they're not just using the playbooks but a lot of people are using the tools it tools by themselves.

44:26 uh the there are 8K users who are uh using the uh tools. So over the quality uh as I mentioned like we put the guard rails on the quality from the day one. So it has uh not I mean decreased so which is a really good thing and also what we realized is we are actually um we also maintain the productivity uh so the productivity has gone up about 20%. Increase in the productivity and we also measure the productivity against the reliability of our system and the reliability has not gone down at all.

45:00 So >> yeah, I had a question. Uh when you're doing the searching for the tools, uh so you're doing the searching and then returning the schema and then actually executing, what uh pieces of the tool call were relevant when you were doing the search? Was it just like the name and description or did you include like pieces of the schema or pieces of um something else in there?

45:27 >> Yeah. Uh that's a great question. So how does uh I guess uh how does the agent figure out the right tool to use based on what we present in the uh results? Um so um the first one is um the name and the description of the tool itself. It is enough for the agent to decide uh which tool to use. And also we have um something called as short description where we have a long form description and a short description.

45:53 Uh so what we realized is uh just the short description itself is enough for the coding agent to um uh use the figure out which tool to use. >> Uh you mentioned that um agents would be able to identify outdated playbooks. >> How exactly the agent would be able to identify anything is got outdated and automatically notify that the playbook needs to be updated based on any infrastructure or any other architectural changes.

46:21 How how would you do that? Yes. >> Yeah. Uh that's a great question. So how does uh the agents detect the outdated information in the playbooks and how do they improvise right? Um yeah so the way it works is um so even without the playbook also the agents can try to figure out what to do. Uh right. So with the playbooks they when it is and when when it finds any um outdated uh information it actually tries the u so what happens is suppose if the command that you gave is not working.

46:54 So it actually first tries the command and it fails and it takes the output and reasons about it that okay so this is the command given in the playbook but it is not working. I need to try something else. So they will try uh of course they will based on their pre-training they can try uh different things and also if they cannot find it then they always have access to the tools for example okay let me go and search in the Google uh docs on uh like the right information or they can use the code search to go and look for

47:27 the right information uh and they can improvise and uh the thing about this uh coding agents or the models like claude they don't give up that easily they keep going and trying uh new things until they exhaust all of the options. Um they generally don't they just keep going right but we also have this instructions in the agent that uh if it finds a dead end and it has uh used up lot of tokens they can always go and ask the user to uh direct them provide some missing information then the user can find the information

48:02 and point it in the right direction. So um these are the mechanisms to uh make them updated. >> So I wanted to know what is the process to control how many playbooks you can have in your company but also how to remove or prevent duplication of playbooks that are created. So what is the system then to control that? >> Yeah. Yeah. Uh that's a great question.

48:29 So um yeah the main uh mechanism is of course every playbook that we create has to go through the code review uh right so it is uh generally um like the uh a human will review what uh human will review uh before it gets merged. Uh so if there is any duplicate uh it gets flagged and also we have automated uh code reviews um for the system where we have a instructions um for example whenever someone creates a PR we also ask the coding agent which is doing the reviews to figure out if they if it is a duplicate or um if it

49:04 is um if it is not needed then they can surface that and also another thing is like the coding as I mentioned um the people are not uh uh doing the uh playbooks manually. They are using coding agents to create the uh playbooks. So we also have instructions on how what is the best practice of writing a playbook and uh also telling the user that um or the engineer that this is might not be needed or um irrelevant right >> and uh yeah sorry >> just uh I want to leave one note uh I think um another important aspect of this

49:45 MCP server is also we are able to uh use metrics to ma measure the usage of all the tools and playbooks and uh We go and look at the metrics for each of the playbook and we can deprecate uh the things that are not used.