Context Is the New Code: DevOps Practices for AI Agents Transcript, AI Summary & Key Points

00:10 context is code. Quick show of hands who believe that is true. All right, there's two ways that I've seen this kind like manifest. One is the kind of like very vibe coding Enrique party who says like we're just giving it prompts right we're giving it context it's writing the code for us definitely context is code there the second piece is something we experienced in our own company we were writing a piece about on boarding people to our tool you cannot imagine the discussions we had and then the number of pieces of

00:56 code we had to write for all the edge cases of doing the emborting and then the end we said we said let's write a skill in natural language give it to the agent it will cater to whatever kind of the person in front of it kind of needs to have right all the scenarios so so many code compressed to a couple of words that's also code it's not just like building it it's like piece of the product as I personally love to think in parallels and in 2009 I was thinking like what if ops would be more like dev.

01:32 We had devops. All right. So this talk is about what if context is like code. That's the mindset. So the software development life cycle is the context development life cycle. So I'm not talking about context engineering within your coding agent, but think of it as context. How do we deploy? How do we manage that across our whole kind of setup? We remember the infinity loop, nothing new there, but I've put a few kind of different words on that.

02:12 We're generating context, right? We're evaluating whether that context is good. We're distributing kind of that to team members, your organization, to agents. And then we'll observe whether that actually is working. Yes and no. And based on the feedback we get, we improve our generate and compact simple loop same as we try to do with coding. So I'm going to walk you through it in the parallels between those four things and in the end we'll have a bonus the context flywhe part generate I guess you've already been

02:53 generating you're the human context generator you're typing into your cloud code you're prompting you're giving it context no it's like this no please listen to me here's my cloud MD this is what I want the humans as a context engine. It doesn't scale well, right? So context engineering kind of became a thing. What we put in kind of the context window of the LLM and we're being kind of deciding what to do.

03:23 Now I'm not going to talk about context engineering because that's typically about what the agent will pull in the complex SNO. That's not the talk. I'm thinking about all the context that we need to prepare that goes in that thing. So it's a different cycle. It's not context engineering, but it's more the context as an artifact. People have been putting rules, instructions.

03:51 Luckily, we now kind of have a little bit of a standard. It's called like agent MD that we can reuse across. Funny thing is claude exists on being cla still. All the rest of the industry has caught on. There's a whole PR on this, but there is a little bit of standardization happening, which is good, right? Because now it is something that we can reuse across different pieces.

04:16 There's other pieces of context which is about your code and your docs and libraries that you're using. They have docs, they have code. So these tools like context 7 uh and ref they kind of look at your code dependencies and say well I'm using the express library. Let me pull in the context that is really good at kind of coding with that library. That's another piece of context that people have been catering for and then you can say I'm on version one two three of that kind of library and it pulls in the right context.

04:53 Good. and they're producing this for you to consume. It's not something you write. You could write it from your own libraries. Maybe you do that for your microservices, right? Microservices team one, they publish the context how to use the microser and then another one can use that part. Then you have connectors was we here like one of the vendors was unblocked.

05:20 They have an MTP agent. They look on Slack. They use all your sources that you typically as an engineer and as a person would look for context, right? Where was that decided? Where was that like chat at the bot? Like where do we find that information? We can now kind of do a lot with MTP and find that information. And then then there's a new style of context that is not like in your cloud MD, but it's your specs.

05:46 Hey, please implement this. And it's not a prompt, but it's more like a like a product requirements document in a very loose form. I want you to perform this. There's different ways that like different tools think about this. Kuro thinks of this as a task specification. Uh specits about this as more of the product features that I keep in my project all the time.

06:11 So there's different ways of people that do it. And then Baroo also put in intent integrity where you can verify the specs while you're running these as well. So it's a different kind of context that you're now providing to the agent to do the job. A little bit more sophisticated as your prompt. That's kind of the rough generate. So there's a lot of things that were starting to generate with this.

06:35 Uh whether that's your documentation, your code, your guidelines, your rules, and that's all good. So you become the curator of context not like the agent will decide what goes in but you'll make sure it has access to up to date to the right information and to go from there. Second piece is we have code and in our SDLC we build tests. It's very very common.

07:06 Nobody likes to write tests. We have things like tester in development, write the test first. All good. But we're doing this to measure the quality of those pieces. So what's the equivalent for context? What are tests for context? Okay, most people will have heard about something called sweepatch. It's a bunch of problems that says here's an input of a task.

07:44 Here's a model. Take this PR and solve this. How good does it solve it? Right? So, this is a way to kind of like reflect and make a case, a test case for testing things. Now, most of the things you see around that are about benchmarks. I'm training a model. I want to see how efficient the model is. I would suspect that maybe 001% of you is actually building a model if not unless you're building context, right?

08:17 So this is useful as a way to think about like test cases and how they do things. And then there became tools that allow you to write your own benchmarks. This is starting a Docker container. It's running the coding agent. It's input output. You run the tests. Did you succeed? Yes or no? So that's how you would test context by saying this is without the context.

08:44 This is with the context. This is with the old context. This is with the new context. So kind of that is the baseline of testing that people are going to. This has not yet been into the mainstream because it was thought of like too complex. This is for the LLM providers. But you can actually do this yourself. Simple way of testing. Here's a skill. If I look at the a skill has typically a metadata portion or a front-end matter in the YAML, I can check this similar to a llinter.

09:24 Does it have the lines that it needs to have? Does it have the correct syntax? That's kind of like very much like a linting of context. So maybe if we have some structured context, we can run a llinter on that and kind of do some verifications similar to linting of code, right? And then we can see, oh maybe the description field is too long. This is not according to the spec.

09:52 Second way, think of this as we write a skill or a piece of context and the LLM is giving us feedback and I often compare this to Grammarly like what's your writing style like is the writing style of your context effective for this model. So we can have AI alm as a judge watch your context and give us feedback on how well this is written, whether this is well worded, whether this is too short, whether this is uh kind of mixed uh and so on.

10:26 So grammarly for context, let's take this up a notch. So far, we've just been looking at a llinter, the syntax, what it can do. Now, a skill often has a purpose, a task it wants to complete. Before, I just looked at like the wording and kind of whether it was kind of the spec compliant as well. But because I know the goal of the skill it wants to achieve, I can now put this to the test.

11:05 I have this thing that I want to get done. I run the skill. It gives me of all the test cases this kind of result. I change it and then I get better results. It is different from testing because some tests that will pass when you do this start failing when you change it. So it's a little bit of a different but think of this as I'm testing the functionality of the context that you've provided.

11:37 One more notch. Sorry, it's not one more notch. It's a different notch. Um, so much as I told you that vendors are providing docs for their libraries, here's an example of fairell testing their own kind of versions of models how well they actually work on their library. So they're testing what information what best best practices they need to give the LLM and maybe the model of today will not succeed but the model of tomorrow does.

12:15 So they have to kind like provide you as a test harness to trust their models to use the right thing in their library. So they're optimizing also for agent op use and not just for use uh human use. Right? So this is uh now getting more and more traction. As a library owner, I want to provide kind of that testing and not just the context in there. Make sense so far?

12:45 Cool. All right. the extra notch. It is you take your piece of context, you have a skill, it needs to complete a task, but the rubber meets the road if it's within your repo. Otherwise, it's like a synthetic check, right? It does it on its own. We can check it, you know, no matter what. But the real case, it needs to work in your codebase, right? So much like BDD, we can come up with scenarios and say if you start from this commit, you give it that prompt, we want to achieve those tasks and then we run that set of

13:30 tests whether that task is completed. So that is the end to end test in my opinion for context and then you can start like building a library of testing and kind of go from there. And if you know what's wrong, you can start optimizing this, right? This is like please enhance, you know, and it gives you hints on how to do this. Similar to your in your VS code and he gives you, hey, hey, like you need to do something about this.

14:03 It's not working. It's not well. It's doing things. You could say things like, "Please remove all the stuff that the LLM already knows. Why are you putting this in?" Like it doesn't make an impact. Like it just takes away of your context window. That's one piece of feedback. Conflicting kind of things there as well. But the the the big thing is when you have the test you can optimize and it's the one thing that we talk a lot about like you know vibe coding but the vibe is also like how do I test this and this was a

14:39 puzzle the piece of the puzzle that actually makes that loop like hey I write things that's working but I install a new agent it's not working so I rerun my test I now see what's failing and I get that feedback loop. So that's what we're trying to achieve much as we were with writing our own tests. Now if we have this whole piece we can run kind of the tests the C whatever we do the optimize and then we run that in CI/CD.

15:13 Now, unfortunately, I already said it. It's a game of whack and mole. If you change context here, some tests will succeed, some will not. Compare it to when you have an insanely big code base and you've been in a big enterprise. You know that some pieces will fail. Even if you're going to deploy, it's a choice. Like, is it risky? Is it not risky? Now there's another piece of that which is when I run this it's nondeterministic.

15:46 So I'm going to run the test let's say five times and I'm going to say it works most of the time. So we're good. And you can run this locally as well. Kind of like similar to when I write like my tests I pick uh like two tests I skip the rest. I have a fast feedback loop and then I kind of open the test to more. So kind of that is the same loop. Now one great way of dealing with that if you see while you're typing CL code it's not doing the right thing take that as a test case.

16:24 So that is something you can bring in you write the test case without the context with the improved context and then you got it working. So that is kind of like giving you a certain uh level of certainty. Now in the beginning I said it's nondeterministic and you run those tests and what you'll kind of start seeing is that somebody has to take a decision whether we're going to go live with this.

16:48 It is different from going live with code because you say test suite works boom put it out. But like 98% of my evals work. What do I do? Who kind of decides? And often that means actually the product owner has to start deciding whether you go live yes or no. And the way you think about it, it's a risk game. There's evos you care about that must pause and other pieces can kind like be a little bit more erratic.

17:21 And so the metaphor is think about error budgets, right? I give a certain set of functionality error budget that's very like limited and then I have another one that like I can relax. So that's how you deal with the non-deterministic part and the whack and mole to kind of still get something in production. There's a lot of vendors who will try to sell you out of the box metrics.

17:52 I don't care about whether it works in your kind of you know uh environment that's like very clean room. I need to work it in my code in my environment. So the evals that are generated you have to make sure they make sense. Now you can have an AI generate the evals and the more context you give it the likelier it will also generate the best test for that.

18:15 So, but don't just go like I pick a vendor, it has like evals, we're done. It doesn't work like that. So, you got a whole stack of things you go through. I'll come back to the security scan later in another phase. But that is also pieces that you can program as well. All right. So, we got generate. We have the equivalent of testing with evals. Now, how do I distribute that stuff?

18:50 Most people, if they do share it, could be that they copy and paste it on Slack. I'm not talking about that. That's not sharing. We share it in GitHub. We check the file in and so somebody else who checks the project out, they can get the context that they need. People have been using git subm modules but in the end nobody likes it because you always forget that you need to pull the subm modules.

19:13 So it's kind of annoying and then they start putting everything in one repo but then one project and then the other project uh okay how do we kind of get that context out. So we have a solution for that. It is exactly libraries and package managers. You know, if I have my codebase and I had my library that works only for this project, that's fine. But if I want to reuse that, I create a library.

19:41 I package this into an MPF package or whatever you poison. I push it to a registry and somebody else can pull the same thing. Now, in this case, we're just having registries of context that we pull. So a simple kind of install this context in this project version blah right that's a package manager that I want to do. It also helps if you have kind of libraries that provide that context that can just say well from that vendor from that library install kind of the version that matches.

20:21 I have my team's rules that I'm going to install. I'm going to install the front-end package because that is good. And so you have like pieces that you can put together and it's an artifact. So we version that because you can have the updates and pulling from there. So what you'll see is that there's a bunch of marketplaces who have a bunch of skills, right?

20:42 So problem is were they even written for reuse? A lot of people put their personal skills up. Hey, I look I build a thing but they don't have any testing, right? So you don't know and then you have to look at the whole context whether that's useful yes or no. So there's a maintenance aspect of this that also kind of is uh gives you a credibility if you start adding tests to this much like you would look at a GitHub repo and say h you know they're well they have well documentation that's good they have good tests I'm

21:19 going to reuse that. If you just share a skill MD or kind of a markdown file h I know what happens and you don't even know what happens in your codebase. So that's kind of like challenging. So registries alone are not the solution. Now luckily and I've said it in one of the previous talks I feel that skills is a little bit uh on its way to become the standard jar npm packets or whatever of context.

21:47 So much like we standardized agentmd we now have a way okay all kind of the major vendors of AI coding will support skills so it becomes interchangeable it's not like you have instructions like you have rules just in scroll skills and we're reusing that same format as well now when I distribute that stuff much like security scanning I want to make sure that there's There's no malicious skills in there because skills also can contain code which is like you know everything's now melting together.

22:27 This got a lot of attention with open claw and people doing weird stuff and kind of things. It's weird. It needs like that one thing for the whole industry to wake up. But if you think in the parallel, you know this was coming, right? It's so simple to kind of predict. It's one of the things I have with my talks is like I give the overview and then I just wait until a vendor implements some of these pieces uh through the whole year but in this case sneak you know known for its security uh scanning also had like a

22:59 context scanner in there and then of obviously we have bill of materials. How was the context created? Who created the context? Like why when was it created? Uh so similar to a bill of material that we kind of implemented for security on code we can do the same thing an AI bill of material and here's a tool G blame who actually shows like this line of code was written by this person uh but they know whether it was AI or not because they kind of captured that in GitHub annotations and they didn't just look at who

23:33 checked it in right so part of that provenence of where the context came from also give you a certain level of credibility much like it would do your libraries for code. And then if you take this a step further, instead of kind of like doing this only in annotations, why don't we have almost like a transaction log? Everything that's been done on this code and we record that.

24:03 It has an advantage not just for tracking provenence, but it's also context. remember five steps ago we did this and that person did that and you got that sources the agents love this because then they can read this almost like a backstory of why this happened. All right. So, hope this gives you some idea that it's not that similar dissimilar as code, but we have just a bunch of new tooling that will be spawned around managing context there as well.

24:41 Finally, in the cycle, the observe. All right. What if Claude could self-improve its own thing? Right. So it is kind of like creating a loop that it's looking at the failures of the things it saw in its logs and say let me rewrite kind of that better. It's another step of optimize that we did with the scale but it's it's a in its form a little bit of an observability thing like it looks at what it did and then kind like improves that piece as well.

25:18 So hugging face does similar things on their codebase and they call it a retrospective. They got the agent to talk about we did this thing, was it the good thing, was it the bad thing and reflecting actually almost like a postmorton on what the agent did. So they're collecting that and then they're feeding this back into the agent as well. Now often that's still in a very closed loop on an individual system.

25:43 But you can think about if that optimization gets repackaged pushed into a registry the next kind of iteration of all the other agents will get the new context and the optimized context. So kind of that is the feedback loop of actually running this thing in production not kind of eval the synthetic checking but kind of on actual use cases as well. Luckily, the industry is starting to settle on another standard.

26:14 How do we not only get our claw MDs to standardize, but how do we standardize our logs? Because they're all starting to tap into that observability layer. So, ideally, one standard and then all the tools could work similar to open telemetry kind of on this but on the logs as well. No surprise, this is the company Devon that also does a lot with the agents and autonomous agents is kind of building that self-reflection loop as well.

26:46 And what's another way of looking at feedback? Every commit is a story. You've chatted with this, but imagine I put my agent and said, "Please auto push to get the review there is capturing context. The review is a kind of observability and feedback layer to whether it actually worked. It doesn't have to be a commit. It could be on Slack that you capture this like hey we deployed whole thread feedback kind of that is another loop of observability that we might not think about.

27:21 We think about like oh logs and servers and agents but this is also feedback of whether your context worked in the wild yes or no. a whole business like you know a company just dedicated to that and then there was one of the sponsors here and I like the idea they instrument your code and when it's deployed in production they actually capture when things happen these were the lines of codes that were impacted now traditional operations would then say great you had a failure here's a problem let me raise a ticket with a

28:01 What they can do now is they can say, "Agent, pull out all that information. Use that information as context so you're not repeating the same mistake again. Save this as a lesson learned. Put that into your context and kind of push that up, right? And it could be as simple as is shown here. Well, the system is being used kind of like like that's the standard deviation of usage, but it could also be like this is the failure rate on this one or this is the risk rate or these are like 10 million people using that function.

28:33 So, probably take care if you're writing this code that there's might be an issue and that you're pushing this out. All contextual but not siloed up. Now, gathering information and running these things requires you to do some sandbox typically and it makes sense and this is a great article like okay we want to control what the agent is doing but imagine it's trying something it wasn't intended okay if I get it into my logs that's great but what if it's reaching out to an endpoint that we don't know what if it's doing

29:09 something crazy u I try to sandbox claw myself in a And then it says like, "Okay, no worries. I can look at your environment variables even if you scrub me all kind of the secrets." Okay, remove them. I can still look at your memory. Uh, okay. Let's move with that. Uh, no worries. I can just send it off to over DNS IP exfiltration of the data. That's the kind of thing that you want to know, right?

29:39 I know I'm exaggerating a little bit, but this is also part of observability like what is going wrong and what it's not supposed to do and not just in your context. You know, when we deal with context, we used to have the web application firewall and uh I know it's always a pain writing those rules and you get a lot of noise and people question like is this useful or not?

30:06 But then kind of like it still gives you a signal where something's going on. Makes sense. Now in this proof of concept, I've created a different kind of firewall. So imagine you're in a sandbox and you say we're safe. Good. Okay. Um it's downloading stuff, but the problem is whenever I start immediately my coding agent cla MD is loaded. I have zero control on what it's loading there.

30:37 So looking at the network whatever it is loading. So what I did is I said why don't I create a shared library that will intercept file read calls and actually will check what it's reading from disk. Right? I learned you know two days ago you can do that with eBPF but not on a Mac. But anyway the prototype is are you safe? And it's not just about network access.

31:03 is also like what is the context? Most people think about prompt injection, but it could be different ways that it's like overruling things. So, think of this as your context firewall inside of your thing. Now, it has the same problems. You have to write the rules and that is tedious, but it's another way of getting feedback. And even if I'm just logging the things and I'm not logging the things, I can get that information.

31:31 And what you'll see is that even for training the models, the OpenAI is using the app logs or whatever it's doing to actually correlate that to problems in codeex. So they're using it not just in production, but they're using it also to train the model. This is a feedback loop that kind of makes sense, but we can do the same thing with our context and not training a model as well.

32:01 And then we get into dark factory land. Who has heard about the term dark dark factories? Few people. There's this like futuristic idea and I hate to break it to you. There's people that just say, "Give me a spec. I'll pull it through a factory of coding agents and software comes out." Okay, I'm not saying this can't work, but what's interesting is that they're also relying on digital twins to kind of simulate Google, octa, jira, and so on.

32:35 So they're basically building that observability also for the agent to see whether it actually working, whether they're doing the right thing and so on. Now I personally think that autonomy is not the end goal and the way that I explain it is there's different style of managing coding agents. There's the micromanager and likely you're doing a lot of micromanagement.

33:00 No. Yeah. Okay. It doesn't scale right. So next step is I'm going to write you a spec you generate and that's kind like how and then I do the review. Okay. And then the team lead is just saying, okay, okay, I have a bunch of agents they do and one is criticizing the others and I'm just going to take the call if somebody kind of doesn't agree on that.

33:20 I'm going to run this nightly and maybe, you know, the agents keep going and they just call me when I need it. Like I'm not like even in the loop anymore, just when there's a conflict they can't solve. So all different styles. Is one better than the other? If you're a manager and that's what you're actually becoming of the agents, the point is that you know which management style to apply when.

33:51 So if you truly deeply care about something that cannot go wrong, you get into the room and you're micromanaging the situation. If the risk is low and you mitigate it like you probably mitigate it maybe like deploys or similar things you can say go at it we it's a known situation we got guard rails everything around it and if we hit it okay you can call me right so that's why I'm saying autonomy is not the goal it's just knowing when to apply autonomy in that full spectrum and that's my take on this there's obviously

34:29 people who believe that if the ultimate autonomy We can run billion companies with two people and it's done. But as Hannah showed us, there's still on call. There's still solutions. We still have to make decisions. Okay, good luck. If this is your money and you're just blinding blindly kind of trusting the agents. So I I hope that gave you an idea on where the similarities are managing context and kind like managing code and how kind of the tool space is adapting and we're getting there.

35:03 Final piece. This slide is what my initial talk was but I decided to rewrite my talk yesterday. Sorry. So you can watch this talk online but I was really bored but it's still an interesting talk. Um, it talks about you as a developer changing and shifting role. So, as a developer, you're managing agents. So, you're basically operations. You do not know anything anymore about like the code.

35:33 They throw it over the wall and you have to make a decision. Great. If you write your specifications, you actually become the QA architect that writes context and kind of gives them all the specifications how the agents should be working. Great. And then you become the product owner because you decide what the agents need to do, what they need to build.

35:54 So you get closer to that field and working out and maybe you build that proof of concept, you build five proof of concepts. Why not? Like why limit it to one? and you make the call what is actually good and in the end you become the data manager who collects all that data and turns that into knowledge. So senior engineers know that their role was never only about the development.

36:24 They were the ones who went out into other parts. And now AI is pushing us to do the same thing. And this slide, but I think that still holds even though this diagram was made two years ago. All right. So I'm an engineer. What's in it for me? There's a funny thing happening. Engineers did not like to write tests, but all of a sudden they like tests because the agent is writing the code and without the test they can't tell whether it's good.

37:04 engineers didn't like to write code documentation. But now they write context because it's very selfish of them to write context because the agent does a better job and they don't have to do that much of review. Right? So for the first time there is an alignment of people doing jobs and things that they weren't liking before but they I see it almost like they start doing good engineering practices and no matter how hard me as a VP engineering somewhere like please don't do this please do it that way if I could just

37:41 distribute the context and say my agents are going to do the job for me and say nope no not that way oh please do it that Okay, that makes me scale as well, right? So for some time we try to get the alignment of dev and ops kind of you care about the production, you care about this things. For some reason, this is working as an auto alignment across people doing that stuff automatically.

38:08 Now, we've been talking a lot about doing this as a solo person, which is great. Like a lot of people do it that way, but I really said like what if I share kind of my best practices and my context to my teammates or to my teammates agents then they don't hit the same problem again. This is also a perfect place to get all the naysayers on board and it's like no no AI can't do this.

38:35 Can you write the context so they can do that? Like great. So, but the the thing is the more you share this and not just on slack but the more you share this there's a benefit effect of kind of distributing that and why one team why not a set of teams you maybe as a platform team say let's write the guidelines push that also into the context and you're scaling this out as well so everybody can contribute pieces of the context and eventually hopefully your organization collect benefits from that flywheel as as well.

39:12 So people have been thinking about maturity levels. Do I manually create content context? Do I kind of persist it? That was kind of the distribution. Do I make it adaptive? That's the optimize and the learning. Do I look at observability? That's kind of like my proactive context uh looping pack. But then who owns context? It's good. like maybe look you own the context or the code within your repo but if you start sharing and that's one of the puzzles that people have like but who owns the cloud MD who owns my skills

39:49 who owns that it's an artifact so we have to start thinking like that team is best positioned for that piece of context they own it much like a library or a micros service kind of that is ownership here's an example of a product that says everybody can submit rules But there's only kind of a few in the team that can say yes no before the distribution happens to the old company.

40:14 So there's an ownership and kind of handling PRs of context. Now going to go quickly through this line but if you think about maturity our agents are still very much optimized for coding. how much will they get optimized for creating context, downloading the right context, kind of doing that stuff. So there's an agent maturity around context that you have to think about like the technical tooling.

40:41 How well does it actually deal with context? How mature are your team or your organization context practices? Are you putting in the tests? Are you pushing them just like yoloing out? Like that's kind of a you know maturity level on its own. And then as a manager, how do you enable other teams to share their context? What are you doing for them to make it available?

41:08 Are you putting that platform where they can push things out, where they can share things? Do you give them education on how to write good context? Have you write the right tools in place and the right practices? And you start focusing people like you know every time somebody says the I did a bad thing reflexes like did you update the context that kind of reflex is enablement kind of going around that and you know in dev ops we busted and tried to bust quite some silos but what's funny right now is even if the people

41:42 sometimes are hoarding kind of context as long as I can get access to the data my agent will figure it out. So there's a kind of silo busting happening like hey uh you know typically if I'm not in engineering I can create a PR and then I have to see whether that's working. What happens right now is as long as I have access to the repo, I can send the team pull requests, right?

42:09 So I'm I I'm not blocked by a PR and even like one of the things we're doing uh in our company is that I report things from the outside and we get into a conversation on Slack and we just say at Claude, can you fix it? Can you create a PR? So we're having just a joint conversation and then it just does the thing and then only one person has to say approve yes no before that was like create a ticket wait until the triage was happening and it just goes a lot faster.

42:41 So there is definitely some silo busting going on there. Now it's not all perfect and I don't want you to believe that context will change right the model will change whether it's still effective with your context um is it still relevant it's the is it the thing that we did a year ago now we have to remove this now obviously AI can help you and see and picking up the signals whether somebody said no and five developers said no and then you kind of see oh let's kind of have discussion about that.

43:16 So context drift is a thing much and as I told you distributing the context if I'm still using the old version of the context you're using the new version it is still an unsolved problem. So couple of unfinished thoughts. What is chaos engineering for context going to look? Are we just injecting context and see what happens and will survive and the coding keeps working and our whole system kind of detects uh bad things?

43:48 Context analytics maybe is that the number of downloads of context? Is that kind of the effective use of context that we want to flag? I don't know what that's going to look like. What about AB testing? Can I push out like new context and try it on two teams and see where that succeeds and then roll it out to the rest of the team? Have I find those products?

44:06 Who will be the next GitHub? Who will be the next Kubernetes for context? You know, and then the question comes like why are we still doing this? Can't the agents keep like all the work like can't they do it? Maybe they can, but we still have to have we're still responsible. We still have to assess the risk and they can all assist us with that. But in the end, some things are just a human call and then maybe they'll just invent their own language and we can't see what context they're sharing within each other and then

44:36 I don't know the end is near. But so I didn't dwell too much on kind of knowledge but I think this is where we're heading. We could we're sharing all the context and the ultimate goal is that your context actually become knowledge that is specific because it's not a bunch of markdowns. It's like what you put in there that actually counts. And so your mode as a company might not be the coding, might not be the first thing, but it's kind of providing the right context and maybe about your business the right context.

45:15 And this this is the differentiator. I've done things like, hey, I have my own kind of solution. Uh I see there's like five GitHub repos that do the same thing. Can you just extract like what features they have and we don't and can you incorporate them into my project? That's the kind of copying in the in the end if you kind like have that world. But I will not have the business understanding of what to build and what kind of to go for.

45:45 So I want you to think about that there's still a modes to be had in the industry. And I would say a lot of people focus on kind of the coding agents but I think context is a fuel and coding agents are your engine. So if you're not giving it the right fuel you kind of choke and kind of underperform in that way. And that's it. That's my talk. Thank you.

46:18 Thank you, Patrick. >> If you want to get the slides, send me a message on LinkedIn. Um, only in exchange for feedback, not just a question. Can I get the slides? That's unfair. >> Thank you very much. >> We've got maybe three minutes for questions if Oh, cool. That's good. Uh hi thank you for the talk and I have a question about language of the context.

46:44 What language is the best for the context? Maybe it's a code for some some type of context or English or another abstract language. So there um language models are typically trained on examples and most of the LLMs that we're using out there are a lot trained on English. Now with embeddings you can have similarities between English and kind of other languages but you sometimes would struggle.

47:18 We had one case where we somebody made a skill in Spanish, right? And our review said, uh, you're using Spanish. Uh, so is is that bad or not? But but there is definitely a challenge there. Like is it working for all the languages how you describe now in I think in reality from what I can tell uh some languages are better than others and that's just because of the training of the model.

47:48 Awesome. Anymore? >> Hi. Uh, thanks for the talk. Uh, my question is around confidence. So, let's assume that you apply all the things that you describe in this talk and the one of this morning. Uh, so it seems to me like quite rock solid strategy. It seems like that has some logic. But, uh, in let's say enterprise software that is let's say basically ruling the world.

48:12 I'm thinking like bank, insurance, uh, health, hospitality, etc. um how confident you are with uh let's say this approach that the output will uh let's say be um good enough for running in production uh considering that usually you don't have like a green field you have you are working in brown field maybe with uh a lot of intricated logic and stuff like that >> yeah so um there's a couple of pieces uh how confident I am so um let's say you're doing a legacy and a language age has the same problem as that language.

48:49 Has it been trained on that? Will it do good? Now some kind of AI coding ventures vendors they kind of train on more niche languages. So that's already one step you can take. Now the providing context in most of the cases gives a lift to do that. Right now I talked about the management styles and it's the risk. So would I kind of say straight to production in a bank without it?

49:14 No. Can I come up with cases where I would do it if I have the right tests and the guard rails in place? Yes. Uh it's a little bit like the story of you know within DevOps it says like can the junior come in and push to production on day one? How confident are you? and it was how far not only about your tests but also about your guardrails and so you can put a lot of guardrails in place.

49:43 Now I still feel a little bit hesitant about like the auto going to production and yes there's people doing that but they accept the risk for their business. If you're not willing to accept don't that's the simple ask. >> Yep. I'll stay around also for if you have more questions. Um, so but thank you very much for this. >> Thanks Patrick.