Agents Made Code Free. Maintaining It Didn't Get Cheaper. - Dmitry Buykin, Maersk Transcript, AI Summary & Key Points

00:00 Hi, I'm Dimmitri. In this session, I will show how I work on maintaining large AI coding projects. Coding agents made code cheap. They made entropy even cheaper. In the Marvel movie, Thanos watched his home planet Titan collapse under its own growth. So, when the universe got crowded, he raised half of everything. This talk is about the one time I did the same in one project.

00:26 Our code was growing the way Titan did. When I found what my agents had done, I deleted tests and boil the plate code nobody used. I had to rewrite the whole project and I lost some useful pieces. And that's the dilemma. Drown and chaos or break what people use. And here we'll build the third option for that project. The rewrite didn't hold. The agents kept writing and the mess grew back.

01:00 So I drew a line above it. There is more change than anyone can check. The third option is to stay under it with small sweeps every week and feedback loops built into the system. Each chapter of this talk is one thing I had to learn to get there. Thanos needed six stones for one snap. Ours are six questions and they light up in the corner as we go. The first one is time.

01:29 How do we stay under the line? Chapter one. To stay under the line. I first had to know what's right means here. I used to think that tests decide that. So what does right actually depend on? Let's keep one task in mind. Someone wants to change the amount on an order we already approved. Two kinds of agents meet here. One wrote the instructions and the other follows them in production.

02:04 We run this in 110 countries with about 10,000 scenarios of pros instructions. Every task lands in a slice. One process in one country. There's a second stone space. It asks where a change is allowed to go. By default, it stays inside its slice. Most of this system is pros about three quarters by my estimate. So pros needs engineering too. At start we borrowed mutation testing.

02:35 You break one sentence on purpose and you watch for red test. Here almost none turned red. And if your wrapper is all code, your agent instructions and your tests are your pros. With agents, a rewrite is easy. The hard part is knowing what has to survive it. For code, tests can tell you. For pros, only people can. Agents can't build a complex system on their own.

03:05 They fix what is easy to see and break consistency on the way. And people also get tired. So we spend their judgment only where it's needed. When business asked check with the team, it hides a real question. Is this a human step or can we automate it? Experts can't answer that. They don't see how the systems connect inside. So developers dig through the traces and logs with their coding agents and they decide which it is.

03:35 In our task, the team leads decision stays human. The limit was data we could connect. Business experts see the rules. Developers see what's under the water. And that's where most bugs leave. They have their coding agents write those links down so the runtime can check them. Our guide for experts says it simply business tell us what we work out how.

04:04 One document to readers. Agents read the blogs. So their blocks have to be exact. Experts read the text around them and they sign it. We keep one copy, a second one with drift and works both ways. Our code needs a plain description for experts too. Experts can only fix what they can see. Each of them reviews a task exactly the way the agent saw it. The node becomes the next change made by a coding agent and test it on the same task.

04:38 So right depends on people. No single person knows it. A group of experts decides and that's why we bring them in from the start. You'll see the soul stone here a lot. We take it in chapter 4. Chapter two. Experts correct one task at a time while agents run the same pros all day. We track every outcome accepted or rejected. The accuracy shows us where people still need to look.

05:10 I used to think the goal was to move everything into code. So where do we still need judgment? Here is our task again. The pro says the limit is 1,500. The code says a,000. The same order is fine in one place and too high in the other and nothing throws an error. Even a line like do not call X teaches the model that X exists. That's the third stone reality.

05:41 Is it true or does it only look true for the model? Pros has no compiler. So we built one. Then the author lints every changed file into ledger just like code. Lind can't stop copies drifting aparts. So we watched the walks. A walk is the path one agent takes through the pros. When agents keep walking in the same path, we turn it into one workflow tool and we delete the pros that walked it.

06:16 Our first workflow of tools made things worse. The pros still named the steps inside them. So agents run those steps again. It worked once. The pros kept only the decisions. Now a walk becomes code only when it repeats and agrees across countries otherwise it stays pros. What still varies stays in pros inside the frame. Code prepares the facts and carries out the decision.

06:47 Experts own the decision table and the model only picks a row. If no row fits, a person takes over. We haven't built the next step yet. The model will take the small decisions and workflows will connect them. This boundary moves both ways. We moved hundreds of lines of expressions out of code and into pros because experts couldn't read them. A reax is still fine for spotting a critical signal.

07:15 When there are many, they move into config. Judgment stays wherever the answer varies in a place an expert can read. Chapter three. Now the model decides and its decision becomes an action. I used to think control meant permissions on the agent. So what does it mean to stay in control? Control starts with what the model sees. That's what the fourth stone mind.

07:44 What does the model know? When we let it choose what to load, it loaded everything. Then it invented a pricing check just because prices were visible. Whatever sits in its context becomes a reason to act. The runtime decides what's the model sees for our task. That's three facts from the order. The full record stays with the runtime and the model makes fewer wrong moves.

08:17 Here how we minimize context blo for a big set. The model passes a handle, a short name for date that the runtime already holds. The runtime checks it before any write. It keeps the context small and still enough for the decision. At the same time, it uh validates context for consistency. A model with the right facts can still do damage. That's the fifth stone, power.

08:41 What made change? We control a change twice. during development and again at the right at runtime only code opens the right path so we can change pros many times a day the open rings are the parts that still depend on people let's zoom in and run right five walls stand in front of it and the sixth checks that it landed if a right looks like it failed an agent retries and the journal makes sure it runs only once means much more than permissions on the agent.

09:24 Chapter 4. Most days I glide over the system. From up here, everything looks fine, so I don't go down into every change my agents made. Then the surface gets rough. How do we know we're losing control? All those controls protect the system. Nothing protected my attention. At the start, I read every change and that's how the system picked up my habits.

09:48 Then the volume grew past what one person can read. I was burning out and I stopped reading every change. That was the price of the last stone. So, who judges and what does it cost? With it, the gauntlet is complete. Every tech lead I know has been here. The one person every change waits on. agents just get you there in weeks instead of years. I needed a way of working that doesn't run through one head.

10:22 And the first gauge was me. When I'm tired and I can't explain what changed, that's the signal. The dashboard can be green while that gauge is firing. So, I stop and fix the process first. I don't wait for a number. I couldn't read every div anymore. So I built instruments that watch how things change and their derivatives. One of them is blast radius.

10:49 Before we built one new check, we replayed its rule over several thousand past sessions. It would have fired on less than 1%. Where the numbers jump, I go down and read. When something looks wrong, we wait. One signal waits until it comes back, even when it comes from an expert. A cluster across slices is different. Then we fix the concept behind it once.

11:15 Left alone, an agent keeps adding conditions until today's tasks pass. The eval score drops when you take them out and you get an honest list of gaps. One red slice is rarely alone. The experts note becomes an eval and an agent searches the siblings for the same failure. Siblings differ on purpose by market and by local law. So we sweep them and the fix goes only where the failure is confirmed.

11:53 At any time 10 to 20 of these loops are open. At one point I had become the message bus between users and my agents. At this scale, every user has to be an evaluator with a screen to give feedback without me. My job as the architect is to build those loops into the system. Each loop writes to a ledger. So nothing waits on my memory and nothing waits on me.

12:24 Agents forget everything when a session ends. So the memory has to leave in files. We keep two kinds. what happened and what is true now. The next session starts from what is true now. Every issue has to connect to its root cause. Without that, the agent decides on wrong facts and the check has nothing to count. So we don't act without evidence. Reality comes back.

12:54 In chapter two, it asked if the pros is true. Now it asks if the change is I used to trust green. So what would make us change our mind? Green was tempting because checking is where the times goes. My agent spend almost 9/10en of that time on a change just checking it and checking took a bigger share of my own week too. Verification speed is the limit.

13:27 We rank evidence by what it proves. A replay on real tasks sits at the top. Before the data arrives, we write down what would prove us wrong. And every new check has to fail once on purpose before we trust it. AI refactors fail quietly. Every condition and every decision table row is a branch. And after a refactor, each one has to land exactly once.

13:56 Here the branch for amounts over the limit got lost. Our [snorts] order would have gone through without the team lead. Real tasks decide what we test. We freeze them as a set and replay the old version against the new one. If no real task takes a branch, we call that no evidence. You can't trust one agent to judge its own work. A second agent checks it with a checklist and without the author's reasoning.

14:30 It can't be talked into agreeing. Over 6 months, it grew to most of my checking. Coding agents finish most of a batch and quietly skipped the rest, especially when their context is full. The ledger shows what was skipped in one backlog. Most items looks covered, but only about a quarter could really be closed. So now evidence we can count is what changes our mind.

15:01 And this is the third option from the start of the talk. I used to think there were two choices. Keep everything or snap. So what is worth carrying forward? This one process over a year. Some branches merged back, some still drift, and some are dead but still look alive. Which ones deserve to survive? A fix is only a moment. These forces keep pushing right after it.

15:33 You've met most of them already. This chapter deals with regrowth and branching. The first gauge was me back in chapter 4. These are the gauges for the code. Time gets its answers here. The bridge line comes back as a number. A ratchet means the count can only go down. Two gauges run today and six more are next. Every week real use decides what stays.

16:07 Anything no real task reaches gets a name and its evidence before it goes. Test the evidence I didn't have when I snapped. Tests kept that variants alive. Agents kept every option a test still mentioned. Now retired variant takes its test with it. Here's the whole method as one week. This is the third option as a process. together. That's the full gauntlet.

16:42 Thanos used his for one snap. We use ours every week in small doses with evidence. I haven't had to snap again. Soul had a second price and I only saw it later. After we deliver it to production, I found my own engineering principles built into the systems design. Its skills and its memory carry my habits and my biases. Conway said the system copies the orc chart.

17:12 Ours copies whoever corrects it. Here are 18 questions, three for each stone. I'm using them when working on my projects. Pause here and pick one your own repository can't answer yet. That's where to start. I learned to use this process when I was already burning out. If you're the one your team's changes weight on today, this is for you. Use your stones while you can still read every change. In the end, Thanos retires to a garden. Maybe he should start using them daily. Thank you.