The Most Important AI Coding Advice You Haven't Heard Yet Transcript, AI Summary & Key Points

00:00 You'll have heard Day Ferraro and others on this channel before talking about the benefits of using ATDD, acceptance test driven development, when you're specifying and controlling development with Agentic AI. Now, today I'm going to look more closely at the inner loop of test driven development. Now, is it a good idea to force the agent to use a strict red, green, refactor for the lowest level of code design?

00:23 Does code quality improve? Is it worth it? That's our topic for today. >> [music] >> Hi, I'm Emily Bates. I'm a software developer, creator of Summon Coaching. Welcome to the Modern Software Engineering channel. If you enjoy the video today, please hit like. Consider subscribing. We are several presenters here with a common goal to bring you great advice on the technical aspects of modern software engineering.

00:53 So, this video is another episode in our AI Briefings series, and you can check out the previous ones on our channel homepage. So, previous episodes were about acceptance test driven development. This kind of approach where you specified the behaviors and the verifiable outcomes you want from the software before you do the coding. And based on conversations with product owners and business people, you outline scenarios, features, examples, which sets you up for success.

01:23 You build the right thing. Now, you also need to build the thing right, which is where the inner loop test driven development comes in. So, today I'm going to interview an expert TDD practitioner and early adopter of Agentic AI, Nizar Sallander. Now, Nizar is a software architect and consultant with Factor 10. Recently, I've been talking with quite a few technical coaches about how AI has changed their approach to test driven development, and it seems to me that there's a lot of variety and innovation going on.

01:52 And Nizar is one of those innovators. So, I began by asking Nizar to explain why is TDD still important when we're using a generative AI? >> It's still important for the same reasons as before. For me personally, it's about confidence. Good tests give me the confidence to make changes at a productive pace without losing my sanity. >> Nizar has created a couple of different tools to use with a generative AI, and I asked him, why did he create TDD Guard to use with his coding agent?

02:25 >> So, at first, I I started by basically giving it instructions just like everyone does, you know, follow TDD, do this and that, and it kind of works, but sooner enough, the agents started reaching for shortcuts. They would skip steps, over implement, things like that, and I had to go and remind them again over and over, and that's just not something I wanted to put my time and effort into.

02:44 So, sooner enough, I started looking into a way to basically automate this enforcement of the TDD cycle. >> So, TDD Guard was Nizar's first solution for forcing an agent to do proper TDD. He went on to create a follow-up tool called Probity, which we're going to see a demo of in a minute, but I asked Nizar to explain why he created this tool. >> So, TDD Guard worked great, but I kept finding myself policing agents on things that don't have anything to do with TDD.

03:14 Things like agents try to disable lint rules instead of fixing the issues, or they would try to commit changes before they run the checks that I want them to check to run. And also, TDD Guard works only with Cloud Code, and my team also used other different agents. So, that was the idea behind creating Probity. Have it run any rules across different agents.

03:34 And there were a lot of opportunities that showed up since I've created TDD Guard. The ecosystem matured. There are simpler ways to go about things that are much easier both to for me as a maintainer and for the end users. >> We're going to see a demo of Probot in just a moment, but I'd like to pause here first and thank our sponsors, Equal Experts, Transfic, Octopus Deploy.

03:58 We are grateful for their support. These are companies that are offering products and services well aligned with the kind of topics we discuss on this channel. So, please click on their links below, find out more. I would also like to thank all of our patron supporters both for this channel and me personally. We couldn't keep making these videos without your support.

04:21 Well, I've been trying to understand what test-driven development looks like when you're coding with an agent. So, I've asked Nizar and others to show me a solution to a well-known code kata, RPG combat. It's a classic coding challenge and the purpose of this is really to show what the TDD process looks like. I'm not actually trying to evaluate the code that comes out, the quality of the solution, because I'm pretty sure that any coding agent can solve this exercise and all the people that I'm talking to are producing

04:49 good quality code in general. What I want to understand is the coding process that they're using. How are they writing code with a Jantic AI? This is a a screen record of the kata. I've just picked out a moment in this kata where it finishes up the the faction story and it moves to the magical object story. So, here for the magical object story, it creates a new test case for the healing objects on the right here.

05:20 And then it creates a new file on the left for to make that pass. And you can kind of see it doing this. It creates another test here on the right and then a little bit more code appears on the left. So, it's it's definitely writing the tests first. But then a little bit later on in this sequence, something happens here that I wanted to ask you about, Nizar.

05:39 So, this here it's it's just added a new test. Do you want to talk us through what's happening here? >> Yeah, so it looks like it's added a test where it checks that let's see it loses one weapon health each time it is used and we can see that if you're on the right but we can also look at the test itself right in that test window here. So we have our basic setup of the attacker and the target.

06:05 We also have a sword which is the magical weapon here. And then we have the attacker attack the target with a sword and then we assert that the health of the sword is now four. So after writing the test we also see that the main agent has run the test. So we see here they run the command npm test. We have one failed test as you have pointed out. And now the main agent is kind of like reasoning out loud and says red and now we're going to go to green.

06:37 We're going to give it a weapon use that yields its damage and wears it down. And probably because it checks all the changes that the main agent is trying to do, it sees the context, the history, the test that was run and the result and it will notices that hey this new method that you're trying to introduce use is not really referenced anywhere and it's not really the simplest change you can make to make that test pass.

07:04 So it flags it as an over implementation and asks the main agent to do this the correct way. Find the simplest way to make the test pass and that's what we'll see here. >> Excellent. Excellent. So Probity has prevented the agent from creating a use method that was not called from anywhere. >> Exactly. Yeah, and that's something that agents tend to be happy about especially in more complex situations where the test seems aren't as clear.

07:32 They would get a bit lazy and you know reach for test that really don't give you the confidence you need. So I used to do this manually. I would you know, just follow whatever the main agent is doing and I would flag and ask it to undo things, change this and that and but it was tedious and I just didn't enjoy it and I'm really glad that now I have a tool that does this mostly for me.

07:57 So, I don't have to police the the TDD cycle itself. >> I think this is very cool actually cuz you can see the agent doing TDD properly. But it seems like quite a lot of trouble to go to. So, I ask Nizar to explain why is this important that it does this? >> The thing is it all boils down to confidence. And if I lose that sense of confidence, if I know that someone after me might try to refactor or make a change, but there's something important for the functionality to work as intended and it's not covered by a test or

08:32 something to communicate to the next developer that this is crucial. I feel like I really lose the whole thing that I'm leaning against that allows me and my teams to develop in a productive way. All of the sudden you start to see bugs, unexpected behavior, things that are not documented and I feel I feel like strict TDD gives me that, you know, automatically, you know, by making sure that every test fails before an implementation, you know that it can fail.

09:01 You know it's a test you can rely on. And then by making sure that you always have the simplest change to make it pass, you kind of like make it protects you against like over-engineering or unnecessary complexity or things that might mislead other people in the future. It just keeps the code clean, concise, tidy, easy to work with. I'm I just can't be thankful enough for TDD to be honest.

09:26 It keeps me sane. >> I really like that. TDD keeps you sane. Exactly my experience. So, what we've seen in this demo and what I think is very common is that the agent is writing all the code. And Nizar is not actually writing any code by hand anymore, pretty much. But I asked him if he still reads all the code that the agent is writing. >> I like to have control over exactly everything that happens.

09:50 So in the beginning, I would actually run one agent and I would just watch it and I would step in as soon as I notice something is not ideal. So yeah, I still read all the changes. I discuss them with my colleagues. I still feel that it's mine. I don't feel that it's code that I just borrowed from somewhere because I'm involved in forming it, shaping it.

10:10 I feel responsible for it and I'm also proud of it. I was not comfortable with letting agents and LLMs write code for me. I kind of let that go, but I think me understanding is critical. I just and I think it's also an important bridge basically between the agents and the real world because I'm in meetings with stakeholders, customers, team. I hear pain points, feedback.

10:33 I know some story of the project. There's a lot of things that just aren't apparent in the code itself. So as much as I would like to make the context rich for the agents so they can work more on their own. There's a lot that they just can't infer from the code alone and I have to be in that loop. And I think it'll be difficult for me to do an effective job at being in that loop and being in charge without also having the mental model, you know, it's I think it's central here.

11:02 >> So this is an important point. Nizar is very clear that he's going to read and understand all the code that the agent is writing for him. And enforcing this TDD process with his tool gives him more of a supervisor role. He told me actually he has normally about four of these agents working at once and he can supervise them all. But one thing I was a bit concerned about though was the cost of this approach.

11:23 So I asked Nizar about that. >> Yeah, so it it does cost tokens. Even though I like to use the latest model on a high effort, I noticed that they all the validation requests were actually getting routed dynamically basically on depending on how complex they are, but the vast majority were being routed to very simple small models. They were actually very cheap.

11:47 So, I understand that maybe if you are on a limited plan or a budget, this might not be the ideal way to work. For me, it's not an issue. The issue really is that it just takes longer. But, I'm also fine with that. I have I I would rather pay upfront than later. So, I would say the time is more really of something that you would notice. >> This could be a good investment, actually.

12:11 You're given some extra time and extra tokens in exchange for confidence in the design of the code. So, I asked Nizar for his best advice for anyone wanting to improve the quality of the code that they write with a genetic AI. >> I would say it's the same old advice that everyone has been giving for years, which I have also learned and found the value of it whether using agents or not.

12:33 It's try to test behavior, not you know, implementation shape. And this is so that your test survive refactoring. They allow it give you the confidence to actually make changes and you don't have to keep rewriting tests. So, make sure you test behavior and it's not about coverage, really. It's It's about confidence. You know, do those tests give you the confidence to ship without you having to manually doubt everything and want to keep checking things yourself.

13:01 >> This is all excellent advice. And if people would like to try out Proverty, where can they get it? >> So, it's there on my GitHub Nizar's {slash} Proverty. It's still an early project and I would happily receive any feedback and ways to improve it. I think this space is developing very quickly. There's a lot of great ideas. And to anyone who basically, you know, has a hunch generally speaking, you know, have a has a crazy idea they're a bit unsure, like just go for it and you really never know where that takes you.

13:31 >> Thank you. That sounds like great advice. >> Thank you. The pleasure's mine. >> I hope you enjoyed this interview with Nizar as much as I enjoyed talking with him. I think it's a fascinating insight to one approach to using a genetic AI. So you can get good results with design and confidence in the code and the tests. You saw a part of a full demo in this interview.

13:51 And if you want to see the whole thing, Nizar using Probabilistic C2 solve RPG combat, we're going to make it available to the Patreon subscribers on the One Software Engineering channel. And of course, I think Nizar also said he's going to publish it on his blog. So do check out the links in the show notes. Happy coding. >> [music]