What the Top Companies Do Differently With AI Coding - Martin Harrysson & Prakhar Dixit, McKinsey Transcript, AI Summary & Key Points

00:01 [music] >> All right. Good morning, everyone. Thanks for hanging with us here. I think we're coming up to lunch, so we'll uh um we're going to do one more session here. Um so I think you've heard from a number of different speakers around what they're doing in their transformations around using AI and Agentech in in the software development. And we're going to talk a little bit more about what are we seeing across?

00:35 So we have the we have the benefit of seeing a lot of these transformations at a lot of our clients. So you know, we're we're from McKinsey. I'm Martin Harrison. This is Prakhar and Shilpa, my colleagues. And so we're all from our practice that work with clients on transforming software engineering and and how that's changing with with AI and Agentech.

00:59 And and so we spend a lot of time is particularly on the people and operating model side of things. And so we're going to talk a little bit about what are some of the things that we're seeing across clients and and what are the some of the things that we see make the biggest difference when it comes to having impact. So I think uh as we sort of take a quick step back and see a little bit like where where are we when it comes to AI in in software development.

01:34 Um I think it's it's probably it's probably hard to put what like one point on this exactly, but I'm optimistic and I think that we're coming out of this trough of disillusionment. Uh I think this is you know, this has been a space that has arguably had perhaps seen the most impact of AI over the last 2 3 years. It was quick out of the gate so to speak and everyone's been about agent decoding for a while now, and I think the expectations became came huge.

02:07 When we When we surveyed a lot of our clients sort of end of last year, it was though it was a huge mismatch between the what where we actually were in the expectations. And so, most most of our large companies that are quoted relatively meager performance improvements, even though everyone was using AI in software development. The reason The reason we think that we're now maybe heading out of this is that when we re- we re-looked at this just a few months ago.

02:39 So, we surveyed a few hundred of our of our clients and and large large enterprises. And this It's clear that there's a There's a top sort of There's a 20 25% of of companies that are really starting to see some some pretty massive benefits. And when we talk when we mean when we mean with that is not just the sort of 10 15 20% like a little bit faster coding and and sort of going through the the PDLC, but real, you know, 2x like end-to-end improvements.

03:16 And so, I don't think everyone's necessarily everyone's dot is like on, you know, all the way out of this yet, but I I'm very optimistic that we're heading in in the right direction. And so, what we're we're going to cover now in the next 15 or so minutes are a few of the things that we see as patterns that that seems to be working really well. And so, we're going to talk a little bit about these four elements.

03:44 On the operating model, what are some of the things that we we see changing there? We see two redesign of the flow. I think we've we've so far seen a lot of our clients trying to use more tools in the same process and and trying to get things sped up and it's it's just not working. On the on the talent and people side, here we're seeing a massive difference.

04:09 I think we're we're seeing a lot of or several of our clients moving away from the typical scrum teams into much smaller pods and we can talk a little bit more about what that looks like and and I know that's probably come up in some of these other talks as well. So we'll we'll talk about what we're seeing across here. And and then on the on the tech side, I mean I think we're all this is a a crowd that is obviously have seen all the tools and so on and and they're very they're really amazing.

04:38 I think it's much more about how do you construct a stack that really works for you and how do you get the right context? I think we heard that in the in from the previous speaker as well. How do you set up like a a data and context foundation that actually serves you right? And then on on the last part, so much about this is around change management.

04:59 We're all fundamentally like changing how people do work and that's that's never easy and it's not really going at agentic speed yet for the most part. >> Thanks Martin. So I'll I'll cover a little bit of each of the four aspects that Martin talked about and what we are seeing as changes across those. So on the operating model side, I think you probably have seen some of this happening in your own organizations as well.

05:24 Many because of the changes that the agents are bringing to the PDLC, the cycles themselves have shifted, right? This traditional two-week sprint, the scrum team model doesn't work anymore in this newly agentic PDLC and we are seeing a lot of our clients shift more towards more of a Kanban style but one-day two-day workflows that are running where agents are running the night shift, humans are validating the output and then rerunning the loop, etc.

05:48 Now that requires setup of the agent flows in a way that you can do it. So that's the work that's needed to be done but once you do it, you can't go back to the older model of doing the two-week sprints. Um and that requires where you basically set the guardrails for human reviews, right? So a lot of our clients are deciding where they want the human review in the loop and when they are confident on that loop, they take the human review back and let the loop go to the next one and so on and so forth.

06:12 So, depending on the confidence and the maturity of the clients, we are seeing that happen more and more where these loops are more autonomous end-to-end orchestrated. Um and I think last year or even 6 months ago, a lot of those loops were more around the build and the test part of the cycle. Now, we have seen that actually expand more towards the product side and the release side as well and I think I will probably talk a little bit about what that means in terms of where the bottlenecks are today compared to where

06:36 they were before. Um and then the last thing I would say on the operating model is it is very critical that um the artifacts or the codification of what we are using to feed into these loops is predefined, right? And they are kind of standardized. You have a template because a lot of our clients and anybody who has more than 5 600 people in their organization have multiple different ways of storing the data and storing the context.

06:59 And for some specific artifacts within the loop and everyone decides what those are, the PRDs, specs, wherever they are, having that um consistent standardized artifacts helps a lot in making sure that you are actually moving towards this operating model. Um on the talent side, Martin touched a little bit of this as well, but we are increasingly seeing the structure and the size of the teams shrink.

07:21 Um that doesn't necessarily mean that people are reducing head count. It means that you have more teams, but smaller teams. Pods of two to four is what we are seeing. That also has an implication on what the responsibilities of those roles are. We are seeing them um collapse in terms of what their responsibilities are and this is true especially for larger enterprises, banks, etc.

07:41 where there were multiple different layers of roles. They're all moving towards like one standardized definition. I don't think the industry has agreed on what they will be called, but you know, the product type of roles are moving towards more what we call a definer role. The dev QA roles are moving more towards type of a builder role, etc. But we are increasingly seeing that happen organically across most of my clients because they are realizing that a lot of the manual activities that these roles were doing can be

08:05 identified and the responsibilities are expanding across um all of those. On the tech side, um I think the maybe this audience is not the right audience to say this, but the tooling part of it, right, is {quote} and {unquote} solved, I would say. Most of the tools that are out there in the market right now are doing great, right? So, it's not a question of whether the tool will work or not.

08:28 It's more what Martin was saying is like, how do you organize your stack around those tools that work in your context specifically? Um and I think the edge that organizations right now have that is that the semantic layer, how they are building it, what's their IP, what's their context, what's their knowledge. Again, for larger clients, the challenge is that a lot of that context sits in the head of like SMEs, people who've been there for 20 years.

08:49 How do you make sure that agents get that context again and again and do not have to depend on those bottlenecks, which is not new problem. This was a problem 20 years ago as well. Now that agents are actually needing that context, you have to document that knowledge in some way that can be used by the agents. Um and can't emphasize enough, you will see a lot of talk around this at AI itself is the trust layer, [snorts] right?

09:10 Like, how do you make sure that these agents are building the things in the right way? What are the things that are needed for governance, for, you know, making sure that they are secure and they are stable? Um that is becoming increasingly more and more important. And then the last part on this is change. Um I think the doing this, and we've all seen this, doing this in pockets and pilots and front-runner teams has always been easier, especially for if you're building something from scratch that new.

09:35 That is for the last year or so, everyone has done that, and that's the reports that get published, right? Like, oh, this team did this and like got 10x productivity in like 6 days, whereas it would have taken 9 months to do blah blah blah. But the real change requires you to like scale this across hundreds of thousands of engineers. Um and I think for that you need to prep a little bit before you actually unleash your agent APDLC on the rest of the organization.

09:59 You need to create a structure where you have um thought about not just what those agents are, but all the things that we talked about, what your operating model is, what the talent structure that to look like, how are you going to organize around the stack? And more importantly these days, how are you going to think about your cost for those agents that you are deploying?

10:16 And so, AI finops has obviously become very important. Um with that, maybe Shilpa you want to take that. Sure. Um I think the last point I would say, and this is connected to the tech point, is on the skills part, the challenge I think is not right now. There is enough skills available in the market, and everyone is creating more and more. I think curation is what has become more critical.

10:38 Um most of my clients, they ask me this question like, "Hey, do you see these like skills? There are so many frameworks. How can we use them?" etc. I don't think the answer is that the skills won't be available or can't be um made available to your organization, right? The challenge is more whether you can curate the right ones that fit in the right context for you.

10:55 Um you will have hundreds of thousands of skills that are easily accessible. There will only be a few handful of those that are actually meaningful for your organization, for your teams. So, curating that is where I think a lot of my clients need to spend their time. And it's changing so fast because it's like there is a proliferation for the same. I was talking to a client yesterday for just writing a simple product requirement document.

11:16 They just literally Google, and they're like 1,700 skills that you can download just for that one task, right now. Which one do you use? And it's like paralysis in decision-making. That's where they are struggling the most. So, that curation becomes a lot more important now. >> Perfect. I'm going to touch upon one aspect of this acceleration, which is actually creating inverse impact on quality, which is the fact that as we get code generation faster and faster, a lot of the reports and data that we're seeing is

11:49 actually pointing towards quality not going in the same direction, and in fact inversely going down. And some of it you would expect because if the frequency of your code commits, your PRs is increasing, you would you're bound to see more incidents, more uh quality defects, and so on. But actually in the data, even on a per developer basis, even on a per PR basis, uh we are seeing the trend downwards in terms of the of the bugs and the incidents that are getting reported.

12:14 And the fundamental of it comes down to the fact that you would want to coach your developers, testers on of course using AI and tools and and getting faster acceleration to generate code, but then the the skill set of reviewing and really looking at what has been generated is that good or not uh, also needs an equal emphasis. And from that perspective uh, a lot of the work that we're doing we are focusing on first of all curating what are good examples that we can actually feed uh, to the agents that are that are

12:50 supporting the PDLC cross and cross and and those examples one of the policies is you don't want that to be the average of your code base. You really want to curate golden examples um, that you can specify as this is what good looks like. And throughout the cycle the the the definition of what good looks like, what are my policies, what are my standards, the more you can actually put that as the context to the agent, the better the outcome is going to be.

13:16 And and and the benefit of doing it throughout the process is that you don't sit at the end of it and look at something that's sub optimal and say, "Oh, but this did not pass X quality gate or Y quality gate." And hence the the the push that we're bringing um, with a lot of our clients then comes down to breaking down at each check uh, checkpoint um, all the way from generating PRDs to what good uh, code looks like.

13:41 Is it maintainable? Is it something that's not complex? Is it something that can actually uh, merge well with the rest of the branches? These are the kind of uh, rules that you would want to encode in the agents that are supporting you across across the PDLC. Um, one interesting phenomenon that we're also seeing is that that the build and test, the one the two blocks that you see highlighted, because there's a lot of acceleration, because those those are hero use cases where we're seeing a lot of adoption.

14:11 Um, while the the time to do those two parts of PDLC, there's compression in those two stages. Interestingly, then the bottleneck in bottleneck is shifting towards the upfront part, which is how fast you also write the requirements and what are the ways in which you do it. Downstream, which is ultimately how frequently can you release? And very interestingly, sideways, and I'll talk just a little bit more about it as well.

14:36 But, the upstream piece comes down to connecting your agents to your environment, to your enterprise context, giving them the logs, the customer logs, the emails, whatever you can feed in. Uh but, then also uh equipping the the connection back to what build has to be done. Similarly, for release, really mapping out what are your enterprise requirements as it relates to release and planning for it upfront.

14:59 Um and and being able to bring acceleration in that part is important. The sideways one is interesting because we see a lot of organizations which solve the the the pieces of puzzle through and through, and you would see agents across the PDLC. But, then specially in enterprise context, there are so many processes, there are so many procedures. For example, when we were creating a customer-facing application for uh for a one of the recent banks that we worked with, they had rules specific to the color scheme that had

15:30 to be used and whether a color blind person can actually click on one button versus other and really understand that. And in the traditional context, you would ultimately have some person then running around to that team and sitting down with that team and really understanding what are those requirements. But, these are the kind of bottlenecks that we start organizations see emerge.

15:48 So, also mapping up out what are your parallel teams, uh risk, compliance, governance, design, all of those requirements also have to be encoded into the agents as as you think about this acceleration end to end. >> I mean may one one one more thing before we uh need to start wrapping up here. I think a lot um one big area that is um you know, still a work in progress, I think, is around uh measurement and and and thinking through how do we think about impact uh when we do all of this, right?

16:24 So, we're we're we're investing, we're using up tokens, we're getting new tools, we're retraining people, and then how do we know if it's if it's working or not, right? And And there's still there's still quite a big gap in this area between like the actual the adoption of AI and and so on versus uh the organizations that feel confident that they're measuring the right things and that they have a good handle on what what the output is.

16:51 I think this is a There is no I mean uh there's no quick quick answer here. Like this has been, you know, a perennial topic, I would say, in software development for as long as as long as I can remember, at least. I mean, I've been in this space for for 20 plus years. And And so, I mean, it often comes down to practically what what our clients are doing is is actually, you know, is measuring a lot of the things that are not that are not new, and then starting to add in I think I have I do have a couple of clients who

17:20 are who are using now AI in some really interesting ways to also think about impact. So, can you use AI to score the value of what you are building to actually figure out the intent, and and they're starting to see that what you want to do is is use your tokens, invest your resources in the things that are are driving the the most value, which sounds very obvious when you say it, but incredibly it actually incredibly hard to do.

17:48 Um And And I think this is one where uh we we're really in the next few months, I think, we're going to see uh a lot a lot happen on this front cuz I'm I'm starting to see this in in some of my tech clients already. Um That will really help like fig- help uh organizations figure out where to, you know, where to put their efforts and and when they're getting value.

18:08 I think the the other thing just very pragmatically that we're seeing being very helpful links a little bit to the the team size point that we were making making earlier. I've several clients that basically, you know, you reduce you reduce your pod size to something that is half or less than half of what it is today and you create twice as many teams.

18:29 And if you just think about the these teams that still in some ways are the same teams as before. So as engineering leaders you you have a mental math saying this is three teams for eight weeks or something like that. If you in the short time just keep thinking like that but the teams are half the size. You see a you ton of acceleration, right? And so that is almost like cheat code in the near term to to get some productivity but that is obviously something that you need to get more sophisticated over time.

19:02 All right, so so maybe in you know, to to wrap this up I know we've only got 20 minutes or so so we've we hit on a number of things. We will we will be here afterwards as well so we would of course love to engage with you on on more of these things that we see as patterns that work. I think this is you know, getting real value from AI in in software development is as with many things it's it's about getting 100 different things right and doing it in a reasonably coordinated fashion, right?

19:32 It's not it's not changing one thing or putting in a new tool and you're going to see this happen. And that is both the challenge but if you the good news now is if you put the effort in and do this we're seeing some some real major improvements whereas in the past you may have only seen you know, things you work for a year and it gets 20 20% better.

19:55 Here it's we're seeing 2x 3x and and beyond type improvements on on many metrics in a relatively short amount of time, right? So I'm I'm really optimistic. We're starting to see this in in the data and I think things are really starting to happen. So you know, go go out there and and and do more of this and and we'll we'll love to hear how it goes. So thank you so much for having us and have a great rest of the day. Thank you. >> Mhm.