Rebuilding CI for AI-Scale Commit Volume — Gautam Korlam, SonarSource Transcript, AI Summary & Key Points

00:00 Hello everyone. Uh welcome to my talk. Uh today we're going to be talking about how we are rebuilding CI for AI scale commit volume. And this comes from a lot of experience from build and merge infrastructure that I have been working on for a decade plus at this point. And kind of want to walk you through all the new bottlenecks we're seeing in software factories and also specifically for highly regulated industries and banks.

00:26 How does that translate? How do we keep things more compliant and auditable while being able to move fast? So let's dive in. My name is Goautam. I am a co-founder and CTO at QAR and recently joined Sonar as a principal engineer. After the acquisition pre previous to I was at Uber for almost a decade working on build and merge infrastructure for mono repos.

00:48 So everything from build, CI/CD, ids, you name it to make sure developers are productive. And now at QAR as part of Solar I'm trying to make sure that code that's been checked in is verified and all the code written by agents is of high quality before it goes in. So before I get started I want to just paint a bit of history picture. Uh at Uber we had agent scale coming volume before they were agents.

01:16 So what I mean by this is we had one giant monor repo and we were doing like thousands of changes a day and with very long running builds. This meant that if you just kept everything serialized and merged them one at a time so as not to break your uh main branch you would take a really long time up to days so 20 days in this case to make sure that everything was green.

01:40 And if he skipped everything and just let everyone merge to me, we would see red main all the time because changes would conflict with each other and even though the CI would be green on one of them combined, they might actually break. So we would see like our iOS monitor would be like red all the time and that was not productive for our developers.

01:58 To fix this, what we did was we made a system called summit cube which we have a paper on and we tested all the likely possibilities of commits that are coming in uh and making sure that we can like reduce the wait time so we can test these things in parallel. We could also predict what could break by testing all the combinations. So once we know that a PR might break, we might remove it back from the queue and let the author know that they have to go fix it before they can push it back up after rebasing.

02:28 And finally like to keep the trunk clean only if it passes with all the combination of commits coming in it would be let into go directly onto the trunk. So you want to make sure that the overhead was worth it for keeping our trunk clean. And this is still running today at Uber. I'm sure now we have like merge cues that GitHub came up with and all the other code hosts but for back then there was not a system.

02:50 So we developed this. [snorts] So the problem with this approach is currently all these cues that we have. So that's either PCI or review are a centralized bottleneck in the SDLC. So all these cues were built for human speed and once you build it from human speed what happens is they are not able to keep up once you go to agents because humans generate changes used to generate changes like few times an hour.

03:13 So we could do testing and review one at a time. Now with agents developers are writing like several PRs uh in many in a day and they don't always wait for the queue they all want to like come in at the same time which means your CI and review cues are piling up and having pressure and add to that like the patterns of software factories and background agents triggering on like various workflows suddenly you see like everything slowing down to a crawl because your bottlenecks are still going BCI in review and they're

03:45 over capacity in this So what does this mean uh for teams today? Right? So just to put things into perspective uh our own uh rate of commits has almost like 10xed since we started guitar uh about 3 years ago and it's just like uh unbelievable with the same size team we're still very small we're able to like push so much more code uh and we even see this in the industry trends on guitar.

04:12 So this graph shows you the number of PRs that we see that guitar reviews over time. And if you look at the growth trajectory, it's pretty exponential. Every month almost we're growing 25% month over month of just the amount of PRs that have to review fixed CI and other things. So this is also I think like corroborated by evidence that industry players like GitHub have been putting out where they're seeing like uh staggering growth in PRs and this is now coming for every single team not if our small team is seeing this

04:43 I'm sure like the larger teams and banks are seeing it like at a much higher degree as they are going through their AI adoption journeys. [snorts] So the problem I was talking about right so if agents are opening PRs and if you make them uh check everything through the CI system uh individually they might pass but once you kind of merge them back on domain your truck might still be broken.

05:05 So how do you like solve for this like how do you keep things green uh at this high rate of pace right so this is very important kind of like how we solve submit Q the changes themselves are not uh testing against each other so there needs to be some sort of change in either the CI system as well as how you architecture code and in the cues to be smarter to be able to solve this and the problem we're also seeing with everyone trying to adopt like software factories which is having agents write all the code uh from the

05:35 spec and then just having humans be responsible for the spec and review is that the factory line that exists today was built for human speed as I mentioned which means that as agents push more changes uh every part of the factory line is going to be slowing down. So for example with build and test you might have lots of rebuilds whenever uh compilation breaks or you might have integration tests you might have flaky tests that are not reproducible locally and that just slows everything down.

06:04 So because you have to retry them and it could take hours in many cases with a complex build. Add to the fact that now you have review and approval. So a lot of you probably already use like AI to review your code. Hopefully uh you're not doing every single review with with humans. You ideally are reviewing with AI as a first pass. Now the reviewers become the cue if you don't have that system in place.

06:28 And even after you review, there might be some criteria you might want additionally to make sure that it's approved fully because what's happening now in the industry is a lot of folks are just rubber stamping them because they cannot keep up with the high rate of PR. They were relying on other signals now to be able to just get through. So the process has to change.

06:46 And the other problem right now we're having is like a lot of these checks are running. There's uh a lot of records that we keep that are expiring. So like your old logs don't exist as they expire out. There is no stamp of like what was actually checked because there is like just an approval stamp. So there needs to be some record of these things for auditability and compliance checks to happen.

07:10 The other issue we're seeing is that as the code volume increases the time taken for repair also is adding a ton of overhead. Uh what we see on our platform at least 13% of PRs had a recorded CI failure. This is just one or more CI failures. Which means in all these cases there is a separate round trip to go either back to the developer machine or for an agent to understand the logs root cause of failure fix it push another commit and then keep iterating till it's green.

07:40 And this is just CI because most of the time if a human reviewer wants to look at a PR they're not going to take a look unless CI is green because that's the first line of defense. So that also like slows uh the review process because they can't go in parallel because usually we want to look at something once it's finalized and every lab is now paying for review.

08:01 So if your review comments come in and you're doing another pass same at break again and it kind of like repeats the cycle. So how do we make this better like what what do we do to make this problem more manageable as agents are pushing down more code. So we have discovered this uh obviously back to but also in guitar steam right now. But the most obvious thing is of course like make your CI really quick and fast which means that you may want to like remove a lot of the expensive checks.

08:29 Try to offload them to like post merge even and try to make the CI step itself really quick because your repairs will be much more efficient. So you can paralyze things, you can chart, you can do like uh caching. And we'll talk about some of the techniques that we have used as well. You also want to make sure the review is fast. So not just like a deep review, but you also want to have maybe multiple passes that are like giving quicker signal to your developers.

08:55 So they're able to like use them as a way to understand in what scenarios they may want to go deeper for higher risk changes or maybe for like lower risk ones even like let the AI approve and merge them. And for the checks that are already running in CI, if you can move them to the left, it's way cheaper to catch them. And this actually uh is very relevant to checks that happen even past CI.

09:16 For example, you might have like expensive regression tests or like things that might fail in production. Uh not every time it's possible to like move things to the left, but as much as possible, if you can move them, it means you have less bottlenecks down the line. And locally, agents have done more context anyway to fix the issue that they're facing.

09:34 And you also want to have policies. This is very important for regulated industries. So a story we have at Back at Uber is we lost like the license to operate in London because there was a glitch in the software that allowed drivers to get on the platform without all the proper checks and that was because of a feature flag change. So we had to put in a process in place so that every change affecting London that was going through the SDLC would need some sort of like a a lineage of okay does this change affect London?

10:04 If so, what are the controls in place and we were able to win our license back? Now, a lot of you might be facing these kind of like challenges where agents just make it much harder to understand what's happening in your codebase. So, how do you ensure your policies are actually being respected? So, you need to have a way to like enforce these rules beyond CI like just because the code compiles doesn't mean that it's following a particular specifications.

10:27 It could be like HIPPA. It could be like uh Tora. There's so many other policies that these agents are not going to be aware of unless your agents are enforcing it on the other end. And also you want to make sure that you record that it was enforced for auditability. So just a quick story about how we made our CI faster as well. So we were at a baseline, we were I would say like still fast but like very annoying when you're trying to like push more code.

10:54 uh we were around like 10 minutes for our CI and we wanted to like bring that down significantly. So we started think looking at how can we cache outputs uh we use rust for our build system. So we started like caching as much as possible and reusing them. And we also looked at ways to like reduce churn in the compiler. So we added some more fixes to like ensure that the compilation part is more cachable and is invalidated less often.

11:18 And by adding more fixes on top of this, by sharding and by removing like these overheads, we're able to bring the time of CI almost like less than a minute for incremental steps, which means once the cold CI is warmed up, incremental builds are way way quick quicker and we can push a ton more code. The other issue is also review. So just because the CI is green, what was funny was when we were measuring our uh P90 time of like PRs making into production, the vast majority of time was spent in review back at Uber and

11:50 even today to be honest like most teams spend more time reviewing code and doing other kinds of testing before they are comfortable merging the code into main. So this is something that needs to be addressed on top of CI as well because right now what's happening is agents are of course like generating code at the same time the AI agents are also like finding issues on on your uh code review system and these automated reviews are also findings that you would have to now go back and address.

12:21 So either you have to pull context locally for your agent or have an agent on CI iterate and fix and humans are able to review whatever is left after the AI took its first pass. And this needs to be part of the equation so you're not wasting precious human cycles on more and more agent generated PRs. Uh here's an example for us. Uh we were refactoring our code base to have faster CI.

12:46 So we were using next text to parallelize our uh tests and we had this like shared database that we were spinning up to make sure that we could set it up once and all our database tests would reuse it. But the problem was that there were some guards we didn't put in place which meant that when CI gets very busy uh there is like uh crosscontamination and test might become flaky.

13:09 Guitar actually is a code reviewer. It caught it on our PR before we merged it. So we didn't have to waste time after. And GAR also fixed the issue for us because it suggested the fix, committed it, and added the engineer as a co-author. By doing this, we're able to like prevent expensive mistakes before the code made it all the way into main. The other thing that we do at Qar uh interestingly and as we see more and more of our customers start doing is we're using policies and criteria to decide when is it appropriate

13:38 to actually even like approve changes. There's a lot of changes that are like routine or lower risk documentation dependency bumps of course if they're not breaking any APIs security patches uh even things like uh lower risk changes for internal tooling that can potentially be auto approved. Uh so guitar supports these things out of the box and you can also add additional custom rules which are plain English that you can use to like create the approval and merge on and we are seeing like teams uh who are like very AI

14:10 native starting to adopt these more and more. We see I think on our platform the rate at which folks are adopting like the auto approval and automerge grow really quickly across teams even from large teams to like 100 plus,000 plus developers. So here's an example of approval policies. So if it's a dependency bump and doc changes, we should be approving them.

14:29 Any changes under sensitive stuff like payments or o might need a human approver on top. And then of course you don't want to have the agent who made the change approve its change because that would not be uh zero trust. So in this case as agents are generating PRs the PRs are getting approved. When there is a new change that doesn't meet the criteria then we would have to escalate to another approver.

14:55 And this can be done with your own agents of course but also gitas out of the box if you want to try it. And the other problem with uh just relying on CI today is that just because CI is green as I mentioned doesn't mean the code is compliant in many cases. So in this case uh the CI is passing and there is some metadata that we have about it on the rerun a flaky test uh is going to fail which means that when you force push to fix that thing now the branch is pointing at another piece of code.

15:27 So the problem with this is that over time most CI data is going to be expired. So unless you have a record of what happened and what was fixed, you will not be able to like be compliant or be able to audit what actually happened. Which is why I think things like agent trajectories and like your uh outcomes after each push are very very important. So whatever information you can record on the PR or any other system of record is going to be a source of truth as your artifacts expire.

15:57 So the control here is making sure that the change was approved and tested by someone who did not write it. If you don't have that information, it could be a problem. So here's an example. We have a pipeline that failed in the left and a pipeline that passed. But the difference here is that if you record properly why the failure occurred and someone else was able to approve it because it was a known failure mode and [snorts] it was saved with the change that made it in.

16:24 This thing actually passes your checks versus something where you don't have a record of what happened but you just blindly trust the agent pushing code because it could be a test that was actually uh not high quality. There was no record of all the compliance checks that were like recorded. So you want to make sure that on the uh more sensitive uh industries you want to have all of these decisions recorded as rules or compliance checks on top of your PR in addition to code review.

16:53 So the what are some of the kinds of records that we want to keep. So here's a few that we keep on which I think are just examples. It could be different for folks. So for example, which coding agent made uh the PR? Where did the PR originate from? What was the session that originated it? And who actually started it? Could it be a trigger from like automated workflow or could it be just an engineer doing prompting?

17:17 And for each of these like CIS that run, what is a commit where CI was trusted and a save test report, not just a status? And making sure that each finding has like a record of how it was resolved. Was it dismissed? Was it followed up? Was it like fixed on the PR? And then making sure the approver is someone a apart from the bot that was prompted by the same person.

17:40 And making sure that any overrides are explicitly named. So if there is a red build or like a control code coverage skipped, you want to make sure there's a reason for when why that check failed and why we're okay with that in that scenario. And making sure that the code that merges actually has all this and it's not like a small change on top that was merged.

18:01 So all the checks are done every single time. So in this case we want to make sure that agents cannot approve their own changes. Engineer who prompted it is also not an author. And if you have these practices in place now we can actually start removing some of these bottlenecks because the CI being faster will unblock your builds. So your flow is much nicer and making sure you can like move some checks left will also make sure that your test load is going to be reduced quite a bit.

18:29 On the CI side, if you can automate review for the most part and only escalate to humans for exceptional cases and potentially even auto approve lowrisk or internal changes, then you remove further bottlenecks and then you have less that has reaching humans uh for a software factory to perform to its uh potential. And of course after every single approval and merge you want to record what happened because as factories are going to ramp up if you don't have an idea of what happened in the past that context is going to

19:01 be very useful on how to like iterate and make your software better over time. Uh that's it for me. Uh would love for you folks to also try out guitar and see how you can improve your software factories. And with that uh thank you so much for for your