AI Writes Code Faster Than Uber Can Merge It - Preetam Dwivedi & Manjari Akella, Uber Transcript, AI Summary & Key Points

00:04 Uh hello everyone. Uh we are here to present our talk uh merging at um AI scale. Uh this is essentially uh Uber's journey from going to a fully green trunk to somewhat broken trunk. Um I'm Priam Devi. I'm a staff engineer at Uber working in developer platform. >> Hi, I'm Maji. I'm a senior engineer working alongside Priam. uh in the developer at at Uber.

00:32 >> Great. Um so here's the outline for the talk. Uh we're going to go over like how the merging um works at Uber at the moment and then how AI changed the landscape and then I'll respond um to that how we adapted and then discuss the results and um talk about what are we planning next. So at Uber uh we have a six language based monor repos where more than 5,000 uh developers uh are contributing over 36,000 commits uh 27,000 PRs uh every week and uh pretty much every changes uh to the services libraries are being done

01:15 in one of these monor repos. uh for the purpose of this talk we're going to focus mostly on go code when we talk about anything because that is our the biggest one and that's where the most of the scaling challenges uh we face. So let's understand the problem a little bit and um and understand like why we need a merge cube here. So uh given here basically we have two users uh user A and B creating PR A and B um on a base edge uh from the trunk and when the PRs are created and CI is run uh the CI is green builds are

01:53 passing everything is good. Now uh when a is merged into the trunk, we can see the bills for the merge commit are still green. Um bills are still passing. But when the PRB is merged into the trunk, um bills are no longer green. And the reason is that B was never tested uh with the changes that were merged by A. So uh we never tested the the combination of those two things before merging and hence the the when B is merged it's broken.

02:30 Um now imagine when we have thousands of uh engineers working in a monor repo and everyone is making changes um like hundreds and thousands of those every day. Uh this is a constant problem uh where trunk is broken um very frequently and there is a burden on fixing and there is a toll across the company to sort of um figure it out and uh fix it. Um so here uh we're going to present a solution uh to that problem uh and that solution is summit Q.

03:02 Um that is our Uber's merge Q um that we deploy uh to solve this problem. And in this case we can see our two examples here P A and PRB. And here we essentially what we are doing is we are predicting the futures that would be u that could be true on main. So there are like three things here. A could land on on its own to the main B could land on its own to the main and then the combination of A and B.

03:29 In general I think uh what we want to do is that we want to land the change that is going to keep the trunk green as a whole. So in this case when we land A and B together the trunk is green and uh and that's how basically uh sort of summit Q works and keeps the trunk green for our monor repos. Now how does it scale in this uh version of it? Um there are three things that summit key employs.

03:56 Uh the very first thing is target analysis. Uh what it means is when two users are working across two different services uh they don't really have any overlapping changes. Um so those two things can be built and tested independently of each other and then can be merged. So this allows us to basically get higher throughput and not serialize every change that is entering into the queue.

04:20 Uh second is probabistic speculation. Um what it means is uh we are using historical um the build information and also the the change characteristics to predict uh what is the success score for a given change and then we are using that to basically decide what all builds are likely needed in order to land that change. And so in that case we prioritize the build that are most likely needed and are important for that change to be uh merged to the trunk rather than like every other build that uh comes from the speculation.

04:53 The the third part is bypassing the large changes. Uh what it means is that if we have a large chain sitting in the queue which is taking longer to build and at the same time if we have a smaller changes entering into the queue um those can be built faster and so though those changes don't need to wait for large changes to land before they can land.

05:12 So in this case we can land the smaller changes without landing the larger changes first and this optimization allows us to basically move fast and get a higher throughput of the queue. So this is what has worked um when it was just humans uh landing changes creating PRs but um then we had AI and the changes at which the uh PRs were created and landed it all changed.

05:42 So to understand it a better let's look at some of the numbers. So uh in January 2026 we were doing uh roughly 90 requests uh every hour and by March we were doing 130 requests every hour. So that's roughly a 44% increase in two months and the reason for this increase is primarily because of the AI generated changes. Now given the way the system was built um uh it had the rate of uh these changes had effect on it.

06:17 So um looking into those like we had 30 minutes of uh P95 line times when we were doing 90 requests every hour and that went to roughly 3 to four hours when we were doing uh 130 uh requests every hour. So in this case we were still keeping the trunk green uh at all times but at the cost of developers essentially waiting for 3 to four hours to land their changes to the trunk.

06:44 Um now I'm going to hand over to Mry uh and she's going to talk about our response to it and how we adapted uh to the situation. >> Great. Thanks Pritham. So, Priam just uh walked us through a few optimizations that Submit Q had that worked so far in keeping up with the demand uh that humans generated. But once AI came into the picture, uh the optimizations we had were no longer sufficient.

07:11 So we our response essentially was another set of optimizations to our existing submit Q and we approached the response in three different ways. The first was we introduced this concept of admission gating which helped us to shape the traffic that we accept into the queue. Uh next we introduced the concept of deferrals which is a secondary queue that we introduced that helps with developer experience uh from the pain that comes with admission gating.

07:42 And lastly, we implemented what we call conflict relaxation which helps us buy back more parallelism uh within the queue itself. Uh next we will look at each of these uh interventions in a little bit of depth and understand exactly what each of them are doing. So the first one admission gating uh is basically what it sounds like. There is a gate at the admission of every request into the main submit que.

08:11 We look at the request that comes in and we categorize it as one of two things. Either it's a small change or it's an expensive change. And the definition of this uh could be anything. One of the definitions that we use is the target analysis that Pitham talked about uh earlier in the slides. Uh a small change touches fewer targets. An expensive change might be something that spans across uh multiple targets, the entire build graph, something like that.

08:42 So a smaller change is more likely to give you higher parallelism and we basically say admit it during peak hours and add it to the active uh submit Q build and test as usual uh and move along. But an expensive change has the capability to basically be a bottleneck by holding resources uh needing to validate against everything else that's in the queue.

09:09 So to protect the rest of the queue, what we do uh in this uh phase is just to reject any expensive change during peak hours. What this means is that the developer has to come back and retry their expensive change outside peak hours. Uh so this gives us control into what enters the queue by moving the burden onto a very subset of changes and developers.

09:41 Uh let's look at the trade-off of admission gating. Uh after we implemented this, we definitely observed an increase in our throughput. We were able to achieve over 20% increase. Uh our land rate went from 130 to about 160 during business hours. But it came at a cost and the cost is that developers become the scheduleulers. uh not all developers since we do protect the smaller changes.

10:07 But you can imagine if you are a developer who owns any types of libraries uh that span the entire code, you will be particularly targeted in this because the work that you do uh naturally falls into this expensive change bucket. Uh so we managed to protect the capacity during peak hours but had to make developers wait. Uh next slide. So the next thing that we do is targeted more towards bettering the developer experience uh of uh the cons that come from admission gating.

10:42 Uh what if instead of outright rejecting expensive changes, we are able to record them and replay them back into the main queue uh based on the status of the queue uh when we have let's say a better uh handle on the capacity. And that's exactly what deferrals implement. During peak hours, if an expensive request comes in, we no longer reject it, but put it into a deferred queue where it is persisted.

11:13 And once the gates permit, usually upon uh for example, we can say after peak hours uh we put this persisted request back onto the active submit queue. So essentially we uh shape the traffic in a way where we release more expensive changes when the queue is able to handle it. So what this solves is a developer submits once and then we automatically release it for them when the capacity allows.

11:43 Next slide. So the trade-off here is it solves the problem for the developer. They don't have to wake up early, stay up late, try to find a particular window where the queue looks good and hit the button on their end. We will handle that for them. But the other side of this, the bad that comes with this with this is that it doesn't really solve the waiting for the developer.

12:06 You still have to wait the same amount of time for your expensive change to go in to the queue. Uh, next slide. So before we look at the next uh and final optimization that we did uh I want to touch a little bit about the uh on the concept of what a target overlap is uh and how that serializes work in something. Um so recall that Fam mentioned the target analysis that we perform uh to determine whether two requests should be serialized or can they go in parallel.

12:43 So an example that we are looking at here uh is a strict one connected component. What this means is that each of these requests A through F touch some subset of targets and all of there's at least one overlap of targets between each pair of views. So like A and B overlap on at least one target, B and C overlap at least one target. Similarly for C, D, E, and F.

13:10 And what this makes uh submit Q do is group all of these requests into a single component. And that means that all of these have to go together. And this is basically what prevents uh us from doing work faster. Uh and the next optimization we do is how do we break this? Is there a way where we can break this up in a smart way? Um next slide. So conflict relaxation is what helps us break these links uh in an informed fashion.

13:45 Uh before is basically the status quo. Any relevant overlap between two requests uh makes them conflict with each other and makes uh them serialize one behind the other. But with relaxation we essentially do three steps. First we rank the overlap. So it's no longer a binary decision of is there at least one overlap between these two requests. Think of it more as a how strong is this overlap.

14:15 Uh once we have that we keep the stronger links and relax the weaker ones to buy back parallelism based on a config value that we set. Um next slide. So the first step is essentially scoring these edges uh or these overlaps. We use a jakard scoring uh for computing this dependency overlap. We basically have overlapping targets. Uh we check for that and any new dependency edges that are introduced.

14:46 We compute a combination of these two into a continuous valued score between zero and one where a higher value of score indicates a stronger overlap and a lower value of the score indicates a weaker edge with fewer targets that overlap between requests. Next slide. Once we are armed with this knowledge, we basically uh make this conflict graph again.

15:11 Right? So taking the same example and adding some numbers to the uh conflicts that we have uh from the earlier slide. Let's say we have these six changes and the config value we set says we want at least 50% parallelism in the queue at all times. What this means is that we need at least three separate cues that are working uh independent of each other and based on the u overlap scores that we have computed we are able to make this choice of dropping certain edges in order to be able to achieve that target uh Q size

15:55 that we have. So in this case uh the weakest edges are between B and C and B and E and we drop those two which gives us three separate connected components. Uh and this is how we essentially go from a serial chain of six requests to a parallelized chain where two each exist in three separate cues. Next slide. So next comes a question of um where exactly do we make this determination of uh dropping the edges right so there are two places where we can do this uh one is at a global level every time uh we are trying to

16:35 assign uh config graphs and building them uh we can basically recomputee this every time and what that does is it introduces some level of instability into the graph which is determined by any new requests that enter the queue, any work that finishes and exits the queue. So uh essentially what we we can do that uh but what we chose to do instead was a more uh local assignment uh of overlaps and the place that we choose to do this at is only at the new arrival uh checkpoint.

17:15 So once a new request comes in, we essentially take that and compute the edges of this with everything that exists in the queue, make the decisions and that's how it stays. The next decision making will only happen when another new arrival comes in. So the concept is simple. We assign once at entry time uh and basically reuse this. Um what this helps us do is any work that we schedule based on previous conflicts gra conflict graphs that we have created is preserved uh and it helps us hang on to speculative builds that

17:55 can help us move faster uh towards a finalization of request. Next slide. So the tradeoff of this final um intervention was the biggest that we saw. uh we saw over 50% increase in the throughput. So we went from about 160 lines per hour to almost 250 sometimes even greater uh during peak hours. Uh so it worked in that it not only helped us uh significantly improve our throughputs while giving us uh extremely good uh P95 land times but it came at a cost as everything does and the trade-off that we have given away here

18:38 is basically what Prium alluded to in the very first slide uh talking about the integration testing some combinations of changes will land untested because we are dropping edges from our conflict graph. What this means is main can now be broken. Uh and this creates essentially a new obligation where we have to validate uh what lands after it has landed.

19:08 Uh and to talk a little bit more about how we do this, I will hand it back to Kam. >> Great. Um thanks Mandre uh for walking us through the uh those three um sort of the solutions that we employed. So uh taking back this example that we had initially we had basically two changes here A and B and say we do a premerge validation independently after relaxing those uh conflict between those two things and then say we land it.

19:38 So now our the landed state is essentially a lands first then b and now the combination of a plus b uh fails on the trunk. Right? So this is where we started um at the start of this talk and so now we have a broken trunk again. So now how are we dealing with this? So we have a system uh here called stowpipe and stowpipe is our like post uh land uh validation system.

20:10 Um it's responsible for validating any commit that lands on to the main and then making sure that uh it it detects basically if bills are broken and if so then it creates a uh recovery uh strategy for that. Now uh here we have basically two things. One is every change that is being landed by summit Q um lands onto the main trunk and then we have the second reference here or the uh trunk um the pointer if you call is the verified main and the verified main is essentially the one that is being uh managed uh by store.

20:50 So let's take an example to understand it better. So imagine we have this comet A u where main and verified main both are pointing to it. This is a good known comet. Uh this is a building uh this is green on the trunk. Now let's say we have uh another set of comet that land onto the main say B C and D. Now main moves to point uh to the D uh commit. So at this point a STO pipe is going to take this range of commits essentially B, C and D on top of A and validate this newly uh landed changes to see if this new state is

21:32 still buildable and is still passing or not. In this case we can see it fails. And so now store pipe is not going to move the verified main pointer in this case. it will just keep it at a because that's what the good known last uh commit is on the trunk. Now over time more commits uh land onto the trunk and main moves to commit G. So now between uh the new state that we have G and the previously uh good known state which is A we have more commits in between.

22:05 So we have B C D E F and G commits in between. So now store pipe is going to take those commit uh range and basically validate and see if those are buildable and if bills are passing. So now say in this case builds are passing and so the store pipe will basically take the verified main pointer and point it to the newly uh validated uh good known uh commit uh g in this case.

22:36 So that's what a store pipe does is it is essentially observing the newly uh landed commits onto the main uh verifying if they can build or not and if they pass move the pointer if not then uh continuously look for more changes and keep validating. So now let's look at like when uh a comet landed on main u breaks the trunk. So what happens in that case as I said earlier stopipe is essentially validating and then detecting that it failed and on failure it basically alerts.

23:11 So on call gets paged uh and as a response to this um broken trunk on call is responsible to prepare a fix or revert that change um get it removed and then land through summit cube and then eventually it merges back into the trunk and then a stow pipe continuously validates the new state to see if it has been fixed or not and then accordingly move the pointer um to the newly landed comet.

23:42 So uh with all of this um I want to just recap on basically what we have achieved. So uh at peak times we are able to do roughly uh 300 uh lands per hour. we are still able to maintain our P95 land time to 30 minutes and this is essentially a 233% increase uh since the January uh numbers that we had which were like 90 lines uh per hour. So u with this u we are doing all this work um in the open u we have summit Q and store pipe being uh developed and uh everything is in the open source.

24:24 So if you want to explore and contribute um do scan the QR code explore the project and uh feel feel free to uh contribute to the project. Um what are we doing next? And um as we have already done this work uh we are still uh looking ways to uh make it better uh optimize for throughput handle the uh the recovery process in better way and also like improve on our like how we how we sort of tag what is green or not.

24:58 So as a first thing we are thinking about is on summit Q side basically batching uh related changes together so that way we run less number of builds and we can do more work uh with the same uh same amount of resources. Second is uh today we have a single pointer verified main which points whether a repository head is green or not or a given commit uh is green or not.

25:24 We want to do better where basically we can say a given service is green, given project is green. So even if a comet on the trunk is broken, uh we can essentially tell which part of the trunk is broken and not all of it. So that would allow us to basically do a better um uh sort of rollouts of those services and changes um and we don't have to block everything.

25:49 And the third one is basically uh as I said the the recovery process from a broken trunk is manual. So uh the most of the investment is around automating the fixes or reverting the fixes and then landing those uh to summit queue uh in automated way so that our time to recovery can be uh reduced and we can fix the broken trunks faster. And so as a closing point uh I want to call out that we chose a hybrid approach uh to validating the changes.

26:20 Um one we are still doing the premerge um uh validations. Um we are still uh making sure that we test the changes uh where it matters and relax the changes uh where we can take that hit and valid validate them later uh after they land to the uh trunk and then we are running the whole exhaustive uh validation with the help of a store pipe um uh post uh landing to the trunk and that allows us to basically have this balance where we are validating uh whatever whatever matters before and then validating everything at the

26:59 end to make sure it is still good. Um and uh our consumers can consume the the good estate and with that um that would be our end of the talk. Um um if you want to learn more about Summit Q and inner workings of it, feel free to scan these QR codes. Um these are the previously uh talks that we have done um on summit Q in depth and they go over like in detail on how how it functions internally and again thank you. >> Thank you.