Letting AI Approve Low-Risk PRs: What Worked in Production Transcript, AI Summary & Key Points

00:10 I'm a software engineer at Duolingo. I've worked there for about 2 years and most of that time has been dedicated to the team that I'm on called DevX AI. Um, so you might be familiar with the traditional DevX team, you know, working on things like merge cues, CI/CD, the works like that. But my team kind of takes a different angle at that and we're focused on how do we enable our engineers to use AI effectively in their workflows.

00:35 So that's kind of what my talk today is mostly about. So what I'll cover in this talk uh kind of where we see the industry going in terms of culture change with AI adoption. um how our internal AI literacy programming um through education kind of helps our engineers, how that can enable autonomous systems um and AI agents in your company with the case study of redesigning our code review processes and then kind of how you can scale that in your organization.

01:09 So let's talk about what we're seeing in the industry today. Um most organizations do have pretty good tooling access. Um, raise of hands. How many of your organizations have cursor or cloud code in your organization? Okay, so maybe half the room. Um, how many of your organizations have a dedicated team towards helping you use those effectively? Okay, so maybe more like 10%.

01:34 So maybe by the end of this talk, you'll advocate in your organization that you might want a team like ours in your org. Um, furthermore, we see AI as becoming like more prevalent in work workplaces. So maybe last year we saw, you know, co-pilots taking off, but now agents are kind of the big thing and um, with that becomes a AI has a lot more responsibility.

02:00 Um, but where does AI adoption in your org get hard? Um, I think the first one is that engineers can be skeptical. I see some nods in the room. Um, I mean it's a big change to what people are used to, what they studied going into school. Um, so that that can be a blocker for teams like mine. Um, core systems like code review are viewed as sacred. There are best practices that have been honed for years that are maybe now getting challenged by the bottlenecks that come with AI being able to write code faster.

02:31 Um, AI also challenges and changes accountability. So this is a conversation my team has a lot you know with something like code review maybe we saw both the engineer and the reviewer responsible but when you one of those is AI like who's responsible um and then finally um trust is different kind of same as accountability it's slow to build fast to lose um with AI I think a lot of people are afraid and skeptical when it comes to like trusting more and more of their processes to AI so I'm going to kind switch gears now

03:08 and talk a little bit about like in depth some of the things that my team does to kind of make our engineering organization feel more comfortable using AI in our workflows. The first thing that we do is structured lab style workshops. Um so these are things that are pretty hands-on that are outlines. Um people can work through them at their own pace.

03:29 And um the topics that I have in this screenshot include learning how to use MCP servers, learning how to use like cursor rules, um learning how to do batching requests to LLMs and do like evaluations. But um one thing that we really have found with doing lab style workshops is that um engineers trust like the prevetted content from their engineering orgs a lot more than say you know bringing in a vendor and kind of doing a more generic solution.

03:58 Um, so this is a screenshot that we ran at one of our company all hands last year and 95% of our engineering org reported learning something new and that they would recommend this style training. So I would definitely recommend giving this a shot if you guys are focusing on, you know, some new features coming out that you want to train your organizations to be more effective in.

04:23 Um, one thing that can really help you with figuring out what to put in your workshops is actually building AI observability dashboards. So, this was kind of a small initiative that my team ran with last year that we've kind of seen a lot of success with in return on investment for and kind of continue to build out today. So we started with just doing like a daily active user chart for um one or two vendors and in-house tools and then over time we've kind of scaled that up to include you know looking at what functions

04:54 like engineering design finance are using different AI tools. We've added more AI tools. We have charts around like what developer community for example in our organization that's iOS, Android and backend but it could be definitely different in yours. Um and then we also look more at like what the breakdown of AI consumption is through things like what idees and programming languages people are using token usage cost model family lots of different groupings.

05:23 And then finally just kind of anything else that may be interesting or useful to your organization. Um, and one thing I do want to call out is we did start with this as a vibecoded project and it has continued to be vibecoded. So we could just kind of go to our AI tools dashboard and say okay add me a chart that looks at this now. And that's been really effective for like communicating well with our leadership around like what people are doing in the organization and really helping us grow from there.

05:55 Another thing my team does is live office hours for AI support. Um, so this is just a 15inute time slot that different people in our organization can book. Um, and it gives them a lot of support in terms of like I might want to use AI in my workflow, but like I don't know what to do or what are the best practices. And one thing I have found really interesting with the office hours is it's not necessarily used by engineers, but it is kind of used by these fringe parts of like people that might touch engineering a little

06:26 bit that like now need more support. So, I think people like our learning designers that might need to know now how to use GitHub because they're vibe coding more projects or um people like our QA support team like using AI more and more in their workflows and figuring out how to like actually scale what they've done off of their computer. So that has been really cool in terms of like the the impact can be broadreaching beyond your or engineering organization and it's just really awesome to see like those people have

06:58 those light bulb moments and take them back to their their teams and kind of get them to use AI more. Another thing that we do is encourage shared learnings. So we have um a slack channel for engineering building with AI and then um in parallel we have an bi-weekly engineering building with AI meetup and in those we kind of nudge people in our organization to share like what they're doing with AI in their daily work and kind of keep it low stakes but make it something that the culture of learning even sharing failures

07:34 or successes is really like shared and encouraged and that makes people a lot more comfortable using AI in their work. Finally, uh I think I put this slide up to talk about investing in your AI vendor relationships because that's been a really important part of enabling AI for our organization. Uh the first point is around it accelerates your access to new AI tooling through beta programs.

08:07 One example that I have for this is we had um we have a relationship with cursor that kind of nudges us on new things before they're available. So one of their new features last year was around adding team rules and that was a kind of a pain point we had had in our org before they had introduced this feature. So things like letting the AI agents know like what CI/CD platform we're using, what are our best practices when it comes to writing Python code, um what libraries do we have in house and how do you use those?

08:39 Um so when this AI vendor came to us and said, "We have this new feature. Are you guys interested in trying it?" We said, "Yeah, let's see if it can solve this painoint for us." And that had been really successful. Um so we do this with a lot of different companies. Um, and we're very fortunate to have like a very active le partnership with our legal and security teams where we can kind of say, "Hey, we're trying to get these new betas turned on.

09:03 What do you guys think of this? Does this fit within our security and legal boundaries?" And quite often the answer is yes. And we can kind of test out these tools, provide feedback to vendors, and advocate for our organization through that. So that's kind of the third bullet point here also is our vendor partnerships help us advocate for the organization.

09:25 So when we have like these long-standing conversations with a lot of the different AI vendors, we can tell them, hey, we're seeing pain points here. You know, vibe coding for design is a great example of one of them where we've communicated that a couple times. um so we can really you know nudge them to develop products that help our organization have that be a two-way communication.

09:51 Uh finally a great part of our uh a AI vendor relationships also is it can kind of help us identify where we are maybe like lagging behind the best practices of industry. So one example that I have for that is around eval. Um so as we were kind of bringing on online more AI agents um we realized that like oh a lot of these vendors are actually using evals in their organizations and seeing success with that.

10:18 So maybe we can talk to them about how they're doing that and copy that and do the same thing. So that that has been a really successful partnership also is just helping us figure out where the gaps are that we can learn from them. So what are the results from Dualingo investing in so much AI education and literacy for their organization? Um over the last year we've kind of hit the tail end of getting to 100% AI tool usage adoption for the engineers in our organization.

10:51 Um before that it had been 80% which is still quite high but um I think we can definitely say now that like the whole organization is far more literate than they were a year ago on this topic. Um engineers understand the strengths and weaknesses of using AI in their engineering systems which I'm going to touch on a lot more in our case study and it kind of reduces fear uncertainty and doubt in um using AI in their in their day-to-day job.

11:20 And then finally, engineers are more ready to adopt AI agents into our existing engineering workflows like code review, which I'm going to talk about in the next section. So, this is kind of a case study of the work that I did um over the last year around kind of how can we improve the bottlenecks that we see in terms of code review slowing down shipping code from AI being able to um generate PRs a lot faster.

11:50 So this is a little bit of a look at what the user experience now is like for us in terms of code review at Dolingo. We have a system called the PR risk assessment. Um and it essentially buckets code changes by low, medium and high risk or if your code review is far too big then undetermined because it exceeds the token limit. And then um if it falls into the lowrisk bucket and it meets a bunch of other criteria then it will get auto approved by our friendly auto approved bot here.

12:21 Um and what this means is that we allow you to merge your code without a human reviewer in these cases. So why target code review is an effective AI agent to run in your organization? Um, well, in my opinion, there's a lot of pros and cons when it comes to code review. Um, the pros obviously being catching bugs and getting feedback from your peers on best practices.

12:52 Another pro being mentorship, especially for junior engineers. And then finally, going back to earlier, shared accountability between the author and the reviewer. But in terms of cons, I mean, there's quite a few cons of having all of your code changes reviewed by humans as well. Um, it's inefficient for a lot of portions of code changes. Um, you know, thinking, uh, we switch teams a lot at Duolingo and so, you know, moving around what team owns a code base.

13:20 Um, it's pretty predictable, but it doesn't necessarily like need to have an actual code sign off. Uh, frequent it frequently interrupts developers. Um, you might be aware of Dualingo's famous notification system. Uh, and we try to kind of replicate some of our gamification in our company's internal practices. Um, but then you do get lots of notifications.

13:46 So, we see that a lot. Um, it's also an inconsistent developer experience. So, how many of you have sent your code at one point to maybe a person that really likes to nitpick and you've had multiple cycles where you've needed to turn turn that around? I'm seeing seeing some head nods in the audience. Yeah. Versus, you know, sometimes you just really want to get that code change in, so you send it to your work friend and they sign it off without looking at it.

14:14 So, it's it's not consistent. Um, and then finally, as I've kind of mentioned already, um, it's easier to make PRs now, and so we see the bottleneck shifting to code review for shipping quickly. So, how does our AI auto approval bot work? Uh, it's pretty similar to other CI checks you might be familiar with. um on a pull request push, it triggers our risk scoring pipeline and then that updates the pull request CI check and bot approval status.

14:52 How does our PR risk scoring work? So it takes in a bunch of parameters. Uh the PR title, the verification steps, what the actual diff is um a user prompt mainly consisting of you are grading this code change on its risk levels and then an actual risk prompt which buckets what kind of code changes are low, medium, and high risk. Sends that to an LLM and then we get a risk tier and a risk explanation from that.

15:24 And then I do want to call out too that Meta kind of pioneered this work with their diff risk score. Our system is a lot more scaled back because our engineering org is only 300 people and not 300,000 people. Um but the kind of the core principles are still the same. I want to talk for a second about what is covered in our risk assessment. So uh at a high level lowrisk changes look like you know changes to markdown files uh straightforward and small code changes.

15:57 So these are actual uh feature differences in the repos and then you know things like approved metadata and configuration changes that engineers have signed off on to say hey this is routine routine update I make I just want this auto approved. Um, in terms of medium risk, we see things like updating the API versions for our learning app. While this can be a straightforward change, the feedback has been, you know, if the API version mapping is wrong for some reason, it's just too high risk for us to like ship that as

16:31 an auto approved change. We we would love to have two people be accountable for that. um updates to foundational components like off logging things that kind of have a larger blast radius even if the code change is small. And finally, large feature code changes. So, you know, adding a new feature, but it's kind of straightforward still. And then finally, high-risk changes.

16:54 I mean, there's a lot that goes into high-risisk code changes, but the three I boil it boiled down to here are changes to user permissions. We always want to get that reviewed. um changes to largecale infrastructure and then uh changes to specific and core features of the Dualingo app. There's also a lot I have not included in here that is very Dualingo specific.

17:21 So if you are going to want to do something like this for your organization, you're you're going to see a lot of things come up and learn a lot from the organization around like what things are wide reaching. Uh, for example, one thing in the Dualingo world that I didn't include in these slides because it's not relevant to everybody, but you'll see things like this is um we have just like these asset animations and changing those can have pretty wide ranging impacts.

17:47 You know, think of like Duo at the end of a lesson like popping up and waving at you. So, kind of things like that we have bucketed in here. Um, but it it'll change from organization to organization. So you might be thinking at this point, Sarah, it is completely crazy to not have people doing human review for all code changes. And to that I would say we have a ton of guard rails in place to kind of mitigate risk.

18:16 Um so the first chunk of guard rails that is included is providing restrictions for or like providing config options for the auto approved bot in your repos. So the two that really come to mind are um one locking down the auto approved bot to only code owner changes. So for our repos we typically have like five or six people that own one codebase and you know maybe 20 or 30 people contribute to that codebase overall but only those five or six people would have the option to like turn this on only have autoproof from

18:54 them. 80% of the code changes in that repo are already from the small group of people. So it is effectively the same. Um but it's just like a nice sense of security that that group of people has. Another restriction option we provide for configs is including and excluding directories in the repo. So you might have like some parts of the repo that are very safe for an autoproof system and some that are maybe not safe.

19:18 You know, I think of something like our CI/CD repos where we have like the finance CI/CD definitely no auto approve, but we might have like the scratch CI/CD totally fine. So, kind of a wide range of things can live in repos from time to time. Um, another thing we have this completely disabled for, and we kind of debate this from time to time, but it's how it is today, um, is disabled for changing any AWS resources.

19:48 Um that's just a preference from most of our engineering org. Um it's disabled legally for repos that have higher auditing requirements like socks or um ISO for our uh Dualingo English test. So no auto approved running in that area and domain. And then we also have restricted it to new engineers during their onboarding period. And we've kind of shifted that around a little bit, but um we found that for you know people's onboarding periods, they don't know the best practices for the organization.

20:17 And so uh it is just a little bit safer to kind of have that block time window before enabling auto approve for for those people. And then um fi a final guard rail is just around opt-in notifications like per day. So some parts of the industry have kind of advocated towards moving towards a ship show ask framework where um you know you ship most of the code changes you show people before shipping and then you ask for permission in terms of uh code review before showing or asking.

20:53 Um, and so opt-in notifications are a really nice way to like have the auto approved bot but still have that visibility in terms of um, like moving fast and seeing what's been shipped like once a day instead of seeing what's been shipped every time you get notified that somebody wants you to review a code change. So the feedback loop portion of the PR risk scoring I think is really interesting and ties into where we see um the benefits of like our AI literacy programming uh really take hold.

21:34 So um I have a Slack channel project risk score and in it people report inaccuracies that they see with the bucketing of um our risk scoring so far. So the top one we have is um hey this PR probably should be marked as high risk because it changes some services or this PR should be auto approved and it's not. Um and then you know I'm changing how we connect to every endpoint.

22:02 This should probably be medium risk instead of high risk or low risk. So things like that I probably get a couple messages a week in the Slack channel. Um, and I think what's really powerful about this is that uh since we've kind of really invested in like AI education, we see that people are like pretty like indifferent and willing to accept this system.

22:24 Whereas before people had been like, hey, I might not want to turn on the auto approve bot at all. Um, but now we see people are like, I really like having auto approve on in my systems, but I just think there's an inaccuracy here that I want you to fix. So, we take these reports and essentially put them in an email list. We say, "Hey, this PR should be low risk, medium risk, high risk."

22:49 And then pick out keywords in the explanation that we just want to like evaluate and validate um the risk scoring on. And then from there once we have this eval data set we can fine-tune the risk prompt um and as new developer feedback comes in. So the risk prompt kind of grows and changes over time but is backwards compatible with all these different reports that I've gotten from people in our engineering org around um what things they want to be like low, medium, and high risk.

23:21 So it's a really cool like flywheel and feedback loop system. And I think that that this is really what is enabled by having excellent like AI training and support in your organization. Uh I also wanted to talk about the results of our auto approved bot system because I mean they have been pretty dramatic and like pretty awesome. So this top line we have here is our median time to merge of hours for all pull requests that are in our organization.

23:57 Um, so this is over pretty much the last year and we see it study at about 18 hours uh per poll request. And then um this bottom line here is the percentage of all poll requests that are being merged through our auto approved bot. So it climbs from about 0% to about um 10% over like the last sixish months. And we see that with that climb is the correlation of our median time to merge dropping from about 18 hours per pull request to about 12 hours per pull request.

24:34 Um and so some really exceptional things about this this 10% of PRs being merged through our auto approved bot. Um that includes all of the code changes that like we can't do an auto approved bot for from compliance or um you know resource change reasons. So this is like everything. It's not just what is eligible for auto approve. Um and then also like this is not just the low risk changes.

25:03 So it kind of shows that like when you accelerate the low risk changes there is that cascading effect that like the median risk and the high-risk code changes also help it helps them to like it gets everything merged in faster. So kind of to reiterate what actually changed in like the culture to enable um like an AI agent in our code review system. I would mainly argue that like the educational training for our engineers like helps us build an understanding for our AI systems that enables um AI agents to work more for

25:45 your organization. Um and this foundation like it lowers the skepticism, it lowers the uncertainty and the fear around adopting AI in your organization and kind of helps people to use AI more effectively. And then finally, this is just a case study, but we see um or yeah, I'll get to that in a second. So, yeah, like I was saying, pull request auto approvals are possible because our organization was ready for AI workflows through our education program.

26:22 But um I do want to emphasize that our auto approved bot is not special and our engineers are creating and expanding their own AI workflows um mainly through our support for AI literacy and training to kind of enable them to scale. So some other examples that immediately come to mind of where I see people making AI agents in our org. Um, we have a support bot for our organization just to do question and answers.

26:51 That has been really effective in kind of helping get people unblocked more quickly. We have a um an AI agent working to fix some of the bugs for our app, especially things that are simple and straightforward. And um we are using AI a lot in terms of um generating some of our content for the app. So like these these are not all just me. These are all different parts of the organization like using AI effectively like in their work.

27:25 So kind of branching out in terms of like how to scale AI in your org. Um it starts with access like we've seen here you know around 50% of this room um is using some sort of agentic coding uh in their organizations. But then we kind of move towards if you want to scale that more looking at enabling experimentation you know through things like hackathons and learning sharings um doing literacy through things like workshops um creating observability through things like your tool dashboards to be able to understand where

28:04 is AI being used in your organization and sometimes the tooling actually kind of can help you say, "Oh, you know, I noticed somebody in our Android or is using AI way more this week than last week. Did they learn something that we can have them do sharing on and kind of expand their knowledge?" Like, so we've I've done that a lot actually. I see people using in new ways spikes in our observability dashboards and kind of nudge them to share their learnings.

28:33 Um, and that's been really positive for our organization, too. you know, some people just need that kind of nudge to share what they've been working on. Um, additionally, guardrails. Um, I did the case study and talked about how we're guard railing the auto approved bot. Um, but that's something that my team works a lot with a security audit, legal to kind of identify what guardrails we want to have in our organization to help us use AI more effectively.

29:02 Um, and I think that that's a story that every organization is definitely working through now. And then kind of finally, once you kind of have all those building blocks into place, you can look at where autonomy and expansion are occurring in your organization. Um, so that's pretty cool. So before you add AI agents and automate, I want you to ask yourself a a couple of questions.

29:29 The first is, have we built AI literacy in our organization and not just AI access? Do engineers feel empowered over the AI systems that they have in place in their organization? Um, is learning encouraged and visible? And would AI systems feel empowering and not just imposed in your organization? So if AI is going to participate in your workflows, you need to empower your organization first.

30:01 So thank you for coming to my talk. Again, I'm Sarah. I'll be around for questions. If you want to connect with me, you can do it on LinkedIn or you can follow me on Dualingo. Hey. Yeah. Um, we've got some time for questions. So, if you like Oh, cool. Great. Someone actually um I'll bring in my mic. Uh, hello. So, I'm part of a team very similar in my company.

30:36 I was wondering if you were uh looping uh the the actual result of code review like meaning like when they do manual validation they choose to manually go through the code review are you looping on which of the manual code review are indeed uh going to do a change of the pull request. You get my pointer like which manual code review are really um involving a change and getting a feedback loop on the teams to say maybe this kind of code review you should make it automatic.

31:12 >> Okay. So your question is about um for code review changes are we verifying that like we're getting the right set in that lowrisk bucket to make the code reviews automatic? Uh it's more like when they choose to make it manual. >> Okay. >> Does does this code review lead to code change? >> You sometimes they review manually but in the end you see like a thousand of exchange.

31:39 >> Yeah. >> That doesn't lead to any change of the poll request at all and it gets validated anyway >> which is costing a lot of time to our developer to be fair. Um, >> is it clear enough or >> So, are you asking around like how often when people do manual reviews that they are actually leaving feedback that is resulting in another change? >> Yeah.

32:07 >> Okay. Um, yeah, that's a great question. So, I think my answer is kind of twofold there. Um, one part of um, the research that I did to like get the risk bucketing accurate was actually looking at what code changes were people essentially just rubber stamping already. Um, so I found that like we actually could bucket pretty well like what code changes were low risk and correlate that with what code changes were already just kind of getting a sign off with no comments.

32:38 Um, and that was one thing that made me feel really comfortable to say, hey, we should actually scale this in our organization. We have like 30% of our code changes with like no feedback, you know? So, like what's the point of pulling in people if they're not going to be giving like any they don't need to give feedback, you know? It's like a oneline change.

32:56 Um, so that's part of it. Um, and then I guess the other part of it is, you know, our auto approval bot is optional. Um and so sometimes people still want to get a manual review um just in case there is like they do want that feedback or that mentorship on a code change. So I would say maybe like maybe like 20 to 30% this is just like ballpark numbers of the auto approval bot still people kind of just reject it and say I want a code code review anyways.

33:28 Um, and that's that's fine. You know, we make it optional for that reason so people can have that mentorship. Um, but we don't see too many complaints around uh the auto approved bot like being too aggressive to where we would like be missing that kind of feedback that does cause churn of like um I guess one thing I do want to call out is this to my knowledge has never been involved in like a site incident.

33:54 So like it does have that level of like conservatism where um it's not really causing the churn and outages that you might be like afraid it would to turn this on for your org. Yeah. >> Yeah. Uh I'm here. Uh so first of all thanks for an interesting talk and um I I think you mentioned that you're right now the hund around 100% of developers already like you know using AI and efficiently using AI.

34:24 So my question is how do you measure the actual you know kind of efficiency or in in other words I mean the u what's the real value for your organization right I mean >> I saw your of course you have this um like speed up for example in number of reviews or uh time for reviews right but um >> this is you know not necessarily obviously this is because of your AI lit Right.

34:55 But not necessarily >> because of the you know developers using AI, right? >> Sure. >> Uh so yeah what what's your like gains the the value from uh from this uh AI like adoption right? Yeah, I mean I think that is the golden question in industry right now overall is kind of around you know we we now see a 100% of our developers like using AI in their workflows but uh what impact does that have to the organization um and I would probably say it's twofold for us it might be different for your organization um the first is

35:32 like people do feel more empowered to use like AI to build like to integrate in their systems um So we see this in terms of you know even in our product like we have a feature called video call with Lily where you can video call in your other language and um you know having our engineers be familiar with AI makes adding like cool AI features to our product valuable um and then in addition to that let's see I would say it also empowers our engineers to be a little bit like more ambitious in their work maybe to take on

36:10 projects that are slightly larger than they would maybe even a year ago. Um that's something that my team talks quite a bit about like um we face all these new issues now of like I'm trying to work on this new project that I'm not a subject matter expert in. I would have never ever been assigned work like this if I didn't really have the support of like using AI and kind of helping me to like speed myself up and answer questions.

36:35 So we kind of see that like throughout our organization too. All right. Yes. Good news. >> Uh, so you you've uh showed very nicely the effect on reducing mean meantime to pull request approval. >> Sure. >> Um, have you noticed or have been able to discern any change in defect rate with uh um automatic approval versus manual approval? Has there been any signal at all in in in the data there?

37:03 >> Yeah, that's a great question. Um, for the most part, no. Um, I would say that we see so we have about 200 pull requests that go through a day if my math is correct. And I would say maybe one or two of those are reverts of the auto approved bot requests. Um, but it's kind of also hard to determine like how many of those reverts are because like we needed a human reviewer versus just like a human reviewer would have, you know, I think engineers do a lot of things of like, oh, I'm going to test the CPU going up and

37:40 see what happens and oh that was bad. Let's revert it. But like they like kind of know what they're doing. So how many of it is like those cases versus like oh we probably should have used a reviewer the first case but I think this project probably would have been shut down almost immediately if we had seen like a significant defect rate like that. Yeah, >> great talk.

38:00 Uh I'm here. >> Okay. Hi. Uh thanks for sharing your experiences. Um a similar question to the previous but not so similar. So when you actually went to this journey, what were the objectives that you set yourself as an your success metrics and where do you think you are uh in in terms of your objectives overall? >> Yeah, that's a great question. Um I think that when I first set out to do this project, so um Meta's they call it their diff risk score, but it's essentially the same concept.

38:35 They actually use this in um primarily in a completely different way which is um gatekeeping high-risk code changes to go through at like safer times and I think that's a phenomenal idea too that like has a lot of merit. Um but when I kind of first started to introduce that idea um Dualingo is a company that likes to experiment and likes to ship fast.

38:58 So adding friction was just not acceptable. Um but from there we kind of could pivot and say like oh actually like this is still a great system for like for the lowrisk changes and how can we like enable our engineers to ship faster and you know it's something we've seen in our developer surveys and like and in the metrics is like people really do love to have something like this in their organizations.

39:22 We've one of the highest pieces of feedback in our most recent engineering survey was like, "Love the auto approved bot, like you guys should keep it up." Um, so kind of a pivot, not really what the original intention was, but like having an organization that supports pivots too is always great. Yeah. >> Any more? Nice. Thank you. A big round of applause for Sarah. Thank you.