225,000 Lines of Tests, Written and Merged by Agents - Kush Agrawal, Duolingo Transcript, AI Summary & Key Points

00:01 Hey everyone, thanks for tuning in. Today I'm going to walk you through how we use AI agents at Dualingo to write and ship more than 200,000 thousand lines of unit tests on iOS codebase. I'll share what we built and what we learned along the way. Uh I'm Kush. I'm a staff engineer on the platform team here at Dualingo. My job is to build developer tooling and AI assisted workflows for the 100 plus engineers who contribute to our client apps.

00:18 Before Duelingo, I spent close to a decade at Stripe and Uber working across a few teams there. Let's start with some background on the scale of mobile at Dualingo. Uh we ship our apps to the app store every week. Um we're also a very experiment-driven company. So at any given point of time, we have more than 600 AB tests running live and our code bases are pretty mature.

00:36 Both the iOS and Android apps have been around for more than 10 years at this point. Here's also the number of executable lines of app code in our iOS repository. So as you can see back in 2019, we were at about 200,000 lines and today we're hit close to 2 million. And as AI assisted coding picks up, you can see that curve starting to bend upward. And it's not just the amount of code that's changing, it's also who's writing it.

00:57 So last year about 200 people contributed to the iOS repository. And this year we've already crossed 250. And a lot of these folks aren't even iOS engineers. We have backend engineers, Android engineers, even PMs and designers all shipping code to the iOS app. And that's mostly been unlocked by AI coding tools like cursor and cloud code. Um, another thing to note about our codebase is that it's a codebase in transition.

01:16 So a few years ago, we standardized on the MVVM architecture pattern. Today about half the code in our repository follows that pattern and the other half is the legacy code that we're in the process of migrating. Since the MVVM half is the easiest to test, that's where we focused on this project. Here's where we started back in March of this year. We were at overall unit test coverage of 11%.

01:35 The MVVM layers were a bit lower, repositories were at 11%, view models at 9, data sources at 5, and combined we are about 9% for the MVM slice of the app. So why does this matter now more than it used to? As agents and humans write more code and write it faster, writing code isn't the hard part anymore. Verifying it is. So when someone puts up a PR, how do you know that it's actually safe to ship?

01:57 The bottleneck is really shifting from writing to verifying. And that's where unit tests come in. They make up the base of the testing pyramid. They're pretty cheap to write and pass to run. And they also act as guardrails for agents. So when an agent's change breaks a test in CI, it gets immediate feedback on what to fix. They also act as living documentation on what the code is supposed to do.

02:13 And here's where we ended up 8 months later. Overall unit test coverage went from 11% to 24%. Repositories went from 11 to 72. View models from 9 to 58. Data sources from 5 to 41. And the combined MVM slice of the app went from 9% to 59%. So what did we actually end up building? On paper, it's a pretty simple loop. First, you identify the files that need tests.

02:32 Then you write those tests with an agent. You verify that the tests are good quality and they pass CI. Then you resolve the PR, which means getting attributed by an engineer merged. And finally, it on the feedback so that the next batch of PRs is better than the last one. But in production, it's a little messier. This is what our system ended up looking like.

02:48 I won't go through every box right now, but at a high level, it's a few temporal workflows that coordinate through shared state stored in S3. And every box you see here maps back to one of the five steps I just talked about. Here are the building blocks that we ended up using. Temporal handles workflow orchestration and execution. Cloud code acts as the coding agent.

03:03 S3 holds a shared state that keeps our workflows in sync. And we use GitHub for actions, labels, and PRs. Data dogs acts as observability layer where we keep an eye on the health of our pipelines. If you haven't used empodoral before, it's basically a durable workflow execution engine. You write your workflows as normal Python code and if a worker crashes, the workflow just picks up right where it left off.

03:22 This fits agents really well and agent session can run for seconds to hours and it can call out to tools like GitHub, CI, S3, and Claude. We also lean on a few built-in features of Temporal. There are schedules which are basically chronike schedules for our jobs. There are composible workflows so that we can fan out work in parallel. Then there's first class support for retries and timeouts and item potency so that we never kick off duplicate runs of the same file.

03:45 To split up this work, we split this project into four milestones. The first was local validation. Basically, is this project even viable? The second was a scheduled backfield workflow to write tests for existing code. The third was a PR life cycle workflow to actually get those PRs merged. And the fourth was a label trigger so engineers can get tests for the new code that they're writing.

04:03 Now, we'll go through each of these in order. Before building anything real, we wanted to do the least amount of work possible that would tell us whether an agent could actually do this job well and do it at scale. We started with a very simple local cloud code setup where we had one cloud code command that acted as an orchestrator. It ran an Xcode build script to generate a code coverage report locally.

04:19 Then it fed that report through a file selection script that we wrote and then it handed each selected file off of sub two sub aents. One wrote the tests and the other one opened the PR. We ran this in a loop and generated a bunch of PRs. We basically ran this in two batches. The first patch had 17 PRs and we merged eight of them. The second had 40 PRs and we merged 19.

04:36 So that's roughly a 50% merge rate. But the goal here wasn't to merge a lot of PRs. It was to learn how the agents fail and to fix those paper cuts before productionizing anything. Here are some of the failure modes that we saw. Since we're a large legacy code base with both XC test and swift testing, the agents would sometimes get confused and mix the two in one file.

04:51 We fixed that by giving the agents clear rules on when to use each. We also had a giant user client singleton class. Some code passed around the concrete version of this class which is really hard to mock whereas others passed around a protocolized version. So we migrated the entire codebase over to the protocolized version and added lint rule to enforce it.

05:05 This made the whole host of files easier to test for agents and for humans. We also ran into six concurrency related issues and agents writing timing based tests which are a big source of flakiness in CI. So we improved the state for both of these with better rules and examples in our testing skills. Now this brings us to milestone 2. Since we were so far behind on code coverage, the biggest leverage that we saw was backfilling tests for code that was already merged in.

05:27 That's where the scheduled backfill workflow comes in. This is a system design diagram of the workflow as it runs in production and it essentially does two jobs. The left half is to identify which is picking the files that need tests and the right half is right which is having the agents actually write those tests. Let's start with identify. Every time a PR merges to the main branch in the iOS repo, we kick off a GitHub access job that generates an Xcode coverage report and uploads it to S3 that becomes a source of

05:50 truth for the latest code coverage. The back fill workflow then fetches that report and runs it through a selector script which picks the next set of files to test. Let's get a deeper a little deeper into the selector script. Its job is essentially to score every Swift file based on how confident we are that an agent can write a good test for it. The strongest signal is file type.

06:07 So view models, repositories, and data sources get a 2.5x boost while legacy files get pushed down. We skipped UI code completely for now since snapshot tests are better fit there. And larger files also get a penalty. So we prioritize code that we're confident that we can test well. The next step is to actually write the tests. We kick off one child workflow per file, all running in parallel.

06:26 In each one of these runs, the agent clones the iOS repository, reads the testing skill that we keep in the repo, which spells out how to write a good test. Then it writes the test plus any mocks and fakes it needs and opens a PR with a generated test label so we can identify it easily later. These agents run on Dualingo's cloud coding agent which is a platform that also runs on Neporal.

06:45 The simplest way to think about it is as an abstraction on top of cloud code and codex running in the cloud. We can invoke it from Slack, temporal or GitHub. It talks to a bunch of our internal tools for MCP and it logs a bunch of events so we can see what each agent run is doing and how much it costs. We started running these on Linux machines and then later move them to MacRunners which will become important in a minute.

07:03 Here's what a generated test PR looks like. Every PR description follows the same template and it says explicitly that the PR is AI generated. The reviewer has four options. They can approve it and the PR automarges. They can push changes to it directly. They can reassign it to someone better suited or they can close it out with feedback on why it isn't mergeable.

07:21 Now once we had the backfl running in production we had a slightly new problem. We suddenly had tens of tens to hundreds of open PRs that nobody could manage by hand. So how do we get all of these to a mergeable state and in front of the right reviewer that's where the PR life cycle workflow comes in. You can think of it as a state machine. It runs every couple of hours looks through every open generated test PR and takes exactly one action on each.

07:42 It covers the next two steps of our loop. Verify which is does this PR pass CI and is it high quality and resolve which is getting a human to review it and monitor it. The logic for this workflow is a simple decision tree. If there are any merge conflicts or the PR has been open for about 3 weeks, we close it out and mark it as still. If CI is still running or a reviewer is already on it, we skip it.

07:58 If CI is green and there's no reviewers, we find one assign them. If CI is red, we try to fix it. And if a reviewer rejects the PR, the file goes into 30-day cool down. So, we're not spamming engineers with the same PR again and again. When CI fails, the first thing we check is which job failed. If it's pre-commit, a GitHub action applies the fix. It's cheap, it's deterministic, and there's no agent needed.

08:17 If it's a real test failure, we kick off another agent that reads the CI logs and pushes a fix to the PR. We cap that whole loop at five attemps. Past that we close the PR rather than trying to fix it forever. Once CI is green, we need to find the right reviewer. What we do is we run git blame on the source file being tested. Find the top contributor and assign it to them and turn on auto merge.

08:34 If they approve it, the PR gets merged. If they rejected, the file goes into the 30-day cool down, which I previously mentioned. This brings us to milestone four. Once all of this was back filling tests for existing code, we wanted to make it just as easy to get tests for the new code engineers were writing. Now any engineer can add or write test label to their own PR, it hooks into the same temporal infrastructure we just walked through.

08:54 The only real difference is selection. Instead of the selection script, we take the file straight from the engineer's PR. The agent writes the test and you get a draft SPR link back to your original one. Now let's talk about some of the challenges that we ran into. Our agents originally ran on Linux machines which don't have X code or Swift installed.

09:07 So when an agent would open a PR after about 20 or 30 minutes CI would fail frequently on lint. This wastes the agents time and CI time as well which is an increasingly scarce resource. For a while 30 13.6% of our PRs were failing due to lint and then when we dug into it 62% of those failures came from just to lint rules. We fixed this in layers. The first was better examples in the prompt along with pitfalls to avoid.

09:27 The second was a GitHub action that auto fixes lint errors in CI. The third was an audit of the rules that couldn't be autofixed. We essentially removed the ones that weren't high signal and this made life easier for both agents and human engineers. And finally, the real fix came when we moved the agents to run on Mac runners. So now the agents could build and lend the code locally before ever opening a PR.

09:44 The second challenge was PR quality. When we assign a PR to a human, we wanted to be something they'd actually feel comfortable proving. But agents sometimes take the shortest paths to green CI. We saw agents editing production code to make a failing test pass. We also saw them modifying, deleting, or silencing existing tests. And then we saw them adding long weight to tests which can make CI flaky.

10:03 Again, there was no single fix here, so we built three layers. The first is the deterministic rule. If a PR touches any file that isn't a test, a mock, a fake, or a stub, we close it out. The second builds on our AI code review. At Dualingo, we use Claude and Codex to review every PR. And those B bots often leave comments. So, we built an agent that resolves any P 0 comments before a human ever looks at the PR.

10:22 And the third is an LLM as a judge. It's a reviewer agent built just for these test PRs. It reads the PR diff and rejects production edits, changed or deleted tests, disabled tests, and long weights. And as you learn more, we keep adding additional checks to it. So far, the LM as a judge has reviewed about,200 PRs and rejected 73 of them. Every rejection comes with a comment like this one, saying exactly why the PR was rejected.

10:44 We then read that feedback and fold it back into our prompts and skills, so the same mistakes get less likely each time. So, let's talk about what all of this added up to. Since we started, the pipeline has merged more than 700 test PRs, adding about 225,000 lines of test code. Every one of those PRs was reviewed and approved by a human. MVVM test coverage went from 9% to 59% and overall coverage went from 11% to 24%.

11:04 Most importantly, we now have a code base so engineers and agents can ship faster and with more confidence. Here's a few more interesting stats. Since March, the pipeline has written 34% of all new test files in the repo, 44% of all new test cases, and 10% of every swift line added to the codebase. And interestingly, that share has actually dripped in the last few weeks because engineers are now writing a lot more tests themselves, which is honestly the end goal.

11:27 Um, and here's the coverage curve since April when the backfield started running on a schedule. You can see it bent slightly upward around June and July, which is when we really ramped up the pipeline. So, what have we learned while doing this project? The first thing is that this pattern generalizes the loop we talked about identify, write, verify, resolve, iterate.

11:43 It isn't specific to testing. Unit tests were just the lowest hanging fruit and any job that fits the shape can run on the same infrastructure and that's exactly what we're starting to do at the company. We're applying the same loop to modernization to modernizing screens to MVM and Swift UI, migrating code uh on Android to Corines and generating snapshot tests.

12:00 So before I wrap up, I want to leave you with a thought experiment. There's a new class of fast general purpose classifiers like Jev. They taken some state and output an action in milliseconds. So what if we replace a handwritten decision tree in the PR life cycle workflow with one of these? PR state goes in, the classifier picks the next action and we execute it all in many seconds.

12:19 I also want to thank a few folks who worked on this with me, Jesse, Gordalup, Hunter and Jiho. They built a lot of the platform pieces this project stands on and helped review a lot of this code. That's it for me. Thank you so much for making it this far. If you want to go deeper, we've written about this pipeline on the dualing engineering blog. And I'd also love to hear from you if you're building something similar. You can find me on LinkedIn or next