How to Build an AI Code Review Agent - Daksh Gupta, Greptile Transcript, AI Summary & Key Points

00:01 [music] >> Um all right, my name is Dax. I am one of the co-founders of a company called Griptile and we build agents that review and test pull requests with the full context of the code base. And by reviewing and testing these pull requests, we reveal bugs, logical errors, code quality concerns, primarily from the AI-generated code. Now, to give you a sense of scale, many thousands of companies use Griptile every day.

00:37 About 1% of all pull requests on GitHub go through us and a variety of many different types of code bases, whether it's the CUDA code base at Nvidia or the wallet code base at Coinbase and many, many others. And every now and then, we meet a company where we're actually not the right code review or code testing product for them. Their needs are so specialized and so esoteric that it's actually beneficial for them to build their own.

01:02 And when that happens, we tend to help them for free. There's two reasons for this. One, we have a lot of specialized knowledge about how to do this really, really well. And second, we get to learn a lot about how we can serve customers like those in the future. And in this presentation, I want to tell you what we tell them, which is how to build a really great AI code review agent yourself.

01:22 Now, there's kind of three things that people like about Griptile and I'm going to talk through how to implement those yourself. The first one is code base context. When customers use Griptile on their pull requests, they find that it really understands their code base. As a matter of fact, before we were a code validation company, we were a code base context company.

01:39 We were teaching language models from 2023 how to understand the very large code bases and as a result of that, we got very good at indexing code. Uh the second thing that people like about Griptile is that it runs your code in the sense that spins up your stack in a fully loaded dependency-rich sandbox and it clicks around and tests things using computer user through pressure testing API endpoints and so on.

02:00 And the third thing is we're actually very efficient allocators of inference. We pick the right model, the right quantity of inference budget for each subtask in the review and validation process. As a result of that, we're able to get a lot more performance per dollar than an internal tool could or any competitor could. So, there's kind of two reasons that you would want this presentation.

02:18 Uh the first one is you want to build your own review agent. And the second reason, which is a bit more subtle, is if you're trying to pick a review or testing agent for your company and you want to understand what makes a good one and what makes a great one. All right. So, let's start right in. Now, blank slate. You are part of the platform team at your company or you're just the uh local uh AI psychotic member of your team and you want to build the review validator of your company.

02:48 Um Companies are writing lots of code. People are vibe coding. Uh you have, of course, engineers vibe coding. Some of them are a little too close to the sun. Uh you have marketing people writing code. And uh the new problem is the review bottleneck, the validation bottleneck. And so, that's what you want to address. The first thing that you should do is you should get yourself a sandbox.

03:07 There's many, many companies here that will sell you a sandbox. Uh you can also just use firecracker or something like that. And you'll want to put an agent in that sandbox. Uh the agent can be Code X or Cloud Code. Personally, I recommend going model independent for reasons that I will get into, so you can use something like Open Code or Pi as your harness.

03:23 Those are both open source. And uh you will need to then connect your GitHub or your GitLab. Have a PR webhook send it the contents of a pull request to any pull requests that are opened. The agent can have some type of prompt that just says, "Hey, review this and send back some feedback." And then you can wire that back using GitHub's review APIs. Very simple.

03:41 In fact, many people think that's actually all it takes to build a review agent. There's a few things that are going to go wrong when you start using this. But what will happen is you will, in fact, have a pull request go live. The contents will go to an agent and about 5 minutes later you will get some feedback on your pull request. But here's what's going to go wrong in the time that it takes you to build this.

04:00 The first one is you're going to miss a lot of bugs. As it turns out that actually bugs are interactively large. They're In fact, I saw a a tweet not very long ago that said that bugs are fractally infinite. So you will never run out of a of bugs in a particular piece of code. That's the first thing that's going to go wrong is you're going to miss bugs.

04:14 Of course, that's the one thing that your system needs to do is to prevent at least very catastrophic outages and incidents. Second thing that's going to happen is it's going to miss context and it's it's going to misunderstand parts of how your code base works. It maybe won't understand where you're going with your code base. It won't understand how a specific part of it works.

04:30 It maybe will not understand how one of your libraries works. It's possible you're one of those companies that never recovered from the microservices apocalypse of the 2010s and so you just have thousands and thousands of repos and you have to have context of all of them to be able to operate. So that's the second thing that's going to go wrong. The third thing is it'll be very expensive.

04:47 I think people are very surprised by this. So generally from people who have seen try to do this, they'll find that they're spending between $3 and up to $25 per pull request on reviewing it using coding agents. These tokens can add up really really fast. Very quietly they got very expensive. The fourth problem they'll find is nitpicky comments. Now this matters more than you might think.

05:07 Nitpicky comments are comments that are technically correct but overly pedantic. There's two things that go wrong when you have nitpicky comments. If you are a as a human reading those comments, they're very annoying. They're like a boy that cries wolf. You'll just see lots and lots of comments and it'll be kind of annoying for you to just read through them and figure out which ones are just minor stylistic complaints and which ones are actually catastrophic problems.

05:26 The second thing that will go wrong is they will wreck your code base because typically the way the review agents work is they'll leave comments on GitHub and your coding agent picks them up and resolves them. And coding agents are not discerning. They're going to look at review comments and just address all of them and you'll end up with extremely painfully defensive code as we found the coding agents are particularly astute at writing.

05:47 The next thing that go wrong is that every time you run a review agent on the same pull request, you will somehow find new bugs in the sense that it will keep catching new things in the pull request very annoyingly, and you will And like I said, bugs will appear to be fractally infinite. So, the first thing that you want to do, the lowest hanging fruit, what is you want to optimize the prompt.

06:06 You started with a very simple prompt, something like review this PR and then output the comments in GitHub's API schema format. Generally, there's two themes within the bugs. Look for specific bug types, and second, skip the nitpicks. And that's what you want to orient your prompt around. You'll want to ask for specific bug types, and you can kind of do some internal data science on existing outages and incidents that have happened in your code base and figure out what types of bugs you have the highest propensity to

06:31 produce. Create a catalog of them, and then have your prompt look for those specific bug types. The second is you'll need to expressly tell it what a nitpick is and not to look for them. And nitpicks can be anything from asking why there isn't three layers of defense on this particular thing, or why error handling is done in this slightly shoddy way.

06:51 Now, the thing that you'll run into when you change these prompts is when you change a small thing in the prompt, it'll have a weirdly big impact on the number of comments that your agent is writing on your pull request. This is quite scary because it means you don't really know if it's missing a bug or not. You don't know the counterfactual. The validation agents, or really any reviewer or tester, can can tell you about the presence of bugs, but it fortunately can't tell you about the absence of bugs.

07:12 That's that's sort of a a limitation of the format. And what makes this particularly difficult are bugs that are extraordinarily rare and getting rarer by the day. In fact, according to our data science, only about one out of every thousand pull requests has a generally catastrophic bug in it. Or maybe one out of every thousand lines of code has a catastrophic bug in it.

07:30 And so, because these bugs are very rare, you might release a version of your agent that you think is very good, but it secretly misses a catastrophic bug in a pull request, and you wouldn't know for a long time because you don't produce those bugs very often. And so that brings us to the second improvement. You will want a data set of historical PRs.

07:49 This is relatively easy if you're one of those companies that very nicely catalogs every time something goes wrong in your code base. If you have that, then you'll have a list of pull requests that have caused outages and incidents in the past. You'll want to catalog them and then essentially have any version of your review agent run through those and see with what frequency it can catch those bugs.

08:07 Now, this will allow you to tune the prompt a little bit more confidently. But at the same time, the interesting thing about bugs is in a changing world, the topology and ontology of bugs changes over time. And so you'll need to have a real-time updating data set. And this is tricky because unless you have millions and millions of pull requests, it's actually hard to have reasonably high coverage of bug types.

08:27 It's true that about 80 or 90% of your bugs will be just repeats of prior mistakes, but increasingly as code agents get good, they're going to start making novel mistakes. They'll fail in new ways that we've not anticipated before. And so this is where you will start to hit limitations. Now, quick side note, this is kind of what we do at Reptile, except we actually have a lot of code from a lot of different companies and we can test across a very, very large corpus of pull requests and therefore improve our agent very

08:50 confidently. It's one of the reasons why we're able to eke out little bits of performance on the margins in ways that internal review agents can't. But you can get decently far, especially for your particular company, to do this with your internal data. The next thing you want to do is model inversion. So, for in case anyone was at my talk earlier this year in San Francisco at AI Engineer, I talked a lot about the ways in which coding agents fail.

09:13 And I discovered an interesting pattern through that research, which is that the types of bugs produced by Claude models or Codex models are actually quite different from each other. And they're different in a number of ways. As an example, Claude models are something like three times more likely to produce one-off errors than GPT models. GPT models, on the other hand, are significantly more likely to produce auth bypasses than Claude models.

09:33 And these are all just artifacts of the post-training, as far as I understand. Now, to prevent this, you'll find that if you're using the same model to review the code written by the same model, um you'll actually have relatively fewer bugs caught because they fail in the same ways during review that they do during code generation. And so you'll want to invert the model.

09:51 If you've used Claude code to write the code, you'll want to use a GPT model to review it and so on. Now, doing this is somewhat straightforward for the 80th percentile and quite tricky for the last 20. So, if you're using coding agents, you'll notice they leave some type of fingerprint to tell you what coding agent was used for a particular pull request.

10:08 For instance, Codex will name branches in a codex/branch name format. Or Claude is going to write a co-authored by Claude in the PR description and commit message. And about 80% of the time, this will be true. In the final 20, when you happen to manually create a branch name or there was some other reason that a different coding agent was perhaps used, it's significantly harder to figure out which model was used during generation.

10:28 And you have to develop some type of heuristics to figure out what are the signatures of models being used. Often you might have an internal catalog where people report what coding agents are using these days or have some type of telemetry that's coming out of your dev machines and and your engineers' computers. We found that there's a 10% higher catch rate if you flip models.

10:45 And this might sound very small, but you have to know that 10% is in terms of incidents and outages is actually quite a lot. It means that there's a 10% higher chance that something will go wrong if you're not using model inversion, which can be quite catastrophic. And so we highly recommend that you use some type of model inversion in your process.

11:04 The third improvement is code base context. Now, this is something that I care a lot about. Before we were doing code validation, we were doing code base context. Uh we were trying to figure out better ways to do rag over very large code bases. And even today, people consider grep to effectively solved all of code base context. I actually think that's not true for a number of reasons.

11:21 One, grep is extremely imprecise. And second, it's very expensive. To give you some sense of scale, single-digit percentage points of GDP growth last year were inference. A very large percentage of inference was coding agents. And if you actually look at a coding agent trace, the vast majority of tokens used by coding agents are all the same. grep, file read, cache write for the file contents.

11:41 That means a non-trivial percentage of GDP growth is grep, file read, cache write. That's very bad. And this is because grep is very imprecise. It's reading files in very large quantities, and I don't know, there's perhaps some adverse incentives for companies paid by the token to figure out ways to burn more tokens. But even so, you'll find that you'll need to use slightly more surgical methods of figuring out where you should be reading.

12:00 And the way we do this is we produce an internal knowledge base where we essentially check across the entire code base all the different parts, catalog them, and then index saying, "Okay, well, this is authentication. Here's how it works. Here are the flows. Here's a diagram of the flows. Here's every file that's involved." One example of where code base context matters is, for instance, when you're trying to catch an off bypass.

12:20 It's actually very difficult to figure out if an off bypass has occurred because you need to have a pretty concrete understanding of whether you're in an authenticated environment when you're in that part of the code base or not. And that's very tricky. And so, you'll find that you will need some type of uh of sort of code base context to be able to do this.

12:36 Now, I've been very centered around the concept of bugs, but code can be very, very bad even if it's not buggy. For one, it can be terrible quality, but second, it can just not fulfill the expressed intent of the author. Recently, I saw a post that said that people were increasingly using Slack as their IDE. Now, I think that's a little funny, but at the same time, there's not that much ex- expressed intent in a Slack message.

12:58 You're only saying maybe a sentence or two, if even, asking an agent to go and fix something, which means the there's a lot of ambiguity in your instruction. As a result of this, you might find that the coding agent has either misinterpreted your instructions or just it was too ambiguous and it didn't know exactly what to do. And so, your correct code can actually still be quite wrong.

13:16 Now, one of the ways in which we fight this is by giving agents increasingly more context. And we find that the two types of context that matter most are docs and tickets, and then of course, the code base from your code base indexer or knowledge base. You can get remarkably far if you give it access to your Jira and Linear, and then your Confluence or Notion or whatever it is that you store your docs.

13:35 You'll find that there's a lot of tribal knowledge that is subconsciously helping you produce better code. And when you have agents do the same thing, it feels like they just don't know what's happening. And because you you underestimate how much depth of knowledge you yourself have. Now, the way that Reptile approaches this problem is we actually disintermediate all of these context sources with a living wiki.

13:57 This wiki is essentially a sort of world model of how your company works. It's everything that it knows about your code base, everything it knows about your application through production telemetry, through logs, everything it knows about your road map from your documentation and your linear tickets and so on and so forth. And it uses this wiki to supply context both making the agent more context aware and making its code searches significantly more precise because it will never read a file that it didn't need to read.

14:26 The last thing that you should do, or this might be four out of five. The next thing that you should do is autonomous testing. >> [clears throat] >> One of the things that we think about a lot is that for instance, for UI changes, you might be able to read through a bunch of changed files for React code and conclude that there's nothing wrong with the code.

14:42 But in fact, when you render it, it might be, for instance, hidden on an iPhone. Now, this is one of the more technically difficult problems to solve. The way that we approach this at Reptile is we essentially build these dependency-rich sandboxes, which are completely configurable virtual machines, and we have customers, we work with them to essentially spin up their stack in these sandboxes.

15:04 And then Reptile's agent uses computer vision to test them end to end. If you haven't been paying attention, computer vision has gotten very good, and it's like far better in these new generation of GPT models and Opus models than it was in the previous. For the first time, I think end to end QA is nearly solved. The thing that you need to do though is you need to document all the flows through your application.

15:21 And second, you need to be able to have a sufficiently simple build script so that the agent can spin up your stack in your environments. We have a team of forward deployed engineers that help companies do this. We found that it's one of the trickier parts. Unless of course, you have very good dev boxes already, in which case congratulations on working at Stripe.

15:43 So, oh, I This is the diagram I'm going to show you. But yes, you will need a dev box, you will need PR branch dependencies where you're running the app, and you'll need some type of flow map that says, "This is the auth code and then this is the sign-up flow." and so on. And using this, your agent can now test your code end-to-end and use vision models to say, "Oh, like this button technically appears, but it actually is hidden on iPhones, or it's not in the first fold on the iPhone duo, etc."

16:05 Now, if you do all of this, you'll find that the costs are still quite high. It can cost you anywhere from three bucks to 15 to 20 to do a single pull request review. And the thing you'll want to solve for now is to use cheaper models for some things. Now, to me, frontier models are still generally speaking more capable than open source models. This has been true for some time, even though the rubber band is tightening.

16:26 So, our share today is about 20% frontier models for the most critical tasks. And then the long tail of tokens, which is exploratory, file reads, and so on, we use a variety of smaller models. And so, if you do this, you can actually reduce the cost of running your review agent quite drastically. So, this is sort of a illustrative example of what we do, which is we'll use a small model for our finding context, we'll use a very lightweight computer use model for testing user flows, and we use a frontier model for things

16:53 like analyzing bugs, applying the review process, and first submission. In practice, we have actually 20 different subtasks, and each of them has a very specific set of models that we use for each of them. And we have a small research team that is constantly iterating exactly what models we use for what task. You can probably get 70 to 80% of the way there using something of this kind where you just swap out very simple tasks with very simple models.

17:15 Now, this is another example of where it really helps to have a very large data set. If you don't have a good data set, you might be able to have trigger happily replace a lot of different sub agents with smaller, faster models, and you won't know that it's this agent is actually now quite a bit worse at catching bugs, and you would only find out when it misses something that then causes an outage or incident.

17:34 And you will also find that you can kind of combat this problem of overactive file reads using a cheap model that essentially looks through all of your context pieces and figures out which ones actually should be included. In theory, you should be able to use a decision model for this, something like Java, or the Decisions API, to do effective re-ranking on all of the different things, and deciding what should be and shouldn't be loaded into Frontier Model Cache, which is in some ways your most expensive and most

17:55 scarce resource. If you do all of these things, you might be able to actually get quite far in building your own review agent. And if you do, please let me know. My email is daksh@prod.com. I'd love to help you. And uh if you choose not to build it, then you can also just use our product, um which is free in comparison. But, uh thank you so [snorts] much.