60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood Transcript, AI Summary & Key Points

00:00 Picture a pull request. It changes a schema in one of our data pipelines. It goes green in CI. The tests pass. The linder is happy. And everything about it looks clean. So, is it safe to merge? Because a change like this can still break something downstream. Maybe it blows a freshness SLA. Or maybe it breaks a consumer three teams away who never saw it coming.

00:24 Or maybe it exposes data that never supposed to be reachable. That gap between CI past and this change was actually safe is the gap my team has to close every day. Hello everyone, I'm Jas. I'm an engineer on the data platform team at Robin Hood. I design and build the distributed data injection and processing frameworks behind our data lake. The ones teams across the company rely on to make decisions with data.

00:54 Notice the title of this talk. The bots in it are the ones opening the pull requests and as you will see the ones reviewing them as well. Today I want to show you how we answer that question when the PRs are coming in faster than any of us can read them. So here's the problem in one sentence. Pull requests to our production data pipelines now arrive at machine speed.

01:21 But reviewing them safely still runs at human speed. First, a quick word on the scope of this talk because it shapes everything else in this talk. AI coding assistants have made pull requests cheap to open. A new query, a schema change, or an access grant now takes just a few minutes. And an assistant can open pull requests far faster than any team of humans ever could.

01:50 The PRs I'm talking about today are the PRs to our data pipelines like schema changes, pipeline code, configuration changes, and grants that give someone access to data. Data platform is the team that owns reviewing that kind of PR. So when those PRs pile up, they pile up on us. And the reviewer I'm about to describe is the system we built to handle them.

02:15 So the people who feel this are a small group. The engineers who carry the context to say what is safe to merge into production and everyone who is waiting on them. So why is this a problem? AI open PRs just keep arriving. And to be fair, the safety net we already had works. CI checks together with a human review have always caught a lot of real bugs like broken code, inefficient queries and risks like downstream breakage or sensitive data leaks.

02:50 But the human half only works at the human speed and only with the context a person happens to carry in their head. Who reads this table? How big is it? And how sensitive is the data in it? The PRs are coming in at the machine speed, but the number of people who understand what is safe to merge into a production data pipeline did not grow at all. That's the mismatch.

03:12 The queue builds up in front of the few people with the context and that's where things start to slip. So the problem we set out to solve was this. Build a PR reviewer using AI that can keep up with the speed at which PRs are being opened. So here's the whole answer before we get into the details. Three duties, three owners. Scripts prove what can be proven.

03:40 An agent judges what needs context and people own the calls only they should make. We took a hybrid approach. Scripts for the facts and AI for judgment. We did not want AI reviewing every change when some questions already have an authoritative answer. And that's layer one. Deterministic scripts check a source of truth like the schema registry or a catalog.

04:08 A failed check turns CI red and blocks the PR until it is fixed. Layer two is the agent. The agents rules tell it what layer 1 already checks. So it focuses on the risks those checks cannot catch. It reviews the PR, scores the risk and approves or escalates to a person. Both these layers review the same PR. Merge requires passing checks and an approving review.

04:40 And this also keeps AI costs down. We use AI for judgment, not to rediscover facts a script can look up. So here is where we are going today. We have covered the problem and the approach. Now I'll walk you through the hybrid reviewer we built and the decisions we deliberately keep with the people. After that how the reviewer learns from the human feedback and what results we have seen so far.

05:06 And we'll finish with the lessons we would pass on to you. Three things to listen for as we go. Scripts for facts and agent for the judgment and people for the protected calls. I'll come back to all three at the end. So to make this concrete, let me show you the one PR will follow through the rest of this talk. This PR makes two changes to a table in our data lake.

05:36 It adds a new column and widens an existing column from integer to big integer. It's the kind of PR we see all the time. On the left is an upstream source table. Our injection pipeline reads from it and writes into a table in the data lake. call it our table. The first change, the source has gained a new column and this PR makes injection bring that new column into our lake table.

06:05 The second change is to a column of a table already has its type gets widened from integer to big integer so it can hold bigger values on its own. That sounds completely harmless. Widening a type is about the safest schema change you can make, right? Well, whenever a column changes, the real question is who reads it downstream? Some of those readers are registered in our lineage tool.

06:30 So, we know about them. But keep a second question in the back of your mind. Are those all of them? Hold on to this picture. We will come back to this exact diagram each time the PR moves forward through the reviewer. So let's start with layer one. Layer one is the scripts and their whole job is checking facts in the live systems. Things the code alone cannot tell you.

07:00 Here are the two of those checks and what they said about our running PR. First data contracts. When a PR changes a schema, a script compares the schema and keys against a source of truth, the schema registry. for our PR. The question is simple. Does that new column actually exist upstream as registered in the registry? It does. So that check passes.

07:29 Second require tags. Every table needs an owner and any PII has to be labeled in the catalog. The script checks the catalog directly. Does our table have an owner? Yes. Is its PII labeled? Yes. pass again. And if either of those had failed, there is no debate and no model involved. CI goes red like any other failing checks and the PR is blocked until it is fixed.

08:00 Contracts and tags are two of the layer 1 checks. Next, the two that look at downstream impact and performance. So these are the other two layer 1 checks. They look past the table itself, what happens downstream and whether the job will still run in time. First, the lineage blast radius. When a column changes, like the one our PR widens, a script walks every downstream job the lineage tool has on record for that column and asks a simple question.

08:32 Does any of them break? It's a real check against a real system. But notice the boundary I have drawn. It can only see what is recorded in lineage. Its view of downstream is exactly as complete as that record. No more. Second, the performance. A correct change can still be too slow or too expensive. So the CI performance test runs the change in staging at the same data volume as production using mock data.

09:04 So that tells us two things before the PR merges. whether the cluster is sized right for the job and whether the job will still make its freshness SLA. Those are the kind of problems that otherwise show up weeks later as an incident. So that's the full set of layer 1 checks. Let's see what they said about our PR. So back to our running PR. Same picture as before.

09:30 Here's what the layer one said about it. The data contract check went to the schema registry and asked whether the new column really exists upstream as register. It does and the check passes. The leanage blast radius check walked every registered consumer of the widen column. The jobs the lenage tool has on record. None of them break when the column goes from integer to big integer.

09:57 They all look fine. Along with that tags pass, performance passes. Every check is green. As far as the scripts can see, this PR is safe. But I chose those words carefully. As far as the scripts can see, everything on the slide is a statement about the registered picture, the jobs and the columns our systems have a record of. And keep that in mind. It is not the whole story.

10:26 Now let's look at the agent with the diff. What actually changed? The PR description and the commit messages explain the author's intent, but intent never explains away a real finding. If the description says it is safe and the code says otherwise, the code wins. File tools let the agent read, search, and list the files across the repository. It follows changes beyond the diff into pipeline definitions and access grants looking for the relationships the author may have missed.

11:01 These are the repository repository tools only. The agent cannot query the live catalog lineage tool or production state. Life facts stay layer 1's job. The agent reasons over the code. So why walk that code for a schema change? Because registered lineage is not the whole picture. The rules define where the judgment belongs. They are configured by the change type in the PR knowing what layer 1 already covers.

11:30 They target query anti patterns, sensitive data asks and consumers outside lineage. For example, an unpartitioned cross join can run correctly and still hammer a cluster. A grant can look ordinary and still open sensitive data. And for the schema changes, the agent walks the pipeline definitions and access grants in the repository to find consumers that were not registered.

11:55 It's a tedious search uh a person would otherwise do by hand. Let's return to our PR and see what that search found. So back to our running PR. Same picture. Layer 1 said every registered consumer of the vid column looks fine. Now the agent walks the code and it finds a job that lineage never knew about. It's right there in the repository. It reads our widen column from our late table but it was never registered in the lineage which is exactly why the layer 1 check could not see it.

12:34 And that job takes the value and writes it into another table. And in that other table, the column is still an integer. Now follow it through. Our column is a big integer. Now the first time a value comes through that's bigger than an integer can hold that right fails. A valid schema change green in CI passes every registered check and a downstream job still breaks.

13:03 The agent flagged exactly this chain in its review comment with the reasoning. So a person could see the whole path from our column to the failing right. We'll get to what happened next in a moment. So why did the lineage miss that job? Because the enterprise data world is messy. Workloads run across different query engines and each engine has to publish its own lineage.

13:25 If one does not publish, that relationship is missing from the picture. There's a single catalog that automatically sees everything. So the registered picture is useful but it is not the whole picture. Now I don't want to oversell the codewalk. It's genuinely useful but it cannot see everything. The agent can search what is in the repo. Pipeline code access grants and the jobs defined there that read our tables.

13:56 That's where it found our unregistered job. But plenty of data use lives outside the codebase. Ad hawk queries that someone runs by hand. Dashboards, those are completely invisible to a repository search. A consumer out there never shows up, no matter how carefully the agent looks. So we make the agent honest about that limit. If it does not find anything, it does not get to say no downstream consumers exist.

14:22 It says it found none in the places it searched. None found is not the same as none exist and the agent never gets to claim an allcle. So the agent has walked the code. What does it actually produce? A review the same shape a good human reviewer would give you. First review comments on the specific lines that matter each with its reasoning. Not awake looks risky but this line changes this column.

14:53 And here is the job that reads it. Second, a risk score. How severe is what it found. That's what drives the next step. Then it makes a call. If there is no serious risk, it can approve. If there is a risk that needs a person, it escalates to a human and the comments and reasoning go with it. So the person has the context to investigate instead of starting from scratch.

15:22 And to be clear about what it takes to merge, layer 1's checks have to pass and the review has to approve either the agent or after an escalation a person. Both have to say yes. Now let's see what that looked like for our PR. Here is how our running PR actually ended. Same picture. The registered consumers look fine and the core walk found an unregistered job that writes our widened column into a table where it is still an integer because the agent found a real downstream break.

15:56 It scored the risk high and escalated to a human with the whole chain spelled out in its review comment. Now the human does the part the agent cannot. The agent only reasons over the repo. It cannot see whether the job is actually running in production. human reviewer checked and confirmed that the consumer the agent found was live and active. So this was not a theoretical risk.

16:21 The PR waited the f the fix was about order not about rejecting the change. The downstream table was updated first so its column could handle the wider type and only then did our PR merge and deploy. No incident. The scripts in layer 1 said safe. The agent found the gap and a human made the call. And that was not a one-time thing. Most changes can be cleared by the agent.

16:54 When the checks pass and the agent is confident in its verdict, it approves on its own. But there's a whole category of changes where the agent never gets to say approved. And no matter how confident it sounds, grants touching sensitive data are one. Who can reach sensitive data is a decision we want a person to own every time. A migration that looks like it changes intent, not just mechanics, is another.

17:21 If a change alters what the data means rather than just how it is moved or stored, a person decides. And if the agent itself is not confident in its own verdict, that's the third. It does not get to round up to approve just because nothing concrete came back wrong. In all the all the three cases, its job is only able to surface the risk and its reasoning.

17:45 The decision stays with a person. That is one PR. Now, how does this hold up across every PR we see? And how does the reviewer get better over time? How do we know the flags are useful or whether a rule change helps? We start with the agents verdict and its reasoning in the review comments. A person replies agreeing or disagreeing and explaining why.

18:12 That explanation is what makes the feedback useful rather than just a vote. Those replies are collected every day. Once a week, the on call reads the feedback and updates the review rules by hand. Those rules shape the next review. I know this is manual today. Nothing retrains itself and no rule changes automatically. But we have an idea to let an incident investigation agent help with that weekly pass.

18:41 But it is not built. Without the feedback, every rule change would be a guess. So let me show you an example where a human corrected the agent and we use that feedback to improve its next review. So here's a real one. And note the label in the corner. This is a separate example, not our schema change PR. A column was marked as personal information in the source database.

19:10 But by the time it lands in the data lake, the injection pipeline has already encrypted it. So the PR proposed labeling it nonsensitive in the data lake even though the upstream data is still PII. The agent flagged this. From where it stood, the upstream says PII, so a nonsensitive label looked wrong. The author pushed back and asked the agent to look at how similar columns on the other tables were already classified instead of going purely off the upstream tagging.

19:42 A human agreed with the author. The encryption before write made the nonsensitive label look okay for this column. And that kind of correction is exactly the signal that teaches the rules to get smarter because the next time a similar column comes through that context can stop a needless escalation. One caution though, this depends on the specific pipeline encrypting before the right.

20:07 It's not a blanket rule that encryption always makes data nonsensitive. So how do we know whether the reviewer is actually doing a good job? We look at two kinds of mistakes. Bad PRs it approved and fine PRs it unnecessarily send to a person. That matrix separates four possibilities. A fine PR approved and a bad PR escalated are the two correct outcomes.

20:36 A bad PR approved is a miss. That's dangerous because the change ships. A fine PR escalated is a false escalation that costs a person's time. A gate that escalates everything may avoid misses, but it has not reduced the review load. As of August 31st, 2026, our miss rate was about 2%. Mostly from the gaps in the rules rather than the failures of the judgment.

21:03 We don't have a clean single false escalation number. But the dominant cause is missing context like the encrypted column example that I just gave you. Misses give us new rules and false escalation show us context the reviewer needs. So here is one of those misses. Again know the label. This is a separate example, not our schema change PR. We have an open copy of the sum of some data.

21:32 some of its columns are about to be rettagged and there is a downstream reader that reads those columns from the open copy. Privacy retag those columns as personal information. When that happens, they move into a more restricted copy of the data and they are gone from the open copy. The retag looked fine and it merged. But what nobody checked was whether anything downstream was still reading those columns from the old open copy.

22:01 something was. And once the columns were gone, that reader broke. That's a miss. A PR that should have been escalated got approved. It wasn't a failure of judgment. It was a gap in the rules. A coroner case nobody had written down. So, it became a new rule. When a column gets retagged as sensitive, search the codebase for anyone still reading it from the copy that's about to lose it.

22:25 And that's how the rules get better. the misses get found and get turned into new rules. So let's pull it all together. You remember the pull request from the start, the one that was green in CI. Three things happened to it. First, scripts proved what a source of truth could prove. The contracts, the tags, the registered lineage and performance all passed.

22:54 Second, the agent read the code and found what the scripts could not see. A job that Lineage never knew about, quietly waiting to fail on a wider column, it flagged the whole chain and escalated. Third, a person confirmed the job was live. Uh, the downstream table was fixed first and only then did the PR merge. No incident. And behind all of that, the reviewer keeps getting better because every correction a person makes can become a rule.

23:22 These results are as of August 31st, 2026. So you might ask, Jas, does it all work? Yes, about 60% of the data pipeline PRs merged without a human review. We have seen about 75% less manual review effort. The agent specific comments and reasoning help focus the review rather than making a person reconstruct every risk from scratch. The time saved comes to about 132 engineering hours a month.

23:55 Incidents tied to these PRs are down about 30%. And that does not mean human free review. The protected decisions still reach a person by design. The goal is less review work, not removing accountability. And those are the results. Now, let's look at the architecture choices we got wrong. Here is what we got wrong today. Layer 1 and layer 2 are separate systems triggered independently and running in parallel.

24:28 Each produces its own result that seemed like an obvious design when we built it, but neither sees the others pass fail or outage state. Layer 2 knows what the scripts are configured to cover, not what happened when they ran. Merge still needs passing checks and an approving review, but no single system has a full runtime picture. The agent also cannot call live layer 1 checks.

24:56 Its tools only read, search and list the repository files. If a judgment needs a live fact from a live production system, it cannot fetch that fact itself. When one site cannot finish, we wait for a human. There is no automatic fallback. These are current limitations and they motivate the proposal. On the next slide, platform team is exploring one entry point.

25:22 This is a proposal and not what's running today. The left panel is the current design and the right is where we want to go. A classifier looks at the whole PR then decides which changes to review separately and which to review together. This is a whole PR decision, not independent routing of each file. The agent would also get the layer 1 checks as collable tools alongside its repository tools.

25:49 When judgment needs a live fact, it could request the appropriate check instead of being limited to code alone. The aim here is one system with the whole PR in view. We haven't built this yet. So, it's a direction, not a result. Let me close with what we would carry forward. So, here is the whole talk in three lines. Deterministic evaluations. First, if something can be proven against a source of truth, let a script prove it.

26:17 Don't spend a judgment call on a yes or no fact. It's not just cleaner, it's cheaper, too. Every fact a script proves is review work that never needs AI. So, the cost of running this stays down. Flag never clear. Not for every PR, but for the changes that actually scare people, like sensitive grants, migration intent, and low confidence verdicts. There, the agent's job stops at surfacing the risk and the reasoning behind it.

26:50 The decision stays with the person every time. Count the false escalations. A gate that escalates everything is not actually saving anyone any time. It's just moving the work around. measure that cost the same way you measure a miss. So I started with a pull request that was green in CI and a problem PRs arriving at machine speed and too few people with the context to review them safely.

27:16 We built a hybrid reviewer scripts for the facts and agent for judgment and people for the protected calls. We walked one PR through all three saw it get lighter without going human free and saw how it learns from the corrections. We told you what we got wrong and we left you with three lessons. Prove what you can cheaply. Flag and never clear the scary changes.

27:42 And count the false escalations. So who reviews the bots? Scripts for everything a source of truth can prove, an agent for everything that needs context and judgment, and humans for the calls only they get to make. Thank you for listening.