Give Your On-Call Its Own AI Teammate - Swaapnika Motapothula, Chinmay Gandhi & Vlad Tsvang, Meta Transcript, AI Summary & Key Points

00:01 Hi, I'm Chinmai and with me are Swapnika and Vlad. We're excited to share our journey as well as our learnings around building an AI on call teammate at Meta that now runs real on call rotations everyday and around the clock. We are part of the engineering team that builds the tooling platform and infrastructure for incident response on call experience and mitigations at meta.

00:27 That space is evolving rapidly with the increase in AI capabilities. AI changed how fast we write code. About 3/4 of new code is now written with AI and more AI generated changes reach production every single day and some of it can cause production failures just like any other core change can. But on call did not speed up with it. High quality code can pass review and still break in production.

00:56 And when it does, someone still gets paged at 2 a.m. The bottleneck moved from writing code to operating it in production. And on call is where that operational load lands. So we asked if AI is writing the code, can AI also help operate it? And we think it can. Now, let's quickly run through one of those on call pages with a simplified example. It's 2:00 a.m.

01:27 and 12 alerts are firing. Same night, it can go two different ways. On the left is how most of us do this today. And notice we already have good AI. First, you sort the alerts by yourself. Checkout, payments, disk. Which one to pick first? Let's say you pick checkout. Then, you ask your assistant why P99 latency is high. It's genuinely useful. It pulls the metrics and finds the deployment at 149.

01:56 And when you ask, it reads what changed and checks the runbook. It figured out that the time config was changed to a lower value. And based on the runbook, it suggests to roll back, which is all very helpful. However, if you look at the shape of it, it waited to be asked. You picked every next step. You were the glue between the questions and once it answered, it went dormant.

02:26 At 239, you know the cause, but the roll back, the changes, and the follow-ups are still yours. Now, on the right, the same 12 alerts, but an AI teammate started before you woke up. It had already ranked them. Checkouts first, payments merged in as the same cause, and the other 10 handled separately. It read the dashboards, diff the same 149 a.m. config change, and matched two past incidents.

03:01 You get a hypothesis, a 30-cond way to verify it, and a fix waiting for your review. At 2011, you verified it, and you're back to sleep. That gap is what this talk is about. So why were you the glue? Every team was bolting automation onto on call, one silo at a time. An alert script for alerts, a triage bot for tickets, a deploy bot for deploys, a fix it agent for incidents.

03:36 Each one solves its slice, keeps its own context, and forgets it. Now replaying your 2 AM page, the alert script saw P99 latency above 2 seconds. The triage bot saw three tickets saying checkout is slow. The deploy bot knew about a config change at 149 and the incident history had two similar past incidents. Every clue existed, but nothing connected them.

04:02 So the on call became the glue exactly as we saw on the left side of the previous slide. Our bet was one AI teammate per rotation instead. One loop across every signal, one team memory that learns every shift, its own identity so it acts as itself and a human gate so you approve instead of operate. But giving one agent all that context and reach raises the stakes.

04:32 Which is why autonomy has to be earned. Under the hood, the teammate or the AI teammate is the simplest agent. There is model using tools in a loop acting on an environment and reading the feedback. Everything else in this talk is what we built around that loop to make it safe to leave it running. And for that I will hand over to Swapnikai and Vlad next to get into the actual details of how it works.

05:08 >> To achieve this we started with an autonomy ladder. Think about how you onboard a new engineer to on call. First the shadow. They investigate and suggest but they don't touch production. Then the pair they write the fix and another engineer approves it. With time they gain knowledge. We give our agent exactly the same ladder. At level one, it investigates and tells you what it found.

05:34 It changes nothing. At level two, it acts with approval. It drafts the fix, but nothing reaches production until a human approves. At level three, it clears lower criticality work on its own and only interrupts you for what matters. High criticality work stays with humans. So, what earns the next run? We use a simple test. How costly is a mistake? And how easy is it to catch?

06:01 So an action climbs only when mistakes are cheap. They're caught by tests, CI or review. And the agent already has a track record on that exact action. That's why the agent can close a noisy alert on its own while a production mitigation still waits for a human. At level one, the investigation is layered and stops at the first confident answer. The agent investigated the issue using the team's memory and runbook.

06:31 It also shared the root cause analysis, but human had to apply the fix. At level two, it writes the fix with the root cause and a test plan. It answers reviewer's comments and fixes CI, but nothing reaches production until a human approves. Most drafts don't land and that's fine. Review is the filter. At both levels, the agent does the homework, but you still had to grade every single paper.

07:04 The help got much better, but the tasks requiring human input didn't get smaller. Level three shrinks the pile. Look at this example. 23 items came in, but only three need you. On the right, the agent handled the long tail noisy alerts, duplicates, routine fixes that went through normal review and CI. Two more are still being worked. You don't have to drive each one to completion.

07:34 In the middle is your much smaller decision queue. Each item makes clear what it needs from you with a run this button that has a call to action. And remember our 2 a.m. checkout page that still stays human owned. The agent brings the investigation. The human owns the critical decision. It's the new hire ladder again. Trusted to handle routine work.

08:00 Ready with the homework when it needs help. Not just less work per item, but fewer items waiting on you. That board is where the work lives. But nobody stares at a board all the time, right? So the agent is also available across many surfaces like chat and native incident surfaces. At 9 every morning it posts a digest. 23 came in three need you. 18 handled.

08:27 Each thing that needs you come with a one-click next step. So approving the quota bump takes one tap from your phone. The 18 it handled are summarized not listed. and the two that are still in progress tell you what it's doing about them. And when you want to dig in, you just reply in the thread. As you see on the right here, you open the checkout incident and ask for the evidence.

08:55 It already has it. Errors start at 8:41, 2 minutes after a config push cut the payment client timeout. The same pattern as two past incidents. You ask a follow-up, is it only EU? And it answers with a breakdown. It drafts the role back. You approve it right there and errors start recovering. You never opened a dashboard and you never had to explain the context because it was already there.

09:23 That's what a teammate feels like. A morning handoff a colleague you can ask. Now under the hood of this AI teammate, it's AI automations. The first investigates on every new signal, auto triage classifies it, checks its criticality, and kicks off the right agent. An alert goes to the incident investigator, a task to the team's own agent, a test failure to a coding agent, a lookup to memory, and code search.

09:58 The second keeps the system healthy. Unattended automation fails quietly. So on a schedule, it looks for stuck or failed runs, retrace and repairs broken schedules, and it only pages a human if it can't fix it itself. Nothing stalls silently. The third hands off at every shift change and every morning at 9, it summarizes the shift. What needs you with the fix ready and what it handled on its own?

10:28 Posted to the team chat. Not only in the morning but every urgent issue that needs user attention gets notified in chat with findings and draft fixes. It also tags the next own call in the chat during handoff. But the entire cycle still continues. And because it belongs to the team, it runs the agents and the skills the team already trusts. That's the slot in the middle.

10:56 Bring your own agent, add your own skills. If a team already has an agent that has the team's runbooks and the knowledge base, the automations leverage it. Alongside that, it has our own incident investigator and a deep dive coding session. Work comes in from alerts, tickets, incidents, team posts, and chat. The agent itself is generic. What makes it good at an on call rotation is the team's agents skills, memory, and playbook.

11:28 And every run is logged so anyone on the team can inspect the trajectory or resume the conversation. Now I'm handing off to my teammate Vlad to talk more on the shared agent. An uncle agent can't belong to just a single person because the uncle itself doesn't. So it belongs to the team. There's one teammate per rotation on one board. Whoever is on call can ask it, correct it, or pick up its work mid investigation.

11:59 It carries the team's on call expertise, the playbook of lure profiles and past decisions from one shift to the next one, the way a good handoff should. Here's what that looks like. Monday at 2 a.m., Sam is on call and tells it, remember, if checkout times out after a config push, roll back the config first. It saves it to the memory. Three days later, Jordan is on call, hits the exact same alert, and just asks for a fix.

12:28 The agent answers with some fix, says where it came from, and has the roll back drafted. Teach it once, the next call gets it, and it accesses the team its own identity owned by the rotation with only the access the team is comfortable granting it. And it learns from every uncle shift. Handoffs, decisions, and corrections all flow into memory. So shift number 30 is better than the shift number one.

12:57 The same way a new uncle gets better every week. But memory erodess over time. So ours has to earn its place. It retrieves semantically uses what it finds, scores what it actually helped, and promotes or retires entries automatically. When someone says that was wrong, the item reopens and the correction becomes memory. Three things made this work beyond our team.

13:28 First, a loop that never stops listening. Every piece of feedback or course correction reopens the item and ultimately gets absorbed into memory. Feedback is an input to the system, not a story read once a quarter. Second, customize instead of forking. Teams bring their own playbooks and skill workflows and it's still the same teammate underneath. Third, what we call one rotation to end.

13:58 Identity, config, and memory are per team. So add a rotation becomes a setup, not a new project. The agent is generic while the expertise is owned by the team. So what has changed for the people on call? Well, first their job shifted from driving deep investigations to making key decisions. They start from a hypothesis with linked evidence, not a blank task or dashboard.

14:24 Second, the load is getting lighter. Early signs of on call load reduction are promising for teams that run it automatic autonomously and we want to measure it rigorously before we put a number on it. And the third, every change that agent writes still meets a human. Fixes land only through normal review and continuous integration. And most drafts don't land.

14:48 That's the point. Review is the filter. Encle stops being attacks and becomes a review cue. Here's what we're focusing on next. First, mitigations that get approved, not just drafted. We grade every mitigation that agent proposes and feed the outcome back so it earns its next level instead of being handed it. Second, skills are written from the chat so a team can turn its runbooks into skills without writing any code.

15:16 And third, measuring the time we give back properly with a rigorous study of call time saved rather than usage counts. And the next rung on the ladder is uh earning medium criticality autonomy with evidence. On that, thank you so much for listening to us and hopefully that was helpful. Bye.