The Agents Were Fine. Everything Around Them Broke. - Joey Wang, Harvey Transcript, AI Summary & Key Points

00:00 Let me start with one number from last month. On average, more than one agent has started on our platform every single minute, day and night. And honestly, the agents were the easy part. Hi, I'm Joey. I bu the cloud agent platform we built inside Harvey. After 10,000 PRs, the biggest lesson I can share is this. The thing that broke was an agent. It was everything around them.

00:33 So, what does one submit add up to? 10,000 PRs that an agent wrote. And it isn't just Tavac team playing with a toy. Last month, about 460 different people at Harvey use it. And over the summer, it stopped being only about code. People started asking a data question in Slack and getting answers back. My favorite recently someone on our marketing team dropped a prototype and a talk track into Slack and ask for a story book for a product video.

01:10 Once the harness exists, people bring the work that you never planned for. So a promise up front. This is not a talk about models, evos or leaderboard. This is talk about hardness, odd things around the model, checking out the ripple, holding the key, opening the PR, getting the merged. When we first wrote about Spectre back in April, we put it like this.

01:36 At some point, a harness become the product. Now, whenever I tell people this, I get the same question. Why on earth would you build this when clock or cursor already exist? My answer, we don't compete with them. We run them. And here's how I see it as my own view. Agents on your laptop are powerful, but they don't scale. You can realistically run five to 10 of them.

02:08 Pass that you need a cloud. And cloud needs our secrets, our real stack, our alerts and tickets, one set of rules and enough capacity that nobody gets starved. Those single two owns that for us. So we build layer and that layer exactly what broke. So I want to tell three stories today. How we stopped competing on writing code. how we let any agent drive and were broke once they did.

02:38 The first story start with the question we keep hearing. More of our engineers were using cursor and they ask how do I get our secret there? How do I test against our real stack? We could have rebuilt all that inside cursor and then again for next to and the one after that. Instead, the answer became one sentence. Just ask cursor to use vector. And that's really the whole idea.

03:08 People will always have a favorite agent and the favorite keeps changing. So pick yours. Cloud code, codeex, cursor, trag your own hands. Start on your laptop, push the session to the cloud, put it back. That's how Spectre sits underneath all of that. Because once the code was written, the real group starts picture appears day. Alert files in Slack at 2 in the morning.

03:37 An agent picks up nobody typing. It writes a fix, spins up real copy of our stack and post the proof on the PR. When CI fails, the follow-up loop reads the failure and pushes another fix while everyone still sleep. By morning, the PR sits in one review queue under the same rule as other PR whichever agents wrote it. The owner gets a notch and only once your working day starts and once approved a agent can automerge it.

04:17 If you only build one of those stages that I described, build a follow-up loop. We switched on for everyone in early October and that months. Four out of five agent PRs merged into our backend PR had been through it. That's the difference between agent that write code and an agent that finished the job. And then there's the knowledge. Every team has that one person who knows exactly how to test the tricky service.

04:48 Uh which flag you never touch. And that knowledge leaves in their head. of their laptop. So we gave it a hope projects people write down what matters. Adrian write down what they learn. Team pings the skills you trust and there's a record of who rent what. Any agent working in the project with the same page. My personal view whichever agent you pick the audit trail and the checks should be the same.

05:19 The second story is about rule. Back in May, if you want a spectctor to do something, you open a browser tab or tacking Slack. The whole month, exactly one task came out from terminal. So, we made a rule. If a person can click it, an agent can call it. Start a task. Turn on the follow-up loop on automerge, publish a page, all from CLI, which means your laptop agent can call it.

05:51 A agent inside the sandbox can also call it. Even CI can call it too. And then people ran with it. By October, the command line was starting about 6,500 ts a month. The web app didn't shrink. Everyone else just grew around it. Task started from the terminal were merging mops and task started in browser and the new kind of users showed up. People who start hundreds of tasks a month and never open the app at all.

06:25 And this is where it gets fun. In October, I had a big task. So I bought code session and asked to be the coordinator. It split the tickets into 10 different milestones. K one for me and the rest to Spectre with hundreds of tasks. Each has its own sandbox to validate end to end. Then at the end of the October, our it give everyone trag now watch Spectre for me.

06:53 It list the question Spectre is waiting on draft my answers and once I say yes, send them back. I actually DM to my friend like that. I just use cursor clock code or whatever to spam spectre. Now, well, that's the dream, but it's also the problem. A quick detour. When we look at where our task time actually went, the slowest step wasn't model syncing.

07:20 It was get just checking out every ripple we own took over a minute on average. The fix keep a copy of every ripple on every machine and let each task borrow it. The third story is the one where it broke in mid August. A release goes out with a bug. That bug triggers alert that fires once per workspace. So one bug become hundreds of alerts all landing in one single S channel.

07:55 Someone has set up workflow that tag our agent on every alert and every tag started task. On normal day those task started about 50 task. That day in about four hours they started around 900. 900 agents are chasing the same bug until that wasn't room left for a single note. The fix was almost boring. A repeat alert now joins the investigation that's already open.

08:25 Two months later, it happened again. And this time, it wasn't a bot. On October, the engineer did exactly what we've been asking everyone to do. Hand work to Spectre. About 950 tasks from the command line. In about 3 hours, nothing in our platform can work. The cluster filled up and everyone else work sat in line behind one laptop. That's not a user problem.

08:55 That's our problem. 4 days later, we had a real fix. A fair launch queue. Now starting a task gives you a place in line and the dispatcher started when there's room. People sitting at keyboard goes first. Everyone gets a fair share and can borrow more than the cluster is quite. And whenever you start a task, you can see a placing line. When we hit it with a 300 task burst in low test, the task lit went from almost 2 seconds to 70 milliseconds.

09:32 So did it work? When it's the busiest day of every week back in August, two certain task and about one in 10 didn't finish. Then in November 4th, more than twice that and much better in the finish line, but we still have some room to grow. I will be modest, we change several things at once, so I cannot tell you which fix mattered the most, but the birds kept getting bigger and the platform kept up.

10:06 And just when that was on the crunch show, the model neck moves. At the end of the October, our dev platform team raised an alarm. PRS across the company had jumped around 60% in a single month. A merge that used to take 15 to 30 minutes was taking over an hour now. And merge Q kept having to be switched off. So now a PR rebase onto main rerun CI and merge if it's green.

10:37 If main breaks everyone CI breaks with it. Alterately we haven't cracked this one yet. But before I go there's one more thing. While we were busy fixing all it something else was quite growing. Most of what I've been showing you start with a person asking. The fastest growing source of the work on Spectre doesn't these are automations agent that wake up on schedule on an event and just do the work tri alert read the PR write the post.

11:16 Last month about 110 people had automation running and we learned to give them a guardrail on day one. If one fails five times in a row, it switch itself off and it can carry a budget or expiry date because when no one's watching, an agent can start thousands of tasks, too. So, here's what I'm chasing next, and I don't have an answer yet, honestly. What if every Slack channel had its own agent that never locks off?

11:46 One that follows competition and start work on its own. What if every pure prove it works before a human even look at it? Today only one merge in 20 carries that kind of proof. And how do you merge when agents never stop opening PRs? Ask me again 6 months. I will tell you what broke. If you're building something like this when you work, here's three things I do again.

12:16 Oh, what happens after code is written. Make Eric action something an agent can call and cap what an agent, human or alert can start before they find out what they can do with it. And one last thought, that's my own view. Engineers can write code with whichever agent they like. The work that's lost is the harness that runs it, checks it, and get it merged. Thank you so much for watching. Come find me on linking if you are building similar things.