How to Build a Software Factory That Improves Itself - Zach Lloyd, Warp Transcript, AI Summary & Key Points

00:01 [music] >> So, I'm going to be talking today about software factories and uh specifically like an infrastructure approach to software factories. Um a little bit about me though before we start. So, I'm Zach. Uh I've been an engineer for I've been for a long time at this point. I was a principal engineer at Google. I used to lead engineering on the Google Doc Suite, helped scale that up to hundreds of millions and I think maybe now over a billion people who use it.

00:37 Uh I'm currently the founder and CEO at Warp. Uh Warp is a company in the agentic development space. Our current focus is on helping companies build their own internal software factories. So, we have a platform called Warp Factories which we do that with. We're also uh pretty well-known for Warp Terminal which is an agentic terminal. It's open source around 65,000 GitHub stars, almost a million active developers.

01:04 Um our focus as a company has always been helping developers ship better software more quickly. Just to set some context on where we're at uh in terms of agentic development. So, most companies that I speak with uh are at this like phase of what I think of as interactive prompting. And so, they've deployed tools like Cloud Code or Codex to their engineers.

01:29 Uh engineers are working on their laptops. They are they might have a bunch of these going, but generally they're like interactively prompting agents to build features and fix bugs. And where uh most companies I speak with want to get to is something that is more like automated development. So, the reason for that is really there's this um you know, they want to get to a world where they are deploying this transformational technology which is agents and LLMs to get the most possible business value at the lowest

02:03 possible cost in a way that is governed and secure. And they're running up against the limits of interactive agents, which are like you you're bottlenecked by a person running them. Um it's hard to control costs if everyone has their own setup, and it's hard to govern. And so, just to double-click on that, I think for this audience, like local agents have this security problem where uh if there is a problem or a compromise of a local agent, then it's basically has the ability to take over whatever the user who's signed

02:35 in on the their laptop is, and so that's not a good thing. Um local agents are also hard to govern. So, right now, we're in this situation where you know, everyone is kind of configuring their own MCPs, their own skills, their own tools, and it's like you can't see what's going on, you can't audit it. And so, the solution uh to both these problems is to sand box your agents and move them to the cloud.

02:59 And the product category that is emerging to support this uh is what um you know, we call a cloud software factory. >> [snorts] >> And so, this this term is kind of overloaded, but what it means to me is basically, every company is going to set up a loop in the cloud where um a combination of humans and agents do work. And so, humans are responsible for putting work in, agents will do different parts of the software life cycle.

03:31 So, triage, they'll work with the human on a spec, they'll do implementation, they will do computer use verification. And as a as a business, what you're going to want to do is set these up internally, define all the rules for how humans and agents [music] interact, and then get to a spot where you can really, really measure and improve over time. Because getting this concept right is going to be the key driver to how your organization ships software.

04:02 Um it it warps like our belief is that you should think of this as infrastructure. So, not as like a product or an AI teammate, but really like an infrastructure stack that you can build on top of and scale. And the key aspects of this are like it should be open, it should be adaptable, and you should think of your factories as something that you define in code.

04:24 And so, by open I mean you want a uh factory model that can support different uh different models, whether it's open weight or frontier. You want one that can support different harnesses. And the reason is just like the best models and harnesses are constantly changing. You want adaptable means like the factory should work with your existing infrastructure.

04:45 And then finally, defining code means like your factory should be a thing uh that has like a strict definition. And that definition should include like what are the workflows? What are the skills? What are the MCPs? Where do these run? And when you define it in code and version control it, then you get the ability to like test different versions of your factory.

05:06 You get the ability for coding agents to update your factory. So, these are some key characteristics. Uh I'll say another thing that I'm hearing from from businesses is like AI sovereignty matters. And so, the concept here is like as you think about factories, you should think about uh owning all the data that goes in and all the data that comes out as well.

05:32 You should think about how do you have these agents running on your own infrastructure and own compute. Um, this is what most larger enterprises that we work with want. How do you provide your own inference? And then finally, like I said, any model, any harness. And so, really preserve optionality and treat um treat these software factories as something that's very key to your business and that you want to, you know, mostly take ownership of the core ingredients of.

06:01 Um, another really important point here is like as you think about building your software factory, like if you were to look on Twitter on any given day, you would probably see different model. Like someone would say like pie is the best or someone say grok or like there's all these different things in the world around like what is best and you might even see benchmarks like sweet bench or terminal bench.

06:20 But the world that you want to get to um as you automate more development is actually being measurement first on your own data. And so you should whatever factory model you approach have the ability to score meaning like eval how your coding agents are doing uh feed those scores into agents that can improve your factory and then also create your own benchmarks.

06:45 Meaning like be able to take all of your real coding agent tasks and try them against different models and measure how it goes. So I'll also talk a little bit about like um how engineering changes in this factory world. And [snorts] so the way that I I conceive of this and talk about it is you will still have engineers who are very much responsible for building useful things for your customers.

07:11 And so they'll be be building the features, they'll be fixing bugs, they'll be doing product engineering. And they'll be doing it like through the factory. So they put work into the factory. But there's a second kind of engineering which I call factory engineering which is uh either uh your platform team or just your general engineers will have to get good at like trying to make the factory better.

07:34 So improving the skills setting the context, changing the model mix, all of these things are going to become more important as engineers are like building and improving the system that actually builds the software. So I'm going to I'm going to show what some of this looks like in practice and again this is going to be in the context of our own product warp factories, but these general concepts I think will translate to however you're thinking about building your factories.

08:06 I'll try to leave a little bit of time questions at the end if I can. So, I'm going to start with um the view of the factory that the factory engineer or the manager has. And so, this is like if you are someone who is trying to get better uh productivity across your team, you're going to want some view of like what is happening from a productivity standpoint?

08:35 Like how automated is my factory? Uh you're going to want a view of um what's it what's the cycle time? So, like a DORA metric view. You're going to want a view of uh spending. This might take a second to load. Um but basically, you want all of this data tracked. And the way that you get these views is by having a system where everything that your team is doing I don't know if you all can see this, but everything that's happening on our team, and this is Warp's internal software development that I'm leaking here, but

09:10 it's like I can see what I can see what every engineer on our team is doing, and I can see what all the automations that we're running are doing. And not just can I see them, like I can see all the artifacts that they're creating, but all of this is getting stored. And so, all of these agent traces, you know, I I could actually go in I could see what this this one is doing.

09:30 I could go in like see the actual conversation that someone is doing. But the important thing here is that um all of this data is getting stored. And what your coding agents are doing is the raw material for improving that. And so, uh you might be wondering, well, how does how do engineers actually do this work? Like how are they putting stuff in? Um and the answer on this is that they put it in through the tools that they're already using.

09:56 So, on our team, they're using Slack uh to do a lot of the work. Um they put in uh agentic work through our task tracker, which we use Linear, but it could be Jira. They They the same context whether they're uh responding to a GitHub PR comment. And so the the idea here is like the product engineers are using the tools that they're already using, but all the work goes into this centralized location where you get a centralized view and all of this stuff gets recorded over time.

10:24 By recording everything, just to go back to this dashboard for a second. Actually, let me let me not do that for 1 second. The other thing that's important here that I I talked about earlier is like the factory workflow itself, defining it as code. And so this is our kind of like default factory workflow where think of it as like workflows in, we have this orchestrator agent that will figure out, okay, should this next go to a triage step, should it go to a spec step, an implementation step, code review.

10:57 The actual details of this need to be very very flexible. If you click into one of these, you can see like essentially you are providing an agent definition. Uh which is like what are all the secrets, the MCPs, the agent prompt, the model, all this stuff. The The cool thing though that I would emphasize that however you're approaching this, you probably want to do is that there's an actual code repo that sits behind this where if you go and like look in that repo, you can see, like I said earlier, a definition of the

11:33 entire factory at any given point in time. And so that definition includes Think of it as like everything that goes into the process of automating development. It's your It's your agent definitions, it's your automations, your benchmarks, it's where these agents run, the skills, the webhooks, everything. So everything gets stored in in a repo that it allows for automation.

11:56 Um And I'll I'll I'll show a little more why that matters in a second. The other thing, just to return to this view for a second cuz I think it's interesting, you get these metrics around productivity and spend, but you also want to set up something where there are like we call them scores. You could think of it as like online evals. And what what you're looking at here is basically we have a set of agents that are looking at all of these runs that are going through our system.

12:25 So remember I go back here it's like you can see everything that everyone's doing on our team. You then have a scoring mechanism that goes and looks at those runs and sort of figures out by using LLM as a judge whether or not um those runs are good or not. And so for instance we have a score here that is looking for agents that are writing redundant tests which is a common failure mode with these coding agents.

12:53 And so for the most part our our agents seem to be doing pretty well here. But if you look back in time you'll see there's some set of them where an agent is deciding okay this was this thing wrote tests that didn't need to be written. And so every one of these failures is an opportunity for improving how your system works. And so as you gather this information and you know you could go look at this as a person and grade it but it's much more efficient to have agents do it.

13:19 As you gather this information you can then use it to feed what we call a self-improvement loop. And so the idea behind a self-improvement loop is that agents will go and they'll look at all these failure modes. They'll aggregate them. They will look for patterns. And then they will come up with ideas for how do you improve the underlying factory definition so that going forward you don't have this same failure mode.

13:49 And so you know for instance I don't know I'm going to pick a random one here. Um This is like read match more positive skills before implementing. The key thing here is like it's saying I saw this fail 37 times. And then it's saying okay we can actually um oops. We can actually fix this um with a PR to the factory definition. So, this is like closing the loop where the agent has identified a problem and it's suggesting a change to actually drive a better result in the factory.

14:30 So, this is like a a core concept I think as you think of like how do you really improve development? It's it's about creating these loops where agents can identify problems and then fix them. There's one more thing that I want to show and then maybe I'll take a couple questions. Um, I mentioned early on like you can't really trust the public benchmarks [clears throat] as like a way of understanding how good a certain model or harness is going to be on your own workflows.

14:57 And so, what you want to get to a point where you can do is basically take any past run. So, we can do this like with any anything that we've done in our system. I could go into it here and I have the um, Sorry, I picked a bad example. I um, Uh, maybe I'm picking a bad example. The The idea here is like you can basically take past runs. I don't know why I can't do this here and say okay, I want to add this to a benchmark suite.

15:32 And what this is doing is okay, this is like a representative thing that I did in my factory. They could be useful to measure by testing it with different model configurations. And so, think of like building up a a suite of like front-end tasks. And the question that you're trying to answer is like what's the best model for me to use for front-end tasks?

15:50 And like you might look on Twitter, but you might also just be like let me actually test. And so, what you can do or should try to set up for your own system uh, and this is built into Warp Factory. So, I'll show it here is basically curate one of these benchmarks. And what what it does is it will replay your real past coding agent tasks and then generate for you what looks like a basically a cost versus quality trade-off graph.

16:22 And so, this is something you'll see when uh you know, the public benchmarks come out, but this is on your own data. And so, in in this case, for example, like we could see GLM 53 flash uh was 93% correct on these tasks at a cost of $2. GPT 56 saw was like 95% correct, but it cost three times as much. And so, you can take this information and then build a model routing strategy that you have real confidence is not just a generic one, but for your actual code, your stack, is actually providing value.

16:59 And so, um this is like I I think a a good example of like closing the loop. And then, just to go back, I'm going to finish up here in a second. The the sort of like way that we think about this is like at the bottom of your stack, it's defined in code. Think of a data layer, what's the data going in and out. Where do these agents run? You really want to sandbox them.

17:29 You want inference that's flexible. You want a self-improvement layer that gives you something like benchmark scoring. You want orchestration so you can have multi-step workflows, and then you want access from all these existing tools. And so, that's like the infrastructure that I think is going to look it's going to look something like this um no matter how you approach this problem.

17:51 Uh if you're interested in our approach, obviously we're we're we would love to talk to you, but let me pause there. Hopefully, this was interesting to folks, and I might have time for one question, I don't know, or I might have to go. Yes. >> What uh what advice do you have for testing and verification for the factory in general? >> Awesome question.

18:12 So, the you know, one thing that could happen if you do this approach is you get like a pile up of PRs, right? And so, you want some way of like getting them through. The big things that I advise are spec-driven development, so you could actually test um make sure that the changes are adherent to a spec. And the other thing is computer use verification, which I didn't show, but actually um you know, if you can have agents that verify or produce evidence from the user standpoint that the thing works, that can go a long

18:42 way in terms of like lessening the code review burden. So those are the Those are the two things that I recommend. Yes. >> So how do you plan um the budgets when we are starting a project? How do you plan this verification cost? >> Yeah, so the the way that we do this is like we give people the tools to measure like what's your cost per PR, and then what percent of the cost is going towards like scoring or benchmarking or stuff like that.

19:11 Uh and then you can kind of extrapolate out, like are you willing to spend X amount on a project or whatever, but the key the key thing um like that I want people to leave with there's one idea is like really set up something that you can measure and improve. And once you can truly measure in a closed-loop system, you can you can do all of these things in a way that's just never been possible before.

19:36 Any other questions? >> Great question in the back. >> Yep. >> Go ahead. >> Can you talk a bit about your data curation process measure cost like that? >> Yeah, so so the question was around uh how do we do cost per PR? So this is >> data curation. >> the the data curation. Um Are you So I just want to make sure we're talking about the same thing. So you're talking about like when I for the the benchmarking or for like our general >> Yeah, for the benchmarking.

20:10 >> Okay, yeah. So the way that we we do this is like you can either build up a benchmark set by hand, which a lot of people will do. They'll you know, and that's kind of what we recommend to start. Uh we will pick like for ours we pick like a set of sort of small medium hard tasks. Whatever you want to be representative. Or you can actually ask the factory to do it cuz the factory is aware of all the past sessions and so and everything is stored in code.

20:35 So it's possible to have the factory itself look at prior sessions and identify interesting ones for a benchmark set which is pretty cool. Yep. >> How is correctness evaluated on this chart? >> Yes, everything in this chart is driven by LLM as a judge which we we may I think another way of doing it is human grading which makes total sense. We just haven't added that yet so I would suggest that as well.

21:00 But basically anything that is defined uh in the system as a scorer so you define these scores which basically grade how well something is doing can be used as a benchmark evaluation criteria. Yep. >> Is this Warp platform is this source or is this only for enterprise? >> So it is is not open source. It's it's like very flexible and like like the idea is to make it open so you can bring any existing tools into it.

21:31 It's not only for enterprise. Like this is for small teams and you can try it out on your personal project. It is best for teams though I will say because like if you're if you're an it's just overkill for an individual. Like you're probably going to want to use like your cloud code sub or something. But once you have a team of like maybe 10 or more engineers having a system like this is very valuable.