Roblox Engineering Director: "Prompt to Prod" Is Broken Without Trust Transcript, AI Summary & Key Points

00:04 myself. I'm Andrew Swedllo. I currently work at Roblox. I support the um engineering acceleration team as well as the core services and platforms group. Uh I'm a manager. Before this, I was at uh Instagram for a couple of years and I was supporting safety, AI, fairness, teen mental well-being and a few other sort of contentious people areas. And then before that I was at Google for my entire adult life for almost 16 years.

00:28 And I worked on uh a lot of different products like um Google Assistant, YouTube, um Android ecosystem, mostly around developer tooling as well as privacy and security. Um and so I've I've worked at some of these larger companies and I've seen how software is developed at scale. Uh and I tried to make this presentation sort of fit uh some of my experiences around transitioning to AI for Roblox.

00:49 Roblox itself is actually a 20-year-old company. Um, and so you we've been around for a long time and we're doing things in a very classic way, but more recently we decided we need to accelerate with the rest of the industry and this presentation is about that. So to start off with, uh, I'd love to get a show of hands here. Who in the last week or two has shipped AI generated code?

01:12 Okay. And keep your hands up. Keep your hands up. Keep your hands up. And how many of you people uh that have your hands up trust that code? Leave it up if you like totally trust that code. 100%. Okay, like three people. Okay, so like that gap is actually the main uh gist of this uh presentation here is that we've gotten really good at generating code, but we haven't gotten great at digesting the code and trusting the code.

01:33 And so um this is the paradox. Um essentially, we've solved typing, but we haven't solved trust. And um this creates a lot of back pressure on engineers. It creates a potential for a lot of sevs. People that are having to wake up at three o'clock in the morning to debug an incident but don't understand how the uh code works is a real problem right now.

01:57 And so um this is the problem that we tried to solve at Roblox. And wee affectionately called our effort u prompt to prod because we wanted to get to a point where you can go purely from a prompt all the way into production without any human intervention. And so a lot of the techniques and tools that we built were to support that effort. And um I think it's probably obvious after listening to all these talks here, I think everyone in the room knows, but you know, we we made this this this strong transition over the

02:25 last I'd say 6 months from really sophisticated autocomplete, which I'd say copilot was at the beginning, um to trying to get to more autonomous software development. And um that's with agents that are running not just to generate a line of code or a file but potentially rearchitect your entire codebase um in a short period of time and build features end to end for you.

02:47 And that's a change a fundamental change in how we need to think about many different aspects of the software development life cycle. And uh part of it I would say is not is not just a uh a change from how we're thinking about what are the best models, what are the best harnesses, um but more fundamentally it's about the infrastructure that's used to build your applications.

03:14 So um just as a quick show of hands here, who is where where are people at in the journey? Is anyone here allowing agents to ship directly into production right now? put your hands up. Okay. And when you do that, do you like do any code review? Like how do you maybe can we get a microphone over here? I just want to ask questions. Um how do you like or are you just yoloing it?

03:38 >> Got it. So So you you have like a set of gates that you're using to try to make sure that the code change doesn't actually like cause a a major a major set. >> Yes. >> Okay. Well, you were the the only person that put your hand up. And so I'd like to kind of point out that there is a pretty big opportunity here because if we're all we're doing is generating code, but we're not actually doing the full endto-end product life cycle, it means that we're never really going to get the full promise of AI.

03:59 It doesn't mean it means that we're not going to be able to, you know, really increase the productivity um in the way that the industry hopes that we are and the valuations of the companies. Um, and so, uh, what we think, uh, at at Roblox is that if you just increase the speed of of autonomy without focusing on safety, you're actually just creating technical debt.

04:21 And what I mean by technical debt is I mean that um there there there will be a lack of people that understand the code, how it works in production. uh there will be security incidents and uh potentially people will have to wake up at 3 o'clock in the morning to answer a pager. And so um the the rest of this talk is going to focus around uh these three areas here.

04:44 Uh the first around is around alignment and guard rails. Um this is how we encapsulate expert judgment and build operational systems. The second is around security and access. This is how we let agents uh run real workflows without compromising the company. And then the third area um is the one that's most nent. It's about how do we rethink measuring productivity in the world of agentic software development.

05:09 And I'd love to have a conversation with you all about that one. Okay. So let's start with alignment and guard rails. Um, just as a uh data point here, most of the LLMs, including state-of-the-art frontier models, we're chained on pabytes of data, which is a lot of data, but actually um if you look at all the data that's locked with inside businesses and corporations right now, you're talking exabytes of data.

05:35 Does anyone know what the factor difference is between a pedabyte and a and a exabyte? It's a million. It's a million. And so it's like a million times more data that's locked within uh within companies than what was trained for the model. So there's no um it's it's not unsurprising that when you try to leverage these state-of-the-art models within your company that they don't always get it right and they make a lot of mistakes because they don't have access to all that data for their training.

06:02 Um and so uh another issue that we ran into when we first started our journey here was that uh we wanted to start allowing ages to be more autonomous. And the first reaction from like folks in in security and throughout the company was no that's too dangerous. We don't want to have a data leak. We don't want to have a sev. Um and so you know no just don't do it.

06:24 But at the same time we're also spending a ton of money on tokens and agents and not seeing the true value of them. We're seeing 10 20 30% you know productivity improvement but not to the extent that we're spending without the autonomy piece. And so most of what we had to build was the trust infrastructure needed to allow these agents to run more freely.

06:44 And um security was probably the largest uh bit of work that we had to do. Um there were lots of fears around prompt injection around agents running with inherited uh permissions um that were acting as you. Um, there was an interesting case from one of my friends who works at a at a frontier lab where he asked uh an agent to do u to work on a juror for him overnight and it ended up um slacking other folks on the team to ask them to merge the PR that the AI generated as my friend um and saying that it should just skip

07:18 the checks. It's okay. Uh because it was trying to be helpful and and and solve the problem. We've seen similar issues as well at Roblox where uh an agent trying to be helpful does something that's actually not great for security. Um and so lots of like very interesting problems to solve and our approach to trying to solve these things were sort of these three main key areas.

07:38 One of them is around sandboxing and uh like maybe just show of hands here. Who here runs their agents in a sandbox? Whoa. Okay. Does that mean that everyone else is not running their agents in a sandbox? Okay. Uh, well, I think I think you all are taking a lot of risk if you're not running in a sandbox of some sort. And a sandbox is pretty configurable.

07:59 U, most of the major frontier labs offer like sandboxes that you can install in your enterprise and like put in place controls. We built our own sandboxes because we wanted an an even higher level of security and safety. But your sandbox is there to protect not only the underlying host and all the files that are on the host, but also, you know, the network access that the agents can do.

08:21 Um, we spent a lot of time thinking about uh policy gateways, making sure that uh, you know, the the agents only have access to what they need when they need it. So, just in time permissions, least uh, privileged um, access, and we also tried to make sure that we didn't have any uh, longunning secrets that could accidentally be leaked by by the agents.

08:42 Um and then finally uh making sure that the uh the agent identities have a separate uh auditable uh annotation from uh from a from a real human. So for example that Slack message, it should be obvious in an audit that uh that Slack was sent by an agent and not by a human in that case. So um uh the next piece is once we saw the fundamental security pieces it was reliability.

09:12 We felt like reliability was secondary because uh security is something that where you lose trust of your customers whereas reliability we have reliability issues all the time. We know how to deal with those. We have a lot of operational runbooks and processes for those. Um but still important to the business because we lose revenue when we have a se.

09:26 And so a a model can be um really great at at solving industry level problems, but it might not know your organization very well. And so we had this idea that the the the the AI could operate best in our environment if it if it um was able to act like our best engineers. And so we tried a bunch of different things at first. We tried to fine-tune a bunch of open source models.

09:52 We tried to do like special uh system prompts that tried to teach the the models to act like our engineers and none of it really worked in the way that we wanted it to for many different reasons. Um and so finally uh we we we we came up with this idea that uh our expert information at Roblox was actually already stored in our codebase. It wasn't at like the top of tree in like the the the head of the codebase.

10:19 It was actually in things like code reviews. So we had this uh idea that if we were able to look at every single code review and we're able to like extract insights from all the different code reviews and then cluster them by similar feedback that we'd be able to automatically pull out uh special rules that we call exemplars that are fundamental pieces of institutional knowledge that are encapsulated in like a YAML file piece of code and that are testable and that are extensible and used by AI to improve future

10:48 operations. Um so we call those exemplars and then we have a system that we use to um to make those exemplars useful in agentic interactions called the alignment engine. It's actually like a react loop. Um and this is sort of like the you know system diagram here where you can see that I got the laser pointer here. Let me try that. Um so we take a look at the historical data from our uh repo and it's actually was about 1.75 review comments u which is about like 7 seven uh 7 700,000 PRs over 3 years and we extracted um

11:23 all of the useful pieces of feedback from those code reviews and then we did a clustering and then we extracted the exemplars and then we actually um at first weren't sure whether or not these would be high quality and so we built a system a UI for people to see um the the exemplars that were extracted from their repo. And so what they saw was they saw a UI where you would have a list of uh of of specific rules that we thought might be important to your repo.

11:51 And then the interesting thing about that is that we found that the ones that people ended up saying were the most useful were the ones that were annotated by the authors that originally provided that piece of feedback. And so we found that like if you showed uh our employees the best engineers and like their exemplars that they'd be most likely to adopt these within their repo.

12:12 We wanted to make it opt in at first so that way we weren't um inadvertently causing friction. Um but being able to like understand like who an expert is, annotate an exemplar with the expert, show it to the repo owners and allow them to adopt it for their repo was really transformative for us. And this is sort of like one of the UIs for how if you wanted to hand author not manually not automatically extract an exemplar you could create one.

12:36 You essentially have like the rule description you have some patterns and then you have uh the uh performance of that exemplar on the codebase um mostly from coming from code review stats because we inject the exemplars in the alignment engine during code review and there's a quick test feature here which is a playground. you can actually give it a PR and say hey run this exemplar on this PR and see if it finds the issue that um that you hope it did and if it doesn't you can kind of tweak or uh improve your exemplar to

13:05 make it more effective and so um exemplars in action like when we first started building our automated code review agent this is a an autonomous standalone single-purpose uh agent for code review um the accept rates were really really low in terms of uh being able to uh the suggestions that that that the coded reagent made. But over time uh what we saw is that uh adding exemplars ended up getting us to it's like almost 60 70% acceptance rates of all the suggestions that were given in code review mostly because of the

13:38 exemplar addition and you know that's pretty great since I don't know if you've seen this or not but even only 55% of human code reviews are accepted. Um so our AI agents are actually producing better results than our human code reviewers are at this point. Um let's see here. So um we'll move on to uh to access. So I I told you before that the name of this presentation was called Promptra.

14:05 And um the inception of the project was pretty simple. Um for those of you that don't know, Roblox is a pretty popular gaming platform. We have 150 million monthly active users that come to the platform. It's really popular with kids. You've probably seen Roblox on your credit card bill as maybe like the first time you've ever seen the words Roblox.

14:25 Um but uh um uh so so so our homepage is like really like the main place where everyone starts on the on the platform. And we had this idea that you should be able to go do you want to come take a look at this? Um is that you should be able to go from from a prompt all the way to a production change with no human intervention. And so it sounds pretty simple but then when we started to unpack that it got pretty complex.

14:48 And so we decided that we should be able to author at least an experiment. The blast radius is pretty small. It's an experiment. You can have a small population. And so um you know that would be like at least the entry point for for this project. But as we started to do that, we unpacked what it takes to actually create an experiment on the homepage.

15:06 And it actually takes multiple weeks. Some of it is because you have to let the experiment bake, but a lot of it is because there's, you know, 18 different human touch points along that journey. And a bunch of these things were inaccessible to agents. None of them had APIs. There was no MCP integration. All these tools were missing the fundamental plumbing to do agentic software development.

15:23 And so big part of the work at the beginning, it wasn't AI, it was essentially plumbing and infrastructure. And it was to try to make sure that we could get to a point where all of these different systems had uh CLIs and APIs and MCP access. U we used uh Playright, which is pretty pretty pretty pretty cool tool um to help like convert your your UIs into CLIs and make them agentic friendly.

15:44 Um, and that took us it was it was about we did a a four-week sprint and we were able to get this full stack of things um plumbed and uh we were able to to to get to the point where where an agent could actually author and experiment and make a change on the homepage. So that part was great. Now the plumbing worked and uh the next piece was around uh safety.

16:07 So making sure that you could actually trust the fact that um you weren't going to break production. And what we found is uh we were missing some of the basic fundamentals. We didn't have, you know, great unit test coverage. We didn't have integration testing. We were still not doing like some staging like deployments for some of these changes. We didn't have like an auto roll back and an auto revert for like these particular changes.

16:28 And so once again, the hard work actually wasn't AI work. It was good infrastructure work. It was about making sure that those things that we've been saying for 20 years in the industry like testing and canary and telemetry and all that stuff was there. Um and we spent a lot of time working through those particular problems. Um and you can see here these are some of the uh different systems that we had to instrument with telemetry and make sure that we had um or instrument with as CLIs and make sure that we had the

16:54 right telemetry for. Um and the other piece here just go back to this other slide was around policy. Um we we ran into like some some pretty serious friction around what can be allowed to be done. And so there was only one person that said that they allow PRs to be automatically merged into production. Um we ran into a similar problem as well when we talked to our our policy and compliance folks and said, "Hey, we would like to, you know, still do a code review, but it's only going to be an AI code review.

17:20 Are you okay with that?" And do you know what their answer was at first? It was no. It was no. It was no. Actually, one of the reasons why we have a lot of the data around um the human effectiveness of code review suggestions and AI's effectiveness of code review suggestions was because we wanted to build some confidence that um making sure that the AI code review that we would replace a human code review was actually as good or better.

17:41 And so um but it was a policy change for us. we had to like rethink some of our underlying assumptions for software engineering at the company um that every code change requires a uh human code review and and and and essentially say that's not that's not going to be true anymore. Is anyone else here thinking about removing human code review? Anyone?

18:00 Okay. I I I I'd love to like know why the people that didn't put their hands up are are not thinking that because I see it as so fundamental. Imagine you're producing 10 times more code. How the heck are you going to review that with the same amount of people? There is no way. There's no way. And and then not only that, but you will have uh not gotten the gains that you need to from from AI.

18:20 And so I really encourage you all to definitely think about the policy pieces as well. We also had other rules around like deployments and when deployments could happen. We would do a production deployment moratorum between certain hours. We had to remove that. We had to get get to the point where we trust AI enough that it can uh build the code, it can review the code, it can ship the code, and it can revert the code.

18:39 And so, uh, this is essentially where we wanted to get to, which is like write, review, fix, approve, deploy, and iterate. And, um, and we tried to like make this as transparent as possible as well. So, we didn't want this to be unstandable by humans. So, we tried hard to make sure that anytime that an agent would author code, would uh, modify code, would do a code review, you could see that action.

19:05 And you can also provide negative and positive feedback signals to the agents to be able to say this was a good change, this was a bad change. And then we take those negative and positive feedback uh pieces and put it into a cycle for self-improvement. The way that we do self-improvement is we make sure that any of the negative feedback is added to our eval set for those agents.

19:23 um as well as we use it for extra context for um some of our rag operations as well as um use it as uh to distill it down to exemplars. Okay. Um let's see. So, okay, we built all these things. We built like sandboxes. We've changed policies. We've um uh fixed a bunch of security things. We're spending a whole bunch of money on tokens. But does it actually matter?

19:49 Is it actually making us more productive? And I'd be curious to know here. Um, who here thinks AI is making you more productive? Put your hands up. I hope it's everyone in the room here. You're at the conference, right? Um, so a lot of people, but but let me choose a random person. How do you know it's making you more productive? Who put their hand up?

20:07 Okay. How do you know it's making you more productive? >> Well, I think now it's >> Got it. So, so you're saying you do you talk to the engineers? How do you know that? That's >> engineer. >> Okay. You're an engineer. So you feel it right now that it used to take you 3 days and now you can ship it in a shorter period of time. >> Used to take three months and now it's like >> Yeah.

20:31 And how do you how do you measure like that besides sort of like the feeling of it? Do you like track your future velocity in some way? >> Okay. How do you how do you do that? >> We have like ticket tracking. >> Ticket tracking. So through Jira kind of kind of style ticket tracking. Great pretty features. What about anyone else have a interesting metric for how they measure productivity?

20:52 Okay. Yes. >> PRs. Okay. Yeah. So, I think like in the past a lot of us would probably look at PRs and cycle time and all that kind of stuff. But the point is is that I think most of those metrics are are failing us at this point because generating code itself and like reviewing code and all those things. Like I said before, if they're all automated, like what are we measuring?

21:11 We're measuring maybe the productivity of the AI, but certainly not the engineers at that point, right? And so um I think most of the metrics that we relied on with space and Dora and stuff like that need to be rethought. And so this is the beginning of like how we're thinking about doing that transition ourselves. Um and uh as you can see here I'm trying to be a little bit contentious by saying things like lines of code and PRs and things like that actually should just kind of go away in terms of productivity

21:37 measurement. I think we need to like reprogram our language to stop talking about those things. Um and uh one for us where we invest a lot of time and energy in is around agent quality. And so when I say agent quality, I'm talking about two different types of agents. Either the sort of like focus single purpose autonomous um agents like a code review agent that you know you build and it sort of like runs indefinitely.

21:58 It's not inherited permissions or anything like that. And then also when we bring in a new harness and a new model, we kind of want to know whether or not that's better than the other one or is this vendor better than the other. And for us to get confidence in any of those decisions, we rely very heavily on eval. And so I'd say, you know, maybe 30 or 40% of all the work that we do is around evals and data.

22:18 It's about making sure that we know how to measure the quality of the agents that we're building, the single purpose agents, as well as um uh the harnesses and models that that we want to spend money on. And so, um, the example here might be is that, uh, we found, you know, a whole bunch of, uh, code in our, um, in one of our repos that didn't have any unit test coverage at all.

22:40 And then we take a state-of-the-art foundation model with a new harness and we run it on and say like, hey, how much unit test coverage can you get without human intervention? And if it can get 60% unit test coverage and the previous model was 50%. and those tests look good, then we actually promote it and say, "Hey, we're going to adopt the new thing."

22:58 If it's a regression, then we don't adopt it. Um, and so eval are super important for all the work that we do. Does everyone here use evals? Does Okay, only one person as well. Okay, you you seem to be like, we should talk afterwards. You But, um, is it So, does everyone here know what eval are? Put your Who doesn't know what an eval is? Okay. So, like an eval is a way for you to be able to say uh look at the quality of something that was generated by AI essentially.

23:25 And so, um because generative AI can be hard. It's not binary, right? Like, you know, if you say, "Hey, generate a unit test for this file." Um you want to know whether or not it's a good unit test. Um and now it might be a really crappy unit test that's testing something that's not that important. Um, and so, uh, an eval gives you the ability to, um, start with a with a set of a set of data to run it through AI and look at the output and gauge whether or not the output is good or bad and then compare that to a

23:54 previous version of the AI. And so, if you're building like a single purpose agent, like a let's say a code review agent, you saw that graph, right, that that I had there. um all of the ways that we knew that we were getting better and better like all the things that we were doing that were making an improvement were all measured through through eval.

24:11 And so definitely if you're not thinking about evaluation should definitely get that on your radar. This is really important. It's a fundamental way in which for you to be able to make sure that the agents are are actually high quality. Um the other one here is feature velocity. So, I was really happy that you said features, by the way, because as much as I would love to be able to tie um our our AI efforts to like topline metrics like R&D or revenue or like those sorts of things, it's really really difficult.

24:37 Um and so the best proxy that I could think of right now is feature velocity. So, the idea is that if you're shipping more features as a company, you're probably going to be um shipping a lot more innovation. And if you're shipping a lot more innovation, then you know you're hopefully going to be a more successful company. But what is a future, right?

24:55 Like I think that's the big question is like in the past we always really struggled with how do you define a feature is like adding um like a new capability to a backend service a feature. Probably it is. Um but it was very hard to like annotate the code and know whether or not there was a feature and if the features are increasing over time. But because we're, you know, living in an agentic world, it's actually gotten a lot easier because now you just ask an LLM to analyze the PRs that are being submitted and say,

25:19 "What kind of PR is this? Is it a feature PR? Is it a uh configuration change? Is it a bug fix? Is it a refactoring?" And guess what? They're pretty darn good at that. They're like actually quite good at being able to determine the intent of a PR and classifying into a category. We actually do a primary and a secondary, but this this chart here is just for primary categories.

25:37 And then so now you have all the features that um you have you have an idea of the different types of PRs you know how many features are coming coming out of your system. Um the question is how do you actually use that and turn it into something useful. For us we look at median number of features per engineer. So we kind of normalize it because actually all the buckets go up with AI right because people are able to do refactoring able to do more configuration changes able to do all of their work faster.

26:02 But the thing that we care most about is actually like the number of features per engineer going up. And so we we mostly look at like a P50 here and we can see like over time 22% in feature velocity in the last you know 6 months or so. And so um this is a way for us to kind of say uh more concretely that investments in AI are actually translating into innovation for the company and hopefully making our product better.

26:28 Okay. Um this is the one where this this metric here is the one where I'm not as confident is the right thing. So I would really love feedback from the group here. Um when I started this talk I talked about how we how we want to do prompt prod but ultimately like our goal is to get to something like 24 by7 AI right so we have AI agents that are continuously working on your behalf even when you've left the office shipping into production safely you know scaling it out by 10 parallel instances doing all that kind of

26:58 stuff right and so for that level of autonomy you have to say well you know you have to give a task to an agent and it has to be able to go and do something big. But what we're seeing, you can actually see in this data snippet here, is actually that the P50 turn time, this is like you talk to your agent, you tell it to do something, is 40 seconds. That's not very autonomous, right?

27:19 That's 40 seconds of work. And so that means that a human needs to attend to that agent, has to sit there every 40 seconds, tell it what to do, right? You can't get to 24 by 7 AI at that at that rate. And so the idea here is we want to look at um what I like to call long-term rate. Um and so um looking at how if you uh if you give a spec to an agent, will it go and actually go and converge on the right output of the spec?

27:46 Will it problem solve by itself? Does it have access to the right tools? And will it converge on something that's good? Because you can also have a longunning uh turn that isn't good. And then did I put the errors in this one? Okay. Yeah. So so you can see here like we also try to filter out like turns that work out in weird ways. So, you know, missing a tool, like, you know, some like network interruption, all these kind of things.

28:07 But our goal is to be able to orchestrate the work in a way that allows people to do these longrunning turns and go for eight hours and get a giant chunk of work done on on their behalf. Um, and so right now you can see even even where we're currently at there, the P99.9 is only 2.1 hours. I mean, that's barely my commute home. And so, not much work is happening while I'm out of the office.

28:29 And so this is our uh main success metric right now for 24 by7 AI. We want to look at this and see uh whether or not we can get to like an 8 hour 8 hour window. And by the way there are some like circuit breakers in the aentic tools themselves that will actually just stop after 4 hours too. So, you know, you have to not just solve like the the the coming up with a good spec, giving it the right tools, putting it in things like Wiggum's loops or like those sorts of things, but you have to like also make sure that the

28:54 tools are configured in a way that allow you to burn those kind of tokens overnight. And um and so I think like if you kind of combine all those things together, this is what's what we're trying to do to get to 24 by7 AI. It's about the um the alignment guard rails. um making sure that you uh you have institutional knowledge that's accessible by your agents.

29:17 Um it's about making sure that you put in place like the right security foundations, sandboxes, access to all the tools, and that you're also measuring the things that matter the most. And and I would definitely keep away from just measuring lines of code in PR because it is trivial or even less than trivial to game that at this point. And so, you know, please like rethink rethink some of those some of those metrics.

29:39 And I think with that uh we're going to go on to Q&A. >> Thank you for the lovely presentation. You got me thinking some some stuff uh especially about the autonomy. I was thinking differently in terms of human intervention rate and uh one thing that I instead of agent agent autonomy and stuff like that right because the number of interruptions that you have it will get the people thinking between tasks so it's context shifting right and that takes like 14 minutes for the human to shift context which is >> I think like

30:20 there's a there's a term for this that's emerging, right? It's the orchestration tax, right? And so like like if you if every 15 minutes you're getting interrupted by an agentic workflow because you have to resteer it in a different direction or answer like a clarifying question or deal with, you know, an error, um you're paying a tax. Um and I don't think that's where we want to be.

30:38 So u most of the our our goal is to try to try to avoid those interruptions. And uh the question over that comment is how you're measuring the interruptions you know because I I was >> so in in that uh long running turn time th like those are single turns no interruptions >> but autonomy wise you have certain measurement and how like the autonomy wise you have to measure how autonomous is what is the actual metric and how you're capturing it.

31:10 Oh. Uh, I like maybe I don't quite understand, but um I was thinking about it in terms of like the length of the turn itself. So a turn is like every sort of prompt that you do to the agent like how long will it go before it successfully completes the the the task that you assign to it. And so if you're doing spec driven development, if you give it the spec, you put it in like a like a wigs loop, then how long will it go before it actually, you know, stops either because of an error or anything like that.

31:36 So, so our goal is to like get it to like to to take a long period of time but also su successfully complete. >> Thank you very much. >> Yeah, no worries. Okay, next question. Okay, let's let's let's go down here at the front. Yes. So, >> Eric will get it. >> Come down here. I like how many hands went up. This is great. Okay, I'm glad that I spoke quickly and got through the slides because I prefer the conversations actually.

32:01 >> Thanks. Um so I I I know you mentioned like in order to get the agent truly u truly automatist uh you basically have to do a lot of plumbing work for the existing code like setting up MCP and stuff. What kind of tools did you use and um how long did it take for your company to do that? >> Yeah. So so it is interesting right? Um one of the pods that I I support is called the the AI pod but actually almost none of the work that they're doing is AI.

32:27 It's actually all like infrastructure work. It's, you know, adding the the the CLIs. It's like increasing test coverage. It's, you know, providing extracting and providing context to the agents. Um, I wouldn't call any of that like the traditional context engineering AI work that we've seen in other other previous um uh presentations. And so we were we were given a very ambitious target by our CEO.

32:50 So this is about 3 months ago. He said to us like, "I want you to be able to unlock, you know, full AI for everyone at the company, like autonomous workflows in 6 weeks." And so we sort of did like this companywide code ride. We could bring in anyone that we wanted to to like work on on on this on this exercise. And so within six weeks, we went from I would say like more casual use of AI to every single person in the company using AI tools and shipping into production.

33:17 Um, and a lot of that was the the precursor work for that was to make sure that they their tools all worked with their agents. And so about six weeks um a wide variety of different types of technologies. There's not one specific type of technology, but we did use Playright for example was um you know something that we leaned on to help get us to CLI capabilities faster.

33:35 Um I probably shouldn't be saying vendor names and stuff like that, but we try we try almost all vendors. We have a rapid eval process which is a way for us to like quickly onboard and try new state-of-the-art tools and see whether or not we should add them to our tool chest or or build our own or um or use a combination. And so that that's also another fundamental problem that I've seen in the industry which is it takes too long for people to get on boarded into new tools.

34:01 I know some folks that like still don't use any of the frontier lab technologies because they haven't been able to get security approval, budget approval, any of these things. And I believe there's going to be a Blockbuster and Netflix moment in the industry right now with that that the companies that fully adopt AI are going to be like Netflix and and and really excel and the ones that uh don't will become like Blockbuster and become out of date pretty quickly.

34:24 >> Do you mind I ask one more question? >> Okay, sure. One more. >> Uh you mentioned the evolve framework, right? Like how what's your recommendation for coming up with Evolve framework? >> Yeah, I I really shouldn't be like recommending vendors. My my com team told me not to recommend vendors. Uh there are many out there that that you can look at. Um but we actually built our own eval uh framework and harness to to start with as well because there wasn't anything that was great with coding like a year and a half ago.

34:46 There are some like better things out there now. Um you can look at like I don't know weights and biases and things like that. We don't use them but like you know they're they're pretty popular with people. They have some sophisticated tools. Um cool. I'd also uh recommend investing in observability for your for your agents as well. Um so you know using OTL to collect tracing data to be able to debug all your agents like that is like one of the most important things that we invested in as well.

35:08 Okay. Uh let's go to the different part of the room. So okay you we have a question there. Okay. Yes. >> Hey uh thank you for the talk. I'm still very skeptical about you know shipping everything all the way to products AI but you know I won't trust you. I trust you with all the beautiful graph. However I think you know in reality problem happens right?

35:26 You still can have issues right? uh remediation have a good remediation plan is very important right I'm I want to understand did you ever have any uh problem with shipping all the way to a product causing problem right troubles losing datas how do you remediate right how do you also make sure uh same thing won't happen uh in the future or if things happen again right how do you uh minimize impact in your system right >> yeah I I think I think maybe like uh uh even higher level question is like what is the role of like

35:56 humans in this in this new world, right? Like like I you know I try to write some code myself still and um when I see like the volume of of changes that are happening even within like the repos that I'm working in, it's it's more than any human can really like gro and and and review. Um and so so I believe that uh most engineers will not actually understand most of the code in their codebase moving forward that that it will be kind of abstracted away from them and that they will live above this like AI abstraction

36:24 layer. And so like to make that actually work and deal with like problems in SES, we need to like also have like AI abstractions within production. So you know if you're not investing in agents that can help you debug production incidents, then you should probably be working on that because that's certainly it's sort of like the order of operations is coding agents, code review agents, production agents, right?

36:45 And like if you bring all those things together, then that's when you can start to, you know, really kind of unlock the full agentic life cycle. Um and and I I I agree it is scary to like say that that that we will um you know allow AI to do these these these acts on our behalf and we don't trust AI for everything. For us you know it was a bit of a maturity process.

37:04 It was saying like hey do we have all the right plumbing in place? Do do the agents have access to the right tools? Do we have the right policies? Um do we have the right graduated promotion like technologies? Does it go through canary? Does it go through staging? Does it have integration tests? Can it can the agent do a roll back if it detects a problem and a revert?

37:23 And if the answer is yes to all those things, then you might as well try experimenting on some of the lowrisk scenarios. So we also started with um trying to use AI to identify low-risk PRs to automerge, right? Because you can actually say from an LLM, does a blast release of this look like it could be something that could take down production and they're actually, you know, do a pretty good risk assessment.

37:43 Not always, right? But sometimes. Okay, how about how about over here? Hi, >> if you've managed uh you hear me go >> if you've managed automation from coding through review to production feels like the bottleneck's going to move forward then to product. >> Yes. >> Are you thinking about how to validate product requirements? >> I I think I think it's a I think it's a good question.

38:10 We haven't hit that bottleneck yet. I think that human imagination is unbounded and like with like AI writing tools like PMs can still write PRDs but even super fast now. Um but I I do think you know certainly you know the the if you compress 5 years worth of road map into one year you need to like write 5 years worth of road map and so that that does take time.

38:28 Um my hypothesis is this is that uh the it used to be expensive to do stuff and you you wanted to get it right and so you would do a lot of user research and like customer interviews like all these kind of things to try to make sure that you don't ask your engineers to do something that's going to be thrown away. But now like the building part is becoming a lot faster and easier and the deployment part is becoming a lot faster and easier and so like maybe the the the the barrier for coming up with ideas and making sure

39:00 the right ideas is not as not as as hard anymore and that you can actually just pump out more things and your agents will also tell you through your experiments which things are the most effective and your product will self iterate that way. And I think that's where you'll see a lot of value is like if you can like take the loop from, you know, generating code, testing the code, building the code, deploying the code, but actually like full-on experimentation, iteration on experiment var uh variance, then you'll see a

39:22 lot of a lot of people moving faster. But there might be a new way of thinking about product management and uh you know coming up with the right features and that kind of stuff. So no no answer for that one quite yet. Yes, right beside you actually there's a person since you have the microphone there. How are you solving engineers continuing to build a theory around the problem and the solution of the problem?

39:43 Like are you are you reaching to pair programming? Are you reaching to other kind of things that are happening before PR review? >> Sorry, tell tell me more what you're thinking. I I didn't quite understand that. Uh so the theory building the mental model we keep in our head of the system the domain of the system and the how we that this problem is solved and continue to keep that mental model as we develop it over time.

40:07 >> Yeah. Well you know I I spent a lot of time working at Google uh and is there any Google people here at all? Google love design docs. That was like the thing when I first started working there. We all love design docs. We spent a lot of time on design docs and and the idea was like you know you you would you would create this design doc. you get a lot of feedback on it.

40:23 It was a way to kind of like build that mental model to get feedback, all that sort of thing. Um, but I'm going to argue and say design docs are dead and that you can you can build a prototype as fast as you can write a design doc now. And so, you know, what's better than like a prototype? You can touch it. You can feel it. You can see whether or not it's the right thing.

40:41 You can inspect the code. You can look at the architecture. You can do all that kind of stuff. And so, um, I think, you know, that that that ability is sort of unlocking a a higher fidelity shared understanding of, you know, the things that you want to build because you're making it very concrete. You're kind of showing with an example. And so, um, I see a lot more of that happening right now that instead of like even a junior engineer coming with, you know, a design doc and iterating for a long time and trying to get

41:04 that shared consensus, they just come and show us what they're what they think they should build and then we're like, "Yeah, that looks good. Ship it." So, a little bit of a of a of a contentious take there. >> So, >> okay. Uh, yes. Yes. >> I'm okay. I got a question. It was It's So, I had a I had a similar question to the previous one, but I have another one.

41:22 >> Okay. >> My experience has been when I'm say vibe coding for a while. It's working great. Code's going nuts. I'm I'm adding features left and right. And then after a while, it's like, "Oh, you have to fix that bug. Oh, wait. That created another bug. Oh, that created another bug." And then I go and I look at the code and I'm like, well, no wonder this is a complete to utter mess.

41:41 You know, it's everything is repeated everywhere. No wonder it can't stay in a steady state. So then I go rewrite it or rearchitect it a little bit and then I'm back again, you know, for a while until again it's messed up all the state. And I'm wondering if you've experienced anything like that, if uh if you have any mechanisms to try to detect and pay down and refactor tech debt and so on.

42:05 >> Yeah. Um I think I think it's interesting. I feel like I see more of the whack-a-ole style, like you you try to fix a bug and you create a new one. Um, with more of like the vibe coding kind of like prompt this, do that sort of style of of of AI software development. But if you're doing more like spec driven development where you're kind of talking about like APIs and contract boundaries and like convergence criteria and all these sorts of things then I find that like the code is not quite as messy and that you know

42:33 you have some semblance of order rather than sort of like you know things changing all the time and um bolting on like new features and that kind of stuff. >> Well then I have a quick followup which is yeah how are what what format are you using for your specs? What does spec driven development mean to you? U well it changes on a week toeek basis uh and team to team it's very personal but it's it's the difference between sort of specifying like us like I want to build a specific feature to talking more about like the

42:59 properties of the systems that you're trying to build and the constraints around it. So more of like the the requirements and the and the boundaries and the architecture of the system. And these can be pretty large documents like we have like you know giant like specs that are multi-pages and have to be chunked up sometimes to process. Um and so uh I don't have a perfect answer for that right now but what I've seen is that the people that are just saying like fix this bug or like add this feature like that's where we

43:24 see more of the whack-a-ole and sort of the um u less sophisticated um agentic software development. Okay, we've got I think five minutes left. So, okay, five minutes left. Okay, let's go in. Should we go in the middle here? Oh, sorry. This gentleman here has had his hand up. So, yeah. >> All right. Uh, thank you. Uh, so, uh, I'm from medical device company, so I have to admit that we deliver zero code, you know, using AI for half years.

43:50 >> Zero code using AI. Okay. >> Yes. Because we have to go through >> just brain coded. >> Yes. Uh, so a couple question. One thing is you're talking about the uh code review that accept rate for humans 55% and the machine you AI >> turns out that humans aren't always that great either right >> yeah so how do you determine this metric because when we do the code review it's not necessarily rejected or accepted sometime it's just a suggestion or it's good and then follow-up discussion >> we we we look at like just the

44:19 basic uh accept rates within GitHub so whether or not we see if if someone made a made a suggestion and then whether or not the diff against the merge code was the that that the suggestion was actually accepted. So we just do it programmatically by looking at the the comments and the GitHub uh diff to what gets merged into production. >> I see another one how you train your team to get on the same level similar level using the AI because different people have different acceptance for the AI.

44:47 I I think like the um we we sort of have this other metric I didn't put it on here but it's sort of like AI intensity um and it's about like how much you use AI and what you use AI for and so you know we do look at at that amongst engineers and we do see like a broad spectrum for you know some people although not many people not adopting at all a large cohort in the middle that are sort of like what we call like casual like AI users they kind of use it on a day-to-day basis but like you know not to do all of their work

45:11 and then we have like our highintensity AI users um the high-intensity ones are just self motivated and have like learned to do this by themselves. And then the middle cohort is sort of adopted because they see everyone else doing it, but they haven't quite learned how to be highly effective 10x engineers like the top cohort. And then we have other folks that are just like, I don't trust it.

45:30 I don't trust it. I don't want to see it. I don't want to touch it. Um and uh we're not really sure like how that's going to play out in in the industry. But as a manager, when I'm doing um calibrations, you you're starting to see the difference between people that like accept and use these technologies versus people that don't, right? Just just based off of like the output.

45:48 Like this person was able to get through this many road map items and someone at the same level only got through a fraction. What's the difference there? Right? And how do I as a manager start to like calibrate and and and and promote those people? That's still an open question right now. And um I don't have an answer for that. We do try to do a lot of info sharing sessions.

46:04 Um, there are ways for people to kind of like share like their prompt history and show how they've been building things. Um, we do like lots of lunch and learns, all that kind of stuff to get people excited. But I think right now people are more excited than not excited. Like I actually get like hate mail from people at the company like why haven't you brought in this tool yet?

46:24 And I'm like, oh my god, it just came out two weeks ago. Give me give me some time. Um, so yeah, it's a different Okay, maybe one more question. Okay. Go over here. Yes. Okay. I haven't been to this side of the stage yet. So, hello. >> This is the best side of the stage. >> Yeah, I agree. >> And uh so you guys do no human looks at the code prompt straight to production right now at Roblox.

46:50 >> Not for everything. This is like a small subset of our production changes right now. Yes. Yes. We we like the homepage changes and experimentation like that's like been the main focus area and we have like eight or nine other like development workflows that we've been automating. >> Sure. So the followup is uh so code review is initially based on you know what is being accepted in GitHub.

47:08 Uh at the moment a lot of our code review is done by AI. Eventually all the code review that the AI is looking at is going to be done by AI. How do you retain confidence in the code review once the human is out of the loop? >> Yeah, I'm I'm not sure the code review matters is is is like you know like in the way that it did before. It's it's still an important uh validation checkpoint to make sure that nothing breaks going into production.

47:33 But um you know the point of it I think before there was like two main reasons why you do a code review. One is to make sure they don't ship like a bug. One is to make sure that the uh the institutional patterns are upheld within the company and the third one is probably just like for education and knowledge sharing. Right? Those are like the main reasons why you might want to do a code review.

47:50 And um and certainly from the like the quality standpoint and like the adherence to like a standard, I don't think those things are going to matter as much and that an AI can do as good or better than a human for those the making sure that everyone understands the code and like that we have institutional knowledge and share that knowledge. I think that's a different problem that needs to be solved and it's uh it's I think it is going to be a problem. Okay, I guess that's it. Thanks everyone.