OpenAI’s Performance Strategy for 900M ChatGPT Users Transcript, AI Summary & Key Points

00:03 [music] So today I want to tell you guys a story about two accelerations that are happening somewhat at the same time. Uh first run is around growth. Uh how we actually growing user base faster than ever not just openai chatpt but in general companies are reaching their first few million users a lot faster than before. Um the second one is how our development workflows are changing given you know aentic coding and all those things and how that is changing the rate of shipping of things we're getting out of the door and

00:42 how that is actually affecting how we um keep our products fast efficient so overall performance engineering. So my name is Martin. I I lead the chatt performance team at OpenAI. Uh my whole technical career or almost my whole technical career was in performance engineering. I think I was doing that for I don't know 16 years or so or more. The bulk of that time was at Netflix uh where I spent I think almost 9 years.

01:09 Uh Dio mentioned the early cloud days. That's around when I joined when people thought will I actually leave my my data in the cloud that's like magical thing there so it feels like the moment is is similar today uh I spent some time at snowflake expedia and a few other companies doing that as well so prior to openai I was with another AI company called parasale we did inference as a service I was leading the whole engineering team there and prior to that I was at a fintech out of Brazil called Pikay uh is leading

01:43 whole infrastructure platform uh and developer experience. As you can imagine, uh my whole career is really in this central organizations. This an area I really like is I call it the plumbing of the house and no one sees that it's there, but once it it blocks and it breaks, you know, all hell breaks loose. So, it's a always a fun part of the company to be at.

02:06 So, three things changed. Uh the first one is is growth pace. Uh not just openAI like I mentioned companies are reaching their first million, first 10 million, first 100 million, maybe more um users a lot faster than before. The curves are like exponential but a lot faster than before. Um also we shipping more code and more things out. We're getting more things out of the door.

02:35 uh with aentic development there's probably some talk here around um you know AI slop and everything that's going out so we are shipping more logic out the door um but the third thing is previously um we always had this assumption that there was a human that understood all the changes that were going out like someone actually coded that and you know architected it and know what's going on before they decide to push something out um that it's not entirely true anymore.

03:10 Um at least from my point of view today. I know it's I might be a slight bubble but uh the abstraction layer is a bit higher. Uh people are delegating more things to the agent. So they don't know the details of everything that might be going out the door. So it's something we need to keep in mind. Um and of course with with all of that it puts pressure not just on performance.

03:31 performance is one of those uh uh problems but um everything else that goes around it uh anyone from you know GitHub GitLab here or anything like that I'm really sorry for you guys it's it's putting a lot of pressure in that infrastructure um but performance too I we're just using more and more resources um we adding logic that adds latency so introduce a lot of challenges and we need to keep up we need to evolve our practices so we can keep up with that you know fast pace of growth and the fast uh pace of development

04:06 going into chatt a bit of the history just to give you a bit of context about the scale I'm talking about uh chattyt was not launched in 2022 late 2022 if I remember correctly as a full-fledged consumerbased application that was built from scratch to support hundreds of millions of users it was launched as research preview. Here it is. It's pretty cool.

04:32 You guys should try it out. Um and quickly users started coming and once you have those user coming and coming pretty fast, uh that research preview becomes a product and product has requirements. You have latency requirements, you have reliability requirements, you have support requirements and everything that goes uh uh behind the scenes to make that work for a large volume of users, especially if they're relying on that for their day-to-day on the things that they u need to do.

05:08 The growth was quite impressive. Um I think very few companies uh can tell that they reached their first million users in 5 days uh from launch in a research preview. Oh, this is pretty cool. Oops, we have a million users here. You can imagine how um behind the scenes the team supporting the infrastructure were um at those times. U incredible growth, incredible for the product, lot of pressure on everyone behind the scenes.

05:40 But that growth, it did not stop. It did not slow down much. Um, Chatt continued to grow and and it's still growing pretty fast. The last official number I can share is from February this year, I believe, when we reach 900 million weekly active users. That's 900 million people using ChattP every week. just to let that sink in. It's that's almost 11% of the whole human population.

06:08 It's a lot of people and as you can imagine chat it's not the lightest um u thing to do compute wise. So there's a lot that is necessary on a back to make that happen. Um that growth was not smooth. It was not just adding a few million users every day and kids getting pretty predictable. It was full of spikes as you can imagine. We were launching products all the time.

06:40 Some of those products became quite viral. So huge spikes on on users uh and that of course drove adoption. Uh just this one you you might remember the image generation spike from last year. Maybe you generated one of those images that I cannot say the name right now. Um but within just the seven first days 700 million over 700 million images were generated by over 130 million users.

07:06 So image generation pretty quick and I mean that was putting a lot of pressure in the teams behind the scenes. So um scrambling this is the sort of thing is quite hard to predict when something like that becomes um hugely popular. But again it's something we want. Everyone wants their product to grow and this is really great. Um chat is not a it's not used only in the US it's used all across the globe.

07:38 uh and with that I mean if you support infrastructure architecture all those things you know that adds complexity to things like where are my pups where are my actual uh application servers in our case where are GPUs are they closely together do we have enough capacity in all those regions um or I don't know how is the network connectivity between all those different data centers uh the architecture where is data how is data being replicated where is data located giving all the different constraints you have across the

08:12 globe. So it adds a lot of complexity to the architecture um to support a good experience or to be able to do and and provide a really good experience to all our users across the globe. Um it was not just growth um but the product did not stand still either. uh it wasn't just a lounge or just users joining. Um there was a lot of change beyond all just the model changes.

08:37 The models was extremely important but there were a lot of launches along those four years less than four years now. Um there were GPTs that were voice there was agents um all different things image generation a lot of product launches some a lot more viral than others. So the product was changing constantly. So a lot more things different workloads that had to be supported.

09:06 Uh more complex architectures behind it to support all of that. So the product was changing while it was growing really fast as well. Um and this is I wouldn't call it new uh but it definitely accelerated more since late last year. um especially since the launch of Codex our shipping rate changed dramatically I mean dramatically the I'm not discussing value or anything like that that's a way deeper discussion but just the volume of change entering our systems increase a lot um when everyone thinks about agentic

09:53 development the first thing that comes to mind is you know automating the proper coding task like you know develop this function or that application or refactor this, refactor that um and reduce the overall time it takes to get that done. So shorten the time to get from an idea to a PR that gets merged to production. But the part that um really surprised me is not just that um especially with newer models where the agents are a lot more capable, they can take more complex tasks, we can delegate things that will run for

10:30 a lot longer. Um developers started going multi-threaded. They started actually not working on a single thing at a time. Uh developers are working with maybe seven, eight, nine, 10 different things at the same time. uh and they keep on blocking them as things go. Uh I don't know if how how much you're coding these days with agents like you're probably hearing the pings saying you know I you know I need the user input right now.

10:55 Um they have multiple development boxes or maybe you're working with workre but in in general a lot more parallel things going on. So it's not just faster to ship but shipping more things in parallel. Some of you might be familiar. Oh, I have a laser thingy. There we go. Cool. Um, some of you might be familiar with this uh with this bench from DX uh where basically they study a few companies, got some numbers about the volume of PRs that get shipped merge uh per engineer per week.

11:30 Uh and then you you split that into you know different size of companies and uh the type of company as you can expect you know smaller tech companies um are shipping more PRs they're getting more things out but at PK your P90 there you're almost at five PRs a week per engineer which is quite fast I would say but we are a lot faster than that uh we're shipping a lot more these days um unfortunately the the latest number I can share is from late last year.

12:02 So October, November and you if you've been following you know there was quite a huge development since late last year uh in coding agents but even back then back in October 25 um the volume of PRs that our engineers are shipping every week increased by 70%. Um almost every engineer at the company today um is using codeex on a weekly or probably a daily basis.

12:29 um every single PR gets automatically reviewed um by by Codex. Um and it's not just the coding tasks. Uh like I mentioned, Codex became the workbench to do everything you need to do during that whole development workflow. So it's not just coding, but um I found a bug in production, go troubleshoot it. or I want to understand a metric a regression go talk with the observability tools and fetch the metrics and explaining what's going on or to do um you know data science and data analysis I use codeex to do that as well

13:10 um and all the the other boring things like please summarize that slack thread for me because I don't want to read it all um a lot of productivity textes just make your day-to-day a lot um a lot easier and and not just engineers I I manage the team I probably don't spend too much time coding these days anymore but um I use codec for everything um to prepare my 101s to summarize and write documents and even this presentation um I started it on codeex to go over the outline and to discuss what I actually wanted to to

13:44 present and how to slice that and all the images that got generated all the visuals in this presentation so it's being used for everything on your day-to-day not just your pure pure coding tasks >> [snorts] >> All right. Um, and with that, I would say evolving of the software engineering workflow, we're shipping things faster. It takes less things to takes less time for things to get out.

14:12 Uh, we're working on more things in parallel. We're delegating more. Um, that agentic development increased throughput and parallelism. And that increased the volume of changes going into our system. I'm not saying value feature I think just changes things that are going out. It might might be a feature might be a bug fix might be a refactor might be something else but just the overall might be a config thing but changes going out.

14:42 Um but shipping all those things, every new feature, every new line of code, it it has a hidden cost. Um ideally that hidden cost um adds value. Uh it's something you want. It's something you want out. U it's a new feature, a bug fix. Uh but it has a cost somewhere. So the cost might be tiny initially, it might be invisible. You might not it might be you might not notice it in the first place.

15:17 U and that increased rate of change just exacerbates that. It's nothing new but it's happening bit bit faster. Um and all those different all that change going out all those new extra if conditional statements every extra network request you're making every data structure that you decide to hold in memory. Um that's all consuming a is consuming from a shared budget.

15:50 um your latency it can get so bad to certain extent until your users decide to stop. So if you're adding logic you're doing more things you're adding more checks whatever you're doing that always adds a little bit of lazy just because you're doing a bit more and it just expands to the point where hey like this experience is not good and I don't want to use your product anymore.

16:13 Same goes for hydro resources. You only have a finite amount of memory. you only have a finite amount of RAM and yes everyone will raise oh cloud can expand forever no it cannot uh uh it's not as elastic as you think um and just makes things a bit bit less effic so it's sh it's consuming from a shared budget and I'm not talking purely about those big regressions like things that you clearly see um from uh one deploy before and after that something drastic changed I'm talking about even small things like I said that

16:50 small extra if statement that you added to check something that it's important. Small things that on a database they don't make a difference. It's no difference at all when you um comparing before and after. But all of that compounds uh you keep building up a little bit little bit little bit uh until it becomes a bigger problem. It becomes a a a concern.

17:13 uh you might be running out of headroom. You might be, you know, running well making your latency is too bad for users to have a pretty good experience in the app, but it it just compounds and with that increased rate of change, it's just compounding a lot faster. So, we shortering that time window until we have to actually start worrying about those things.

17:37 Um and that damage from all those small changes and sometimes big regressions it it will show up later. Um for the users it will show up in uh slowness in the app itself as you can imagine. Uh for efficiency no you will be paying more to run exactly the same workload. You add more logs you do more things you still have the same workload you're just paying more for it.

18:05 for a scalability uh you have less and less headroom to grow and as you guys probably can imagine uh a scalability problem uh will become a reliability problem at some point when you cannot absorb that spike anymore when you cannot grow anymore so the damage will appear later so you need to address it um as fast as possible this is nothing new um I mean we've been developing features and adding features and adding logic our whole careers.

18:35 Um nothing new. Uh but in one side you have people shipping features and shipping things and on the other side you might have a performance engineering team or you might have a few perfminded individuals that care about efficiency and care about latency and making things as efficient as possible. Balancing that so you have two opposing forces and keeping things in balance.

18:59 The problem is that with that increased rate of change, uh, one side of the equation got a lot faster. So, we're shipping more and Perf engineering, uh, or folks working in performance problems, they need to adapt. Um, we need to keep up. All right. Um, of course, the the answer to that, that one was open. That's not mine. Uh, I don't know. Maybe I put something there.

19:30 Um the answer to that is of course we don't want to hire more performance engineers. Uh it's uh it's hard to find Perf engineers in the first place. Uh but again not not the best answer. We need to make our uh to make our workflows faster as well. We need to speed up the cycle on the Perf engineering side too. We need to speed up our reactive flows, our reactive loops where we detect things a lot faster.

20:03 We can u profile that and we can get to the root cause of the problem faster. But not just that um also understand have the agents understand the codebase uh propose a solution implement that thing and get it out and get benchmark and get tested. Uh so when we actually get a notification, hey great I already have a fix for that um the benchmarks are looking good again we can continue.

20:30 Um also on the active side we can have a lot of agents in parallel looking for possible optimizations. We can have a lot of different skills um working on different types of optimizations. You can be looking at the latency hot path. You can be looking at uh what methods are actually taking most CPU and try to optimize those. Uh you can be looking at allocations and continuously improve that or bundle size.

20:58 Um they all specialize looking at different things but working non-stop trying to find those um those optimizations trying uh so you don't have to do that in serial. Um and great we need to adapt. we need to move a lot faster. What happens if we don't if we don't adapt as fast as we need? Um, of course, there'll be a lot of chaos behind the scenes, but beyond that, uh, what's the first few things your users will feel?

21:33 Um, first things are generally snowless. Things you add logic, things get slow until it becomes a problem. your users stop coming, they stop using the app, they cancel their plans, or even worse, maybe got bad to a point where you're just throwing more errors, you're down for longer periods of time, things take longer to get fixed, but your users are suffering and um for especially for a large consumerbased application like like chat and it might be the case for you know where you guys work um you don't want that that

22:07 has a deep effect in the business bottom line. So performance affects user retention and acquisition. I model that at Netflix. I model that at OpenAI. Um I'm pretty sure if you guys try to model that as well in your companies, you'll see the correlation between things getting slow and you're throwing more errors and users just cancelling plans. So uh direct impact in a business.

22:32 Um, one thing I've noticed too, especially for um, AI applications is everyone is very very focused on inference. Like inference is the the hard part. Inference is the uh, the resources really to watch all the time. So they're really focusing on uh inference components like uh your time to first token or uh your tokens per second, your throughput uh which are important.

23:01 Your your component metrics are super important. In inference is a big chunk of of the problem, but it's not the whole journey. It's not the whole problem. Uh we need to measure and keep watching what the users actually feel. uh when the users come to your product they generally trying to solve a problem they're trying to do something perform an action and during that process um while they're trying to perform that action they have different feelings they have different expectations and we need to be able to capture

23:37 that and then we need to drill down into uh take for example this this is bit more focus on on a chat app but the idea is the same I use the same thing on Netflix I think Chrome uses is the same thing as well. The different the users had different expectations along the action journey I would say. So when he types in a message and clicks enter or clicks submit um he generally expects something to come back really quickly instant just a feedback to that action just to know that I'm not hanging here or anything.

24:11 Great I got a feedback I might got a spinner. I might got a thinking thing going on there but something is going on. it's not on me anymore. You know, the the product itself, it's working. I just need to wait a little bit. Uh then uh comes something I call first visible value. When can the user continue doing what he wanted to do in the first place?

24:34 And in our case is if you have a if it's a chat request, um the user wants to start reading that answer. uh and so the time it takes to get the first few tokens and the user can start reading that um and you don't want the users to stop reading because the the throughput the stream cadence is slow. So you need to make sure that the stream cadence is pretty good so the user can continue reading without getting blocked and frustrated again.

25:04 uh and then the time it takes to finish that message, the time it takes to uh uh complete the task itself. [gasps] Even that has a lot of ws. Uh where do you stop the timer specifically? Um for uh a chat response, do you stop at the first token? Do you stop at the first few words? When can the user actually extract some value? It's, you know, depends on the product.

25:27 Uh when you have a thinking response, where do you stop the timer? Do you stop at the thinking tokens or do you actually stop when the user can the actual final message in an image? Do you stop the timer when the final miss the final image is generated or the first low resolution image is generated? It's all there's a lot of new answer. There's a lot of thinking you need to do around product to actually have those numbers right.

25:52 Um, having those intents is super important as a dimension as well because the expectations they also vary depend on what the user is trying to do. He might be asking a very stupid not stupid sorry a very simple question uh like what's the capital of France? Uh and well I can go to a search engine and get a response pretty quickly. So my expectation is that the response should be pretty quick to sell.

26:17 Um, great. Uh, so I need to respond faster on those cases. Hey, I actually fired up a gentic loop uh that needs to go and check a lot of sources. It needs to do some code edits. You need to do a lot of things. So, I expect that to take a bit longer. So, the expectation for that is longer as well. Image generation heavier. The same thing. Users have different expectations.

26:41 Cool. I have all those different timers, those different wall times, different intents, a lot of different dimensions. Now to be able to act on that then I go and decompose that into u system evidence into specific components I want to track things that compose that large wall time. Um I like to use this sort of chart. There's a lot of different options.

27:05 My team likes that as well. So your latency breakdown chart or layer cake where you have your distribution on the x-axis and uh actual latency on the y-axis and the different layers depending on what's going on. The chart is not important part here but um there's a lot that happens uh beyond inference. Of course this is not to scale numbers or anything.

27:29 It's just demonstrate. uh but yeah there is client work there is networking time there is time like uh to fetch data to do serialization to do tokenization there's a lot of that goes on beyond um just inference and that breakdown into layers it help us actually focus our efforts and also the intent dimension help us focus our efforts if we see that the large chunk of time is not inference itself is maybe data fetching we can focus on that and try to improve that.

28:05 So having visibility the whole path it's super important otherwise you just default to inference uh which is the first thing that comes to mind to everyone. Uh inference is just part of the problem as you can imagine. Uh take for example our chat application. Chat is very simple. The interface is simple on purpose. You go, you type your message, you send that message, you wait for a bit, you get the response back.

28:36 Um, but I don't know if many of you developed this sort of system before, but we can't just forward that message to the inference engine. It does not keep that state of your whole conversation and everything that needs to um it needs to properly um respond to that. Um there's a lot that happens before um a message gets sent to the inference engine. So there's client work on of course um assembling everything that needs to be assembled at the client side.

29:05 Uh u fixing things there once things come back. We need to verify the user identity. We need to verify the client identity. Uh we need to check the user's plan. You need we need to check quotas. We need to check the actual user state. There's a lot of that needs to happen. Then it comes a fairly heavy part of it which is we need to fetch context. uh conversations they can get pretty long.

29:31 We need to fetch all that conversation history. Um the user might have uploaded files like PDFs, images, whatever um to that conversation, we need to fetch those files as well. U we if you are in a project, we need to fetch that project context as well. So a lot of data that needs to be fetched until we start assembling what actually gets sent to the inference engine.

29:54 Great. I have all the data. I have everything I need. Um there's encoding uh there's tokenization and of course every module is like different tokenization. Um great everything is tokenized. We need to look at the context window like is it exceeding the context window the user has like maybe we need to do truncation maybe we need to do some compaction.

30:13 Then we need assembled uh everything uh to be sent to the inference engine. So there's a lot that goes on even before something gets to the inference engine. And then there's a lot that goes on after like we're streaming the message back because we need to store the conversation somewhere as well later too. The message as you can expect what the user sent that maybe a small string a small question is just one small ingredient of that whole um that whole equation and especially for long conversations um they are pretty

30:49 data heavy. You might have conversations that are megabytes long, tens of megabytes long, maybe hundreds of megabytes long. Uh, and there's a lot of requests. So, there's a lot of data moving around our systems. Uh, and that data as you can imagine it consumes all those resources I man I I mentioned before. So we are of course consuming a lot of you know database requests you know blob storage there's IO moving things around there's a lot of CPU serializing d serializing things uh there is a lot of RAM because you know

31:28 you probably need to hold things in memory for a little bit as well so it consumes all those traditional I would say uh uh resources that are not GPU so message I want to give here is just when people think about AI the First thing that comes to mind is GPUs. We need GPUs. GPUs are really important. Don't get me wrong. It's super important. But um everything else, all your CPU workloads, they are extremely important as well.

31:56 There's a lot of resources that are taken to make the whole experience work uh the way it it works. So, we need to watch out for those things. um bringing this up to you because as you can imagine there are code changes, there are improvements, there are shipping rate changes to the actual inference engine, but the rest of the product is a lot larger uh as there's a lot more components.

32:22 So there's a lot more code change going to the other components that affect CPU uh and memory, IO, all those things sometimes even more than GPU. So bringing that up Why that's important? It doesn't matter um that I have extremely fast model, the fastest model, the fastest inference engine if the rest of the path is slow. The user will get a slow product end of the day anyway and it will decide not to use it.

32:51 So um we need to focus in the whole experience, the whole path of the request to make sure things are moving slower uh moving not slower faster. Um, so, uh, the users get that, you know, pretty good feeling when they're using the product. Uh, awesome. Uh, summing things up a little bit, uh, the product path is getting broader. Uh, and it's more than inference.

33:20 Uh, it includes all that data, context, tokenization, streaming. uh that path is getting heavier and heavier with all the agent coding and all the changes we're making and that's just accelerating. Um we're consuming more resources and those resources uh they are not just GPU. Um keeping in mind we need to evolve the way we think about Perf engineering.

33:46 Um again lots of changes need to speed things up on our side as well. Um we of course don't want to ask our engineers across the company to slow down you know or add roadblocks that makes things slower. Uh no every AI uh uh PR that is open needs to be reviewed by five like humans. So we don't want to add things that will slow down that will just take away the whole advantage we have of of using AI agents.

34:14 Um, we actually need to speed up uh the different loops that we use in perf engineering. Both the reactive loop when something requests and we need to fix uh we need that to be faster. We need that to be closer to the actual um code or or where the problem originated in the first place. Uh and the active loop itself that needs to be a lot more parallel.

34:43 Uh usually uh our reactive loop it it takes quite a bit of time. Uh I don't know any Perf engineers around here that they did profiling and optimization things in past. The loop is quite slow. You you find a regression maybe you from one release to another you see that your CPU regressed a few% what do you do next? You need to do some profiling. you need to go and capture uh your CPU profiling where which stacks are actually in CPU the most time and hopefully you have uh uh continuous profiling so you can actually

35:18 compare a before and after and easily spot uh uh where things regressed um if you don't then things become even a bit more complex uh uh you need to look at the code changes try to map that to the stacks that are running CPU and see what regress an optimization then you need to think of optimizations and you need to implement those optimizations. Then you need to go and ship it, test it and see if things fix and all of that is serial being done by one person.

35:46 Uh and that's just it just takes a bit too long. the path I'm exploring within the team uh and the direction I'm seeing things go is is having things always on having our agents always on working non-stop uh in in both loops both reactive and also the active loop. Um we want to automate not only uh the large regressions. So things you can spot uh easily from one release to another.

36:20 Uh where generally you can have an agent triggered automatically by that regression. Uh the agent could go and fire a profile. The agent could actually compare two profiles. We have that today. We have performance skills on on codecs to do that. Um pretty straightforward. If you have all that things, all those things working in your systems, uh agent has access to the codebase.

36:45 Um he can compare before and after what PRs got into that um into that code change and he can go and propose a co a fix and hopefully your systems are safe enough to the point where maybe the agent can deploy that uh and and test if that fix actually had the expected effect. Um, same goes for drift, small regressions. We can have fine grain metrics of, you know, maybe code complexity, maybe the volume of network requests that gets executed during a benchmark.

37:17 We can have um small measures to to find that drift and act on that drift as well. Uh, and of course the active optimization loop that I mentioned before. So the idea here um this is more of a end scenario. Uh that's where we're developing things to uh that where we want things to go to. Um and there's a lot of intermediary steps on that. We we want everything that happens pre-production.

37:47 So ideally before a merger, before something gets deployed, uh all the boring things be caught by a CI. Uh pretty straightforward. We've been doing that for a long time. Nothing new there. Uh but as you can imagine there's a lot of things that um don't really especially on the perf side they don't manifest until they get to production until they get a specific combination of user workload.

38:12 So there's no way of catching that pre-production in the first place. So we need to catch that and we need to fix that once we see and again active profiling. Um a few examples. So we can have uh we can have microbenchmarks running all the time on every single PR uh agents can actually help develop those uh uh microbenchmarks and we can find a lot of different things.

38:38 We can find though that performance that that method regression uh we can actually monitor all the methods that are part of our um our critical path our latency critical path and find specific regressions on those. Uh we can profile memory allocations and we can see if any specific function is allocating more memory and we can act upon that. Um we can watch bundle sizes like as if especially publishing to mobile um you can watch bundle size if these things are increasing and if that exceeds your budget or not.

39:16 Uh the message here is is we've been probably been doing that um for a long time but we can automatically act upon we can go deeper. We can look at the metrics. We can look at the codebase. We can check all our architecture diagrams and all the information we have about the system running. uh and the agent can propose a few ideas of a few ways of fixing that, can go and implement that and ideally you can actually benchmark run the same benchmarks again and compare before and after.

39:50 And yes uh many many times that fix will not work. It will not fix the problem and you can just throw that away and continue doing you can fire multiple threads with multiple ideas. That's that's the that's the point. Um, of course, this is end state ideally. Um, there are a lot of steps to get to that in the first place. Uh, you can actually have a set of skills in codecs that generate that that just make that process faster.

40:17 Instead of manually triggering profiles, collecting things and analyzing, you can have an agent collect that information and do that comparison for you. I actually wanted to do a demo of that today. Uh but then folks told me it's not a good idea to do that in the opening keynote. So I decided not to do it. But it's something that it can be easily achieved.

40:35 The the agents are pretty good at chugging a lot of data and comparing things. So pretty easy. Can generate a flame graph out of it if you want. So fairly straightforward especially with microbenchmarks where we can actually get you can reproduce things pretty easily. Same goes for production. You want pretty good observability in production. You want to catch all those different regressions that you see uh and again act uh upon those same thing you get the same information.

41:05 The difference here is um the safety um things are a bit trickier if you asking an agent to deploy something to production and you don't want your users to feel any issues, feel any pain. Uh so there's a lot of best practices that you need implemented even before you could do that. But you can get almost all the way there. You can actually have an agent submit a PR that uh implements an experiment of a Perf optimization.

41:32 You can do that. There's a lot um if you're not at a stage where you can fully trust an agent to deploy to production without any supervision, you can get almost all the way there. U and should work really well. Um the always on loop is the one that probably excites me the most. Um in my team we have a lot of really really great Perf engineers and we're in the process of how can I actually translate uh all that skill set into a set of skills uh to my agents uh that can continuously be looking for optimizations in my

42:10 environment. can they could be looking at the code, they could be looking at my traces, they could be looking at logs, whatever. Uh but continuously finding optimizations and it's funny too the the the habit people have when they have a very specific niche skill set, they name the agents as themselves. So I have a Ben agent, I have a Brendan agent, I have a lots of agents uh name as people and then you you start treating them as as people at some point too.

42:35 So it's kind of weird. Um but that always on loop trying to find opportunities. Um a few best practices of things that have been working well for us. Um when you can get agents to complete full loops, um things become a lot more autonomous. You can have a lot of agents doing a lot of things at the same time if they can complete the loop. meaning I I start a loop from some event or a continuous loop.

43:13 Um I can have an idea. Uh the agent can implement that and he can do the whole thing all the way to um actually measuring if that change he made was positive and it worked. Uh and so it can continue doing that without any human intervention. So sometimes that's not that easy. Um, and in other times it's a bit easier. But if you can focus in trying to build all the safety to allow agents to complete full loops without any intervention, things will become a lot faster.

43:45 Um, a lot less supervision, a lot less bottleneck on people having to monitor and then you know just have people not clicking accept accept accept or or just do yolo mode and let the agents do their thing. Um as you can imagine too um the basic things we've been doing for a long time they become more and more important you know your engineering maturity um becomes more important.

44:10 Having really good task coverage it's important. Uh otherwise how would the agents know what they did was right or it's working it's functionally working or not and benchmarks the same way how can the agents measure if the the improvement they implemented was was good or not. So having those benchmarks having those tests extremely important became even more important.

44:34 Now u you need your contracts expectations between the different um parts of the system. Observability super important. You need really good coverage um about your system so the agent can understand what's going on. If there's blind blind spots, it's very likely that it might go in the wrong direction. Uh because it doesn't have that say intuition about the system that user you the humans might have.

44:59 Uh that safe uh roll out and maybe a canary analysis, automated canary analysis or a blue green push all those safety measures to get the code out. uh even if maybe something uh maybe there's a bug there but that shouldn't affect users so roll out in a safe way. end of the day um all those pers optimizations that that's a that's a search problem right um and the agents can try the different alternatives they can try the benchmarks uh they can compare things um and it's really cool because they can do all of that a lot

45:41 faster and they can do a lot of those things in parallel u but the agents they do need a a clear signal again to know if they're moving the right direction if what they did is right or wrong or if they can discard some work or actually use it. So they need all those metrics. Um if you actually trying to reproduce a production workload, you need actual um reliable workloads to be reproduced by the engine itself.

46:07 Otherwise, you might be optimizing something that doesn't really match production. Same problem you had before as well working with Perf. Um fast feedback is important too. Um the faster the feedback loop um the more the agent can do without waiting. If your feedback loop might be 12 hours uh then you know uh maybe your agent doesn't really add that much value because the human interruption is is not that much.

46:34 Uh but if you can have a fast feedback within a few minutes I can know if I'm in the right direction so I can you know make a decision and move or in one way or another um things become a lot faster. um that measurement you have uh it just guides the agent. So having the right measurements is super important. Just if I'm measuring the wrong thing, the agent will optimize the wrong thing and I'm not actually improving the product in the first place.

47:00 So defining the metric correctly, making sure you're capturing that. That's why I focus a lot in what the user is perceiving uh during the whole process. So focus on that. And agents like I mentioned they can chug through a lot more information than humans. U of course you need to manage the context window and what you actually sending to be useful but uh you can send a lot of information.

47:27 So all your observability uh all the details about your deploys uh all your code base uh if you have actual written down definitions of the architecture of how the system works is extremely important. all context that the agent uses to define things. Uh different than humans, you don't have tribal knowledge here. Well, you might have tribal knowledge in memories.

47:49 Uh but in general, you want to provide that context so agents can do the right thing. Um general idea we are working towards a world where humans are just setting the direction and our agents are just doing all the work at least the boring work that um is not making the decisions uh and defining what is good or bad. And to wrap this up and right on time which is good.

48:17 Um a few takeaways uh from quick talk today is not just growth growth is increasing of course but the development flow changed significantly. Um we're shipping and making changes a lot faster than we were before. There's argules around if it's good or bad. I don't want to get into details of that but we are making a lot more changes than we were before.

48:41 uh and that has implications in other areas of development too. Uh not just performance but all your developer experience whoever support your CI engines or um your uh code repositories there's a lot of impact in different areas to support that you know rate of change. Um when you're thinking about AI first thing that comes to mind is GPUs. It's not just GPUs.

49:07 There's a lot of optimizations, a lot of things that happen even before anything gets sent to be processed by a GPU for inference. Uh last but not least, um our perf workflow needs to change. All the different workflows that are affected by that different rate of change, that increase rate of change, it needs to evolve to keep up otherwise like our our balance would be wrong and we are at time.

49:31 Um just quick shout out to um uh the teams behind Codex and Image Gen. Uh all the cute nice images that you guys saw here today were generated by Image Gen 2 which is really cool. So quick shout out and um if you want to want to chat after this feel free to add me to, you know, social networks. I'm always inclined to talking with you know Perfminded people.

49:57 Uh and I will be here all day very likely at the at the coffee coffee table there. So feel free to reach out and and have a chat. So it was a pleasure. Thank you. >> [music]