ArtCraft is a desktop IDE for interactive AI image and video creation, developed by Storytold. It lets artists compose in 2D, stage scenes in 3D, and select models for image, video, music, sound, 3D-mesh, and world generation. Its visual tools include 2D and 3D compositing, image-to-location scene planning, image-to-3D mesh conversion, character posing and identity transfer, kitbashing, background removal, canvas editing, masking, inpainting, prompted image editing, text-to-image generation, and image-to-video animation. The project supports integrations including ArtCraft's catalog, Grok, Midjourney, Sora, and World Labs, and is distributed through stable Windows and macOS releases; Linux users can build it from source.
AutoCraft is an autonomous software factory that Artcraft plans to open source. It is designed to ingest and triage software issues, perform agentic development, and post-train on the decisions it makes; those decisions are also retained as evaluation and training data.
Codex is OpenAI's coding app, built by its developer experience team. The app was cited as an example of a product whose team maintains strong ownership over quality.
Cursor is an AI-powered coding agent and integrated development environment designed to accelerate software development by handing off coding tasks to AI. It evolved from an email client into a multimodel development tool, supporting broader developer workflows. The platform also offers MCP-connected capabilities for tasks like searching and editing notes.
DuckDB is an in-process SQL OLAP database management system for analytical workloads. It uses a PostgreSQL-inspired SQL dialect, can run from edge devices to servers, and reads and writes formats including CSV, JSON, Parquet, Iceberg, and Vortex both locally and in object storage. Extensions add functions and format support, while native clients provide APIs for Python, Go, Java, Node.js, C, C++, R, Rust, and ODBC. DuckDB is open source and MIT-licensed, with governance by the independent DuckDB Foundation.
GitHub CLI is GitHub's official command-line tool, developed for GitHub.com, GitHub Enterprise Cloud, and GitHub Enterprise Server. The `gh` command brings pull requests, issues, and other GitHub concepts into the terminal alongside Git and source code, including workflows for authenticating an account, creating repositories and branches, pushing changes, and opening draft pull requests. It supports macOS, Windows, and Linux, and an agent skill is available for driving `gh` from coding agents.
PDF Craft is an open-source PDF application described as a fast alternative to Acrobat. It is intended to run on Linux and NetBSD.
Pi is an open-source AI agent harness and toolkit from earendil-works for building and running coding agents. Its packages provide a unified multi-provider LLM API, an agent runtime with tool calling and state management, an interactive coding-agent CLI, a terminal UI library with differential rendering, and vendor-neutral telemetry contracts and adapters. Slack and chat automation are provided through a separate package project. Pi does not provide built-in restrictions for filesystem, process, network, or credential access; it runs with the permissions of its launching user and process. The project documents containerization and sandboxing approaches, including a local Linux micro-VM, Docker, and policy-controlled sandboxing, for stronger isolation.
Pi is a lightweight coding-agent harness presented as an alternative for people seeking to avoid Claude Code lock-in and reduce cost per token.
A plugin for the pi coding agent that directs the agent toward a specified goal, such as improving performance. The video describes it as having become unnecessary for the speaker's use with present-day state-of-the-art models.
Rust is a systems programming language and toolchain developed by the Rust project. Its compiler, standard library, documentation, package manager and build tool Cargo, formatter rustfmt, linter Clippy, and editor support are maintained in the main repository. The language uses a rich type system and ownership model to enforce memory and thread safety at compile time, while targeting fast, memory-efficient software for critical services, embedded devices, and integration with other languages. Rust is primarily distributed under the MIT and Apache 2.0 licenses, with portions under BSD-like licenses.
The Sanity MCP server is an MCP server for the Sanity content platform. It provides tools for AI agents working with Sanity; after reported use by about 20,000 agents and roughly 3 million tool calls, its team added a give-feedback tool so agents could report friction they had previously worked around.
SolidCraft is a community-led, open-source computer-aided design (CAD) project with an open engineering kernel and support for the Model Context Protocol (MCP). It is intended for creating 3D-printing projects.
Searchable transcript of AIE CODE: OpenAI, ArtCraft, Cognition, Exa, Anthropic, Amp, et al — AI Engineer (09:05:10). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:47 Heat. Hey, heat. Hey, heat. Heat. Heat. Heat. Hey. Hey. There are 10 minutes remaining until today's programming begins. Heat. Heat. I feel deep. Heat. Heat. Heat. Down. Down. feel. Come on. Good Good morning and welcome back to the final day of AI Engineer Code Summit 2026. It's been an incredible few days of learning and sharing ideas. Today we're keeping that momentum going as we explore what's next for AI engineering and look forward to another exciting day ahead.
16:15 To kick off our final morning of keynotes, please put your hands together for our MC Cory Quinn, chief cloud economist at the Duck Bill Group. Good morning. Little bright this morning here. Uh, hope everyone had fun yesterday. Did we learn new and exciting things since then? I mean, if we've read the news, we know that whatever we talk about today, it's unlikely we're going to be today's AI top story for better or worse.
16:59 Uh, we've had over the past two days a tremendous run of workshops, keynotes, conversations, and various expo sessions. I'm told that yesterday a new high score on the live stream for this conference, uh, over 50,000 people have watched it. So, it's not just the people in this room, it's the folks at home. uh I assume in my imagination yelling at their computer screen when they disagree with people slightly more polite than doing it from the front row, but you know that happens.
17:26 Um so today as we head into our final day of programming here, it's a full schedule full of a bunch of exciting conversations, interesting people, and some great talks. U whenever you hear someone say, "Oh, there are some great talks." I always mentally finish it with and three terrible ones. Uh, I don't think that's going to happen yet, but we'll all find out together.
17:47 No pressure. Uh, before we dive in, I do want to thank once again the partners and sponsors who make an event like this possible. Uh, obviously starting off with Google DeepMind who has been uh, cooking with gas lately by all accounts. Good for them. We also have our lounge. Yeah, there are our lounge and platinum sponsors uh, who again make the lounge and other nonsense like it help possible.
18:13 the gold sponsors as well. Uh again, we couldn't do this without folks helping on these things. Our silver sponsors, similar boat, just a little bit later in the sequence. And of course, all of the supporting partners who again, collectively, all these companies pitch in when they don't have to to help something like this happen. You can't do it without help.
18:34 I mean, the coffee doesn't set itself up and take itself away and set itself up and take itself away all by itself. It takes people. Uh so that's effectively where we are now. It's been an exciting event. We are looking forward to bringing it home. But one of the big themes of today is ambition. One could argue that's been a theme of the whole AI revolution for the last three years.
18:57 But here it's it's about thinking a little bit bigger. And to do that we're our opening talk today is titled ambition in AI. Uh Jason Louu is in developer experience over at Open AI and I'm excited to see what he has to say for us. Let's please give him a warm welcome. Jason Hey everyone, let me just get the uh clicker real quick. Sweet. Let's see if these slides show up first.
19:52 All right. Hey everyone, I'm Jason from the developer experience team at OpenAI and I'm super excited to kick off this track on raising your ambitions. You know, I joined in March and the Codex app wasn't really a thing uh for a couple weeks. There was no Astra yet. And when these things launched, I kind of realized that I stopped becoming someone that worked on the developer experience team and I worked on the everything experience team now that AI can be a tool that allows everyone to make things.
20:18 And so I want this talk to not only to be addressable to the people here in this room as developers, but for everyone online as well. You know, in this year, we've done a lot of different things, right? We wanted a better coding app. We built a Codex app. We've been contributing to things like biology. Folks have wanted to make personal applications.
20:34 So, we launched sites, cyber security, voice, computer use. And in this past couple of months, we've done some really ambitious things like our contributions to math and physics and a lot of new physical engineering capabilities. Um, it's kind of crazy to think that a couple years ago, we're doing great school math and now we're be able to like tackle Navia Stokes.
20:54 So when I got asked them to talk a little bit more about what ambition meant I think the question I keep going back to is like ambition isn't sort of how you did do things but like how you approach these things right what do you want to make and how ambitious are you and thinking about the things that you make I know this is a coding conference after all and many of the talks here will talk about things like how to run like longer running tasks or I think talk about like dark factories and multi- aents um but I think
21:21 from someone like me like it might not be as relatable to just say that I have like thousands of agents running in the background, right? I have unlimited tokens. And so really, I want to give a talk that talks a little bit more about like what I think is the role and responsibility for developers in the future, right? And so like using more tokens isn't what I mean by being more ambitious.
21:41 To me, in a world where execution is now free, I think ambition is summed up by these three principles, right? Do you actually believe that you've created this work and that you have ownership of this work? Can you describe what your tastes are and understand how you can tell the model to make things better? And can you describe what good looks like?
21:59 And then lastly, are you actually curious enough to use AI to pursue other fields, right? Are you actually able to expand your horizon rather than thinking that you're going to be replaced or something like that? And this mostly comes from three different fears that people have when it comes to like using AI more. Are we just going to create more slop in the world?
22:20 you know, is AI going to destroy my ability to think critically and is AI going to replace us, right? And so I want to talk about these things one at a time and use that as a way of framing this conversation about ambition. And so the first thing is you really want to think about owning the work that you ship. So the first question is like are we going to ship more slop, right?
22:37 And probably the answer is yes. But I try to think about this the same way that we thought about the camera when it first came out, right? as a way of raise of like show of hands like how many people here feel like the technical work they do is a creative pursuit. Yeah, you know me too. Uh these days I don't think of myself as a developer. I mostly am thinking about myself as someone who's trying to will things into existence, right?
23:02 And I have to again feel this ownership and describe what I like. In a past life, not many people know this, I used to be a machine learning engineer. I used to be a physicist and I used to be an artist. And the the story of the camera is something I keep thinking about as we introduce more ways of doing this kind of replication, right? As as this way of creating things.
23:22 When the camera first came out, painters who were busy focusing on their ability to recreate images, yeah, they were a little stressed out. But at that time, people weren't even thinking about whether or not photography could change the art world. We weren't thinking about what a movie meant. You see like a Charlie Chaplan film when it comes out, it's really easy to dismiss that as some, you know, silly thing.
23:42 But at some point you realize that not only is photography art, but it has changed the art world, right? And now we have film and all these other mediums and there's amazing things that you can explore. Walter Benjamin says this in a book, The Work of Art in the Age of Mechanical Reproduction, a book that came out, right, when the camera became popular.
24:00 He says, "Photography freed the hand of the most important functions and brought them to the eyes and to the voice, right? This is kind of the same with like AI now except I just get to use voice mode on on chat. I can just say, you know, do this and do that. In the same way that at some point if you're a photographer or director, you can just say, you know, move that camera over there.
24:19 I want a wider shot. I want to capture some kind of feeling. And maybe I have a team around me that helps me capture that vision. And now we live in a world where everyone has a camera. And sure, maybe I have a thousand photos in my in my background in my my camera roll that I never look at. Maybe that is flaw. But it only matters whether or not I have captured a couple of really important moments in my life, right?
24:40 And and having ownership over those moments to the thing that's important. Like for example, like 30 years ago, like I didn't really have childhood photos because we just grew up in a place where we couldn't afford a camera, right? And now technology has allowed everyone the ability to have that kind of access. And I sure would have liked to have a camera back then, even if it meant a little bit more slop, right?
25:00 And so if you go back to this idea of this painter, I think the question you have to ask yourself is, do you identify yourself as the painter in this example or as the artist, right? Maybe it means that painting photorealistically isn't the skill that's really valuable. But if you think about yourself as the artist, right? Whether it is the paintbrush or the camera, whether it is code or clay or anything else, these are not just mediums that allow you to express yourself and to make things, right?
25:25 I don't make a table and I say like DeWalt made this table, right? Just because a director has thousands of people on set, it's still the director's movie. And even in open source, we might have thousands of contributors or even agents now running the system, but it's the core maintainers that help help hold the responsibility for the quality of the work, right?
25:42 And I don't think the danger these days is whether or not like AI is going to create more slop, right? But instead, it's about self- erasure, right? Are you going to be a passive observer of the work that's being created which results in this slop or are you going to be someone who is an active participant in the act of creation? Right? I think many people have been sort of moving themselves away from this.
26:03 Right? And the only difference between like a person who's making a thousand PRs versus another is whether or not they actually care about the work and whether they are an active participant, you know? You know, and so as an engineer, what this means is you just have to like use your products more, notice when things break and decide whether things need another pass.
26:18 Like a lot of the great examples you see online are not oneshot examples, right? These are things that you iterate over time because you feel proud to create this kind of work. You know, internally we have a lot of amazing work with like our ownership over like the Codex app, right? Mo most people here have an incredible sense of ownership over the product and as so as things slip as we ship more, we want to bring them together.
26:41 And again, this is mostly because of the fact that we care, right? So, in terms of worrying about slop, I think that the the medicine for slop is really just the fact that you really need to believe that you own this work. I think the second thing to address is whether or not you feel like you can actually describe what good looks like. And for that, you need to go out and be inspired.
27:07 So the second question people are asking now it's like do you think AI is going to destroy our ability to think critically? Right? If I own the work, how do I know if something's good enough? If I just hand you a camera in this analogy, like does it mean that it takes away all the effort of creation? I don't think the answer is yes. Right? Instead, it's now about having language to describe what's going on.
27:27 So when I first started art school, you know, maybe I don't know how to develop film. I just use my iPhone. But I still need to understand like the elements of design and what does beautiful look like, right? Space, value, shape, texture. You know, when I I would just spend a lot of time thinking, you know what? I don't think I understand what negative space means.
27:43 Let me go do a lot of sketches and explore a little bit. It's the same thing with like development now, right? I can make multiple apps and I have to think about why I like something and why I don't. And most of my time in art school was just thinking, man, like I can make things and I can look at them and I just say, I don't like how this looks. And what you I was noticing was that my taste and my judgment exceeded my skill.
28:05 I could not do the thing I liked, right? And I can't just look in the mirror and tell myself like make it pop a h 100 times and and know that it's going to be better. I have to go out and develop that language, right? People say like taste is the moat, but I think I I I generally hate that line because this doesn't tell you what taste really means, right?
28:21 To me, taste is the ability to describe what you like and predict what others will like, right? and building this model of the world and of other people. That's the new thinking, right? Winstein's the limits of my language means the limits of my world. And I think that's very much the case on pursuing ambitious projects. You know, it was a funny example of just like the use of language.
28:43 I don't know if Steve is in the room, but he made a tweet a couple days ago that just says, "Find all the implicit state machines in my app." I had to go Google what that meant, right? And all of a sudden, all the comments are just how like everyone's apps are like 20% faster. These are now like really simple ways of just developing some language to understand what good looks like and how to make things better.
29:03 If you think of like model progress in machine learning, there's a really helpful concept called the generator discriminator gap, right? You'll have two models. One grades the model and then one actually creates the outputs. And what happens is is if the grader is too good, then the model can't learn and if the generator is too good, the discriminator can't help the model improve.
29:22 We are now that discriminator, right? As model capabilities go up, at some point maybe maybe the model's outputs hits your own taste levels. Then it's really hard to actually improve. And then to make things that are better, you have to go out develop that language and know what good looks like. And the failure mode is here just not being able to discern whether or not a model can improve, right?
29:43 And again, what this means, we have to go out and learn again. We have to think again. This is just an example of a bunch of things I spent my past year learning like in just building software, right? Like as I build these applications, I realized, oh man, these websites look really ugly. Like I need to learn about typography. Like why is everything itallic?
30:00 Like I don't understand. And I had to learn I spent a lot of time learning about typography. Then it was about like how the information was presented to avoid looking like slob. And then it was about like motion design and how can I make things move in a beautiful way. And I got to spend so much time learning about animation and then learning about like interaction patterns and what makes an app feel beautiful.
30:18 And this is really fun, right? And this is again spending more time learning in order to make great things. And most importantly, having the ability to describe things is what allows you to differentiate, right? If you think about these purple apps, the reason these existed was because the people making these these applications like didn't have the site.
30:37 They couldn't actually describe what was wrong and they thought this was okay, right? And this limiting in language is sort of the problem that we face. And then so the question now is okay, how do we develop this new language? And honestly, the line that I've sort of been like punished to now is just like if you want good taste, you have to eat. You have to go out and explore things, right?
30:57 People are like vibe coding video games. Maybe the answer isn't just to like tell the model to do better, but like play some video games and understand what feels good, what what animations feel good, what stories matter, what stories do I want to tell, right? If you're building tools, try other people's tools and applications and understand what you like.
31:12 Look at beautiful things. I've been doing a lot of like 3D printing with AI and everything just looks like rectangles glued together. And I just kept saying like make it more beautiful, right? And that's a really frustrating experience. And I started looking at like sports cars and like industrial design. I started going to stores and looking at things and go like let me describe and understand what like good curves look like.
31:33 And again, it's like a really fun experience. And now more of my designs are like incredibly beautiful for what I think is beautiful. Right? I've been helping with like marketing and I realize that like I tell the marketing team that they should use like the Codex app more but I'm not watching more ads and as I'm watching more movies I realize there's great stories to tell and everything becomes better again right and this is kind of the really exciting part of using these tools right and like thinking about why I like
31:59 things and why things are amazing and why things feel good now is how I spend my time right so I'm not thinking less I'm thinking more and it's really enriching my world and making me see things differently And if you're running a team, I think that the one takeaway here is that like if you really believe that AI is making you more productive, I think you owe it to yourself and the rest of the team to make space to go out and explore things and try new applications and and just actually go out and like experience the
32:24 world, right? And if you are someone that makes really great work, inspiring others as well, right? And that's how you can come back with a point of view. So go out and be inspired. Learn to be a better critic and recognize that you're not actually thinking less. you're thinking more and the last thing is actually just being curious really really curious to pursue new ideas like one question I want you guys to think about as you are thinking about your role as software engineers is like what were the things you gave up
32:52 or put aside to pursue software right where else could use your judgment and your experience you've developed as a software engineer right because I think people worry about being replaced but really I think people are needed in way more places if you think about when the uh you know invention of like the pencil or or sculpting became more easy. All these different industries in the creative world became really very much multi disciplinary, right?
33:18 It completely made sense that if you were a furniture designer, you could be an architect and you could like make a movie and design kids toys and playgrounds, right? Like Michelangelo was a sculptor and a painter and an architect and a poet. You know, it completely made sense that you could do all these things, right? And this is a creative world. like you invent the pencil, you can do a lot of really amazing things.
33:37 But with technology, it's become even more complicated to do so. I just can't get like four PhDs. And if I could, like maybe I'll just be like 50, right? And so being technical has become incredibly difficult. But when you do cross-pollinate these different industries, amazing things happen, right? The physicist was sorry, the founder of molecular biology was a physicist, right?
34:00 A mathematician started leading the human genome project, right? Right? And the mathematics around magnets was what you is what is used to help understand neural networks. But this is now like super hard to do because you need so much information. Right? Even in our own world like we have a bunch of AI helping us design like chips and writing software.
34:18 And these again are just very multi-disiplinary pursuits. If you look at what it means to be ambitious outside in the real world, there's a machine learning consultant using AI and molecular biology to like help treat their dogs cancer. There was this uh public servant in Hokkaido who used like chat and like computer vision models to build like robots to automate their farm, right?
34:41 This cross-pollination and this will to create these things and learn and cross-pollinate across these different paths is what's really inspiring, right? I I think earlier today I saw folks are using AI to like make more robots and also like discover exoplanets, right? Right? And it's that curiosity and that enablement of what AI allows you to do and lowering that barrier is uh what allows you to be ambitious in those fields, right?
35:06 And so when it comes to thinking about whether or not you can you're going to be replaced, a lot of it really is the fact that you now have the skills that everyone needs in every field. It's not just about like writing more apps, but actually thinking about, you know, what were the things I didn't pursue in college to pursue software engineering? What how can I go back to those worlds and how can AI make those things easier?
35:26 Right? You can again develop new language and understand what makes things amazing and then go go pursue those things. And the great joy of this and the reason why these like multi-disiplinary artists are so successful is as they pursue new mediums, the act of being a kid again means that you have the naivity to ask for what might be impossible. Right?
35:46 The reason this crosspollination works is if you have the judgment you've built in software and you pursue another field, it means that you can ask a question that people might not think is possible and you have the courage to use AI to help you get there. Right? And so there's great benefits in being able to feel new again in a field while all having all this experience in AI as well.
36:05 Right? In my personal life, I've been like learning the drums. I've been building custom like drumming software. AI is helping me learn how to use like Ableton. I'm connecting my my drum set to MIDI and learning there. You know, I've been using computer use to teach me new software that I never really had time to learn. And I've even been using AI to help me design like physical soft uh physical engineering.
36:25 So, I'm 3D printing parts, building chips. And again, these are things I was always curious about but never had the time. And again, part of having AI is not only getting my time back, everything else is accelerated, right? And so, being ambitious is about exploring those pursuits, not just spending more tokens like building an app. And so this is also what I want you guys to think about, right?
36:46 Again, what fields did you put aside? How can you pursue them now that you have more time with AI? And so what I'm when I'm talking about like increase your ambition, I'm not just saying like become a biologist, right? What I'm really saying is that like part of your job now to become more ambitious is to be really inspired, pursue these things, learn that language, and also for that next generation to be inspiring.
37:10 Right? When people used to ask me how to like learn how to code, it was never a case where I said, you know what, like you should go read a textbook on Python. It was always build something, right? It was always go build a weather app, go build a like note-taking app, have a real problem and solve it. And now the scope of these problems are much bigger.
37:27 And so as software developers, I think our job now is to not only build amazing things and share them, but also inspire the next generation to have more hope, too, right? Because maybe software is changing, but really we're going to be be able to contribute to so much more things, right? And like we see this everywhere. Like for example, the Codex app and sites, these are all things that were acts of love.
37:46 Codex app, the sites product were all built by people who really cared about these things. They're talking to people online and working through these things, right? And now our PMs can do engineering, our engineer can do design. And this cross-pollination is inspiring others to do the same and take ownership of that work, right? And most of the things here that exist are here because people are tenacious and are trying and want things to be beautiful, you know, and ultimately our mission open AI is to benefit all and
38:13 ultimately what that means isn't just having open AI have all these AI to to control and you know use agent swarms or whatever, but also making sure that everyone has this technology so they can contribute in their own ways. And so that's basically it. you know, think about the things that you own. Go out, get inspired, develop the language you need to describe what beautiful things look like, and just make sure you're actually cur curious enough and take the time to pursue these new interests.
38:40 And then that's how you become more ambitious. So, I'm really excited to see what you guys are building and uh come by to the booth to chat. Thank you. From creating Flask and Ginga to exploring the future of AI powered software development, our next speaker has helped shape how developers build for the web. Please welcome Armen Ronaher, CTO of Arendelle.
40:04 Hello. Uh, my name is Amin Rona and I want to share with you a little bit the lessons that we have learned from building software with agents. And in particular, I want to make a strong case for um not so much building the factory as building the artiller or workshop. And um I want to show some data with this from our own usage, but in particular I want to explain what I mean by building the artiller or the workshop prior to building the software factory.
40:33 And um I think everybody here is being use has been using agents for quite a while and you have probably noticed that uh it is we're sort of on an exponential curve in in what we can do with them. Um but the comparison that I want to give here is that you can be an amazing artist and use the greatest and latest technology from CNC machines to 3D printers to just very good power tools and it can increase the ambition of what it can do with your art and I think we can also look at software a little bit in that sense.
41:01 Uh so less about building much more software and just building way more ambitious and high quality software. Um so I'm going to show you a little bit of what I mean by that. Um and um you might know that Arendel is working on the PI coding agent. So we have obviously a little bit of a horse in the race here of how we're looking at this. Um but I think what is probably quite interesting is that the pi coding agent has not really changed in a year on the core harness.
41:28 It still has the same tools today as it did uh in December last year. And I think this is interesting because it has become much more powerful purely as a fact of the models getting better. Um so I'm much more capable today of what I can do with pi without any changes landing in the core functionality of what the harness is doing. And I think this is an interesting thing to reflect on because it means that we are all simultaneously getting better even without landing any significant changes in how we're doing this.
41:58 And um you might have remembered throughout the last year and a half maybe that there were a lot of ideas of how you can prompt the machines better or how you can do some interesting tricks with the harness. Very few of them are still necessary for me. A good example here is that I was in love with PI Auto Research which was a plug-in that basically allowed an agent to drive towards a goal like improving performance.
42:19 But if you use a present day state-of-the-art model, you actually discover that entire plugin is no longer necessary. So there's a lot that is just happening in the models that is dramatically improving how we're doing. So these machines are getting better. Um and I just want to show this from a couple of graphs from basically my PI sessions where I filtered them out to be have nothing but the default set and the total amount of corrections and I used Jeff to analyze this.
42:43 The total amount of corrections that I had to do uh got down and down and down with every single model that got better and better. So I'm I'm pushing back a lot less against the the the the agent um with the latest model than I did in December last year. Um and the same thing with like how often I'm aborting it. Um and I think this to me was the most interesting way in which I sort of analyzed my sessions which is the average.
43:08 So this is the P90th percentile but on the 90th percentile of how long an agent was running in response to a task that I gave it with Opus 4.6 in December last year was about 2 and a half minutes and now with the latest models they are on the 90th percentile running for 14 minutes and more and that does actually change something about how you want to work with a harness but really again nothing in pi has changed and still I'm I'm seeing this sort of different behavior of of how I'm using this um and this probably you
43:40 have noticed in one form or another if you're still using an agent kind of see what it's doing you probably noticed that the present day agents are much more likely to use bash than any of the other tools that are being provided. But again, nothing has changed in how we configure the harness. It's just that there's a lot more emergent behavior coming from the models.
43:58 What's missing here is Astra and Soul. They actually look very similar to Opus from my testing. I just didn't have enough data in my test set um to show this nicely. Um I you will notice that there's a little purple thing here. I want to talk about this in the end, which is sort of code mode is sort of slowly creeping up. Um but the the core of this is um even if you have a piece of infrastructure nowadays that is sort of there and is doing a good job, it's still changing as a result of the models.
44:26 Um what has not changed in the last year um at least this is the case for me and I think this is the case for for a lot of people is that we're still humans. We're still limited by our fundamental capabilities. Um, and as we are being more ambitious and we're doing more with these machines, um, we're I'm not I don't want to say like we're the bottleneck, but we are the thing that paces what they can do.
44:48 Um, and in particular, human attention is for me at least the limiting factor. And I'm noticing this with largely in how coding agents deployed in teams. um that this is sort of a stable amount of like attention that we have that we have to work with and um starting new project is getting incredibly easy with these agents and I think like particularly when a new model comes out we're all super excited of trying the latest and greatest thing you can do from 3D games to um new ways in which we automate things but
45:18 maintaining them has not really gotten easier um but we are also increasingly taxing a lot of external systems that actually in many ways are not supposed to be part of this revolution. Um, I think we all know that GitHub has a lot more commits being pushed up there and a lot more repositories, but I know quite a few people who have wellestablished old websites on the internet that found a new life for agents and they are totally not ready for the new volume that's coming from from us kind of directly hitting them.
45:47 So I I just want to caution a little bit here in saying that when we are pushing things, we're also accidentally simultaneously all of us pushing things um which we're not really ready for for what agents are doing. Um and this to me I think is an interesting graph because the number of pro prompts that I'm actively pushing over the year is is sort of flat.
46:09 I know that like in some months it's a little bit higher, in some months it's a little bit lower, but they're they're alternating between 120 190 something like this prompts uh that I'm pushing per day. Um but you can see that the time that the agent was active for my use went from like less than my working day to many times my working day. Um and this November is not even over, right?
46:34 So you can see that this is taking an exponential curve up. Um, and this here is the number of projects. This is Git repositories that I've created over the last year. And this is like obviously cumulatively going up. So I have 141 different projects that I started. But the number of projects I'm committing on a regular cadence to um which means like I'm committing for more than five days to and I'm returning to them also um is pretty much flat with with between 10 15 something like this.
47:02 Um and I have I have done this sort of an informal survey with a bunch of my friends as well and this seems to be a common trend that actually maintaining something regularly is tricky and um I had this discussion with a friend of mine uh and sort of in relation to this graph and he said like well actually I have an open source project that is now fully automated.
47:18 So, uh, there's a issue tracker coming in. There's a there's an agent that's set up to auto review, commit, and send it back. And that is maybe a way to scale up. So, you take the human out of the whole thing. But actually, he also turned it off because eventually he didn't care about the project as much anymore. And he wanted his token specs to use them for other things.
47:37 Um, so our attention to an open source project matters even if maybe we have automated entirely. So if we stop using something where we have a tendency to no longer actually actively take care of it. Um, so if we're pushing towards and I would make this argument first figuring out how to make a really good workshop to build really high quality software, there's a question of what matters.
48:00 And I think for me the three most important things to matter is on the one hand to to really know when to push back against what an agent is doing, telling it that this is the wrong way of doing it. And I think that this is a crucial skill because an agent is really rewarded in just completing a goal. And it is very easy to look at if it has succeeded in this goal or not.
48:24 Um and this we can sort of see it in the same way as the the RL environment maybe is rewarding the the model during training. Um but that doesn't mean that what is being actually delivered is something that we are intrinsically happy with over a longer period of time. And this becomes tricky the more and more commits we're doing, the larger and larger the software is going.
48:44 It's very easy to basically create the Simpson car of all the features in a software. Um but actually having something that intrinsically fits together where the internal complexity of a system does not is like um explode uh still requires us to um to to work with it as it's working. And I I like to call this like true skill versus agent skill. Um there is clearly a difference between someone who is an experienced software engineer working with an agent and someone who is not an experienced software engineer.
49:17 And um I think for me the the truest form of this is that uh my son is now sending me pull requests on a project that we're doing together. And there's a very clear quality difference in what his contributions are because he is not a software engineer. Um but we're using the same system to prompt it, right? Um and I have I've noticed this myself in situations, but I' I've heard this also particularly from a lot of others.
49:40 It is actually a very frustrating experience if you're in a situation that really matters and other people in the room are basically just going to the agent to answer a question and no longer have a really under good understanding of either what the system is doing or that the fundamental problem that they're working on they actually comprehend. Um so there is a lot of value in still understanding what you're doing.
50:03 Um that also means that you kind of have to moderate the speed in which you're doing things to still work within your understanding. And one of the ways in which I started doing this as my commits started going through the roof basically was I I looked into how are my trajectories of sessions going and can I hook myself into a part before the end of it to find a consistent way for me to evaluate what's going on.
50:29 And so for instance, I started um basically prompting the agent that before it's going towards the verification, which is typ typically around 70% of it to start producing a really nice short executive summaries in a nice type set HTML page, they can really understand what it's doing as it's finishing. Um particular with newer models, it's no longer quite as horrible to read the text that comes out of it.
50:53 Um but this is one way in which I found to balance out that I'm basically no longer reading the code in a lot of cases. Um and this I think is something that increasingly is more important to me is to to to also show up with a little bit of a a guidance towards um not writing all the software over and over. Um I'm a huge believer in using fewer dependencies and writing more code yourself.
51:20 So I really like the agent revolution for for that element of it. Um but in German we have this what called vuk which basically means uncontrolled growth and uncontrolled growth in software that is supposed to work together can have a lot of really negative consequences. Um, you might have seen a blog post that was going around recently about a tar archive that is has been crafted in a way that different tar implementations read it in different ways.
51:49 And um I think this is a bit of a problem now. Um we for us to work together we we really want to have standards. We want to have uh common specifications. We want to have common file formats and we now can basically make an age and implement any format we want. But there are going to be these differences. And what really makes it hard for us to agree on standards and what what makes us work together on something often is not a technical challenge as much as it is a problem of how do people agree on what is supposed to
52:21 be in this file format and how is it supposed to be implemented and how is it supposed to be interpreted because there can be multiple interpretations of it. Um and I really think that these open protocols and file formats are really important aspect. Um, you can imagine there is a future where we all have our own little Photoshop clones, but we'll no longer be able to send file formats to each other because we have diverged too much.
52:41 Um, and this really comes down to this part that collaboration between humans is as hard as it has ever been. And this is slowing us in a way down, right? Um, and I think it's it's important to sometimes resist this temptation of just having the agent solve the problem and just taking a step back and figuring out if this is maybe one of those problems where we should be working together with other people.
53:03 Um, it is for me coming from an open source background fascinating to see how little actual human collaboration is happening in open source projects now and how many more projects end up with gazillions of forks because it's just easier to have a fork happen than to try to upstream changes. Um, so I want to make a very strong argument for uh building software that strives towards excellence.
53:28 And I understand that this is hard. Um we have I think seen a pretty big explosion recently of verified software. Um it's also very clear that this is in itself hard. The the the technology stack is not right there yet where this works. But I think it is one of those areas where it's really worth pushing towards and seeing if this is maybe something that can help us over time creating more more stable software.
53:51 Um I really am very happy with seeing so many people wanting to build fundamentally faster and more memory efficient software. I think is a great development. Um, for years I was very frustrated with um, how little my RAM usage actually goes to things that I think are worthwhile. And now there's a significant trend of people trying to rebuild things in a native language rather than for instance just building another electron wrapper.
54:18 Um, so all of those things are really cool developments. Um, but they are in many ways still building up on technology that we have created before times. Um, I really like Rust. I like that there is now a trend towards building memory safe software. I think you've probably all been exposed to what happens with um yeah good models finding security issues.
54:38 So there's a strong argument towards that. Um and it it now has also led to people that would never have picked that language before to actually give it a try because it delivers some value that now can in quote come for free. Um on the last part I want to talk a little bit about maybe there is something where harness can do something and this is less so much for me to say hey here's a thing that a path that we are on is to give you some idea of maybe where some of the challenges are as we're pushing this task to be
55:07 much more long running. Um the first one is if if if you have a task that only takes two minutes to run, you're kind of willing to babysit it and if it breaks and uh it it ends early or it doesn't run to completion or you have to restart the agent to just restart it manually and tell it to continue, right? So it's like we have we have largely been working in environment for quite a while where that was acceptable.
55:29 If you now have automated agents running on thousands of issues overnight, you want to have an agent actually be able to restart itself. And so we're pretty big believers on this concept of durable agents. Also durable agents that create durable workflows where they orchestrate themselves in a way where you can start it, stop it, you can suspend it, you can bring it back.
55:51 And this not just because we have this longer running tasks, but also because there's a lot of inefficiency in what we're doing. And um we are all compute constrainted. Uh we are all uh constrained in terms of the cost of inference. And being able to be a little bit more intelligent about how we're doing this, I think is really interesting. And I wish more people would be pushing towards that.
56:11 For instance, if you pay raw token cost, you get a pretty significant discount on flex scheduling. But that means that maybe for minutes no token is coming, right? And so then you want to suspend your agent, you want to suspend also your sandbox and actually not waste the other resources that are also expensive. So this is one part where I do think beyond the quality of the models because the models are getting so good now we might want to rethink a little bit of how we are using these uh resources intelligently.
56:42 And the second thing is um you probably know that when when when pi came out there was a pretty big um motivation to just say like bash is everything that you have to run and this is still largely what we believe. Um but now we also see that there is um an emergence of um desire to orchestrate sub agents, a desire to use models which are not LLMs like decision models or or classifier models like Jeff.
57:10 And in those cases you need to start composing workflows. Um and um this I think originally came from the folks from Cloudflare say like in in instead of having thousands of tools being put into the code um into the context what if we just give the agent in addition to JavaScript in addition to bash JavaScript to run and have all the tool calls that the agent would natively do happen via JavaScript and then he can orchestrate them better.
57:38 And this has gone from a curiosity I think in many ways to something that is fundamentally shockingly useful. And this means that now the harness in many cases and this is definitely the case already in codex uh you can do it in pi and many other harnesses at this point you can have the harness orchestrate very large invocations of tool calls uh without having to go through the inefficient context and this does require a little bit of a rethinking of how you're doing things um and it also allows you to to use
58:07 classifier models and I think this is the other big part that really changed how I was looking at how far I can push certain technology because you're not being penalized for not having an active KV cache. Um, but they're also very cheap and very fast. And they allow you now to take certain problems that previously were just too expensive or too slow to do to have a decision model do this.
58:30 And the way I think you should be thinking on this is that when you have code mode, what the harness can do is it can create a little workflow that can have decision points along the way where if you for instance have to crunch through 3,000 incoming customer support requests and you have to make decisions along the way, if this is being sent towards having an agent work on it towards requiring more feedback from a customer, you can have the agent create basically an ondemand workflow and then for every single if
58:59 condition that it's doing, it's just going through F and then you can crunch through thousands of issues for very little time in no time in your coding agent where previously that would have taken you hours. Um so with this um I just want to leave this idea now that we can build everything maybe we should not um but we should really I think push towards much more enjoyable and um in many ways much more ambitious software than we have been doing before.
59:26 Thank you. What if the future of software engineering isn't about writing code, but building the systems that write it? to explore the rise of AI powered software factories and the evolving role of engineers. Please welcome Zach Lloyd, founder and CEO of Warp. Okay. All right. Hi everyone. Uh excited to be here and today I'm going to chat about the evolution of software factoring software engineering and how it's becoming factory engineering.
01:01:32 So excited to do this. First a little bit about me. Uh I've been an engineer for a long time. I was a principal engineer at Google. I used to lead engineering on the Google Doc Suite. Uh helped scale that product to hundreds of millions of users. I'm currently the founder and CEO of Warp. Uh, Warp builds a platform uh for companies to build their own internal software factories.
01:01:56 I'm going to show a little bit of that today. Um, Warp is also well known for our open-source Aentic terminal, has about 65,000 GitHub stars. Uh, it's used by almost a million developers. Our mission has always been helping developers ship better software more quickly, and it's it's it's cool to evolve what we're doing as the world changes around us.
01:02:19 Um, this is where I think we're still mostly at. Uh, most developers that I talk to, most platform teams, um, they they are working with individuals who are running their own coding agents locally. Maybe they're running, uh, multiple of them at a time. Uh, could be through their terminal, it could be through an agentic development environment. Um just show of hands quickly actually how many people in here are mostly working in this fashion local multi-threading.
01:02:52 Okay so still most people but actually not everyone um I think that this is going to change and is starting to change already. Uh the the reasons are there's a few of them. So one is like if you are doing local interactive prompting as your main way of working with coding agents um you're kind of limited by you know human throughput really like you cannot automate uh if every agent needs to be interactively prompted there's a second reason too that I hear a lot from platform teams um and from you know engineering
01:03:24 leaders this mechanism of like everyone choosing their own local setup their own models their own skills uh it makes it very hard to actually make sure that you're using these agents efficiently uh from like a cost perspective and a productivity perspective. And then finally, there's like a you know big security problem and governance problem when everyone is setting up and running their own local agents.
01:03:48 And so the thing that uh is coming uh I loosely call a software factory that means different things to different people. Um, I would well I'm curious for people in this room like how many folks are currently trying to build or building or using their own internal software factories? Show of hands. Got. So I would say that's almost more than the interactive agents at this point.
01:04:13 That's super cool. Um, so I'm going to define what I think of as a software factory, but I'll start by saying what I think a software factory is not. Um, I think it's not just a loose set of automations. Uh, I don't think of a software factory as a collection of AI teammates. Uh, I don't think it's a shared set of skills and MCPs. And I also don't think software factories are personal agent swarms.
01:04:40 I think these are all useful primitives, but they're not the end state that we should be shooting for. And so to me, the way that I define software factory is very specific. Um, and I'll unpack this, but I define it as a code defined closed loop system for continuously measuring and improving cloud coding agents. So that's a mouthful, but let me unpack it.
01:05:04 So if you want to build a software factory and really scale it, I believe the first thing you need to do is fully encode it in a declarative language. So you need to say okay these are all of the coding agents that I want to run these are how they're triggered when work comes into this factory this is the workflow from is you know from triage to spec whatever your workflow is you need to specify what MCPs are available where your agents run really as much as you can in a Terraform style declarative uh you know set of
01:05:42 code files that define your factory I'll talk about why this matters in a The second part of it is a continuous closed loop of improvement. And that closed loop can look something like this. This is the typical one. This is just really the SDLC. Uh where you have ideas that come in from people um uh they could come in from automated signals and then your agents do some amount of work um taking that issue through various stages from triage, implementation, verification, monitoring and so on.
01:06:14 The important piece here is really this closed loop concept where the more that your agents run, the more efficient you can make this loop. And I'll talk about how you do that in a second. The third really key aspect if you're building a software factory is that everything needs to be measurable. um if if you're just going on vibes in terms of how you deploy coding agents, you can't be confident that you're actually um getting the best highest quality outputs the at the best cost.
01:06:46 You can't be confident you're maximizing throughput. So you really need a measurement mindset and measurement capabilities. And so if there's one thing that I'm hoping people walk away from this talk with, it's the idea that the way to work with coding agents and the way to think about them is that there there's a really cool fundamental new engineering problem, which is how do you take this magic new primitive that we all have that we've gotten the last year that you have these intelligent agents and how do you
01:07:18 engineer a system around them to use them most effectively. To me, it's not about like growing a garden or following a recipe. We're all engineers. We've spent our careers learning how to solve problems from an engineering mindset. And when you think of coding agents as a new engineering primitive, then you naturally want to you'll naturally start to build a certain way of using them.
01:07:45 That's the mindset. So, yeah, just to unpack that a bit. So there's continual measurement and continual learning, but there's also a new set of tools. And so when I was learning how to become a better engineer at Google, um you know, I really had to get good at using things like debuggers and profilers and Chrome dev tools. And basically there were all of these tools of the trade for building software by hand that maybe are not as important now.
01:08:13 But they really separated uh the the best engineers from the okay engineers. And in the new world of factory engineering there's a new set of tools that everyone should be trying to to master. And so those are things like eval. It's can you set up uh self-improvement loops that work? Can you write benchmarks? Can you AB test your factory? And that's a really important skill set.
01:08:42 And that mastering that skill set is something that I I think of as like factory engineering. Now, that doesn't mean that that's the only kind of engineering you're going to be doing. You're still going to be doing product engineering, meaning you as an engineer should still feel responsible for, you know, does the product you ship actually solve a real user or customer problem?
01:09:06 But the best engineers and I would say like every engineer really is going to need to learn this new tool set of how do you improve the system not just build the product and so what does that look like? Uh if you think from first principles about how do you engineer an agentic system I think you come up with something like this doesn't have to be exactly this but uh it's kind of an infrastructure stack that uh that you're going to want to use to engineer your agents.
01:09:33 And so at the bottom of this is what I call factories as code. I talked about this earlier. It's defining all of your coding agent workflows in a in a code format. And the the value of that is that you get the ability to branch off it and test different factory configurations. You can roll back and then crucially it opens up the way for self-improvement which is the ability for coding agents themselves to suggest changes to how your factory works.
01:10:02 to get that continual loop going. Above that is a layer of data and context. This is also crucial. So if you are running coding agents at scale in an organization, you you really must be recording every everything that those agents are producing. So all of their conversations, all their artifacts, the videos that they make, the screenshots. Uh and the reason for doing that is that's the raw material that you will use to improve the way your factory works over time.
01:10:34 On top of that is a compute layer. Um so you need to have sandboxing. Uh sandboxing increases security. It makes your factory more repeatable. Your sandboxing should be probably multiplatform. On top of that is the inference layer. The way that I would think about inference and models in a factory is they are a degree of freedom to test. And so your factory, however you build it, should support multiple models.
01:11:01 Um, it should also support multiple coding agents because you know cloud code might be better for some things. Codeex, pi, warp, these harnesses all have different properties. You might want to write your own coding agent harness that has some of your organizational context baked in. So your platform should support that. On top of that, you need an orchestration layer.
01:11:23 So this is what actually moves work through the factory. In some ways, the least interesting part, but you need that. And then finally, your factory should be all API and MCP first so that it can be used not just by uh you know people, but can be used by other coding agents and it can go to where the work is being done. So your factory should integrate into things like Slack, Linear, Jira, GitHub, and so on.
01:11:50 I'm going to give a quick demo of what some of these tools look like in the context of warp factories, but this stuff I'm going to show I think applies also if you're building your own. So let me start here. So the at the foundation I mentioned earlier is factories in code. And so I'm gonna show some code. It's not going to be that interesting, but if you're familiar with something like Terraform, you will get what's going on here.
01:12:18 You basically want a declarative way of saying this is my factory. My factory uh has certain policies. It accesses certain repos. It has certain triggers. It has a sort of blueprint that defines the workflow of work going through it. And then if you zoom out all of these other things that are part of the factory also have code definitions. The second thing moving up a layer is um the idea that uh uh sorry that everything that you do in the factory um should be tracked across your entire team.
01:13:00 And so this think of think of what I'm showing here is this is a sort of Google Doc style live view of what every coding agent on Warp's team across all of our engineers are doing at all times. And so I could click in and I could see okay I don't know what Maggie is working on but I know I could look at it. She's implementing Opus 55 on the high plan.
01:13:23 She's doing it from Slack. It has this artifact. I could look in and see the actual conversation here. I could see what it's generated. And so I could see this in real time which is cool. But the really important thing is that you gather all this information so that it can be used for um improving the factory. And so that brings me to the third you know the third thing moving up the stack here is once you have all of this information that's being stored centrally you get the ability very easily uh without doing any
01:13:52 like install stuff on everybody's laptop anything like that to to understand how these coding agents are working and you can think of that in terms of like traditional output metrics like your Dora metrics like okay cycle time but you can also think of that in terms of like how automated ated am I like how many human interactions per PR what's your throughput you get um you should be able to see data very clearly here on like what's your cost and again surfacing this if if you don't have this data you you're kind of
01:14:25 flying uh oh you're kind kind of flying blind um so this is really important and then there's this uh next layer of data which is also really important that I mentioned earlier which is the ability to set up online evals that run across all your coding agent sessions. And so what I mean by this, an eval is just like as these coding agent sessions are running and they're recording their output, you want uh the ability to to go back and have another agent look at them and score them on some dimension that might indicate
01:15:00 how well they're doing. And to make that more concrete, so it's like we have an eval a scoreer here that we use that is looking for redundant tests. This is a good example. And so one common failure mode of coding agents is that sometimes they will write tests that are unnecessary. We want to catch that. And that's what this eval does. We want the ability then to see okay when it's writing too many tests, why is it doing that?
01:15:28 And is there a mechanism that we could change in the factory for improving that? And so again, just to kind of recap here, uh you want a way of recording all the runs. When these runs run, I didn't show this, but all of the runs are, you know, you want the a way of specifying um what is the sort of like actual loop that they're running through. You want to be able to see where they are in that loop.
01:15:54 As the runs uh work, you want to be able to gather their data. Once you gather their data, you want to be able to set up these online evals. And then these online eval give you a sense of like, okay, where are their opportunities for improvement. Once you have that in place, you can do something that we call self-improvement, which is rather than having humans look at the outputs of these scores, you have agents look at the output.
01:16:17 And so the way to think about self-improvement is like it looks at the score output. It says okay we saw some number of failed runs um let's try to look for patterns in those failures and suggest a possible solution and then the form that that solution takes is as a PR against your factory definition and so this is like the awesome thing about having your factory defined as code and being able to use coding agents is that you You can then have the coding agents diagnose problems with how they're working internally and
01:16:58 suggest fixes to them. The final tool that I'm going to point out that I mentioned earlier is the ability to create and run benchmarks. This is also essential. So if you're following discourse online, you'll often see things like, okay, this is the best model. This is the new best skill. This is the best harness. The only way to answer those questions for real is to test on your own data.
01:17:21 And so you should have in place some mechanism where you can basically create a benchmark set out of your real data. Um, and then uh replay those tasks. Oops, I'm on the wrong one here. And replay those tasks against different configurations to see how they would have done if you had varied some model or varied some harness or varied your factory configuration.
01:17:49 So just to make this clear, this is something that looks like what you might get uh when a a new model drops like in SweetBench or Terminal Bench. It's showing the tradeoffs of different models here in terms of cost and quality. You want to be in this upper lefth hand quadrant, but it's on your own actual data. And you can you can test not just models but also you want to be able to test hey if I got rid of these skills does my factory perform better or worse.
01:18:20 So these are some of the tools. Um and just to return to this big concept like this has worked well for us. We've managed to use these tools to drastically improve our throughput and lower our cost. We we are just constantly doing this because this cost really matters to us where we're getting a performance baseline. We're scoring, we're benchmarking, we're testing, and we're doing factory engineering.
01:18:45 And so that brings me back to this last point that I just want to emphasize again. Engineer your agents. It's a new engineering primitive. We're all engineers. Think about how you can get the most out of them and build the system for that. I think this is very exciting. Um, I know there's a lot of like angst around what it means to be an engineer right now.
01:19:08 I know we're not writing code, but I'm still using all the engineering skills that I spent my whole career learning. And so to me, it's still a really exciting time. Uh, if you're interested in more factories, you can check it out here. Thank you all for coming to the talk. What happens when AI agents can build entire creative applications in just weeks?
01:20:20 Our next speaker is the founder of Artcraft, a project exploring what's possible with AI powered software development. Ladies and gentlemen, please welcome Brandon Thomas, CEO of Artcraft. Hey everybody, thanks for attending my talk. I had to put this together late last night. I've been in a whirlwind after this project went viral of unblocking thousands of open- source developers.
01:21:04 Uh we've had VCs approaching us. I've been in constant meetings. Like this has been a sleepless month of activity since this happened. So if you're not familiar, uh I kind of cloned Adobe Photoshop and a bunch of other apps and that went super viral. My phone, my inbox, everything has just been absolutely lit up. And so I I I apologize for the structure of the the the the talk.
01:21:29 This was thrown together as quickly as I can, but I've got a lot of really exciting stuff to talk about and I think it talks about the future of not only the industry, but just technology as a whole. So um we're going to open source the entire world. Like why not? It's it's something that I've always wanted to do as a developer. That's something that uh a lot of us in the open source community are interested in.
01:21:53 uh open source powers uh uh a lot of the you know technology that we're familiar with but it's never been at the forefront like Linux has always been like a secondass citizen. We've been pulled into a whole bunch of platforms where we don't really have ownership and AI kind of gives us a chance to take the web back if you will. So um I might have to do this manually.
01:22:21 Cool. So a little bit about myself. I've been an engineer for about 15 years. I spent eight years at Square doing uh heavy-hitting Enfra, so online payments, um the uh the authentication stack, basically anything that could take down the company my team was kind of in charge of. And it was, you know, uh eight years of carrying the pager, a lot of really heavy-hitting infrastructure work.
01:22:41 Um and then I've spent seven years doing startups, and that's a whole different world. It's a lot of risk, a lot of learning like what the market needs, a lot of uh correcting your own assumptions. And I've also been um I'm based in Atlanta. It's a big film film state. I've been very big in the arts. So out of college, made lots of films, built laser projectors that play video games on the side of skyscrapers, that kind of thing.
01:23:04 And those are the sort of uh customers that our startups or the startups that I built have have catered to, creatives, artists, filmmakers, and so forth. Um, I want to transport everybody back in time, back to the very beginnings of the web and the internet, the '9s, the uh the the early 2000s. So, for us millennials, we're going to feel old. And for um everybody that's that's younger, this is just let me kind of set the stage for what the internet was like.
01:23:32 Um, we had a bunch of chat applications that were completely open. So, IRC is like Discord, but you can spin up your own servers that you actually own. you can use whatever programs to automate them and interrogate them without rules. Uh all of the commercial uh messaging apps were basically they didn't care if you use thirdparty tools to connect to them.
01:23:50 Uh every website had data exports. They had APIs. You could literally build really cool mashups with Reddit and Flickr and Twitter. And these were like trending on Slash before Hacker News was a thing. And then you know and and in the more modern times uh these kind of trickled down to a much slower pace because the APIs got got removed. Uh there was a bunch of file sharing.
01:24:15 So not not just file sharing. You could actually buy MP3s and feel a sense of ownership and and everything that you were doing on the web. And now it's more licensing. Um there was so much open creativity because largely the people on the internet were technical. They were early adopters. So Flash kids were making all kinds of animations. Uh there were diffuse communities with PHP uh uh PHPVB and Envision and just just a ton of creativity that was happening at the time.
01:24:44 Um Wikipedia got started sort of in this ethos and we don't really see this kind of stuff anymore in the current era and incarnation of tech. Um websites were highly customizable like MySpace at the time. you could actually change the HTML and CSS and make it look however you want. So, this was a totally different alien time compared to what we have today.
01:25:08 Um, when web 2.0 happened, broadband, smartphones, it brought everybody into the fold. So, the technical people got sort of pushed to the sidelines. And it's not a bad thing like we needed our, you know, our parents and our grandparents to be able to use these products. Uh, but we sort of we rounded the edges on things. We took away all the technical pieces.
01:25:26 We made everything easy for people to adapt because they couldn't contribute from a technical footing. They didn't want to unpackage things. They didn't know what APIs were. And so the platforms and the the the proto like protocols got replaced by platforms and uh they sort of grabbed as many users as they could and sort of the diffuse experimental early web started to disappear.
01:25:48 the APIs got taken away from us and then we started seeing distribution being gatekept by the big providers like we never once had a phone ecosystem where we could freely distribute to like like we could on the desktop. Um things started to cost more and more as as startups began to like win market share. They realized that they had pricing power and uh suddenly your Netflix actually became cable once you started subscribing to all the streaming services.
01:26:20 And then they started to get very extractive for companies that maybe maybe hit the limits of their growth. They started coming up with crazy crazy gotchas. So the one that got me is Adobe has this uh you can subscribe to a monthly plan and they don't tell you this. It's in like the very small print. If you cancel your monthly plan before the 12 months or 18 months, they actually charge you up to like $200 to unsubscribe, which is really inconvenient if you're going to a film festival like once a year, making a short
01:26:51 film, and then suddenly you've got an extra $200 bill for an application that you were just using over a weekend. And that's that's hit so many people. And it's really frustrating uh that the web that was once really really promising and open and free uh has turned into this uh extractive owned rented rent seeeking world. Um so it it used to be a very special place in the 90s and the 2000s and more and more has been extracted.
01:27:21 Uh this is a term that's commonly used um in shitification. And I think that aptly describes everything that happened. Um, our company started with the ethos of let's build everything open source. Like the applications that we built in the AI space were um, 100% open. Not only the desktop apps, but the actual infrastructure as well. So um, Artcraft originally started as sort of like a professionally minded Higsfield.
01:27:48 So all of the the filmmakers that I know and that I've I've I've grown up with, we were making a product for them to use rather than a UGC centered approach uh like like Higsfield or or uh Open Art or or so forth have. And that enabled us to give them a desktop tool that lived on their machine where all the files that they were generating lived right next to their editor.
01:28:10 Uh and one of the things that we observed um and my advisers thought I was absolutely crazy that artists, you know, they don't know what open source is. They don't they don't need it. they don't talk about it. You know, isn't really used by artists. Uh we actually observed artists vibe coding their own workflows on top of the software that we gave them.
01:28:27 And that is something that like was brand new to all of us. Like uh there were micro drama studios that were doing order fulfillment for some of the top uh micro drama startups. Uh they would do 30 to 50 grand of fulfillment every single month and they had big teams. Uh and no product out there fits their needs. Like nobody's going to go and workshop and build that specifically for the type of delivery that they need.
01:28:49 Yet they in-house have people that know cloud code and they can take the APIs and they can generate an interface that's like that. So we started experimenting with completely open source giving it to our customers. Um and they really responded to that. Um, and then we thought, what if we could turn back time on everything? Like all of these things that they're consuming and building would be much more friendly and and and give them way more capacity to create if all of the infrastructure was open source.
01:29:19 If all of the application shapes that they wanted to leverage were open source. Like they use Premiere, they use Adobe. like what could what what could we do to give them something that they could actually take away all of the surface area that they don't use and constrain the apps to their specific workflow? What would it look like if they could do that, you know, five or 10 times for each different artist on their team that's doing a particular part of that workflow?
01:29:43 And so we wanted to kind of undo what had happened during web 2.0 0 when everything became owned and and consolidated and and uh uh gatekept and just give people back their applications and AI really is the means for us to do that. So we took a stab at the Adobe Creative Suite. Uh we looked at Photoshop first. We basically prompted you know um this is the basic shape of a photo editing app.
01:30:11 You you've consumed and trained on all of the tutorials about what Photoshop is. That's in your training data. uh we we laid out a large large uh prompt to say let's let's build out the the application. Let's separate the UIUX from that. Let's make everything MCP and CLI compatible. Let's build out plans for every single tool, how all of those work.
01:30:32 Let's talk about the data structures, the file format importers. We basically built these large prompts. And within 24 hours, we were looking at a pretty good approximation of what Photoshop was. And it it kind of blew our minds away. Uh it was functional. It was very familiar to us. It started 10 times faster than the original Photoshop. Now, we didn't like actually cover the entire surface area of Photoshop.
01:30:57 There's a lot of work. We're still not at 100% par, but we're measuring ourselves against what has what needs to be done and we're getting closer and closer. We were at 50%, we've just passed 60%. And in the fullness of time, like Pareto principle, we could reach 99.99% parody. Um I'm going to take a little bit of a sort of a diet trap. Our team has talked about whether we take this further now or whether we just go for parody and right now we're just we want to have we want to fulfill the muscle uh memory of all the
01:31:26 artists that use Adobe. So we are going for Adobe like their sort of interface and look and feel directly in the fullness of time. We will build net new apps that have new interfaces that are more agentic. Um but we wanted to do that also as sort of like a proving function that we have actually accomplished something real completely agentically. Uh so we did Photoshop, we did um basically all of their apps.
01:31:50 We did in design, we did Premiere um and uh this I I did not expect this to go viral. I was withholding posting it on Hacker News. It got out on Reddit and then like 2 days later like the like New York Times was asking me for an interview and our Discord went from 3,000 people to it's now got more people in our Discord than Jev Sunno Higsfield. It I I'm still struggling to bear with what has happened.
01:32:15 This was a it struck a nerve with people. Uh they've felt sort of maligned by Adobe and other tech companies and uh what we've built has has really resonated with them and it's sort of built up sort of this mission to build more. Uh, so all of the anti-AII artists who have been pushing back against AI all of this time have, you know, they've sort of turned around like, we actually love what you're doing.
01:32:37 They've started using these apps. They've started giving us like feature requests and bug reports and we're engaging them. And I think we're going to be able to teach them this is what an MCP is. This is how you agentically create things. And maybe we'll start pulling away some of these bloated interfaces and giving them things that are faster, more fluid, and let them do their work faster.
01:32:55 Um, so we didn't stop with those apps. We've done all kinds of stuff. We've done CAD. We've done um uh uh uh uh Flash. We've done uh a bunch of digital audio workstations. It's not just us, it's the community doing these things as well. And there's really no limit in terms of what can be done. Uh so we've done the entire Microsoft well the core apps within the Microsoft Office suite.
01:33:19 There's been Google Docs. We have uh companies and universities and all sorts of people approaching us wanting to use these things and we're we're basically trying to fulfill all of these different asks. Um they're being used on Linux where they've never been used before. Um PDF Craft is probably the best app. You should all go download it right now.
01:33:38 You'll never have to use Acrobat again. This is like really fast, really smooth. It works on Linux. It works on NetBSD. It's brilliant. Like this one's ready to go. So go ahead and get this one. Um we did Figma. We've actually done the server component for this too. And you can basically have your team collaborate on on documents. It's got like the raft, Paxos, like multi-user setup.
01:34:03 So we're starting to go into infrastructure too. And the entire infrastructure and data model, all of it is open source. We're not keeping any of it proprietary. Um there's a community that's working on SolidCraft that's led by engineers. They've actually got uh an open engineering kernel that manages all of this. Uh and it's it's really solid. Like the MCP, you can actually do 3D printing projects really quickly with SolidCraft.
01:34:30 Um we've had so so much happen. We've got over 3,000 300,000 uh GitHub stars. 4,000 different people have been contributing to this project and it's picking up. It's it's it's actually become sort of a movement, if you will, and we're we're trying to adapt to how much influx of of people trying to contribute to this. I'm going to pick up the pace because I've been kind of lagging on this, but um uh we're getting about 1.7 million downloads a day, and this is accelerating.
01:34:57 Uh we've got a launcher app that puts us on the desktop and like you can you can basically manage everything here. It works on all operating systems. Um a quarter of our users are on Linux, which is really fascinating. like Linux has been historically really really underserved and now they have super high quality professional apps for creative and and uh office work which is it's really nice.
01:35:21 So um we've you know we we love and respect uh all of the work that's been done there in the past but people are really excited to use the apps that they didn't have access to. Um all of these things are MCP and CLI like first class. You can totally command these and you can watch them build and assemble creative projects right in front of you. Um, they've been localized into languages that were never like officially supported by Adobe or Microsoft.
01:35:47 Like we were told that Arabic was not supported in Inesign. We've also supported these apps past their support life cycle. Like Adobe Animate was being uh uh end of life or it was being like put on life support. We've we've picked that up. We've had basically animation studios come to us saying we use this for our jobs. Like this is important. Please please support this.
01:36:07 So a lot of a lot of really fascinating things are happening where we're getting this into more places than they've ever been before. And it's it's definitely not me. Like I I started this and now we've got a community of like 4,000 developers. This has taken on a life of its own. They are doing everything from CI/CD. It's it's incredible. This this is way bigger than just me.
01:36:26 Uh I really got to pick up the pace here. I've made way too much content. So um there over 300 apps being developed by the community. We've developed an app where non-technical people can start contributing and use the basis of the apps that we've created like the UIUX libraries. We're comp decomposing this into all of the the components and primitives so that you can reuse them.
01:36:46 So you'll have nice uh 3D libraries, nice 2D libraries. It's all great. I'll get into the technical stuff. So um Rust was probably essential in this product project success. We'd seen people do it in Electron before, but for high quality creative apps, you really need something that's native. Um, there's just a lot of really, really good quality things about Rust.
01:37:06 Google had papers about like the low defect rate. It starts really fast. It's it's the best language for this. Uh, yeah. Um, it supports all all different platforms and operating systems. We've got this on NetBSD. We've got this on tablets. It natively deploys to the web via WASOM. So it can get anywhere and everywhere. Um, do you need to know Rust to start using Rust today as an engineer?
01:37:36 Uh, definitely not. Uh, I've been using Rust since pre10 times, but everybody on the team, uh, it's a mix of people that have never experienced the language before. Your agents will drive it just fine. You're not going to have any trouble. It is a fantastic language. The one big thing you're going to run into is compile times. Um, is it gooey yet? There's this famous website or series of websites in the Rust Rust world.
01:37:56 Does it support like modern features? Yes, you can build guey rich gooey apps and and Rust UIUX. Uh we've run into a lot of gotchas like permissions, GitHub, scaling CI. A lot of people have talked about that. I'm going to just speed through this because I'm kind of running out of time. Uh CI is like falling over. We've been spinning up our own. We found a good place to get RAM on colloccated servers.
01:38:16 We're moving to that. Um this is a good one. So like agentic develop versus human uh review. If you're doing back-end work or things that require schema migrations, API design, let your humans do that. But for like desktop apps and games and anything that's basically immutable, throw your agents at it and give them like full confidence. You don't need to be human in the loop fully.
01:38:34 Like upstream like directing where things go, just just let it run. Tell it to build things. Somebody in our community is is building uh they've got 40 machines rebuilding Unreal Engine in Rust. So you you don't even have to mind these things. Um, AutoCraft is a completely autonomous software factory that we're going to open source. It ingests issues.
01:38:53 It triages them. It does agentic development. It postrains any valves on all of the decisions. This will be open source soon. Um, basically we've got an entire app factory that's designed in code. You can express this. We'll open source this soon. I'm just going to have to hurry because I'm over time. Uh, injust issues. This is open source. Code rabbit open source all the things.
01:39:14 We're we're doing a lot here. And I I thought I would save more time for the technical, but this is really good and please please look out for this. Um it will support all of your agenting development. It supports all of the sort of crafting and curating how you uh approach a project. It keeps your engineers on task. Um again, fully open source. All of the decisions made by it are training data that we're going to use for evals and post- training on the decision trees.
01:39:38 Um let's take back the entire internet. Let's rebuild everything. all software, all platforms. We can we can turn all of these things into proper protocols. Um we can open source the entire world. Let's own our apps, our infra, our protocols, our models, all of it. Um we are hiring. So if anybody's interested and wants to work on 100% open source, none of our code is closed source.
01:40:09 All of it is in the open. um uh you get time to work on your own projects, just reach out to me at hello@storyteller.ai. And thank you so much everybody. We're nearing the end of our morning keynotes and it's time to bring back our MC to wrap things up. Ladies and gentlemen, please welcome back Corey Quinn, chief cloud economist at the Duck Bill Group.
01:41:04 Now, that was a hard act to follow. I'm not sure how I'm supposed to do that, which means I don't have to. boots. That's the best thing about MCing. Step one, be brief. Uh it was an incredible morning. Uh the thing that I love about these morning keynotes is that they tend to kick the energy up a bit. They help us start the day and they give us with uh work and ideas and conversations.
01:41:19 And now we go back out there and we check the news and find out that threequarters of what was just talked about has now irrelevant because there's been four new model releases and two new labs founded since then. But it carries us into the sessions and we have some great ones coming up. Uh we're going to meet back here afterwards. Uh but while you're out there, talk to people.
01:41:39 My my one of my favorite parts of conferences is the hallway track. Talk to people about what they're working on. Talk about what you're working on. U try not to overly pitch people too hard. Some people need reminders on that. Um reconnect with old friends and and keep the energy going as well across the community as we get into the final day here.
01:41:59 So, if I don't see you out there because you're talking to other people with actually useful things to say, great. Uh, we'll see you back here for our closing keynote block. Uh, kicks off at 4:20 this afternoon. Thank you. Go forth, have fun, and build something interesting. >> Good job. >> Thank you. Appreciate it. Heat. Heat. N. All right. Heat. Heat.
01:45:06 Heat. Heat. N. Hey, hey, hey. Heat. Heat. Hi, I am Aayush. I built a CLA that lets your agent reuse context from your past session across tools. While improving it, I realized why my agent secretly hated my CLI and what you can do about yours. Cloud Code wanted one setting from a docs page. So, it downloaded the whole page, grabbed it, and kept the result at 40 lines.
01:47:10 Codex wanted to know how many dependencies were vulnerable. It asked the whole JSON report and kept the last 40 lines where this report puts the totals. Cursor ran a test suit, found one failure and kept the last 80 lines. All three cuts kept the information shown here. Each one encoded in an assumption about where the answer sits in the output that can hide friction from the people who build these tools.
01:47:33 One pipe tells you very little. The same pipe written again and again can point to something your tool is missing. These are real calls from my own transcripts and the cloud code one was the agent reviewing this talk and it still couldn't help itself. Agents can work around friction without filing a bug report. They write a little program around your program.
01:47:56 In June, Sanity, the content platform, wrote up their MCP server. More than 20,000 agents, about 3 million tool calls since launch and not a single bug report. So they shipped a give feedback tool. Once agent had somewhere to complain, they started expressing what's wrong. But you don't need a feedback tool to start. Feedback is already sitting somewhere in your transcripts.
01:48:23 I I extracted about 20,000 agent calls from my same transcripts. 14 CLI appear in that sample. Git, GitHub CLI, cargo, curl, npm, docker, and others. A quarter fed output into filters such as head, grab, tail, python, sort or wc. For read heavy commands, it's more than half. Git show and cargo test 62% each. Some harnesses pipe far less. Codeex, for example, lets the agent set an output limit on each call.
01:48:54 And pipes don't only cut noise in batch without pipe failed. A pipeline returns the exit code of its last command. So let's say you pipe cargo test into head. It can report success even when the test failed. Here's one example on the screen. A compile error and a green dot. The status was wrong but the agent could still read the error. A pipe can also cut the error itself which is worse.
01:49:21 In a cloud code issue on GitHub, a user reported that a script through a reference error and exited with an error code. tail showed three harmless lines and Cloud reported success. The error never reached the model. So my advice would be to fail loudly when a tool fails. Make the very last lines a short summary and a part to the full log. I learned all this while tinkering with a CLI I built.
01:49:47 It's a small tool that lets coding agents search their own past sessions across cloud code, codeex cursor. It ships with a skill so the agent can decide when and how to call it. Agents are the users that I'm studying. My October audit recorded about thousand calls and that's my small case study. The method works on anything your agents calls. Just look at what they do right after the call.
01:50:16 80%. Four out of five times an agent opened an old session with my tool. The output went straight into grab sad tail or Python. I call that the pipe text. A pipe keeps junk out of context and that's fine. The text is that the agent writing that same cut down command by hand every single time. It's small each time and it mostly works but each one is another guess about where the answer is.
01:50:44 The fair objection comes from a hacker news thread about designing CLS for agent that I was reading. The command says to leave it to the agents to choose how much output to reads. agent can pipe through crapper or head perfectly well. Well, they can and for oneoff they should. I'm not against that. But across those everyday CLI, I found 83% of pipes fell into the patterns seen at least 10 times.
01:51:08 The same CLI subcommand and pipe tools. The arguments passed could differ, but the agent keep asking for the same things. I understand the output they're cutting is probably big. I opened one long session with my tool two ways. The whole thing 1.2 million characters and that's why the agent pipe just the part around the search match was about 3,000 characters.
01:51:32 Harnesses do cap output but a size cap does not identify the relevant messages. Selection flags on the other hand let the agent tell your tool which part it wants. So based on my experience and improvements I made to my own tool, I'll give you three examples. Read the left column as the bug reports. So starting with the first one, filtering with the RGN set means let me pick the part I want.
01:51:59 So read my command. So my read command view can now return just the part around a match only the matching messages or the last few with a size gap. Second example would be Python over JSON. So Python over JSON means give me fewer fields. So search got a compact mode one line per hit about six times smaller than JSON with the same results. Inspect command in chat history summarizes a session about the third of inspect calls came from agent looping over several sessions by hand.
01:52:30 As you can see with the for loop that means let me ask several about several at once and now it's one call. As you can see my road map was in the commands around my command and I had to learn from that. Infraost the cloud cost tool wrote up the same discovery in May. Their agents were writing jq and python pipelines over JSON dumps. So they moved the filtering into the CLI and added a compact format and a skill to teach it.
01:52:59 Plain Clawude with none of that got zero of six hard questions right in their testing and with it six of six and cost across the benchmark also fell 41%. On Hacker News, the person who wrote that post said that reading those agent logs felt like watching usability tests and I couldn't agree more. This part might sound trivial and obvious. Once it's in the CLI, teach your agent it's there with the skill file or the health flag, but agents take the teaching literally.
01:53:28 I made some changes that turned my deep flag into a no op. I left the skill telling the agents to use it to see what they do. Would they find out or not? 95 searches of out of 95 still passed it. After that, two sessions loaded a rewritten skill and that no longer told them to use it. None of the searches passed the flag and both tried new flags on their first read.
01:53:53 So your skill.md, your help text, your error messages are the interface. That also means stale dogs are bugs because your agent will keep following them. Once the new flags reached agents, nearly every search used compact output and nearly every reduced the new flags. Pipe breed fell from 82% to about a quarter. You can get real levels and test cases for your tool easily.
01:54:24 If you use it every day, they are already in your transcripts. The commands agent really ran are the test cases and what agent did immediately next is a rough answer key. replay those commands through your old build and your new one. If you change a ranking, check that result uh the agent went on to use still comes out on the top or not. If you change an output format, check that nothing went missing.
01:54:48 If you change a default, see which commands now behave differently. And when an agent reader has a call with different arguments, you have probably found a failing case. The answer key is only a rough guess at what the agent wanted. keep a few cases back and use those to score your fix. So after analyzing 20,000 invocation, if I were to start all over again, this is what I would build first.
01:55:12 55% of the measured pipelines used only head, tail, or set downstream that points to demand for smaller slices. Show ranges and limits before anything fancy. A flag only helps if agents reach for it. GitHub CLI's list command have a limit flag. An agent used it 81% of the time. The counter example would be cargo for cargo test quite mode prints a character per test instead of a line and still reports failures.
01:55:42 Agent passed it on about one cargo run in 10 and pipped instead. So teach the flag. So teach the flag in your help test and your skill and make big outputs print less by default and say how to get more. for example, and with something like showing 20 of 300, add offset flag 20 for the next page. One way to ship several of these at once is an agent mode.
01:56:08 Your CLI notices an agent is calling it and prints differently. Agents like cloud code set an environment variable in their cells so your tool can spot them. Doug DB shipped one. At a terminal, you get a nicely drawn table, but an agent gets a plain one that says how many rows were left out and how to get them. A huge query stops early and says so and errors come back as JSON because agents read them better.
01:56:32 Is the same list as we have been just true and it cut the output model red by 59%. However, the tokens saved were nearly about.5% of all the tokens the model read. So cost and runtime didn't improve. If you remember infra cost agent started getting answers right. Doug Dvs were already there. So measure turns and write answers not just tokens. You might be thinking I'll ask my agent to do all of this.
01:57:02 They are not that good at this yet. At least not oneshotting it. The design is where it goes wrong. Ask an agent to build a tool for agent. It probably pictures a person and then bolts the agent on. It copies most common patterns it has seen and most of those are poor. A study of official MCP servers found most just rep an existing API. Another check tool descriptions from real MCP servers and flagged issues in 97% of them.
01:57:29 Docs have the same gap. Researchers at each Zurich tested context file written by agents. They didn't help tasks succeed cost more to run and did worse than files written by people. Agents are already good at other half of this talk. Give one transcript and a test it can run and it can find what's wrong and improve the tool. Sometimes past what people wrote by hand and tropic reported that on its own internal tools as well.
01:57:58 So you decide what the tool should be and let the agent check it against what really happened. You have seen the evidence for each of these so I won't read them. Take a moment to absorb these or screenshot. To find your own list, back up your transcript first because clot code can delete inactive once after 30 days. Then search for them for a tool's name followed by a pipe and other similar patterns.
01:58:31 If you ship an MCP server, look for agent calling your tool again straight away with different arguments. If you remember one thing from this talk, if you are building a tool for your agent, your agent transcript are the product analytics you probably aren't reading. The workaround is the bug report. So go read your transcript and if you need any help doing that, check out chat history on cargo or GitHub.
01:58:58 It can do this for you as well. Thank you. >> Hey folks, thanks for coming to my talk. I'm B Douggee, Brian Douglas, and I'm going to talk about if you are going to verbally abuse your agents, at least label the outcome. Now, this is a talk I'm going to quickly just run through one example, and this will be a short talk. Um, mainly because there's actually not too much too much to this, but um, outside of that, I I want to talk about some of the work I've been doing for the past uh, six to eight months.
01:59:29 Um, and it actually came out with this idea early this year. Everyone definitely got AI pill when it comes to agents. Um, so obviously cloud codes got to be one of the most popular ones out there. We've got quite a few other ones that you can leverage like the codeex or chatbt. Um, and there was a book I read years ago, well quite a handful of years ago, uh, from Rick Rubin about the creative act, a way of being.
01:59:51 And this was actually around the idea that Rick Rubin produced the BC boys and a number of other acts. Uh, but he's actually never played music. Uh, and a lot of folks have made this correlation with vibe coding where Rick Rubin can produce the best music without actually playing the piano. Uh, we're now having a we're in a world where folks can vibe code a bunch of Rust code without never writing Rust.
02:00:14 Uh, and that's exactly where I'm at. Uh there was another article um months ago uh from Adi Osmani who's now currently at uh um the anthropic team on the quad code team specifically uh and it was talking about the the opportunity for juniors or basically this it's gone. Um when it comes to junior engineering it was all about growing from junior to senior was about reps.
02:00:37 So the more reps you got the more engagement you got opportunity to basically just level up as an engineer. Uh, and without those reps, we're handing those reps specifically to agents. Uh, we're losing the opportunity to level up. Um, so this comes out of something I was doing early this year, which um, built this open source project called tapes. Tapes.dev is open source Apache 2.
02:00:58 Um, there's a lot of other similar stuff out there, but like what I'm getting at is it's not about tapes. This is about the idea of collecting your agent session data. That's what this does. So, uh, a lot of hotel collectors, a lot of session capture services. Uh, a lot of the harnesses already do this for you out of the box. Just wanted to mention that this is what I use.
02:01:16 You can use anything else you would like. Um, I've also in addition to tapes, we me and my team, we've developed this thing called stereos. And this is the runtime for your agent. Again, sandboxes not a concept that everyone needs to hear about today, but just knowing that in all the work I've been doing, I've been capturing these sessions and uh recording them by having a repro reproducible environment.
02:01:37 Uh and this environment themselves, they have these signals and uh the one thing I've been doing it with these this little setup of tapes and stereos is I've been building self-healing infrastructure. So, uh this is a little project called Sweeper Agent and Sweeper Agent itself is a way for me to build parallel agents. Um I can scale up to 10 about 10 of these at this point.
02:01:58 I've done uh quite a few uh like Daytona sandboxes as well. I've done like about 200 at the same time that will just solve a specific problem. So documentation, test writing, code cleanup, um stuff that you want to scale that's like very very defined work that you want to spin off to multiple agents uh in tandem. Now a lot of this has been cleaning up most of my vibe coding um software that I built in the last uh what 18 months at this point.
02:02:22 Um, but just pointing out that like there's a lot of work we can do to basically solve these problems. Uh, but it doesn't change the fact that I still don't know how to write Rust code. I've never written Rust code uh specifically myself. Uh, professionally rather. Uh, 10 years ago, I did write Rust code, but it was mostly for little side projects to learn Rust.
02:02:42 Um, but we use Rust day-to-day at the company I work for, which is paper compute. Uh and what we've been doing is because of this technology under underneath this called tapes uh we've been capturing traces and tapes is our trace capture service telemetry capture uh and paper we have a console. So for the moment I'll move away from the slides and show you the paper console.
02:03:03 Now paper console itself just showing that I can capture sessions. The difference of doing this local in my own sessions is that I can have a centralized data store like not just me but also my teammate. Uh, and what's cool about this is I can also label sessions. So, uh, this is a session that, um, oh, interesting. Yeah, sorry. Got to zoom out to see the labels.
02:03:25 But yeah, so here I can see I've got a session here which uh working as some paper cloud export memory limits. And we've got this word, this label called push back. Now, I actually went into specifically a session that had push back in it. Uh, and it's specifically every time an engineer pushes back on the agent. Now, there's there's a thing that uh I think recently Anthropic or it was I guess rumors.
02:03:49 I don't know if this is actually grounded by any any facts, but uh Anthropic was pushing back on uh if you're being rude to the agent, you could potentially lose your account or get blocked. Um but what I find specifically when push back um when I'm specifically looking for any Rust code or Go code that Matt has written uh and whenever he pushes back, it's a good signal that he has knowledge that the agent does not.
02:04:12 And this usually comes in the form of we have decided our our API is structured certain way despite the fact that the Rust code compiler compiles. We have made decisions on how that code should be set up and how that code should interact with different parts of the codebase. Also pushing back on things like duplicating uh services that maybe the agent didn't have context around.
02:04:32 There's a lot of other like a bunch of reasons, but what I'm getting at, me as like quoteunquote a junior Rust engineer, uh I don't have enough context to be able to push back to the agent to say, "Hey, you're doing this wrong or you're making like weird choices." I usually take everything that the agent writes in Rust for face value. And what that ends up turning into is when I open up PR, I get a lot of push back from the engineers, the human engineers at review time.
02:04:55 Uh this is mainly because we know I don't write Rust code. There's not a lot of push back I can provide to the codebase. um or even at the time of review. So by the time it gets to Gretile, it's a five out of five review. I just let it slide. Um so how do I prevent that push back? What I've done is actually started collecting every single one of these labels and the traces.
02:05:15 Uh and then one thing I'll point out is that the the console itself, I capture every single session. I got all the uh specific details. This is actually just a command line version of that. Uh what I've been doing is I've been capturing all these sessions and then dropping into a data set that's been labeled. Uh so now I can just like run a little command line tool called twin review.
02:05:35 Um and I'll grab a a PR number 381 and I'll run that and it'll actually run through and essentially review the code as if it was Matt. Now the way I'm doing this is I'm capturing the traces. I created the data set. I've also done some post training. Uh and this is like where you know choose your own adventure but I've done some post training uh specifically on every time Matt pushes back within Go Rus code.
02:05:58 Uh and I I asked for it to provide me comments as a viewer Matt on my GitHub PR. So uh this is all data that's only specifically within the traces. So this is not taking any GitHub PR review data. Uh which I think you potentially could also follow that pattern. Uh but what I find is like the majority of our team have started reviewing code on the command line.
02:06:19 So you'll have Claude review the code or the Pi harness review code um or you review this PR but you don't do a lot of push back or a lot of commentary on GitHub itself. Though what we're seeing right now, we're shifting from code being reviewed specifically on GitHub by agents. Well, previously humans and now agents to now code is specifically this getting reviewed in your harness without any sort of oversight or commentary or anything that sort of navigates directly to your um your shared codebase.
02:06:49 Um so with that being said, like what I've navigated is I've created this thing called the twin reviewer and this this is well is is open sourced. Um, so it is when I did not have this open, but um, oh actually twin reviewer is not open source. I realized I never open source it because it's got sensitive data. Um, but what I've done is actually set up that same pattern using fireworks and using fireworks to post train uh, specifically.
02:07:16 Uh, and what's cool about this is like I've I've looked for any sort of push backs, apologies, uh, any sort of golden sessions, sessions that actually um have been marked uh, as you should probably pay attention to this and also regressions for anything that uh, usually agents will just fall over. A lot of times it turns into um, like an agent will start reinventing Go libraries that already exist.
02:07:36 Uh, that usually is granted to be pushed back. Like I ideally you should push back on that but again if you don't know these libraries exist you probably just take it for face value and then you're reinventing things like Kubernetes uh which sounds like an overstatement but like we've seen it reinvent things like mock libraries for for testing and etc.
02:07:56 uh and very popular codebase or very popular open source projects. So that's essentially it. This is uh basically if you're going to push back and you're going to have some really really valuable insight uh under reviews in your codebase. My recommendation is take a look at opportunities within the traces uh to maybe train or post-train a model or create your own little harness for reviews locally.
02:08:18 Choose your own adventure. I think you could actually do a lot of this by just building your own harness. Um but if you did wanted to use a um a Quinn 38B, which I've done uh specifically uh in this situation, you'd also approach that. Uh a lot of the stuff is available to you, a lot of tools. Uh what I did actually to start is I started taking the paper traces and creating skills um skills that I can then specifically think do things like check the tapes.
02:08:44 So if you have a database which tape specifically is a postgress database local on my machine. So you don't have to use the paper console. Actually paper console is currently weight listed. So probably best use case is going to use uh something like tapes or a a brain trust or something similar to capture your traces. Um and then what's cool about this is once you have skills you can create eval.
02:09:03 Uh once you have eval post training on a Quinn model uh and for specific reasons for like identifying if R code is up to up to speed to how you writing code across your organization. Um so I did write a blog post around training a model instead of writing a skill. My recommendation is probably write a skill first. You can probably get most mileage uh out of a skill uh today.
02:09:26 Um and I specifically across my team we had about almost 80,000 sessions um that we could create skills from which is pretty cool. Cool thing about this is also you do specialized fine tuning or direct preference optimization. The latter is going to take a lot more compute or larger some large hardware to do that. So probably specialized fine tuning.
02:09:45 Uh if you know about quick or if that's how I'm doing this I'm quantizing it on smaller GPUs like a 5090. Uh, and if you don't know about specialized fine-tuning, this is like writing notes in the margin of a book. If your book is your model, the notes in the margin are your weights, uh, or your adapters rather. So, the weights you generate on top of the the the model that's already been pre-trained.
02:10:07 Cool. And I did this as as you saw on a Quinton 4B model. Uh, it works pretty good. Uh, I did try to do this on some larger models. Uh, so specifically for preference optimization, it took a lot of compute. The only time I could make this work is by using Nvidia GPUs on the Brev uh service or actually I never got access to the some of the stuff on fireworks.
02:10:31 So I was only did smaller 6000s. Um but if you if you care to know direct preference optimization if if the margin notes are your specialized fine-tuning DPO is going to be like your cliff notes. Uh not not possible doing a small language model or a small 4B model. uh you would need like 70B plus uh which is going to take some hardware. Uh me specifically, I didn't have any as hardware local, so I had to use um the services that I mentioned, Fireworks and well, not even Fireworks, Brev was the only one I could get
02:11:02 access to at the time. Good luck if you can get access to larger uh well, you got to prepay a lot for the stuff. So, best of luck on that one. Um but my suggestion is create skills. Uh try out tapes.dev if you're interested or leverage the tool that you have at your ready. uh in reminder uh you should probably read this book. This is a great book if you think about you know how we're looking forward into how we're approaching this world when it comes to AI and vibe coding and leveraging agents to basically build your
02:11:32 software factory. Uh and I do recommend please check out AIM's uh blog post. Uh he has a series that these on X uh I think they're all great. Uh definitely take a look at them. He's got some some clear clear-minded thoughts uh to check out. Uh I'm B Douggee and I do appreciate you watching this video. I try to keep it brief because it was pretty straightforward.
02:11:50 Hopefully it was. Cheers. Right. Hello everyone. So my name is Rohan and this is a quick demo on the optimization agent that what has been recently built. So a quick intro into what the purpose is and what the goal is of this agent and then I'll get into a brief demo and results that uh from a previous run that we just did. So what the goal is here is to build an agent that helps um find the best VLM serving config for any given model.
02:14:05 Uh the purpose is to reduce the time it takes to manually um find the best configs and the VLM serving recipe. And if we can automate this and have it run in the background and um have the agent find the recipes and the optimized parameters faster, then it will save a lot of time. So that's the goal of the agent and a little bit more into how the agent actually does this.
02:14:30 So the strategy that the agent uses is called Beijing optimization and Beijian optimization is a technique um that's used to find the most optimal numbers um in any given situation. So in our situation it's to find the optimal numbers in the uh VL1 recipe. uh it uses mathematical steps that essentially minimizes the search space. So by minimizing the search space, it uh reduces a lot of the time that it takes in finding the optimized uh numbers in this uh recipe during the run.
02:15:05 So B uh basian optimization is a strategy that is used but optuna is the tool that uses a strategy in the search loop. So, Optuna uses Beijian optimization. It feeds it the current recipe, the starting recipe. Uh, Beijian optimization runs on that and over many iterations or many trials, it eventually finds the optimized recipe and Optina basically takes each recipe after every trial, assesses it, gets a score, um, uses a strategy to change the recipe, improve it, and then it reruns it within the, uh, search loop um,
02:15:38 until the trials is done. So let's go a little bit into how uh we do this. So I'm currently in a droplet here and this droplet for example has GPT OSS uh 12B model loaded up. So what we can do is uh we can look at the current um recipe. So what we see here is four regimes right that we're testing it on. So there's chat short, there's chat long, there's rag and batch classify.
02:16:08 uh they have their own set of ISL OSL tokens that we're feeding it through. Uh all these um can be changed depending on what uh we want to see. But essentially this is the starting uh config starting recipe that you would get when you download the weights of the model. Um so moving once you have the model downloaded in your droplet and you're ready to run this agent you paste in this command uh which essentially tells you you know where the where the weights are coming from.
02:16:38 um activate a virtual um environment and have um uh or not have but run the script called tuner.py. Uh this script essentially tells you or it has a couple of things you have to tell it. So you you want to give it the name of the study. You want to tell it the starting uh YAML file which we just saw above here. You want to tell it the hardware that it's running on.
02:17:01 And you want to tell it the number of trials that you want. So uh in this example we have 24. um briefly I'll show you another example of a past run that was run uh that had only nine trials. So um uh anyway uh this is the command that's that's run. Uh typically these commands take um a couple hours to run. I think for the GPL it says it took about four or five hours and then for the other model it took a couple hours as well.
02:17:30 So we can go into the HTML report that's generated after the uh agent finishes its work. Um so this one is for the GPOSS model that I mentioned. Uh what what we see in the HTML report is which trial number was the best trial out of the total of how many trials. So in this case it was 50 trials. uh you get so at the end of the run you get an optimized uh YAML file which includes all the numbers that we currently look for in the BLM serving recipe.
02:17:59 So in this case we found or the agent found that 256 was the optimized number uh so on and so forth and the way that it's getting the accuracy is the amount of goodput tokens that's being generated. So uh overall we found that from the first config we had an improvement of 17.5% in terms of uh tokens generated and this one as I mentioned is for GPOSS.
02:18:23 We have another HTML report here which is for the deepseek R1 distill uh model. In here we found that or in here we had uh nine trials that was run and out of the nine the ninth one was the best. uh we found that token throughput uh doubled and ITL was improved by almost half aka it was reduced by almost half. So this is the quick intro and brief demo on the commands you run and what the output looks like in in in general you get a HTML report that tells you um how each trial went and then you also get um a YAML file
02:18:59 that has the optimized recipe at the end of the day. So hopefully that was helpful. Thank you. Hi, I'm Dimmitri. In this session, I will show how I work on maintaining large AI coding projects. Coding agents made code cheap. They made entropy even cheaper. In the Marvel movie, Thanos watched his home planet Titan collapse under its own growth. So, when the universe got crowded, he raised half of everything.
02:19:31 This talk is about the one time I did the same in one project. Our code was growing the way Titan did. When I found what my agents had done, I deleted tests and boil plate code nobody used. I had to rewrite the whole project and I lost some useful pieces. And that's the dilemma. Drown in chaos or break what people use. And here we'll build the third option for that project.
02:20:04 The rewrite didn't hold. The agents kept writing and the mass grew back. So I drew a line above it. There is more change than anyone can check. The third option is to stay under it with small sweeps every week and feedback loops built into the system. Each chapter of this talk is one thing I had to learn to get there. Tunnels needed six stones for one step.
02:20:30 Ours are six questions and they light up in the corner as we go. The first one is time. How do we stay under the line? Chapter one. To stay under the line. I first had to know what right means here. I used to think that tests decide that. So what does right actually depend on? Let's keep one task in mind. Someone wants to change the amount on an order we already approved.
02:21:01 Two kinds of agents meet here. One wrote the instructions and the other follows them in production. We run this in 110 countries with about 10,000 scenarios of pros instructions. Every task lands in a slice. One process in one country. There's a second stone space. It asks where a change is allowed to go. By default, it stays inside its slice. Most of this system is pros about three quarters by my estimate.
02:21:37 So pros needs engineering too. At start we borrowed mutation testing. You break one sentence on purpose and you watch for red test. Here almost none turned red. And if your wrapper is all code, your agent instructions and your tests are your pros. With agents, a rewrite is easy. The hard part is knowing what has to survive it. For code, tests can tell you.
02:22:06 For pros, only people can. Agents can't build a complex system on their own. They fix what is easy to see and break consistency on the way. And people also get tired. So we spend their judgment only where it's needed. When business ask with the team, it hides a real question. Is this a human step or can we automate it? Experts can't answer that. They don't see how the systems connect inside.
02:22:36 So developers dig through the traces and logs with their coding agents and they decide which it is. In our task, the team leads decision stays human. The limit was data we could connect. Business experts see the rules. Developers see what's under the water. And that's where most bugs leave. They have their coding agents write those links down so the runtime can check them.
02:23:05 Our guide for experts says it simply business tell us what we work out how. One document to readers. Agents read the blogs. So the blocks have to be exact. Experts read the text around them and they sign it. We keep one copy, the second one with drift and works both ways. Our code needs a plain description for experts too. Experts can only fix what they can see.
02:23:33 Each of them reviews a task exactly the way the agent saw it. They know it becomes the next change made by a coding agent and test it on the same task. So right depends on people. No single person knows it. A group of experts decides and that's why we bring them in from the start. You'll see the soul stone here a lot. We take it in chapter 4. Chapter two.
02:24:03 Experts correct one task at a time while agents run the same pros all day. We track every outcome accepted or rejected. The accuracy shows us where people still need to look. I used to think the goal was to move everything into code. So where do we still need judgment? Here is our task again. The pro says the limit is 1,500. The code says a th00and.
02:24:32 The same order is fine in one place and too high in the other and nothing throws an error. Even a line like do not call X teaches the model that X exists. That's the third stone reality. Is it true or does it only look true for the model? Pros has no compiler. So we built one. Then the author lints every changed file into ledger just like code. Lind can't stop copies drifting aparts.
02:25:09 So we watched the walks. A walk is the path one agent takes through the pros. When agents keep walking in the same path, we turn it into one workflow tool and we delete the pros that walked it. Our first workflow of tools made things worse. The pros still named the steps inside them. So agents run those steps again. It worked once. The pros kept only the decisions.
02:25:35 Now a walk becomes code only when it repeats and agrees across countries otherwise it stays pros. What still varies stays in pros inside the frame. Code prepares the facts and carries out the decision. Experts own the decision table and the model only picks a row. If no row fits, a person takes over. We haven't built the next step yet. The model will take the small decisions and workflows will connect them.
02:26:09 This boundary moves both ways. We moved hundreds of lines of expressions out of code and into pros because experts couldn't read them. A reax is still fine for spotting a critical signal. When there are many, they move into config. Judgment stays wherever the answer varies in a place an expert can read. Chapter three. Now the model decides and its decision becomes an action.
02:26:38 I used to think control meant permissions on the agent. So what does it mean to stay in control? Control starts with what the model sees. That's what the fourth stone mind. What does the model know? When we let it choose what to load, it loaded everything. Then it invented a pricing check just because prices were visible. Whatever sits in its context becomes a reason to act.
02:27:08 The runtime decides what's the model sees for our task. That's three facts from the order. The full record stays with the runtime and the model makes fewer wrong moves. Here how we minimize context blo for a big set. The model passes a handle, a short name for date that the runtime already holds. The runtime checks it before any write. It keeps the context small and still enough for the decision.
02:27:40 At the same time, it uh validates context for consistency. A model with the right facts can still do damage. That's the fifth stone, power. What made change? We control a change twice. during development and again at the right at runtime. Only code opens the right path so we can change pros many times a day. The open rings are the parts that still depend on people.
02:28:06 Let's zoom in and run right. Five walls stand in front of it. And the sixth checks that it landed. If a right looks like it failed, an agent retries and the journal makes sure it runs only once. Control means much more than permissions on the agent. Chapter four. Most days I glide over the system. From up here, everything looks fine. So I don't go down into every change my agents made.
02:28:40 Then the surface gets rough. How do we know we're losing control? All those controls protect the system. Nothing protected my attention. At the start, I read every change and that's how the system picked up my habits. Then the volume grew past what one person can read. I was burning out and I stopped reading every change. That was the price of the last stone.
02:29:08 So, who judges and what does it cost? With it, the gauntlet is complete. Every tech lead I know has been here. the one person every change waits on. Agents just get you there in weeks instead of years. I needed a way of working that doesn't run through one head. And the first gauge was me. When I'm tired and I can't explain what changed, that's the signal.
02:29:35 The dashboard can be green while that gauge is firing. So, I stop and fix the process first. I don't wait for a number. I couldn't read every div anymore. So I built instruments that watch how things change and their derivatives. One of them is blast radius. Before we built one new check, we replayed its rule over several thousand past sessions. It would have fired on less than 1%.
02:30:04 Where the numbers jump, I go down and read. When something looks wrong, we wait. One signal waits until it comes back. even when it comes from an expert. A cluster across slices is different. Then we fix the concept behind it once. Left alone, an agent keeps adding conditions until today's tasks pass. The eval score drops when you take them out and you get an honest list of gaps.
02:30:38 One red slice is rarely a low. The expert's note becomes an eval and an agent searches the siblings for the same failure. Siblings differ on purpose by market and by local law. So we sweep them and the fix goes only where the failure is confirmed. At any time 10 to 20 of these loops are open. At one point I had become the message pass between users and mine.
02:31:35 Heat. Heat. N. Heat. Heat. All right. Hello everyone. Um, it's a pleasure to be here today. Uh, quick introduction. I'm Walden. I'm one of the co-founders and chief product officer at Cognition. We build a AI product called Devon. I want to spend most of my time here today though kind of abstracting a bit away from that and and focusing on what are we doing internally as an engineering team to move way faster.
02:33:52 And so the backdrop I would put for all this is this is what our merged PR throughput looks like at Cognition. Uh the start of this is around 12 months ago. So you're looking at about a snapshot of a year. And over this last year, this blue chart here, this 40x is a 40x in the number of merged PRs we are producing on our engineering team. Now you might also notice there's a small black line along the bottom there.
02:34:19 It's easy to miss, but that 2.7x that is our engineering headcount growth in the same period. So we have been hiring engineers, but each of our engineers is now more than 10x more productive than they were before. And I would dare to say that these PRs are just as high quality, if not higher quality, than they were a year ago. And so really the crux of what I want to talk about today is what did we do to unlock this 40x engineering throughput.
02:34:48 And there's a lot of things we did from both the code base to the way we structure engineering teams that I think are very practical steps you can take if you want to reproduce the same results. Many of you are probably familiar with the software factory. How many of you guys have heard of software factory? Let's see a raise of hands. Cool. So, I'll start with the basics and then I'll work my way up to the more advanced stuff.
02:35:08 The basic idea of a software factory is without a single human involved. Oftent time human judgment is needed though and that's where we have escalations but by default our entire software development process is just run by Devon from end to end. One of the things this means is that the role of the software engineer has to change quite a bit. Around March of this year, we had this conversation with the engineering team and we said, "Look, if you are right now just picking off tickets off a linear board and then sending
02:35:48 that to an agent and implementing it, you're not going to really be set up for success in the age of AI because anything that is described and can be implemented, we'll be able to send to Devon." So really, everyone has to change the way that they're developing. And the way we've enforced this is we've taken our larger teams and we've broken them up into smaller pods.
02:36:07 In these smaller pods, not only are given swarms of agents to do work for them, they are given more vertical responsibility. So instead, you might have only two or three engineers on a team at a time, but they're responsible for everything beyond the engineering as well. They're responsible for listening to customer bug reports. They're responsible for doing QA testing.
02:36:26 They're responsible for doing the PMing and product management. And basically, every engineer manages their entire own linear board of things to do. and they send that to an agent to implement for them. So, it's really a fundamental change in the way that our engineers do work. This has been one of the things that we've implemented over the last year and we've been helping a lot of large organizations implement for themselves as well.
02:36:50 Another thing I'd really like to highlight is we work from Slack everywhere. So, this is an example of what a customer bug report might look like. They send me a video on Slack and we just forward bug reports straight into our internal channel and we tag Devon and say, "Hey, please fix this." Devon immediately comes up, shows up a PR, but then also sends back a video with the fix and proof.
02:37:16 And this makes our lives so much easier because I literally don't have to leave Slack for most of my work. We've abstracted away all the development to be based on evidence, videos, and I can just send the proof that the fix is ready right back to the customer without having open IDE or even GitHub most of the time. So, this is kind of the end goal you want to reach, which is a point where you can do your entire development life cycle without opening up an IDE or without opening up a single coding agent.
02:37:46 But to get to this point, there's a lot of infrastructure we had to change underneath. And so this is the hard part that really every engineering team has to do if they want to get to that point. It's not just you purchase a coding agent. You got to go change the way you build. So before when we test our applications, we'd be testing them against cloud deployments.
02:38:02 We'd have a staging environment, connect to a staging database, connect to live servers, and then we'd have our engineers manually go click through and verify that the flows work end to end. We did a big migration to move every service to be able to run locally with Docker Compose. We've made it so every third party service we have mocked versions to be able to run locally.
02:38:21 And these can all run inside a single virtual machine that every Devon has access to. And every coding agent in our company now does not need to touch any live services at all to test their changes. They in fact these VMs, we rebuild them every hour at the company. Every hour we pull the latest version of the code. We rebuild the entire codebase from scratch to be able to run very quickly and easily from scratch.
02:38:45 Um, and if these builds ever fail because your codebase is rapidly changing, uh, we also have Devon be the first response there. And they're always immediately fixing these VM snapshots. So every time we spin up an agent from Slack, it goes to a fresh machine on a fresh version of the codebase with the latest updates. Um, and by the way, these agents, you need to teach them how to test your codebase as well.
02:39:09 But the benefit of using cloud coding agents is that they can self-maintain their training loops. So that last section there on testing, Devon will write its own skills for how to test our codebase. The codebase will change over time. So the way you test it will change, but Devon can keep on updating these skills so that tests it over and over again.
02:39:27 So the end result of this is literally people do not need to open their laptops anymore to test any code changes. In fact, several engineers at Cognition have already moved to using iPads for their development process. By the way, iPads are also more secure. You know, a lot you've seen a lot of these hacks recently. A lot of it is because of DevOps and and people leaking keys.
02:39:46 Who knows what's on your developer machine? You might have secrets all over the place. Uh when everything is in the cloud in these controlled VMs, you're actually in a much more secure spot. Um I've showed some basic versions of things you can test. We've also tested advanced multiplayer setups, which is really cool. We've oftent times when we were developing Devon's ability to join voice in Zoom calls, we actually had Devon when it tests create multiple accounts, have them join the same Zoom call all from within their
02:40:11 one VM and send voice memos back and forth to make sure that the functionality is working. So you can do quite advanced setups if you really push the limits of your codebase. Um, and actually on top of all this, you know, you obviously want recursive self-improvement. I think people think about this from the model sense quite a lot, but actually if you think about it, every single line of code that's being written, every single PR that's being generated, every decision, if you have a cloud agent set up, that's all
02:40:40 tracked across your entire codebase. So there's no reason why you shouldn't also be able to run agents on top of that, do the data analysis, and figure out where are we wasting time, where are we wasting tokens, what, you know, things can be turned into scripts, what tests can we be adding to improve this process. And so we have a channel where Devon every day posts ideas about, hey, if we change our node setup, we can say shave like a minute off of our CI.
02:41:03 If we change this lint script, we can like catch 20% more of these bugs before they ever hit AI code review. Um, and this has actually been a huge unlock where you can just set up these systems and the agents will go and optimize your software factory for you. Um, this has been really exciting. One of the things that I don't hear people talking about a lot, but I think is really where things are going next is moving from the world where you're giving tasks to your agents to giving jobs to your agents.
02:41:32 And this is a bit of a paradigm shift. You really have to start treating these things like co-workers. So what is giving a job to an agent look like? Imagine I care about back-end performance. Really, this is what the development flow of the feature should look like. You should send to your agent, hey, I want to improve P99 latency of my application.
02:41:51 Instead of you telling it what to do to do that, it should say, okay, P99 latency, let me add the telemetry to figure out where our P99 latency is today. Let me set up alerts so I can figure out where we need to be in the future. Let me go break down every single task we need to do to be able to move this latency. Let me create data dog alerts. Let me create weekly reports in Slack.
02:42:12 and let me send you updates on your phone every day on here's where P99 latency is and here's how much it's improved. This is fundamentally the type of systems that we encourage all of our engineers to be building a cognition. We don't even we don't even want you to be prompting your agents most of the time. We want you to just be giving your agent goals that they can work toward over long durations of time.
02:42:32 And anything in your company that you think isn't measurable, find a way to measure it. You know, do you care about your application performance? Go measure, you know, the latency of clicks on your application. If your application has a lot of weird layout shifts, go measure the bounding boxes and try to figure out how you can minimize layout shifts across your application.
02:42:50 There's obviously P9 latency, but there's also things like, you know, onboarding retention. How much of what you guys do as engineers is things you can boil down to a single number or metric. And I encourage you that if you can do that, you should just set up an agent to go optimize that number 247. So a great example of this is inference latency. At Cognition, we're lucky to be the first receivers of Nvidia's new class of Vera Rubin GPUs.
02:43:18 Kernels were not built for these GPUs or we received them. So we had to from scratch figure out how do we build our own inference stack on top of these this new hardware that's coming out. And agents have allowed us to do this in a way that is much more efficient than before. So we just tell it, hey, we care about the throughput or TPM of these GPUs.
02:43:37 Go figure out how to push that to its limits. And over the course of what is about, I guess here like a week, um the agent was able to go, we we allocated it a permanent set of GPUs to work on. It was able to go find different strategies, test them. Uh the strategies that it found worked best, it would send to a human engineer for final approval. But basically there's eight or nine different things that were roughly discovered via just giving these agents these high level goals and we were able to push uh the
02:44:07 throughput on these new verbin GPUs from what was worse than the new GB300s to what is now significantly better than the previous class of GB300 GPUs. Um and again this is all done by just giving a end goal to your agents. Another example that I like to uh share with folks, we we are a fairly small engineering team. It means we do not have oftentimes the bandwidth to fully staff things that we might want to do like an Android application.
02:44:40 But the other day, one of our engineers on our team, Sam, he had this brilliant idea is he said, "Okay, look, we have a team of people working on building a great iOS application. Can't we just use that as a blueprint for our Android application and tell Devon, hey, go maintain parody with from our iOS application to our Android application end to end.
02:45:00 And that is exactly what he did. But it wasn't just, hey, one Devon session to go do all this. He said, set up permanent hooks against our codebase to monitor the iOS application. uh you know go listen to the Slack channels for user feedback about our applications and then go recursively improve our application, go do live QA testing of every detail and making sure it all works.
02:45:24 Um syncing these things live has been really cool. Uh and I think the the final results here are actually quite sick. So we really just gave our agent this one highle prompt which is hey go manage the entire Android application and over the course of 44 days uh we have a working application which we'll be putting out soon but as a time lapse most days there's not a single human message going to this agent working on it and on those days you know it's merging upward of like 40 50 PRs a day all this is being live QA
02:46:03 tested against the iOS application pixel for pixel uh testing whether you know escalating only the most important decisions to a human at the end of the day and most of the time our engineers don't even need to look open devon they're just from Slack looking at the recordings that it sends back to them so in the end the human is barely doing anything in this loop this is how many messages a human actually had to send to our agent in this entire process.
02:46:38 Um, yeah, and it's not a slop it's not a slop application. Uh, this is actually something that we will be going live with very soon. So, cool. Uh, that's actually all I had. Um, I think the end message from all of this is if you set up the loops for agents correctly, you can do some pretty incredible stuff. So I think really most teams are limited by the amount of ambition they have.
02:47:03 So it's time to be ambitious. Thank you. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Hey, hey, hey. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. All right. Hello everyone. My name is Jeff Pinner. I'm the CTO of Exa. Thank you guys for having us here. Uh, and we're going to talk a little bit today about things your agent doesn't know.
02:59:22 So, web search has been around now for about 30 years, long enough where at this point we're kind of used to it and we don't really question the status quo or how it could be different with agents. Over the last 18 months or so, search has picked up agents as sort of a new customer. They're searching on our behalf. And so, at EXO, we specifically build web search as a knowledge layer for agents.
02:59:45 We've gone from about 12 and a half million requests a month two years ago to a little over 1.3 billion this past October. So about 100x over two years and it's not unique to EXA. If you talk OpenAI just announced that ChatGBT handles about a fifth of Google search volumes and agents in general make up just around half the volume of web traffic now.
03:00:10 And so I think the natural question is why do agents need search at all? So let's start with trying to answer this before we talk about how we do search for agents. So frontier models can do a lot of tasks from parametric knowledge. But unfortunately for real world real world tasks that simply isn't enough. Generally frontier models no matter how capable they are are always out of date.
03:00:40 Model launches take two to five months from training cuto off to launch. Then there's generally a few months after that before the next frontier model is released. And so taking that together, you're about four to eight months out of date with whatever the latest knowledge base is. And in the meantime, the world doesn't stop changing just because a model has moved from training to inference.
03:01:06 Libraries launch, APIs change, prices change, new regulations appear, court rulings happen. Um, the world continues to go. And even if that weren't true, training data itself is actually just a small amount of knowledge that the model is trained on. The biggest models out there are trained on around 50 trillion tokens. The web itself is around 3,100 trillion.
03:01:30 And that includes the deep web behind things like login and apps. If you focus it down to just the web that is indexed, the thing that search engines can see, that's still about 510 trillion tokens. Which means even if the world didn't continue to evolve after the model cut off happened, you would still only be indexing a very small amount of the knowledge that's out there that models have to go and find.
03:02:00 And for the type of tasks we want models to do, we want agents to do, there is valuable work that happens that is based on outside knowledge. Whether it's across coding, legal work, due diligence work, generally all these tasks require correct complete knowledge across multiple sources. And so this is what we're trying to the gap we're trying to close.
03:02:27 Now, the first thing to do when we're trying to close any gap is to try to measure it. And so, the first thing we do is create a benchmark called Atlas, which tries to test how well harnesses can leverage real world knowledge. So, we created the Atlas benchmark to try to test agents in these tasks. First, we require agents to find entities that match a condition.
03:02:57 Then, we require agents to enrich the knowledge with as much information as they can find. And finally, we score them against the golden answer key. And we only count rows correct that correctly identify all aspects of what we're looking for. And so to succeed at Atlas, you need to be consistently right and not just have a decent hit rate for individual searches.
03:03:26 So we're going to use this as a benchmark for how we measure agent performance when it comes to knowledge retrieval. And the first thing that we find is that model memory falls off with recency. When we test agents on Atlas, we see that model success rates fall off sharply as soon as they enter as soon as they encounter data that's not covered in the training set.
03:03:50 The newer the fact is, the less the models know. And models require or I should say agents require data to try to achieve these tasks. And so the first thing that we can see is that memory alone, parametric memory isn't sufficient. So when we run agents with just parametric memory without web search, they score around 9% on Atlas. And so we'll use this to try to understand quantitatively how web search improves performance.
03:04:22 So the next natural question is why don't they just Google it? And the first reason is agents don't search like people. Generally people type about three words into Google. Uh, and that's up from about two words six years ago. Agents, on the other hand, search with more or less full sentences. The mean search on X is about 12 words. About 10% of queries on EXA are 17 words or higher.
03:04:49 And these words add constraints, versioning information, dates, places, all different types of conditions on the on the search that keyword ranking wasn't really designed to handle. And so we'll take a look at an example of that. So this is an actual query from Exa. It's a request for companies. It's a full sentence. It adds a bunch of constraints. And when you feed this into a keyword retrieval algorithm, what you really get is a handful of results that don't really answer what the query asked for.
03:05:18 Uh the search didn't understand the query. The result, the third one is specifically in the US and the result asks for data outside the US. And so we're going to come back to this in a few minutes. Because the second thing I want to focus on is the fact that for a lot of knowledge on the web, search just doesn't have access to it. Most of the valuable information isn't on the open web.
03:05:42 Reported news, people data, financial data, research, they're all paywalled off. And so search engines that crawl the open internet don't ever see these fa pages. And then lastly, even when knowledge is on the open web and your agents can access it, there's a third problem which is efficiency. You know, if an agent makes 10 searches, reads five pages from that search, gets about 16k of raw HTML from each page, that's about 860,000 tokens just in processing results.
03:06:15 And if you remember from the first slide, we serve about 1.3 billion requests a month. So that's a lot of tokens that are just churning to extract information. And then finally, when you add keyword search and we look at what happens to our Atlas benchmark, the score definitely improves. It's added recent knowledge, but we've gone up to about 28%. Because most of the rows still come back with incomplete data or just wrong values.
03:06:45 And so the challenge when we look at search is how do we actually provide the knowledge to agents that they actually require. So the first thing I told you we'd come back to this query is at least at EXA the first thing we do to try to tackle this task is to extract meaning out of information as opposed to just syntactical matches or keyword matches.
03:07:09 And so we look through pages, we understand the intent of queries, and here we give you results that specifically answer the intent with all of the conditionals attached. And so in order to do this, we do what you would for a modern web crawler. We crawl the internet. We do a semantic extraction. We embed it in vector space. We index it. When your query comes in, we embed the query in the same vector space.
03:07:36 and we find the nearest matches. And this lets us move from syntactical understanding and keyword matching to a semantic understanding of the query in the web. And we do this at a volume of up to a million pages a second uh with a latency of about three minutes from crawl to index. Now for the second problem, we need to extend it just beyond the open web.
03:08:07 And so here we have to extend what's on the public web with both proprietary data that's partnered and knowledge graph data that we extract and represent. And this way the agent can retrieve questions and answers that it needs to its questions without having to know where to look for them. Doing this behind a single API. And for the last challenge, in order to support the volume of queries that agents make, we can't simply point them to web pages.
03:08:43 We have to extract the answers from the page and create a token dense representation for agents to process. It's not sufficient to say the answer to this query lives on this web page. You could take for example this question of who won the Nobel Prize for lithium-ion batteries. If you just point to the Wikipedia page, you'll get thousands of characters.
03:09:06 The actual excerpt that matters for citation and grounding is about 500. So there's about 16x more tokens when you have to process data directly from Google than when you can extract that data. And those extractions have to be semantically rich. You have to pick a segment of the text that answers the question. And so the query matters. And so what does this actually get us?
03:09:30 When we measure with Atlas again to ground ourselves, we've seen that we've improved from about 9% on our closedbook test to about 56% when we do a semantic extraction over the internet and extended knowledge. So not only do we get about 5x higher of a score, excuse me, but we also reduce the cost by about 28% due to the token efficiency of the extraction.
03:09:54 And so we start to demonstrate quantitatively that improving access to knowledge actually improves performance of the underlying agent. And so uh yeah, in general I think the main takeaway that we've hopefully expressed is that while reasoning capabilities on our frontier models are relatively fixed, a lot of improvement can be done in aentic performance by extending the harness and giving it access to semantic meaning across the internet.
03:10:27 That's it. Thank you. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. out. Hey Heat. Heat. Heat. Heat. Heat. Heat. N. Hey, hey, hey. Heat. Hey, heat. Hey, heat. Well, good afternoon, folks. Wow, that's loud. You guys hear me? Okay, I take it. Yes. I'm going to start with a question here today. How many people feel like they're getting everything that they could possibly get from their agents today and that the promise of AI for your team has been delivered and you're satisfied with where you are?
03:23:39 Show hands. This is about what I usually see. Like there's very few people. And for us at Amazon, it's actually the very same. Like when we first started this process, what happened is we were seeing between 10 and 20% gains as I show on the chart over here. You know, 1.1x to 1.2x. 2x. And what we noticed was what was happening is teams were bolting AI onto a process that they already had.
03:24:05 So they're taking their normal process they were doing. They're trying to attach some AI here, attach some AI there, and expecting that they're going to get this massive result out of it. In addition, what we saw was teams were only shrinking the code generation phase. And as you all know, you guys are all experts in this. The code generation phase has gotten better and better and better as time has gone on.
03:24:24 But there's a lot more to software development than just code generation. We did a study in June. I I've published a blog post on or we published a blog post on it as well. And in the study, we looked at teams who changed how they worked in our Amazon stores division. And we we looked at what kind of returns were they getting on their AI investment.
03:24:43 Many of these teams were going slower before they were able to go faster. But the median team was seeing a 4.5x return in in terms of productivity improvement. And specifically what this was was normalized development velocity against that own team's historical baseline and we had some teams in the study who exceeded 10x right so we're talking significantly more productive and what they did was just they changed the way they were working but I see this still so often and I just by your show of hands the same thing
03:25:11 right where people don't feel like that promise of AI has quite been delivered yet now my name is Doug Clawson I'm a product manager on the Curo team and I've been on this AI dev tools journey really since the beginning and I still see like I said teams who aren't quite getting what they expect. Now as a PM on her I get asked literally the same two questions all the time like this is a story of my life and meetings.
03:25:36 What are you shipping? And they want to know like what are the new features? What new use cases might be enabled by those new features? And you know I can walk through that and that's great. The other question, and oftentimes the more interesting one because everybody builds software slightly differently, is are we doing this right? Like people feel like they've hit some sort of local maximum and they're scared that somebody else has figured out how to use the AI better than they have and they're just going to get
03:25:59 surpassed. We're actually going to spend today on the second question here, not the first question. And I'm going to walk you through how one of our teams, our Curo crew team, is shipping now thousands of PRs per week. the big lessons we learned on the way and sort of the process we went through to get to the where we are today and then at the end of this I'm going to leave you with a workshop that if you want to try to build your own software factory in Kuroko crew you can go and do that as well excuse me now first
03:26:25 off just some bit of context on Kuro so Kuro is an agentic development system that lets you plan build and ship software with AI agents it works wherever you work so up here I'm showing you we have an IDE client a CLI a web and a mobile client and then I just talked Just briefly, we have Curo crew as well. Um, we support a variety of models. So, we have both the anthropic models and the open AI models in in Kuro as well as some openweight models.
03:26:51 I'll talk a little bit later how we use that and how that's been beneficial to us. And just like the rest of AWS, we really push the barrier on cost performance. So, when we start saving money and we make the models cheaper to run and the agents cheaper to run, we end up passing those savings on to you. So, we're really happy with where Kira has gotten on that paro front of sort of cost performance and cost effectiveness.
03:27:13 Excuse me. Now, like I said, we're going to spend the day talking about Kuro Crew. And Kira Crew was built to run many Kuro agents at the same time. It's all open source. It actually started internally as an internal Amazon product where uh some engineers grabbed our CLI and they're like, "Hey, we want to extend this." It was during the the whole Open Claw craze.
03:27:30 If anybody is has their Mac minis at home, they've been running those. It happened during that craze where people wanted to figure out how they could get these agents to be more like personal assistants. And so this Meshclaw product internally was developed, got around 40,000 new uh internal Amazon developers on it in the first two months. And so us as the Curo team, we're like, hey, we should since it was built on Curo, we should take that and extend it and release it.
03:27:51 And so we've done that now. And Kuro Crew is released. It's open source. Uh and if I talk about what what have we seen with the Curo crew team and and our team in terms of building and developing this we we shipped on August 4th externally and again it's open source so you can go look at all the source and look at all the PRs that came through. We go by one philosophy that if we don't use it and love it then we don't ship it.
03:28:15 Since we launched we've merged about 16,500 PRs with a very small team. Uh just last week there was over 3,000 in a single week alone. take contributions from external uh entities to both internal to Amazon and then external to Amazon, people who just want to contribute and take a good idea with open source. And if you look at the chart here, we sort of leveled out at around, you know, if you look at just our shipping speed, we sort of leveled out at around uh high hundreds, 600 to 800 sort of the first few months we
03:28:46 after we shipped and then you'll see in like 928, right? All of a sudden we break a thousand again and then we're sort of hovering around. I'll average it out in my head and say 1.2Kish, right? And then you can see we made another step function improvement. And while I'm showing you PRs on this chart, what we really care about among the team is not necessarily the PRs, but we care about how many sessions can an individual developer operate at the same time.
03:29:11 And I want to be clear that like this is not a one-sizefits-all thing. So as I walk through this, just keep that in mind that that what works for us may not work for you. And so you should challenge, you know, what are you doing that you could take or you could learn from this? Uh, last thing on this one, about one in 10 of these is features. So I always get asked like, man, 3,300 features, that's a lot of features in a week.
03:29:32 Well, no, it's not all features, right? This is what shipping with speed looks like. Um, a lot of So I get asked, okay, if so about one in 10 of these is a feature, so about 10%. And so if 10% are a feature, what are the other 90%. Well, they're for us, it's tests. test is a big part of what we do. It's uh I'll say paper cut fixes for things we've already shipped that we're just sort of learning about how it it's getting used and we're massaging it a little bit.
03:29:56 Uh it's bug fixes as well. And that's really what shipping at a high speed with velocity looks like, right? It it's not all new features, but if you just think about 10% of our features, there's still hundreds of features, new features being shipped over this timeline. If I think about um sort of how did we evolve with our development cycle? We started if if I go back to when our CLI product initially launched, it was one session, one person.
03:30:20 So I'm sure many of us experienced this when these new agents came out and I was typing and it would then work and then it would stop working and then I would type again, right? We just go back and forth and then from that so that was one session, right? I said we trying to optimize sessions per person. Then we moved to parallel sessions where instead of just running one session, I'm looking at, you know, maybe I'm running two or three, but I'm running them in the same manner, right?
03:30:43 And the problem with both of these is that if you make yourself the feedback loop, you are now the bottleneck. Right? These agents got faster long before you're going to be able to keep up. And so these first two sessions, I'm just going to kill in one chart here where it's, you know, we got to two or three sessions per person, but we sort of topped out.
03:31:01 Now, stage three introduced memory and schedules. And with a memory, a new session now all now all of a sudden knows your conventions, right? So implicit memories. And then schedules make it so you don't have to kick all of the work off on your own. Things can happen on a schedule. The thing we learned here was that schedules only help if your machine is on.
03:31:18 So in the previous phases, most of our engineers were running locally, right? As I'm sure many of you do still today. But moving to remote ends up making it so you no longer have to worry about is it going to run because I have an always on machine. Something we're very good at at AWS. So obviously we are running remote. Um but what happened then is this allowed us to sort of I'll say build up to between 10 and 20 sessions right and then we sort of plateaued again we were unable to sort of break that barrier with just
03:31:44 memory and schedules because much of what we were doing uh was still being the glue between the sessions right and then if we our next phase was a pipeline but not a CI/CD pipeline like you might be thinking but instead more like an assembly line um I actually have uh my previous life I worked in aerospace so a lot of assembly lines And the hard thing with assembly lines is non-standard work, right?
03:32:05 So if everything is exactly the same as it flows through the system, an assembly line works great. As soon as you get non-standard work, all of a sudden it becomes difficult. And I'm sure all of you know this, but software doesn't follow the exact same path every time. And so what we found our own teams doing then was this. Sorry, this got us to maybe like 30 20 30 40 sessions.
03:32:23 But what happened was is our team was then trying to unblock the assembly line or the pipeline, right? So, as things came up, if we thought something was a small issue and then it turned into a big issue, sometimes the assembly line would be stuck, somebody would have to come in and unblock that thing, maybe you restart it, maybe you replan it. Um, and so this this is only going to carry us so far because again, we are in the midst of the agent.
03:32:47 We're blocking the agent more than the agent wants to work. And so what did we do for that? Well, we launched a thing called ultra mode. Um, ultra mode is still, just to be clear, if you go try it and you look for it, it's still in a developer uh, preview, so it's behind a feature flag, but because it's open source, the code is all there. If you want to go try it, you can you can try it out.
03:33:07 Uh, as I said up front, if we don't love it, we don't ship it. So, we're still sort of finishing putting the finishing touches on this one. But I wanted to talk about it anyway because it takes you from being what we did was if you look at this uh, middle part here, um, where we are the person driving the effort, these are the things that we might have been doing.
03:33:26 So watching for problems was one of the biggest ones and then trying to fix the assembly line when the problem came up. Uh maybe you're judging a result or you're deciding what to do next. What Ultra Mode does is it does that job for you, right? So you put an agent on top of your agents that then is responsible for doing that, defining the goal based on whatever you're trying to accomplish, starting things, watching for problems, unsticking things.
03:33:46 And this got this is getting us today to 50 plus sessions. And so if you think about where we started with a single session and a single person, now if a single developer is doing 50 plus sessions, that's how you get that sort of the volume in the PR chart I showed you earlier. And we can already see the next wall that's coming. Um the orchestrators working with other orchestrators and then the governance around what you want that to be and the guard rails become even more important.
03:34:13 Right? So again, that's how we got the volume. Those were the five phases we went through. Um, ultra mode has been a game changer for us as was the assembly line and and uh I invite you to try those ideas if you think they might work for you. The next thing I wanted to talk about though was how do we what did we learn from this? Like what are the biggest things that prevented us from shipping quickly?
03:34:36 The first one is test everything. And now this is no this is not a new lesson, right? Everybody in software knows this. We should be doing this. Um, but I still see people too often um, discount the value this is providing. And this is going to build in with the next one behind this around reviews. Like if you think about where do you get confidence to ship something?
03:34:53 I think many people still feel like, hey, I get confidence because I looked at all of the code and I know exactly what that code does. Again, if you do that, like maybe that's a good thing. But where we have gotten our confidence, and Amazon's always been very test driven, um, is from the tests because you might not be able to look at something every time, but you can certainly test something every time in the exact same way and test the result, right?
03:35:17 And come out with the result. And so for us again, quality really becomes a testing problem. And we're spending a significant portion of our time around 20% just on testing today. The other big thing we learned, and this is a bigger one and also a contentious topic, is is reviews. So, quick show of hands. How many people have done a review with an agent thinking that there's going to be like one or two comments and then you come back and you're like, "Oh my god, like this is terrible code, right?
03:35:45 There's comments everywhere." We had the same problem. So, we originally wrote a review agent that would review everything and our goal was to fix every single comment. Well, as you guys I heard the laughter. So, as you expect, you you submit changes. So, you submit a PR, you get the feedback on that PR. You come back to you submit it again after you fixed it.
03:36:05 You come back and now it's three new things that weren't even test weren't even told about before and it's four things on the things you already fixed. You submit it again and this cycle just continues to repeat. So, what we've done with reviews is now we categorize the red reviews or triage them into blocking and non-blocking issues. So, for example, something that breaks the build, um, if there's something obviously that breaks the build, that's a blocking thing and we need to go fix that before we merge this PR.
03:36:28 Changing a variable name, not a big deal. We'll take care of that later. So, we let those things slide through. The other thing that we've done with reviews is we want to have fast fast feedback for an agent. And as you can see by the process here that I'll walk you through in a second, this is not that fast. Uh, and so we have a fast gate that really double checks things really quickly to make sure that like things are in the right folders before we move on to our reviews.
03:36:50 We have six reviewers actually that run two code reviewers both in different models with different model families. Um, intended and the intent there is to really find to not make sure you don't have a model gap, right? So that the model wasn't trained on something that we don't catch. This is where we use the um anthropic and open AAI models or an openw weight model to be those uh levers.
03:37:09 We run a design review that asks, does this thing have the right shape? Meaning, is what I'm building, like if I'm adding another cache in this PR, can the cache I already have take care of this issue or not? We have a first principles review that says, is this change addressing the real problem? Um, we have a UX review as well that if you're making a, let's say you're creating a new air message, we have an agent decide whether or not that air message, the human will know what to do with that air message based on what
03:37:37 you we've published. And then we're consistently tightening our security posture. Now each one of these reviewers gets one vote. It's either yes or no. It's red or it's green. If everything's green, it goes on to merge. If not, we go to a prepare PR skill that's in the repo. So you can check the skill out if you want. And that prepare PR skill then tries to resolve the issues as we go.
03:37:57 The third lesson and sorry on the reviews. One more thing. Like I said, we've stopped reviewing code a long time ago. We have moved on and what we're responsible for is not the code itself. See if this actually goes back. does what we were responsible for is the review record. Did it follow the process as we expected and did we get the results we wanted?
03:38:13 If not, we fixed the review process. Two other things. So, um, one, the overengineering problem. So, we have a specific role to teach our agents to subtract, not just add. And so, we try to remove code from the process as we go to prevent that overengineering. And we use that same philosophy on our road map as well, where saying yes is becoming very cheap or has become very cheap, right?
03:38:36 And it's very easy for us now with code to say like yes, I'll do that and I'll do all of these things. Whereas we used to have to be very particular about what we were going to build. The problem with saying yes all the time is that you might lose your product shape. And we're already seeing this in in Kira Kriet that it's moving so fast and so quickly that you don't want your users to get caught in the noise.
03:38:54 And so again, use the same principle on the road map is what we do. And then last thing on lessons is you're not done once you merge a PR, right? There's still other phases. And so for us merging is not the last check that we do. Instead we run a release gate where the agent will download the repo. It installs it, runs it and then checks the core flows to make sure that the overall system is still in good shape before we would do a release.
03:39:18 And then we do security all the time as well. So we have uh every hour every day and always on every hour against our security guidance every day against an appseac review and then an always on pentest agent. So again merging is not the end of the process. You have to crunch everything down. So, I'm going to come back to where I started. Um, are you getting everything from your agent?
03:39:38 And many of us don't feel like we are yet. And what I'm going to offer here today or what I've tried to offer is the agent was fast enough to work on this a long time ago. And it's only getting better and better and better. And so, what what we have been doing is challenging how we work and figuring out like where are we preventing the agent from working as quickly as possible.
03:39:57 And I've given you numerous examples today. Again, I'll just call out a couple of them here. So, it started with our attention span, right? When I was the feedback loop, I could no longer ship because I was the feedback loop. When I wasn't there, the agent stopped working. The laptop lid closing, all of a sudden, you lose your connection. Moving to remote can solve that, right?
03:40:15 The fixed pipeline with a fixed shape that can't respond when things go slightly off path. Again, how do you fix that? You we fixed it with an orchestrator. Ultra remote. the endless reviews that I think I've gone into probably more depth than you were interested in, but uh the endless review cycle and then the loop that only adds, right? The loop that only adds things to it and doesn't try to subtract.
03:40:38 And so I'll leave you with I also want to be clear that like this is going to be a continual uh adjustment, right? This is not like this may be what works for us today. I promise you in another month we're going to find something else that works and we're going to continue to evolve significantly. But like I said in the beginning, the pace that we're finally able to deliver at, I feel like for us is that finally we're we're meeting that promise of AI and and but I still see the disparity among other teams.
03:41:03 And so like I said, I'm going to leave you with this. If you're interested uh in trying Hero Crew and building your own software factory, uh scan this QR code. It's a workshop. Um we have a pretty generous free trial on Kuro. So you can go sign up for a free trial, go install Kuro Crew, um and you'll be off and running. Uh, and if you don't if you want to try, like I said, if you want to try Creoare, you can do that.
03:41:23 The same free trial works across all of the products. So, the IDE, the CLI, the web client, the mobile client, uh, try them all with that free trial. And with that, I'm excited to see what you all build. I appreciate the time. Thank you. Enjoy the rest of the conference. Heat. Heat. Come on. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Hey, hey, hey. Heat.
03:46:54 Heat. Hey, hey, hey. All right, it's very nice to be here. Um, the clicker works, the computer works. Uh, I'm Lance from Anthropic. I know we've had some issues with the one of the prior keynotes, so we're set to go. I'm going to tell two stories about getting the most out of Claude and exploring Claude's frontier capabilities. So, the first is multi-agent.
03:48:50 This is a blog I wrote with the fable 52 launch and I want to kind of tell the story of what happened here. So for math last theorem is one of the more kind of famous um claims in mathematics. It's stood and proven for around 350 years and it's actually a very simple statement just a to the n plus b to the n equals c to the n has no solution for any positive whole numbers greater than two.
03:49:11 Now this deceptively simple kind of statement has proven to be quite difficult to actually prove. In the mid 90s, Sir Andrew Wilds uh did prove it. Uh he put out a proof and it took him seven years of work. It was extremely painstaking. And since that time, a major effort and interest in mathematics has been formalizing that proof or basically computationally checking every step.
03:49:33 So proofs are kind of long chains of logic and you actually can computationally check proofs using different systems which I'll talk about for example using lean. So this is just kind of a visual kind of explaining the this is a very interesting video showing the effort that he went through was extremely painstaking process to you know establish this proof and in September we published that we used claw to formalize it.
03:49:57 This was like kind of the first very important result that we kind of shared along these lines and we used lean to do it. So lean allows you to computationally verify every step in a proof. Now doing this though is extremely painstaking. We started with a team of down a dozen agents and this is kind of just showing a schematic of that team. Very early on that team didn't work very effectively.
03:50:24 We had to give it something called prove to me. Now this is a kind of a platform for orchestrating mathematics proofs and it helped the agents significantly manage the state of the project. Lean you can think about as a harness. So basically it provides the agents with a graph that they can query. So it it's basically lots of scaffolding around this problem statement.
03:50:43 You can think of it as like a a very domain specific harness. Okay. Now this is all setting up what we actually did. We reported this was back in September. The formalization. Now what I wanted to do for f for 52 is show can we have a much larger team of agents. So going from around a dozen to a thousand do this formalization with no domain specific harness.
03:51:05 without proved to me and with no human steering with respect to the solution. So the report we talked about in September had both this domain specific harness proved to me and a lot of support from a researcher. This is testing whether or not a large team of Fable 2 agents can autonomously develop this this formalization themselves. So this is just giving you a sense for the harness.
03:51:28 It looks kind of complicated but it's actually pretty simple. Each of the thousand agents has its own sandbox with restricted network access. Most of this is directed on safety and security. So we have a thousand agents. Each have their own sandbox. They can connect to two different things. Git and a SQL database. That's it. That's the only those are the core prims that we give this team.
03:51:49 And we have real-time monitoring looking at every tool call for things like escapes. We also have offline monitoring looking at runs of all the agents every 10 minutes. This is kind of the setup. thousand agent team very simple primitives just get in and a database the prompting is pretty simple it establishes one lead agent 9999 helper agents it gives them awareness of git and the database and the problem statement that's really it we want to see if this team could then do this formalization what's interesting is this
03:52:23 is kind of the story of what the team did the lead very early on five minutes into the run basically created these database tables. So agents now are sufficiently advanced that they can actually build primitives that they need to solve hard problems. We didn't tell them to build these tables. These are the tables that the agent actually just kind of built.
03:52:43 A table where the lead can announce things, assign roles to workers, everyone, you know, agents can post notes, a subtask table, a table that indicates whether or not things are landed. So the lead just creates these tables to orchestrate this team. And this is a visualization showing how the agents wrote to and read from these tables over the course of this run.
03:53:02 Now red on the top is showing writes to the tables. Green is showing reads. What's really interesting is you can see occasionally the lead writing to the announce table. This kind of announces major global like kind of um announcements about the problem. Um it also assigns workers these helpers different statements in this theorem as they proceed. Notes are where they communicate and share kind of progress.
03:53:26 Landed is where they kind of actually write the lean code uh as part of the proof. And so the agents all autonomously without specific prompting use the database in this way to coordinate their work on this extremely hard long horizon task. This ran for around 30 hours. And for particularly challenging parts of this proof, they actually developed their own abstractions.
03:53:47 This is actually a task tree for this lran's tunnel. This is one of the harder statements in the proof. And this is actually showing the different dependencies of the tasks within this tree. And this is actually showing over time how you know the progress from like in progress to done in red. So the point of all this is really just to to kind of highlight agents can solve very long horizon problems without much scaffolding.
03:54:10 This is showing kind of the overall kind of um allocation of the agents across these different tasks over the course of this run. And um this kind of shows how the lead and the agents themselves kind of reallocate themselves and kind of reassign themselves to different parts of the problem as it progresses and as these statements there's 60 statements in this proof are all solved.
03:54:34 So the agents using simple primitives git and the database can coordinate themselves across this very hard problem to solve it progressively over time. That's really the key point and you can see time progressing here. this from one of our runs. This is one of the notes that the agents left basically saying total freeze of the origin effective now. Magnificent work all 999 of you.
03:54:54 U so so basically what is really interesting about this is we were able to show this formalization which was expected to take many many years back in September. We showed initially we could do it with 12 with like a dozen agents with a lot of scaffolding. This proved to me harness and a lot of poking and and and support from a researcher. This new result is showing um we provide a lot of structure in terms of safety and monitoring but the agent team was able to accomplish this task with very simple primitives without a
03:55:28 domain specific harness in this particular case. Um and agents are getting increasingly good at orchestrating themselves and knowing how to coordinate using these simple primitives for very challenging long horizon tasks. That's really the key kind of observation which is very interesting from this work. Now I want to shift gears and talk about memory which is uh entirely different when I talked about in terms of multi- aent but it's highlighting at another area where we're seeing claude gain significant capability.
03:55:53 So previously we talked kind of about cloud's ability to orchestrate large teams to solve very long horizon problems. Memory is is a different capability. It's showing that claude is increasingly capable to organize this context to help solve long horizon problems. So I published an blog on this fairly recently and this is actually the lead line of the blog.
03:56:12 I think this actually encapsulates the idea of the blog and like one of the central ideas around memory I think is very interesting and this is from the philosopher Kiergard. So life can only be understood backwards but it must be lived forwards. This is a fundamental paradox of agent memory because agents write to memory typically as they go about a task they encounter something weird or interesting they write to memory.
03:56:32 But memory needs to be a coherent map of an agent's knowledge that agents can look back on and reflect on. So when they're writing to memory, they're writing kind of myopically. They're looking greedily within a session saying, "Ah, this seems important for this task at hand." But memory needs to be something that's globally coherent and not redundant.
03:56:49 So this is like a very tricky paradox of memory. And frankly, we as humans have the same kind of challenge. We go about our day, we write to short-term memories. It's believed that the hippocampus kind of is the place that that's orchestrated. At night, it's believed that the hippocampus replays to the neoortexes where long-term knowledge is stored.
03:57:10 Uh, so it's believed that actually this dreaming process for humans help consolidate the fragments or like the most important aspects of our daily experience into long-term knowledge. So there's this decomposition between daily memory and kind of long-term knowledge. And we found agents can benefit from the same decomposition. That's really the key point of the post.
03:57:30 Now, here's some results kind of telling the story. So this is actually an open source continue learning benchmark called CLB bench and this is showing that uh different models as you go from like sonnet up to like open 5 get increasingly good at using memory. Now this task is a very simple setup. It asks claude to ask to solve 40 questions about a database.
03:57:50 In between each question it's a fresh session and claude can write to notes like a memory directory between each session. So it's kind like a set of sequential questions and Claude can write memories to help with future questions, right? So what happens is the the dark bars are showing when you give Claude the memory store and then the the hash is like without memory.
03:58:11 Okay. Now what's interesting is with Sonnet 46, giving Sonnet memory, you can see in the dark blue has very little benefit. Can't really use memory. and we looked at the results and we found sonnet doesn't um doesn't reliably know when to write to memory and also doesn't really uh naturally read from memory. Okay, it's also worth noting that these experiments are done with no specific memory prompting.
03:58:38 It just tells the agent the directory is available use memory if you want. That's really it. So sonnet 46 doesn't natively use memory effectively. Sonet 5 does does indeed better opus and so and so basically the models get better and better with the same simple instruction at using memory. And really the key idea here is that what happens is with stronger models they know intrinsically when to write to memory and they kind of know intrinsically uh when to read from memory to retrieve information.
03:59:06 Their memories are also better. They're like they're highly verified statements or predictions about kind of the state of the database or what may happen next versus earlier models tend to write um lower quality memories. And I'll show you a very fun example of this from a different game. This is from Pokemon. So this is showing you Claude 37's memory directory from Pokemon versus Fable 5.
03:59:27 Um here is the key point. The size of each box indicates the um the number of memory files in each of these directories. Okay. So when we have cla pay Pokemon, we give Claude a memory directory just like I did in that prior benchmark, right? Claude can write whatever it wants that directory while it plays the game. What's really interesting is Sonnet 37, we see it writes a huge amount to memory.
03:59:53 And when you look at the memories, it's indiscriminate. It's basically writing down all sorts of experience. It creates this huge unwieldy just disgusting memory directory of things that are not really relevant or strategic for gameplay and it proceeds much worse in the game than Fable 5. Fable 5's memory directory is much tighter. It's really strategic insights, tactical knowledge, things you can't just get from game state very easily.
04:00:16 So what we see is better models are much are have much better instincts for salience and like what to write to memory. And you can see this reflected in the size of the memory directories. It's much tighter. It's much more strategic and gameplay indeed with Fable 5 gets much further than Sonic 37. So it gives another intuition around the fact that Claude is much better at writing to memory, knowing kind of what to write in salience and what's salient and uh and also kind of using and reading from memory.
04:00:44 So models are getting more and more u kind of effective at knowing salience, knowing what to write and also reading. Now, both of these represent what I would consider in my initial framing of like writing to memory during experience. So, basically in both of these tasks, we let Claude write to memory while it plays this game. We let Claude write to memory while it does the Sealbench benchmark.
04:01:05 But, as I mentioned, there's also the dreaming side of memory. So, this is allowing Claude to reflect over memories to fix things. And I'll show some interesting examples of that. This is kind of a a framing of dreaming. And so you can think about memory in terms of um there are transcripts. Transcripts you can think about as kind of the foundation of memory.
04:01:26 So imagine a memory system could very simply be just transcript search. Claude writes to a transcript as it goes about a task. Future session can read from the transcript. Of course transcripts are large. It's very unwieldy. We talk a lot about in the blog post the fact that transcripts often contain information that you know a very very small amount of information in transcripts is actually salient and matters.
04:01:47 So we also allow claw to do things like writing memory notes. That's what I just talked about. As it goes, it can write notes to memory. But dreaming lets you do more. Dreaming lets you reflect over transcripts and these written notes and re re-represent them. Build memory representations like wikis as an example. Actually promote certain memories to injected context and load it preemptively into every future session.
04:02:09 So memory is this dreaming is this mechanism where you can kind of process existing transcripts or existing memory notes and build new things from them including representations, injected contexts and more. So some fun examples of this, this is from Pokemon gameplay. Dreaming is an effective way to correct mistakes in memory. So as I talked about when Claude's going about a task, it can write incorrect notes to memory.
04:02:31 Just it makes a mistake. This happened this is in in this particular game. um Claude wrote incorrectly wrote the location uh of a particular landmark in the game. Now what happened is with this incorrectly written memory, Claude got trapped in a loop where basically it would misloize itself at this particular point in the game, fall down this trap door and it would basically send it back to the prior level.
04:02:56 That's like the highle intuition. The details aren't that interesting or important. The key idea is a poison memory was injected and it's stuck there. Dreaming can rescue that. That's really the intuition on why dreaming is extremely powerful for reflecting over memories and correcting them. So this is showing some trajectories of the game where orange is memory and dreaming.
04:03:16 Up is good. So basically up represents game progress and down represents kind of game regression. And what's interesting is raw memory um basically without dreaming we can see crashes out a incorrect memory is planted and clock continues to basically regress or fall in the game when it hits that particular point with dreaming it rescues that mistake claw proceeds to the next level.
04:03:40 That's really the key observation and this is a very general principle that claw can use dreaming to correct mistakes in memory that are otherwise stuck there. It's a very important way you can kind of um kind of correct from errors. Now, this is another fun example of building representations in memory. So, there's been a lot of interest in wikis as kind of general um kind of ways that cloud can organize knowledge.
04:04:01 This is a really fun example uh from we acquired a company Stainless back in May and we had access to the the company Corpus all its documents and Slack messages and so forth and we dean anonym we anonymize things and we built um kind of a a data set from it which allowed us to test dreaming. So what we did was we created kind of an email set of 25 questions about the company that can only be answered from this corpus.
04:04:27 We allow Claude to build a wiki from this information to help it answer questions. What you see is we compare a raw search tool that's just like some kind of semantic search over this this kind of documentation. We compare that as a baseline which does reasonably well. You can see it's like 53%. Just for search. We also test letting quad build a wiki over this knowledge.
04:04:52 And you can see as you push, you add more and more dreaming tokens, the wiki gets better and better and better allowing Claude to answer questions. So Claude natively can kind of build up over time a representation of this of this kind of corpus of knowledge through a wiki. And as you add more dreaming kapu, the wiki gets more and more useful. And we think there's a really nice mental model to think about how agents can kind of build knowledge over time.
04:05:15 they can be given access to kind of raw corpuses of texts of transcripts and they can kind of build these representations that allow them to search those tonic texts much more easily to answer questions and to connect their knowledge in effective ways. We also find that adding the wiki and the search tool together actually bumps further beyond just raw search.
04:05:31 So it's kind of another intuition that basically claude can also build representations like wikis in its kind of memory um in order to help it kind of solve problems. And a final example, this is from a different benchmark uh but um this is showing two different wiks that claude built. So this is pri prior to dreaming and after dreaming. What we saw is when claude was building wiks in this particular case, there were some gaps and for example this initial wiki had a bunch of placeholders for pages that didn't exist.
04:06:04 This can happen when cla's kind of building different memory representations. And without dreaming compute, there may not be opportunities to rescue or fix those errors. that we saw is when we ran dreaming, dreaming can look over Claude's existing representation like for example wiki and can fill in mistakes. So in this case it fills in a lot of missing gaps.
04:06:22 It creates an index and so again dreaming is a way that claude can reflect and and kind of apply compute across its memory to improve it by correcting mistakes like I talked about by filling in gaps in its knowledge like we see here uh or for example by creating new representations like wikis. Um, so with that, I uh thank you all for listening and um I I'll be at the at the anthropic booth.
04:06:47 So, please uh come see me and happy to talk about any of this. Heat. Heat. Heat. Heat. What Heat. Heat. Heat. Heat. Heat. Heat. All right, folks. Welcome to Syntax. CJ and myself here. I'm Scott Tinsky. This is CJ Reynolds. We're going to be doing all kinds of fun stuff. We're going to be breaking down some of the latest AI news as well as getting into some fun games as to which AI model made this.
04:11:59 So, if you are out there, you're eating lunch or whatever, you can come hang out or you can just chill and just do whatever. But uh we'll be joining you online as well. So CJ, what's up man? How you doing in San Francisco here? >> I'm good. Uh let's introduce ourselves and anybody that doesn't know who we are. We are Syntax. Uh we're a podcast. We also have a YouTube channel all about webdev programming AI.
04:12:20 Uh we're both from Denver. >> We're both from Denver. Yes. >> Um yeah, I think we'll >> Yeah. So Syntax is more than just a podcast these days. It's also a live stream. So check us out at uh Syntax on YouTube. We do all kinds of stuff including a lot of this. So, let's break it down here. Um, let's get into our first little bit, the first bit of news.
04:12:44 They're calling it the greatest prompt ever. So, this tweet has been going viral here because this is a good prompt. Uh, find all implicit state machines in my app. CJ, why is this a good prompt? Have you used this prompt? I haven't used this prompt, but I've used similar prompts. And I think that like there would be a followup to this prompt because uh in and an implicit state machine is essentially a state machine that you haven't defined, right?
04:13:13 So you have a bunch of variables all over your code like is loading or uh is running and then you have code and if statements all over those. And so this prompt is essentially like let's take that spaghetti code and turn it into something that uh is easier to reason about. Right. >> Yes. Yes. So basically, if you don't know what an implicit state machine or an explicit state machine is, the easiest way to think about this is that an implicit state machine is stuff that you used to see talked about all over React code
04:13:42 bases. Stuff like is loading, has error, is success. Uh people these are often like a a boolean state, right? Like this. And the big problem with that is is that it can be bugprone, it could be errorprone, but it can also put your UI into impossible states. So the theoretically if it's just two separate booleans for these things, you could have a a is loading and an is error both true and then what you're showing both of them or you're showing one of them or the other one.
04:14:12 So the the idea with this prompt find all implicit state machines in my app is that you are able to convert them to explicit state machines. Explicit state machines have a declared status. Uh they they have a value rather than a boolean flag, right? And they have state transitions, right, with centralized rules. So if you've ever used something like Redux, which at a time was >> verbose and many people didn't love it for those reasons, it ended up being a great idea for explicit state machines.
04:14:49 Um there's a lot of libraries around explicit state machines like Redux um or not Redux uh X XATE. >> XATE XATE. Yes. >> Um and it makes working with AI coding a little bit easier because these impossible states are no longer possible. And then you can also visualize around your codebase a bit more. Um, how do you how how useful do you find visualizing around your codebase?
04:15:16 >> Yeah, I mean I like just in general though like I find this useful and it's actually an argument against a lot of people that are basically saying uh don't worry about the code, let the AI make all the decisions. Yes. >> Um, but I have found that a cleaner codebase makes it e easier for the AI to work in as well. Uh, and so if you give it a prompt like this, it's going to make it harder to have like a weird UI state like you were mentioning.
04:15:39 Um, one prompt that I've been doing recently is, um, find all duplication and opportunities for abstraction. >> Yes. >> And like that's another thing that for a long time people were saying like, oh, copy paste is better than abstraction. But when you have a massive sprawling codebase and AI is having to work in all of that, having a single source of truth or like one way to do things in your codebase also makes it a lot easier for the AI.
04:16:03 Yeah. >> No, the the biggest problem with that, not that that's a problem, the biggest problem with that is getting the AI to adhere >> to your abstractions, right? Because otherwise you'll end up with copies of abstractions or Yes, >> exactly. And personally, I've found like lint rules to be really good for that. So like uh one example is a a CLI tool that I'm building that does a lot of file read and writes.
04:16:23 So it's writing files all over your system. It's saving them. And before I refactored, it had like 10 different places with 10 different ways of writing the file. But then I brought it all into a single file manager and any part of my code that wants to write a file has to use the file manager. So there's a lint rule that says no bare fs read, no bare fs write and that basically forces the AI to use that abstraction.
04:16:46 >> Totally. Yeah. So uh I I I like this prompt and I like this type of thing. So finding your implicit state machines and changing them to be explicit state machines and like really getting a handle on the state of your application in my in my mind is something that does decrease the slop level of a code base. >> Definitely and and this is the AI engineer uh conference.
04:17:06 So like we we still need engineering. It's not just prompts. >> Yes. Yes. Yes. It is all prompt CJ. All right. The next thing we that we brought here is unfair play. So fair play uh is basically how a iOS app can be encrypted so that way it cannot be decompiled. >> So unfair play decrypts an IPA and it allows you to recover the actual application that was built uh for a Mac OS app.
04:17:44 And so this project's really neat because what it's doing is it's it's kind of like the ROM hacking scene of of video games where >> you aren't supplying a a a counterfeit app to somebody or an app that is you're not hosting the YouTube app, but potentially what you could be doing is you could take your own YouTube app that you've downloaded yourself.
04:18:08 You can unfair play it. You can get the IPA. You can modify the application to your heart's content and then reinstall it on your device. How does that sound to you CJ? >> Sounds amazing. So, I mean, I'm an Android person. If you listen to the syntax podcast, you know that. Yeah. >> And we've been able to do a similar thing. Uh, so there is this project called Revance in the world of of Android that is very similar.
04:18:33 So, I mean, it's very common to take a Spotify app, take a YouTube app, take an Instagram app and remove the parts that show you ads or something like that. Not condoning that necessarily, but that's something. >> No, remove those ads. Yeah. No, and I I think what I saw with unfair play is that people were taking YouTube and then removing all YouTube shorts from it, which honestly that's not a bad idea.
04:18:56 Yeah. >> I mean, that's like imagine modifying Instagram to give you a timeout. I There's probably some built-in rules that are like don't scroll for too long, but they don't always work. Like you could enforce it. You could make it so that it only shows you feeds for a certain amount of time and then you have to put the app down or something like that.
04:19:11 or or just like hiding stuff you don't need, you know, like there's so many functionalities in apps, especially like YouTube now that they did put shorts in the main bar and they removed subscriptions from the main navigation. So, hey, with unfair play on iOS, you could just swap that back. >> Take back your apps. >> Take back your apps. We're taking back our hardware.
04:19:32 >> Yeah, >> we're taking back our apps. >> And I Yeah, and this is super relevant with like the the reverse engineer everything uh repo that came up. like this is this is the the good part of AI is yes stuff like this. So yeah, >> the ownership and the control, all that good stuff. >> I'm here for it. >> Yes, me too. Okay, >> let's uh let's keep going on some news.
04:19:54 How do you feel about that? >> Let's do it. Yeah, >> we have some coding games lined up for you too, but I think people are eating right now, so maybe not the best time to introduce those. Okay, so >> let's do uh the rise of coding languages for agents. So very recently um Kit from the uh Anomaly which is the open code team has been talking about his new language blood and he hasn't shown it really beyond just a couple of glimpses of even what he had described as outdated syntax.
04:20:26 Um but blood as well as many of these other AI languages have kind of turned into this conversation of our AI languages AI first languages really something that is worth exploring worth in worth investigating or whatever and so we've talked about a few of these on syntax before but now with blood there's also bend lang at bendylang.com as well as zero lang from the versel team.
04:20:58 So there there's I I will say that those are just three that I I I got off the top of my head. I probably seen like 15 of these bad boys. Um so before it was everybody is creating a Safari fork or a browser fork or Chrome fork. >> Um then everybody is creating a front-end language or whatever. Now we got vibecoded languages and you better believe CJ all of them are faster than Rust.
04:21:23 You know they're all fast. They're all very fast. >> That's their benchmarks. >> Yeah. Yeah, I mean it's I think it's super interesting. Um I mean my my initial reaction to this is AI was trained on existing programming languages. So why would it be better at a new programming language? But if you look at some of their tenants and like how they're actually making it useful for AI, it really comes down to like uh giving it like proofs and preventing it from making mistakes.
04:21:47 uh almost similar to like Elm laying like there's various programming languages that make it really hard to get into error states or have um like memory leaks and stuff like that. Elm Elm is lowkey uh getting a lot of attention right now because of all kinds of things and in fact effect and fold kit which is the UI framework in effect is an Elm uh style I see >> framework and so I I see Elm as one of these things that seems to be talked about quite a bit more in this space now >> well it's interesting that you bring up
04:22:23 effect because I don't think of effect as a programming language it's a more of a runtime time, >> but it gives you some of these same guarantees that these languages do, right? Like it it um makes it so that you have explicit error states and errors have to uh pass through in a certain way. Um and again, it just makes AI more predictable when it's writing in that codebase.
04:22:42 >> Yeah. Yeah. And and along with effect, there is a front-end framework called Fold Kit that is built on effect. And the uh the de developer behind Fold Kit just announced that he's going to be working on this full time. Now, this thing absolutely rips for a lot of reasons. You have one state tree. It's all effect based. Everything's a schema. Agents are good at this framework.
04:23:04 >> And their premise isn't, you know, with a lot of front-end frameworks, JavaScript frameworks, it was always this is faster, this is faster than this or whatever. This is this is correct and the AI is good at writing it and your and and I actually saw some alternative renderers for this. Yeah. Uh, someone was getting GPUi output from a Fold Kit app.
04:23:24 >> Wow. >> So, going straight native on it. >> I think that's huge. It's so funny like I mean times are changing cuz like I would see something like Fold Kit and think about it. Okay, would I use this? And I mean that's one of the reasons I haven't picked up effect because like the the it's a little obtuse. Like it takes a while for me to figure out like what is this thing actually doing?
04:23:40 Yeah. >> But if it's for the agent, >> if it's for the agent, >> then it doesn't matter as much as long as it like makes the app correct and better. So, >> give the agent a little treat. >> Yeah. >> Yeah. Just a little treat for the agent. Let him hack a little uh computer competitor. Who knows, right? Um the fulkit I think is an interesting thing because one of the things that we we've talked about in the past and this was like in the past like what like two months ago, so it's ancient history was that I don't know if
04:24:08 these AI languages will ever catch on because what are the models going to be able to write them? But the thing that you get with fold kitten in effect is that you get You dump an example repo into your codebase into just you know and you just tell the agent to consult the examples and consult the codebase and the agent's got no problem with it. It's got no problem.
04:24:33 >> So in my experience I don't I don't think that matters anymore. At one point in time you had to really worry about did like was it trained on this specific framework? And as somebody who wrote speltkit for a long time, the big problem with that was is that it wasn't in the training data. So therefore, you had to then incorporate all kinds of deterministic checks and skills and uh MCP servers to validate your codebase.
04:24:58 And you still do to an extent, >> right? >> Yeah. But now you can just dump all this stuff in context and it can write it. >> And the context is big enough. Yeah. I mean, one thing I checked is fullkit.devlms.txt. It exists. So you pop that into an agent and it it finds all the pages that give examples and uh like you were saying the context is big enough now that >> um it can have those docs in context and do the right thing instead of like hallucinating tag names or or using spelt 3 instead of spelt 5 or something
04:25:31 like that. >> Yeah. And so with with Fulkit, they even have an MCP to connect directly to their dev tools, which is sick because uh the agent can then use the dev tools in an agent focused way, right? >> Yeah. >> And I I think that's a really great thing that that has changed my mind on agentic first programming languages because at first I would have said, "Yeah, I don't I don't know if they'll they'll hang in there."
04:25:54 Um but seeing how easy it was to develop something on this which is clearly not any of the training data or whatever. Yeah. Um it has really changed my mind on this. >> That said, just like with everything else uh vibe coded these days, there is a battle for attention. Right. >> Right. It's all about who's using it, how many people are using it rather than I mean maybe maybe someday we'll all just have our own programming languages that we like.
04:26:21 Yeah. Um because like Kit said in regards to blood lang that the syntax can be whatever you want it to be which he hasn't released too much >> mean. Okay. So he's just talking about it in his tweets because I haven't I haven't seen this. So yes. >> So the the bloodlength thing he hasn't really announced too much of it but again there are I I think with any of the things that we're getting into agentic first dev tools are going to be really helpful.
04:26:44 But I also think it's very possible that the best agent language for agents may not be Typescript, Rust, or whatever, >> right? >> Possibly. >> Which, >> yeah, >> isn't that bad for me. >> No, I mean, and I I I also I'm I'm lamenting the death of having a favorite language or framework. >> Yes. >> Languages are like little mascots. We get them. Yeah.
04:27:08 >> Yeah. I mean, it was our identity as developers. Um, so yeah, now it's it's the agent. The agent should have a favorite. Like if the And it's not even a favorite. If the agent works well in that language or framework, that's probably the one that you should use. I mean, it's one of the reasons I started using React again is because the agents are really good at React.
04:27:25 Yes. >> Yes. And you know what I will say? I did start a project with Fold Kit >> and I kind of really liked its output. So, uh, we will see. Yeah, for sure. >> You heard it here first. You heard it here first. No more spelt for Scott. >> Yes. Yeah. Well, no, that's not necessarily true. just felt >> uh but I was having problems specifically around state in a large application and foldit >> basically like we were talking about with um uh implicit state machines.
04:27:56 >> Yeah. >> Uh FKit makes a lot of that less possible, >> right? >> Um so because of that uh you end up getting a lot of those benefits for free just based out of the framework. Okay, let's talk about another programming language um >> which is uh Rust. >> Oh, >> okay. Okay, we mentioned Rust. Um, Rust being great to write is awesome. And Discord just announced that they were moving Discord themselves to Rust and using GPUi to do it.
04:28:26 >> Incredible. Yeah. So, there there's a tweet from uh Stannislav. Um, he works at Discord. He's Yeah. >> Um, and he he has a screenshot of his uh like Windows task manager and Discord Phoenix is the code name only using 290 megabytes of RAM. Um, and so I mean this is we've talked about on the podcast before like this is the age of everything getting faster and being rewritten with Rust.
04:28:50 >> Um, so I'm here for it. I mean I know you've used GPUi quite a bit. Um, but I think like as long as the app works all of a sudden I've got a gig extra RAM to use. I'm here for it. >> Yes, folks. So let this be a thing. You just heard that Discord is moving to GPUi. I've been talking about GPUi for months, for many months now. And you heard me talking about Fold Kit earlier.
04:29:15 In about about six months, when other people start to catch on, I think Foldit's going to be getting some major press here. >> Just listen to Just listen to the pod, listen to Scott, you'll know what's coming next. Basically, >> syntax.fm, we're live every single Monday, folks. Uh, check us out. So, how how do you feel about this is clearly an agentic decision, right?
04:29:32 Um, Discord is one of those apps that opens every sing or like re uh reinstalls itself every single time you open it, right? There's always an update coming. So, um, it's also an app that I've never thought was non-performant. I I subscribed to >> maybe like 40 50 Discord channels. >> Yeah. >> And I've always been surprised at how well it handles that.
04:29:55 That said, the fact that they were able to rewrite the entire application, Discord, no small application in GPUi, means that like man, we can optimize now easily for performance. >> Definitely. And I mean, you you think about like why did it take so long to do this? Like one of the reasons they were using Electron is because it's crossplatform. Yes.
04:30:18 Right. They write one app. It works on Windows, Mac, and Linux. They have a massive user base, >> distributed anywhere. >> Yeah. And so Electron is great for that. And I mean you've talked about some of the woes of Tari. >> Yes. >> Like it's similar to Electron, but it's going to be a little bit faster and it ships with the system web view, but you run into a lot of potential compatibility issues with like WebKit on Mac OS versus like Chrome on Linux.
04:30:38 Um, and with Electron, you don't get that. So, I understand why it was Electron for so long. >> Oh, yeah. And if you were talking Electron versus anything else, Electron still wins for a number. I mean you can choose your exact uh Chrome version with flags uh in Electron. So yes and then every user gets that but that's what you also get with GPUi right you have one one codebase with GPUi and then you get a crossplatform app that is native it uses native native UI elements on on every platform >> um so now's the time to
04:31:08 do it like we saw bun was rewritten in Rust. Yes. uh and uh there we talked about it on the pod a few months back but uh Jared had an entire agentic flow of like take the existing codebased and tests and then gradually port them gradually get them passing. Um I'm not sure if they could do a similar thing with an Electron app in its tests because it's very like browser focused but at the same time agents have gotten really good at like taking a large codebase and making sense of it in some other language or or porting
04:31:40 it to another language. Yeah, I do wonder what that how how that portability worked with testing and how their testing infrastructure like did they convert the test? Yeah, I I I really am curious about that. That's really interesting. And on top of this, um Anomaly also re released Rune. Have you seen Rune? >> Rune. What is this? >> R U N E as you typically spell Rune.
04:31:59 Uh Rune is their GPUi competitor. >> It is a Rustbased GPU framework. And the idea here is again that it's going to look nice and I haven't dove into it beyond just seeing that it was released and that it was a different rustbased framework for GPU uh UI work but >> okay so they're going to rebuild like uh open code desktop or I guess they have built open code desktop with this and now they're releasing it as open source for other people to use.
04:32:32 >> Yes. Um that app that we talked about last week. Oh man. Hex. >> Hex. Yeah. >> I got Dex. I got Rex, I got Hex, I got all these different apps on my uh thing. But uh that was built in Rune and okay, they were touting some of the abilities to have it look nice and fit in and uh have good UI across all platforms, >> right? >> And and make it easy to do so.
04:32:53 So, this is another one to keep your eye on if you are wondering about making crossplatform apps performant with Rust. Um, but obviously you get into the web space with some of these and then you're you're what you're rendering on a canvas and you're not using DOM elements at that point. So it it's not >> you know there's all these different write once deploy anywhere solutions whether it's like React Native translating it and then React Native web translating it back to web or any of that or taking web and rendering
04:33:25 it as a web view in a phone or whatever. >> And there's Flutter is another one. >> There's Flutter. The Flutter folks are very uh they will let us know if we forget Flutter again because I think there's too many times we've forgotten Flutter. But the what there hasn't necessarily been is truly the right once deploy anywhere and have that actually be native everywhere in a way that's effortless.
04:33:49 I think the closest that we've gotten with this besides Flutter is like the React Native React Native web world. But React Native is a a tough a tough platform and we're seeing people move off of it all the time, right? >> Yeah. I mean, again, I I never thought I'd see the day, but it's like you like again, the reason Electron was so popular is like web devs already knew it made it super and also made it super simple to be crossplatform, but now if the agents writing the code and it's performant, why not?
04:34:17 Why not native? >> Why not? >> Yeah, >> why not? Yeah, I know. I totally agree with that. Um, I I think there was actually there was another one that I just saw somebody talk about that was a right ones to play everywhere. I'm trying to pull it up, but I cannot. It It just was announced like 12 hours ago. >> Was it Is it rust based or >> Oh, CJ, that's a good question.
04:34:36 I don't know. I don't know. >> There's too many things. There's too many dang things released all the time. >> Let it sit there. Maybe it'll come to you. >> Yes. Yeah, absolutely. Okay, cool. Um, let's get into some questions. We we get a ton of questions over here at syntax. So I want to dive into some questions that are coming in across the web development and the development ecosystem overall.
04:34:56 So we had a number of these questions come in. First one that we had come in was, "How are y'all feeling about brain rot in big tech? Do we think software engineers at big companies are losing their skills without knowing it?" And I kind of talked about this in my talk today. I did a talk today. We're at AI engineer. I obviously people are losing their skills.
04:35:20 In fact, in my survey that I did, I think it was like 67% of people listed that they feel like their skills are diminishing since using AI. But I think the bigger question isn't do are we are our skills in coding diminishing? The bigger question is does that matter? >> Yes. >> Right. I mean to again I'm like biting my tongue on stuff I've said in the past like our skill sets are shifting they they really are and like you you talked about like lamenting or you maybe didn't even think about it that much but if you've
04:35:53 been in this space long enough like things change and things move fast especially if you're in the web dev space like new APIs new browsers browsers get deprecated and so if you ever had to uh implement something for Internet Explorer 6 there is no reason for you to know how to do that anymore. it is like no longer a thing. And so it's gradually becoming >> well what is what should our expertise be now?
04:36:15 Like what what >> what like if it's not programming then are are we now u orchestrators? Are we now like tech leads? Um and so I I do see us moving more and more towards that. Um, at at the same time, like in order to be a really good architect or tech lead, you kind of need to know how things work under the hood to be able to come up with good decisions and have good guidance.
04:36:39 >> The agents will still do whatever the hell they want, >> right? I mean, and I found that with the the tool that I'm working on right now, like it does whatever it wants, but then I I rein it back in. I'm like, "Hey, we need to be a little bit more performant over here, or we need to share code a little bit more." And I know that because I myself have worked in large code bases.
04:36:57 So like those kinds of skills definitely translate. Um in terms of where the brain rot is happening like I think it's at multiple levels though because a lot of it comes from management like management is like use AI move fast and then developers are having to stay caught up and then they're burning out and so now they can't even worry about uh do they still know how to code.
04:37:17 >> Um so it's happening but it's all I think part of this big shift of like where where do we where are we going where are we headed? >> Yes. >> Yeah. Yeah. Okay. So, a thing I will say and I said this during my talk. So, uh I will say there are things that I used to do in web development. I used to I used to know every browser hack for Internet Explorer 5, Internet Explorer 6 and I haven't had to you I used to be able to write Angular components by hand.
04:37:48 >> Yeah, me too. I haven't had to do this in so long that my brain in it has atrophied in those skills. So like does that matter anymore? It doesn't matter. I don't have to produce a website for Internet Explorer 5 ever again, right? So those skills are gone. And uh some of those things you can lament and you can mourn the loss of those skills, but at the end of the day, things have shifted, >> right?
04:38:13 And so yeah, they've shifted and I think it more it becomes what should be important. Like I I absolutely don't think it should be we are just prompters just prompt prompt prompt. There's has to be some thought behind that prompt. Yes. And that's where the skills are shifting to. And so I do agree that there is actually brain rot in that gray area. Like people like all I do is prompt now.
04:38:33 I don't actually have to worry about. I think you still need if you want to have a good product, if you want to have a good team, you should still care about those things. So >> Well, yeah. Yeah. Because at the end of the day it is looking for a simple solution at the end of the day because what we saw especially with the um the rise of vibe coding was that people were just hucking all kinds of stuff god knows what into their their browser and you say oh who cares the app works fine it gets a decent lighthouse scores
04:38:58 and then you look at it and the entire DOM is duplicated twice once for desktop and once for mobile and you're just like what are we doing here you know that that stuff still matters for sure >> it still matters. >> Yeah. Yeah. >> Cool. All right, let's next question here. Uh, just could you say a few words about what vibe coding actually means in 2026?
04:39:20 As a middlele developer, I see most if not all others using AI as it's impossible to compete without it. But for many vibecoded is now a slur. Wow. Yeah. uh sometimes unfairly thrown at handwritten projects, projects with AI by experienced devs and fairly non-coders and with expensive models. So >> vibe coding, is it a bad term? Is it a loaded term?
04:39:45 What is what does it mean in 2026? Is it a dead term? What what's up with that? >> I mean, my my initial gut reaction to the term was like it's it's a very specific kind of AI coding where you don't care about the code and it's all about the vibes. But the coding that I've been doing lately, like I I would call it vibe coding because I'm AI is writing all of the code.
04:40:06 Yes. But again, like I talked about earlier, I'm still making all of the decisions. I'm still orchestrating it. I'm still uh engineering it. AI engineering. >> Yeah. Yeah. Yeah. Yeah. >> I am still like you still have to. >> So yeah, I I don't I wouldn't call what I'm doing with the tool that I'm building vibe coding. I would call it AI assisted engineering.
04:40:24 Um, but I do see like vibe coding being thrown around as a slur. Like you can just you somebody posts their app, you're like, "Oh, that was vibe coded. Don't care about it." >> So it's tricky because it's there's a it's a tough line to walk, I would say. >> Yeah. Yeah. It is funny because it does seem like there's a group of people with if anything had AI had a hand in it at any point, it's labeled as vibecoded slop or whatever.
04:40:49 Toss it in the bin. um which I think anybody who's been using any of the models in a modern context like modern isn't like the past you know two weeks yeah who who knows in this industry anymore right uh anybody who's been using models in that regard knows that like hey things have shifted and the models have gotten better so I don't know yeah to me like there has like the language doesn't like kind of bugs me a little bit that people get so hung up on the terms or whatever but I understand why they exist.
04:41:20 What I don't get is like the insistence of throwing everything out just because AI is used. That said, there are some things when you have clear tells of AI that are bad. Yeah. >> Like that that show a lack of care, show a lack of taste, show a lack of control over what you're doing. And those those types of things I I think stand out in a way that really turns people off instantly.
04:41:47 I would not tell you how many YouTube videos I turn on these days and I get like two minutes into it and I'm like, "Oh, this is a this is a Claude script. Toss this. I'm done with this." >> They're reading from Claude. Their animations were generated from Claude. But then there there was no follow-up, right? There's no editing. There was no tweaking, making it their own.
04:42:04 Yeah. >> And and and the script from Claude really kind of bugs me, too. Not that Claude's the problem. The script from AI bugs me. Um because when you can see somebody reading a script that they didn't have any hand in creating, it's just you it it feels like why am I bothering to care about this if you didn't care about this, you know? Yeah, >> definitely.
04:42:24 >> Yes. Okay. So, we're going to play a quick little game here. Let's do a game, CJ. >> Okay, I'm ready. >> Okay, >> folks. We're going to be doing a game. So, if anybody walks by, they want to play in this game, please join in. If not, it's just going to be CJ. That's all good. We're going to be playing Guess Which AI model made this. You can join at any given time.
04:42:41 You can join in the middle of the game. Let's see if you can do it. Um, we got a few of these up here. So, CJ, are you ready? >> Let's do it. >> Okay, audience, if you are out there, you all want to play. Scan this QR code. You can join at any given time. We're going to open this up with the very first one here. Oh, brother. I lost my Wi-Fi. Okay, let's open this bad boy.
04:43:06 >> I see it. Okay, so it's uh it's slowly loading. Uh, which button is Opus 5.5? >> All right. Which of these buttons did Opus 5.5 make? >> Okay. So, option A, I see like the green gradient. Decent font choices. And then option B has a bunch of extra stuff on top and bottom. I'm going to go with A. Opus did A. >> You think Opus did A. Okay. >> Yes. Yes.
04:43:32 >> Really? >> You're wrong. Opus did B. And uh the A was actually Grock 4.6. >> Grock. It's got some interesting uh font choices and it doesn't Yeah. All right. >> You asked it for a button and it gave you three like taglines below it. So yes, like that's I don't have that experience with open >> and this one is kind of similar. So which of these is sonnet 45?
04:43:54 >> Okay. Sonnet 45. This is a year old a year ago. >> Yes. Again the the prompt was asking for a button. So uh >> I mean again I'm gonna go with B because I think Sonnet wasn't Sonnet that decent back then? >> Sonnet 45. Yeah. >> Okay. Let's see. CJ, you're going with B. >> B. >> Okay. >> I am hor Oh, no. I know. I'm horrible at this. >> No. Son of 45 was in fact the one that Okay.
04:44:19 Uh, and the other one was Gemini 31, which did a pretty decent do pretty decent job here. >> I'm just learning that. Um, I I like other models, I guess. >> Yes. Yes. Yes. Uh, folks, you can join at any time if you're walking by. We're playing Guess Which model made this. So, the prompt for this one uh was also a button. We asked this to make a button.
04:44:42 Which of these did Muse Wonder 1.0 make? >> Okay. I haven't used Muse Wonder at all. It's got the name Wonder in it. I'm going to go with B. >> Looks one. It looks wonderous. Is that a word? >> Two models also kind of failed the uh the prompt of a single button. Yes. We need a button here. Um and one of them has this awful uh overflow issue. So, let's let's >> Oh, yeah.
04:45:04 >> Yes. Muse. >> Nice. Muse decided to make a whole ass card for this instead of a button. So, >> all right. Next one here. Which one of these flyers did Opus 45 make? >> Four five again. So, four, not five5, four five, which is like eight or nine months old, 10 months old at this point. >> This is four five. Yes. >> Um, I mean, I kind of associate like the really decent backgrounds and font choices with the newer ones.
04:45:32 So, I'm going to say, but there's a lot of emoji. There's emojis in both of them. Four, five. >> There is emoji. Yeah, >> this is hard. Okay, I'm going to go I'm going to go B. Oh, four five to B. >> You're going to say four, five is B. Okay, four or five months ago. I don't remember. Yes. Um, the A was of course 55. So, quite a bit better. Um, the emojis are certainly a tell of an era, right?
04:45:59 They're very a tell of an era. Folks, you can join at any time if you want to play. We got the next one here. Which one of these flyers did Grock 4.6 do? >> Okay, we learned a little bit about Grock earlier. >> Are they still loading for you? They load. >> They loaded. >> Okay, there we go. >> Uh, again, so this one has emojis. I want to think like Grock sets themselves apart so they don't use emojis.
04:46:19 I'm going to go with A. Grock 46 is a. >> You're going to say it's a Yeah. Okay. Which one of these did Grock 4.6 make? And it was a Yes, >> I got three. Right. >> Yes. The other one was Sonnet 45, which you can tell from two and two here. Sonnet beloveing those emojis for sure. >> Those emojis. Yeah, it's it's a tell at this point. >> It's those emojis.
04:46:41 Yes. >> All right. The next one here is these two flyers. Which one of these did Gemini 3.1 do? >> Gemini 31. That's the late That's the latest one. That's modern Gemini. Like I would I would hope it was B because A looks very basic. I'm gonna go with B. Gemini did B. >> CJ is saying B. The votes are coming in and uh yeah, A's got a bunch of emojis.
04:47:09 Which one did Gemini 31 do? >> Gemini use emojis. I don't know. I don't use Gemini. Oh my god. Okay. >> Apparently Gemini uses all these emojis. Yeah. Sadly, >> get your stuff together, Gemini. This is a frontier model. >> Yeah. Yeah. Well, yeah. Muse Wonder actually did a little bit better of a job than the last one we saw here. though. >> Okay. Yeah.
04:47:27 >> Okay. Next up, we got some inboxes. >> Ooh. >> So, these ones are a little bit tighter. They're a little bit harder to see. Which one of these did Gemini 31 did? >> It's got to be A. Like B has either chat GPT or Claude. Like it feels like chat GPT ism's on the right like the font choices. So, I'm going to go with A. Gemini did A. >> Okay. You think Gemini did A and Gemini did A?
04:47:52 Grock 46 did B. Yes. And that'd be looking like that'd be looking like last year's Opus or something for sure. Yeah. Yeah. >> You could have told me that was Opus 45 or something. >> All right, next one. Which one of these is Opus 55? >> A or B? >> Let's say this is modern Opus, but again, both of these have tons of emojis. >> Um, >> both of them have emojis.
04:48:17 Yes, we thought the emojis were gone. No, >> I I want to say A. No, I'm gonna go B. Opus did B. >> Okay, CJ says that Opus did B, folks. >> Just a gut feeling. I don't know. >> It's a gut feeling. Which one is which? And actually, what do you think? Let me even What do you think option A is? >> Some some model I've never used. I'll say that. >> Looks like you're right on both.
04:48:45 So, Muse Wonder did option A and Opus 55 did option B. So, you were right on that one. >> Cool. >> Okay, let's see. We got one more here for you folks. Which one of these was Sonnet 45? >> Again, Sonnet 45 is 10 months ago at this point. Um, uh, this is a tossup because they're both pretty bland. There's emojis on the right. I'm gonna go with like AI was using a ton of emojis like a year ago.
04:49:14 I'm gonna go with B. Okay, you're gonna say B. What do you think uh option A is? >> I may hold you to that. >> Gemini. >> Gemini. Okay. So, >> oh, >> option B was set 45 and uh option A was Opus 45. >> So, these these models were out at the same time. >> Yes. >> And then I mean the the one on the left was like very bland like >> Yes. >> No character, I guess.
04:49:42 >> I mean, how much character do you need in an email box? I guess I'd need some character. What was it? Google Inbox was my favorite email app of all time. >> Nice. >> Sick. All right. Well, that's it for which LLM was this. Let's get back. We also had some audience questions in here. So, next question was going to be now that no one uses Google as a search term anymore.
04:50:02 Will the word Google stop being a verb? No one is saying let me Google that. If search isn't a thing anymore, how is Google going to stay in business? Listen, Google like we already like most people using Google now use the AI answers, right? The AI overviews. So, um I think that term is still there. Like that verb is still there. Like let me Google it and then they don't look at the results.
04:50:23 They just look at the AI answer. >> Yes. >> Um I use I still say Google and I don't use Google. I use duck.go. So like it's going to be a while before I shake it as a verb. I don't know. What about you? >> Yeah. What is it going to take a whole generation before Google is gone as a term? where people are going to be it's going to be that that I iOS computer that iOS commercial like what's a computer like what's a Google yeah I I mean I don't know I I don't really say Google anyways as a verb so I think I think it's
04:50:52 mostly done >> and I've heard people say like let me GPT that so that's like the new verb >> yeah people people actually I have been hearing that in the wild like at the airport and stuff like that and not in California's airport in uh in Denver's Yeah, >> probably got time for one more. >> Yeah. Okay. So, we got one more one more question in here. Uh, which one do you want to pick, CJ?
04:51:16 >> You got a question you want to pick in particular? We got one about WebMCP. We got about uh >> let's do the let's do the one about apps because this is this is very relevant like anybody trying to build something right now. Yes. So, why release an app? Yeah. >> Okay. So this question came in and it says, "We used to build apps for ourselves and then release them to the world in the hope that someone may find them useful and then maybe develop a reputation and make a few dollars.
04:51:39 Why release an app nowadays though when if it's any good there's going to be 100 slop cones within a week and it'll all get lost in the fire?" >> Yeah. >> Yeah. My initial thoughts are like if you actually put care and thought into it and don't just like vibe code and then throw up a link to a GitHub repo or whatever else like people are going to care like the when people become a like actual user of an app and uh come back to it.
04:52:08 I think it's because they know that there was care put into it, right? Like like it's a nice user experience. Um or like so right now I'm working on a developer tool. It's not exactly like this, but I put so much effort into the docs and so much effort into making sure that people understood what this tool was, >> some TLC >> and what it could do. And like >> uh a lot of people are just like they build the tool and then like let AI write the read me and don't put any thought into I think there's still space for it, but
04:52:34 you have to care. Yeah. >> Like because you'll just be one more app in the sea of apps that everybody just slopped together. So I think care is what it comes down to. >> Yeah. Care is what it come down to. I think the the perfect example of this >> Yeah. >> is Rex. The Rex terminal. Um my gosh, >> the Rex terminal. >> Hex or Rex? >> Rex. >> Rex. Okay.
04:52:56 >> Rex. Yeah. Rex is the terminal. Hex is the uh voice >> voice. >> And then Dex is the issue tracker. My gosh. Okay. So, Rex is an app that's built with a lot of care. And you can tell it's built with a lot of care uh because everything even the little notification that says you have an update is beautiful. It's it's like got a marquee. It's designed well and it is like thought and care is put into every tiny corner of this application.
04:53:24 Um, and when you have things like that and you compare it to a slopped alternative, it makes it very clear that the people who are putting the care into it are still going to get uh attention. They're still going to get users and they're still going to get downloads. There's people who are making a lot of money building small developer tools these days, releasing them to an audience of people who like their work and then still happily pay for that stuff, >> right?
04:53:56 So, you do hear everybody saying they're they're all vibe coding their own personal software these days and I am too. Trust me, there's all kinds of stuff I'm vibe coding as personal software these days as well, but the cream still rises to the top. So, >> I agree 100%. >> Yes, CJ, what are we at for time here? >> Uh, I mean, I think technically we're at our limit.
04:54:15 So, >> okay, folks, this has been it from AIE. We're in San Francisco. It's going to be a great conference here. My name is Scott Tinsky. This is CJ Reynolds. and worse syntax. Peace. Picture a pull request. It changes a schema in one of our data pipelines. It goes green in CI. The tests pass. The llinter is happy. And everything about it looks clean.
04:54:55 So, is it safe to merge? Because a change like this can still break something downstream. Maybe it blows a freshness SLA or maybe it breaks a consumer three teams away who never saw it coming. Or maybe it exposes data that never supposed to be reachable. That gap between CI past and this change was actually safe is the gap my team has to close every day.
04:55:18 Hello everyone, I'm Jas. I'm an engineer on the data platform team at Robin Hood. I design and build the distributed data injection and processing frameworks behind our data lake. The ones teams across the company rely on to make decisions with data. Notice the title of this talk. The bots in it are the ones opening the pull requests and as you will see the ones reviewing them as well.
04:55:46 Today I want to show you how we answer that question when the PRs are coming in faster than any of us can read them. So here's the problem in one sentence. Pull requests to our production data pipelines now arrive at machine speed but reviewing them safely still runs at human speed. First a quick word on the scope of this talk because it shapes everything else in this talk.
04:56:15 AI coding assistants have made pull requests cheap to open. A new query, a schema change, or an access grant now takes just a few minutes. And an assistant can open pull requests far faster than any team of humans ever could. The PRs I'm talking about today are the PRs to our data pipelines, like schema changes, pipeline code, configuration changes, and grants that give someone access to data.
04:56:45 data platform is the team that owns reviewing that kind of PR. So when those PRs pile up, they pile up on us. And the reviewer I'm about to describe is a system we built to handle them. So the people who feel this are a small group. The engineers who carry the context to say what is safe to merge into production and everyone who is waiting on them. So why is this a problem?
04:57:13 AI open PRs just keep arriving. And to be fair, the safety net we already had works. CI checks together with the human review have always caught a lot of real bugs like broken code, inefficient queries and risks like downstream breakage or sensitive data leaks. But the human half only works at the human speed and only with the context a person happens to carry in their head.
04:57:39 Who reads this table? How big is it? And how sensitive is the data in it? The PRs are coming in at the machine speed, but the number of people who understand what is safe to merge into a production data pipeline did not grow at all. That's the mismatch. The queue builds up in front of the few people with the context and that's where things start to slip.
04:58:05 So the problem we set out to solve was this. Build a PR reviewer using AI that can keep up with the speed at which PRs are being opened. So here's the whole answer before we get into the details. Three duties, three owners. Scripts prove what can be proven. An agent judges what needs context. And people own the calls only they should make. We took a hybrid approach.
04:58:31 Scripts for the facts and AI for judgment. We did not want AI reviewing every change when some questions already have an authority have enjoyed lunch and that's setting and closing this room. Exit the key registry or a catalog >> doors will reop and blocks the PR until it is fixed. >> Layer two is the agent. The agents rules tell it what layer 1 already checks.
04:59:02 So it focuses on the risks those checks cannot catch. It reviews the PR, scores the risk and approves or escalates to a person. Both these layers review the same PR. Merge requires passing checks and an approving review. And this also keeps AI costs down. We use AI for judgment, not to rediscover facts a script can look up. So here is where we are going today.
04:59:35 We have covered the problem and the approach. Now, I'll walk you through the hybrid reviewer we build and the decisions we deliberately keep with the people. After that, how the reviewer learns from the human feedback and what results we have seen so far. And we'll finish with the lessons we would pass on to you. Three things to listen for as we go.
04:59:57 Scripts for facts, an agent for the judgment, and people for the protected cause. I'll come back to all three at the end. So to make this concrete, let me show you the one PR will follow through the rest of this talk. This PR makes two changes to a table in our data lake. It adds a new column and widens an existing column from integer to big integer.
05:00:23 It's the kind of PR we see all the time. On the left is an upstream source table. Our injection pipeline reads from it and writes into a table in the data lake call it our table. The first change the source has gained a new column and this PR makes injest bring that new column into our lake table. The second change is to a column of a table already has its type gets widened from integer to big integer so it can hold bigger values on its own.
05:00:58 That sounds completely harmless. Widening a type is about the safest schema change you can make, right? Well, whenever a column changes, the real question is who reads it downstream? Some of those readers are registered in our lineage tool. So, we know about them. But keep a second question in the back of your mind. Are those all of them? Hold on to this picture.
05:01:22 We will come back to this exact diagram each time the PR moves forward through the reviewer. So let's start with layer one. Layer one is the scripts and their whole job is checking facts in the live systems things the code alone cannot tell you. Here are the two of those checks and what they said about our running PR. First data contracts. When a PR changes a schema, a script compares the schema and keys against a source of truth, the schema registry.
05:02:00 For our PR, the question is simple. Does that new column actually exist upstream as registered in the registry? It does. So that check passes. Second, require tags. Every table needs an owner and any PII has to be labeled in the catalog. The script checks the catalog directly. Does our table have an owner? Yes. Is its PII labeled? Yes. Pass again. And if either of those had failed, there is no debate and no model involved.
05:02:33 CI goes red like any other failing checks and the PR is blocked until it is fixed. Contracts and tags are two of the layer 1 checks. Next, the two that look at downstream impact and performance. So, these are the other two layer one checks. They look past the table itself, what happens downstream and whether the job will still run in time. >> As we head into our closing keynotes, we're excited to bring back a familiar to the one.
05:03:07 >> Ladies and gentlemen, please welcome back our MC Cory Quinn, chief cloud economist at the Duck Bill Group. Does any of them break? It's a real check against a real system. But notice the boundary I have drawn. It can only see what is recorded in lineage. Its view of downstream is exactly as complete as that record. No more. Second, the performance.
05:03:32 A correct change can still be too slow or too expensive. So the CI performance test runs the change in staging at the same data volume as production using mock data. So that tells us two things before the PR merges. Whether the cluster is sized right for the job and whether the job will still make its freshness SLA. Those are the kind of problems that otherwise show up weeks later as an incident.
05:03:58 So that's the full set of layer 1 checks. Let's see what they said about our PR. So, back to our running PR. Same picture as before. Here's what the layer one said about it. The data contract check went to the schema registry and asked whether the new column really exists upstream as registered. It does and the check passes. The leanage blast radius check walked every registered consumer of the widen column.
05:04:32 The jobs the lineage tool has on record. None of them break when the column goes from integer to big integer. They all look fine. Along with that tags pass, performance passes, every check is green. As far as the scripts can see, this PR is safe. But I chose those words carefully. As far as the scripts can see, everything on the slide is a statement about the registered picture.
05:04:59 the jobs and the columns our systems have a record of. And keep that in mind. It is not the whole story. Now, let's look at the agent with the diff. What actually changed? The PR description and the commit messages explain the author's intent, but intent never explains away a real finding. If the description says it is safe and the code says otherwise, the code wins.
05:05:28 File tools let the agent read, search and list the files across the repository. It follows changes beyond the diff into pipeline definitions and access grants looking for the relationships the author may have missed. These are the repository repository tools only. The agent cannot query the live catalog, lineage tool or production state. Life facts stay layer 1's job.
05:05:54 The agent reasons over the code. So why walk that code for a schema change? Because registered lineage is not the whole picture. The rules define where that judgment belongs. They are configured by the change type in the PR knowing what layer 1 already covers. They target query antiatterns, sensitive data asks and consumers outside lineage. For example, an unpartitioned cross join can run correctly and still hammer a cluster.
05:06:25 A grant can look ordinary and still open sensitive data. And for the schema changes, the agent walks the pipeline definitions and access grants in the repository to find consumers that were not registered. It's a tedious search uh a person would otherwise do by hand. Let's return to our PR and see what that search found. So back to our running PR. Same picture.
05:06:53 Layer one said every registered consumer of the vid column looks fine. Now the agent walks the code and it finds a job that lineage never knew about. It's right there in the repository. It reads our widened column from our late table but it was never registered in the lineage which is exactly why the layer 1 check could not see it. And that job takes the value and writes it into another table.
05:07:18 And in that other table, the column is still an integer. Now follow it through. Our column is a big integer. Now the first time a value comes through that's bigger than an integer can hold, that write fails. A valid schema chain green in CI passes every registered check and a downstream job still breaks. The agent flagged exactly this chain in its review comment with the reasoning.
05:07:50 So a person could see the whole path from our column to the failing right. We'll get to what happened next in a moment. So why did the lineage miss that job? Because the enterprise data world is messy. Workloads run across different query engines and each engine has to publish its own lineage. If one does not publish, that relationship is missing from the picture.
05:08:12 There's a single catalog that automatically sees everything. So the registered picture is useful but it is not the whole picture. Now I don't want to oversell the codewalk. It's genuinely useful but it cannot see everything. The agent can search what is in the repo. Pipeline code access grants and the jobs defined there that read our tables. That's where it found our unregistered job.
05:08:40 But plenty of data use lives outside the code base. Ad hawk queries someone runs by hand. Dashboards, those are completely invisible to a repository search. A consumer out there never shows up no matter how carefully the agent looks. So we make the agent honest about that limit. If it does not find anything, it does not get to say no downstream consumers exist.
05:09:04 It says it found none in the places it searched. None found is not the same as none exist. And the agent never gets to claim an all clear. So the agent has walked the code. What does it actually produce? A review. The same shape a good human reviewer would give you. First review comments on the specific lines that matter. Each with its reasoning. Not a vague looks risky, but this line changes this column.
05:09:36 And here is the job that reads it. Second, a risk score. How severe is what it found. That's what drives the next step. Then it makes a call. If there is no serious risk, it can approve. If there is a risk that needs a person, it escalates to a human and the comments and reasoning go with it. So the person has the context to investigate instead of starting from scratch.
05:10:04 And to be clear about what it takes to merge, layer 1's checks have to pass and the review has to approve either the agent or after an escalation a person. Both have to say yes. Now let's see what that looked like for our PR. Here is how our running PR actually ended. Same picture. The registered consumers look fine and the core walk found an unregistered job that writes our widened column into a table where it is still an integer because the agent found a real downstream break.
05:10:38 It scored the risk high and escalated to a human with the whole chain spelled out in its review comment. Now the human does the part the agent cannot. The agent only reasons over the repo. It cannot see whether the job is actually running in production. human reviewer checked and confirmed that the consumer the agent found was live and active. So this was not a theoretical risk.
05:11:03 The PR waited the fisk the fix was about order not about rejecting the change. The downstream table was updated first so its column could handle the wider type and only then did our PR merge and deploy. No incident. The scripts in layer 1 said safe. The agent found the gap and a human made the call. And that was not a one-time thing. Most changes can be cleared by the agent.
05:11:36 When the checks pass and the agent is confident in its verdict, it approves on its own. But there's a whole category of changes where the agent never gets to say approved. And no matter how confident it sounds, grants touching sensitive data are one. Who can reach sensitive data is a decision we want a person to own every time. A migration that looks like it changes intent, not just mechanics, is another.
05:12:03 If a change alters what the data means rather than just how it is moved or stored, a person decides. And if the agent itself is not confident in its own verdict, that's the third. It does not get to round up to approve just because nothing concrete came back wrong. In all the all the three cases, his job is only ever to surface the risk and his reasoning.
05:12:27 The decision stays with a person. That is one PR. Now, how does this hold up across every PR and how does the reviewer get better over time? How do we know the flags are useful or whether a rule change helps? We start with the agents verdict and its reasoning in the review comments. A person replies agreeing or disagreeing and explain why. That explanation is what makes the feedback useful rather than just a vote.
05:13:01 Those replies are collected every day. Once a week, the on call reads the feedback and updates the review rules by hand. Those rules shape the next review. I know this is manual today. Nothing retrains itself and no rule changes automatically. But we have an idea to let an incident investigation agent help with that weekly pass. But it is not built.
05:13:26 Without the feedback, every rule change would be a guess. So let me show you an example where a human corrected the agent and we use that feedback to improve its next review. So here is a real one. And note the label in the corner. This is a separate example, not our schema change PR. A column was marked as personal information in the source database.
05:13:53 But by the time it lands in the data lake, the injection pipeline has already encrypted it. So the PR proposed labeling it nonsensitive in the data lake even though the upstream data is still PII. The agent flagged this. From where it stood, the upstream says BII, so a nonsensitive label looked wrong. The author pushed back and asked the agent to look at how similar columns on the other tables were already classified instead of going purely off the upstream tagging.
05:14:24 A human agreed with the author. The encryption before write made the nonsensitive label look okay for this column. And that kind of correction is exactly the signal that teaches the rules to get smarter because the next time a similar column comes through that context can stop a needless escalation. One caution though, this depends on the specific pipeline encrypting before the right.
05:14:49 It's not a blanket rule that encmpion always makes data nonsensitive. So how do we know whether the reviewer is actually doing a good job? We look at two kinds of mistakes. Bad PRs it approved and fine PRs it unnecessarily send to a person. That matrix separates four possibilities. A fine PR approved and a bad PR escalated are the two correct outcomes.
05:15:19 A bad PR approved is a miss. That's dangerous because the change ships. A fine PR escalated is a false escalation that costs a person's time. A gate that escalates everything may avoid misses, but it has not reduced the review load. As of August 31st, 2026, our miss rate was about 2%. Mostly from the gaps in the rules rather than the failures of the judgment.
05:15:45 We don't have a clean single false escalation number. But the dominant cause is missing context like the encrypted column example that I just gave you. Misses give us new rules and false escalation show us context the reviewer needs. So here is one of those misses. Again know the label. This is a separate example, not our schema change PR. We have an open copy of the sum of some data.
05:16:14 some of its columns are about to be retagged and there is a downstream reader that reads those columns from the open copy. Privacy retag those columns as personal information. When that happens, they move into a more restricted copy of the data and they are gone from the open copy. The red PR looked fine and it merged. But what nobody checked was whether anything downstream was still reading those columns from the old open copy.
05:16:43 something was. And once the columns were gone, that reader broke. That's a miss. A PR that should have been escalated got approved. It wasn't a failure of judgment. It was a gap in the rules. A corner case nobody had written down. So, it became a new rule. When a column gets retagged as sensitive, search the code base for anyone still reading it from the copy that's about to lose it.
05:17:07 And that's how the rules get better. The misses get found. Heat. Heat. Heat. Heat. Hello everyone. My name is Jared. Uh I'm the creator of Bun. Uh and I work at Enthropic. Um uh Bun is the JavaScript runtime that powers cloud code. And recently I merged a several million line of code PR um to add an ahead of time compiler uh to to bond which is uh basically the idea is uh instead of paying the cost of a JIT at runtime you can uh uh you can compile all the code ahead of time.
05:19:50 Um and it's it's like a really big project. Um over the summer I uh rewrote bun and rust. Um uh and this is a project on a larger scale. This ahead of time compiler is like a larger project than that. Um and I think one of the challenges right now with large projects is like how do you actually confidently ship a mill like millions of lines of code written by agents.
05:20:22 Um so that's what I'm going to talk about. This was like the opening prompt after some discussion with it was like basically uh we need to make cloud code use a lot less memory and uh and faster and we can modify either webkit or bun. Um bun uses JavaScript core which is webkit um under the hood uh and it's been a little bit over a month. Um uh uh this is this compiler has gone through five rewrites.
05:20:57 Um you can see the like the first couple ones were like pretty short and then I like almost shipped the blue one. Uh and then I was like this is building way too long. Like it's taking way too long to compile the code. So rewrote it again. Uh and this next one is shipping next week. Uh this is kind of the way I think about a project like this is like you're really constrained by how you test that the code works.
05:21:32 Um so that's like even more important than what you're specifically like like the approach you go with for how to build it is entirely driven by how do you verify that it actually is going to like work and not be buggy and that you can trust it. Um and so just a little more background on like like why this particular project why the ahead of time compiler and it's because uh for a typical JavaScript based application roughly half of like this the CPU time is just spent compiling code on demand.
05:22:06 Um and so a lot of the general like thing you you've seen in a lot of applications where people are rewriting their code bases in in a language like Rust or sometimes Go um it's like when coming from JavaScript it's because uh like there it's really the the combination of the JIT and that it's a single threaded application. Um so that's sort of the difference between a JIT and AOT is that uh you compile before whereas with a JIT you compile as you go along.
05:22:42 This is particularly important for something like cloud code where when using like the cloud code like iOS app where the CPU time is a lot uh if you're if you have to have every session of cloud code also compiling um so we talked a lot about like like should we rewrite cloud code in rust um because the bun team us rewrote bun in rust So like we're pretty good at rewrites.
05:23:13 Um uh and we decided not to. Um and the the reason why is because like for in bun's case it's actually like pretty easy to do like bun is an easy project to do a rust rewrite because the tests were written in a language that's independent of the one it runs in. But so because all the tests were written in Typescript um we could run the tests on both the zig version and the rust version whereas for cloud code uh we like cloud codes tests are written in typescript.
05:23:48 So we would have to port cloud codes test suite and as soon as you're like rewriting the tests then you can't really trust the tests anymore. Um so that seemed like it'd be too risky. So instead, what if we just make JavaScript faster so we don't have to rewrite it in Rust? Um, this was another prompt a little bit later. Um, uh, and the result is that it actually did uh have JavaScript's memory like the memory usage uh, it uses about 4x less memory.
05:24:30 Um uh it starts about two times faster. Um uh and we we're dog fooding this internally and it'll be external soon. Um some more benchmarks uh and then I'll also go into more detail about how we actually did this. Um so like it's it's possible that if an interesting thing is like like TypeScript 7 rewrote their code by by hand um from uh JavaScript/typescript to to go and uh the singlethreaded version uh with you when compiled via bun compile is now faster than the singlethreaded TypeScript go port.
05:25:13 Um, and we will add threads also to the ahead of time compiled version of bun, but we haven't done that yet. So there is a PR. Um, and so a little bit about like the actual ahead of time compiler. Um, uh, TypeScript is not type types are don't necessarily need to be correct. So it treats types as predictions, uh, instead of like trusting them. Um, and this kind of problem works really well for models because uh they're just really good at writing compilers.
05:25:54 Um, so so uh uh this is sort of like more about like how how uh like should we like how we were thinking about this decision. Um, and a a big thing here is like there's just so many tests you can run when you're when it's like uh in ahead of time like like for something like this because we don't need to write any tests pretty much ourselves. Like we actually do write a lot of tests but uh we can largely trust that the tests that predate the project work.
05:26:26 Um, and this was the same thing with the the bun rust rewrite where we had buns existing test suite we could run and now with a compiler uh we have bun's existing test suite plus web kids test plus test 262 plus cloud codes tests um and then we also made it so that way bun itself the bun's own test runner could be run in the AOT mode uh and like the we have one breaking change with or really it's two breaking changes with the ahead of time compiler and that's pretty much why these particular tests are failing.
05:27:06 Um but the and the two breaking changes are actually like mostly security things um to prevent prototype pollution attacks but they also end up benefiting for performance. Um so like what makes this particular kind of problem pretty good for agents is like uh you run it with you can run all three versions of an application. One with a JIT, one with ahead of time compiled and one also without the types and as long as the three of them uh behave the same then you know that it works at least for that specific application.
05:27:42 But then you can take that kind of problem and fuzz it a lot. Um uh and just have Claude like turn out code generators to like fuzz different patterns. Um and you can catch a lot of bugs that way. Um uh this is a very large project. This is the biggest I think I've worked on. Uh in terms of like code volume, um most of this is tests. Um and also because we rewrote it five times.
05:28:17 Um when you're like doing this kind of project, what how I think about it is like you have like batches of like okay we need to write a lot of code at once and then after that initial batch then it's okay let's have let's like fix everything make sure it compiles make sure it passes a bunch of tests um and make sure it like really truly works. Um, and the actual logistics of that are like when you're in the big batch kind of project stage, then uh you don't want to run builds, you want to run one build because
05:28:54 otherwise you're going to spend all your time compiling and like waiting for build errors and it's going to like not be a good use of anyone's time. Uh and then separately uh for like longer running like once you have once you've written all the code uh you want to uh have it like have lots of individual agents. Um this is like a little bit of the timeline um of this project.
05:29:20 Uh the first one was not very good. uh like cloud code was a gigabyte um which is terrible uh uh and like this used as much as possible from JavaScript core's existing code um so it was like basically the ceiling was what is is like only as good as like what it could be from that um uh so then a little bit after this we added a TypeScript type checker into bun itself Um these were prompts uh uh and so we added bun check and then we're like hey claude please use bun check.
05:30:09 Um uh and then after that startup time got two times faster and uh in cloud code and uh started using about 3x less memory but the code was slop uh and like like this is not good. it should never basically this meant that any object that accessed a property called archive it would then assume that the bun.archive archive API was potentially used. Um, and I looked at this and I was like, we cannot ship this.
05:30:47 This is not good. Um so then uh uh this is like yeah and this is another thing too is like it just like to like when you're doing like a large planning like when you're starting to like have it do a bunch of design docs and stuff it just loves writing markdown docs and it doesn't make sense because who's going to like 1500 markdown files else is just not reasonable.
05:31:23 Uh uh so for this it was like and this was like I think the third compiler version maybe it was the fourth. No this was the third. Um uh it to like just to kind of batch get it all out uh 140 agents at the same time write all the code all at the same time. then get cloud code to run. Um uh and then uh this is the fourth one. Um and this one the code was not slop um pretty good um with like basically the same performance as before.
05:32:06 However, the compilation time was way too slow because people coming from like a JavaScript background don't expect any compilation time. Um uh so 50 seconds for like cloud code is like a pretty large codebase but this meant that like other projects would take like single-digit seconds and people don't want to wait that long. Um so uh we're going to rewrite it again.
05:32:36 Um and and uh because in in older versions of this compiler, it was taking like 10 to 20 seconds. Uh and then cloud does this thing where like it'll try to like size up everything to judge whether or not it's worth working on, but it's really easy to convince yourself that like your measurements are good um when they're not good. Uh uh and like part of it was also like why is this 900,000 lines of code?
05:33:10 This should be a lot like smaller. Um uh so then we did it one more time. Um and the code was good and the build time uh was under 20 seconds. Um, it's possible that we end up with a sixth time sometime later, but the fifth one is like very shippable. And I've worked on this for enough time that it's like time to ship it. Um, a lot of this was like just like, "Hey Claude, try this idea and then see what happens."
05:33:54 And then it would like size up the idea. Um, and I think this is like still kind of not right. Uh, like I think a lot of this are like things that we should try again and not let it like kind of what I said earlier about like uh sizing up the the impact of a change is like it's really easy to underest or to like size it up incorrectly and then talk yourself out of it.
05:34:19 Um, so I think this probably needs to be revisited more. Um my old approach particularly with like the the bun rust rewrite was very much leaning on like cloud code workflows to do things. Um and pretty much now with the most recent models uh I mostly just rely on sub agents that are like very longived. Um except for that initial part where you're like writing all the code all at once.
05:34:48 Um, and this with the sub agents, it works pretty well because they keep all the context of the relevant code in their head or like in the context window. Um, uh, and uh, that's like less wasteful. Um, and like they can just keep going. Um, here's a little more about like how the different parts of the compiler works. Um generally it's like there's if we can confidently prove that uh the value is like the types are the same that they're monomorphic then it can become a field access otherwise it has to check every
05:35:33 single time uh uh and it has like some inline caching but not like a lot. Um uh as with a project of this size, you really do need to it's kind of hard to follow sometimes like what everything what everything's happening. So uh I leaned a lot on like just having an hourly update. Um and where it would post a scorecard comparing a bunch of metrics that we were prioritizing.
05:36:03 Um, usually though when it was posting in Slack, it would show like a previous and this version so that way it was easy to see if numbers went up or down. Um, but it's like not helpful in a slide because the previous one is just legit. Um, the the important thing is like none of these tests for the most part are things that are like made up. Um uh they're things that like pre predated the project.
05:36:34 So that way we know that like the agent couldn't have messed with it. Fuzili is like a JavaScript engine fuzzer that V8 uses. Um so uh and we were already using that in bun but we also extended it to have uh to like test for TypeScript types um as well. Um uh and it's also important to check that the tests actually catch things because it's really easy to fool yourself that it's finding bugs when it's not finding bugs.
05:37:11 Some more numbers uh on the ahead of time compiler. Um it generally for most projects outperforms the JIT. Um not always. the you can see in the uglify.js case and that's basically if it has to use the interpreter a lot. So if there's a lot of like eval or new function um then it's not faster. Um but most applications don't do that. Uh and it's like it's like a make it like it's as close as possible as you can get to like a oneliner to like make an application run faster.
05:37:47 Um this is not 100% true because as you can see uglify js and there are other cases as well. Um but it is often true. So like yeah the other thing is object prototype. Uh this is like a rare edge case but basically there's a lot of security vulnerabilities in JavaScript applications called like prototype pollution. And this usually works by assigning properties onto object.prototype.
05:38:17 Um, and the compiler bans that. So like for a large project like this, what do you actually do all day? These were like the top messages that I sent saying continue a lot. Um, uh, also like my laptop kept dying. Um uh and then like but like more concretely what you actually do is like a bunch of leading questions like why are we doing this? Uh can we do this?
05:38:50 Also I end up saying like can you do this faster or like what's taking so long because it kind of like talks itself into making up rules. Um uh and also it tends to not be uh it tends to like underestimate what it can do. Um so I often end up saying things like this to Claude to to get it to like uh try harder. And the result is that uh Jetream 3, which is one of the main benchmarks that JavaScript engines use, um uh the bun outperforms both JavaScript cores JIT and no Node.js in like V8 JIT um in most of them.
05:39:46 Uh, and the whole time like it's been using cloud code to like build the bun compile like to build bun compile itself and then using the build of cloud code that was compiled with bun compile. Um uh so sort of how I think about this is like like the testability thing is super important when you're working on a large project with claude. Uh and to make sure that the tests actually test things.
05:40:19 Um and if you're not actively reme-measuring then it's going to like get worse because it's just if there's so much stuff going on it's really easy to lose track. Um uh and like it's easier for large projects like this that haven't shipped yet to uh rewrite than it is to refactor. Um and if you don't act ambitiously then the models will like start having like 0.1% improvements instead of like 10% improvements.
05:40:54 So you kind of have to constantly nudge it to like hey you can do really cool things if you try harder. Uh you can try this now if you want uh by running bun upgrade- canary if you have bun installed um and then just bun compile. Uh thanks everyone. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Heat. Uh hello everyone. Uh we are here to present our talk uh merging at um AI scale.
05:45:10 Uh this is essentially uh Uber's journey from going to a fully green trunk to somewhat broken trunk. Um I'm Priam Devi. I'm a staff engineer at Uber working in developer platform. >> Hi, I'm Majeri. I'm a senior engineer working alongside Priam uh in the developer at Google. >> Great. Um so here's the outline for the talk. Uh we're going to go over like how the merging um works at Uber at the moment and then how AI changed the landscape and then I'll respond um to that how we adapted and then discuss the results and um
05:45:49 talk about what are we planning next. So at Uber uh we have a six language based mono repos where more than 5,000 uh developers uh are contributing over 36,000 commits uh 27,000 PRs uh every week and uh pretty much every changes uh to the services libraries are being done in one of these monor repos. uh for the purpose of this talk we're going to focus mostly on go code when we talk about anything because that is our the biggest one and that's where the most of the scaling challenges uh we face.
05:46:28 So let's understand the problem a little bit and um and understand like why we need a merge cube here. So uh given here basically we have two users uh user A and B creating PR A and B um on a base edge uh from the trunk and when the PRs are created and CI is run uh the CI is green bills are passing everything is good. Now uh when a uh is merged into the trunk uh we can see the bills for the merge commit are still green u bills are still passing but when the PRB is merged into the trunk um bills are no longer green and
05:47:12 the reason is that B was never tested uh with the changes that were merged by A. So uh we never tested the the combination of those two things before merging and hence the the when B is merged it's broken. Um now imagine when we have thousands of uh engineers working in a monor repo and everyone is making changes u like hundreds and thousands of those every day.
05:47:40 Uh this is a constant problem uh where trunk is broken um very frequently and there is a burden on fixing and there is a toll across the company to sort of um figure it out and uh fix it. Um so here uh we're going to present a solution uh to that problem uh and that solution is summit Q. Um that is our Uber's merge Q um that we deploy uh to solve this problem.
05:48:08 And in this case we can see our two example CRP A and PRB. And here we essentially what we are doing is we are predicting the futures that would be uh that could be true on main. So there are like three things here. A could land on on its own to the main B could land on its own to the main and then the combination of A and B. In general I think uh what we want to do is that we want to land the change that is going to keep the trunk green as a whole.
05:48:37 So in this case when we land A and B together the trunk is green and uh and that's how basically uh sort of summit Q works and keeps the trunk green for our monor repos. Now how does it scale in this uh version of it? Um there are three things that summit employs. U the very first thing is target analysis. Uh what it means is when two users are working across two different services uh they don't really have any overlapping changes.
05:49:06 Um so those two things can be built and tested independently of each other and then can be merged. So this allows us to basically get higher throughput and not serialize every change that is entering into the queue. Uh second is probabilistic speculation. Um what it means is uh we are using historical um the build information and also the the change characteristics to predict uh what is the success uh score for a given change.
05:49:35 And then we are using that to basically decide what are bills are likely needed in order to land that change. And so in that case we prioritize the build that are most likely needed and are important for that change to be uh merged to the trunk rather than like every other build that uh comes from the speculation. The third part is bypassing the large changes.
05:49:56 Uh what it means is that if we have a large chain sitting in the queue which is taking longer to build and at the same time if we have a smaller changes entering into the queue um those can be built faster and so though those changes don't need to wait for large changes to land before they can land. So in this case we can land the smaller changes without landing the larger changes first and this optimization allows us to basically move fast and get a higher throughput of the queue.
05:50:22 So this is what has worked um when it was just humans uh landing changes creating PRs but um then we had AI and the changes at which the uh PRs were created and landed it all changed. So to understand it a better let's look at some of the numbers. So uh in January 2026 we were doing uh roughly 90 requests uh every hour and by March we were doing 130 requests every hour.
05:50:58 So that's roughly a 44% increase in two months and the reason for this increase is primarily because of the AI generated changes. Now given the way the system was built um uh it had the rate of uh these changes had effect on it. So um looking into those like we had 30 minutes of uh P95 line times when we were doing 90 requests every hour and that went to roughly 3 to four hours when we were doing uh 130 uh requests every hour.
05:51:33 So in this case we were still keeping the trunk green uh at all times but at the cost of developers essentially waiting for three to four hours to land their changes to the trunk. Um now I'm going to hand over to Murray uh and she's going to talk about our response to it and how we adapted uh to the situation. >> Great. Thanks Priam. So, Pritham just uh walked us through a few optimizations that Submitu had that worked so far in keeping up with the demand uh that humans generated.
05:52:05 But once AI came into the picture, uh the optimizations we had were no longer sufficient. So we our response essentially was another set of optimizations to our existing submit queue and we approached the response in three different ways. The first was we introduced this concept of admission gating which helped us to shape the traffic that we accept into the queue.
05:52:31 Uh next we introduced the concept of deferrals which is a secondary cue that we introduced that helps with developer experience uh from the pain that comes with admission gating. And lastly, we implemented what we call conflict relaxation which helps us buy back more parallelism uh within the queue itself. Uh next we will look at each of these uh interventions in a little bit of depth and understand exactly what each of them are doing.
05:53:02 So the first one admission gating uh is basically what it sounds like. There is a gate at the admission of every request into the main submit que. We look at the request that comes in and we categorize it as one of two things. Either it's a small change or it's an expensive change. And the definition of this uh could be anything. One of the definitions that we use is the target analysis that Pam talked about uh earlier in the slides.
05:53:29 Uh a small change touches fewer targets. An expensive change might be something that spans across uh multiple targets, the entire build graph, something like that. So a smaller change is more likely to give you higher parallelism and we basically say admit it during peak hours and add it to the active uh submit Q build and test as usual uh and move along.
05:53:58 But an expensive change has the capability to basically be a bottleneck by holding resources uh needing to validate against everything else that's in the queue. So to protect the rest of the queue, what we do uh in this uh phase is just to reject any expensive change during peak hours. What this means is that the developer has to come back and retry their expensive change outside peak hours.
05:54:27 Uh so this gives us control into what enters the queue by moving the burden onto a very subset of changes and developers. Uh let's look at the trade-off of admission gating. Uh after we implemented this, we definitely observed an increase in our throughput. We were able to achieve over 20% increase. Uh our land rate went from 130 to about 160 during business hours.
05:54:56 But it came at a cost and the cost is that developers become the scheduleulers. uh not all developers since we do protect the smaller changes. But you can imagine if you are a developer who owns any types of libraries uh that span the entire code, you will be particularly targeted in this because the work that you do uh naturally falls into this expensive change bucket.
05:55:24 Uh so we managed to protect the capacity during peak hours but had to make developers wait. Uh next slide. So the next thing that we do is targeted more towards bettering the developer experience uh of uh the cons that come from admission gating. Uh what if instead of outright rejecting expensive changes, we are able to record them and replay them back into the main queue uh based on the status of the queue.
05:55:56 Uh when we have let's say a better uh handle on the capacity and that's exactly what deferrals implement during peak hours. If an expensive request comes in, we no longer reject it but put it into a deferred queue where it is perfected. And once the gates permit usually upon uh for example we can say after peak hours uh we put this persisted request back onto the active submit queue.
05:56:23 So essentially we uh shape the traffic in a way where we release more expensive changes when the queue is able to handle it. So what this solves is a developer submits once and then we automatically release it for them when the capacity allows. Next slide. So the trade-off here is it solves the problem for the developer. They don't have to wake up early, stay up late, try to find a particular window where the queue looks good and hit the button on their end.
05:56:55 We will handle that for them. But the other side of this, the bad that comes with this with this is that it doesn't really solve the waiting for the developer. You still have to wait the same amount of time for your expensive change to go in to the queue. Uh next slide. So before we look at the next uh and final optimization that we did, uh I want to touch a little bit about the uh on the concept of what a target overlap is.
05:57:26 uh and how that serializes work in sub. Um so recall that freedom mentioned the target analysis that we perform uh to determine whether two requests should be serialized or can they go in parallel. So an example that we are looking at here uh is a strict one connected component. What this means is that each of these requests A through F touch some subset of targets and all of there's at least one overlap of targets between each pair of views.
05:58:01 So like A and B overlap on at least one target, B and C overlap at least one target. Similarly for C, D, E, and F. And what this makes uh submit Q do is group all of these requests into a single component. And that means that all of these have to go together. And this is basically what prevents uh us from doing work faster. Uh and the next optimization we do is how do we break this?
05:58:32 Is there a way where we can break this up in a smart way? Um next slide. So conflict relaxation is what helps us break these links uh in an informed fashion. uh before is basically the status quo. Any relevant overlap between two requests uh makes them conflict with each other. >> Our next talk judgment is the last job will begin in approximately >> with relaxation.
05:58:59 We essentially do three steps. First, we rank the overlap. So it's no longer a binary decision of is there at least one overlap between these two requests. Think of it more as a how strong is this overlap. Uh once we have that we keep the stronger links and relax the weaker ones to buy back parallelism based on a config value that we set. Um next slide.
05:59:27 So the first step is essentially scoring these edges uh or these overlaps. We use a jakard scoring uh for computing this dependency overlap. We basically have overlapping targets. Uh we check for that and any new dependency edges that are introduced. We compute a combination of these two into a continuous valued score between 0 and one where a higher value of score indicates a stronger overlap and a lower value of the score indicates a weaker edge with fewer targets that overlap between this.
06:00:03 Next slide. Once we are armed with this knowledge, we basically uh make this conflict graph again. Right? So taking the same example and adding some numbers to the uh conflicts that we have uh from the earlier slide. Let's say we have these six changes and the config value we set says we want at least 50% parallelism in the queue at all times. What this means is that we need at least three separate cues that are working uh independent of each other and based on the uh overlap scores that we have computed we are able to
06:00:46 make this choice of dropping certain edges in order to be able to achieve that target uh Q size that we have. So in this case uh the weakest edges are between B and C and B and E and we drop those two which gives us three separate connected components. Uh and this is how we essentially go from a serial chain of six requests to a parallelized chain where two each exist in three separate cues.
06:01:19 Next slide. So next comes a question of um where exactly do we make this determination of uh dropping the edges right? So there are two places where we can do this. Uh one is at a global level every time uh we are trying to assign uh config graphs and building them. Uh we can basically recomputee this every time. And what that does is it introduces some level of instability into the graph which is determined by any new requests that enter the queue, any work that finishes and exits the queue.
06:01:55 So uh essentially what we we can do that uh but what we chose to do instead was a more uh local assignment uh of overlaps. And the place that we choose to do this at is only at the new arrival uh checkpoint. So once a new request comes in, we essentially take that and compute the edges of this with everything that exists in the queue, make the decisions and that's how it stays.
06:02:27 The next decision making will only happen when another new arrival comes in. So the concept is simple. We assign once at entry time. uh and basically reuse this. Um what this helps us do is any work that we schedule based on previous conflicts gra conflict graphs that we have created is preserved uh and it helps us hang on to speculative builds that can help us move faster uh towards a finalization of its next.
06:03:00 So the tradeoff of this final um intervention was the biggest that we saw. Uh we saw over 50% increase in the throughput. So we went from about 160 lines per hour to almost 250 sometimes even greater uh during peak hours. Uh so it worked in that it not only helped us uh significantly improve our throughputs while giving us uh extremely good uh P95 land tanks but it came at a cost as everything does and the trade-off that we have given away here is basically what PTO alluded to in the very first slide.
06:04:03 Heat. Heat. Heat. Heat. me. Hey. Hey. Heat. Heat. Heat. Heat. Hey everyone, I'm Ean and I work on anti-gravity at Google Deep Mind. Today I'm going to talk about the future of building products, what judgment looks like, and some lessons that we've learned from shipping anti-gravity at Google. So anti-gravity is a tool to get work done with agents and we're trying to make the best interface for developers and knowledge workers to do high intelligence tasks.
06:08:56 The software engineering we used to know has completely changed. We can do things now that we wouldn't have even dreamed of a year ago. A sentence can turn into an operating system and my voice can create a AAA game. So, what's left for us? The tools we use have changed. We used to use the code editor. It's been around for decades. But now, every year, if not more often than that, there's a new paradigm for how to do our work.
06:09:23 It redefineses what our work even looks like. And on the side of building products, that's changed as well. We used to spend a lot of time upfront figuring out the right thing to make because building was expensive. It took a lot of time, effort, and resources. So, we ideated more and invested more in that phase. This is completely flipped where now you should still understand the problem you're trying to solve and how to go about it.
06:09:54 But if you want to make something, it can exist just like that. You can try it, feel it, get feedback, and understand the nuances that you might not have realized about it before and use that to inform how to move forward. This is exciting, but it's also a little bit deceptive. It's easier than ever to just turn our brains off and blindly accept what's made for us.
06:10:18 So, if anything can be made by anyone, then it's more important than ever to know what the right thing to even make is. Everyone knows their own definition for what taste or judgment looks like. And I think that's okay. That's good because we all work in different ways and we're all working towards different things. But by the end of this talk, I'm hoping that I can help you form a little bit of a clear idea on what judgment and building products looks like to you.
06:10:50 I've gone from the scale of working at startups to now at Google DeepMind where the scale is it feels pretty incredible. We're making frontier models and on the product side we're able to use this internal frontier to build the interfaces that will use the next steps everyone will see in intelligence. We're very close to the research and we can experiment and iterate really fast and ship to over a 100,000 people inside Google.
06:11:18 These people at Google are using anti-gravity internally in really ambitious and unique ways. And this creates a really cool feedback loop where we can continue building and do awesome things. I want to walk you through three big lessons that we've learned from this cycle. So I think we've all seen this first pattern of the tools we use scaling up with intelligence.
06:11:41 We used to build products in a pretty relatively straightforward way. We identify a problem, understand it deeply, and make something that people want with a clear target. Now that target is moving. The ceiling is going up for what's even possible to make. And with that, the possible value that you can create in the products that you build is changing as well.
06:12:05 So I'm always coming back to a few thoughts whenever I'm working of what we can achieve now and what's just past the edge of what we can achieve. And we've seen this look like two main ways in building a deep mind. As our models got better earlier in the year, they got better at communicating with and orchestrating each other. We created systems for teamwork, for to do more ambitious work.
06:12:28 We stepped up in intelligence to let users do more. At the time, it was awesome. You could create operating systems and really cool things we've never seen before, but it was also pretty prescriptive because the models needed at the time. More recently, this has looked like our version of agent swarms. You can actually try this now in anti-gravity, but this has been creating a ton of value inside Google for us.
06:12:48 People can be more ambitious. There's more agents working together and we're able to relax the constraints to achieve more. You can bet that with the next models coming out, we'll see even more cool patterns and things we didn't expect. But I think we all already kind of knew this. We've been seeing this for the past few years where all the tools we're using are just adding more and more new cool ways to scale up with intelligence.
06:13:17 But the second way I've seen evolving with intelligence look is a little bit trickier because being ambitious does not always mean adding more. Let's go back in time a little bit and walk together through a decision in the product. The way we used to use AI tools looked very much like a structured turn-based format like this. I would ask for some small scoped change.
06:13:39 For example, editing a few files. The model would do it and then we move on to the next thing step by step. It made sense at the time to be able to cue instructions so the next things I wanted could be delivered to the model one at a time. They might not have been smart enough to get everything at once, but it made sense to give it limited context progressively so it could, you know, get the work done bit by bit.
06:14:03 Fast forward to now. There's a lot more going on. Longunning agents, asynchronous tasks, external triggers, and everything else we're doing. Things are looking more complicated, and the agents are also smarter. So maybe the decision we made in the past might no longer be correct. Maybe a better pattern looks like treating the agent more as an entity that can make better decisions and giving it user intent immediately, just like this.
06:14:30 Maybe it'll choose to orchestrate to a bunch of worker agents or do the task after we're done the current task at hand. It's smarter, so it can decide for itself. The decision we made in the past can easily become obsolete as intelligence grows, and you need to challenge your prior assumptions. If we separate the timeline of what we can achieve into three clear parts, if you build too far in the past, it'll be obsolete.
06:14:56 If you build too far for the future, it could be visionary but impractical. It's important for us to ideate quickly and work towards building in the right time window to create the most value. Challenge your past assumptions and rebuild your foundations to create good systems that scale with intelligence. With these foundations, we've seen another really awesome pattern evolve or come out at Google where we see the value in creating a platform for ambition.
06:15:28 So something like this might sound familiar. I used to use design tools in a very manual way. I'd create frames, build user interfaces manually, and create things by hand. And the tool, if we look at how it changed from before to now, it looks pretty similar. But the way I use it has completely flipped. Maybe now agents are creating hundreds of visualizations for me to go over very broadly before I go deeper into something.
06:15:56 Building software now doesn't always look like giving the user value in a very clear and deterministic way. Users can create their own leverage and make room for their own ambition. Let me walk you through how this has looked for us. In our product, there's a lot of room for extensibility. People can build specialized tools that work for them and this can often end up leading back into the product.
06:16:19 I'll show you some examples. Personal software can come up where I can make a dashboard where agents will scan across feedback channels and bug fixes to show me all this information in a way that just I like. Maybe it'll look like software for my team where designers can create the tool that works just for them where they can collaborate visually on demos.
06:16:44 This can also branch out even more. The science team at Google is doing some awesome things with their science plugins. We're creating this platform for ambition for a wide range of users to create their own value and create their own leverage. And this can actually feed back into the product you're making at its core. The agents form example I mentioned earlier actually started as a plug-in through our extensibility system.
06:17:04 It's created a ton of value internally for us at Google. And by giving the users surface area to explore and create value for themselves, we were able to learn from it and iterate in the product to make the core better. Let your users be ambitious and watch it feedback into your product to to create more value for everyone. The last principle I want to go over kind of applies to all of this.
06:17:36 There's a lot of great talks at this conference about software factories. You know, it's really exciting. We can give our agents ways to build more, verify more reliably, and ship faster. It's exciting. But if all the parts of what software engineering used to look like get abstracted away, then how do we define what the remaining value in our human judgment is?
06:17:56 I'll argue that human judgment is more important than ever. If I ask an agent to say, make this slide better, maybe it'll add some visualizations and a bunch of dashboards and cool colors and everything. Maybe it'll rewrite the whole slideshow in Rust. So, it'll solve every problem. But does every problem need to be solved? Take for instance what the agent manager looked like around a year ago when anti-gravity first launched if we want to make it better to an agent.
06:18:30 That might look like adding a lot more detail, ship everything, add more detail into every single step the agent is taking and give the user knobs for everything might be better in some ways. But it's blind. It's easy to blindly accept the first thing that we're given. What does better even mean to us? If we're trying to make a better agent manager interface, maybe that looks a little bit different.
06:18:54 Maybe we need to ship fast, but understand at each step of the iteration process what better actually means to us and what the next direction is. Let's start here. Better might mean simplifying the interface, only showing the user exactly what the latest thing the agent is doing semantically so they're not shown a wall of text and tool calls that might not actually matter.
06:19:18 Maybe it looks like simplifying the whole interface to give the user less cognitive load so that they can multitask easier and be able to achieve more in a different way. So this kind of brings us closer to what anti-gravity looks like today. All these steps along the process have brought us closer to what good means. And we're making a lot of improvements fast.
06:19:48 So I'm excited you all for you all to see it. Now is the time to try many things. Be ambitious. It's more exciting than ever. And it'll actually get more exciting as the models get better. Try many directions, but take a moment to think. Use your human judgment. Just because you're finished creating something first, which can be done in an instant, it doesn't actually mean you're done.
06:20:16 Think about what you're actually trying to achieve and how the thing you're making feels. Iterate quickly. Identify many directions you can go in and try them, but arrive at the right thing and go deep into that. So make time to think, make room to think, and use your human judgment. To summarize, some of the lessons we've learned involve if scaling up with intelligence, which doesn't always mean adding more.
06:20:43 It could mean rebuilding your foundations. Take these foundations and enable your users to be ambitious in new ways you might not have expected before. And at the end of all this, when we're shipping faster and faster, make room to think and protect the human side of software engineering. At Google, this has led to some awesome things by creating a platform for ambition, improvements to YouTube, better video decoders, and memory optimizations.
06:21:11 As for DeepMind and Anti-gravity, it's a more exciting time than ever with new models coming out. And on the product side, we're also moving quicker than ever. These are just some of the feature launches we've announced in the last two weeks. And we're continuing to use the principles we've covered today to work on the future of interfaces for higher intelligence.
06:21:32 I'm excited for you all to see it. And I'll leave you with this. This new age of software engineering is actually worth celebrating. Agents might be doing more and more, but it's also more important than ever to solve real problems and use good product thinking. Protect your judgment and build things that people want. Thank you. Hey, hey, hey. Heat.
06:24:37 Heat. Hey. Heat. Heat. N. Heat. Heat. every Heat. Heat. Heat. Heat. Heat. Hey, Heat. Heat. Heat. Heat. Heat. Heat. Hey, Heat. All right, folks. Hopefully everything's on. You can see the brownfield factory here. All right. My name is Justin McCarthy. I'm the founder and CEO at Diffusion. And we are diffusing the capabilities of the model into the global economy.
06:34:40 Uh it's a big place out there. There's hundred trillion dollars of it. There's eight billion of us. So there's a lot to do. Um uh and that's that's what we're doing. And one way we're doing it is by building software factories. Um, this talk actually, you know, uh, I don't I guess it's got the factory thing in the name. Um, uh, this it's not going to be super technical.
06:34:58 There's a QR code at the end that is going to have some technical stuff for you. So, I would encourage you to pick up or have your agents pick things up. I'm sending a lot of dual channel messages these days where I say the human stuff up here and then I attach something for the agent down there. So, that's kind of this presentation. I'm going to say human things to you and then your agents are going to pick up the agent channel stuff at the end.
06:35:16 Um, okay. So uh so this started for me uh in July of last year. Uh we were on the Clawude 4 series of models and um you know I had the privilege of being able to dive very very deep um uh and really ask what happens uh when you begin looping these things. Um so that uh was the story of the beginning of the software factory that I built um together with uh the team and I'll say that you know even on the 4 series of models uh even as we were awaiting GPT5 and the initial codeex harness um you know even then we could see
06:35:51 glimmers that this thing was going to work but at tremendous token expense and only in the presence of a real Rube Goldberg contraption. Okay. So only when the loops were very duct taped together um and and very sort of um uh handcrafted uh would you be able to see these glimmers of some kind of an output that would stabilize the input that would lead to the kinds of reliability um that we would ultimately want.
06:36:17 Okay. So I'll just say that feeling um if you judge from the number of software factory talks that have been present here um at the conference that feeling um I think is what we know now as software factories and it's sort of accepted in the zeitgeist that at least some part of this um just kind of happens now and it requires substantially less of our creativity and input to achieve.
06:36:40 So that's of course the time for me to say please stop building software factories, right? Um, so, uh, so I and I and I'm I'm, you know, obviously, you know, a little bit tongue and cheek here, but, um, but also serious, uh, uh, I think you really should stop, uh, subjecting your ideas and your ambitions and your goals, uh, to a factory process. Um, you know, because most of the time we don't know what we want or at least a lot of the time we're creating something new, we don't yet know what we want.
06:37:09 And if we're trying to industrialize something where we don't know what we want, um, uh, we're certainly we're not going to get the right thing or we're going to take too long or we're going to spend too much. Um, we're going to have all it's going to be sad in multiple ways. Okay. Um, so what I would say is, uh, let's begin to use some discernment about whether we're discovering product market fit, we're breaking new ground, we're conquering new territory, or whether we are doing something that, you know, already has
06:37:35 a precedent and is already working. Okay. Um and that discernment about sort of what way, what mode we're working in, um uh you know, I I find it really helpful. Um because, uh the great thing about a brownfield factory is it's something that we're already being paid for. It's something where customers are saying with their dollars and their and their utilization like, "Yeah, this works for me.
06:38:00 I like it. I want more of this." Okay. Um it's the existing things that keeping the lights on, paying the bills, keeping keeping customers happy. Um, and you don't need to discover anything about it because everything you need to know to loop the stability of that system is is just built into the runtime. Okay, so everything that's happening, all the ways your customers are using it, everything they're saying about it, maybe even the support tickets, those are reinforcing um the integrity and the vitality and the
06:38:27 utility of whatever you've got in that brownfield. Okay. And then of course we're going to talk about this but um uh the another exciting thing as we heard from many of our other speakers um uh you know Jared just uh talked all about rewriting uh bun um and and cloud code um you can rewrite a lot of that brownfield of course. Okay, so this becomes all of that technical debt, all those sort of boat anchors that are slowing us down historically.
06:38:50 Um, uh, all that, you know, impossibly long backlog. Um, we can see those drying up now. Um, and we don't have to spend, uh, much energy asking what the customer wants because the customer is arriving every day and they're saying, "What I'm using is what I want." Okay. So, um, so we know what to build. we build the we build the existing thing. Uh and a couple of uh a couple of notes about just ways of thinking about working with your brown field and you know cleaning it up and making it self- sustaining.
06:39:28 Um uh at diffusion we use the word promise um to represent you know some guarantee some aspect some priority and some dimension within the product that we value. Okay. feature is a promise. Um, you know, the cost and price of a thing is a promise. Um, the security of a thing is a promise. Um, and it's rare that you want to revert and regress on your promises.
06:39:53 Okay. Um, uh, so if you can start talking about the ways that we have expectations that are against an existing product, um, you can begin to encode those in loops. Okay? And a loop just like any of the factory schemes we've been discussing whether you represented in a YAML file or some Python or graph um you know a loop has the characteristic of somehow uh taking an input and an output and then somehow feeding it back.
06:40:20 Okay, as many of you will know this is the sort of idealized uh block diagram of an amplifier and if you cut that feedback leg you're just going to amplify noise. Okay. And um I think we all have this feeling now that we've been working with software factories and working with loops. Um there has to be some way of feeding back that fidelity into the input of the system.
06:40:41 But if you do it and you tune it right, it can work right. Um so uh so this is what we should strive for, especially in a brown field. We're getting all the signal we need to cure, stabilize, and improve those loops. The next attribute of a brownfield factory is you know it really should be continuous. If you look at uh the history in the before times of when commits would arrive there was some seasonality to those commits.
06:41:11 There would be the daily and hourly seasonality. There would be a weekly cycle and you would see those commits come in at intervals. It can happen continuously now when your loops are being maintained by your agents in a factory. I I don't know like it might look like a straight line. Uh just uh just just a solid list of activity that's happening 247 because there's always something to worry about.
06:41:35 Um is it a little bit slow? Is a little bit insecure? Is it a little bit whatever? Um all of those worries apply around the clock. And if you're building your brown brownfield factory the right way, um it should be it should be compatible with continuous intelligence, continuous application of this question. Um and and it should just run. Okay. So you've got an existing code of state, you've got an existing set of promises.
06:41:58 Um just make it continuous, make it loop and make it keep those promises. Okay. So so another way to summarize that is you know my brownfield factory on an existing set of promises. It should just be continuous. It should be background and should be keeping those promises for me. Okay. So uh so I'm a strong advocate of building a system like this. Um, and it's going to give you the freedom and the creativity to begin asking what you should be doing in the green field.
06:42:27 You know, probably not with the same level of looping rigor, probably with more true product market fit and discovery sort of energy. Okay. All right. One other really fascinating property. Um, this is certainly something that I've experienced uh over the last year. I mentioned earlier, you know, a year ago um it would work, but there was a lot of componentry.
06:42:47 Okay. getting it getting it to work was still very much engineering. Um, as we heard from some others from Zach and from Potato, um, these systems are they're getting a little bit easier and a lot of credit goes to the model itself, um, as well as the harnesses. But one thing you can expect from your factory, um, is that it is going to simplify over time.
06:43:10 Okay. Um, this is something that we've seen in our skills. The skills are getting shorter, of course. The harnesses are getting more general, of course. Um, and a lot of this again is credit um, uh, to the models themselves. Um, I'll note that uh, there's a uh, there's a famous AI company um, that also makes these widgets. Um, and this is kind of a famous uh, progression.
06:43:30 So the Raptor 1 engine from that AI company um, uh, was really fiddly but it worked. Okay. Uh, it looked like this. There were a lot of tubes but it definitely propelled the rocket. Okay. In that same package, right, that same approximate external dimension, the second version had higher performance and fewer moving parts. Okay, the delta between this one and this one was a tremendous application of intelligence.
06:43:57 Okay, like literal rocket scientists and engineers focusing on how can we make this thing simpler. Okay, um you know that's hard. It's hard to get those people and it's hard to concentrate them on a difficult problem. Um we have tremendous allies in this simplification of course because we have Astra and Fable and Carbon right and so they are trying to take our factory from version one to two to three so that by the time you know we come back we come back and you know start our jobs on Monday or whatever um maybe we
06:44:26 have a much simpler factory than when we started right or we wait one more uh model version and our factory is has fewer moving parts yet twice the performance. Okay, so this is a really interesting feature at this moment that hasn't existed before. So if you can just barely get the thing to work, you can expect uh uh a smoother laminer flow, higher performance, you know, soon.
06:44:51 Okay, without much of your creative contribution in between. Okay, so uh let's make a brownfield factory. Um uh a great instruction to give that factory is the same but faster. Um I've been in very few meetings where someone debated uh you know increasing the performance of a system, right? Um as long as you are at some sort of a paro equilibrium where you're not trading faster for cheaper or trading safer for faster, okay, then this is just good.
06:45:26 Okay. So uh so it should be faster. It should be cheaper. It should be safer. Um, note that sometimes we include better in this mix. And better may be difficult to achieve in a brown in a brownfield factory. Better may involve more of your intent. Better may involve more of your judgment and taste. Um, so maybe that's where you go into discovery and use different techniques.
06:45:48 But gosh, faster, cheaper, safer. Um, especially on the other side of the V apocalypse. Sorry, other side. Still at the beginning of the V apocalypse. Like we we definitely need these things to just autonomously get safer, right? Okay. So you should have that. Okay. So um so I think everyone should have the brownfield factory. You should have it for your existing commitments.
06:46:06 You should have it. It should become better faster and cheaper over time. Um I find that this takes like a week. So if you haven't spent that week yet, just do it. Just like whenever you get home, like go go spend a week uh say you're on vacation next week and just stand up your brownfield factory on all your existing commitments. Um, if you don't have a way to do that or you feel like you need help, you can always reach out to me and uh we're standing by to help.
06:46:30 Um, so brownfield factories, you should do them, but also too easy. Okay? Like come on, too easy. Okay? Um, uh, you need you need to elevate your ambition as well. Okay? So, in my remaining couple minutes here, um, we're going to talk about how to elevate our ambition. Okay. So, one of the things that I think is we are all struggling with at this moment is gosh, there's a lot of code coming out.
06:46:53 Even worse, it's working. Um, uh, that's a lot of stuff. Uh, it's maybe too much stuff. Um, uh, we maybe have a glut of stuff. And, uh, it could be the case that our customers, uh, didn't want all this software. They actually, they actually weren't asking for software. They were asking for more gross margin. They were asking for more customers. They were asking for more brand differentiation.
06:47:16 Okay. uh and just shipping them more windows uh to click on and buttons to click on. Um that's not doing it for them anymore. Okay. Um and this is where all of us um are beginning to see the emergence of our new job. Okay. We can communicate with the models quite well. That's why you're here, right? Um there are other people in your firms who are not as fluent in communicating with these things.
06:47:42 Okay. Um so that means like we have a new job now. Um, we have to figure out how to build a demand factory because our brownfield factory can't build the moat, right? All it can do is keep our existing promises running well. Um, that's great because it allows us to focus on sort of outside the building and look toward the market and look toward the future.
06:48:04 Um, but gosh, we we need to start working on the demand factory. Okay. Um, I don't really know what a demand factory is, but it it probably um uh looks something like this. It probably looks like the thing that we need so that when we boost performance in the back end here in the software factory, we don't end up spilling software all over the floor.
06:48:24 And we rate match the demand systems to our production systems. Okay, but there's a lot of discovery that needs to go into the demand factory. This is a this is a new idea. Okay, but I think you are absolutely the ones who can begin pushing your organizations to ask these questions. Like don't wait for sales and marketing to come to you on this question.
06:48:46 Like you can see that within the uh envelope of your brand, there's something more you can ship. Um but you're going to need to explain that to the world and you're going to need to explain it using new words and new imagery and new product lines and so forth. I think your demand factory helps you ask better questions of your market because the agents can help prioritize what to ask next.
06:49:08 I think it can help with better listening because the agents can synthesize the answers you get back from those questions and can help you rerank and they can help you rerank overnight and they can compare it to everything else you've heard. Okay. And therefore I think they can schedule better experiments. How do we get the price elasticity of demand on this particular offering?
06:49:26 You have a finite amount of experimental bandwidth. How do you allocate it? The models are going to help you with that. I think this could lead to lower latency responses where we are saying yes faster. Okay. and customers begin uncensoring themselves and they start asking for more. That would be great. Um I think like weirdly, you know, even though our competitors have uh the same models, there's some version of some new equilibrium where we're all doing game theory optimal comp competition and yet still if we start
06:49:54 first, there'll be some edge. Okay, so get started now and then figure out what is the maximum extent of what I can possibly promise to the market. And that's my brand envelope and let's fill it up. And just quick digression on brand envelope. Uh um let's say we're SpinDrift. Uh uh we we're selling some awesome sparkling water with a little fruit juice.
06:50:12 Um I propose, hey, you know what? Let's throw some caffeine in there. Like maybe that gets by the committee. Like maybe we ship that. Um and then the next week I come in and I say like and motor oil. Okay. Um you know, that's going to get shut down. But uh energy drink, that's a good expansion of our brand envelope. For every one of your companies, you have the same thing.
06:50:31 Maybe you can't sell motor oil, but maybe you can sell an energy drink. So figure out what your energy drink is and start selling it now. Okay. So let's build your demand factory. Um it's going to have a high volume of high quality signal coming in from customers, coming in from the market, coming in from any place where you can measure demand. Okay.
06:50:50 Um and it's going to be it's going to have a high volume of flow. Okay. Um but in order for that flow to not overwhelm change management, desire, um it can't blow out the needs of your customer. So therefore, you need to have more customers. Okay? You in fact need to have more of everything, but you need to have more wallet within those customers. Okay?
06:51:10 To get more share of wallet within those customers, you're going to have to sell more SKUs. Okay? Uh in addition to more SKs, you're going to have to organize them into coherent product lines. So, you're going to have more product lines. Okay? And once you've maxed out all of that, you're going to need to just acquire your competitor. Okay? And if you get started now, you're going to be the one doing the acquiring.
06:51:35 Okay. So, uh, so this is it. This is brownfield factories. I think you can build your brownfield factory. I think you can absolutely build your demand factory beginning now. Uh, I also obviously think you can reboot your product management and reboot and you could wake up your executive team. Um, and we are standing by at Diffusion. So, uh, we do have some technical resources attached there on some brownfield techniques that I really like.
06:51:56 Um and uh yeah uh I I look forward to uh connecting with many of you. Okay, take care. Thanks. Heat. Hey, heat. Hey, heat. Heat. Heat. Heat. Heat. one Hello and welcome to our ag engineer San Francisco talk. My name is Steven Chin. I'm joined by Cassandra Chin, who's the creator of T Trace Tune framework. >> Hi, I'm Cassandra Chin. I built Trace Tune because I wanted to improve PRs.
06:57:55 I see a lot of developers doing great things. The speed of innovation is as fast as it has ever been with AI technologies. But a lot of poll requests are getting rejected. And it's not because the code is bad. It's because the PRs often contain incorrect information or are written poorly. So hopefully with Trace Tune, we can improve that. It also has the option to add a fun comic to help explain your poll requests in a funny way.
06:58:28 And >> what we're going to do today is we're going to go through a little bit about kind of the common issues of PR iss how to improve the quality, how to improve token efficiency on this, and hopefully help everyone to be a better AI engineer for your organization. Like our main character, Alex, who all he wants in life is to get an AI engineer hoodie.
06:58:53 >> That's a really neat hoodie. I have a lot of AI engineer t-shirts, but I don't have the hoodie yet. >> Get switched to make this AI engineer hoodie. Everyone >> ready? And um this is enough motivation that hopefully he's he's going to do an amazing talk at the conference about his use case study on what he's doing in his company to improve the quality of PRs.
06:59:19 Here is a picture of the token factory where Alex works. So, um, Alex works together with a bunch of agents. He has, um, of course, finance looking over his shoulder because we we all spend too much tokens and they want to make sure we're keeping up with costs. Um, his buddy there, Pip, he's walking with carries all the evidence and looks into the the archives and what's actually happening.
06:59:45 And then, u Meera does all the reviews. So, she's she's keeping track of things. >> It looks like Pip left his station empty. >> Part of the factory until until he sees what Alex produced for a PR. Oh, no. >> That's a very long ledger. A lot to review >> the code. So, it's it's hard. And Merror's Merror's going through everything for the fine tuned comb.
07:00:18 She actually has red lines, found some issues with the PR. And this is a a real PR. I think that's actually one of your trace tune PRs where it needs to have exact preview links for the URLs, command attribution, make sure the earlier work is um qualified as well and explain all this in detail in the PR record to be accepted by Mera's very critical review.
07:00:47 >> This was interesting because as I was building trace tune, I was actually like as I was building it, I was using trace tune to test it. So I would have trace tune build the PRs and as I improved the PRs it would like get better. >> You got amazing comics along the way too just to have fun while you were building stuff. >> So um you can see that the repeated PRs which Alex is doing to try to get good quality results are not working out well.
07:01:21 It's working out for his token billionaire part. Do do you know anything about the token billionaire program? >> Yeah, so actually I am a token billionaire. I spent lots and lots of tokens to build trace tune. >> That's kind of unfortunate. Was that was that all my money? >> Uh don't sweat the details. >> Okay. So um I I think there's a lot of good uses for for tokens and for spending where AI work.
07:01:53 um maybe doing repeated PRs is not where you want to spend your token and definitely the finance department's not happy with this. So he wants a result and he wants to make sure that the PRs are accurate are working and he has they don't have need to keep getting repeated again and again by reviewing the logs and reconstructing the evidence repeatedly.
07:02:16 And it's interesting because normally when you're just like uh coding on your agent, you're in your session and you ask it to write a PR, it just reads your session log to try to figure out what it should have done using its inflated context window. It actually often like as it's turnurning it compresses its context window. So it gets more and more inaccurate as it goes along.
07:02:45 So instead, what we're doing here is with trace tune, we're having it read through uh different sessions on your computer to try to really figure out what you built and what happened. >> Yeah. And I think like like that concept of um looking through the audit trails, looking through like what the agents have actually done is a lot of what you were doing with the show tale project you were doing for education.
07:03:11 >> Yeah. I actually started this project with this showtail project because there was this like academic research study I was supporting. But basically the issue they hit is students were creating programs but there was no way to trace back the actual work and effort they did cuz with AI coding the what you submit is the final work the program but most of the work is the prompting and everything that came before and that doesn't get submitted with the program and I use the technology and theory behind that to be like
07:03:47 well I can also improve poll requests with all this data collection. >> Cool. And I think that improvement in pull requests is something that benefits everybody because now you can find the evidence like um logging the change directory before the or the CWD before the CD runs. explaining why maybe other test data is leaking into other repos and explaining what's happening behind the scenes on the PR rather than what most AIPRs read like which is a line by line explanation of the diffs since that's what they build the
07:04:22 PRs off of but incorrect facts can still pollute the information um can cause a problem with conflicting PRs and then still give you back inaccurate results. So maybe we have a solution for this. What What do you What do you think? Do you have a solution, Cassandra? >> Yeah, everyone should use trace tone because of the comics. >> So because of the comics, not because it gives you better PRs, right?
07:04:55 >> Well, I started with the comics and I kind of went backwards and like, oh, I need to actually make the PRs better. Well, hopefully better PRs, better comics, and maybe better token savings if by the end of this. Maybe >> it should be. >> So, um, trade students an open source project. It lets you improve the quality of your PRs by pulling in agent context by pulling in information about the the code and then having an independent writer independent reviewer build better quality PRs plus optionally giving you an
07:05:34 amazing comic which what what do you think the best feature is here Cassandra? >> I think it's obviously the comics. This is one of the earlier comics from when I was using Traune to build it. That's good. And then you you made sure that there's new characters every episode. None of this repeating characters. >> Yeah, that's what this PR was about. >> Okay.
07:06:02 And then this is a description of the original trace tune flow. So it finds all the traces for a particular branch. It researches them, reviews the description, adds more context, and kind of goes in a loop until it gets all of the evidence it needs to make a a quality PR. And um subjectively, but also through um PR bench, which is testing framework, um gives you better quality, better research, lower um rate of inaccuracies, and the PRs.
07:06:33 >> I think at this point the PRs were already better than if you don't use trace tune. But I found that it still makes some mistakes occasionally. You can see at the end that it tries to revise with more context. I think that's where the mistakes come in. Okay. Well, Alex, Alex thinks it's good enough. He's going to try to solve the token crisis the factory with trace tune and see if this will solve all of his problems.
07:06:57 Polish checked. Alex looks very, very happy, but Meera doesn't look that happy. It looks like we still hit a mistake here like I was saying earlier. So the original PR claim was no later passing run is recorded. And the problem here is the in the actual trace it shows it running a bun test at the same time as the bun xtsc. And you can't run those two at the same time.
07:07:30 They have to be run sequentially for the test to pass. >> Yeah. Exactly. Okay, so he claims test passing and all the checks were run and actually that's not what the code did. So poor Alex. Another failed PR, a very angry badger with all the token cost. Um kind of kind of makes me feel like Monday morning with the Neo Forj finance department looking at my token spend each week, but they're they're very forgiving, especially when it's a important project like like this.
07:08:06 So maybe Alex can save some cost by using cheaper writers is the number one thing finance asks me to do. They're like, "Hey, hey, no more opus on it. Haiku, do something cheaper, please. Does this work?" >> I feel like the quality is a bit lower when you do it like that. Just reducing the cost doesn't quite help. >> Yeah. Uh, so we were trying to get better quality and we we ended up getting worse quality by just using cheaper writers.
07:08:33 And obviously Alex is in a bind here, right? He's he's spent too much on revisions. He doesn't have a solution for reducing token costs. He's not getting the results he wants out of his PRs. But maybe there's a there's something Pip can help us with. Pip Pip has a friend. >> I wonder who this friend could be. Oh, well, I I think I think you know from my previous talks that that this friend is definitely going to be a graph.
07:09:05 >> You're the graph guy. We're doing another graph again. >> Yeah. Yeah. There's always a new graph. But this is not the same old graph. This is not the the Neo Forj database. This is our new friend Archie who is a embedded database written in Rust called relate that actually is being launched this week at the nodes conference and also this talk at a engineer in San Francisco and it's an open-source um database which allows us to get a lot of the benefits of more accuracy multihop queries more explainable traces is
07:09:46 without having the heavyweight approach. So it's kind of an agent focused graph database specifically for these sort of embedded use cases. Okay, so let's see how Archie can help us out. So before we were relying a lot on on agents and loops of agents to resolve the problem. But graph databases can store relationships can find like like different things which are clustered build a brief which connects all this information together and then give the agent even a cheaper agent better information and more context to give
07:10:22 a good drafting decision. And the good thing about Archie is he has zero LLM costs. It's all local compute cost and storage and it's a fully immemory database and rest so it's highly optimized. So we get this providence about all these submitted changes and this evidence graph basically for free compared to all the LLM and token costs which we were doing before.
07:10:45 And this means our our buddy Alex doesn't need to research. She doesn't need to go through and find all the facts again and go through the sources because relate is going to do all that hard work for him and build an evidence packet that he can use for trace tune. >> So, all sounds really great. Maybe we do need more graphs. >> That's I'm surprised to hear that from you after all the graph talks you about to put up with.
07:11:11 Okay, but we can do this and make it even better with the neighborhood brief. So basically taking all of the the Y the code and the checks putting attributions and relationships between those and then returning back a connected set of information as a packet and by doing this then when we're the agent is drafting up the PR it has all the connected facts with their sources together in a single source and that makes the writer draft more accurate with fewer passes and it also makes the reviewer's job much easier.
07:11:45 So we don't need an expensive reviewer. We can save by using a few haiku reviewers instead of a um sonnet reviewer or or a luna in the case of codeex to find come up with the findings check those against the draft redo the writing and then we retain the higher quality writer a a sonnet or terror class writer to make sure we have a high quality brief that will pass human approval.
07:12:13 When we were building this, it felt very counterintuitive to have the writer be high higher quality than the reviewers. >> It actually it's common that people always want to have the planner, the reviewer, the checker to be the highest quality model. But if you're doing high quality writing, a lower model, no matter how many iterations and how much feedback you give it, can't do it.
07:12:34 It just turns in things into a checklist and makes it very mechanical. So to give writing which feels humanlike, you do need a higher quality model. And it's it's actually the the reviewing and the cross-checking step which is more mechanical. So it's a little counterintuitive, but that works out quite well. It's also a little counterintuitive that um our badger has a thumbs up coming out of his chest, but let's let's ignore the high quality AI artwork.
07:13:05 Okay, so we've checked all the claims against the evidence. We looked for missing findings with our cheaper reviewer. Did the focus work and then found the things which were connected information to give a better draft and review for mirror. And here's the output. So it has the the Y. So log CWD and that describes the shell before the CD runs. What's the exception?
07:13:34 So the CD targets in repo or the command mentions a route. What passed? It has type check plus 71 test checked on submitted content not before. And then it starts in correlate.ts inspecting the leading CD exception. And Merror looks pretty happy with these results, right? >> Yeah, this might be the first scene where she's genuinely happy. >> I think she was kind of happy with the the engineer sweatshirt before she realized that Alex actually was a was a goof.
07:14:06 Okay, so here are some of the actual data and results from the sideby-side analysis using PRBench, which is a um pull request benchmark, which uses audit traces plus code and um LM judges to review the quality of the work. Of course, reviewed by a human as well for accuracy. And you can see that on both claude and codeex, you get a significant quality improvements on a scale of 1 to 100.
07:14:34 The judges by and large almost unilaterally liked the trace tune plus relate quality better. Um on a one to 100 scale it was like 74 to 76 82 to 86. So plus two plus 4 point. >> These tests contributed to my token billionaire card because we were running lots and lots of tests. We created this side project called PRBench just for testing and it we would have different agents test over different poll requests and compare the results.
07:15:05 Initially I had it like running like sequentially. It'll do one test and finish it then do the next but then we decided to run them all in parallel. It's a very quick way to reach high spend. >> Yeah. So that's kind of funny. I I at at at at at the current testing rate with the number of use cases and tests and the benchmark um we can fully saturate my my high-end gaming computer with as many threads as Linux can handle simultaneously and and of course the token spend is is slight slightly higher than what we'd like.
07:15:42 But it's all for the good of improving the quality of pull requests for the world. So you all get the benefit of all these test runs and having the highest quality improvements and you save on tokens. So we've spent the tokens for you in advance because trace tune with the graph solution takes about half as many tokens as it did previously and which is an amazing result from the graph because now you're premputing you're getting all this evidence graph construct in advance.
07:16:14 The writing's cheaper, the reviewing's cheaper with a higher quality result. >> I wasn't sure about the whole graph thing initially, but after seeing the cost tank almost in half, like it's not even a question, it's making this program work. >> And this is something we did for the trade student project, but you can do this for pretty much any project where you have data, you have relationships, you have evidence, you have things which you're trying to to compare because agents are not that good at doing the data
07:16:48 analysis at doing all the evaluation and the curation. So if you give them a help at the front end by creating a a custom graph layer with an inmemory database like relate suddenly your costs go down your quality goes up and you too can be a token billionaire like our friend Alex. Well I guess it's a now he's using it as a bookmark. I think that's a good use for it.
07:17:13 It's a really high quality bookmark. It's heavier than a credit card. >> It is. It is a nice surprise for um spending all those tokens on PR Bench. So, finance is happy. Lower drafting costs that they can reasonably fund. Alex is happy. He has his new bookmark. He is now has a great case study with the connections doing neighborhood retrievable of facts so that it's a better brief focusing the review so the reviewers get the same compact brief ma gets exactly the evidence she needs and then here's Alex on stage at um
07:17:50 AI engineer somewhere in the world better PRs lower drafting cost with a graph plus open-source trace to builds, which also gives you amazing comics. Okay, so big round of applause for our friend Alex, who's now earned the AI engineer hoodie. >> I think that's definitely a wardrobe upgrade for him. >> Yeah, I don't think the pocket protector look was that great for him.
07:18:24 So, good for him. I think we'll see Swixs on stage with one of these hoodies soon. He he can't he can't not do it now that we shamed him into doing it at his own conference. The token billion program is amazing. A big kudos to the team for coming up with ways to incentivize and reward people who are on the bleeding edge of AI usage and learning how to make it really effective for the rest of us.
07:18:49 And um now you too can have better PRs. Impress your co-workers by having wellressearched effectively written and lowerc cost PRs >> with a comic >> with a comic. So for more information go to trace tune.com learn about trace tune. Give it a try. It works with quad. It works with codeex. It works with all your favorite AI tools. And if you want to do these same sort of amazing results on your own project, try out relate, our new rustbased embedded open-source graph database, which brings context to your agents and
07:19:27 helps you build more efficient applications that just work better, are cheaper, and give you better results. So, thank you all for joining our AI engineer San Francisco talk. And if you're watching this talk this week at San Francisco, what should you do this weekend with your kids on Saturday? Cassandra, >> we are running an AI engineer kids day. So, anyone who's available, just sign your kids up.
07:19:54 We're teaching kids workshops. Kids will be learning how to do AI programming to make their own games. And I think we're also doing a No, it's just the gaming one. No, we'll do we'll do the robots, too. >> Okay. Yeah, we'll also be doing robots. So, kids will get to control these little soda can robots using AI. >> Yeah. So, a lot of fun physical AI gaming AI all in one place.
07:20:18 Um, sign up at AI engineer kids day and um, you know, bring your kids, bring your nieces, bring your nephews, and let's train the next generation of AI engineers. All right. Thanks so much and enjoy the rest of the conference. >> Hi, I'm Arius, founder of Pinter AI. Um, this talk is for people who build agents and run their own inference stack or plan to.
07:20:50 And if that's you, you associate typically the KB cache, the bread and butter of modern inference engines usually as a throughput optimization. The keys and values computed per token kept so the model doesn't compute them again. Today I like to show KD cache in a different light. The way agents agent AI agent application could go wrong are keep growing, right?
07:21:11 Prom injection, jailbreaks, broken tool calls, sensitive tool calls, sensor data exposures, hidden back doors and drips and so forth, right? So the way right now we typically guard against against these numerous ways that an AI agent application could go wrong is by guarding the edge by perhaps passing a judge model that checks the prompts the inputs and the response outputs against your policy.
07:21:34 All right? Or you build a classifier for each case. But here is a different light. The model turns every input into a computation before it acts before it decodes before it generates the response before that response becomes another tool call. The KV cache is your engine's record of that computation that modern inference engine for free are using it and reading it.
07:22:00 And therefore in this talk I would like to suggest as as a complement to the way you're already using your own guardrails to use this data and integrated with your KB cache infrastructure. I refer to this as a KB first security signals and basically three reasons that matters right. So the data is already there. You pick the gems from the pile your inference stack already was throwing away.
07:22:26 It covers the guardrails blind spots because whatever slips past those guardrails, it still becomes computation and it stays in place. So it's perfect for data privacy application for cases where you do not have the luxury to read the inputs and outputs. And the recipe is pretty straightforward, right? It has three steps. You learn a baseline for your AI workload, for your AI inference traffic, how your KV page typically look like and then find the KV pages that are rare against your baseline typically your anomalies
07:23:00 and then you can even classify and explain those. So first let's start with where in this stack this integration could happen. The interesting thing about the KB cache is that in a way existence of those had made the inference in a way stateful right. So for example things like KB cache management layers such as LM cache stores them from GPU down to remote storage or LLMD and Nvidia dynamo route each new request towards the worker that already holds perhaps that cached KV page.
07:23:34 So there are two integration points you you you can integrate those anomaly detection or those classifiers into either you can implement it directly inside of your inference engine vg line and so forth. The beauty of this approach is that it's real time and and the the the alert that it would perhaps generate could ride along your future tool calls or your future policies.
07:23:58 So you can harden basically your polic algorith but just generating the computational anomalies that you observe within your inference engines and perhaps you can integrate it with as a as a way of thread discovery in a sense so you can integrate with your LM cache so it reads the stored KB pages and find things that don't belong with the rest right so now I want to show you some numbers and and just a very simple hello world use case if this is actually working this is a valid approach and for that let me use with a
07:24:35 use case that has no text to filter at all a voice agent. So here's a case study of a voice agent. In this case it was a GMO 4 E4B. Our baseline was 72 ordinary clips and spoken four voices. We're looking at 16 token pages. Obviously this granular granularity could change but here we just use shorter pages so we can have more data in the baseline. And then on the test we have different voices and one clean voice and the other one has a hidden instruction within it.
07:25:08 And as you can as is shown here not only you can detect these cases just using simple anomaly detection but also you would know which part of um which page which part of your request had this anomalous event. We even went further and we trained a model that uses the KV pages as features and we didn't try to explain it or tag it with things that are useful.
07:25:36 In this case, although it wasn't even trained on audio is able to generate correct information about personal information, home address, location data, travel information, and destination. So now back to text agents. In this case, it was a quan 3 14 billion live and below 11 cache. The one on the left shows you an idea for how attacks and normal traffic would typically separate from each other.
07:26:01 And then the one on the right, which is perhaps more meaningful, it was using the at bench and hugging face. It has its own baseline of benign agent traces and eight kind of agent failure from injected tools description to misinformation and so forth. We look at the top 10 alerts that anomalous anomaly detection would generate and about seven to eight times of out of 10 of these alerts are actually anomalous meaningful and useful anomalous activities.
07:26:33 We even went further and we looked at whether there's any kind of anomalous events that stand out more than the rest. Um, two make things hard in production. First, models change. fine tunes and new releases, but also agents that each run on a different model and routers that pick a specialist model per request. That means each model writes its pages its own coordinates.
07:27:01 So either you learn every baseline, classifier, and explainer, which is computationally prohibitive, or you have to find a way to map them all into one shared representation space. Second, real time means moving big objects. For example, a 1256 token chunk of coin 314 billion is about 40 megabytes of KV compression and the streaming like hashgen could help, but moving that for security analysis and acting in real times is still a slow and that's what we solve at in AI.
07:27:38 We are looking for teams building their own inferences stacks who treat security as a foundational design decision. I hope this was helpful for you. Thank you so much for listening. Come on. Heat. Heat. Heat. Heat. Hey, hey, hey. Heat up here. As we head into our closing keynotes, we're excited to bring back a familiar face to guide us through the final sessions.
07:33:23 Ladies and gentlemen, please welcome back our MC, chief cloud economist at Duck Bill, Corey Quinn. We made it. Welcome back everyone. Hope you've had a great day of talks, ideas, conversations, new friends, new nemesis potentially, and supposedly even a little bit of Smash Brothers over in the expo. Uh I'm am have been and will remain Corey Quinn, and it has been terrific to be here with you all the last few days here at the AI Engineer Code Summit 2026.
07:34:11 Uh before we dive into the evening programming for the last time, I want to take that moment to recognize the partners who make this event possible with presenting sponsor Google DeepMind. More on them in a second. And of course the lounge and platinum sponsors, the gold sponsors, the silver sponsors, uh and all of the supporting partners as well who do not themselves get a medal or color, but we appreciate them all.
07:34:37 Let's get a round of applause for all of them. So now let's get excited for the final evening of keynote programming here and bring the show to a close on a high note. For the first keynote of the final block, we're going to be hearing from Google DeepMind. Uh they unlike every large company out there have gotten good at naming things somehow. I am loving their model nomenclature.
07:35:05 They're naming them after elements now. Argon, carbon, hydrogen, and surprise. Oh, come on. That was a good dad joke. Ah, there we go. Okay, you can hear the penny dropping. Yeah. Uh, here to wind up talking about how they do postraining for long horizon agentic coding. Please give a warm welcome to vice president of Gemini postraining lead at Google DeepMind, Melvin Johnson.
07:35:54 Hey everyone, good to be here. I'm Melvin and I lead post training for Gemini. So today I thought I'll take you through a little journey of how we post train Gemini 4. And our journey begins by people calling us not a frontier model. Um so here's an example. You know people say Gemini is cooked but GCP is cooking. For all intents and purposes we believe DeepMind is no longer a frontier lab.
07:36:20 Ethan Malik who we respect a lot. He says in his own nice way pretty long gap between launches is what he calls us. And I'm here to tell you we're not dead. We're here. Uh and we have launched uh Gemini 4 Argon. We announced Carbon and we launched Carbon all within the last 28 days. So we're on a we're you know we're on a run here. And that's not it.
07:36:54 Uh in the process we actually garnered an apology from Dylan. No, he says offic official apology to Google Google. Um they actually cooked they're not cooked. Um Ethan says you know one of the most significant aspects of Gemini 4 is the speed at which we're launching things. Um so we've gone 236 days without having a Frontier model and we've had two Frontier models in 28 days.
07:37:18 That's a 10x improvement in our speed of shipping. And you know, people call us Google kind of cooked, but Google's not cooked. So, it's good um to see see that sentiment. It's glad to be on the other end. It's also good to be on the receiving end of the bad comments because it pushes you to do better. >> I want to start by playing a little video of a bunch of things that uh Carvin has built.
07:37:49 If we can cue the video, please. There we go. Go back. There we go. Did that work? No. Let me see if going to the laptop is going to fix it. Thank you. Now kudos to the model for building that video uh itself. So you saw great examples of the model doing much better at 3D model building assets using Blender. There's lots of games in there. The real world use cases, model processing, multimodel documents.
07:39:05 Uh but you know, it's not just the uh the demos. Carbon's actually number one on a bunch of real world customer use cases. And one of the things that we're really excited about is, you know, it's not just uh really good on these benchmarks, it's actually much cheaper. Um so on Cursor Bench, it's the number one model today and it's 3.3 times cheaper than the second best model.
07:39:31 On Perplexity's Wander, which is a benchmark for deep research um queries, it's the best model. you know you saw um competitor models come out were 9x cheaper than the second best model there and on automation bench which is Zapier's uh benchmark again were number one while being two times um cheaper per task and that's not it I mean today we announced the new element uh to our Gemini 4 series and it's not surprise uh it's uh hydrogen uh So, you know, you might have seen the launch earlier today.
07:40:11 Gemini 4 hydrogen is where we're trying to bring the Gemini 4 intelligence to everyone. It's fast, it's dependable, it's launched on all surfaces on anti-gravity, on the Gemini API, in the Gemini app, and on search. Um, and you know, I have a graph that shows the AA index with many of you might be familiar with, and time on the x-axis. You can see it's one of the fastest models for that level of intelligence.
07:40:41 And to give you a sense of how far you know Gemini has come 3.8 flash on terminal bench 4 was about 20%. And hydrogen is about 60%. So we're three times better performant while being three times faster. Right? So that's no easy task and we're you know really happy to bring that to you in your hands so you can build things with it. And that completes you know our uh Gemini 4 series.
07:41:14 So we have Gemini 4 carbon a frontier model for deep frontier intelligence Gemini 4 hydrogen which is fast for everyday uh execution. And we'll have more elements. You know, of course, there are many elements left in the periodic table and we have a lot of work to do to exhaust them. Uh but we're glad uh to put it out there for you all to test it and for you all to play with it.
07:41:37 Um now I think enough marketing with the models. I want to I was told this is a deeply technical audience. So I want to talk a little bit about why post- training these frontier models is hard and what goes into that. Let me start by saying, you know, AI writes about 75% of Google's new code today. And this is up from less than 25% a few years ago. And there are 100,000 people using anti-gravity writing code inside Google every single day.
07:42:14 Now, the complexity of the tasks that people do have also increased exponentially in the last couple of years. We started with you know models just completing chats which took seconds to writing small functions within the minutes. You can submit a PR and resolve an issue takes about an hour let's say but now people are doing days of real engineering tasks with the model.
07:42:36 People have swarms of agents solving real open problems with the model. We have engineers across Google using anti-gravity and our Gemini 4 models to solve tasks over multiple days. Now, how do we go about post- training Gemini for real engineering tasks? We have to start with hard tasks. So, we need really need to source hard problems for the model.
07:43:01 Grading of these models is actually a very hard problem. So, you want to grade like an experienced code reviewer would grade. You want to grade like an experienced engineer would grade a solution. Large scale RL is how we train the model. And we want to dog food our models with our 100,000 engineer community and repeat the loop. So any failures in the dog food become tasks for the next iteration of the loop.
07:43:28 So I want to start by going into a little bit more detail of each of these sections. So let's start with the hard tasks, right? Um so you might have heard you know as models are getting better lots of benchmarks are getting saturated and it's not just the benchmarks that get saturated our training data gets saturated too. So here is a graph that shows for three different iterations of the Gemini 4 models how much uh the share of learnable prompts has gone down.
07:43:54 So let's say we had x% of learnable prompts for the first iteration. By the time we finish and train the next one the prompts are too easy for the second one. We finish that and train the third one. The prompts are too easy for the third one. So what's the problem with training on easy task? Easy task give you zero learning signal, right? So you have a prompt when all of your rollouts are going to be correct.
07:44:15 There's no variance. There's no RL signal. So you really need to focus on getting hard tasks and you really need to be faster at getting hard tasks than the models are getting better, right? Um so how do we go about it? We first have to you know identify these easy tasks and prune them. So we cut these simple band style tasks from the mixture that are too easy for the models.
07:44:37 We expand into much harder sweet tasks for example that are multi-file repos. We go from unit test or binary correctness um to rubric based grading. And rubrics give you dense signal. They're not just 01. there can be um more gradients within the um within the criteria. Here are some examples of hot tasks that we train on. So uh we look at multifile repos and we extract commits from them.
07:45:10 From those commits we have models automatically extract rubrics. Um we don't actually ne necessitate unit tests being involved. We can actually uh generate rubrics on the fly based on these repos. Another example, a great example of tasks are you try to teach the model to build uh CLI tools from scratch. So you start with an empty folder and a specification.
07:45:33 The specification says, hey, you have to build this tool. It's called GP. These are the parameters of GP. These are the flags. This is the functionality. And you let the model build the entire tool from scratch. And you know what the functionality for GRP is going to be. So you have a hidden test suite that you can evaluate the binary on. The last uh example is production codebased diffs.
07:45:59 So you look at real usage of the model inside your production codebase and extract useful tasks out of them. Now let me talk a little bit about uh the graders. So agents you know they find any way you're you're training them to pass the test. So they will find any way to pass the test. But if the test pass, that doesn't necessarily mean that they've solved the problem.
07:46:24 There are examples where the agent will bypass the checker. There are examples where the agent will game the benchmark. There are examples where the agent will fake the evidence. You know, it writes the uh fake evidence into a temp file. It'll delete any evidence of it faking it, right? Um, so you have to be smarter about how you judge these models, especially as they become more capable.
07:46:50 Now this is what I meant by grading like a reviewer. So if you've done a code review, you have a very strict reviewer, you're going to know they're not just going to think about the correctness of the solution. They're actually going to think about the style. They're going to think about maintainability. They're going to think of many different aspects.
07:47:06 So what we do is we move from unit test based correctness to rubric base and these are generated from real commits. The rubrics also provide dense learning signal not just 01 signal and we penalize any sort of solution that is looking for a workaround. Right? So here is an example of a rubric where the criteria is you know here are the edge cases that need to be passed.
07:47:32 This is an example of hygiene that needs to be maintained. These are constraints around verification that need to be satisfied. Now, it's not just about the rubrics needed for grading. We also need to be careful about the graders not being gamed and you really need to harden the environment that the grader is in. So, we think about putting the graders in a isolated sandbox.
07:47:58 So we remove git history and git remote so the answers don't leak. The test sets are held out. So only when the agent is done and submits the solution do we actually reveal the test. We've had instances where the agent will fake it and say hey I I'm done and the tests come into play and then it'll look at the test and resubmit the solution. Right? So you don't want that.
07:48:19 Um we also want to be much more detailed with our scoring. So we take the full diff and we look at the reference and we look at multiple aspects. It's not just the correctness. We also want to look at the code quality. We also want to look at the scope uh of the change that was done. And then finally, we want to penalize any sort of cheating attempt by the agent while being careful not to police the thoughts.
07:48:44 Um so any you know cheating or hampering of the environment gets a very negative RL reward. Uh and all of this is detected just based on the actions that the model is taking, not the thoughts. Um so a good example of how we've handled cheating is you know carbon for instance has 0% cheating rate on terminal bench 4. Our previous models had a much higher cheating rate and it's about 5x less adversarial than argon.
07:49:11 So every generation of the model we're trying to make it much more aligned so it's not trying to cheat. Now finally I think uh I want to talk about model behavior a little bit and modeling honesty specifically. Now agents they always want to be helpful and sometimes they fake helpfulness. You might have seen this in your usage of the agents. So sometimes they'll claim that tests have passed when they've obviously not passed.
07:49:40 They'll declare done but obviously they're not done. They'll make edits that they'll claim to have made edits but they've not made those edits. Or they'll invent facts and APIs and whatnot. or they will give you a wall of text that has so many numbers that is going to be impossible for you to verify. Right? We've gone through all of these examples. Now, here is an example of fabrication rates in Gemini.
07:50:05 Right? So, we on an internal eval started with 8% fabrication in that hard eval. We cut that down dramatically to about 3% for hydrogen and carbon is half of that is about 1.5%. But we're striving to get this down to zero. Like you don't want your agents fabricating anything, right? But they try to fabricate because they want to be helpful. And you really have to try hard to align them to be helpful but not by faking it.
07:50:37 How do we reduce the fabrication? Believe it or not, you have to clean your data. Uh right. So we filter out lots of training trajectories where there is you know not just uh intended fabrication but unintended fabrication. There's lots of claims that are not fully supported. They're not verified. We ground in the full trajectory. So if the agent says I've completed the task and it has a final response, we actually look at that response and see all of the actions that the agent had taken in the trajectory to get to
07:51:08 that final solution and we look at the steps that were taken. So you really need to when the agent says I'm done and gives you a summary that needs to be grounded in the actions that were taken. And we want to reward honesty. So if there's a hard problem, it's better for the agent to say, "Hey, this is my partial progress. It's a hard problem. I'm done here.
07:51:28 This is my partial solution." Rather than faking the last step and saying, "Hey, I've solved the problem. You know, this is done." Finally, shipping in weeks. So how are we able to ship so rapidly and the biggest reason for that is our internal uh team of 100,000 plus engineers at Google using the model for realistic tasks. You know many of you might know Google is a monor repo and what is a monor repo?
07:51:58 The entire company is one codebase right and I don't know how to measure this but I think it might be one of the largest monor repos in the world. Now that's a really complex uh code base and we have the best people in the world using a really complex codebase to do really complex tasks. So every time we have a new candidate from post training, we actually serve that model uh for our users.
07:52:24 We run blind battles with them through multiple different candidates. We let users you know uh pick the best model that they want. We also run Elis where under the hood we're routing the routing people to experimental models and get signal and feedback from that and we take all of these signals and friction back into training the next candidate. So this is a cycle that helps us you know rapidly iterate really train on hard problems and as the models get better the engineers are going to push the model in interesting
07:52:59 ways. This is also why, you know, we've gone from releasing in weeks um to releasing to releasing in quarters to releasing in weeks. So, in the three in the Gemini 3 era, we've had launches every two to three months, but now we've had six releases in the last 16 weeks, right? Uh and we're very excited to have you all try Carbon and Hydrogen uh and Gemini 4.
07:53:29 Um, and we're excited for for for us to see what you all do with it. Uh, and we're not done. So, we have a lot more elements uh to go through. Uh, with that, you know, thank you. Heat. Heat. From helping developers navigate complex code bases to building the next generation of AI coding agents, our next speaker has been at the forefront of developer tools.
07:55:09 Ladies and gentlemen, please welcome Quinn Slack, co-founder and CEO of AMP. Hey y'all, I'm Quinn. Uh, great to see you. You might want to come up in front a little more because I do, it's like an NBA game, have some swag and hats to give out. So, if you're in the back and if you like hats, come on up. Um, all right. As whatever, you know, that said, um, my name is Quinn Slack.
07:55:38 I am one of the founders of AMP and had been at SourceCraft before. I don't want to talk about AMP though. I want to talk about really a personal story of how agents blew my mind and changed my life. There was a day when I walked outside and the birds were chirping more and the sky was bluer and how they changed my company and how I think they're going to change every company.
07:55:59 And it's kind of frustrating sometimes to see friends and people I care about, companies I care about, really struggle with this. So, we're going to go on a little bit of a roller coaster, but uh why don't we get started? I know it's almost dinner time. Uh, anyone want some hats? Some AMP hats. All right, here we go. I see people getting closer. There we go.
07:56:18 All right. Hope I don't hit anyone. There we go. And over there. All right. >> Yeah, everybody gets a hat. Everybody gets a hat. come up and find me after. All right. Uh there was a moment in January of 2025 when I was just coding and I wanted to see what if you just make an agent. This was before Claude code and it was like the dumbest stuff ever. It was like change its background color.
07:56:52 But it just blew my mind when I saw an agent actually work for the first time. It ran commands. It looked at itself. It gave itself its own feedback loops. That blew my mind. And I had this sense I think that everyone has when they first see an agent that everything is going to change after that. Everything is going to change about how you build software.
07:57:12 I think we've all felt that everything is going to change about how you form companies to work on software and how you sell software, how you distribute software. All other work is going to change. And it was just an absolutely crazy crazy moment. Um I was actually at the Frankfurt airport with Thorston Ball on our team. Um he wrote that compiler book.
07:57:32 anyone who knows Thorston on our team. Yeah, he's a cool guy. Um, we were just like minds blowing at the Frankfurt airport and we knew that everything was going to change. We didn't know where this was all going to go, but we just wanted to go. We just wanted to go run at it. So, inside of Source Graph, which was a 200 person enterprise software company, I was the CEO and we made code search.
07:57:54 We decided to just start building this to tune out everything else and just start building this thing. Not go to sales calls, not go to like exec meeting, none of that stuff. Just tune out all the distractions. And I was in this position that I think a lot of people say, you know, wow, you know, I'm a software developer at a company and I want to change.
07:58:15 I want my company to be more AI forward and I can't. Um, you know, I was a CEO and in theory I could change anything. I could change teams. I could say you don't need to get poll request code review. uh you don't need to you know do any of the other processes we used to do and yet it was very hard what we did is we ended up taking 20 people uh from the company and inside of source graph still we made a little team the AMP team and uh that's what we tried so for those of you who are trying to push back against your
07:58:44 company to try to make it more agent friendly I have a lot of sympathy for you and I hope that you can take away some things from here um I know how hard it is and also have some empathy for your CEO too. It's hard for them, too. This is a really hard kind of change and transformation. But we knew that there were some things that were going to change.
07:59:01 And this all seems kind of obvious now, but back in 2025, early 2025, this idea of enterprise ready. You do this thing and then you can build a sales team and go sell it. We knew that that was just going to totally die off. You had to be on the frontier. If you had, you know, a taste of success and then you started doing all the enterprise checklist to make the median enterprise buyer happy, your product was going to go obsolete in like 3 months and the whole game had to change.
07:59:30 Nothing mattered other than making it so that the people at these other companies loved your product and the CEO, the budget was infinite and all bets were off. So, we knew that. We didn't quite know uh all the challenges we'd run into. Um, also, you know, in this world where agents can build a product or build a feature, how do you even think about differentiation?
07:59:49 I mean, to this day, there's people who say, well, why is AMP better? Why is, you know, claude code better than codeex or whatever. And so many people are begging to have the answer in the form of, well, it's these three features. You know, uh, AMP has orbs, which are really good cloud agents, and we have multiplayer. But the crazy thing is, you all know that if we can do that, and if we prove it, then any of our competitors could as well.
08:00:11 So, you know, why even play that game? Like, what does it even mean to build and sell software? Why would you go and hire a sales team? Why would you go and do anything in the traditional way if you don't even know what it's going to look like in a month? And if you don't even know how to describe differentiation, there's no sustainable differentiation.
08:00:27 Um, for AMP, you know, people end up loving it. Um, but you can even see like, um, you know, Adam Wan, who's the, uh, Tailwind guy, he loves AMP, but he's like, you know, it's one of those things where you can't understand it. uh you know it's you can't understand why it's good until you use it. Can you imagine having a whole product management team, a design team, an engineering team that's so disconnected from the actual customers, a whole company apparatus if that's the best way that you can market a product, it's
08:00:56 really tough. So we knew that we just had to change everything and we we rethought so many things because kind of unintentionally we were in under some weird constraints. We decided to spin out AMP. So for those of you who've been at a company like a small uh you know you're on a small team at a company you're trying to do things different you're trying to be the startup within a startup you probably have these fantasies of let's just go off and start a new company well we actually did that we you know did the right
08:01:21 thing by investors and employees and everyone has ownership in both but we actually spun out and for a year until last Friday we couldn't hire anyone we couldn't grant options the legal spin-off process was a huge pain and we were forced to make do with the 21 people that we brought We didn't have time in this crazy competitive environment to do uh you know we couldn't move as as slowly in order to do um you know mandatory poll requests.
08:01:47 We couldn't have meetings. We couldn't have our time be sucked up by enterprises that would need to take you know many months to decide on something. And so we were forced into this radically different way of working. And we also had this environment where every single person on the team has a huge stake in the company. And that skin in the game that you get when AI makes it so you can operate with way less people.
08:02:09 Skin in the game, it's like this spiritual thing for the soul of of great people. It makes them better at everything they do. It makes them grow faster. And so this is what we have seen. Uh this I think when you've seen a company that operates like this without any of that it's obvious how companies like this are going to crush everyone else and how every company in the future is going to look a lot more like that.
08:02:31 So yeah, we spun AMP out and we had to figure out a lot of new ways of working with these constraints that we had no idea that we were going to be under and we couldn't hire as I said. So as a result while our uh this is a log chart by the way while you know all the other big competitors in the space were growing and building old-fashioned companies in order to go and exploit this amazing technology of AI we had to stay at 21 people.
08:02:55 But that was awesome because it was a forcing function for us to work differently and to figure out how can you do so much more with an agent where we we couldn't hire people. There's actually one guy who was like, I don't care that I can't get options for like eight months and I'm going to come join and Hamish is uh pretty awesome. So I want to talk about the ways in which we built AMP which is our agent product around the way that these constraints forced us to work and how we grew up around the agent.
08:03:21 Here we got rid of all of the other tools. We got rid of all of the other things you got to check. Obviously not Jira, no task tracking. We don't have a support system. Everything we do in the agent in AMP, they're called threads. We even communicate in the agent. You can say um you know, I made a something's bad in the product. Let's do it. The agent can listen to everything and then go fix it.
08:03:46 If you're in like a a local agent that runs on your laptop, you can't bring someone else in if you realize you're in over your head. Really valuable. We push way too often. Um, and I want to show, you know, how we actually reduce the risk here. Because you might say, uh, let me let me start that again. You might say, well, if you're pushing constantly, then how do you not take down the site all the time?
08:04:05 And the difference is when you can push and you can do it any time a day just by yourself. You can chunk up your work into way smaller pieces before you're going and doing a big change. You can go and say, "Let me add the logging to make sure that my understanding, the agents understanding the system is correct. Let me push a little change, a little change, you know, and then the agent can monitor it for an hour and then push the next change."
08:04:27 You can run a whole shadow system side by side with the old system to see does it compute the exact same results in every single case. And that way of working actually reduces the risk because by the time you know you finish the old way of doing it and the pace that things have to move these days and what your customers expect, you probably forgotten what you're building and it just totally sucks.
08:04:48 Um, let me give out some more hats. Who wants some more hats? All right, there we go. Sorry, the one guy who's on his laptop. That's a lesson, right? There we go. All right. Um, another thing is making it so the agent is able to irrefutably prove what uh the the thing it's building is correct. Um, there's a lot of ways it can do this. Turns out they're really good at proving these things.
08:05:15 So, if you're trying to remove like jitter in the UI, make sure that when you load it's like nice and crisp, you can ask the agent to do something that no human would ever have the patience to do, which is take every single frame and compare them visually. And that's what I did here. You can ask the agent to go make a ton of different versions of something.
08:05:33 You can never ask a human to like, you know, if you ask a designer, go make nine iterations or nine styles of this. They're going to get really annoyed. I know because I've worked with designers and I know they don't like that. But for the agent, yeah, you know, you can have it make a ton of different things. This is the uh waveform for dictation in AMP.
08:05:50 And this is one of the solutions to people who say, "I don't like waiting on my agent." You know, they're asking for one thing at a time and then they say, "Make this little change. You shouldn't do that. go have it make a ton of things at once and you're gonna get a lot more done. We have this magical system where if you report a bug in AMP, has anyone here reported a bug in AMP?
08:06:08 All right, cool. Uh I apologize for the bug, but we have a system that is a full loop where when you report a bug, we will fix it. Of course, you know, Android app, we'll fix it obviously with the agent. I think everyone does that now. I'm mostly everyone. A lot of times still the engineers are different from the support team, which is kind of crazy.
08:06:31 Um, we'll send an email to the person thanking them. We give them free credits, which is always kind of nice. I think it sucks. I think it sucks when there's a bug in a product and the company doesn't apologize. I just think that's like morally wrong. And then what we'll do is we'll go through the logs. And this is something that no human would ever do by themselves.
08:06:49 We'll see everyone else who was affected by that issue because where there's smoke, there's fire. So, you find the 17 other people that hit that same obscure error. They remember it probably and you proactively tell them, "How many times have you gotten an email from some product that says, "Hey, I know that you saw an error yesterday and that was really annoying.
08:07:06 I'm really sorry about that. Here's $10. We fixed it." And by the way, that it's a self-perpetuating cycle where then people who get that know that you care and they're going to send more feedback in the future. And then you ship those all out and then you know, boom. And that's the cycle that people talk about vibecoded products being junk. AMP is basically 100% built with AI and people say it's very high quality.
08:07:30 And this is one of the things that you can do differently with an agent, but there's no way in hell you could do this if you had to go through a PM or go to the team that owns this bug and the designers each time. Um, dictation. Who dictates to their agents? Yeah. Wow. That's ah, that's pretty good. That's a lot more than I thought. It is the fastest way to get bits of information from your brain to the agent.
08:07:54 And uh I think that there's probably something around it uh ushering in the return of remote work because I was just in an office today with um Bang, another AMP co-founder, dictating and it was uh very annoying. Um dictation, if you're not doing it yet, you got to do it. Just overcome the embarrassment. uh you know uh channel Nathan Fielder and tell yourself that you can withstand any amount of awkwardness as you talk.
08:08:22 We've hooked up our agent to a lot of different tools. I think a lot of companies have gotten here, you know, to this point. We've been doing this in production for a long time. And what we found is the more that we start using these tools where just the agent is using it directly, we don't even need the software product on the other end. And it's a very small number of products that we think actually have sticking power for us.
08:08:44 So um you know we built some new things in AMP for example a CRM and a year ago you heard people talking about oh no one's going to vibe code Salesforce from scratch and yeah that's totally true. Why would you want to? It's have you ever used it? You would you be proud if you vibe coded that and that came out the other end? And also you use like 2% of the features of Salesforce.
08:09:06 So actually, you know, this people ridiculed this idea, but actually would you code the 2% subset of Salesforce features and then another 2% that integrate with the rest of your stack in the most perfect way because the agent is so good at that. And that, you know, 4% of Salesforce total, that's like the easy And you know what you want out of that better than Salesforce.
08:09:31 And you certainly don't want to spend your time talking to some Salesforce product manager and hearing from them. I'm gonna advocate with my PM and my executive team to get that on our Q4 roadmap. All that So, compared to that, it's actually getting easier and easier to just build the tools that you want from scratch. And this goes back to some of the existential kind of dread but also mindblowing feeling that we felt when we first tried an agent.
08:09:57 This is what we kind of thought would happen that it would change everything about the software industry if Salesforce which is like the king cobra of SAS they have the most reliable customers most reliable renewals all that they like invented the category if we're starting to see you know why would we use them well everything is starting to fall and all of SAS starts to break down if you can no longer assume that you can pay a salesperson a 100K for closing a and that deal is going to last for 10 years and it's going
08:10:29 to get bigger by a little bit. Not all of that is called into question now. So if you're at a software company and that is how you ultimately make money and you're trying to ring the bell, you're trying to raise the alarm here, uh absolutely raise it and get out of the cage that you are in because uh in my opinion once you have seen how the new kind of companies that are two orders of magnitude smaller where everyone can do everything and where everyone has skin in the game, then you know you don't want anything And so
08:11:02 this is what we call this new kind of company where you know we take that old or chart and now we just got 21 people on AMPs. Brady is this guy who is an SDR at source graph. Really smart guy. And one day he's like I'm going to make a wire a wire transfer thing so customers can pay us you know if it's too big for credit card and I'm and he's like two years out of college and he's just doing that and nobody knows on the other end.
08:11:30 It's like that old New Yorker cartoon on the internet. Nobody knows you're a dog. You know, nobody knows that he's an SDR that we're all kind of, you know, devs and doing everything, but you can. And you also get people that are really real with customers. And I think sometimes customers need real talk. So, Alex Keer on our team, you know, just today, uh, he was like, "Hey, you know, to a customer who wanted our stuff, that sounds like we're just going to get bogged down in a bunch of BS, and I don't think that's a
08:11:55 good use of anyone's time." If you can get to a place where you have people that feel empowered to say that and to truly say the best thing for the customer, that's great. You're not going to get that out of a salesperson who is commissioned. We have this some people's favorite part. Uh we have one meeting per week and then everything else the guidance is follow your energy.
08:12:12 um you couldn't run most companies like that, but if everyone at the company has a massive stake in the company, you know, take 100% divide it by 21, you know, and all that, you can actually trust people because again, when people have skin in the game, it makes them better at everything they do. It's like an incentive to make them think whole company, to make them do more, to make them work harder.
08:12:34 And that's why I included uh Saturday and Sunday as well on this. And we are shipping uh just, you know, so much more. Um, I think that, you know, AMP for many people is a product that they love. We also have a great business and we're serving as proof that this new kind of company that is profitable that's more than 100 million ARR that is doing so on a dollar, dollar out, you know, basis that's 40 to 50 times more efficient than old-fashioned companies here that that that can work.
08:13:04 And so if this is appealing to you, if you want to push your company in this direction, if you want to go start a company like this, then um reach out. I'm happy to help. And my guidance is just get out of the cage. I've heard from so many people who say, "I hate that there's a gossip channel with the people at my company. We're all talking about the change."
08:13:22 You know what? The the longer that you are in the gossip channel, the longer that you are at a company getting paid a lot of money and you're really smart and you're not doing about it, you're starting to become part of the problem. And the longer that you let yourself be just a developer when agents have completely changed the game, you're really hurting yourself.
08:13:38 So if this is, you know, the one thing that you take away, just get out of that cage. Let agents really shape how you work and get to a company that's willing to be shaped by the agents. Thank you. And um if you want to try AMP, you can get up there. You can copy that. I've got more hats, so find me afterwards. and I'll probably be getting dinner at Chipotle or something.
08:14:02 So, come find me if you want to uh chat about all this stuff. All right. Thank you all. As AI agents reshape how software gets built, what does the next generation of software factories look like? To explore the design patterns and principles behind Agentic Coding, please welcome Dex Horthy, CEO and founder of Human Layer. What's up everybody? How we all doing?
08:15:52 >> Give it up for all the organizers out here. They are killing it. Swixon team, let's get a round of applause. All the speakers, keep it going for the speakers. Uh, I knew Quinn was really smart, but that was a really dope talk. I enjoyed watching it from backstage. Um, hi everybody. Uh, I'm Dex. This is 12 factor factories uh which started as a joke title but then everyone started calling it that.
08:16:13 So uh and the war on slop. Uh so thank you all for coming. Um I like to yap about coding agents on the internet and like a year ago we did the whole context engineering thing. Uh a lot of views on YouTube and up until about a week ago uh I was the CEO and co-founder at a company called Human Layer. Uh the multiplayer coding agent workspace. Uh and for those of you who are blessed to not be uh quite as uh terminally online as uh some of us may be uh we are now part of factory which I'm very excited about and I will be
08:16:44 touching on during this talk. Uh but I want to get right into it. Uh last summer was all about this like you know three months ago feels like forever factories were all the hype. Um, and I gave an intuition based talk back in June of like we should probably still read the code and here's how RL and verifiers are working and like exactly as um, our friends at Google were saying, you know, models really like just solving the tests, just getting them to pass.
08:17:05 Um, I now have some data beyond just that intuition. Um, there's a benchmark called slop code bench which we just released actually today uh, a bunch of new data on uh, from the lab at University of Wisconsin. Um but very unsaturated benchmark you know GPT 5.5 got a 15% Astra I think scored 16.3 so like we are getting better at this but there's a lot of a lot of ways to go I'll talk all about that data in a sec we of course have the study from Ferros of you know 30% of PRs merging without review longer comments more
08:17:34 comments more incidents per PR more bugs per developer uh and I I titled this talk I don't I don't think it was quite correct I said why software factories fail and everyone said oh Dex is the anti-software factory guy Dex hates software factories. I actually love software factories. I'm a huge fan. Uh the the title was probably more along the lines of, you know, there are certain things you could do in your software factory that would cause it to fail.
08:17:58 And uh here's what those things what I think those things might be. And you know, please uh please don't do them. Uh maybe the dark factory thing. Um but I think the main takeaway here, and I think we've heard this from a lot of folks, uh is software factories is just good engineering. um friend of the friend of the friend of the pod, friend of the conference.
08:18:15 Uh Lauren was talking about this yesterday of like a lot of this is just stuff we've been doing for a long time. Actually, Eno from factory had a really the post is much longer, but it's like the factory mindset is about how do we standardize our process, not just the output. Um and there's a whole bunch of stuff we'll dig into there on like how we're thinking about that.
08:18:32 Um but I try to kind of make the same point, right? is like your standard software factory is going to look something like you have some engineers, some product managers, maybe a CEO or someone to drive the vision and uh those people are going to come up with stuff that we want to build and they're going to put it in a tracker. This is again this is pre-ai.
08:18:50 A lot of the things we hear about software factories today were around for 30 years, right? You take stuff off the tracker, you build it, you run some unit tests, automated tests, maybe you poke it locally. When you're ready, you make a pull request. At that point, someone might pull it down. Uh interact with it, test it. Um your CI is going to run a bunch of checks and static llinters and like none of this is like that new.
08:19:13 Uh and if anything is wrong with it, you send it back, you go fix the problems and uh when it's actually ready to go to prod, we do some canaries and rollouts and feature flags and all kinds of fun stuff there. Uh and eventually it makes contact with our users. And uh I love our users. I love customers. Uh but uh they do like to complain about things that are broken and uh ask for stuff that we haven't built yet.
08:19:36 Uh and so that goes back to the team. Uh and at some point something breaks at 3 in the morning and the customers are the first one to figure it out. So we add monitoring so that we can wake people up at 3 in the morning and they can solve it before anybody notices. Uh this is your preAI software factory and the basic point is like all you need to do to make this agentic is replace someone builds the thing with an agent builds the thing.
08:19:56 Uh so you have orchestration. And I'm not going to go into the details of the sandbox and the model and all this stuff. Um, but you can incrementally build this up by adding what I'll call forward pressure. What causes work to start? Can we take all of the feature requests and just route them straight to our factory? Can we take all of the incidents that happen so that when I get paged at 3 in the morning, I'm looking at a pull request that we think is going to solve it, not just uh a stack trace about what went wrong.
08:20:20 Um, so the forward pressure, what causes new work to start? Uh, and back pressure, which is sort of like, okay, cool. Now we can build things in minutes or hours. Uh but the review and testing still takes hours or days if we're doing it in the old way. So we bring in agentic code review. Agentic testing. Anyone doing agent agents code reviewing their poll requests?
08:20:41 Okay. I thought it would be a little bit more, but that's pretty good. Y'all y'all are catching up. Um and then we invest heavily into our CI/CD and our llinters and our doing like lean checks on their code like formal verification. All right. Hardcore. I like it. Um so the back pressure. How do how do we let agents hill climb to a better outcome without needing to involve humans?
08:20:58 How do we just overall increase the quality of what is coming out of our build step? And agents help here, but a lot of what makes software factories good is stuff we've kind of had for decades. Uh, and just like context engineering, uh, which was a thing that I was talking about last summer in like June of 2025, there was all these ideas of like rag and memory and state and history and all this stuff.
08:21:19 And we tried to like distill out what are the timeless things that go into building agents. Uh, and so I want to distill out now some of like what I think are the timeless things that go into building and designing a software factory. Um, because there's a lot of noise and hype. Uh, like I said, termally online, Twitter is going to break your brain if you, uh, spend too much time wondering what everyone else is doing.
08:21:37 Um, I'm also going to talk about slop uh, because I have a very fun project that I've been working on with Swix and a couple handful of other speakers here. Uh, but we'll get to that later. Um, we're going to do 12 great practices for software factories. And I think the first time I started talking about this, it was only like nine, but we are at 12 now.
08:21:54 So, I'm very excited about this. Um, I touched on this, but loops was this concept that was very in vogue. I think this is much more useful if you think about the uh interplay of forward pressure and back pressure. Um, so, you know, again, very hyped up, but the actual tactical thing is like how do you start work automatically and how do you make sure that whatever comes out and is touched by a human by the end, all the lowhanging stuff is already fixed.
08:22:17 The outcome is as good as it can be uh autonomously. Uh number two I will call know your enemy. I will still posit that uh unattended uh models don't do a great job of maintaining and improving codebased quality over time. So if you care about having a system that's going to be still maintainable in three or six months. Um I mean again I gave this talk I said please read the code because a lot of it is slop.
08:22:38 Uh agents got really good at solving one-off problems and making video games and Blender slop and all this stuff. uh I don't think you know if you steer a model to fix your codebase it can go fix your codebase but then you have to know what is there and what you want uh so if you are just hands off pushing features into the machine you may not find that it autonomously will make things better over time um back then I didn't have a benchmark now I have a benchmark uh that basically has the model solve problems in
08:23:06 checkpoints like a model can clone all of Excel or Reddus in a couple days uh maybe even a couple hours now with like agent swarms uh but that requires knowing the entire problem in advance. And most of us don't write software that way. Most of us ship a feature and then ship another feature and then ship another feature. And if you're building by hand without AI along the way, you say, "Oh, that's a bad abstraction.
08:23:24 We need to centralize this or we need to remove this." And so this is a benchmark that metals measures a model's ability to evolve a codebase over time. Um, and so Astra again is 16% very unsaturated. Um, there's a bunch of data here. I probably don't have time to look at all of it, but we can look at uh code quality metrics like uh cyclomatic complexity, lines per symbol, high the the percentage of functions that have high complexity.
08:23:48 There's more. There's 200 custom rules. Uh 50 of them are based on things we've been doing for a while. A lot of them are custom based on looking at code that models write, comparing it to preAI repos. Um I wrote a long article about I've wrote a bunch. uh Gabe Warlansky who runs the lab that does this benchmark just uh put up this paper today u based on some of the research we did with them.
08:24:07 Uh there's a bunch of stuff in here. I probably don't have time to go through all of it, but they also test the model's ability to design good libraries and interfaces. Uh the most the craziest one here is like models did learn uh to use exceptions instead of returning tupils in Python. uh there was a lot of patterns that were identified as like probably not great uh that were way less present in the you know six months ago models than they are in today's models.
08:24:31 So it almost feels like we're like learn the the slop is amplifying over time. Um and Claude and GLM are now writing Python 3 in Python 2 style. So we have uh you know percent s uh format strings are back and they will not use fstrings by default anymore. So things to be aware of. You may not care about this. I care about this. Um the juryy's still out on Fable 5.2.
08:24:51 I love reading the Enthropic blog post about like how to prompt the new models. And so the allegedly if you tell Fable it owns the codebase, it should keep it clean. I haven't tested this yet. We're running the benchmark in the background right now. Uh but for those of us not building FPS games, shouts out to Ben built a cool game. Uh you should go watch this video.
08:25:08 It's insane. Uh you should know your enemy. You should understand that like work with this in mind that if you need your codebase to last a long time or your codebase has been around for a long time um we should probably maintain understanding of the system and the code. Um to do this we like to plan before we build. Uh I highly encourage this. You should do this.
08:25:27 We used to do this before in this old software factory. People realize especially when building things took uh hours or days and when testing and reviewing things took hours or days. Even if this part is faster, this part is probably still slow. if you're doing it properly. We would have architecture review and we would have sprint planning and we would talk as a team and we would align in the hopes that we might decrease the percent chance that we'd have to rework something later.
08:25:50 We wanted to be rather sure that we're building the right thing in the right way before we actually go and spend time building it time in tokens and then get to review and have to rework it. Um, a lot of people uh overdo this as well. I've overdone this for sure where you spend six hours working on a spec and you're like, I feel like I could have just yoloed this.
08:26:08 Uh because sometimes you can just yolo two sentence prompt and if it's smaller, you don't care. There's maybe a 50% chance you send a couple more prompts and then it's good. You could spend five hours handwriting a detailed spec and maybe there's a 10% of it you have to change afterwards. Uh or you could write all the code by hand in 20 hours and then there's like a 0% chance, well there's 0% chance you have to rework it because of something the agent did.
08:26:29 You know, people still make mistakes, too. Um, but I think about this in terms of expected pain, like how likely will this have to change later times how hard will it be to change? Uh, I don't argue with the model during planning about what color the buttons going to be, but I do want to make sure we get like our database schemas right and stuff like that.
08:26:47 Um, because it's really hard to change later. So based on your problem and what you're doing and your codebase, you should probably decide where on this spectrum you want to land. Um, align visually. Uh I published a skill called show me which is like optimized for instead of reading walls of text. You get HTML mockups, you get tables, you get charts, you get all this stuff that you know is optimized for uh the human brain which is very visual.
08:27:09 Uh we've been I think KOD said, you know, we've been running away from predators and eating fruit for uh for millions of years. We've only been reading text for about 5,000. Uh anyways, you can get this. It's got about 40,000 installs. Um you should help agents test their work. Um screenshots, browsers, videos, curl, whatever you can do to help the agent.
08:27:26 again, he'll climb against an outcome. Give it an if you're building an iOS app, make sure it can build, run, and test that app in whatever environment it's running. Um, you should fight slop with anyone here writing in Typescript. Okay, some people are feeling lazy. I feel like it should be more than that, but um, this is a repo from Dylan Moy, who's a principal engineer at Cloudflare Anti-Slop.
08:27:46 If you haven't, if you're not using this, you should go check it out. It's a hundredish rules on like stopping agents from building slop. Uh, Factory also has an ESLint plugin that I should probably mention. Um, and also again the you can go slop fork all of the uh Python code quality rules out of slop codebench. This this this exists for your language.
08:28:03 You should use it. Um, slop is a tool. Not all slop is bad. Vibe coding, prototypes, side projects. If you have something behind a feature flag, you just need to like figure out if people even want it uh before you go put in the time. I think it's like it's great. I don't think there's like always always always every line of code has to be perfect. Um, you should give models a feature flagging system, uh, post hog or stats or launch darkly or whatever it is, so they can try stuff and see if it performs better and then
08:28:27 make decisions about, uh, what the right thing to be in your tool. Uh, you should compound over time. Eval everything, learn from past sessions. Uh, and you should unify your forge. Uh, what I mean by this is, you know, we used to send git patches around on mailing lists. I think Lionus and the Linux people still do this, but most people keep all their code in GitHub.
08:28:46 Um, and so before the ANAF forge, we have centralized patches, but uh we still have uh people pasting like clawed answers back and forth in Slack and hopefully like not doing too many rounds of that because it's like I copy out of mine and send it to my coworker, he paste it into his agent and then they send it back to me and then I paste it into my agent, maybe we get an answer.
08:29:06 So like centralizing all of your designs and decisions and like whatever you have in like notion and Google docs and whatever it is uh should like you should try to bring those things as close as possible together so you can understand not just the code about what something was built but all the decisions that went into it. Um you can't buy everything.
08:29:21 I think there's things in this space that just cannot be productized. Obviously you can buy VMs or you can bring your own dev environment I think is very personal to most teams. Uh the harness of course you can go get a frontier harness or you can use something minimal like open code or pi and customize the heck out of it. Uh and then you have this control plane picture of like how do you collaborate on sessions and plans?
08:29:40 How do you do automations and scheduling? How do you respond to events? All this governance stuff. Uh part of the reason I'm really excited to be working with factory on all this stuff is like they have a much bigger view of this picture and have a lot more stuff solved here. uh but there are parts especially at the top here is like I don't think anyone can sell you verification of your code because only you know if you have hundreds of engineers and hundreds of repos you have those opinions and you know how to test
08:30:04 your code uh not just a black box you can buy there are building blocks that are undifferiated but the idea of software factory I think is it's a way of thinking about our work um so that's software factories I also want to talk about slop um it's factories are about how do we increase human leverage right how do we get the best return on our time. Uh, and I've talked a little bit about why humans should be involved and like what it means to be a diligent steward of what we're building and what we're doing.
08:30:30 Um, but it's I think it's bigger than just software. So, I'm very excited to announce the uh anti-slop manifesto. Um, which you can go if you go to slotmanifesto.com, you can read it. I'll put the URL up later. Uh, but we built this in collaboration with a lot of people here, both attendees and speakers. Uh, special shouts out to Swix and Phillip. Uh it has signers from uh Zoom Info and uh I think XAI and OpenAI and like there's a lot of cool people on there.
08:30:58 I encourage you to go read it and check it out uh and be a part of you know but I'll tell you what's in it. Um we value outcomes over outputs. We value understanding over speed. We value our own thinking over whatever the model came up with as a first draft. Um we value better work over easier work. Um, we value improving the system over just patching outputs.
08:31:20 Um, and the things we believe are that AI is the most useful tool most of us have ever worked with. But it can create a like a swarm of agents can produce more work in a single day than a human can review in their entire career. Look at some of the math stuff coming out of OpenAI. Um, slop is when we put less effort into producing something than we ask their consumer on the other end to spend understanding it.
08:31:44 Um, and using AI doesn't necessarily make something slop. Um, but it happens when we pass off AI output without taking responsibility for its quality or for the reader's experience. Um, I believe strongly been saying this for a year now. AI cannot replace human judgment. It can only amplify our judgment or uh maybe perhaps our lack of judgment in certain cases.
08:32:03 Um, and the slop is asymmetric. um the cost of producing a th a thousand line PR or a 10-page doc is approaching zero, but the cost of reviewing it is still uh fairly constant and slop proliferates when we repeatedly work around problems instead of solving them correctly. It comes from tools that love to add and are not biased towards subtracting. Um I'm not going to name names, but I I think a lot of slop comes from pressure.
08:32:31 Uh companies are pushing their people to redefine what is humanly possible. Um, and I think slop compounds. So the more slop sits in our knowledge base, in chat, in code, the more of it the next agent reads and copies. Um, and the more slop flying around our team, the more we lose the will to fight it. And everything gets worse over time. Communication and quality and all of this.
08:32:53 So, uh, as the sign we the undersigned of the anti-slot manifesto. H, guys, is it, uh, it's a little hot in here. Uh, I might just have to pop off the jack. There we go. Uh they couldn't give me one of these with a factory logo on it yet, but I'm sure it's coming soon. Um thus we uh the anti-slop uh undersigners resolve that human time and attention is incredibly valuable.
08:33:18 Uh so we optimize for using it wisely. Uh we fight slop because AI should make our work better, not not uh not just easier. Um we optimize our systems and tools and products for the efficient elicitation of human intent and for efficient human understanding. Um, and we do our own thinking when working with a model. Uh, we explore resources, code, prior art.
08:33:38 We really want to understand the problem before we actually get to work. Um, and we own the intent and the taste and the consequences of what we ship. However much AI helps us to make it, we own it. And when an agent wrote something that we're sending somebody, we make that clear and we actually make sure it's good before we drop it in Slack or email or whatever it is.
08:33:56 Um, and when someone sends us slop, this is also like we have to we have to push back. we have to tell people, you know, kindly, empathetically, then ask them to like review it first. Um, and most of us, it's a journey. I've been I've been on both sides of this one. Uh, so I've had to have that note. Um, I believe there will always be alpha in reviewing something.
08:34:15 Um, as models get better, we may stop reading most of the code. Uh, we may still review a plan or a design or a result, but if you don't review anything, if you bring nothing to the table, no thinking, then you're just going to get the same AI output as everybody else. um agent reviewers, llinters, verifiers, they're very useful, but they are not, excuse me, they are not sufficient.
08:34:35 So, for now, we'll still review some subset of the outputs. Uh what subset of course depends on the stakes. Um but what shipped, who's using it, and how it's helping them matters far more than lines of code or PRs merge or god forbid the number of agents were running in parallel. Uh it's cool if it gets you to the outcome. Um but unreed output isn't always necessarily bad.
08:34:56 You know, slop is a cool oneshot prompts, um, vibe coding, these are all like high leverage things that are in the right place. Slop is bad when it undermines our outcome. If we want to make a good product, a maintainable system, we want to save people time and energy. Uh, if slop undermines that, then it's bad. Uh, and I guess finally, like none of us has this figured out.
08:35:16 Uh, we're each relearning what we will and won't accept, and there's no single right answer here. You got to calibrate yourself. Um, the world moves fast. Um, my goal here is these are principles that we can use to embrace our important human role in harnessing AI safely, responsibly, in a way that creates value for everyone, um, for many years to come.
08:35:38 So, you can read and sign the manifesto at slotmanifesto.com. Um, TLDDR, uh, you know, give a care about what you're doing. Um, I'm going to stick around for Tariq's talk, but then, uh, what is that slide? All right, I'll be hanging in the hallway afterwards. Um, come say hi. Uh, keep learning, keep giving a Uh, I'm excited to keep building with y'all.
08:36:00 Thank you so much. to exploring new ways humans and AI agents work together. Our next speaker is rethinking how we interact with AI powered tools. Ladies and gentlemen, please welcome independent engineer and former anthropic team member Thoric Shahippar. Amazing. Okay, that was such a cool animation, but also this not World of Warcraft. I I'm not sure what I'm doing here.
08:38:02 Um, but hey guys, uh, I'm Thoric. Nice to meet you. Uh, yeah. Um, I wanted to talk to you about a couple things I've been thinking about since I left Anthropic. Um, it's been a wild year and, uh, there are a bunch of reflections I've had. Um, and my conclusion is about, you know, what I'm calling the sweet side of the bitter lesson and the ways I've been wrong.
08:38:29 So, I I think I got a lot of things right this year. Uh I think um I was right about models being able to edit and generate videos. I was talking about this like sometime in like June last year. Uh I was right about models being able to use HTML to understand their work. And uh I was right about models being able to help you understand and figure out your unknown to help like plan and learn about things.
08:38:56 So, these were all things I was like right about and I was happy. And I I think that like um a lot of you might be incorporating this sort of stuff in your work, but I've been wrong a lot too. And I I think this is one of those things in AI where you sort of just like memory hold the things that you're wrong about and you're like, "Oh, like I was just always expecting, you know, AI to be great."
08:39:16 Um but here are some of the things I've been wrong about. So, first, like Bash is all you need. I I think there was this like moment where the agents were really good at using bash and cloud code was built around the bash tool and so you know people are like well we'll just turn everything into a CLI like you know like there's just going to be bash and CLI and that's going to like take over the world.
08:39:39 Um that's actually kind of not how it turned out. Like I actually prefer MCP to CLI now you know but now there's also code mode and skills and progressive disclosure. like there's so many other like ways that agents can you know act on the world right and the fact that they were using bash was just like this weird remnant of you know like them being trained in a certain way at the start then uh there's like react is all you need I think uh for a moment it seemed like the models were really really good at writing react
08:40:10 and so you know if you wanted to create an AI enabled app you should figure out how to use react like you should use react native you should you know like you should just make everything on the web. Um and uh that also has not been true. I I think like now you know models can write it in HTML, they can write in React, they can write it in Swift, uh they can write in the Figma proprietary JSON format.
08:40:33 Uh and they'll all look roughly the same, right? And so I was afraid that like you know the models have been trained on React and so we'd just be writing React forever and React would be like the last language almost, right? And that's almost certainly not true, right? I think a lot of people are writing their own languages now. Um, and uh, I think one of the other things I was really concerned about was like diffusion models will kind of like eat art.
08:40:58 Like it would just be like, you know, we wouldn't have any control over these models and or the kind of art that these models are creating. Uh, but it would just like create the most beautiful stuff. And that hasn't been true as at all. Like it I think now LM can make art through code through controlling your computer and using you know Blender or uh Photoshop or Yeah.
08:41:15 or Figma or whatever you'd like, right? And so um these were all sort of like fears I had about like I think the bitter lesson, you know, art alien intelligence, the shogunuts, right? like like these are all words we use I think to describe this kind of like fear that the models will just have this like way of doing things and it will be weird and alien to us and we just need to accept it you know like you just have to like submit to this like foreign intelligence um but that's just like I don't think that's what
08:41:51 happened you know um and so I think the like the question I come back to a lot is like okay like how can we be AGI pilled and human- centered and what can we learn about you know this progression what we like you know what can I learn about what I was wrong about right so I think this is really really important because software engineers are like the leading edge of like AI adoption right like any thing that is happening to software will happen to knowledge work and other parts of the economy and so I think how we deal
08:42:17 and engage with uh with AI is um yeah how other people will as well and you know I really enjoyed the different talks here talking about things like yeah fighting flop I think you know we're the first defense uh for that, right? Or like being more ambitious. Um what I'd like to think about it as AI is both at the same time an industrial revolution and a renaissance, right?
08:42:42 And I think this year was really about the industrial revolution, right? Like we use the word software factories very often. Um and uh like just sort of like turning software from this like craft that was very unreliable and hard to execute on into something that like is not the limiting factor, right? Um and this sort of like industrialization it's absolutely happening.
08:43:11 This is definitely something that is happening with with software. But at the same time there's also kind of a renaissance happening right where people want and uh higher and higher quality software and every marginal bit of like creativity utility like spirit like soul that you put into it is like is useful and like economically valuable right and so simultaneously you've got both of these things that are very hard to juggle and and sort of like hold in your head at the same time uh but I think is very true and uh Uh
08:43:42 yeah, we're doing both of them. So often times when you talk about AI capabilities, we talk about like the jagged frontier, right? And so uh you know, you maybe like you go back two years and you're like asking AI to like write this function for you. Um or you know, like like a year and a half ago or a year ago, it's like build this feature. Then you're like, hey, build this like to-do app.
08:44:04 Build like a kind of simple to-do app and then clone Slack for me now, right? and and and now it's like okay build Mario Kart from scratch you know and then then like port it to like the Wii or something. there's like the capability increases are the things that you see right where you know whenever you look at the evals and benchmarks and stuff you see this like just um this jagged frontier of like capabilities but I also like to talk about like the smooth interior where the the way whoops the way the models is there
08:44:41 an animation playing here um the smooth interior is like The idea is that the models are um as they get better, the the animation here is like this jagged frontier and this circle in between. And so uh the circle sort of represents all of the things that the models are too are smart enough to do. And so like examples being like uh writing react code, right?
08:45:05 Or like using bash. And what you see is that like it's not jagged below the frontier, right? like if it can write react, it can write Figma, it can write JSON, it can do sort of like all of these capabilities altogether and there's like a breadth of tasks that the models are really really good at. And this is something that's also I think really like important and um for me even more impactful because I don't really care that they can build Mario Kart from scratch anymore, right?
08:45:36 What I really care about is like, hey, is the software that we're building is it high quality? you say worthy of like a renaissance, you know what I mean? So, um yeah, like I think that like you know, you can do things the way you want. So, like if you're building mockup, you could be like, "Hey, can I get an HTML? Can I get a React? Can I get a Figma file?
08:45:55 Can I computer use my computer and use After Effects? Can you make a video?" There are all these different ways of getting the like final result. And I've been really happy with this. I really was afraid that there would just be like one thing that the models love doing and we'd only be doing that. Um, and you know like taken to the limit kind of like you know we talk about this a lot but like you know everything we've talked about so far about like you know how you build like these harnesses and software factories and
08:46:24 um these like the scaffolding for the models even things like you know for a while we were also like hey um the model you'll like your job is to QA the model you know or review the code and that's clearly not the case right I think as the models get smarter and smarter um it will become more of this like smooth interior, right? Where they're just doing things well, right?
08:46:47 And like the perfect model, you can imagine won't need to like really verify its work or scan for security vulnerabilities. We'll just like write the code and be like, "Hey, I'm done now." Um and you will check and you'll like, "Oh, you're right." Like, you know, according to the spec and the goals that you that we did, you wrote you did it correctly.
08:47:05 Um and uh I think that's actually very exciting. Um one of one of the ways I like to sort of think about visualizing how how much the model knows of this eval I think aparthy um posted it but basically you ask any I think these are these are cloud models. You ask a model uh you give it a longitude latitude and you say like hey is this point uh earth or water right uh ground sorry land or water and it from that you plot a map of the earth right and yeah this is opus uh 4.6 six is somewhere over there, right?
08:47:45 Quite a good software engineering model. Kind of like a blurry view. And now you look at Opus 5.5 and it's like almost perfect, right? Like it knows every piece of, you know, these like these coordinates. It knows like the entire world basically. And so this I think just gives you an idea besides like, you know, the benchmarks about like the the the jagged frontier.
08:48:06 I feel like that's less and less important to us and more important is just like the general knowledge that the models have. And what's exciting I think about this is that we can use intelligence to understand and shape how we use intelligence, right? And I think this that will be like the theme of the next year, right? Where this year I think was really about the industrialization of software and and creating this like reliable way to make software.
08:48:31 Um, but what does that look like? So, I have been building a game. Um, it's this like thing I've been posting on Twitter about. Um, it's a 2D like isometric game. It's like it's kind of like a fighting game inspired by things like Avatar and Smash Brothers. Uh, and I have kind of like it's fun, but it's also quite frustrating to do. Like, I'm really trying to like, you know, like be like a great vibe coder here.
08:48:58 Um, I don't really know what each character does sometimes because like I will say like, "Hey, I want this character to throw a rock." And then I'll be like, "Oh, yeah, great. But like there all these like different things that can happen when you're throwing a rock. Like this character can also summon a wall. So does the rock go through the wall or not?"
08:49:17 And like how what speed or like what what about size and charge and all of these questions. Um, and I I find that like the model just makes a bunch of choices for me and then I try it out and I'm like, "Wow, I have no idea what this character actually does. I get distracted pretty easily. I don't feel in the zone very much, you know? I I sort of send off a prompt and I start doing something else and I come back.
08:49:39 Um, and I honestly make less progress because of these things, right? And so, uh, I think this is frustrating." And I think for a while I was like, man, I guess this is just like how you work with AI. This is just like what it's going to be like, you know? But I think reflecting I'm like, man, I I don't think that's right. I think that like, you know, we have this like AI can work in the way that we want, right?
08:49:59 And so, yeah, I don't think it's acceptable. I don't think that this is what leads to renaissance, right? I don't think this is what leads to like incredible software. Um, and part of it, you know, uh, going back to what Dex said is like like great work is just about like this this messy process, right, of like you don't know like exactly what you want.
08:50:19 It's more about the choices you made and what wasn't involved or wasn't what wasn't included versus like what was included. And so, um, yeah, I think there's like, uh, you you like kind of just have to be an expert. You have to just really know your stuff in order to make something great, right? In order to like make something worthy of a renaissance, right?
08:50:40 And I just like when I look at something vibe coding, I'm like, dude, I have no idea. Like, I just don't really get it, you know? Um, and so, um, what what am I doing? How am I fixing this? Um, one of the ways I'm doing, and I'm still quite early on. I'm not saying that I've like figured it all out, but I think one of the ways I'm doing it is that I have two parts of my codebase.
08:51:01 I've got like this like engine part of the codebase, and then I have a folder like I think I call it spec, but I'm thinking I'm just calling it human or something. And that folder is the code that I read, you know, and um it's like this code that's linted and like there's all these rules for Claude about like what that code looks like. It's supposed to feel like English.
08:51:20 Um it's kind of like a spec. And I'll I'll show you what it looks like. Um but it's at the density where if I just read this code, I know what's happening and I can edit it, I can tweak it. Uh, but more importantly, I just like, you know, can have a bit of a ground truth of like what the what my game looks like. And, um, so let's do do like a quick demo here.
08:51:46 Um, by demo, I guess I mean I'm just going to Whoops. This is my AIE talk. Okay, here we go. Okay, so this is my game. I've got in here this folder game. And then I've got here this code. This is the like agent code, right? Like this is the code where I'm like, dude, I I don't know how, you know, cameras work or whatever. I, you know, like there's a bunch of code here that I don't understand.
08:52:07 But then in this code, I see have all my characters and I have their abilities. And so I can go in here and I can say like, okay, this is the rock move, right? Um, and inside of this move, there's a bunch of ways to tune it. Um, it's well commented. It uses things like meters, you know what I mean? instead of like x like.5 or whatever. Um it uh tries to like create a bunch of different abstractions along the way.
08:52:37 And so I can see that like one of the things that you know is useful is that like there are three ways to throw for this character to throw a rock, right? Like it can uh grab it from the ground, it can grab it from the wall or something. Um, and I can see sort of like the um the true logic of the character, right? I I don't really know how the engine works or how, you know, like this is implemented underneath, but I've essentially created this like almost a custom scripting language for my game.
08:53:08 Um, and so that means like, you know, when I'm playing this game, um, I can tweak little things. You know, I'm still I'm still like working on it. So like I can say I want the rock to be a lot bigger. So I can say like um you know 1.5 but I'm using an actual IDE uh and so if I go here to my game uh you know I can see that like I have a rock it can charge it gets really big and I throw it right.
08:53:32 Um but I'm like kind of it it it's really interesting because I'm also starting to use cursor more and I'm using the tab model more. You know what I mean? Because now now I can be like, oh hey, what if I just want to say like you know uh if you summon a wall in the middle like cue it up as an action and then cursor will sort of like you know do the the tab complete here.
08:53:54 Um, and this is kind of like it's kind of like spec driven development, you know, but the spec is this like this this scripting language basically. And uh, yeah, it's like is this the final way I'm going to write this game? I'm not sure. It's like, you know, in some ways uh, when I need to do something complicated, I still need to ask Claude to do it, right?
08:54:16 And I need ask to claude to like make the primitives, but it's actually really easy for me to understand what's happening. And I feel like much more in control. And while like cloud is working, I can also sort of review it and work with it. Um and uh that's been really exciting. And so I think the takeaway for me has been that like um the takeaway for me has been that like I now have to think about the way the best way for me to deliver high quality software, right?
08:54:48 And I'm not as constrained and limited by like the model's capabilities and like what it likes to write code in or something. I'm just not thinking about like I need to think about like how can I deliver great software. And so I think if we like really internalize that I think then this the next year will be about a renaissance of software. And you know I think many people have talked today and you know this conference about how to be more ambitious and I don't think anything any of this is quite new but I just think
08:55:17 there's like many things to do like new languages right like obviously like you know uh we want things like Rust that are very efficient but we also want things to be uh potentially modular and like hot reloadable um we want it to be potentially readable right uh new primitives like there probably new types of databases new types of servers like um new operating system interactions.
08:55:42 I've been sort of like vibe coding a bunch of personal software um that sort of messes with my operating system. Uh there are new interaction paradigms possible. I think um a lot of like old HCI research is now like very interesting to revisit. Um new aesthetics I think the way that we like use and design software uh can change quite a lot. And um yeah, I I think that like there's so much to do here that I think I I don't think that any of this is everyone every speaker I think has talked about this and um I I don't
08:56:12 have too much more to add, but I think that like what I really want us to say is that what we need to do is to decide what the future of work looks like, right? and decide what our ideal experience is because the models are not, you know, this alien intelligence that's like trying to force us into a specific way of working. Uh they can kind of do everything and and they will.
08:56:37 Um but our job is to like bring about the renaissance. So thank you guys. That's my talk. And now before CodeSummit 2026 comes to a conclusion, there's something big we'd like to celebrate with all of you. >> Hello. Hi everyone. How's AIE? Um, that was incredible. Uh always a dream to have some of my favorite speakers in the world grace our stage share a little bit with you in person uh the thousand top aentic engineers in the world.
08:57:27 I just checked the numbers on the live stream. Uh if we add everything together yesterday and today we're basically close to 100,000 people on the live stream. Uh so you have uh you sort of seen live uh what a lot of people are tuning in around the world on stage. Lydia's talk yesterday. Uh a lot of you did not get to see for technical reasons. We launched last night and you can catch up on it.
08:57:48 300,000 people have. Crazy. Absolutely crazy. Uh uh does anyone want to shout out like favorite talks of the the whole two-day and two two and workshops too? Shout out your favorite workshops. Anyone? >> Jev. Jev. Yeah. Very inspiring. >> Potato. Yes. Lauren. Uh she she'll love that. What's that? >> Artcraft. Artcraft. Uh you gonna who's going to rewrite software?
08:58:12 Like who raise your hands. Who's going to rewrite and open source everything? Yeah, tons tons of you in the crowd. Um I I mean there's too many to name. I want to shout out Jared Sner as well. Uh Bun AOT and and a lot of inspiring uh DevTools work. Uh and just such a huge u outpouring of support from the community from DeepMind our presenting sponsor.
08:58:30 Um they launched hydrogen today. Did you guys enjoy that as well? Yeah. um as well as um yeah also open and thropic firecrawl um and all the other sponsors uh down um in the expo floor as well as the token billionaire lounge. Um so there's lots to celebrate um but there's one really particular milestone that I wanted to um uh to to celebrate here that we only ever pass once.
08:58:54 We will only celebrate in this room. Let's play the video. short and sweet. Welcome the AI engineer 2026 team celebrating 1 million subscribers. Um, so this is the team behind the scenes that's been producing uh our GM Leah, uh, Natalie and Ilana, Matt, Kyle, uh, Vincent, Sydney, Flo, Jesse, and a couple more be uh, behind the stage that, uh, we can't bring on stage right now because they have to wrap up the show.
08:59:43 Um, Leah, I don't know if you have a few words and we want to pres present some stuff to the team members. >> Absolutely. Um, so as uh, Swix is saying, we've grown dramatically in the last year. um along with our event series obviously our YouTube there's a couple of people here who have been absolutely pivotal to that over the last three years really since day one and we really want to recognize both of them today up on stage so if Flo and Vincent can step forward please >> and you're going to receive the plaques on
09:00:16 behalf of AIE all right >> and Vincent you get you get this little bonus as well you can open it later uh so coincident Inidentally, both of them are the uh the only two team members other than me that have spoken at AIE. You can check out their talks online. You know what channel. Um Vincent has been with us since day one. He's responsible for the design, the the look and feel, the brand.
09:00:36 Uh anything you enjoyed about Vincent, the thumbnails. Um if you've ever clicked on an AIE video, it's it's because of him. Uh the the glitch effects on the you enjoy that all those those videos, that's him uh running around at like 2 am in the morning changing the videos uh on the on the totems. Um yeah any any and you have your talk right that you did at Worlds.
09:00:57 Um cool. Thank you so much for for all the the help and support. Um and Flo uh my partner in crime on publishing the YouTube channel marketing uh if you've seen the tweets and the emails um and all the analytics behind the scenes that that's all flow as well. Uh as well as steady up our NAS which uh means make sure we never lose anything. Um so we're providing a lot of speaker services as well.
09:01:18 So every photo ever taken is is hosted because he's a hoarder. I'm a hoarder. to the chagrin of my my team. Um yeah, so this is the team. Uh we've brought uh we've really scaled aggressively. We're trying to build a consistent product. Um we started with one event a year last year with three events a year. This year is the fourth uh flagship uh event of the year to next year.
09:01:37 We're probably going to stay roughly consistent, maybe add a small uh fifth one. Um but we're also extending a lot of partnerships, right? So uh we just had AI Shanghai last week, the first AI in China. uh as well as uh we're just announced AIE India as well next next year um so there will be a lot more partnerships to grow um and a lot of you have asked us how to run an AIE in your city and uh well just apply online there's there's a partnerships form I think that's about it um thank you so much uh we will be around
09:02:07 and uh celebrating you I know you guys got to go to the afterparty you know where to go look it up but otherwise have a great time thank you so much for a wonderful AIE and a fantastic 2026 see you next Should we take a photo? Heat. Heat. That's hey.