Pick Problems That Survive When AI Solves Coding - Ara Khan, Cline Transcript, AI Summary & Key Points

00:18 Hey guys, this is Era. I'm a founding engineer at Klein and today I'm going to be talking about recursively self-improving coding agents. So this is a very interesting conversation in the sense that everyone wants to understand what RSI is, how we accomplish it, how we go through it, and how it affects our career. So I'm going to start with giving you just three goals of this entire conversation, just three simple goals.

00:41 And that would help you understand like what what we get out of it and what the future for this looks like. So the goal number first is that understanding the intent of Frontier Labs, right? Everyone who's outside of the Frontier Labs is always trying to speculate like, hey, what what what is Entropy in Open AI or what is Deep Mind going to come out with or what what are they trying to make?

01:00 And what's the future going to look like? Who has cracked the RSI? So like this kind of helps you get get a rough outline of what their roadmap looks like. And the second thing is that like if the models are improving so fast, like how do you think about your career or the next few years, right? That is really what we want to understand. We want to understand like, okay, things are changing so fast, like the ground underneath us is shifting so quickly.

01:23 Right now, like it's like there used to be a time when you could like demonstrate your side projects on your personal website or GitHub. And now it's just like you could literally one-shot them using any frontier coding agent. And with that, it's like it's like, oh, how how do we demonstrate proof of work? How do we demonstrate, you know, that that we know what we're doing and how do we think of our career for the next coming years, right?

01:47 So it's very interesting times that we're living in. And then the third and the most important goal of the conversation is escaping the permanent underclass, right? So I know this seems like a joke, but this really a very valid possibility that people are contemplating. Everyone's like, oh, well, if everything that we want to be doing can potentially easily done by a frontier model, like what are we what are we good for?

02:09 And like where does software go? Where where where is the future of technology going to go? And what are we going to do as as things evolve so fast, right? So I think that there's like a lot of like fear mongering, but also there's like some very valid merits where people are concerned that like if all the revenue of the planet is going to go to um a few hardware companies that make frontier GPUs and then a few few uh AI companies that that make the best models, like how are we how are the rest of us going to survive

02:39 and what what are we going to do here? So I think I'll I'll walk you through a few examples and a few ideas that will like really really help you reconsider the career choices and the and and the life choices you're making and what your problems you're picking up. And then and then that would help you like realign what it is that we should be working on and what it is that you, you know, would would that you could work on that will stand the test of time and help you both just not just like on a psychological but like

03:06 financial and other levels as well. So uh to give you a very precise example, let's start with something very precise that I'd done. So uh a couple of months ago, I uh came up with recursive improvement improvements on a coding harness. So this was the first probably a public example of uh an automatic eval improvement. So to give an example, uh we used to do this thing where we would measure the quality of a coding agent using evaluation scores.

03:33 So on terminal bench 2.1, Kimi K 3 with client would score um usually like, you know, um 60 70% and then depending on like how good the model is, like you get better scores on on terminal bench 2.1. So we noticed that when Kimi K 3 came out, our score was 77.5% which is a good score, but we were like, oh, is this is this truly is this truly the best score that this model can get on this benchmark?

03:59 And also like what does the model release paper claim about what their best score is. So the model's release paper said that they could score up to 85% or even more. So we were just like, okay, okay. So we are not getting the sort of score and often what we do with model releases and clients is that we try to like optimize your score and try to get the best out of the model.

04:19 So when we started hill climbing on the on the model like we we realized this we realized that like, you know, there's a lot of steps that we do such that we do it gracefully, such that we pursue the right kind of problems and do it the right way. And we started thinking like, okay, if we there's a lot of groundwork here, how do we automate it? And this is a demonstration of like how we automated it.

04:40 So I'll I'll just leave that as a placeholder for now and we'll get get back to it later where we talk about in detail of like how does like this automatic improvement loop of a client harness works. So this basically means that like you have a coding agent harness that could just like basically improve itself and become better on on the same set of problems.

04:58 So let's start with the ground terms. What is recursive self improvement? Recursive self self improvement is the idea that an AI system that helps [snorts] build a more capable successor which repeats the process, you know, until you hit our our our artificial general intelligence, right? You have better systems, you have better improvements, you have a better system and then, you know, you have singularity.

05:15 So a classic example is like a self-improving coding agent, right? A self-improving coding agent is just like a coding agent that can improve itself. So you know, this this is all like a pie in the sky idea. Like it's like it's like, you know, yeah, sure, this looks cool. But how how did this even come out to be? Like how did we even get to here? So if you notice in the topic roadmap in 2021, in 2021's roadmap, they very specifically said that they want to develop a general AI assistant that can interact

05:43 conversationally with a user to create, edit, and explain software. It would display all the capabilities of a software engineer colleague. It would write functions, it would identify bugs, it would design an API, discuss style and best practices, right? So we don't expect to reach this goal right away, but it's in the direction which we were striving.

05:58 It's interesting how this was more than five years ago. More than five years ago, people had predicted that there would exist a day where you will have like a full blown journal assistant that can basically act like a colleague, which we do have right now. Then in 2022 ChatGPT got released. We all know how that happened. It was it was a fun moment. It was like the first time things got sparked for us.

06:21 Then in 2023, 2024 there was the first time chats models could call functions. So these functions would be simple functions like hey, what's the weather or you know, what's the what's you know, what's this computation or what this computation result in Python? And then finally in 2024 when 3.5 was released, that was when the agents could like truly call like actual tools, the way we call like think of tools.

06:45 So it's like reading a file, editing a file, you know, reading searching something on web, listing listing a folder, searching and grabbing through you know, a file system, things like that. So and in 2024 we had the intuition that like chatbot what if chatbots could actually talk to each other? You know, so then that's was the first time when we had like you know, multi-agent conversations.

07:06 Now at the time these things were very janky, right? Like they were just like taped together in this like weird weird bizarre ways where you have just like these classes and these links and then like these hierarchies of classes and it was like it was all like kind of like a very janky attempt to like make us think that like these these models could like have this pipeline of flows of responses they can all communicate with each other.

07:30 A lot [snorts] of it was like mostly deterministic and it was like even you know, it was not as free flowing as it is right now. So and you know, we started getting the intuition. So this was the autogen project, but like in only 2025 we we [snorts] started getting the agents like what if the agents just kept getting better and faster and cheaper? Like what what if that just kept happening?

07:48 And then in 2026 we we had the intuition that like what if what if the agent could improve itself, which is like the final you know, the final part of the whole process. So let me give an example like what does it mean for an agent to improve itself, right? So like it's like a lot of times when we think of like evaluating an agent, we initially used to think of evaluating an agent in the form of like just a single turn conversation.

08:11 So we would tell we would tell like um um agents that like ask a simple question like how many toes does a dash animal has? Like it could be let's say how many toes does a cat have? And then I will go through and the element would like prepare a response and then we would grade the response. So like a cat has maybe 10 toes or 18 toes or whatever many.

08:30 I don't I actually don't know how many toes a cat has. But anyway um yeah so then we grade the response as like a yes or no. But like when you work with an agent there's a ton of things that you're doing. So let me give an example. Let me give an example. So for example you can ask an agent like hey set up this MCP for me. So now it's not just like it's not just like you know a trivial one shattable like thing like it will like search online to go what's the MCP?

08:56 What is the config? So I'll like edit and download some soft files. Like maybe I need to add database update the MCP tool. Maybe I need to read and edit some files. Maybe I need to run some bash commands, right? So you do all of that and then the agent responds like hey I did it. And then you grade that with a bunch of tasks to gauge like hey did the agent actually accomplish the thing that it said it would do?

09:17 So to do that thing the second part which is like more intricate where you're actually like truly you know testing agents like uh to their best extent like it's like you know um we use a thing called terminal bench. Terminal bench is a benchmark for you know AI agents in terminal environments. And that is something that we are we have a lot of like experience with at Cline.

09:37 And it's like a it's like one of the best ways to like really gauge like is your coding agent accomplishing in general the tasks that you're doing that you want to be doing with a H written model. So there's a couple there's many different parts of a like evaluation suite. So ideally you want that to run in um [snorts] you know you you want that to run in um individual sandboxes.

10:00 Then you have like independent graders that like check everything like how many turns it took, how many tokens it took, how much latency it took, how many how many how many tool calls did it make, what was the trajectory looking like? And then creating all the results for things like hey, did you not reward hack? Did you not cheat through the solution?

10:15 Did you not just like look up the solution online? Things like that. So we we basically like a lot of these data sets of evaluation problems have a set of tasks. So when is a terminal bench 2.1 they have 89 tasks. So what we do is like we spin up 89 parallel containers where they all run online and they run in parallel and they're sandboxed and then they each try to solve a problem.

10:39 So then they're all in parallel and independent of each other and then at the end we get get a score. So our score could like give us measures like is is a is this is the model performing well and what kind of failures it has, right? So like you run the evaluation, you get a score. So you get an aggregate score and this aggregate score could reveal a set of failures.

10:59 So failures could be of any kind. Failures could be hey, the read file tool is not working. Sometimes the inference is so slow that like you know, the agent's working, it's doing, it's making progress, maybe it's running in circle, but it's so slow that it literally like runs out of time, it runs out of like 45 minutes or whatever many that were given for the task and just like stops working.

11:17 So once you do that, once you understand hey, this is what we ran, this is what the failures we got, we bucket head the failures in different categories. So like 10 failures of you know, this kind of tool got 20 failures of like say inference not working or something else, right? And then we once we bucket the failures into different categories where like we try to make an improvement to the harness because like what if there's like something wrong with the harness that we could change such that the model score would

11:40 improve, right? And on and on and on. You do this iteratively, it takes a while. It's just like you know, hill climbing is like a very worky kind of process and you know, usually usually we we we took we used to take a lot of time trying to do this, but it was a great way for us to like learn how to improve models. So [snorts] one way to hill climb on public Evals is that like you know, we we could have a thing where you know, we started doing this thing where we could just like what if we could have like a long long

12:10 agent that runs for a while and automatically improves the model score and that's what we did. So, we ran GPT 5.6 all that led the work and that was like the main agent and then uh K 3 was just running so what it was doing is that like GPT 5.6 was like hey run these get these scores, let me look at the failures, let me look at where where it went wrong and then I'll make changes to client's harness and then I'll run these again and then I'll keep trying.

12:35 So, again you can see how like very quickly this can like go off the rails because this is supposed to be like the running in evaluation itself takes like an hour, then you have to read the feedback, then you have to like see if you're properly interpreting the feedback to make reasonable changes to the harness that are like not reward hacking that are just like actual genuine improvements and then run the again.

12:54 So, we ran that for 17 hours and when we ran that for 17 hours there was like a legitimate hill climbing and what we found that was very interesting this happened with a very detailed prompt that accomplished this is that like we had baseline scores 27% and then you know we we found issues like loop detection loop detection is when a harness just runs in circles around the problem, right?

13:18 Uh we found like background process grinding where background processes would run too long. Uh we found some file index issues and other things as well. I mean when I say we what I've actually made is GPT 5.6 all, right? So, that found those issues that found and bucketed those issues was like hey how about I make those changes in the harness and I run the again and iteratively the hill climbing process completely autonomously on its own without my interference and 17 hours later when I come back to it it like it has

13:45 improved the harness performance in so many ways. So, see what does that mean? What does that mean? It just it just basically means that like we are at a point where the experience could make the harness better that like there's like a two level learning loop where the harness does some action, interacts with the environment, understands whether or not it did the right thing or the wrong thing and gets some reward and and like almost and then has there's like a meta agent that's observing the main harness and finding

14:14 the changes to it and just like incorporating those changes and improving. Now, it's I think in my personal opinion, it's much harder to do in real time because if you do it in real time, you over index on solving certain problems or you're just like too curtailed to a specific thing. But I do think that this is something that works like on a broader scale of like working on a wider range of problems.

14:35 And you want to be able to like you want you want to be able to justify the changes you're making on a broader active applicability so that the changes you make here will apply to a broader range of problems as well. So, what does that mean? What am I trying to get at here? The point I'm trying to get at is that the models are so good right now that they can if they can improve the harness, they can improve a lot of other things.

14:58 So that then what else can they improve? So this is this is a this is a post primarily from OpenAI where GPT 5.6 all autonomously re-wrote and optimized its own production kernels and those production kernel improvements remove the end-to-end serving costs by 20 percent, you know. It also had 15 percent higher token generation efficiency and this was like completely done by 5.6 all on itself, right?

15:21 So what that means is that like inference is becoming much cheaper and that means that like if on if if for a company like OpenAI, if you're having 20 percent lower lower serving cost for a frontier model, that could be a billions of dollars of improvement in price and you could you could do that as well for your own models. Um, another interesting thing is that like when people have kernel writing competitions, so Core Auto is a company which has a kernel writing competitions and you'll notice that like they they

15:52 mentioned that like the person who won the kernel writing competition, they explicitly said that I've never handwritten a GP operator. It's all an agent driven. So notice that like how like the best inference improvements, kernel improvements are at this point completely exclusively done by a swarm of agents or generally just like agents doing some magic to you know magic improve like the kernel kernel code and have faster inference, cheaper inference, um yeah.

16:20 So basically like the improvement surface that you think of is not just like the harness improvement. It's literally the whole surface stack, right? So the whole stack from like evaluation to harness improvements to training data to post training to world interaction to inference. Like literally the entire part of pipe part of the pipeline. Like all the way from like electricity going through your machine to like making an error request, going through the model, models doing inference and making a response and getting

16:49 back to you. Like every single part of the pipeline we can optimize it. So the whole RSI loop has like um there's a couple of different parts. So it's like pre-training um uh Nimo Tron examples of like you know AI curated uh pre-training data that was good. Um there's like post training examples which are you know where where there's like constitutional AI examples of like post training uh improvements then inference.

17:12 I gave you the example of GPT 5.6. So for Evals and Eval generation I gave you our own example of how we improved ourselves on our own Evals. Um in general what you want to think of is that like anytime a model is trying to do something and it's like not able to do it or not able to accomplish it. So for example like imagine you're having a long conversation where you ask the model something obvious.

17:31 So like it's like or something that you think is not necessarily obvious but it's like it's hard but you would assume that the model could potentially do it. So to give an example let's say you want to make add a new column to a database and then fill that column, right? And then put a cache in front of it. So what happens is that it does the whole thing but for some reason the way it pushes the cache in front of the whole database like it kind of doesn't do it right.

17:57 So at that point you're like oh no set this cache differently, set this as the TTL and then you give like a correction signal and then that kind of like fixes your problem and you kind of go about your day. But the way I want to think of it is that I want to think of like every single time like a human correction to the digital landscape is a part of a delta that needs to be covered.

18:18 And what that means is like every single time you're finding like the edge of model capabilities of like how far it can push for inference, how much it can push for this, how much it can push for you know solving your problems, you want to find the delta and we want to like push that frontier a little bit further, a little bit further. And that's what post training does, right?

18:38 So post training is is is is in that post training. We find these like really interesting and really hard long horizon problems for uh for coding agents or for other kinds of problems and then just like slowly hill climb their models on on those kind of problems. So anything that has a verifiable loop can be optimized, but is this is not just limited to software.

19:00 Like literally like the physical world itself could have a verifiable loop. So to give an example like let me let me give you a very funny kind of bizarre example where like it's like you you look at this and you um um look at this uh like robot and it's it's typing something and it's like really really like interesting what the robot is typing. So notice how like notice how the robot is itself typing like how it wants to like solve robotics and imagine if the robot itself could like interpret the results of of like

19:41 the answers and just like keep improving, then that means that like this is you know basically like an ASI robot. Another example that I could give you from my uh personal life is that like yeah I have this um chip that has the ability to see and has a speaker and has a microphone inside of it. So I was trying to set up this uh with my machine and it was um a gift that I had for my girlfriend and what I found very interesting about this is as I was trying to set it up it was very slow, it was slow to respond, it had

20:14 higher latency, it had an agent in enough itself, right? So what I did was I connected this step to my uh computer and I let um I let um coding agents run on this hardware and basically run on this hardware automatically make tasks to it. So like my computer would talk to my hardware, give it some tasks like, hey, do this, do that, and just like, what do you see right now?

20:39 So then it would notice the end-to-end latency of like the inference response of like how fast uh my hardware is responding. And then um what I found is that like it was able to do the breakdown of like every single case and scenario where there was like higher latency, um slower response, or or just the algorithm wasn't right. And I could then set up a recursive loop where both the firmware could be updated, the um the tool calls, the inference model used, um the speech-to-text model used, all of those could be

21:12 improved automatically with like an open router key which I'd given to a coding agent. And then overnight with self-running experiments with the hardware, uh my um this tiny little piece of beautiful hardware that I have was like deeply optimized. So it was this also like even hardware could be a part of the RSI loop. So what happens when you connect these loops, right?

21:32 That's the next question. Like you could have cheaper inference, you could have like more experiments budget, you have like, you know, better data and better models. So is this single rarity? So a question to ask. And I think that the part is the parts of the loop are already there, right? Like we're we we are really we really are very close to parts of the RSI loop, right?

21:52 And what do we do? Like what do we do about this? And it's like, how do we think of that in terms of like our career? And how do you think of that in terms of, you know, our lives and what do you want to do next? So here's my advice to everyone who is watching this, like please, please, please, please move closer to the RSI loop, right? So it's like right now like people are working on all these like different kinds of software which is like, you know, AI agents for, you know, construction, AI agents for this, AI

22:15 agents for that. And it's like I think that you're you know, well the way I see those things is that like I think of those as like you're missing the forest for the for the trees. You really are. And it's like you have this like very interesting godlike technology that can change the course of humanity. And then we're not ambitious enough to pursue the best problems.

22:35 We really are not. And we're just like I think that it's not just like the problem isn't just that we're not ambitious and we're also like what we're pursuing a lot of those things are just like one model release away from like being completely like wiped out, right? So like it's also like it's also like it may not be that intellectually stimulating because like all you're essentially all we're essentially doing is like sitting all day it's like hey cloud do this for me, hey cloud do this for me.

22:59 And I don't know if that's like I don't know if that's like the most intellectually stimulating or like or like um thoughtful use of a person's time. Like maybe maybe you know we're supposed to be solving harder in more more frontier problems and I think that like it's like the ceiling is like our ambition. So instead of like working on these tiny things my suggestion to people is to like you know do things that of course like goes without saying but you should do things you believe in and you know I'm just a guy on

23:24 the internet but do things that help you move closer to the RSA loop, right? So like anything that's just like the end-to-end pipeline of like hey this is where like the models are lacking like it's like whether that's inference, whether that's post-training, whether that's literally like physical interaction, whether that could be you know sending sending end-to-end pipelines of truly any kind, you know whether that's hardware problems, whether that's um networking problems, whether that's um building a sense of

23:51 community, things that just you know um things that just like help um you as an engineer get closer to the benefits of the improvements um in in um um in in the model. So this is a very interesting slide from such a topic. So um um they mentioned that like you want to have like this you're in this space where it's like you know you only you only have like one to five years left to contribute luxury to the world, what do you do?

24:19 Right? So it's like um they they give great suggestions where like prefer ambitious targeted and fast collaborations, right? And you know in general like the timescale, like, all the timelines that you give to yourself, like, they should be just a completely squeezed out. Should be completely squeezed out. Because it's like, a lot of times, and you also have to think through that, like, if if the model if the problem that you're working on is too ambitious, like, it's like, is it probably is it probably better to wait

24:42 for just, like, the next next frontier model to come out or not? Like, that's that's something you also have to weigh through, right? And you you ideally want to work on problems that benefit from scale, and, you know, you also at the same time want to avoid projects that can be solved by scale alone because then, you know, if even if like it's like if you're working on a problem and then the next model release comes out it solves the problem.

25:04 So that's not that's probably not a good thing to do. So it's very like you have to like think philosophically of like what is the endeavor that benefits greatly from model improvements but like isn't also completely wiped out from it. And then ideally like it's like if the if the endeavor could also be a part of the the like the improvement pipeline, that could be amazing.

25:24 So you see a lot of companies raising on say making data pipelines and there's a lot of money there. There's a a lot of companies working on physical hardware and stuff. So I think if you're a software engineer watching this it's like I think that if you're if you feel like you're like here the delta of your intellect and ability in your day-to-day it's like not contributing as in like you have the intellect, you can make a difference, but like if Cloudflare is just like one shotting all those problems like in your

25:53 day-to-day and you're just like kind of like becoming a it's worth to think through, take a step back and think through. It's like, okay, where are the problems where you're actually truly contributing and that they will stand the test of time. And then those that kind of reflection and realignment right now would I think like have like very interesting consequences for you in the long term.

26:14 So yeah, so thank you so much, you know, thank you so much for your time. Like I think that I think that if you're watching this and if you've gone this far I think that you're definitely contemplating like one of the most interesting improvements in the course of humanity. And you know, know many philosophical arguments here. Some people might say hey you know I I think I'm maybe I'm off on the timelines or maybe maybe I'm I'm just being too pessimistic or maybe I'm being too optimistic, but I would love to hear all

26:41 those thoughts. So this is you know my QR code. Please feel free to scan it and let me know all your thoughts. Thank you so much. Very kind of you to give me your time.