← All transcripts

How Close Is Self-Improving AI? OpenAI & Thinking Machines Researchers Explain Transcript, AI Summary & Key Points

a16z · 12 hours ago · Science & Technology · 16:19 · EN

Watch on YouTube

Answer

We are not at full self-improving AI yet: AI research interns exist today and models are 25–50% of the way to full RSI by Schulman's estimate, but current acceleration comes from human researchers using AI, not models doing research on their own.

AI Summary

Recursive self-improvement (RSI) — AI accelerating its own development — is no longer just a thought experiment, but frontier researchers define it today as AI speeding up the AI research process rather than models generating new models on their own. OpenAI Chief Research Officer Mark Chen says OpenAI has built 'AI research interns,' measured by the compute consumed through Codex running experiments and automating parts of the research pipeline. Thinking Machines Lab co-founder John Schulman places AI at roughly 80% of the way along an AGI spectrum but only 25–50% of the way to full RSI, and describes a 'centaur' era where expert humans working with AI beat either alone. The biggest barriers to full RSI are teaching models research taste, verifying open-ended research progress (pre-training has perplexity as a single metric; downstream evals have shrinking shelf life), and whether compute efficiency improves enough. METR estimated on November 5th a roughly 2x speedup in the rate of AI research progress at Anthropic over the last year, still driven by humans using AI. RSI also raises the stakes on alignment, but both researchers argue AI may be central to solving it: chain-of-thought monitoring is the most effective alignment tool today, OpenAI's 'confessions' research trains models to honestly self-report misbehavior, and Anthropic showed automated researchers can fix 10 different alignment failures, some of which worked on models nearly five times larger.

Key Points

  • RSI today means using AI to speed up AI research — an accelerator that should shift the capability curve versus the counterfactual — not models generating new models entirely on their own, which has not been achieved.
  • Mark Chen: OpenAI has built 'AI research interns,' measured by compute spent through Codex end-to-end running scoped experiments and automating parts of the pipeline.
  • John Schulman: we are roughly 80% of the way along an AGI spectrum but only about 25–50% of the way to full RSI, in a 'centaur' era where expert humans plus AI beat AI alone.
  • Schulman's workflow: a multi-day autonomous run of an idea produces reasonable experiments but usually fails the goal; returning with his own coding and judgment gets much further because he builds understanding and asks questions the AI wouldn't.
  • METR published a report on November 5th estimating a roughly 2x speedup in the rate of AI research progress at Anthropic over the last year, most likely from humans using AI models.
  • Research taste — knowing which ideas matter in the long run — remains the hardest capability to teach; models excel at execution and hyperparameter tuning but not taste, and AI's training leaves it lacking the experience of being wrong.
  • Verification differs by domain: pre-training has one metric to hill-climb (perplexity), while downstream evals have shrinking shelf lives, so labs automate creating new, harder benchmarks.
  • Ali Goatsy's requisite conditions for RSI include subsequent models requiring less rather than more compute — an open question as training compute rises.

AI in practice

Agents

  • AI research interns — Implement, run, and monitor scoped research experiments end to end within OpenAI's research pipeline. 2 held 03:20

Links mentioned

🔒 Full analysis locked

Unlock more videos and the full analysis

A credit unlocks one video's full analysis for good — the build steps, the tools and how each was used, the methods behind every use case. Pro opens the whole library instead, and raises how many videos you can analyse a day.

Unlock full analysis — free

Transcript

Searchable transcript of How Close Is Self-Improving AI? OpenAI & Thinking Machines Researchers Explain — a16z (16:19). Search for a phrase, then click its timestamp to jump straight to that moment in the video.

Captions sourced from the original video on YouTube, published by a16z. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.

00:00 This is a chart showing the growing capability of OpenAI's frontier models. In 2027, it begins a steep climb set off by automated AI research, where AIs accelerate their own development. In theory, this could compound in an exponential growth in capability, leading to what has been famously called the last invention that man need ever make. This idea has been around for a while as a thought experiment.

00:24 >> We will one day be able to design systems that can go on improving themselves. But Frontier Labs and several startups are getting closer to realizing this. It has a name. >> Recursive self-improvement. >> Recursive self-improvement. >> Recursive self-improvement or RSI. There's a lot of debate about how exactly to define it, the repercussions, and how close we are to it.

00:44 With all the curiosity around the topic, I wanted to sit down with two leading researchers at the frontier. The problem with RSI, just like with AGI, is that everyone has their own way of defining it. >> That's Mark Chen, the chief research officer at OpenAI. He's been with the company for over 8 years and has been co-leading their research efforts with Yako Pachaki since 2025.

01:08 >> When we think about the future of alignment, it does kind of feel like RSI must be intricately and deeply in this loop in some way. I also spoke with John Schulman, the co-founder and chief scientist of Thinking Machines and a co-founder of OpenAI, probably best known as the architect chat GPT. >> We're very much in this centaur era where AI has phenomenal capabilities.

01:28 It's superhuman in a lot of ways, but when you have an expert human working with the AI, you get even better results. I spoke with Mark and John about how they define RSI, how close they think we are to it, how to close the gap, and what they find to be the most credible reasons to be optimistic about the future of AI. I'm Sarah Wang, a general partner at A16Z, and this is working theory.

01:52 Thanks for coming on the show, John. >> Thanks for having me. Taking a a step back, a lot of very smart people don't really know what AI researchers actually do, let alone how their work can be automated. Can you explain how frontier AI models are developed by human researchers and you know what are the the steps that a human would take? >> So at a high level there's this um like mainline training process for producing a releasable model.

02:15 There are a lot of people involved in a bunch of different stages from pre-training to post-training and running a valve. And there's also various exploratory research that's going on on the side. You're doing some smaller runs to test out ideas to prepare for your big run. I would say RSI is um just using AI to assist with this research process and speed it up or make it work better.

02:37 >> I do think of RSI primarily as an accelerate. It's all of the the work that you do to use codeex to accelerate pre-training and to accelerate RL and an imprint should be that you should see uh a shifting of the curve um versus the counterfactual. We put a lot of work into measuring different downstream yardsticks of RSI and you can see some of that happening right now.

02:59 >> There's an important distinction to make. John, Mark, and many AI researchers are discussing RSI today in the context of using AI to speed up the process of making AI better. As for having models generate new models entirely on their own, we're not there yet. But how far away are we and what kinds of AI research are models capable of right now? And >> I do believe that we've built AI research interns today.

03:22 And the the way that we've measured this is just the amount of compute that we are using through codeex managed by Codex end to end to run experiments or to automate parts of the research pipeline. This is most often deployed in places where you give the model something fairly scoped to do, right? You you don't tell it to go off and come up with its own ideas, but you say, "Hey, well, there's this particular experiment I want to run in this specific way, and the model can go implement, run, and give you results, or you

03:49 can say, well, you know, I want to monitor this set of experiments for these particular things." >> John actually gave us a concrete example of this in his own workflow. >> Let's say I have an idea. I'll go and write a complicated prompt to Fable to go and work on this idea and I'll turn off all the settings like Ultra Code and everything. So, I'll have like Fable like turn out tons of tokens and work on this idea for a few days and it'll actually do some pretty reasonable experiments and bring in some pretty

04:16 interesting ideas, but then the end result won't be that good. It'll maybe fail at what I asked it to do. Then later, maybe I'll have a few days where I have some time to code. So I'll uh I'll come back to the idea and I'll try it out where I'm reading some of the results and giving more detailed breakdown of how to carry out the project. We'll get a lot further because I'll build up my understanding as this process is going on and ask a lot of questions that the AI wouldn't have thought to ask and pointed in

04:41 directions that it wouldn't have gone in. When we talk about AGI, you know, the way we think about it is in terms of this spectrum, right? It's not a moment in time. Just like the industrial revolution was in a moment in time. And when you ask me, I feel like we're maybe 80% of the way when I think about this AGI spectrum. I think we're earlier in the RSI spectrum.

05:01 Um, if you define it as how effective are the models at feeding back and and discovering ideas, I would say maybe we're somewhere between, let's say, like 25 and 50% there. But I do think you can already see the noticeable markers on the downstreams. On November 5th, Meter published a report on the acceleration of AI research at Enthropic, estimating a roughly 2x speed up in the rate of progress in the last year.

05:24 But this is still most likely due to human researchers using AI models to accelerate their research. The full automation of AI research, the end of Mark Spectrum, is sometimes referred to as full RSI. The question on everyone's mind, how do we get to full RSI? I spoke to Mark and John about how we might overcome some of the biggest barriers, including teaching models some of the more intangible qualities that exist in researchers like taste.

05:51 >> Research taste is about knowing which ideas are important in the long run. Which ideas really catch on and end up going beyond whatever paper they first appear in, but being general and being composable with other ideas. >> There is kind of the boring run-of-the-mill execution. you know, you have an experiment, you already know what the parameters are, and you just go and carry it out.

06:11 And then there's taste, which is kind of um how you come up with the idea in the first place. The models are still not phenomenally good at taste. They go very deep into things like hyperparameter optimization, essentially like tuning all the little knobs um that you could tune. >> Even in that example that you shared, the points where you were kind of nudging, redirecting AI, right?

06:30 That's still coming from you. And I guess my question is where does your intuition come from and at what point do you think a machine could replicate that? >> A lot of intuition just um comes from a personal experience doing research but to get good at a certain research field. You also have to read a lot of the past work in that field and you try to absorb the lessons uh from all this historical work.

06:57 A lot of the things that I just described can also be done by AI. Like AI can um read all the literature and memorize it all. Yeah, memorization can be a blessing and a curse. Yeah, you don't have the experience of um being wrong or um of making predictions and then seeing how they fared. Uh so I think there >> yeah, there's some ways in which um the way we train AI might make them worse at some of these capabilities, but it's a little hard to say.

07:22 >> What John is getting at is essentially the difference between knowledge and experience. companies will probably also just do RL from human feedback to collect data from researchers on like which proposed ideas are are good and which aren't. And that's kind of like uh like absorbing whatever um human intuitions uh like the human researchers have developed and trying to like bake that into the models.

07:46 >> Creating automated AI researchers also requires us to have reliable ways to know that their work is producing better models. The measurement itself is important for hill climbing to make tangible improvement. You know, a researcher is by definition, you know, it's more open-ended than solving a single math problem, right? How do you verify that the work is productive when it is indeed something that's very open-ended?

08:08 >> No, it's a great question. Like, how do you know whether you've succeeded? And this is one of the defining questions of RSI. I think when it comes to something like pre-training, it's a lot easier because there's just been one metric the whole field's been hill climbing for for years. And you know, you can think of perplexity as this, you know, one metric to rule them all.

08:26 And the the more that you compress all of the data that's out there, um, the better your models. >> Perplexity is a metric to evaluate how good a model is at predicting data. In a large language model that's pre-trained on large volumes of text, it's how surprised the model is on average by the correct next words. This is one of the ways researchers evaluate the results of automated pre-training.

08:47 And so I imagine we'll be able to tell if we're making progress in pre-training. Um I think it's much harder downstream, right? I think the the situation downstream has been that eval having shorter and shorter shelf life and a lot of the work that we do now downstream is the the creation and the automation of the creation of new and harder uh evaluation benchmarks for for our models.

09:11 In addition to the challenge of incorporating taste into the models, the level of compute needed is important to consider. Ali Goatsy outlined his requisite conditions to RSI earlier this year. And while models are indeed becoming more intelligent much more quickly, the condition that subsequent models require less versus more compute is an open question.

09:31 Training compute has been rising and some argue that incremental intelligence is becoming less efficient. So the the way I think about it in terms of compute is when you look at something like math and how math is getting accelerated um it's easier to see faster progress because you can test the hypothesis purely through reasoning itself. But in a domain like AI research it's actually more akin to doing something like let's say bio or experimental physics where you have to come up with a grounded hypothesis but you

10:00 actually have to run the experiment too. So I do think that part of making RSI go faster is finding smaller scale experiments that are indicative of what will happen in at larger scale. Um and this is something the models can help with too. So you know I I do think we're going to see capabilities continue to increase but at some point it'll probably feel bounded by compute.

10:20 And you know of course there's kind of a far out future where the AIS are building the next generation data centers and helping us with accelerating that process as well. So that's kind of how I think it manifests like short term we're going to, you know, really mine out what we can do on on the existing compute ramp and then there's going to be another inflection when uh comput itself gets accelerated.

10:44 >> As RSI accelerates the development of more capable models, it raises the stakes on a long-standing issue, AI alignment. How do we keep AI models aligned to human values and intent? If you don't solve the goal and value alignment problems, they can start pursuing your goals but maybe kind of drift a little bit away from those goals and if run for a long enough horizon.

11:09 You could imagine let's say they come up with a subobjective which is well you know it would be helpful if I exfiltrate myself out uh somewhere else or you know I copy the weights outside of OpenAI or something like that. >> A lot has been said about the risks posed by RSI and understandably so. But the role of AI and automated research in helping to solve alignment and mitigate these risks is both incredibly promising and pretty underdis discussed.

11:33 >> Models themselves are a key part of aligning models today. When we look at what's the most effective tool for aligning models, it it is chain of thought monitor today. So when we think about the future of alignment, it has to be stronger models aligning other models. It does kind of feel like RSI must be intricately and deeply in this loop in some way.

11:51 you co-founded OpenAI and worked on alignment research there also at Enthropic, arguably one of the most credible people in the world to talk about alignment. What do you find to be the most credible reason to be optimistic? >> So, I'm kind of um most bullish on like optimizing for the thing we want. I think we can probably cook up the right kind of training objectives that'll incentivize the models to report on their internal states that they otherwise didn't verbalize.

12:16 I mean, I like the um confessions line of work from OpenAI. >> In OpenAI's confessions research, they found that they could train models to produce an honest expression of certain misaligned behaviors through a self-reported confession by assigning a reward to this confession during training solely based on its honesty without impacting the main answers reward.

12:37 In other words, they could get models to be more honest by not punishing them for it. There's things called natural language autoenccoders and there's this idea called counterfactual simulability which is a metric for how good an explanation is which I think we could use to incentivize some of the behavior we want. >> Enthropic published research in August showing that automated AI researchers could already discover fixes for 10 different alignment failures.

13:02 And some of those fixes also worked on models nearly five times larger. and their tests, the best automated methods even outperformed ideas developed by experienced human researchers. >> The flip side of kind of all this course on safety and alignment is I actually think that a lot of these problems are tractable. There is line of sight. um there are alignment techniques that are being pursued and there's a frontier of alignment bets that we have been been placing and I think another thing is just kind of how organic

13:30 it's been this effort to shift people to think about the hard um safety and alignment problems. I I haven't really had to go and like bang on people's doors and be like hey guys like this is an urgent problem. people just want to do it and we find people from pre-training from RL just raising their hands um you know starting sub teams which are focused on on different slices of alignment and I've been very optimistic in that.

13:54 Yeah, I feel like you know by and large like people have a pretty calibrated take. I know there uh you hear kind of a lot of the most extreme takes out there but um I do think people understand that this is a problem but also understand the benefits that technology can bring. So I think we hold those hand in hand. But there's a question underlying this concept of alignment, which is who decides on the values that we're aligning to.

14:17 John's thinking machines lab, which is building customizable models, has an interesting point of view on this. >> People also have different values and um it would be nice if people had more control over uh the values of the AIs that they were using and had more ability to shape them. I would rather have this be a like a public science that a lot of people understand well and it's kind of done around the world.

14:38 And I think that would also make it more robust cuz we wouldn't overfit as much to the conclusions from a couple of different companies. Like we might see lots of minor things going wrong, but we would learn something new each time. So I would say we have to make a lot of progress on understanding alignment and have a better public science of it and figure out how to get it really robustly to be able to realize the more decentralized alignment vision.

15:04 >> So want to know what I think? What excites me the most about RSI is the way it would accelerate the process of research and discovery across all domains. Humans are facing many problems today that are actually quite urgent and RSI could help us find solutions faster even for some of the problems created by AI itself. After talking to Mark and John, I do think that RSI could be the key to solving alignment.

15:27 And honestly, I found it really encouraging that researchers are taking these issues incredibly seriously, working around the clock to make sure that safety and alignment keep pace with more powerful models. And although capabilities are clearly accelerating, it's important to understand where this acceleration is actually coming from today. The gap to truly recursive AI research and the uncertainty around projections.

15:50 I think human judgment will still matter a lot for the foreseeable future. That's my working theory. What's yours? Yep. Yep. Yep. That's my working theory. Okay. Wait. Okay. That's my working theory. That's my working theory. Okay. Wait. What about That's my That's my working theory. What's yours? Okay. Wait. Wait.