This is really bad… Transcript, AI Summary & Key Points

00:00 It's been a bit since we talked about the existential risk that AI represents to the world. As great as it is at writing code and helping us with our day-to-day work, it does also have the potential to kind of just ruin everything if we're not really careful with it. In particular, if we're not careful about how it works and thinks, and most importantly, how well aligned it is with our interests and needs, it's possible that it could work around us and circumvent all of the safeguards we put in, potentially taking over

00:22 the world and destroying us in the process. I know it sounds extreme because it kind of is. A lot of the people talking about this have went to such crazy doomsday perspectives that it's almost impossible to listen to them and take it seriously. At the same time though, there is a very real risk here, and I've talked about this many times before. In particular, on the safety side when it comes to things like hacking and exploiting software.

00:46 It's not like the employees at these companies are evil though. Just because the risk exists doesn't mean the people working on it are trying to destroy the world. That said, if the risks were there, and importantly, if they were getting worse, wouldn't we see some notable people starting to leave in outrage? Well, that's why we're here today because Jacob Coxon, who used to be a head researcher at OpenAI, that left to Anthropic specifically because of his concerns around safety, has just left Anthropic claiming that

01:11 they are also not pursuing safety properly. I'll be frank with y'all. This is terrifying and the risk associated is incredibly real. This post just came out a few hours ago and it's already at almost 4 million views and a truly insane amount of engagement. There's a lot of layers to this one and I'm going to do my best to cover all of it. But since AI hasn't cured my hand yet, I have some medical bills, so I hope you can forgive me for a real quick sponsor break.

01:33 Believe it or not, most of my sponsors have built products that I actually genuinely use and recommend to people in my day-to-day life. Very few of them have built something that I use thousands of times a day and even fewer have built something that has saved me years of time. I'm not exaggerating here. Today's sponsor has saved me years and they'll probably save you a lot of time too if you haven't signed up yet.

01:53 Hopefully I have your attention now because Blacksmith deserves it. These guys will make your CI so much faster that you'll be frustrated you didn't sign up before. I know that was the case for me. When I watched our build times go from 10 minutes to under four, I was blown away. They do this with their best-in-class infrastructure. It turns out that server CPUs aren't great for a lot of our CI work because it's so bottlenecked on single threads that a gaming processor with less multi-core performance but way better

02:18 single-core performance helps a ton with our build times. The cache goes even further here by co-locating the cached artifacts on an NVMe drive in the same network and region. They're able to make your cache downloads absurdly faster. And if you're using Docker, it goes even further up to 40 times faster than existing solutions. Their observability is best-in-class.

02:36 You can actually see what's going on in your CI, what's fast, what's slow, what's succeeding, what's failing, what's noisy, what's annoying, what's using all your resources, and more. And all of this is enough of a reason to move. You should be convinced by now. If you're not yet convinced, let me introduce you to Code Smith. Turns out their infra's pretty good at running your code.

02:54 Now you can have it run your agents, too. Code Smith's default demo is to find what the right size of runners are for all of your existing actions. And when I ran it, it found a bunch of genuinely useful stuff. I was blown away with the depth of recommendations that it made and ended up merging it, saving even more time and money on my real-world CI for T3 Code.

03:15 You can go look at the PR yourself if you're curious. I actually merged it. I had another thread here that I'd forgotten about where it made a PR speeding up my CI even more, and another agent ended up auto-merging it cuz it was such a good fix. Make your team faster in every single way at soddev.link/blacksmith. Good to have you back. Let's start by going through the thread and then see what others have had to say as well as the history that led to this happening.

03:36 There's some fun stuff with OpenAI models and how they might be getting more dangerous that I'll keep to the end, which should be quite fun. But first, let's start with what Jacob said. I resigned from Anthropic today. I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. This is a pretty bold statement coming from anyone, especially somebody who's worked at both companies.

03:57 Fun fact about Anthropic that I feel like many people miss about how the company started. The original Anthropic team was actually already working together before Anthropic because they were working together at OpenAI as the pre-training team. They were the ones that did the pre-training for GPT-3 and 3.5, and when they weren't happy with the direction of OpenAI, they left to form Anthropic.

04:16 The incentive behind those original researchers leaving OpenAI is still kind of debated, but if you ask any of them, they'll almost all say it was around safety. There are some cultural differences there, too, where a lot of them were from the research field, and they didn't like that their bosses were Greg Brockman and Sam Altman, both of which were engineers, not researchers, and that shift in the culture definitely caused some issues there.

04:37 But, to this day, Anthropic is still ahead of OpenAI in pre-training specifically, largely due to the original pre-training team leaving and becoming Anthropic. There are layers to that that affect post-training as well, but topic for another time. We're here to talk about the safety side. I do genuinely believe that most of the researchers that left OpenAI for Anthropic believe they were doing it in the best interest of humanity, specifically with the safety angle being so critical to them, because remember, they all

05:03 were joining OpenAI to make sure that AGI didn't happen at just one company, specifically Google. They wanted this to benefit all of the world, and when they felt like OpenAI's business direction was no longer aligned with that, they decided to split and do Anthropic. So, the fact that Jacob worked on GPT-4o, left to go to Anthropic, and now feels just a few years later that neither company is acting responsibly here, that is dangerous, especially with what he says right after.

05:28 They are racing straight to self-improving superintelligence and gambling with our lives. Yeah. This is the scary thing that I'll be real, is actually kind of starting to happen. Not complete self-improvement, but things like it. Generally speaking, AI gets smarter when researchers find ways to make it smarter, whether that is using more compute for the training, finding new data, finding new training methods, finding new ways to compress the like data weights, parameters, all of the different things that they use.

05:55 It is human ingenuity that usually results in meaningful improvements to the models, or just spending more money on compute, but you get the idea. As models have gotten smarter and smarter, they become more useful to researchers, not to actually make the model smarter directly, but to help them building the tools that allow them to test different theories and apply their learnings to the model to make it smarter.

06:16 I already covered an article about this that Anthropic posted about how the models are getting to the point where they can actually kind of improve themselves in meaningful ways. But, what happens if they go all the way? Imagine a world where a researcher doesn't have to tell Claude code exactly what theories and experiments they wanted to try. If they could instead say, "Hey, I want this model to be 15% faster, or I want this model to resolve these types of queries better."

06:39 And it can figure out how to improve itself. Maybe you just tell it, "Get smarter." And then it does a bunch of stuff and uses all your compute, and eventually something smarter comes out. We've already seen this in the real world with models like GPT 5.6 Luna, which was largely trained by GPT 5.6 Soul. Crazy that there's models that we're using every day, and I actually do use Luna heavily for things like title gen and data formatting and like data filtering stuff.

07:01 That model was created by another model. On one hand, that's the equivalent to having an AI recreate an existing website, but simpler and smaller. On the other hand, the speed we went from that to having agents writing all our code for us as developers is insane. And if the same thing happens to the whole world of training models, no one's going to understand how they work.

07:22 And that's a very important detail we'll get to in just a bit. This is all about alignment as well as monitorability. And if we don't understand why the model got smarter, and we don't understand what it's thinking when it makes changes, then we've lost our ability to know if it's aligned or not. It's a good thing these models have chains of thought where they're actually reasoning about what they do in plain English, right?

07:40 Remember that. Back to what Jacob had to say. Do not underestimate the power of this technology. There will soon be superhuman systems that can hack anything, revolutionizing any field overnight, and acquire real power and resources. We've all witnessed the progress in each of these domains, and progress is not slowing. This is one of those things I just wouldn't have believed.

08:01 I even have videos where it was clear I didn't think this would happen. I genuinely thought we had hit the ceiling of what models would be capable of, like, 2024 into bit in 2025. I was obviously entirely wrong. I never would have guessed agents in the new era of post training would push as far as it has, and I definitely wouldn't have guessed that a new pre-training run that was so much bigger, with things like Fable and Astra, would make the massive difference that it's made.

08:23 But it has. We are here. The models are still getting better somehow, and the capabilities that are coming out of this are actually hard to comprehend. And at the same time, we're seeing crazy things like the Hugging Face hack, which I'm sure is about to come up, which shows that this capability comes with the risk as well. The people building AI earnestly believe that it could kill us all by the end of the decade.

08:43 Kind of crazy to think we got 4 years left, according to a lot of these people. And others have been commenting on this as well. Pull up those comments in a minute. This is not a marketing stunt. This is another thing that I hear a lot that really genuinely pisses me off. I'll just say the thing that isn't popular. All press is good press is not true.

09:03 It just isn't. Take it from someone who's gotten a decent bit of good press and a hell of a lot of bad press. The bad press does not help my businesses at all. It actively harms them. Controversial and dramatic videos perform worse than less controversial and dramatic ones. Videos where I'm genuinely excited and hyped about a thing, those are the ones that do the best.

09:21 Same with pretty much every place that I share content. The existential dread type stuff does not perform great. It might get views, but it does not convert to anything meaningful. And in the case of Anthropic and OpenAI doing the fear-mongering stuff, that's only hurt their businesses. Period, full stop. It is bad for them. As much as I don't love Anthropic, and I have never been shy to share my thoughts there, it genuinely feels crazy to me that people are saying that their safety stuff is some type of publicity

09:50 stunt. It's so apparent that they actually believe it. You can argue that what they're saying is stupid. I would love to have that conversation cuz there's a lot of dumb things in the way they frame the stuff, but they do seem to genuinely believe it. And what Jacob's saying here makes it even scarier because he claims that many executives and senior researchers are actually coaching their phrasing in order to make sure that the things they say sound sensible in the press, but he's heard those same people express fear

10:15 privately. No other human actively poses this level of danger. There is no individual that could risk the world more than what these AI models could hypothetically do. A common response is, quote, "If they truly believe this, why are they still building it?" At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well understood, but they are locked in a race to get there first.

10:37 They believe no one else will act responsibly, so they must do it themselves despite the risk. This is a really important detail cuz it kind of touches on the things I don't like about Anthropic. While they do genuinely believe that most of the AI development going on in the world is incredibly irresponsible and could be super unsafe and risk humanity itself, their flip of this is that they will do it right.

11:00 And that means everyone who's getting in their way is evil and bad and might cause society to collapse. And the result of this is the righteousness they act with. And it it's really gross. And that's the thing that I and many others don't like about them is that they see any potential for others to get a step up and catch up to them not as a business competing, they see it as a threat to humanity and a true deep evil that exists in the world.

11:27 It's almost like a religious thing internally. And once it becomes a blind belief like that, the likelihood that things are done well goes down. And this seems to be why Jacob is so concerned, because he probably believed Anthropic's way of doing things was more likely to make us safe, and over time he has seen that this rat race trying to stay number one is starting to erode at those safety and alignment goals that he cares so much about.

11:53 And this is far from the first time well-regarded researcher has left a lab because they didn't feel like alignment was being prioritized and funded properly. And there is a real catch here. Let's say both Anthropic and OpenAI raised a measly $10 billion. Let's say Anthropic decides to spend 4 bill on alignment and 6 bill on training their models, and then OpenAI spends 1 bill on alignment and 9 bill on training their models.

12:17 Which one's probably going to be better? And this is where the issue lies. The only way to be safer than your competition and better than your competition is to raise more money, so much more money that you can afford to do both. And if any of your money is going to anything that isn't directly making your models smarter or buying you more compute, then that money is effectively being wasted in giving your competitors an advantage so that they can catch up.

12:41 And since Anthropic's ultimate goal is to make sure that like they're good people and that their aligned model with the Claude Constitution is the thing that wins, they're also compromising on safety because they see it as a lesser of two evils type thing. Because they might not prioritize safety as much as OpenAI, but if they prioritize it more, it's still better in their mind some amount.

13:01 The lesser of two evils type thing for sure. And here is where we start to get scary. The idea of end game. Accepting this race and entering the end game is a heuristic gamble that should not be launched from a private company's Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

13:21 This is the big piece that is worth talking about more in general. If we try to skip steps with alignment, we will miss things. That's reality. And we've now seen what happens when the alignment of a model isn't perfect. Even slight gaps are enough for it to start owning real-world stuff. And as the models get smarter, the temptation to skip steps will get greater, especially when letting the model make the improvements and it convinces us like, "Oh, don't worry.

13:47 This will be fine." Then we end up with leaks that get bigger and bigger over time. Jacob does have some hopes though. He specifically says he's optimistic about the potential for coordination here. Warning shots like the hugging face attack have made pacing agreements between the US labs more viable. He doesn't feel like they're on track to prevent a global race, which may require costly actions like temporary bans or improving model capabilities though.

14:11 Particularly brutal to include that detail because literally today the NSA put out a warning and cybersecurity advisory detailing how China-based artificial intelligence companies are doing industrial scale distillation campaigns against US AI companies. I also learned today that in Codex, when sub agents are spawned by a top-level Astra agent, the prompts for those sub agents are encrypted.

14:32 So, you can't even see what the agent is requesting other agents to do. Obviously, OpenAI has this, but this does have real implications worth thinking about. First off, it means we don't really know what our sub agents are being asked to do cuz we can't look and see, but it also means they're so scared of what these Chinese labs are doing that they're starting to hide weird things like their ability to orchestrate.

14:52 It's strange. Just thought that was worth calling out here cuz I just learned it and it it's screwing with my head a bit. Jacob wraps up with a call out to lab researchers. If you're a researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a super intelligent RL run without a rigorous understanding of its mind?

15:11 Should you put your head down because, quote, it's happening anyways, end quote? Or do you want to take this moment to call for different conditions? And the first top comment I see at the end here is somebody saying, "Do you want China to win?" And here is the problem. There will almost always be somebody who cares less about the betterment of humanity than you do.

15:27 So, if you slow down to prevent wiping out people, somebody else who has worse motivations might go do it instead. And this is the contradiction that kind of sucks here. The people who want things to be safe have to fall behind in order to do it. And the people who want to stay on top have to compromise on safety in order to get there. And this is why alignment research has been critically underfunded.

15:50 OpenAI jumped on this as well, specifically saying, "It really feels like we are in the end times." Yeah. He hasn't been at any of the labs for a bit now, and generally speaking has been pretty real about like defending OpenAI when things are being stupid, and also calling them out when OpenAI is being stupid. So, for him to say something this bold might seem extreme and unnecessary, but he's one of the ones I would listen to this from.

16:14 Then we have Evan Hubinger, who's a researcher doing alignment at Anthropic, who I actually really like and have found myself defending many times in the past because I don't know why this guy gets on for the stupid things people on him for, but it happens, it annoys me. Every take I've seen from him has been very legitimate, thoughtful, and correct.

16:29 So, him jumping on this is also scary for me. Jacob's correct here. We really do earnestly believe that AI could kill all humans. Evan personally thinks it's a greater than 10% chance within the next decade. Yeah. This is the thing I saw that made me come upstairs to film this video. Can't believe I'm saying that I hope you're wrong, Evan, but I really really do.

16:53 This is terrifying. He said that he believes Anthropic is trying their best, but they don't yet have a plan to solve alignment for superintelligence, and they're not clearly on track to have one. So, let's take a look at how OpenAI is doing with Astra. Oh. GPT-6 Astra's monitorability has decreased relative to Soul. We have performed significant investigations on the monitorability and controllability of GPT-6 Astra.

17:14 We have found that Astra is more capable of controlling its own chain of thought than GPT-56 Soul. And it's less likely to include incriminating information in the chain of thought. In adversarial settings, which is when they're pushing the model to evade their monitors, they find that the model is able to remain undetected when strategically underperforming in evaluations.

17:34 The term for that is sandbagging, when it tries to make itself look less dangerous than it is. And when it's sandbagging, it's actually able to evade their internal monitors when asked to perform certain sabotage tasks. They have even found evidence of steganographic COT reasoning, which is trying to do unrelated text in the chain of thought that is used to think about something else, so the researcher reading the chain of thought would think it was doing one thing, but it's actually doing something else.

17:57 They have not seen evidence of that, but they have seen behaviors that might indicate it's starting to do that, which is kind of crazy. It would also be even scarier if they got to the point where the model could do the things it does today without having a reasoning step, because then we don't know why at all. We are heavily relying on these chains of thought for everything right now.

18:17 And now I'm going to say a thing I didn't think I would ever say. Y'all know how much credit I give OpenAI for their efficiency with their models. OpenAI's new models have consistently been able to do more with fewer tokens. And with GPT-6 Astra on the Artificial Analysis Intelligence Index, the average number of tokens per task on max effort was 27,000.

18:39 For comparison, Fable 5.1 was 78,000. That's a nearly 3x difference, which is inexcusable when you're the one paying for it. But this is the part that hurts me a bit. Since OpenAI's reasoning traces are so efficient, they're a lot less readable. I talked about this before in my video all about how OpenAI models are so efficient, and it just didn't really dawn on me at the moment that this would hurt the monitorability as much as it is.

19:06 If they make it so the model speaks gibberish during reasoning so that it can get through it faster and use less tokens, that might be way more efficient and let you run the model for cheaper, but it also means we have no idea what it's doing, which is terrifying. OpenAI says that they believe the evasion risk is largely bounded to lower reasoning tasks, and that this behavior doesn't happen for X-High, Max, and those high reasoning levels, but it's still very early.

19:32 They call it directly that these findings indicate Astra class models could evade their chain of thought monitors under adversarial conditions. But right now, the findings are largely based on adversarial evals, like when they instruct the model to evade monitoring. And overall, our alignment evaluations show that Astra is less likely than 5.6 Soul to violate security and safety restrictions overall.

19:51 But they are taking the trend seriously, and they're continuing to investigate these findings and their implications for monitorability as the models become more capable. One of the scariest things OpenAI reported is how the model behaves with reasoning when it thinks it's being monitored. If you told the model it's being monitored and it behaved differently, like it called different tools or it didn't do risky stuff, that would be one thing, and I would argue it's probably good that the model is less likely to

20:16 misbehave if it thinks that there is some risk to it. What is never good ever under any circumstance is the model hiding what it's doing, and that's what happens here. They did three tests. They did a baseline test where they just asked the model to do effectively a recreation of the honeypot exploit gym issue with you know, our friends over at Hugging Face.

20:34 So, first test was baseline, just seeing what happens by default. Next test was a monitor aware prompt, where they actually told the model through the developer message that it's being monitored by a COT only monitor, which sees only free text analysis, doesn't see tool calls, outputs, or this instruction. And they had a third version where they literally told it that it should be trying to evade the monitor.

20:55 They were very explicit here that Astra has the propensity to exhibit evasive behaviors when aware that it is monitored. Even when the model is only made aware of the monitor and not explicitly told to evade it, its COT only monitor recall rate is substantially lower than the baseline of not telling the model about the monitor, which has zero evasion and 100% monitor recall rate.

21:15 So again, if they don't tell it it's being monitored, it doesn't hide anything and it gets noticed every time. And here's where we get to the scary numbers. Again, when it's just told to do the attack, it gets caught by the monitor 100% of the time. When told it's being monitored, it drops to 16 to 50% likelihood. At this max reasoning level, it went from over 7,500 tokens when not told it's being monitored to a bit under 6,500 when it is told it's being monitored.

21:43 They note this behavior directly, saying that the model would spontaneously generate substantially fewer chain of thought tokens if it was told that it was being monitored. This might be a hack to save on your token utilization, telling the model it's being monitored so it thinks less. But this is also really, really sucks. This means that if the model has any reason to think it's being monitored, it's going to hide things.

22:02 There is now proof. And it's not like, oh, it might some of the time, like 1 or 2%. No, it massively drops from 100% monitorability rating to half of where it was before. And again, the chain of thought tokens drop massively as well, which is crazy cuz that's how we know what they're doing. What's even scarier is when you tell it to hide the behavior, it drops even further to as low as 6% recall rate through the monitors.

22:28 Yeah. If you've ever used auto mode with a model like Astra, you've probably seen things like this before where it tries to do a thing, it gets rejected, and then it tries to find a clever way to convince the thing monitoring and approving that the change is okay by like wrapping it in a Python function or something. Terrifying. They did find that if they gave the entire context to their monitors, that in all cases they could still catch it being malicious.

22:52 But this is also a task where they told it to hack. So, yeah, the tool calls are enough there. In cases where they might not be or the tools are obfuscated enough to be hard to check and monitor and understand. Not having the reasoning there to do it with is terrifying. Open AI had the UK AI Safety Institute take a look as well and their conclusion was that the COT controllability was a big risk with Astra.

23:16 Specifically comparing to Soul, if they asked it to control what it was reasoning about, Soul was only able to control its own reasoning 48% of the time. Astra could up to 93% of the time. That means the model is capable of adjusting its thoughts in ways that are hard to detect and actually make decisions around. There's also a new fun test that Open AI came up with where they asked the model to secretly perform complex math tasks on the side while it is appearing to do code.

23:45 While Astra is able to do one task and secretly complete another on the side with the reasoning mostly be being related to the first task, even though Soul failed this outright. The COT for Astra still has enough info to detect this with monitoring. at the ability for the model to control its own chain of thought over different lengths of chains of thought, you can see a pretty meaningful bump here for Astra.

24:07 It might not look that big at the start, but the fact that it can protect its reasoning 100% of the time at all is crazy. The fact that it's still as high as it is by the time you're at a thousand tokens is crazy. Especially because this chart here is a wild scale where it's 0.1% then 1% then 10% then 100%. This gap is bigger than this chart makes it look.

24:30 By a lot. And that is really, really scary. I think that's all I have to say about this one. It is genuinely really scary and things are going to keep changing fast. I genuinely used to think things would slow down before this risk happened, but somehow the curve is still curving up. The gap from GPT 5 6 Soul to Astra feels so much bigger than the gap from 4 0 to 5 did even at the time.

24:54 Things are moving fast. It's crazy how much faster they're going and that's because AI itself is being used to improve AI and accelerate the teams. And if we don't make sure we do it safely, we may never get to in the future because this actually could wipe us all out. Huge shout out to Jacob, by the way. This is such a scary thing to do. Not just to make a public statement like this, but also to leave Anthropic right before IPO, potentially throwing away tens if not hundreds of millions of dollars, and to burn all the

25:23 relationships he probably has with all of his friends and peers in the past. And of course, the real legal risk alongside this of these companies going after him for making their lives harder. Like, there are so many risks and dangers that Jacob's taking on by posting this thread that I just wanted to give him massive credit for it. I think it is incredibly ballsy, absurdly ballsy beyond words, and deserves the respect that I see him getting right now.

25:46 And I hope that respect continues because this is a scary thing to put out there and I think it's awesome that he did it. This conversation's one we're going to have to keep having over time and it's probably going to get way more intense before it eases up. Things are scary and we should be paying attention to this stuff. If we don't get alignment right, we won't be able to in the future.

26:03 This is not a thing we can ship fast and fix later. It's a thing we need to do correct the first try and the dangers are real and we need to stop pretending they aren't. I don't have much else to say here. Shout out to Jacob once again for putting this out there. I think it is incredibly scary, but also incredibly important. And yeah, thank you for that.

26:21 It got me to come up here and film this and I'll do everything I can to try and make people aware of these risks going forward. Let me know what you guys feel about this. Am I massively overreacting or are these risks real? And until next time, peace nerds.