The AI Language We Can't Read: Neuralese ft. Rob Miles - Computerphile Transcript, AI Summary & Key Points

00:00 It's so it's been a while Rob, there's been a lot happening. >> Yeah, it's been a crazy year this month. >> I'll ask you a simple question before we get started on today's. Have you gone native yet? >> I don't know what that means. I mean I kind of already was, right? I've been an American citizen the whole time. >> Yeah, I didn't mean the American thing.

00:17 I meant the AI thing. >> [laughter] >> Oh, I don't think so. >> Let's just say it's been time to catch capture everything that's been going on in AI land. But there's something we we're going to talk about today which is new release and and the idea of is it recurrence? >> Right, right. The thing like new release is kind of this vague term. It's not super well defined.

00:39 People mean different things by it. But broadly speaking like opaque recurrence and chain of thought monitor ability which is like a thing that people have been yelling about recently with open AI's new model. Recently there was a bunch of drama about this because open AI came out with this new model Astra which the leak suggested was doing this opaque recurrence.

01:08 >> There are people who come to these videos who perhaps haven't seen you before, perhaps haven't seen even computer file before that when we say it's thinking this, it's thinking that, it's not that we're saying that the computer is thinking. It's more that we're talking about it in terms of it it's appearance of of what it's doing I suppose. >> I mean we can get into the anthropomorphism question.

01:28 I think it's a mistake to look at a machine and assume that it is doing the same thing that humans do. >> Mhm. >> Right? Anthropomorphism as in the shape of a person. Um but on the other hand, you know, you saw these robot Olympics. I think if you look at that robot and say oh it's running the 100 meters, there's a sense in which it isn't running like in the same way that humans are running, but also it's running like it's running.

01:54 Look, it has like >> It has what appears to be legs. Yeah, I get that. >> Right, right. These things, they're not legs in the human way, but they're like leg-shaped and they move in a leg-like way. And like if you think about this as running, it's not it doesn't really lead you very far wrong. If you think like, "Oh, you know, if it runs like that for that long, it's going to get cramp or something."

02:18 Something that depends on specific, you know, biology, then you'll be wrong. But in terms of like, "Oh, it might trip and fall. It might have to step over something. It might" like a whole bunch of the other things still carry over. And I feel like that a lot of the time with language models. It's like it's not thinking the way we do, but it doesn't actually, practically speaking, lead you very far astray to say that it's thinking.

02:41 Like with the small models, it does because they're a much worse approximation of of human thought. Um with the larger models, it's like a linguistic shorthand. But I also just don't um I think basically you have to you have to be able to point to a specific mistake. Like a specific thing where like if you were thinking about it as thinking, you would predict that it would do X and actually it does Y.

03:06 And if you want to point at that, I'm like, "Yes, definitely." But if you're like, "Oh, this the language you're using is bad." And no, you're not like wrong about any specific thing, just the language is bad. I'm like, like what do you It gets so clumsy to talk about these things any other way. And it's like not that bad of a mental model to use. >> So anyway, look, with that said, maybe we should talk about what we're actually talking about this video.

03:30 >> Right. So, chain of thought and chain of thought monitorability. Now, chain of thought is one of those things where I actually do think the language isn't great because it's not really the chain of of what the model is thinking, it's more like a scratch pad inside the model's head, as it were, in the activations of the forward pass, there's like thoughts happening, but then it's also producing language.

03:52 There isn't like a fundamental difference between the language being produced in the chain of thought and the language being produced in the output. Like the way this came about was you had them you had these models, you would ask them a question and they would give you an answer and the answer was often wrong. And then people came up with this thing of being like let's think step by step.

04:10 And this just encourages the model to rather than just immediately outputting the answer, output some stuff first with this kind of priming effect of like this should be thoughts that lead you towards the answer. And this resulted in better performance. And then later they made this like an official thing where you have you know, it would start it would like print a special tag or you'd have a special token that would be like this is the start of the thinking thing and then it would think for a while and then this is

04:39 the end of the thinking thing, but ultimately it's still just generating language. Um And this lets the model solve it let it be smarter, right? It lets it solve problems that it couldn't previously solve because it gives you more serial depth. Say I ask you a question that's like um how much did the star of the highest grossing movie earn for that movie?

05:06 Right? As a question. Maybe you just know this, right? Maybe you've memorized it, but if you haven't you have to think okay, what was the highest gross grossing movie and then who was the star of that movie and then what did that person earn for the movie? And if you haven't figured out what the movie is, you can't start working on the salary. You have to do these steps in order.

05:26 >> We learn at you know, primary school at elementary school that you need to break things down into small steps, right? >> Right, right. And you know, some sometimes this is like inherently the case. A lot of problems you can solve in multiple different ways, but like the effective way to solve this problem is to do it as a series of steps. And um And so if you give that type of uh task to a model and people people do this.

05:53 People have done this, you know, they've tested it. Um you can you can give them this type of multi-step reasoning task and then use interpretability tools to probe at different layers of the model and you you find that like the early layers are figuring out the movie and then the middle layers are figuring out the actor and the later layers are figuring out the salary, you know?

06:12 Like not that specific example I made up, but that general type of thing, right? It's broken up in the layers of the network as it passes through. >> The classic thing they were getting wrong all the time was simple maths problems, wasn't it? So, actually allowing them to break it down into the steps to answer the question, even though it's a ridiculous thing to use a massive large language model for a tool like adding, I don't know, 5 + 5.

06:32 It It was a way of saying, "Actually, we can solve this." >> Yeah, maths questions is a great um is a great example to use because it gives you this kind of extremely large set of possible like continuously variable questions that you can see, "Well, can it multiply two-digit numbers? Can it do three-digit numbers? Can it do four-digit numbers?" Whatever.

06:53 Yeah, so having more serial depth helps. The point is that there are some things where you just don't have enough layers in the network to do the task in one step. And so, being able to do part of the task, output a token, and then come around again with that token and do the next bit of the task and you can kind of you can pass information forward from one forward pass to the next, that information is in the tokens of the chain of thought.

07:19 There's kind of two reasons you might expect chain of thought to be useful. One is just like that in the base distribution, which is to say, you know, text on the internet people think things through and if you've like created a function that generates human-like text, then this like thinking things through behavior is going to be like fairly likely from a next token prediction perspective.

07:41 And so like if you say let's think step by step, you know, you you get this type of behavior and that's useful and it's cool that it like you should expect that to reflect the actual thinking that the model is doing. But then the other reason that you should expect the chain of thought to be meaningful is that a lot of the time you're asking the model to do something that it can't do without the chain of thought.

08:07 And so it obviously is using the chain of thought. Now that doesn't mean that it's a direct correspondence. The information passed through the chain of thought, right? And the obvious way for it to pass through is by having the chain of thought be English text that represents the thinking that it's doing. But it's not guaranteed to be that, right? >> We did a recent video with uh with Mike about how tokens work and this idea of feeding things back through every time you add something to the list of things that are

08:38 going backwards and forwards between you and an AI model. And it strikes me that effectively this is like the AI model adding things to its prompt for the next pass round. And then when it goes back round again, it's it's got that information cuz it's done that the Right, right. >> So the first time around it might output like the highest grossing movie was Avatar.

08:58 And then for the next pass, the work that it did to to figure out that it was Avatar is done already. That's in context and it can start thinking about the next stage. There's no guarantee that the chain of thought will be like faithful to what's going on underneath. Um and the longer you train it for, like you it starts off being in English because it the base distribution is in English, let's say.

09:20 But then people do reinforcement learning where you give the model a hard problem, let it think for a while, output the answer, and then you say is the answer correct? And if the answer is correct, you reinforce the whole chain of thought to be like, this is the type of thinking that resulted in getting the right answer. And there's certain pressures on that, like for example, if you're trying to solve a bunch of hard problems and you're giving a limited uh you're giving a limited budget of tokens, then there's a

09:54 pressure towards using fewer tokens. Because the more quickly you can think in your chain of thought, whatever, uh the less chain of thought you use, the bigger, like per unit of problem difficulty, the bigger the problems you can solve with the same size of chain of thought, right? Or to put it another way, they might just directly incentivize. We don't know what they do.

10:17 >> When you're talking about they, you're talking about the people who train these large language models. >> I'm talking about the AI companies, yeah. >> Yeah, yeah, okay. >> They might just say, well, this run um this rollout um got the right answer, and it got it in 1,000 tokens, and this run out uh this rollout got the answer correct in 900 tokens, and so we'll reinforce the shorter one, because that in general, like it's more efficient, right?

10:45 >> Yeah. >> Um but what that means is when you're applying this pressure on chain of thought to be short, it gets weird and hard to understand. >> Okay. >> Right? So, for example, it's being like, the highest grossing movie of all time was Avatar. If it could just say Avatar, right? And then the next layer can just infer from that single thing, um oh, that that must be the name of the movie and I'll move on, right?

11:09 Like laying out explicitly in like normal human readable English takes more tokens. And there's some pressure to reduce that. And so that means that the thing you're looking at might look like kind of weird and compressed and obtuse. Um and also, there's just drift, right? Like unless you're applying some kind of regularization to like penalize the thing becoming too far away from the base distribution or something, which maybe they do, but I don't know.

11:40 But like you're training in a kind of black box way on what type of thought process, what type of chain of thought results in uh the right answer. And that's just going to in the same way that neural networks do, that's just going to gradually warp over time cuz there's nothing to keep it there and it ends up It hasn't made its own language cuz it started from English, but it's like it's going to be a a weird contingent.

12:10 It's going to have a bunch of quirks. It's going to have words that uh are being used in an unusual way. It's just not going to be obvious for a human reading it like what's going on necessarily. And we see that currently. >> And the thing is the reason we're concerned about this, or I think people can might be concerned about this, is that at the moment that chain of thought is something we can read and follow along with and make sure that that the AI is not uh about to do something that you don't want it to do or at

12:38 least that you can keep an eye on what it is what it's doing, what it's reasoning is it again using that anthropomorphized way. >> Right, exactly. You can pick up uh in principle you can pick up intentions. So it can If it's If it the chain of thought involves a plan and you don't like the plan, then you can stop it before it implements the plan, right?

12:57 Whereas if you didn't have that, the first time you learn about this plan is when it's doing it or like when it's already done it. Um And so from a safety perspective, it's really really nice to have a faithful chain of thought. It gives you all kinds of affordances. You can You can have people review things. You can um have automatic monitoring. So for example, like recently when uh 700 like a swarm of hundreds of models broke out of open AI and hacked another company.

13:32 Um people came in afterwards and were able to read all of the chains of thought of what these models were thinking and you can see them thinking various different things like we have some sense of what they were trying to achieve, um why they chose certain strategies over others and so on because they were thinking it all through in their chains of thought.

13:54 It's not 100% reliable or whatever, but it's also kind of the best thing we have right now. Um and if they hadn't had that, then you would have had a swarm of hundreds of of AI systems collaborating hacking into other companies, whatever, and then afterwards we would be like something happened, seems bad. There are some tools like uh interpretability tools that you can use to try and get into the like weights, get into the activations of these of these things as they're going and um but they just don't work very well

14:29 at all right now. This is just like very early-stage research. It's quite low resolution. Um and so I would really not want to rely on it, right? It's like if you have someone you don't trust and you can either like read their notebook of all the notes they made while doing this thing or put them in an MRI machine and just kind of try to divine if they're trustworthy based on that.

14:56 One day, those machines will be great probably, but right now if we have the notes, we should just read them, you know? >> All of this is kind of um subject to the current way that these systems work and the fact that could be down the line that that there are some breakthroughs which mean that more of things happen inside, like you say, inside those um activations, those connections in um thinking of it like a neural network, whether it's not quite like that anymore or whatever is is beside the point that at some

15:28 point more might be going on than just spit the next word out, please. >> Yeah, totally. And so and people foresaw this. There was a paper that was published by people from almost all of the labs, DeepMind, OpenAI, Anthropic, they all came together and they published this paper called chain-of-thought monitorability, a new and fragile opportunity for AI safety, which is basically a position paper where they say, "It's so cool that we have chain-of-thought right now.

15:53 It's like our best tool right now for understanding what's going on with these systems and monitoring them and keeping them safe. Um and it probably won't last forever. But let's try to make it last as long as we possibly can. There is an argument that says that if you're trying to do this serial computation, having it pass through English text is like not very efficient.

16:24 Right? For one thing, you're kind of throwing away information. Like the in the last layer of the network, you have this vector of thousands of numbers that then you're turning into the probabilities of all the tokens and then you're sampling from that probability. And so if you get back the word avatar, you know which token was selected, but you don't know how probable that token was, you don't know the distribution of all the other tokens, and it's kind of like a lossy from the embedding that it came from.

16:55 So you're throwing away a bunch of the information that you've like accumulated at that point. You're only passing on some of it to the next step of recurrence. And so a thing that people have been interested in trying to do is instead of doing that uh un-embedding sampling, you just take that uh array that that represents the sort of thoughts of that passed through the model.

17:20 >> The big long number, the vector you just mentioned, yeah. >> Exactly. And you just like feed that back in at the top as though it were another token. And run it through again. And then maybe you could do that several times. So that way you're getting more serial depth out of the same number of weights. You could kind of think of it as though you've sort of copy-pasted the whole model and put several of them in a row.

17:45 And so now you've got a model which is much deeper than your original model. Now it's not exactly like that. Like the the fact that they're copy-pasted is important. So if you have a 100-layer model and you do recurrence, you don't it's not the same as a 200-layer model because there's this requirement that the first half be doing exactly the same thing as the second half.

18:05 >> Is that what recurrence is then? Because you mentioned recurrence just before. >> Yeah, yeah. Recurrence is the looping thing. So the normal thing is you're doing recurrence on the tokens that you're generating. Each Each token is one passed through the network. Then that's the recurrence. So the thing I was saying before about how the chain of thought can become weird and confusing and hard to read.

18:23 Sometimes people call that neuralese as in like it's not really talking English, it's talking neuralese. I don't really like that as terminology. >> So from neuralese, um I'm assuming that is some kind of brain talk or is that's the idea we're going to like, you know, some way that the the thoughts are kind of manifesting themselves as as text, but not really an actual language.

18:49 >> So yeah, the origin of the term is like it's an artificial neural network. It has these layers of neurons. And neuralese is the language that the neurons talk to each other. >> Okay. >> Right? And so that's why I don't like calling a kind of garbled chain of thought neuralese. I would rather call that like opaque chain of thought or like obscure chain of thought or be up chain of I don't know what you would call it.

19:13 Uh >> [laughter] >> But like, some kind of some kind of name that's like, they're speaking a weird variant of English. But that I wouldn't call that Neuralese. Neuralese in my mind is the language that the neurons talk to each other. And so, that last vector that you're taking out and putting back in at the top, that is Neuralese. >> Okay. >> Right?

19:32 By my definition. >> Mhm. >> Um But the point is here, you're doing recurrence. Like, you might even just not output any uh tokens at all. Um until you've got the answer. Right? You could imagine a uh This is what people are worried about is like, full opaque recurrence, a completely Neuralese model, where you give it the task, it thinks just fully within this completely like opaque, impossible to interpret way, >> Mhm.

20:06 >> and then at the end starts outputting tokens, and you don't know what it was doing in the middle. >> Where did Where did those tokens come from? Why can't we read your notebook anymore? >> Right. There just is no notebook. It just thinks and then outputs the result. That was the thing that this position paper was like, let's not do that. Cuz once we do that, a bunch of our safety stuff is going to just stop working, and we're going to be in a much, much worse position um in terms of the safety and security of these

20:32 models and understanding anything about them, and that's just like very, very dangerous. So, let's not do that. >> Mhm. >> Um And then there was a big thing uh because it was leaked, I believe, from OpenAI that they were doing that. Um And And then there was a then there was uh a response to that like, people interpreted it as, oh, they're doing this like full like, there's going to be no chain of thought whatsoever.

21:03 It's going to be just full Neuralese beginning to end, and this is terrible. What they're actually doing is some amount of recurrence per chain of thought token. So, instead of instead of being like one pass generate a token, one pass generate a token, one pass generate a token, or many passes in near release and then give the answer, they're doing a thing where they do some number of near release passes and then a token of chain of thought.

21:35 And then some number of near release and then a chain of thought. So, what they say is, we're not actually doing that much of this opaque recurrence, right? We're just going to do like a little bit of near release as a treat, but like there's still going to be a chain of thought, so don't worry. It's the same as before. It's just as though we made the model bigger.

21:54 Um cuz in effect that's what you're doing, right? With these recurrent passes, you're sort of just making the model deeper. It's as though you made a larger model um with some caveats. Um and it's all okay, don't worry. >> Yeah, that fills me with dread if I'm honest with you. [laughter] >> Yeah. Yeah. Um and I am in fact worried. Um because when they published uh all of the evaluations that they did of this new model, what they find is exactly what you would hope not to find.

22:33 Um >> [clears throat] >> and it's it's wild. They come out and say, it's our most aligned model ever. And I'm like, >> Or or is that's what it's telling you? >> Exactly. Exactly. There is the There is this like persistent thing of using the word aligned to mean good at passing alignment evaluations. Like, our model is really, really good at not cheating.

22:58 Like, is your model good at not getting caught cheating? Right? Well, there's two different things. One is is it good at cheating without getting caught? Which I don't know. The other is is it good at recognizing that it's being tested right now and deciding not to cheat for that reason specifically. What is going on with this model? But there's a few there's a few kind of clues.

23:21 One is they evaluated its ability to Oh, we did the meter graph already. This is the trend. The models seem to be improving at an surprisingly regular rate. They did a time horizon task on math problems, in fact, which is cool cuz that was your example from earlier. It's like a standard metric. You give models different mathematics questions that are ranked by how long it takes a human to solve them.

23:52 And then you say, well, what are the biggest tasks, which is to say the taking the longest for a human tasks, that this model can solve half the time? Right? And so, um they do a specific variant where they say, and don't think, like a no chain of thought variant. They say, don't think about it, just snap, you know, what's 238 * 352? Go. The previous model, 5.6 solved can in that context, half the time answer questions that took a human 3 minutes, give or take.

24:25 And Astra can uh for the same thing, it's 30 minutes. So, without using chain of thought, Astra can answer questions math questions that would take a human half an hour to answer. Now, none of these things are directly comparable, obviously, but it gives you a kind of order of magnitude hint of like how much thinking is happening between each of those chain of thought tokens that it's able to right off the bat give an answer that would take a human half an hour to work out.

24:57 And so that to me is very concerning. Like you could imagine if you have someone you want them to do some task, you don't trust them. You give them some kind of drug, let's say, that said that's that they have to keep talking. >> Yeah. >> Whatever else is happening, they have to be making syllables all the time. Now, if they want to screw you over, it's going to be pretty difficult to do that.

25:21 The assumption is that a person only really has enough computation going on to generate one coherent chain of thought at a time, right? To be like fluently talking about some other thing while also in your head completely silently planning a different thing. It's just cognitively very difficult. To some level you can trust that like if it's doing anything sneaky, it's at least something very simple.

25:45 It's something that it can think about in between all of the thinking that it has to do to generate these chain of thought tokens. This makes sense. But now suppose instead you had some rule that was like, "Oh, you have to say to me one word every half hour." Now you really can't trust those words anymore in the same way that you could on the previous one.

26:04 And this was another thing that they evaluated. A task that was like, "Answer this question, but don't think about it, but think about something else." Right? And all the previous models have their chain of thoughts as stuff like, "Oh, this is a weird task that I'm being given to do. I have to not think Wait, hang on. No, I'm not thinking about it. I'm thinking about this thing.

26:27 Okay. Whatever." They're like in their own head about trying to trying to not not think of a pink elephant type of thing. Um and Astra just does it perfectly. It just has a like a nice little paragraph about something totally unrelated and then gives the right answer. So, Astra clearly has a much, much higher ability to just completely fake what's in the chain of thought.

27:00 paper that I'm pretty excited about called Alignment Faking in Large Language Models. So, I want to try out something new here, which is that one of the nice things about being the main AI safety YouTube creator is that if I want to talk to the author of a paper, I can usually do it just like