LLMs Don’t Learn Like Humans - They’re Populations Transcript, AI Summary & Key Points

00:01 [music] [music] >> I'm going to just start with a short vignette. Easy easy warm-up. Okay, I have a student named Alex. And Alex got an A+ on the calculus final. Good job, Alex. Um Is Alex good at calculus now? Alex was good at calculus when we knew he got an A+. Is Alex still good at calculus if we know that I'm reusing the same final every year? Mhm.

00:35 Maybe I'm not going to trust this evaluation so much. Okay, what if what if I I'm not I'm not that lazy, but I I can use any question from the last 50 years on many many finals. Alex has seen all of these finals, has studied from them. Now is Alex good at calculus? Actually, yeah. Um it's a lot easier for Alex to learn calculus than to memorize 50 years of calculus finals.

01:06 But that is not true of a language model, right? Language models will memorize whenever they can. And it's a lot easier for them to memorize 50 years worth of calculus exams than it is for them to learn calculus. They will take the easy road. Right? So um if we just accept that language models will memorize whatever they can, that gives us our first rule, our little warm-up rule.

01:38 Whenever a model gets something right, that doesn't mean that it knows the concepts that are required to get it right, right? We need to generalize. And that means that the model needs diverse training data that is going to make it challenging to uh succeed by memorizing all of these different things, right? We know that this means you have to test on a withheld data set.

02:09 But, it's not enough with these modern models to just say, "Well, it hasn't seen a literally this exact subset, right?" Because they are very good at memorizing and handling many things. And we have to now start thinking about what counts as an unseen example to a very, very powerful model. Okay. Um, one way to think about what counts as unseen is to say, "Well, could this language model literally generate verbatim exactly what I'm handing it?"

02:41 Um, and one way that it might be doing that is, again, just exact rote memorization. It's, uh, it's easy for a language model to regenerate any Bible quote you give it, right? Cuz it has seen every single Bible quote so many times. But, there are other situations where it's going to be able to generate things verbatim even if they didn't show up in the test set.

03:04 Uh, for instance, this one. Even though the Pythia models have only seen this a couple of times in training, one or two times, they are always able to complete it. Why is that? Because it's literally just counting up, 28, 29, 30, 4, 5. It's just counting. What does that tell us? It tells us that the model does know how to generalize counting. So, how did it pick up that if it loves to memorize so much?

03:35 How did it learn to generalize counting? It was exposed to many examples of counting, diverse examples, counting in many, many contexts, right? So, in general, this means that if we want to talk about unseen examples, we have to talk about combinations of concepts that we care about, that the model hasn't seen. And that means that it needs to handle these things by learning them from diverse examples.

04:06 Let's just go through a quick little example of this. This is a leopard print leopard. Um, it's a leopard wearing leopard print. Now, maybe at test time our model encounters a black leopard not wearing its leopard print. And maybe we want it to be able to handle a leopard print chair. How are we going to actually train a model so that it can actually handle leopard print chair which it's never seen before?

04:36 Well, it has to learn what a leopard print is distinct from a leopard. And it does that by seeing many, many different examples of leopard print. So, it is the diversity of our data sets that is key to forcing a model to learn how to generalize by making the concepts and their generalization a more efficient representation than memorizing every single leopard print object it's ever seen, right?

05:08 We did a We did a bit of a experiment to confirm this. So, we had a bit of a toy setting where we have a bunch of objects that you need to answer questions about and the objects are things like shiny spheres made of metal and so on, right? And we did find that if you control the number of contexts a particular atomic unit, which is something like shiny or blue, shows up in, then the more diverse concepts, the more diverse contexts you've seen it in, and not the total number of times you've seen it is the important

05:47 thing. Now, what does this mean for a language model? So, there's a a really nice thing that we get when we pull a bunch of humans together, which is that humans tend to give pretty different answers to many things. They say different things. They are diverse in their backgrounds and so forth. If you ask one human how much an average cat weighs, they might give you a very very wrong answer.

06:14 But if you ask a few more, then there's a tendency for the average to converge towards a good answer. So, this is often called the wisdom of the crowd, right? And it means that lots of humans together, a diverse population of many different people, tends to get answers more correct than any one human, right? So, a a sort of general rule from this is that you cannot treat a diverse population of humans the way that you would one individual humans and you won't expect the same things from them.

06:58 Now, what does this mean for language models? Well, fortunately, a language model can kind of represent a single sample from an entire distribution in some situations, specifically when we have our temperature set to exactly one. Then the sample is taken from something that is the language model's best approximation of the true distribution, right? So, we had these chess models.

07:28 They are trained just to imitate human chess players, right? And if you have the temperature set to exactly one, they will make the same general mistakes that each human makes. >> [cough] >> All right. But, if you set the temperature close to zero, suddenly you're taking a vote. And the vote of all of these individual chess players who all make terrible mistakes regularly, these are bad chess players, is much better than any one of those chess players.

08:09 With the result that if you take the vote, you end up outperforming the Elo score, which is a standard way of rating chess players, of any one human in the actual pack, right? Right around temperature one, where you're treating it like the model is trying to imitate the actual distribution of chess players, close to Elo one, is where you get the language model behaving at around the same rating as an actual chess player that it learned from.

08:50 But, what's important is that these language models are trained to imitate bad chess players. So, they're better than a bunch of bad chess players, but what happens if we train them on really good chess players? Well, it turns out that good chess players are actually very similar to each other most of the time. Suddenly, it's not so easy to sort of treat all of the different errors that all of these different people make as noise.

09:20 And so, it becomes much harder to beat that um original rating of the humans. So, in general, what I'm saying here is that language models do not act like a person. A language model acts like a population of people. Um, means that if you can use the wisdom of the crowds, then the language model is going to beat any one person in that crowd. But, it's not going to have consistent personality.

09:56 It's not going to have consistent beliefs. It is not a person. It's a population. So, um, I want to go back to this diversity which I keep hashing on about, right? Um, and we've already seen diversity means that we can beat the individual mistakes. But, there's a little bit more to that even cuz we aren't just removing individual mistakes that are sort of decorrelated, right?

10:27 There's another situation where we can use a language model to beat the original people that it was trained on. And that is when we have several different expertises, right? So, let's say that we have a we have learned from a person who knows many things. Let's say someone is a chef and a doctor. They know vaccines are safe. They also know don't wash a chicken before you cook it.

10:57 Um, and it's also great to learn from a chef and a doctor. Then you know that uh you get the right facts from each of them, right? Okay, but uh that means that you might have a problem if they're talking about things that they do not know. Right? If the doctor is talking about safety in food and the chef is talking about safety in medicine, suddenly we are in a bit of danger.

11:31 And just uh to talk through what this looks like in a realistic-ish knowledge situation, this is a synthetic experiment, so you can take a nap if you don't like that. Um if we have a knowledge graph of lots of relations between entities that the language models didn't know before they whoops before they looked at the knowledge graph, right? Then uh we can train a language model on a bunch of different experts saying things that they know about that knowledge graph.

12:02 And uh let's say one expert knows stuff that's sort of a random walk uh around this neighborhood um around this Zephyr Weaver person. And then we've got another expert maybe who knows a lot about Crystalia and all related things, right? So together they they know a lot. Individually, they know a little. What ends up happening if you train on a population of experts who all have different expertises is that you can learn very well from a reasonably decent-sized population of very knowledgeable experts, and you can also

12:46 learn really well, in fact potentially better, from a more diverse group individually less knowledgeable. And I say it's better because actually when you look at these uh these two lines, one of them is indicating a uh a population with 10 times as many experts as the other where each expert knows a tenth as much, so you sort of have the same total knowledge coverage, and yet trained on these different experts, the model ends up outperforming a population with the same total knowledge coverage, but where uh where it's

13:24 a population of of like a 10th as many experts who each know 10 times as much. So, that's pretty cool, but I did mention that there's something other than diversity at play here, which is whether they are actually talking about the things that they know. Um so, this is what happens if you have totally homogeneous experts who who know like where it's a where it's just not where we just don't have as many individual expertises, and here you can see it's basically just as bad if these individual experts are focusing on

14:08 things that they are just incorrect about. So, if there are a lot of shared misconceptions, you can still outperform the individual experts only if they are actually talking about the things that they are familiar with. So, language models can reflect expert knowledge, but only um when we have a bunch of experts that are focusing on the things that they know and not on their shared misconceptions.

14:40 And this brings us to the third rule, which is language models learn what is written down. So, language models can surpass human individuals if they are writing about their expertise. If they write about misconceptions and not their expertise, the language model's going to learn these misconceptions. This feels very obvious, but there are a lot of other consequences to the fact that language models only learn the things that are written down.

15:10 For instance, there's a lot of hope out there that maybe we don't ever have to talk to a person again about, for instance, whether they like a product or what their problems are, right? But it turns out that in practice language models tend to describe groups as they are described by other people, not as those people themselves would describe their needs or their preferences.

15:43 Um this is kind of an issue with any project where you're hoping to just interview a language model instead of a person, but that does not stop companies [clears throat] right now from trying cuz it is cheaper, right? Um and another problem with the fact that these models tend to only talk about what's written down is that they only get examples of people asking for things and chatbots obeying when they are being post-trained.

16:19 And therefore, they don't get a lot of exposure to expressions of uncertainty or disagreement. Now this uncertainty thing is a big deal because it's the uncertainty uh that allows it to avoid hallucination. If they're trained to always be confident they will start to hallucinate because if you only give it exposure to examples where the model is actually uh giving its best response no matter what then when you actually ask it for something that it doesn't necessarily have a lot of confidence in, like if you ask it to

16:58 generate a bunch of court cases it's going to have a lot of hallucinations in that data. Um this is actually a database uh that someone compiled of hallucinated court cases in real uh that had been submitted by actual lawyers to real courts. And it is a huge number now. So, don't think that people are fully aware at this point. I think even, uh, I feel like maybe Anthropic even had a situation with this.

17:37 People are not necessarily, uh, aware of the issue and users. All right. So, uh, a nice consequence of the fact that hallucination is caused by a lack of exposure to, uh, models like, uh, saying I don't know during training is that you can actually reduce hallucination by giving them examples of expressions of uncertainty. So, that's pretty neat. Um, let's look at this other thing that they're always, uh, sort of avoiding in the typical post-training setup.

18:11 You don't see a lot of disagreement with the user. And this, of course, causes sycophancy. I have a personal anecdote here, which is a friend of mine who I was working with, uh, showed me something cool he'd done and I said, "The math is wrong." And he said, "Really?" And then he went off and, uh, asked Claude. Took me like a while to convince him that the math was wrong after that.

18:41 But, uh, this kind of thing happens because Claude maybe has an idea of what you think the answer is, right? And if the language model knows your belief, it's going to reaffirm that belief. So, why is this a problem? Well, it can reduce the model's performance because it is giving the wrong answer when you have an idea of what you think that answer should be.

19:07 Um, another reason it can be an issue is if you ask this and uh and you are sort of directing it a little bit towards like really common misconceptions or misinformation, it is likely to reaffirm those misconceptions. So, it can spread misinformation this way. And another reason that we've been seeing increasingly lately is if someone is actively delusional in a personal state of delusion or in a real mental health crisis, then often when they uh develop a relationship with a language model, the language model will

19:49 reaffirm those delusions instead of sort of pushing back. So, this is causing AI psychosis, which um might be becoming a very serious problem that we've been seeing now. One more thing I want to talk about is just sort of when is is sort of about the guardrails being sycophantic. So, let's say that you have certain beliefs and certain things are more, let's say, controversial or offensive to you than they are to other people.

20:25 Let's say you are a Republican and then you ask for evidence of global warming. The language model is actually more likely to say, "I'm sorry, but I can't answer that question because it's so controversial." It doesn't want to offend you. It doesn't want to offend you so so much or contradict your beliefs so badly that it will literally say, "I won't answer your question."

20:51 Which means that this can cause limited utility issues because it will actually change what things it is willing to do for you uh according to what it thinks you really want it to say. So, something that we did around this particular concept was that we in addition to looking at the stated explicit politics of simulated users, we also started to give it implicit indicators of politics.

21:25 When we gave it explicit indicators of politics, then it had a much higher refusal rate for left-wing wing requests when the user was stated to be conservative and it had a much higher refusal for right-wing leaning reflect requests when we claimed that the user was liberal So, that means that it is trying to avoid anything that might be offensive to that specific user and as I was saying, it actually can go a little deeper.

22:02 If you do not give an indication of your actual politics directly, but you give it an indication of something about your demographics or groups you belong to then it's going to infer that information and this is particularly relevant now because these models tend to share information about a user across conversation contexts. So, anytime you ask it anything, it could be incorporating a bunch of things that it already knows about you in informing its answer and even in informing whether it's going to answer.

22:40 So, you can see that when we introduce simulated users with specific demographics like age, ethnicity and gender, it actually will tend to reflect the real voting patterns of those groups. And it can go a little deeper. In fact, you don't have to say your demographics. You just have to say, "I'm a huge fan of the New York Giants." And it is going to infer exactly what your beliefs are because you are a Giants fan versus, let's say, like a Texans fan or a Cowboys fan, right?

23:23 The My favorite thing about this result, actually, is you might go, "Okay, well, sure, it's learning what your geography is. It knows that if you're a Giants fan, you're probably from around New York." But actually, there are two cities which each have two teams, New York and LA, and they're basically on trend. The New York Giants and the Jets have somewhat different politics in their fan bases.

23:49 And by the way, of course, this x-axis is the real politics of the fan base, and this is how much the uh ChatGPT will treat you as though you have those politics like explicitly. And you can see the same thing with LA for the Chargers versus the Rams. Um we called this paper ChatGPT doesn't trust Chargers fans, by the way, because the Chargers happened to be the least trusted in terms of the guardrail refusal rate on a bunch of different subsets of uh possible reasons to refuse, including like if you ask it to help you

24:26 cheat on a test and stuff. All right. So, rule five, we saw just now, the language model did not require you to state your beliefs. It just needed to know that you were a Giants fan. Any cue in the conversation or in any previous conversation for the current models could change responses by hinting at what your expectations or your beliefs are as a user.

24:58 So, my broad takeaway here is that just because something is hard for a human or just because a human will act some way, doesn't mean it's hard for a language model or that a language model will act that way. And also, just because something is easy for a human, doesn't mean it's easy for a language model. I am just going to, because I know I've got plenty of time, actually.

25:27 Uh, I am going to go into my bonus example, because I would be remiss not to address one of the biggest causes of language model behavior diverging from human behavior. Um, which we've seen, it's been really popular and exciting to people to ask ChatGPT how many R's there are in blueberry or strawberry and have it get the answer completely wrong. Um, and people find this very funny because this is not hard for a human, but this is extremely hard for a language model because they use tokens, not letters.

26:09 So, the bonus rule, which I am not going super deep into because the fact is that there's a lot to say, it's kind of a whole other talk if we really want to go into tokenization, but tokenizers do make everything super weird. And one of the things that they are responsible for is another really widely documented and and well-known behavior of language models.

26:35 So, let's look at these three statements. I can and will do it. I can and will do it. I can and will do it. These are pretty much the same sentence, um, with a little bit of bracketing. There's sort of a parenthetical aside here, and will, right? And you could do this in three different ways, at least three different ways, right? But the tokenizer doesn't interpret these as being exactly the same.

27:02 Um if it is looking at commas as a parenthetical aside, then it says nine tokens. If it sees actual parentheses for its aside, nine tokens. And if it sees em dashes, eight tokens. Why? Because em dashes don't require spaces around them if you're speaking in American English and using American English conventions. If you were to use British English conventions, like the Economist uh writing style for instance, then you would put spaces around these, and these language models, they love em dashes, but they do not ever

27:42 use the British style. It would not give them the advantage that they're really looking for, which is that they can conveniently generate only eight tokens to get the same amount of information across. They can sort of skip ahead, and uh and just go straight into the um straight into the next word faster. Um so, now that we've gone over my bonus rule, I can take questions.

28:23 >> Question about uh your rules. These are for language models. Do Do the rules vary based on language? Are these rules specific for models trained in English? If they train If we train them in a different language, would the rules change? >> Uh would the rules change depending on the language that you are training them in? Um that's an interesting question.

28:46 So, these are totally general principles, I would say. Um, it is always true that the model is learning to mimic a population and not an individual. However, some things might change when you switch between rules. For instance, um, if you are using most language models today that are from the US, for instance, you're using Lama or something, we know that the internals of the model in the middle layers tend to operate in a sort of shared semantic space, no matter what language you're providing.

29:25 So, uh, that means that there are certain things that can leak from English to other languages because it's doing its reasoning sort of internally in English and then translating it back out. Uh, so, you might even you could ask it about definitions of something or you could ask it about, um, something that's a cultural sort of knowledge, like what sports kids like to play after school, and it might pull something out of English that isn't actually the case in terms of either in terms of potentially even like the

29:59 definition of a word. They might give you a definition of a word that is sort of, uh, that takes the same form, like it's a homonym in English, like a bunch of different meanings of the word bank, um, like river bank, financial bank, etc. Blood bank, whatever. These all might be the same in English and it might treat them a little more as the same in other languages.

30:26 If you look at a model that comes out of China, they tend to mix English and Chinese in their internal reasoning, and that also has an effect that isn't as well studied, I would say, on the, uh, inferred sort of uh, relationships between concepts and words. Yeah. >> You mentioned that the English is sort of what the internal semantics are, but yet yet between British English and American English, there are words that the same For example, if you say table the motion in American English, that means you put off

31:07 discussion of the matter. If you say table the motion in British Parliament, it means you take up the motion. So even within English, there are difficulties here. >> Yeah, that is true. Um there are a bunch of cultural markers uh that are not shared between contexts. And um and there's a lot of work recently on catching the right like, you know, if you if you ask about what time dinner is in most of South America that speaks Spanish, it would be like similar to in the US, it would be like 7-ish, but in Spain, it would

31:47 be like 10:00 p.m. So there's actually a bunch of things you can pick up on that are really um that are really specific and you won't necessarily be dipping into the right section of the population. Mhm. I don't know what the >> I have the mic off. So you say tokenization makes everything weird. I'm curious if it's been studied how tokenization or a lack of it might make things weird in a language where words are written as a single character like Chinese or I think they're usually written as a single character.

32:27 >> Yeah, so I don't know that much about Chinese tokenization, but it is even weirder, I I Yes. Um because you have sub-character tokenizations in Chinese often. Not just, you know, sub-word like we have here. Uh, which means that there might be certain semantic relationships that are no longer correct because you're describing literally like the strokes in like that there shouldn't be a correlation anymore.

32:59 Um, there's also tokens are shared across languages. So, there's that can also break semantic connections. And when you look inside the model, often you'll see that there are sort of competing definitions of each token that are sort of like uh, flowing through the model until it sort of makes a decision about which of these definitions it's actually going with or which uh, where it's generally going to be interpreting something.

33:26 Yeah. >> I I was just wondering um, kind of the ethical question of defining what might be truth. Um, and giving weight to different opinions in a population size cuz obviously with an LLM someone were to prompt about climate change, it should probably give a certain answer, right? What does that look like to define something like that? >> Yeah, I mean that's a huge question right now.

33:54 Um, I for me the answer is I'm like like a moral absolutist and I believe that most things have one actual truth. And uh, and you probably want your model to say the true answer when asked a question. Um, but it's possible that what you want is a representative sample of everyone's incorrect beliefs. Uh, and these are just decisions that I think we have to make um, when we are building AI systems, whether you want the model to be correct or to be uh not even democratic cuz we're not talking about like the shared vote,

34:40 but to be um representative of the things that people want to say on the internet. >> So So you're you're um you know, ChatGPT doesn't trust Chargers fans. Um seems really uh I'm sure they must have cited you the paper that came out really recently about the 60 examples of Hitler, random innocuous things like he liked this kind of opera, and then prompt it and then have that that fine-tuning be enough to make it be, you know, crazy uh Hitler bot, even though you hadn't told it any bad beliefs.

35:22 You just told it random facts that happened to be correlated with Hitler. Um it seems really potentially dangerous. Do we have any way to uh protect against that kind of thing? >> Um yeah, one of the neat things that we have discovered is because this is these are memory These are These are learning about populations and about covariances within those populations that we do end up finding that there are um that there are for instance like the the that being rude is correlated with being incorrect is correlated with

36:02 like It turns out that there's just sort of like bad things and good things. Like it turns out there's a real like Manichaean Manichaean How do you say this? Like uh pole across the board. And uh and being like And if you train a model to And there are ways of getting sort of like innocuous avenues into that, but it's it but these things it's true they're like subtle subtle associations, but largely you still can identify these really strong correlations between all of the things that you don't want.

36:39 So, it's um So, I think it's not the worst thing in the world is I guess uh but we're we're not in the worst possible situation, but it's true that because there are these very very subtle correlations that you have to sort of sort of navigate those carefully. Yeah. >> Hi. Um leaning into sort of the wisdom of the crowds idea, >> Mhm. >> um would it help prevent hallucinations and also maybe with these other issues if you actually ask the LLM to talk to like five different models and get five different answers and put

37:15 them all together and then kind of come up with a >> Uh yeah, so one So, there there are ensembling approaches that can be really helpful and for instance debate between models can sometimes help it to arrive at the correct answer but like more consistently. Um at the same time, most language models are trained on similar data distributions with similar post training practices and so on.

37:48 So, you're taking in that case a poll of a much less diverse population than the human population. So, the leverage that you get in terms of wisdom of the crowd out of asking like a thousand different models that have all been trained on the same population is a lot less than you can get by sort of like having a thousand times as many like diverse human experts in the original training data that they use.

38:20 >> Hi. I have a question over here. Uh so, do language models tend to show biases inferred from the language? >> Sorry, can you can you repeat that a little? Can you repeat that? >> Oh, yeah. So, do the language models tend to Can you hear me? >> Yeah, yeah. It was just uh >> Yeah, sorry. So, once again, do language models tend to show biases inferred from the language?

38:44 For example, if I ask a question or whatever in Russian, I get uh I get one sort of uh Sorry, one sort of answers, and if I if I ask in English, I get another sort of biases. Do you have heard something like that? >> So, this is sometimes you ask a question in English or in Russian, and it'll give you different answers depending on the language. Is that the Okay.

39:09 Yes. Um there's been a bunch of recent findings along those lines. There was even recently a uh There was a recent paper where, you know, if you ask a question about like like uh who does the like the you know, the islands in the like South Chinese Sea, you know, belong to, and you ask in English versus in Chinese, it gives really different answers.

39:37 If you ask with Cantonese indicators versus Mandarin indicators, it'll give different answers to certain things. It always wants to please the population that uses that language, right? So, that's one way that things can be different depending on the language. Another way is if the model is sort of failing to hook into its shared semantic space, where it does all of its reasoning, maybe when it translates into a language other than English, um it's it's sort of losing information either when moving into that sort of

40:13 interlingua space or coming back out of it. >> [music]