Model distillation is the process of transferring knowledge from a larger teacher model to a smaller student model, where the student learns from the teacher's soft labels rather than hard ground-truth labels.
Searchable transcript of What Is Model Distillation? How Smaller AI Models Learn — IBM Technology (13:33). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by IBM Technology. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:00 Have you ever wondered how AI models get small enough to run on devices like phones and laptops without completely losing their capabilities? The answer is often something called model distillation. At a high level, model distillation is the process of transferring knowledge from a larger model to a smaller model. The reason this matters is pretty simple.
00:38 Bigger AI models can be incredibly capable but they also require more memory, more computing power and generally cost much more to run. So the idea is what if we can take what a large model has learned and transfer it into something smaller? What if we could transfer this knowledge over? We may be able to get a model that is faster, cheaper, easier to deploy and still very capable.
01:10 Now first, let's clarify what do we mean when we say a model is smaller? Usually we're talking about the number of parameters in the model. You can think of parameters as the internal numbers that the model learns during training. Those numbers help determine how the model will turn an input into an output. A large model might have billions of these parameters, maybe something like 70 billion.
01:37 A smaller model has fewer, maybe something like 7 billion. And fewer parameters in a smaller model can mean less memory required to store the model, less computation required to run it. Lower latency, so higher speed and a lower cost. It may also make it possible to run smaller models in places where a very large model simply would not fit. For example, on your phone or on your laptop or on any other hardware with limited memory or compute.
02:14 In model distillation we usually give these two models names. The larger model is called the teacher and the smaller model is called the student. That analogy is actually pretty useful. Instead of asking the smaller model to learn everything completely on its own, we allow it to learn from the behavior of a model that is already more capable. The teacher already has knowledge, the student is trying to capture some of that knowledge in a smaller package.
02:49 Now, so far we have understood what model distillation is and why it's important, but now we're going to ask how does it work? Now, before we get to that, let's review how a large language model works in the first place. So, an LLM. I'm going to use an LLM as an example here although model distillation can apply to other kinds of models too. When an LLM generates text it works one token at a time.
03:20 A token is roughly a word or part of a word. At each step, the model looks at everything that has come before and tries to predict what token should come next. So imagine I have an LLM and I give the model the following input sentence "The cat sat on the..." Now what's going to happen is I'm going to feed this into my LLM. And in order to come to an answer, the model is going to put...
03:54 is is going to consider every single possible next token. So internally what the model will do is it will assign different likelihoods to all of these possibilities, so we can imagine this LLM saying something like mat. Is a very possible output, right? The cat sat on the mat. And it might assign a 60% chance to that. For example, floor would also be possible.
04:20 Maybe we give that 20%. Chair is also maybe an output give it 10%. And it does this with every single word in the vocabulary even trying, you know, thinking about words like calendar which it might give, you know, a very small percent chance to. Again, the exact numbers are not important here. What is important is that the model has information about all of the alternatives, not just the one it eventually chooses.
04:53 Now we have understood that a language model will output soft labels for every token in its vocabulary. Let's go back to our distillation setup. Recall that we have a teacher, which is our larger model and a student, which is our smaller model. The goal was to have the student learn from the teacher. But first, the teacher has to be trained itself. We do this by training the teacher the normal way.
05:16 Unlabeled data. What do I mean by this? Well, let's start with what our data would look like. Let's say we have this data set as a sentence and the sentence is "The cat sat on the..." Okay. And because this is, you know, a sentence in our data set, we know what the answer should be. We know that the answer is mat. In this case. Now, how do we train our teacher model?
05:45 Well, we take this data point, feed it into the teacher model. Okay. And remember we already learned that an LLM will output, you know, it'll assign a probability or a likelihood to every possible word that could come next. So if our teacher is, you know, doing a good job it might say something like okay, mat should be 60%, floor should be 20%, chair should be 10%, and so on.
06:18 And it will assign a number, a likelihood to each, um, token in the vocabulary. Now what do we compare it to to see if the teacher is doing a good job? Well, we know that the answer is mat. Okay. So this is what we call a hard label, um, and so we're going to compare the teacher's outputs to the ground truth, which would be, we would want the teacher to output mat 100% and everything else, like floor and chair should all be 0%.
06:52 Right? Because we know what the answer actually is. This right here is what we call a hard label. And this is how we go about training the teacher. We feed it a bunch of sentences that we know the answer to, we see what it's outputting and we try and match it as close to the hard label as possible. Once the teacher is trained, the key idea behind distillation is that we don't necessarily need to use these hard labels.
07:28 We might be able to use the teacher itself to generate training data for the student. So now comes the key step in distillation. Instead of training the student on hard labels, like we did for the teacher. We are going to train it on the teacher's soft labels. So what do I mean by this? We are going to take this same data point and we're going to feed it into the student model.
07:58 All right. And the student, again, is also an LLM so it's going to output what it thinks the answer should be. So it thinks that maybe mat is a 40% chance, uh, floor is a 5% chance and chair is a 1% chance. Again, these numbers are completely made up. ut at this point we might notice that, look, the student's performance is not making that much sense, right?
08:18 Mat is the correct answer so it should be a little bit higher. Now the difference is instead of training the student to match the hard label, like we did before with the teacher, we are going to have the student match the labels of the teacher themselves. So, this teacher is already trained, we're going to see what its labels were and we're going to compare the student's output to those labels.
08:48 So, in this case, that would be mat being 60%, floor being 20% and chair being 10%. Okay. Now these are what we call soft labels. So the idea here is that the student isn't just told "Look, the answer is mat." It's literally told, well, the teacher thinks that mat is 60% likely, floor is 20% likely, chair is 10%, and so on. The student is then able to adjust its own parameters to try and match that exact same distribution.
09:33 Now here's the interesting question. Why is this better than just training the student on hard labels directly? Hard labels only say what's right. Soft labels say what's right, what's close and what's completely off target. That extra information, the relationships between all the possible answers, is often what we call dark knowledge. It's called dark because this information is invisible if you only look at the teacher's final top prediction.
10:11 All right. Let's look at a new example to understand dark knowledge in action. The example is going to be "For breakfast, she poured milk into her..." and the goal will be for the model to fill in the next part. Now, let's say we put this into, uh, our model. What could soft labels look like? Soft labels could look like, um, so soft labels from here could look like cereal, maybe 60%.
10:52 We could have coffee 15%, bowl 5% and all the way something down to shoe, which might be 0.001%, for example. Notice that here, cereal and coffee are both pretty reasonable answers so the model will output pretty high probabilities, um, for those two options. Bowl is a little bit unusual, but it's nothing too crazy. Shoe in this case doesn't make any sense, it's pure nonsense.
11:29 Now, if we trained our student model on the only the hard label, it would only know that, for example, cereal is their correct answer. It would never learn that coffee is also a reasonable answer, right? Everything else other than serial would be equally wrong. Now, in contrast, a student trained on these soft labels will learn and understand that serial and coffee are close in this context, while shoe is way off.
11:58 This is exactly dark knowledge in action. The student is learning the shape of the teacher's understanding, not just its final answer. So distillation isn't magic. There's a trade off, as you might expect. The student model is smaller, faster and cheaper to run but it usually won't be quite as capable as the teacher. The goal of distillation is to shrink that gap as much as possible.
12:18 To get most of the teacher's performance in a fraction of the size. Done well, the student can end up surprisingly close to the teacher while being far more practical to deploy. As you can see on the line in front of me, on one end we have very capable models and on the other end we have very efficient models. Our goal is to find a balance between the two.
12:42 So now we can put all of the pieces together. Model distillation is how we transfer knowledge from a large teacher model into a smaller student model. The teacher is trained on hard labels, whereas the student learns from the teacher's soft labels, not just the answer itself. These soft labels carry dark knowledge, the relationships between possible answers, which helps the student learn much more efficiently than it could from just hard labels alone.
13:14 The result is a smaller model that's faster and cheaper to run while still holding on to much of what the teacher knows.