What Is Quantization? How AI Models Get Smaller & Faster Transcript, AI Summary & Key Points

00:00 Quantization is everywhere from digital communication to signal processing to the edge devices that you use every day. In our modern digital world, quantization is key. Quantization is the process of taking numerical information and storing it at a lower precision in order to save space or computation. Most of you may have heard the term in the context of AI, specifically around quantizing LLMs or large language models.

00:32 Today, we're going to take a deep dive into LLM compression via quantization. Quantization is one of the simplest, most effective ways to shrink a model's footprint without losing much accuracy. And it's used almost everywhere in production AI today, from the models running on your phone to huge inference clusters. A model as we know it is just a huge collection of weights and activations or parameters stored together.

00:57 You can think of these each as little dials tuned during training to represent knowledge. But knowledge is complex. So models have tons of these parameters. Gymaphor for example has a dense model with over 30 billion parameters. Llama 3 has a model with 70 billion parameters and frontier models are estimated to have trillions. Every parameter has to be stored somewhere.

01:23 And how much detail you store for each determines how much space, memory, and compute the whole model needs. So let's do the math on a 70 billion parameter model to see why quantization is so important. Most models store information in FP16 or BF16 format. This means that they allocate 16 bits or two bytes to each parameter. Multiply that across 70 billion parameters and it would take about 140 GB to store.

02:08 This is more memory than a lot of single data center GPUs even have. So we need to shrink how much space each parameter takes up. The problem is you can't just chop a number in half and call it lower precision. These numbers aren't stored as a simple string of digits that make up a value. They're split into separate parts that each encode information.

02:26 Something like the sign, the size, or the detail of the number. If you cut off the wrong bits, you could end up with a completely different number. We need an actual process for converting a value down to a lower precision and back again consistently. So let's use a simple example to see how quantization works under the hood. A model is made up of layers and each layer contains a set amount of these weights.

03:02 Quantization like inference happens layer by layer. For each layer, we'll look at the full range of values present. Let's say that this first layer holds values from 0 to 100. We'll map that entire range down to a smaller set of values that fits in fewer bits. Let's say here we want to store numbers from 0 to 10. We'll scale down any value from our original range and round it to the closest one in that smaller set.

03:40 For example, let's take an original value of 53 right here. Scaling it down to our smaller size that would be 5.3 and then rounded to the nearest representation would end up being five. Done well, this mapping will hold on to as much of the original range as possible and minimize the information lost. This is definitely an oversimplification of the values that would be manipulated in a real model, but it's essentially the idea.

04:12 In real life, this smaller set of values maps on to a few standard formats. Like we saw earlier, FP16 is 16 bits or two bytes. FP8 and int8 are, as you could probably guess, eight bits or one byte, and int 4 is four bits or half a byte. Going back to this 70 billion parameter model from earlier at full FP16 it will take 140 GB at FP8 or int8 it would take closer to 70 gigabytes.

05:04 And down here at end four, it would only take 35 gigabytes to store all of these weights. This is much more manageable. You'll often see the precision in the model names. For example, something like this means the model has 8 billion parameters quantized to FB8 or floating point 8, or this means that the model weights are at 8 bit precision while the activations are still at 16 bit.

05:33 But quantization giveth and taketh away. As we reduce the precision of the weights, we reduce the memory footprint, but also the level of detail of information stored. It's kind of like a photo becoming more and more pixelated until it's hard to decipher the original image. Too much quantization and accuracy starts to drop. In IN int 8 quantization, most models hold on to their accuracy and show significant improvements in latency.

06:03 IN4 is a bigger cut, so accuracy loss becomes a real risk. With well-built mapping, however, INT4 can still land close to the original model's accuracy. It just takes more careful rounding to get there. And to be clear, when I say real risk, I'm talking in the order of tenths of percentages to 1%. Typically, accuracy stays above 90% of the original model's accuracy.

06:25 There's also two different points where quantization can come into play. First, post-training quantization or PTQ. PTQ takes a model that's already fully trained and compresses it afterward. It's fast and it's the more common approach. The other side is quantization-aware training or QAT. QAT simulates the lower precision while the model still training.

07:00 So the model adjusts for that precision loss as it learns. QAT costs more time in compute, but it holds up better at more aggressive quantization levels like INT 4, where PTQ is more likely to lose accuracy. So to recap, quantization takes the numbers a model is built from and stores them with less precision layer by layer down to a level like in8 or in4.

07:28 It can happen after training with PTQ or during training with QAT. Other why, either way, you're trading some precision for memory and speed. And how far you push that trade depends on what your model needs to hold on to. Drop your questions in the comments or let me know what precision you're running your models at in production.