How AI Models Scale Beyond a Single GPU Across LLM Workloads Transcript, AI Summary & Key Points

00:00 The largest AI models today can have over a trillion parameters, which would take something like two terabytes of memory to store. But the largest GPU on the market holds about 288GB, which is nowhere near enough to store these huge frontier models. So how are we running these models behind the chatbots everyone uses reliably for millions of people?

00:24 The truth is, there is no single mega computer hosting the AI models you talk to every day. They're running on a coordinated system of dozens, sometimes hundreds of GPUs. Models need to be stored, run and support real user traffic in production. Confining this whole process to one GPU quickly introduces real constraints, bottlenecks and latency to your system.

00:48 So let's fix this with distributed inference. The process of running an AI model by splitting the computational load across multiple interconnected devices or servers. A model is composed of layers, all storing "weights", or the knowledge of the model. To run a model, all the weights need to be initially stored in memory. On top of that, as the model runs, it accrues what we call working memory.

01:13 Additional information about the conversation that allows it to remember what was previously said. This context, which we call the KV cache, is held in memory as well on top of the model itself. And lastly, every request takes a set amount of time to move through a GPU. If there are too many requests coming in at once, they get backed up in a queue and every user waits longer for a response.

01:37 If we're hosting this for real users, we might have hundreds or thousands of people trying to reach the model at once. So we need to be able to concurrently serve many users. Serving a model in production means solving these three constraints at the same time. Model memory footprint. Whether the weights physically fit on the hardware, you have, KV cache growth, the content that keeps growing while you're serving and requests throughput, how many concurrent users you can serve.

02:04 So let's start small. Let's say our model and its KV cache fits fine on one GPU, but we still have too many users to serve them all at once. What we can do is copy the entire model onto multiple GPUs, and then route request to them intelligently based on things like how loaded each GPU currently is, or which replica already has relevant cached context.

02:29 This is data parallelism. Every replica is a full identical copy of the model. There's no coordination needed between them. You're just deciding which door each request walks through. But realistically, many of the smartest models can't fit on a single GPU. Recall that a model is really just a series of layers holding individual knowledge. Running inference means passing your input through these layers one at a time, kind of like an assembly line.

02:58 So if our whole model is too big to fit, we can split or shard the model by layers. Say layers one and two live on the first GPU, layers three and four live on the second and so on. Real models will have a lot more layers, but you guys get the idea. This is pipeline parallelism. Every GPU holds only certain layers of the model that can comfortably fit in its memory.

03:25 Now, if we send only one request through at a time, most of the GPUs sit idle, waiting for their turn to take full advantage of pipeline parallelism. Think assembly line. Keep requests continuously flowing through so every station stays busy instead of waiting on the one before it. We split the model vertically Now let's look at splitting it horizontally.

03:42 Tensor parallelism divides each individual layer itself rather than dividing the model by layers. Every GPU gets a slice of the same layer, computes its piece of the layer's math, and then communicates with the others to combine their partial results back together. They then repeat that process for every single layer. That combining step is called a collective operation and its blocking for each layer because the GPUs have to constantly communicate mid computation.

04:12 This only works when they're connected by a high bandwidth, low latency GPU interconnect, typically sitting inside the same server. Sending that chatter across a slower network could actually cause us to become a bottleneck rather than an optimization. There's also a more specialized case, a type of model called a mixture of experts or MoE model. This model, instead of holding one block of weights that runs for every single input, holds smaller subnetworks or experts that are meant to specialize in different parts of

04:51 response generation. One could lean towards code samples. One could do punctuation. Maybe one does numbers because you've got so many subnetworks bundled together. MoE models tend to be humongous overall. Expert parallelism spreads those experts across GPUs, so they each hold a slice of experts instead of a copy of the entire model. At every MoE layer, each token gets dispatched to the handful of experts its router picked.

05:23 Modern models will pick about eight experts out of 256, and then it'll all be combined back. This means far less computation per token, but paid for in all this GPU to GPU network traffic of dispatching the individual tokens and then combining the information afterward. Let's look at one last optimization. Prefilled decode disaggregation. During inference, you'll go through two main stages.

05:48 Prefill and decode. And Prefill is where the model reads and understands your input. It operates on a fixed set of model weights and requires a lot of parallel computations at once. Its compute-bound, meaning the speed of this step is determined by how fast it can perform that math. This step builds the KV cache, our bank of working memory with each token it processes.

06:23 Decode is where the model builds its response to the user. It generates a reply one token at a time, and it has to pull the entire KV cache for every step. Unlike these models weights, the KV cache is not fixed, so the demand for memory grows with every step of decode. Decode is memory-bandwidth-bound, meaning its speed is determined by how fast the data can be transferred from the memory.

06:50 If co-located the decode phase's, increasing demand for memory can start to eat into the prefilled stage's territory as this KV cache grows. Because these two phases stress completely different parts of the hardware, it's actually more efficient to split them onto their own separate pools of GPUs, each one individually optimized for its own bottleneck.

07:11 That said, this split only pays off if this handoff itself is super fast. The whole KV cache has to move from the prefill pool to the decode pool. So if the connection between the two pools is too slow, you'll burn through the time you just saved. In practice, that means the two pools need to sit on dedicated, low latency networking designed for high throughput data movement between compute nodes, not just a regular network link.

07:42 Plain TCP over standard Ethernet usually isn't fast enough to make this split worth it, so traffic gets solved with more replicas. Too much model gets solved by splitting the weights by layer, by tensor, or by expert and too much growing memory doing two different kinds of work gets solved by giving each phase its own dedicated pool of hardware. In production, real deployments layer multiple techniques together at once.

08:05 Tensor parallelism within a server. Pipeline parallelism across servers. Data parallelism across replicas. Expert parallelism for mixture of expert models and disaggregation, splitting the serving pipeline by phase. This is called multi-dimensional parallelism. And on top of it all sits an orchestration layer. Routing every request to the right pool of GPUs.

08:32 Balancing load and handling hardware failures without dropping requests. This is a super complex topic, so drop any follow up questions in the comments or let me know how you're optimizing your model deployments with distributed inference.