LLM Inference Explained: 12 Concepts You Actually Need to Know Transcript, AI Summary & Key Points

00:07 continuous botching, page attention, prefix caching, speculative decoding, prefield decode disagregation. If you've been anywhere near a conversation about running your own models lately, you've heard every one of those probably in the same sentence, probably from somebody saying them very quickly. And there's a decent chance you nodded. So this is everything you wanted to know about infinence, but you were afraid to ask.

00:38 We're going through the whole machine in one pass. What an engine actually is, what it's holding on the GPU and every bit of jargon stacked on top of it. 12 ideas, give or take, and a couple of minutes for each. Now, one thing to listen for as we go. Those ideas don't all arrive at once. Some bite the moment you deploy anything at all. Some wait until 50 people are talking to it.

01:05 Some you may genuinely never need. Who knows? Before we get inside the inference engine, there is one problem sitting above it. Agents do not only send prompts to models, they call APIs, query databases, open tickets, and deploy things. Every one of those actions needs credentials and handing the agents a real API key means trusting everything that can influence that agent with the same key.

01:30 That is where the sponsor of this video comes in. In physical, the open-source secrets platform and specifically its agent proxy. Instead of placing a real credential in the agents environment, you give it a placeholder and route its request through the proxy. In physical swap scene, the real secret only when the request leaves, so the agent never ever ever receives the credential itself.

01:55 More than 30 services work out of the box and each call leaves an audit trail giving you one place to see which agent accessed what and when. You can learn more about agent proxy at infysical.com or through the link in the description. Big thanks to Infysical for sponsoring this video. Now, let's get back to the program sitting between a model on a disk and somebody typing a question.

02:20 An inference engine is the program that sits between a model on a disk and somebody typing a question. That's the whole category. Its job list is short and it's worth going through because nearly everything else in this video attaches to one of these lines. It loads the weights onto the GPU. Weights are just the numbers that came out of training, billions on them.

02:42 And we'll come back to why that number matters more than you would expect soon. Separately from those weights, it keeps a running record of every conversation in progress. That record is the expensive part, not the model, the record. It decides which request gets worked on next and how many to work on at the same time. It runs the actual arithmetic and it speaks HTTP.

03:05 So your code can talk to it roughly the same way it talks to OpenAI. So load, remember, schedule, compute, serve. Five things you've used one for sure whether or not you ever run one. If you started OLama on your laptop, that was an engine. If you deployed VLM, the same. And if you never touched either, every request you ever sent to Entropic or OpenAI still landing on an engine at the far end, somebody is running it.

03:38 It just isn't you. Now, they differ and how they differ is a whole other video, but that job description doesn't change. And here's the useful part. Every item on that list has a point where it starts to matter to you. And those points are a long way apart. Some by the moment you deploy anything at all. Some wait until 50 people are talking to it at once.

04:03 Some you may never hit. Working out which is which is most of what this video is for. Now the engine does not work alone though. So before we climb inside it, here's the whole machine in one picture at the bottom. Kernels. Now not the operating system kind just to be sure. These are the little pieces of code that do the actual arithmetic. Everybody benchmarks them and almost nobody chooses them.

04:29 So that's the last you will hear about kernels today. Above them, the engine, the five things we just went through. Above that, a control plane which deploys and manages engines because sooner or later you have more than one. On top, very top, a gateway routing traffic across models and replicas. Now, most of this video lives in the engine part, not because the rest doesn't matter.

04:56 We climb back out of at the end, but because almost every word you'll hear thrown around like patching, paging, prefix catching, quantization lives in there. And none of it makes sense until you know what the engine is trying to do. Everything the engine holds on that card is one of two things and uh two behave nothing alike. The first is the weights.

05:24 A weight we said is a number that comes out of training. There are billions of them and the number takes up room and the precision most models ship at is two bytes each. So the arithmetic isn't complicated. 8 billion parameters, two bytes a piece, 16 GB. And those 16 GB have to sit on the card itself, right next to the thing doing the math, not in the machine's ordinary RAM, on the card.

05:50 Take the Nvidia L4 as an example. It's about the cheapest data center GPU you can rent and 23 GB of memory on the card. 16 of them are gone before anybody has asked a single question. But here's the thing about that 16. It never changes. One user or a thousand users, the weights cost exactly the same. You load them once and you're done thinking about them.

06:18 The second cost is not like that at all. When a model writes a sentence, it looks back at everything that came before it. Every token, every time. Recomputing all of that for every new word would be ridiculous. So the engine keeps those intermediate values and reuses them. That's the KV cache. Same card, same memory, sitting right next to the weights and competing with them for room.

06:47 And it grows in two directions at once. It grows as a conversation gets longer because there's more to remember. And it grows with every extra person having a conversation at the same time because everyone of them needs their own. 10 people deep in long conversations need 10 times the cash of one person in a long conversation right simple math now one cost is fixed the other one multiplies so when does each of them start to matter to you the weights immediately if they don't fit you haven't got the service at all the

07:23 cash later and exactly how much later depends on how many people you have and how long they talk to it that second one is what almost nobody thinks about until they're already past it. A GPU is very good at doing many things at once and fairly ordinary at doing one thing quickly. So the way to get value out of one is to work on several requests in the same pass.

07:49 That's botching and it comes in two flavors that behave nothing alike. The old way is to collect a handful of requests, run them together, wait for all of them to finish, and then pick up the next handful. It works, but if one person asked a yes or no question, and another asked for a thousandword essay, the yes or no finished ages ago, and it seat in that batch has been sitting empty ever since, doing nothing until the essay is done.

08:22 Now the modern way is continuous botching. A request that finishes leaves straight away and a waiting one takes its place. The batch turns over constantly. No seat sits empty and nobody's waiting on a stranger's essay. That's the difference between a queue that drains and a queue that only grows. The other half of serving many people is fitting them.

08:46 Every conversation needs room in that KV cache and you don't know up front how much. So, the obvious approach is to reserve the most a conversation could possibly need per request in advance, which wastes an enormous amount because hardly anyone uses their maximum. You end up with a card that's almost entirely reserved and almost entirely empty at the same time.

09:12 Now, paging is the fix. Instead of one big reservation per conversation, the cash gets handed out in small fixedsiz blocks as and when they're needed. If you ever wondered how your laptop runs more programs than it has memory for, it's the same trick. So when do those start to matter to you? Batching at your second concurrent user. Literally the second one paging as soon as requests vary much in length which in practice means immediately.

09:47 If the weights are the fixed cost the obvious question is whether they have to be that big and they don't. Every weight is a number stored at some precision. Quantization stores all at less. Think of it as rounding. A value kept to a lot of decimal places gets kept to fewer and takes less room as a result. Half the bits and you roughly half the weights.

10:13 On that 16 GB model, that's 8 GB handed back. And here's the part that isn't obvious. The space you get back doesn't go to the model. it goes to the KV cache which is the thing that decides how many people you can serve at the same time. So quantization isn't really about making a model fit. It's about what's left over once it has. It does cost you something.

10:38 It's small and it lands harder on some kinds of work than others. That's no that's a whole video by itself. So we'll skip that. Now when does this start to matter to you? roughly as soon as you're serving anybody at all on hardware you would want to pay for. Before a model can serve anything, it has to read the question, all of it. And the question is usually a great deal bigger than whatever the person actually typed.

11:07 There's a system prompt, there are tool definitions in a chat, there's the entire conversation so far resent from the beginning on every single turn. So, if you're running an agent, every request it sends opens with the same tokens, thousands of them, and in a session that's been going for a while, hundreds of thousands, because all of it, all of it goes back every single time.

11:33 And the engine dutifully works through the whole lot again from scratch as though it had never seen any of it before in its life. Prefix caching is the engine noticing. It hangs on to the work it already did on that opening stretch and when the next request turns up starting the same way, it skips ahead to the part that's generally new. For a chatbot where everybody types something different, this does very little.

12:03 For agents where every request opens with the same enormous wall of context, it's the biggest single lever available. When does it start to matter to you? The moment your requests have something in common at the front, which if you're building anything agentic is every single request you will ever send. When a model answers a question, it's really doing two separate jobs and they stress the hardware in completely different ways.

12:34 The first is reading the input and the input is everything that got sent. the system prompt, the tool definitions, the whole conversation so far, not just the sentence somebody typed. The engine works through all of it in one go in parallel. That's preill. And what limits it is raw arithmetic. The GPU is doing as much maths as it possibly can as fast as it can.

12:57 The second job is writing the answer. And this one is stubbornly sequential. The model produces a token, looks at what it just wrote, produces the next one. It can't go faster by doing more at once because token five doesn't exist until token four does. That's decode. And it isn't limited by arithmetic at all. To produce each token, it has to hold the entire model out of memory and back 16 GB for one token.

13:26 The GPU spends most of the time waiting on memory rather than calculating. So one phase is limited by compute and the other by memory bandwidth. Same request two completely different constraints which is why a long input and a long answer make your GPU struggle in different places and why nearly everything clever in inference is aimed at one or the other.

13:51 Speculative decoding is aimed squarely at decode. If the trouble is that tokens only arrive one at a time, you bring in a second much smaller model to guess the next few, then let the big model check all of those guesses in a single pass. When the guesses are right, and for predictable text, that's often true. You've had several tokens for the price of one.

14:20 So, when do those start to matter to you? Well, prefill and decode. Well, as soon as your inputs get long or vary a lot in size. Speculative decoding when a single request feels slow, which notice is a different question from everything else in this video. It's the one thing here that helps one person rather than 50. Everything so far has been inside one engine serving one model on one GPU which is perfectly fine place to be right up until it isn't because sooner or later there's a second model and a second replica of

15:00 the first one and a version you're testing before it goes anywhere near production. Now somebody somebody has to decide what runs where, what gets updated when and what happens when a node quietly disappears underneath you. That somebody is a control plane. It deploys engines. It manages them and it takes the deployment YAML you've been handwriting away from you.

15:24 And once more then one engine is running and something has to decide which of them any given request goes to. That's a gateway. You'd be forgiven for thinking that that one is solved. We've been load balancing HTTP for what, like 30 years. It isn't solved. And plain round robin is actively wrong for language models. Requests aren't interchangeable. One replica may already have your conversation warm in its cache while another has never seen you.

15:54 And landing on the wrong one costs real money and real time. Now, why that is and what to do about it is is a video on its own. We'll get there. For now, just the word. The gateway routes across models and replicas. So, when do those start to matter to you? Well, later than most of this video, and then all at once, one model and one replica. And you need neither of them.

16:21 The moment there's a second of anything, anything, you need both. Two more and then we're done. The first is what happens when the model doesn't fit on one card at all. We've been talking about 8 million parameters on a 23 GB GPU. The models people actually want are a great deal larger than that. Large enough to need a rack, not a card. So you split the model across several GPUs.

16:49 That's sharding. And there's more than one way to slice it. You can cut every layer across all the cards or you can hand different cards different layers. Tensor parallelism and pipeline parallelism, if you want the words for them, that's that's how they call them. They have very different consequences for how much these cards have to talk to each other.

17:09 And that also is a video whole video on its own. Now, the second is that your traffic isn't constant. Nobody's is. And the GPU is just far too expensive to leave sitting idle all night on the off chance that somebody will use. So you scale on signals particular to this work. How deep the queue is, how full the KV cache is, not CPU usage, which tells you almost nothing here.

17:35 Right now the catch is real though. A GPU node that scales up starts completely empty. It has to pull a container image then waits before it can answer anybody. And by the time it's ready, the spike that triggered it may be long over. That's the cold start wall. And it's why scale to zero is it's much harder than it sounds. Now one last word to leave you with disagregation.

18:03 Reading the input and writing the answer want different things from the hardware. So run them on different machines each matched to what its phase actually needs. It's the most interesting idea in inference at the moment and it needs just about everything in this video to to make sense. So that's a separate video as well. Now when do those start to matter to you?

18:23 Well, sharding when your model won't fit, which is something you work out before you deploy rather than after. And outscaling the first time you look at the GPU bill. That was the machine. 12 ideas, one pass. If you keep one thing, keep the question rather than the list. For every one of these, what matters isn't what it is. It's when it starts to matter to you.

18:50 The weights the moment you deploy. Betching at your second user. Quantization as soon as you're paying for the card. Prefix caching if you're running agents. Sharding when the model won't fit. Autoscaling the first time you read the GPU bill and one or two of them disagregation most likely possibly never. So don't go and implement all of this. Don't do it.

19:14 Work out which two or three you already passed. deal with those and leave the rest alone until you aren't. So where do you go next depends on which ones you landed on. How many people a single GPU can really serve is a video by itself. So is choosing between engines. So is the gateway and scaling and working out what to measure. Now one thing I've left out entirely past a certain size this stops being one cluster and becomes many across regions or providers.

19:43 And that's a different problem with a different set of answers. Further away than most people think and out of scope here. Everything else everything else in this video though you will hit probably in roughly that order. Thank you for watching. See you in the next one. Cheers.