A GPU fails at three users because the KV cache grows with every concurrent conversation and consumes the memory left after model weights and engine overhead; model fit alone does not determine serving capacity.
Serve a self-hosted model to multiple concurrent users on existing GPU hardware by treating GPU memory, especially KV-cache capacity, as the primary scaling constraint. Reduce weight and cache memory usage with quantization, then validate the resulting user capacity and quality against real workloads.
Searchable transcript of Why Your GPU Fails at 3 Users (LLM Inference Isn't a Compute Problem) — DevOps & AI Toolkit (22:09). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by DevOps & AI Toolkit. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:07 I put the model on a GPU. It fit with room to spare. It loaded. It answered instantly. And for about 10 minutes, I looked like a freaking genius. Then the third person asked it something and the answers just stopped coming. The third, not the 300. Nothing else changed. Same GPU, same model, same prompt. The only difference was how many people were talking to it at once.
00:30 Does the model fit is the wrong question. Is the check everybody runs before they deploy. And it tells you nothing at all about how many people you can serve. And the size barely matters here. Whether you're running something small enough to sit on one chip card or something so large it needs a rack of them. The arithmetic is the same shape. And the thing that runs out runs out for the same reason.
00:56 Now, I've said before that self-hosting your own models is a bad idea, and I still think that, but plenty of you are doing it anyway. Air gap environments, data residency rules, models you fine-tune yourself. Those are real reasons. So, if you're going to do it, let's do it properly. Four layers sit between a model and the person waiting for an answer.
01:22 At the bottom, kernels, the attention implementation, everyone benchmarks and nobody actually chooses. Above them, the engine which loads the weights, manages GPU memory, batches requests, and serves an API. Then a control plane that deploys and manages engines, and a gateway that routes traffic across models and replicas. You can stop at the engine it works on your laptop.
01:45 Maybe on that Mac mini you convinced yourself was an investment. For real inference serving real traffic, you need the other two. Today we stay inside the engine on a single question. Single question. How much can one GPU actually hold and how many people does that let you serve? Which engine to run and everything in the layers above it? I'll get to in other videos.
02:10 So here's how we'll do it. We'll work out the arithmetic on paper first. Then we'll put a real model on a real GPU, throw a 100 requests at it, and watch it grind to a halt on camera, and then we'll fix it. By the end, you will know why inference is a memory management problem rather than a compute one. You'll know which line in your engine's startup logs tell you your real user ceiling before a single request arrives.
02:40 And you will watch that ceiling move by roughly a factor of six without touching the hardware or the model. The mental model almost everyone starts with is simple. The model is smaller than the GPU. It fits. Done. Right? That's the mistake. Weights are the easy part. Parameters times bytes per parameter. At 16 bit precision, that's two bytes each. So an 8 billion parameter model needs roughly 16 GB.
03:13 Everyone gets this far, right? Those weights also have to reach the node. Sounds easy, right? And surely you want a capable model, not not at all. So let's say we want to run Kimik3. It's 2.8 trillion parameters. The weights alone are 1.56 terabytes and serving it wants somewhere around 1.68 terabytes of V RAM. No single GPU comes anywhere near that.
03:38 The biggest one Nvidia sells is B300 at 288 GB. So even the flagship needs six of them. Down at H100, size is 24 of them across three nodes plus a suitcase full of large denomination bills. They run 25 to $33,000 each, assuming you can get an allocation at all. Lead times at channel resellers are 36 to 52 weeks. On a single server, you would move those weights once and forget about it.
04:12 Kubernetes doesn't work like that. Pods get rescheduled, nodes get added and removed, and GPU node that scaled to zero comes back completely empty. Everyone, every one of those events moves the whole thing again while you pay GPU prices for machines doing nothing but copying files. Weights are only half of what sits in that memory. To produce the next token, the model looks at every token that came before it.
04:43 Recomputing all that for every single new word would be absurd. So the model keeps those intermediate values around and reuses them. That's KV cache and it lives in the same VR RAM as the weights. And it isn't a fixed cost. It grows with the length of the conversation. And it grows again with every person having one at the same time. 10 users with long context needs 10 times the cache of one.
05:10 That's the number nobody puts in the spreadsheet and is the one that decides how many people a GPU can actually serve. We are not running Kimmy today just to be clear primarily because I don't have that suitcase to spend on this video. Instead, we are running an 8 billion parameter model on a single L4. And the arithmetic from here on is for that one.
05:33 If you haven't met an L4, it's an Nvidia card built for inference rather than training. 24 GB low power and about the cheapest data center GPU you can rent. Google attaches them to these machine types. Amazon sells them too and you can buy the card outright if you would rather do that. Nothing here is specific to one cloud. So let's ask the card what we're actually working with.
05:54 The output is as follows. 23,000 megabytes. Call it 23 GB to work with. And to be clear about which memory we're talking about because this trips people up. That's the GPU's own memory on the card. The node it's plugged into has 31 GB of ordinary RAM as well. And none of that helps. The model has to sit on the card next to the thing doing the maths.
06:17 RAM is only where the weights pass through on the way there, which matters because the two are not priced anything alike. The node without the GPU is about 27 cents an hour. With it 84 cents, same number of processors, same 31 GB of RAM. Adding one card roughly triples the bill and that money buys us 24 gigabytes of the right kind of memory on the cheapest data center GPU you can rent and you can't buy the memory on its own.
06:49 It arrives wielded to a processor you pay for whether you need it or not. Which is why those cues from earlier aren't really about chips anymore. The shortage moved to the memory. We already worked out the weights. 8 billion parameters at two bytes each. So 16 of those 23. The engine wants roughly one more for itself. Whatever is left and it's around 6 GB is all the KV cache gets and those 6 GB are shared by every single person talking to the model at the same time.
07:25 All that is arithmetic though. Let's find out whether the GPU agrees. We'll be using VLM for this. It's an inference engine. It's the one most people reach for today. It's nothing more than the vehicle. Whether it's the right engine for you is is a whole other video. Now, let's put a model on that GPU. Now, we wait for it to come up. And this generally takes a while.
07:48 So, I'll fast forward. Right. Assume every weight from here on is doing the same. Almost all of that weight was two downloads, an 8 GB container image, and then the weights themselves. And for every second of it, we were paying GPU prices for a machine doing nothing but copying files. VLM tells us what the weights actually cost. 15.27 GB. We calculated 16.
08:14 So the the arithmetic holds and it took nearly 2 minutes just to move them onto the card. Now let's ask it something. There you go. It works. Job done. Everyone in the company can start using it, right? But there is one thing worth noticing before we move on. It's a reasoning model. So it thought out loud before it answered. Every one of those thinking tokens sitting in the KV cache exactly like the answer does.
08:42 Now let's see what happens when a hundred of them do ask something. Let's start at the bottom because that's the part people often get wrong. Every single request succeeded. Every single one of them. Nothing errored, nothing timed out, nothing crashed. If you were watching error rates, you'd be looking at a perfectly healthy service. Now, compare it to the single request we sent a minute ago, which came back in seconds.
09:09 Here, the fastest of the hundred took minutes. Minutes. The slowest took several times longer. Same prompt, same model, same hardware. The only thing that changed is how many people asked at once. Now, look at the shape of that distribution. It isn't a cliff. It's a smear and even spread all the way from the quickest response to the slowest which is exactly what the queue looks like from the outside and through output fell to a fraction of a request per second on hardware that felt perfectly good and quick a minute ago
09:45 and before anyone says hey 100 concurrent requests is a lot it isn't it isn't even 100 people I generate tens of concurrent requests on my own running form of agents. So this isn't a company of a hundred. It's more like a handful of developers, each with agents doing their work for them. None of that should have been a surprise. VLM told us exactly what would happen before a single request arrived.
10:12 We just did not look. The KV cache got about 3 GB. Earlier I estimated six. So let's be honest about why I was wrong. Because both reasons matter. We told VLM it may only use 90% of the card. So a couple of gigabytes sit untouched on our own instruction. And the engine's own overhead is bigger than a guest. Over a gigabyte over a gigabyte of activations, four activations and half a gigabyte again for CUDA graphs which leaves room for 20 odd thousand tokens.
10:44 That's the entire conversation memory on this GPU and everybody talking to it shares that one number. How many people that works out to depends entirely on how long their conversations run. Short exchanges, a couple of thousand tokens each and you fit around a dozen. Let them run to the full 8,000 tokens on the that last line and you fit under three.
11:08 Not 2,000, not 200, under three. And that was sitting in the logs before anybody anybody sent a single request. Now, keep in mind that this is a small one. The model we actually wanted at the start was Kimmy, which is around 100 times bigger than what we just deployed. Now, every number on this screen moves with it. So, what do we do about it? We could rent a bigger card and we've seen the hourly rate, right?
11:37 You know how much that thing bloody costs. We could buy one and we've seen both the price and the waiting list. The more interesting question is whether we can serve more people on the hardware we already have. There's one lever that moves that ceiling. It's quantization. Every weight in a model is just a number and every one of them is stored at whatever precision the model was trained.
12:05 Quantization stores them at less. Think of it as rounding. A value kept to many decimal places gets kept to fewer and take takes less room as a result. Concretely, at 16 bits, each weight can be one of about 65,000 distinct values. Drop it to eight bits and it can be one of 256. Drop to four, it's 16. The weights half, then quarter, and every single one of them gets nudged to the nearest value it's still allowed to be.
12:37 Each of those nudges is is tiny. There are 8 billion of them. So this costs something. And we'll come back to exactly what that is once we've seen what it buys. Now here's the same deployment with one thing changed. Two lines, same model, same GPU, same everything else. We are asking for eight bits per weight instead of 16. Same weight as before and for the same reason.
13:01 The thing worth noting is that we are downloading exactly the same weights we downloaded last time. VLM is converting them to eight bits as it loads them so the transfer doesn't get any smaller. Pre-quantized models exist and and would fix that but and that's a different conversation. The weights lost about a third of their size which is roughly what we expected.
13:22 Now look at where that went. Every gigabyte we freed went to the KV cache and nowhere else. KV cache it was the smallest slice on the card. So adding to it has leverage that adding to anything else wouldn't. The cash roughly tripled. The number of tokens it holds roughly tripled and the concurrency figure and prints at startup went up by about the same factor.
13:48 And that's worth stopping on because it inverts why people think they quantize. We didn't shrink the model to make it fit. It already fit comfortably with room to spare. We shrunk it to buy conversation space and because the cash was the smallest slice on the card, cutting the weights by a third bought roughly three times the users. That's the leverage.
14:15 It's also why will it fit is the wrong question to ask about the GPU. Fitting is table stakes. What's left over afterwards is the thing you're actually buying. Same load as before. Nothing else changed. Same 100 requests, same prompt, same card, and the whole lot finished in well under half the time it took before. But look at where that came from because it isn't what people assume.
14:44 Compare the fastest request to the fastest from the previous run. It improved a little. Nothing like the improvement in the total. And that small gain isn't the model running faster. We didn't make the GPU better at arithmetic and a single conversation is limited by exactly that. What we changed is how many conversations fit in memory at once. So even the requests served first are elbowing fewer competitors out of the way.
15:14 More of them run in parallel. The queue drained quicker and the whole batch finished sooner. The work did not speed up. It didn't. The waiting did. Now one honest note before we move on. The concurrency figure VM printed roughly tripled. But the true output we actually measured bent up by less than that. You never collect all of the ceiling past a certain point.
15:37 Certain point the GPU runs out of arithmetic rather than memory and then more cache space stops helping. Memory was our constraint. So fixing it brought a lot. It didn't buy everything. So we tripled our capacity by changing two lines which raises the obvious question what did that cost us? Now that did not cost us much and I want to be careful here rather than dramatic.
16:07 At 8 bits quality loss on standard benchmarks is a fraction fraction of a percent for big models and a percent or two for models this size. for chat summarizing and following instructions that's below the noise floor. You would not find it without a rigorous evaluation. You would not. Where it does show up is multi-step maths, scientific reasoning, and code generation.
16:29 Anywhere small errors compound instead of washing out, which is worth stopping on because if you're watching this, code and agents are probably exactly what you intend to run on it, right? that isn't the forgiving category is the sensitive one and we are running a reasoning model on top of that. So we're on the wrong side of the line twice over. Right?
16:51 Nothing nothing in this demo would reveal that and pretending otherwise would be dishonest. It's exactly why people quantize casually see no difference and get beaten months later on the work that actually mattered. That does not mean don't do it do it. tripling your capacity for a percent or two is a good trade almost every single time. It means measure it against your work rather than trusting somebody's else's benchmark because the tasks you care about are the ones most likely to notice.
17:24 Which leads to a fair question. If it's this cheap, why isn't it on by default? Hardware partly. FP8 arithmetic only exists on other love lens and hopper cards and anything newer. go back one generation to A100 and there are no FP8 units on the die at all. VLM will still take the model. It just stores the weights at 8 bits and unpacks them back to 16 before every multiplication which is which is important being precise about right because it's better news than it sounds.
18:01 You still get the memory back. The weights really are half the size on the card. So the cash still grows and the ceiling still moves. What you don't get is the faster arithmetic and you pay a little for the unpacking but the bigger reason it isn't a default is simpler. It changes what your model outputs not by much as we just went through but it changes it.
18:25 No engine should quietly do that to you without being asked. Now, there's still a nice irony in the hardware, though. The L4, about the cheapest data center GPU you can rent, handles FP8 natively. And 8100, which costs several times more, does not? Which brings us to the question you're probably already asking. If 8 bits nearly tripled our capacity, why stop there?
18:50 Why not four bits or two? Now, four bits probably fine, and it's what a lot of production deployments actually run. With the better methods, it recovers almost all of the original quality and it quarters quarters your weights instead of halfing them. Two bits h that's where it falls apart. Accuracy drops off a clip. Not by a fraction of a percent, but by several.
19:13 And no amount, no amount of clever encoding has fixed that yet. Below four bits, reasoning is the first thing to go. And there's a decent rule of thumb doing the rounds. If you're running agents or tool calling, do not do not go below four. So the useful range is narrower than it looks. 16 down to eight is close to three. 8 to four is a real trade. You should measure below four.
19:39 Just don't. And there's one more lever I've deliberately left until now because is the most on the noise of all of them. Everything everything we just did was about the weights. You can also quantize the cache itself. That's one more flag. And it roughly doubles the amount of tokens the card can hold for very very very little accuracy cost. Which given that the cash is the exact thing we spend this entire video running out of is worth more than a mention.
20:12 Right. Two more lines. Two more lines. And this time they're aimed directly at the thing we're short of. Now look carefully at what did and didn't change. The weights are identical because we did not touch them this time. The cache is still exactly the same number of gigabytes because we didn't give it any more room either. What changed is how much fits inside it.
20:35 The token count doubled and the concurrency figure doubled as well on the same memory because every token now takes half the space it did before. Add it all up and we've gone from under three concurrent conversations to over 16. Same card, same model, four lines of configuration. That's what the engine claims. Now, let's put the same 100 requests through it one more one more time.
21:01 Faster again. It's faster. The whole batch finished quicker than the last run, which itself finished in under half the time of the first. And look at the speed compared to where we started. Instead of a long smear from fast to painfully slow, most requests now land close together. That's a cue that's mostly gone. But be careful reading this one because the numbers don't line up the way you would want.
21:26 We doubled the ceiling and through output went up by less than half. The that gap is the interesting part. We stopped being short of memory. Somewhere between the last change and this one, the constraint moved and now the GPU is running out of arithmetic rather than the room. More cash space keeps buying less because cash space isn't what's holding us back anymore. Thank you for watching. See you in the next one. Cheers.