Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales? Transcript, AI Summary & Key Points

00:00 These days, it seems like everyone is looking to run local LLMs. And that's right, you can actually run GPT-style models as AI assistants, code editors, and more from your own personal computer or even a laptop. And there's a lot of reasons why people are looking to do this. First off, LLM's can get quite expensive. So running them yourself can save a lot money.

00:24 At the same time, you might've experienced getting an outage from an AI service or even getting rate-limited. And that can be quite tricky. But the last part and the biggest reason that personally I like to use local LLMs is because the privacy and security that you get running it on your own machine. But when we talk about running local LLMs, you typically will hear about two well-known tools that can allow you to do this on your machine, llama.CPP and another tool called vLLM.

00:54 And both of them target different types of use cases. But don't worry, you can still do RAG or retrieval augmented generation. You can build AI agents, even use them as AI code assistants. But by the end of this video, you'll learn how and why to pick between these two tools. I wanna take us a few years back to the release of a particular model called Llama 2.

01:17 And Llama 2 came from Meta and it was a large language model, but it was very unique because it was one of the first commercially successful open weight model releases. Because, sure, ChatGPT and other services were already around that I could use as an API or pay for, but the Llama model was a model that I can go to, Hugging Face or GitHub and actually download this model.

01:41 And I could save that model and try to run it on my own machine. But there was one small problem. You probably actually couldn't run it in your own hardware. That's because this specific model was released in a few different sizes. So we got 7 billion parameters, 13 billion parameters and the biggest at 70 billion parameters which required quite a hefty setup needing a big graphics card which these days is either expensive to get a hold of or in some cases hard to find.

02:16 So instead of purchasing more hardware accelerators and bigger GPUs that can be quite costly, the idea with ;lama.CPP is to make models accessible on smaller pieces of hardware. Models that you can download from open source repositories and run on your own computer because of a few optimizations. So one of these is quantization. And let me give you an example of how this works because when we think about pi, right?

02:39 It's actually 3.141529 and it goes on. I don't know the rest of it, but the way that we simplify it is 3.14. And the same kind of theory exists when we're talking about large language models because you're computing a lot of different calculations within a GPU and the weights are stored at sometimes higher precision. So for this example, that could be base, float, 16 precision.

03:07 And by quantizing it or essentially compressing it down to a smaller value, we can go to integer eight or integer four. And so we take a much bigger model and get it to a much smaller size. And this can take our capacity requirements for a GPU from, say, for example, 30 gigabytes of storage down to, say for example four gigabytes of VRAM needed to run this model efficiently.

03:37 Now, there's also other optimizations such as combining the models weights and associated metadata into one unit. So let's think of the weights here, right, of a neural network. So these could be like the safe tensors or whatever you might be downloading in order to run this model and combining this with the tokenizer with the models configuration and all of these different files into one singular .gguf file.

04:00 So this allows us to easily swap in different models and be able to try out all of rhese different capabilities from a single .gguf file and so this is a big innovation that llama.CPP brought. And finally, we also have the ability to run inference on not just GPUs, but also CPU as well, which is really important because a lot of personal computers just don't have a GPU.

04:33 And all of these optimizations really paid off because suddenly you could run LLMs on a laptop or a Raspberry Pi and you could runs LLM's offline. So think about factories or IOT environments. And this whole llama.CPP project led to other tools like Ollama and LM Studio, which are well adopted by developers around the world. But sure, running one model locally is great, but what happens when we have different types of models for generating embeddings for RAG or for acting as a coding agent or doing tool usage, right?

05:04 Or maybe we have 10 users that are all using these models or. 10,000, right? Or maybe you're deploying to a high performance computer on a VM or Kubernetes, which is famous for distributing workloads across hardware and regions. So what vLLM does is takes the local LLM idea to another level. So it focuses on efficiency at scale with hardware accelerators like Nvidia's GPUs, Google's TPUs, you've got AMD, you got Intel, and whatever else it might be.

05:37 To have this very broad hardware stack that you can use to run these models with. And in the middle, just like llama.CPP, vLLM acts as an LLM engine. And so it allows you to run models to be able to process text, to be to do audio, to be be able do images and video. So all of these different types of model formats from almost every leading model manufacturer out there.

06:05 So all the open source labs. Typically come with this day one support for an engine like vLLM, but not only do we have this broad set of hardware compatibility and model compatibility, but there's a lot of improvements in vLLM in order to handle, for example, batching. So what does batching look like? Well, let's say we have those 10 requests that come in continuously per user, right?

06:30 And instead of showing you the actual request, I'm going to show you pancakes on a griddle because let's say we're cooking six of these but they're going to be finished at different times. So maybe this one is done, this one has done, this one was done. And I don't wanna wait if I'm having to cook a lot of pancakes for all of these to be finished before I put on new pancakes on the grill.

06:52 So that's an example of what vLLM calls continuous batching. Now there's also other optimizations for efficient KV cache usage. So let me give you an example. So when a user's request comes into an LLM, what actually happens is we take those input tokens and run them through each of the layers of the large language model. And as this happens, we're generating more text token by token, but at the same time accumulating what's known as KV cache because of these calculations that we're doing on the original source text.

07:30 And trust me, this can get quite large. We typically see, you know, dozens of gigabytes being used for the passage of prompts through a model to finally get to a response that is the desired output. And the thing is, with hardware accelerators only having limited amounts of space. Well, if we take a A100 GPU, for example, from NVIDIA, we actually split this up into a few different sections.

07:58 The majority is typically going to be the model weights itself being loaded into memory. But the vast majority of the rest of the model is actually this KV cache, so that if we have the same user make the same request multiple times, we don't have to recalculate that entire prompt itself. So there is a lot of optimizations with vLLM on proper caching with a technique known as paged attention in order to make this happen.

08:28 But the thing is, although llama.CPP and vLLM greatly benefit from quantization, there's another benefit that vLLM, also brings in, which is known as speculators. So let me give you an example. Let's say we have a smaller model, which performs quite well, generate a few different parts of a response, and we want to use a larger model to verify that that response is exactly what we want to give back to the user.

08:59 This is what's known as using speculators. And it can also be combined with a technique known as disaggregation with a open source project called LLM-D in order to split up the multiple stages of pre-fill and decode when you're using an AI model for inference. As an end user building RAG or AI agents, don't worry, because both of these AI agents can serve models like DeepSeek or Quinn or Llama, even multimodal models using open AI compatible endpoints.

09:33 So you don't have to change much of the configuration in your actual code base because you're still using the same completions or responses API, which is really nice. It's essentially a drop in replacement for a ChatGPT style APIs. And typically what folks will do is they start with a paid API for easy testing, see those bills start to rise and then switch over to either VLLM or llama.CPP on a bigger environment.

09:58 The thing is they're made for different use cases because for llama.CPP you're typically working with consumer hardware, and with vLLM, you're going to be running this in production workloads, but the goal remains the same running your own AI locally. Now, I'm curious, have you tried or you're currently trying a local AI model? Let me know in the comment section what's worked for you.

10:20 And if you learned something today, please be sure to smash that like button to hack the algorithm. And as always, thank you for watching. We'll see you in the next one.