CAG vs Long Context: How AI Models Use and Remember Information Transcript, AI Summary & Key Points

00:00 Long context and cache augmented generation are two ways to give a large language model access to external knowledge and they actually build on each other in a way that's worth understanding. So an LLM, a large-language model, it only knows what was in its training data. So if it needs to reason over some data that's in a private document or maybe it needs to take a look at quarters financials, well it needs a way to access that information at inference time.

00:37 Now most people have heard of RAG, Retrieval Augmented Generation. My goodness, I feel like we've covered RAG a few times before. Now RAG solves this problem with a retrieval pipeline, meaning using things like vector databases and embedding models. But CAG and long context are two other ways to provide an AI model with this external knowledge. So how do they work?

01:07 Well, it doesn't get much more simple than long context, which is basically to say, just stuff everything into the context window. Send it into the prompt, send it to the AI model that way. Now that's simple, but will everything fit? Well, context windows have been growing pretty fast. So let me illustrate that with a quick diagram. So on this axis here, we're gonna put the size of the context window and then this axis there, this will just be time.

01:41 Okay, so let's kind of map this out. So if we start in 2020, GPT-3, that could handle thousand tokens. That's maybe 10 pages of text. If we go forward to 2023, well that's where GPT-4 Turbo pushed that quite significantly to 128,000 tokens, which is roughly 300 pages. Then by 2024 Google Gemini 1.5 Pro came along. What do we have here? Two million tokens, that's maybe 20 full length novels, and the trend is still climbing.

02:32 So the strategy becomes, if the context window is big enough, then just skip the retrieval pipeline entirely. So we just get all of our stuff, we get all our documents that we want to provide as external knowledge, and they go into the prompt along with the query that we want to send to the large language model and it's all stored in this context window.

03:00 We let the AI model just read the entire thing because we've got such a big context window to go from now and then a response comes out from that model. Now for certain workloads this works really really well. The model has access to everything so there's no risk of the- retrieval step, pulling the wrong documents, or missing something relevant, like rag might do.

03:26 But there are costs. Literal costs. Because pricing for most LLM APIs scales with token counts. So if this is 200,000 tokens of context, you're going to have to pay that on every single query, and that can add up pretty fast, as does latency. A large context can more processing time per response. Then there's a subtler problem as well, LLMs have what's called the lost in the middle effect.

03:55 So performance at the beginning of a context window, well that generally is is pretty strong, and then information at the end can generally be retrieved as well, but the information buried in the Middle of the long context window the accuracy can drop significantly. The model attends to the edges better than it does the center. But maybe the biggest issue is that every query reprocesses all of these documents from scratch.

04:28 So 10 questions about the same set of documents mean the model reads these documents 10 times. Which raises a pretty obvious question. What if the model could read the documents just once and then remember them? Well, that's the idea behind Cache Augmented Generation, or CAG, and the key to understanding it is something called Key Value Cache. So Key Value When a large language model processes text, each layer of the transformer computes what are called key and value matrices.

05:11 And these are basically the model's working memory and they represent how the model has understood and encoded everything it's read so far. Now, normally these get computed fresh on every single request, but Cag says, do it once and reuse the result. Now we can think about how CAG works in three different phases. And phase number one, that's knowledge preparation.

05:40 So the relevant documents, they get created and formatted to fit within the model's context window. And those documents can be whatever you like, company policies, product documentation, whatever the knowledge base is. Then in phase two, we move on to pre-computation, which is to say, the model processes all of those documents and generates the KV cache, which is the internal representation of everything it's read.

06:07 And then that cache gets saved somewhere. So it's persisted. So to disk or to memory. And then phase number three, that's actually the inference phrase. So what happens here is a, you know, a query comes in and it's sent into an AI model to process that query. Then instead of rereading all of the documents, the model just loads this pre-computed KV cache, it appends the new question and then it generates the answer coming out.

06:43 So the heavy computation already happened in phase two, so phase three here is going to be a lot faster. And in fact Research has showed something like a 10x speedup. When we use small data sets in the KV cache and even more with larger ones, something like 40X compared to reprocessing the full context every time. Now, CAG does have limitations and the obvious one is that the entire knowledge base still has to fit within the context window and when the source documents change, well, the entire KVCache has to be

07:18 recomputed. So if the data changes frequently. The cost of constantly re-caching starts in, it starts to eat into any latency savings that you are making. So really, CAG works best when the knowledge base here that it's using is stable. So the difference between long context and CAG, well, it comes down to when the computation happens. So with long context, the model has to process every document that we've put into the context window.

07:54 It needs to do it. Every time on every query that comes in. I mean, it's simple to set up, there's nothing to manage, but the cost and the latency hit come on every inference. With CAG, the model processes all of those same documents, but it only does it once during pre-computation. It saves that KV cache, and then each subsequent query just loads the cache and answers the question.

08:24 Now the first query is just as expensive as long context, but the second query, the tenth query, the hundredth query, that's where tag starts paying off. Now, long context is also a really good fit for a specific type of query, and that is. Queries where we just call something one time, so analyzing a single document, maybe answering a couple of questions about it.

08:51 There's no point pre-computing a cache for something that's only going to be queried one time. Whereas CAG is much better if those requests happen over and over. So repeated queries are really where CAG shines against a stable knowledge base, so an HR chat bot answering questions about company policies, for example, because the knowledge doesn't change very often and the cache stays valid.

09:18 So every query after the first one is gonna be fast and cheap. Now, there is one more piece to this that makes CAG a practical thing to use in the real world. That is called prompt caching. Now major LLM providers now offer prompt cashing as a feature of their APIs. And the idea is essentially CAG as a service which is not a real acronym but that's what it is.

09:46 So you send a long system prompt containing all the documents and the provider handles the KEV cash management behind the scenes. The subsequent requests that share the same prompt prefix they skip the document processing entirely, and the the economics of this are pretty significant because cash reads can come at a big discounts, something like a 90% discount compared to processing those tokens fresh.

10:15 So prompt caching takes what was an interesting research idea, which is CAG, and it turns it into something any developer can use without having to manage the cache infrastructure themselves. So an entire video about giving models external knowledge, and we didn't build a single retrieval pipeline using RAG. Whether RAG is still needed after all of this? Well, that's literally the title of another video on this channel. Go check that one out.