Why RAG Solutions Fail with Complex Documents & Vector Databases Transcript, AI Summary & Key Points

00:00 Humans are complex. Because we are complex, our language is complicated. It's filled with nuance and subtle differences that only our complex brains can decipher. This complexity can be difficult for AI to understand and capture. RAG solutions based on complex documents can lead to confusing or even incorrect answers. In this video, we'll discuss how to build practical solutions when you have a complex set of documents.

00:25 Let's say you ask a friend. Who won the 2020 NCAA football national championship? And they reply, Louisiana State University or the LSU Tigers, of course. Another friend, however, might say that it was the Alabama Crimson Tide. Can they both be right or must one be wrong? The correct answer depends on the intent of the original question. So, when you talk about a football season, it typically lasts from August...

00:57 To December. But when you talk about championships, that game is usually played the following January. This can cause confusion. This is why in common language, people will refer to LSU as both the 2019 and the 2020 champions. In the real world, sets of documents are compiled by multiple people over time. Laws or policies written in the 1990s may contradict with laws or policies written in 2020s.

01:29 People are messy and so is their written history. Documents often contradict themselves just like people do. It is very likely that inside a repository there are documents, for example, that refer to LSU as both the 2019 and the 2020 champion. So what's a RAG solution supposed to do when the document set provides more than one correct answer to a question?

01:53 Let's look at some practical solutions. Let's start with a very simple architecture. So RAG architecture, very simplistic form, is going to start with an end user. This end user is going to ask a question. And that question is going to be sent to. A vector database. Inside the vector database, that question is going to yield search results. And then those search results, and the original question.

02:40 Are then passed to an LLM. That uses the information from the vector database and the original question to answer the question, which is sent back to the original user. So how can you augment this if you're dealing with a complex document set? So number one, you need to make sure you don't make any unforced errors. And by unforced error, I mean specifically that you don't have documents in this vector database that should not be there.

03:16 So let's take, for example, you have a policy that was implemented in 2019, and another policy that was meant to replace the original policy in 2024. You don't want both policies in that vector database, because if you have both of them there, you may confuse your RAG solution. So number one, you need a good document management solution to ensure you don't have any of these unforced errors.

03:44 And everything coming out of the search results really should be there. The other thing you can do is you can use something called a clarification loop. In a clarification loop. Is kind of what it exactly sounds like. It's a mechanism built into the AI solution so that the right question is asked. And you know, this could be something very simple. Like if a user is using a RAG solution and they say, why are banana?

04:21 That's a ridiculous question. So you want your RAG to identify that as a ridiculous and prompt the user to say, hey, you know what, I don't quite understand this. Can you maybe rephrase this? But you can also use clarification loops in your process to make sure the question is specific enough. So for example, if somebody says, who won the championship in 2010?

04:49 There's obviously a lot of championships out there. Is it sports? Is it the spelling bee? I mean, what is it? So that would identify that question as not being specific enough, and it could prompt the user to give more details. So at the end result, the answer is specifically what they're looking for. But probably the most important thing, and this may seem obvious, but you'd be amazed at how often I see this error, is that the AI...

05:22 Has to be less than or equal to your data. So what do I mean by that? So if you ask a question, and according to the documentation in the database, if a human were to read everything in there, they would understand that the answer to that question could be X, Y or Z. There's more than one answer to the question. So you need to make sure that your AI solution is robust enough to understand this complexity.

05:55 So for example, if I ask the question, it didn't come back with just x or it didn't back with just y, it would come back, well, it could be x, y, or z. And where you see this a lot is in policies and laws. It's not uncommon for a law written in 2012 to contradict with one written in 1912. It's not uncommon if you've got some kind of agency that has both federal and state regulations for those regulations to contradict with each with one another.

06:31 You know, again, it depends on what data you have in the database. For example, if you have all you have the database is legal opinions, you don't wanna present those opinions as facts. You want to present them in the context. Of that they should be interpreted with. And what all this means, I think the point here is that you really need to understand the data, the underlying data that you're building your AI solution on before you start to develop that solution.

06:59 Because if you develop a solution, you know, thinking that there's one answer to a lot of questions or to a question, and it turns out there's more than one. If you build a solution, that's... Based on data that's opinion and not fact, and you have that solution yielding information like it is a fact, you're gonna run into problems and people will say it's hallucinating.

07:22 But in reality, it's really not a hallucination, it just is not designed correctly. So tell me, what kind of complex document sets are you currently working with? Let me know in the comments. Thanks so much.