LLM & AI Agent Benchmarks vs Reality: Why AI Applications Break Transcript, AI Summary & Key Points

00:00 So your large language model, it scored really high on a leaderboard, right? Number one, LLM for some particular task, right. And that's great. But the thing is, when you're actually using that LLM in your AI application or your agent, maybe users are waiting a really long time to get back their answers. And the thing, is maybe it's correct, or maybe half those answers are wrong.

00:23 So what happened here? Well, the leaderboard here tested one thing, but your production environment tested. Everything else and that gap between a benchmark score and the real world performance is exactly what we're going to be breaking down in today's video. So whether you're building an application like a chat bot that's answering customer service requests or RAG which is answering questions from PDFs and your other types of data by adding it to your prompt, or maybe code assistants helping you write code or just

00:51 building agents that can interact with your services. The thing is, no matter what you're building, you're always going to be balancing for three different things. And I can kind of visualize this in a triangle here, where at the top you've got accuracy. So how accurate is the model? Is it giving you those right responses? And then secondly, what's the performance of this model, right?

01:13 How long are users waiting to get back response? And then sometimes a toughy conversation is, hey, how much is this costing us? And the thing is you can typically optimize for two of these different qualities in your agent or application. But the third one is going to end up suffering a little bit. So let's say we have a huge accurate model, and we want to run it at a super performative rate, right?

01:38 To be able to handle hundreds or thousands of users. Well, that's going to ended up costing us a little. But we might have a cheap model over here that is super performant. And the thing is, while it might give us quick answers, they might not be as accurate. And so. Benchmarks and evaluations are how you figure out where your system is actually going to land on this triangle here.

02:01 And you want to figure that out before your users do it for you. Now, there are two main types of benchmarks that we're going to be taking a look at. And the first is model evaluation. So how smart is this model? Can it reason? Is it accurate? And, you know, is it able to answer the questions for our specific domain? And then secondly, we have to take a look at system evaluation.

02:25 So how fast is this? Is it able to handle the number of users that we want to be able to scale? And what does the cost per request look like? And so you need to have an understanding of both of these two types. But first, we're going to start with model evaluations. At the most basic level, model evaluation is when you give the model some type of question.

02:46 And you grade the LLM's response to be able to determine if that's correct or incorrect. And one of the most well-known benchmarks is the MMLU or the Massive Multi-Task Language Understanding, or in some cases nowadays, the multimodal MMMU. Now, you can think of it as a standardized test where you're grading the model's responses in various types of domains, like say for example, organic chemistry, or say for example, world history.

03:17 And the answers that the model gives are then compared to known correct ones. And we're able to use this as a reference based benchmark because we know what the answers will be. But for other domains, for example coding, we're able to use other types of domain based benchmarks like the SW bench or the terminal bench. And as the name implies, these make the model perform like a software engineer that's doing coding or using the terminal.

03:44 And for some cases, we can imagine the situation where the model is assigned some type of bug ticket from maybe a Git repository. And we're then using the model to explore the code base, see if it can find the correct files to be able to implement some type of code fix or modify the code. And then run test in order to see if that answer is solved or didn't solve the problem that initially occurred.

04:11 And this is what's known as an execution-based benchmark. But a lot of real-world use cases don't have one right answer or a test SWE that we can run against. Because if someone asks a customer service question, there could be a hundred valid ways to respond correctly to that question. So you also need a reference-free evaluation. Being able to judge things like, is the tone appropriate?

04:34 Is it actually helpful? And did the model hallucinate any details about our company or our products? And that's where something known as LLM as a judge comes in, where you use a powerful model to critique the outputs of the system you're testing. And by critique, we really mean this. So let's say this is our LLM or our AI application here, where we have the user's input right here and the model's response.

04:59 And what we do is we combine those two outputs together, so the input plus that response. And send that to this evaluating model, so the LLM as a judge. And this one uses a rubric that it's been trained on to be able to validate if the answer has been hallucinated or if it's correct and answers the user's original question. And the one catch here is that you can't blindly trust the LLM here in this situation to make that call.

05:31 Here's where you need humans in the loop in order to evaluate if those. Scores match what a domain expert would say. And so we use experts in the field to be able to label data, and then that LLM can scale up what those good responses and bad responses look like, so that we're able to evaluate hundreds or thousands of different outputs from a model by being able to do this using an LLM.

05:57 So these are just several examples of evaluations that we see in the field right now. This in addition to agentic evaluations or language evaluations for things other than English, are commonly used to determine, hey, is the model performing well? And is the the model right for this type of domain? Now, let's talk about the other half, which is system evaluation and being able to understand the performance of our AI application or agent.

06:23 Because your model could ace every accuracy benchmark, but if it can't handle real traffic, then none of this matters. Now, when we benchmark inference performance, where Inference is the process where you give input to the model and the LLM generates an output like Red Hat is a software company, blah, blah blah. Well, there are a few key metrics. First off is the time to first token.

06:49 So how long does it take for us to get our first response back from the model? How long is the user waiting to see that first word? Now there's also metrics that we wanna take a look at such as inner token latency. So how fast do the rest of the tokens stream in after the first one? And there's also request latency. So how long does it take in total to get that response back and being able to measure throughput?

07:17 So how many of these requests can we handle one at once? Could it be five? Could be 1000? That's what throughput is known as. Now, why do these matter? And why do we measure them separately? Well, that's because inference happens in two distinct phases. So the first one is called pre-fill, where the model processes your entire prompt at once. That is what's compute heavy.

07:42 Then there's the stage of decode, where the module is generating tokens one by one. And that's what's memory heavy. And the reason why this is important is that these phases have completely different bottlenecks, which means that the shape of your workload changes everything. So let's think about typical workloads. So I'll give you a couple examples of applications that you're trying perhaps to build or using today.

08:09 So let say for example, a chat application, right? Customer service, chat bot responses. You might have inputs of say for examples 128 tokens and maybe 1098 output tokens. And let's say for a RAG example, working with our data, well, we're pulling in huge parts of chunks of documents. And for this situation, maybe it's 4,000 tokens and an output of 512 output tokens.

08:38 And then finally, a lot of us are using AI coding assistants and agents. So agents might be calling tools, getting back 512 tokens and now putting 512 tokens as well. The thing is, if you're benchmarking with the wrong token distribution, you're going to see radically different results when it comes to the performance of your AI application. And so that's why it's important to set up service level objectives or SLOs.

09:06 And what these allow you to do is measure and define targets. Say for example, for a chat application, we want 99% of requests, which is known as the 99th percentile here. To be under, say for example, 300 milliseconds. And so by understanding what type of request I'm going to have to my AI application and the output I'll get back, I'm able to benchmark against that.

09:33 Tools in the open source ecosystem allow you to simulate realistic types of traffic patterns. So for example let's say we have thousands of documents that need to be summarized. Well, this is really predictable. This is what's known as offline inference. But in situations like code assistance and agents, we do something different, which is known as online inference.

09:52 So requests might come in one at a time, but in many situations, they'll be concurrent or at different intervals. And being able to understand what types of traffic patterns you may see in your application and the different workload types allow you to optimize for these different metrics and benchmark this for your end user, which is really important when you scale up your application.

10:13 So what you're looking for is the inflection point. And let me visualize this for you in a chart like this with the time to first token here. And remember, as this rises, users will get a little bit iffy and the requests coming into the system. And at low traffic, the latency, it stays pretty flat, right? But as you push more and more requests through, eventually you're going to hit this inflection point and this is going to spike.

10:38 And so the sweet spot, which has maximum throughput while staying inside of our latency targets right around here. Is going to be the number that we can use for capacity planning. Now, everything we've talked about so far applies to a single model doing a single task, but agents are a little bit different because an agent isn't just one model call, it's a chain of decisions, right?

11:00 Starting from the user's input or request going to our agent, right, over to some type of orchestrator, right in order to be able to decide if we need maybe some type of routing to a tool call here. Or maybe routing to get information retrieved from a database, or maybe do some type of code execution and pull those inputs together and reason on that in order to make a safe and informed decision.

11:31 But the thing is we can't just evaluate this final output, right? Because all of these steps previously need to be checked, right. Did we understand, say for example, the user's intent correctly, right,? Or did we pick the right tool when we decided to call this MCP server? Or did that retrieval through our vector database actually return the right relevant context in order to generate this response?

11:55 And each link in this chain of decisions in an agent is a potential failure point. So each one needs its own evaluation. You can think of it as a pyramid, right? So in this pyramid in order to evaluate our agents, we've got the base system, right? This is what we just talked about, system performance. Is our system fast enough? Can it scale? Are we able to handle multiple users?

12:20 Next is formatting. Are we formatting our responses correctly in the way that our application and end user expects? Because a system that's not giving that correctly isn't useful at all. And then is next, safety and bias. Say for example, our final output. Did it contain any private user information? Was there any identifying information that we could get in trouble for?

12:43 So being able to check this with safeguards and guardrails is really important. Whatever that means for your specific use case. Next up is factual accuracy. So being to cut down on hallucinations by say, for example, using tool calls, grounding in real world data. And then finally are these domain specific evaluations. And the mistake that a lot of teams make is actually just starting from the top of the pyramid.

13:06 But if your system can't handle the load, if you don't have these evaluations throughout the entire agentic process, well, then everything's gonna collapse on itself. So to bring it all together, benchmarking AI systems isn't just one thing. It's model evaluation and system evaluation working together. So on the model side, you've got standardized benchmarks that can tell you accuracy in different types of domains, right?

13:30 So maybe just general understanding. Or say, for example, applied AI encoding or agentic use. And don't forget, we also have the LLM as a judge that with the help of a human evaluator and annotator, we can scale up the responses and evaluations to different types of open-ended responses. Now, on the other side with the system evaluation, we're looking at things such as the latency, so how long a user is waiting to get their response back.

13:59 But also the throughput. So how many users can we serve at one time? And what's the cost going to look like when we do this at scale, whether we're using an API or we're running our own models. And when we talk about agents, we bring all of this in and layer it into every step of the agentic flow. So from the user's input here to when we're orchestrating and calling tools to being able to perform safety checks before giving the final response.

14:26 Well, we can pull in all of these fundamentals in order to evaluate each step of the process, and the loop continues. But the key takeaway here is that a leaderboard score for a model's performance in a certain area is just a starting point. It's not the finish line, because real benchmarking happens when you test with your data, with your realistic traffic patterns, and your own definition of success.