AI agents need RAG plus observability because each covers a different gap: RAG grounds the agent in documented system knowledge, while observability adds live metrics, logs and traces showing what the environment is doing now — together they let the agent move an investigation from 'something is wrong' to 'here is the evidence, here is what may be affected, and here is where to look next'.
Searchable transcript of Why AI Agents Need RAG + Observability — IBM Technology (09:04). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by IBM Technology. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:00 An AI agent can read every runbook, support document, support ticket and architectural diagram for a production environment, and still be unable to answer one critical question. What is the system doing right now? That is the gap I wanna talk about. And a lot of the conversation around AI agents begins with knowledge. And that makes sense. If an agent is going to help in a production environment, it needs access to validated operational knowledge.
00:26 But operations teams also rely on live evidence. What is happening now and what has changed? And what is connected to these two issues? So, the idea of this board is simple. An operation-focused AI agent needs two kinds of context. System knowledge and live operational context. The first kind is system knowledge. This is where retrieval augmented generation helps.
00:51 Or RAG for short. RAG retrieves relevant information, documentation, runbooks and architectural knowledge from the system. And it uses this to help the model when it responds. That grounds the answer and validates the internal sources instead of purely relying only on the general model training. This can help the agent answer useful questions. How is the application supposed to work?
01:18 Which system does it depend on? What procedures should the operator follow? And have we seen a similar symptom to this before? I think this is as the how it should work side of the picture. It gives the agent memory about the environment, but a runbook may have been written months ago. An architectural diagram usually shows how the design was expected to be.
01:41 And those sources do not necessarily describe what is happening at this exact moment. The second kind is live operational context. And this is where observability comes in. Observability to provide signals, such as metrics, logs and other operational insights, such as topology. That... These are the signals that answer different sets of questions. What is happening now?
02:10 Where the behavior has changed? Which components are involved? And did anything change shortly before the symptom appeared? Metrics can show the response time, the throughput or resource that has shifted. Logs, on the other hand, and events can provide evidence. What occurred. Traces can show where the request spends most of this time as it crosses to the services.
02:33 Topology, this can be how all the services are connected together. Change information can tell us what was deployed or what was reconfigured. The point is not simply to give the agent more data. More data can create more noise. The value comes from giving it relevant signals, and the relationships amongst them. So, these two boxes are complimentary.
02:56 RAG helps the agent to understand the documented system as it is known. Observability helps it to understand the environment and what it's doing right now. Now, let's make this concrete with a hybrid transaction. Imagine that we have a user. And this user performs a balance inquiry on his mobile banking application. To the user, this is one action. But behind the scenes, this request may enter a distributed application, it may move through an API layer, it may also then move through a messaging queue, and then
03:23 eventually arrive here, on a mainframe application system. Operationally, this is a chain of connected systems. Potentially owned by different teams and observed through different tools. The user experiences this as one single service, regardless of the underlining system boundaries. Suppose the transaction has become much slower than normal. If the agent only has system documentation, it can explain the intended path, it can maybe identify the applications involved and even suggest a standard checklist to look at.
04:03 This is helpful, but still generic. It cannot tell the operator where the specific transaction is slowing down today. Now. Let's add live observability to the context. A trace may show the request moves through the system and starts to slow down, maybe here, in a middleware call. The metrics show that a related queue is starting to grow. The logs may show some sort of a timeout pattern.
04:30 And a change request may show that the configuration was recently updated before the slowdown had occurred. None of these facts alone prove root cause. Together though, they give the agent much better basis for helping the operator. This is where the difference in the answer becomes clear. Without system knowledge or live context, a general-purpose model might say the slow transaction can be caused by network latency, resource constraint, database contention or even an application error.
04:59 That may be technically true, but it doesn't move the investigation very far. With RAG, we can say which components are supposed to participate, which runbooks apply, and which checks the operation team normally performs. That is more relevant because it reflects the actual deployment of the environment, but it is largely describing what should be true.
05:23 When observability is added, the agent can connect the knowledge to the current evidence. It might say the documented path is operating normally through the distributed application. The largest increase is in the elapsed time between the middleware layer and the mainframe. A related queue has grown over the same period of time, and a configuration change would have occurred shortly before the response time had increased.
05:53 The logs, in this case, and the traces, support that observation. The agent can also separate what is known from what is only a possibility. It can say the timing makes sense with the configuration change, and it's worth validating. While making it clear that the evidence does not yet prove causation. The result is not a magical diagnosis. It is a response that is specific to the environment, it is grounded in evidence, and it is structured around the operator's next decision.
06:21 The agent can now organize response around three things. First, is evidence. What signals changed? Where did they change? And which logs, traces, metrics and events support this observation? The second, is impact. Which connected applications, services or workflows depend on the area now showing this abnormal behavior? In a hybrid environment that could view across multiple distributed teams and across multiple distributed components.
06:54 Middleware, and even mainframe boundaries. Third, is next steps. Based on the runbook and current evidence. What should the operator validate next? That may be checking the configuration that had changed, examining this growing queue, comparing this behavior with a baseline or even involving the team that owns the affected component. This is the bridge from AI output to operational reality.
07:22 The agent can bring together information that an operator might otherwise gather from several tools, documents and even other teams. It reduces the manual searching and gives the operator a clear evidence-based starting point. The operator remains in control. The agent can retrieve, summarize, connect and prioritize context. The human still validates the evidence, evaluates the operational impact and decides what action to take.
07:48 Telemetry may be incomplete, a dependency relationship may not prove cause, a recent change may be relevant, or it may be coincidental. The goal is not to remove the operator. It is to give the operator a more complete picture, faster, while also keeping the evidence invisible and keep the decisions accountable. So, the takeaway is straightforward. RAG helps an AI agent to understand documented system knowledge.
08:17 Observability helps it to understand what the environment is doing. In a hybrid environment, those two sources and contexts are needed across the same boundaries to build the application workflow itself. When they come together, the agent can help an operator move from something is wrong to here are what we see, here is what may be affected, here is the supporting evidence, and here is where to look next. Better inputs produce better outputs. And better context is what makes AI agents more useful to operations.