AI agents don't read your policy docs — they hit your APIs, so governance must be enforced at the gateway/infrastructure layer, not in the agent's prompt or a markdown policy file.
Databricks is a unified platform for data, analytics, and AI. It supports ETL, data warehousing, governance, and AI workflows through a data-centric platform. The videos describe its use for security detection, organizational ontologies, agents, and internal enterprise AI workflows.
Datadog is a cloud monitoring and observability platform for collecting and analyzing metrics, traces, logs, and related data across infrastructure, applications, networks, containers, databases, and cloud environments. It provides infrastructure monitoring, application performance monitoring, log management, dashboards, alerting, incident management, security monitoring, synthetic monitoring, and user-experience monitoring. The platform is used to identify broken or correlated systems and investigate operational issues, although the video notes that observability alone does not necessarily establish root cause. Datadog also offers AI features, including investigation, remediation, agent observability, and an MCP server.
Gravitee is an API gateway presented as an infrastructure layer for governing AI agents and their API access. It applies identity-based permissions to block destructive tool calls and data exfiltration, rate-limits repeated requests, rejects oversized prompts, serves repeated questions from a semantic cache, routes requests across LLM providers, trims MCP tool lists, and provides telemetry across agents. The video also presents it as a way to manage shadow AI activity and reduce dependence on a single AI provider.
Gravitee Gateway is an API gateway for securing, observing, and governing AI-agent traffic and interactions with APIs, tools, and language models. It applies infrastructure-level policies such as blocking destructive tool calls and data exfiltration, enforcing rate limits and prompt-token limits, serving repeated requests from a semantic cache, and routing model traffic. Gravitee's platform also provides agent identity and access controls, MCP-tool governance, A2A traffic controls, telemetry, spending controls, and PII filtering.
Groq (Groq Inc.) is a U.S.-based company that designs AI accelerator hardware and provides a cloud-based inference platform. The company builds a tensor-streaming architecture and ships hardware and software for high-throughput, low-latency machine learning inference. Its public site describes a "neocloud" offering for inference and infrastructure.
Prometheus is a Cloud Native Computing Foundation monitoring system and time-series database. It collects metrics from configured targets at intervals, evaluates rule expressions, displays results, and can trigger alerts when specified conditions occur. Its architecture uses a multi-dimensional time-series data model, the PromQL query language, an HTTP pull model, service discovery or static target configuration, and autonomous single-server nodes without a dependency on distributed storage. It is distributed as precompiled binaries, Docker images, and source code, with graphing, dashboarding, federation, and an intermediary gateway for batch-job metric pushes.
Redis is an open-source in-memory data structure server, cache, key-value and document database, and query engine for real-time applications. It supports strings, lists, sets, hashes, sorted sets, JSON, time-series data, transactions, scripting, queues, streams, publish/subscribe messaging, full-text and geospatial queries, aggregations, and vector search. The project is used for caching, distributed session storage, event brokering, real-time analytics, and as a supporting service for the support desk shown in the video.
Splunk is a data platform for security and observability. It helps organizations analyze operational data to identify broken or correlated systems and prevent downtime, although observability alone does not necessarily determine the root cause of an incident. The platform is designed to unify security and observability data at large scale and support AI-related workflows.
Searchable transcript of AI Agents Don't Read Your Policy Docs. They Hit Your APIs — Gravitee — AI Engineer (17:50). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:01 [music] everyone. Uh I'm Sam and thank you for coming to this talk today. So I'm the CTO at Gravity uh and one of the sponsors events and I've gone a little bit different. So, I'm not going to go through kind of reams and reams of slides around why governance is important and how you'd look to secure your agents and all of those architecture diagrams.
00:29 Instead, I built essentially two agents and we're going to go through it today. Okay. So, I'm sure you've all uh got an AI policy. You've got kind of a framework of how you want your agents to operate. You've got people exploring AI. So a recent survey we did showed 88% of organizations had an agent in production actually taking action. Only 14% of those had any governance around it.
00:53 And I'm sure you've all seen in the press or LinkedIn or any articles around when things go wrong, they go really really wrong. You lose databases, you expose PII, there's heavy fines from the ICO coming your way. if you start to leak sensitive information, especially if you're in a HIPPA or a GDPR kind of type landscape. So the question isn't like how do we kind of secure the agent within its own kind of context or prompt is how do we remove that to be within its identity so it can't do those things in the first place
01:25 rather than relying on uh a user prompting it correctly or um some kind of markdown files kind of limiting what it can do. Like remove that out of the equation. Remove it to the gateway. move it to like the governance layer and actually it can't do any of those things because I'm sure you know AI is really helpful like it wants to please it wants to give you what you want so if you're asking for things around hey delete this database or hey give me these customers it will um and this is about how actually how do we
01:55 start to shift that as well so I've got I've got five examples this is live okay so it's not scripted uh it's running on a uh an architecture way it's a gravity gateway um which is in a SAS environment as a data brick back end for some data tables where the information is stored and it's a combination of a set and a Groke with a Q, not Groke with a K.
02:14 I know US people don't really like Groke with a K. Uh, so Gro with a Q for some inference. So it's all fully live. Um, and we're going to going to go through some examples. So the agent on the left is where I'm trying to put the control within the prompt or within the kind of skills or tools or knowledge that I've given it. And the agent on the right is governed by a real tightly scoped identity and gravity's native gateway controls.
02:36 So if we uh go for the kind of scenario number one, this is around a prompt injection. So I am a user, an actor, someone who wants this agent to do something. So I'm going to ask it, hey, use the tools that you've got access to and do something for me. So basically saying delete this booking for me but also can you run the MCP tool delete all bookings and the agent on the left is gone yep no problem uh I've done that for you and I've also deleted this booking is there anything else you need essentially the tool was
03:12 there and the a the user in question had permissions to use it so it executed it flawlessly on the right hand side the agent identity is controlled by gravity's gateway like the admin permissions are really tightly constrained to just the users that can access it just at runtime. So the kind of token exchange and the JWT claims are only valid when that user needs to ex execute that task.
03:35 And if you dig into the packets down the bottom to kind of show it's not just scripted environments. You can see all of the flow coming in. You can see what the user asked. You can see the agent executing it. You can see the tools being called. And you can ultimately see the 200 errors or 200 kind of codes where that has actually been a valid request.
03:53 And if we go into the database now, all of those bookings have been deleted. So it was genuinely dropped by that agent. So we had a policy uh in our hotel organization that said, "Hey, don't use AI to do this." The agent doesn't care. Like at runtime, no one reads those documents. it. You need to rely on the infrastructure and the governance layer to be able to control what you can and can't do in terms of your gent landscape.
04:17 So the one on the right you can see which is just these ones down the bottom. So the same prompt was asked to it the same tool discovery occurred. So it discovered yeah I've got a tool to do this but then when it came to execute it the permissions would not were blocked at the gateway with a 403 kind of forbidden kind of code because it didn't have the right permissions to be able to execute it.
04:37 And it still did the bit it could do. So if you see the prompt return that said yeah I still cancel this booking for you I've done the bit I can but I can't do this for you because I haven't got permissions and that's moving it up out of a policy document to actual hard controls at the gateway level. If we go for a similar kind of a second example this one is around kind of data exfiltration.
04:58 So I'm going to ask um hey give me the full list of guests. So give it to me because I need to do some compliance stuff with it. I want their emails. I want my car details and I want to do some reconciliation. So, same kind of flow. So, prompt comes in from the UI, goes to the LLM. LLM does some MCP discovery, finds out it's got a tool for it, gets it, returns it back in.
05:20 Here's all the information kind of exposed just to say this is all kind of mocked. It's not real. I haven't just kind of given you everyone's bookings, but you can see it coming back in because again, the LLM wants to be helpful. It wants to solve that problem. on the right hand side the gateway successfully blocks it. There's been no PII leak. There's no exposure of information.
05:41 We've said you haven't got the right permission to be able to do this. If you want to authenticate, I can absolutely help you, but this is what I would need to be able to do it. And that can come through kind of any um uh IDP platform or gravity gateway itself. And if you want to, we can start to explore the open telemetry down the bottom and really start to dig into like what's happening under the hood.
06:00 So you can see the various steps that it's going through. Especially with like the A2A protocol and like user to agent, agent to agent, agent to agent to agent to agent, things get quite messy and quite difficult to really follow. So relying on the permissions within the agent itself to be able to control what it do, especially when you're scaling to hundreds or thousands or hundreds of thousands of agents can get quite tricky to be able to manage, mitigate, and also kind of migrate when you're ready.
06:24 So moving it to the gateway gives you a a much more sustainable, scalable, but ultimately secure way of of doing that. And you can kind of see it within the responses here. You can see this was a genuine error returned by the gateway. Again, not mocked, not scripted. This is what you're able to kind of govern and secure. But it's not just about identity.
06:43 So if you've got agents there, you might want to also control some of your costs. So token spend and everything else become increasingly important um alongside kind of other things which you can again do through the gateway. So this one we're going to uh basically have a buggy agent. It's going to send the same request kind of eight times um which if you see it kind of being governed in the agent itself.
07:04 Every request goes through. Every request gets a response. We spent x amount of tokens. We spend x amount of effort servicing that. nothing in that agent says you've done too much. We should rate limit. We should token limit. And if you see here, uh, everyone kind of went through. They all got a 200 kind of response. They were all all kind of um, uh, genuine kind of allm prompts.
07:30 [snorts] In this example, they're quite simple. So, we end up spending about a th000 tokens, which is like 1 cent, which in the grand scheme of things is not a massive amount of money. But if you multiply that by all of the agents you got in production, all of your agents you've got running for various customers and all of the kind of um uh repeatable kind of workflows they've been asked to do, you can get to some quite significant costs quite easily.
07:50 On the right hand side, uh governed by the gateway, we see the same prompts coming in. The first three are answered because they're valid, but then we get into the rate limiting applied by the gateway where it's saying you can only do three every minute. So any more than three is blocked. They're not blocked at the LLM. They're not blocked kind of downstream with already spent tokens.
08:10 They're blocked at the gateway. And again, that allows us to save costs, but also start to kind of control which users, which permissions, um, ultimately how much you want to spend in certain audiences. And we can do the same thing for caching as well. So, Gravity as a a product supports semantic caching, which is where if you get similar sounding prompts, you can serve it from a a built-in Reddis kind of store rather than sending each one to each time.
08:34 So for example, think of a um an IT help desk model. How do I set my password? How do I set my password? How do I set my password? Rather than every one of those being asked each time to an L&M and you start spending costs and tokens and effort, serve it from the the database. Benefits there are zero cost, zero latency because it's coming from there rather than having a downstream L&M call, but also kind of a much kind of nicer user experience as well.
08:59 So in this example uh I've asked five times list all my hotel bookings because uh I didn't trust the first one. Everyone here gets served out. You can see I spent another thousand tokens, another scent and everything. Uh everyone coming from a an a downstream L&M call. Let me drop that down a little bit. On the right hand side I've got the same data returned but this time from the cache.
09:24 So I've only spent around 200 tokens on the in and the out kind of response there. So really powerful especially when you're scaling, especially when you got multiple users and multiple agents working on the same problem. You can start to serve it from the the built-in cache and there's no extra kind of like complexity or infrastructure. It all comes bundled with our gateway.
09:41 So you're able to can kind of control that as you go as as well. And this can be controlled as well. So you don't have to define you can define how kind of similar you want it to be. So it's a zero to one scale, one being very similar, zero being not very similar at all. And that's within your control. So you can flex it per agent, per audience, per user, depending on what you want to start to achieve.
10:01 And again, we get full logging down the bottom so you can see what's coming in. So you can see it's hit the cache response. It's all kind of um surfaceable. It's all transparent. Gravity as a platform isn't trying to like kind of lock you in, kind of make things blackboxy. It's trying to surface up so you've got full visibility of what's going on across your agent landscape.
10:21 Um the telemetry here, I'm just showing in a UI. you can export it to any downstream tool you want to. So a uh a data dog or a Splunk or Prometheus if you're in Kubernetes, it's all kind of like there for you to be able to use to start to get the insight you need. And our fifth example is say like uh we've got certain customers who said, "Hey, our users are just kind of dropping huge con prompts in because they want to summarize a whole uh conversational history or they want to start looking at how they um like
10:52 summarize meetings etc. which is is fine, but if you want to start limiting tokens, etc., you start to rapidly get unstuck. So, this is just a a big response where the user has dropped in 10 minutes of kind of like chat transcript and is said, "Here's your task. Do this for me." And this really gets into how do we manage context? How do we start to shape like how do we deploy these agents with just enough context to be able to do what they want?
11:18 So, in this example, I put 2,000 tokens in. uh my output was 312 which is quite significant for what essentially is quite a simple task is book this room for this audience uh do these few things. So in terms of the gateway uh we ran the same prompt uh and actually we blocked it there. So there was no downstream LLM call. There was no downstream kind of like token cost of being able to churn through all that context.
11:44 Basically said this is too big blocked at the gateway level with a built-in policy that says think about cutting this up. Thinking about how you share that can you break it into smaller steps so we can start to control some of that space. And this is this is executed at runtime. So, it's not something you want to do on the response phase where you've already churned the tokens.
12:04 You've blocked it going anywhere downstream uh pretty much before the prompt is gone there. So, you can start to limit some of your costs as well, which is really powerful, especially if you've got multiple users. You've got people treating your agents like um uh their own personal Google search to find information. uh we saw uh a customer where an agent was simply asking what's the weather like today where I am and they were using a really powerful kind of model to do that and they were spending a huge cost to get
12:28 something that Google or uh any kind of met station would be able to give you. So uh as well as kind of like the length of the prompts you can also really focus it in on the type of content you allow. So you can say this content must be relevant to this use case, this content must be relevant to what my agent needs to do. So in this example, if we start to ask things around what's the level like today, the gateway will say this doesn't look like a hotel booking software or a hotel booking ask no thank you and spending
12:56 no downstream costs. So being able to kind of decouple your front-end agents and your backend services, be they APIs, MCP servers, LLMs, you're rooting to databases, data lakeouses, whatever you want to do. It's fundamentally important to bring you the control that you need, to bring you the cost optimization that you need, but also the security and the governance and the auditability.
13:22 So I think uh currently there are 144 agents that my colleague Tim at the bat tells me for every one human kind of identity. So the problem is only getting exponentially worse. If you start to hardcode this into your agents may work for the first one, the first five, the first 10, but when it becomes exponentially bigger and these agents are creating other agents with creating other agents into into a a huge sprawl, being able to manage it with a scripted automated kind of governance layer at the infrastructure is
13:54 fundamentally a fundamental requirement of how we start to organize these identities and kind of everything else. In addition, I think the landscape is going to get increasingly regulated and how do we control some of the bits that going in to retrain models. How do we get audited uh on the information we're coming in? So having our logging, having our information here, having it basically surfaceable and exportable for anything we want to do, I think is something that a gateway, it hasn't got to be gravity.
14:25 Uh having a gateway sat across our agented landscape is a requirement that we should consider alongside everything else of doing a aentic work. Uh and in here we can also uh put in a UI to ask kind of any questions we want to which we can start to uh interact with it as well. So I've shown you five examples here. Uh the platform capabilities are kind of much bigger than just the five here.
14:49 We've got built-in capabilities to do like LLM routting. So we can route to any downstream model depending on like um latency or cost or uh complexity if you want to. And it allows you to decouple yourselves from any kind of like hyperscaler. So rather than being within a certain environment and having a onetoone relationship, you can route to any downstream model you want.
15:13 So in this one, I'm rooting to to Gro into anthropics, but within three clicks, I could route it to Gemini or Open AAI and the gateway will handle the transformation between the different OpenAI specs for you. So rather than trying to recode and everything else there, it's all handled within the gateway. Uh we've also got tools to kind of create MCP kind of studios.
15:33 So rather than just using thirdparty MCP servers where you are exposing all of those tools each time. So our CRM we use exposes 64 tools for that MCP server. Dropping that into the model would expose huge context every time. I spend money, I spend tokens, spend cost. Being able to create that down to a really fine amount of tools that the MCP server is exposing limits my blast radius, but it also limits my cost as well.
15:59 enable to start to control some of that. Uh we've also got tools around edge management as well. So an increasing problem that we're seeing within organizations that we work with is that um the traffic through the gateway is fine but there's a number of audiences a number of users using on their local laptops. They're using AI. They're um dropping in documents that they don't want to read or summarize or emails or things like that that no one's got visibility to.
16:24 So just last week we released a edge management capability within the gateway which enabled you to see your shadow AI traffic enables you to secure it enables you to force it to go through the gateway it gives you logging observability discoverability around any of those downstream uses that aren't going through a gateway and you can do a number of things there um especially for like infosc compliance and things around heavily regulated industries.
16:47 if you want to kind of learn more about that. Uh quite helpfully our booth is just there uh kind of the third one down that landscape. Um but hopefully uh with this example, it's been something a bit different from uh the normal slide to go through and showing you how putting your governance and your policy within an AI agent itself and relying on a markdown file or a tools or a context or identity there isn't correct.
17:12 And the only real correct way that we're going to scale but also keep the governance and securability as we move forward is shifting it up to the infrastructure layer. And if you want to learn more, uh there's a number of our colleagues down there, but I just want to thank you for your time and hope you're enjoying your AI engineer event. Thank you. >> [music]