Use the KV cache that inference engines already compute as a security signal: learn a baseline of how KV pages look for normal traffic, flag anomalous pages, then classify and explain them. This complements edge guardrails (judge models, per-failure classifiers) because anything that slips past input/output filters still becomes computation, and the data stays in place, which suits privacy-sensitive deployments.
CacheGen is a system for compressing and streaming key-value (KV) data generated by AI inference engines. It is intended to make the real-time movement of large KV objects feasible, addressing the size of KV data when it must be transferred between components or systems.
Gemini is Google's AI model and assistant accessible through gemini.google.com. It processes mixed media including video, audio, images, and text, and can convert recordings into SOPs, summarize meetings, extract decisions, and draft follow-up emails. In research and development contexts, it serves as an alternative LLM backend for tasks such as adversarial audits of requirements and resume evaluation.
Hugging Face is an AI and machine-learning collaboration platform that hosts and distributes models, datasets, and applications. Its Hub supports public model, dataset, and application repositories across text, image, video, audio, and 3D workloads, while its open-source stack includes Transformers, Diffusers, Safetensors, Tokenizers, TRL, Transformers.js, smolagents, and PEFT. The platform provides a unified API for accessing models from AI providers, GPU-based compute and Inference Endpoints, and paid team and enterprise features such as private datasets, access controls, single sign-on, audit logs, and dedicated support.
Intrinsec AI is an AI-inference security company developing products that use KV-cache data as a security signal for production LLM inference. It learns the normal computational shape of a workload from its inference traffic and surfaces unusual KV pages, rather than inspecting prompts or relying on a separate detector for each attack or failure. Its Tessera product analyzes KV cache at rest, while Plato acts on anomalous KV data in flight; the system is intended to cover text, voice, tool-using, and agent workloads and can integrate with inference engines or KV-cache management layers.
LMCache is a KV-cache management layer for AI inference systems. It stores pages of the KV cache from GPU memory in remote storage and provides an integration point for reading stored KV pages, including for security analysis and threat discovery.
NVIDIA Dynamo is a developer toolkit from NVIDIA for managing distributed key–value (KV) context caches used in large‑language‑model inference. It provides KV‑aware routing, cache offloading, and prefill/decode disaggregation to move KV cache and related activation/state across GPU clusters, reducing memory pressure and supporting scalable LLM serving. It is intended for integration into GPU‑based inference and serving stacks.
SGLang is an open-source serving framework and inference engine for large language models and multimodal models, developed by the SGLang project. It provides an alternative serving stack for deployed models, with runtime and kernel optimizations for inference workloads. Its documented capabilities include RadixAttention, a zero-overhead batch scheduler, cache-aware load balancing, structured-output decoding, speculative decoding, disaggregated prefill and decode, model parallelism, and support for GPU and TPU backends. The project also publishes integrations and optimizations for current open models and multimodal, image, and video-generation workloads.
vLLM is an open-source library and inference-serving engine for deploying large language models on GPUs and other hardware accelerators. Originally developed in UC Berkeley's Sky Computing Lab, it is maintained by a broad community and provides an OpenAI-compatible API server, as well as Anthropic Messages API and gRPC support. Its serving stack manages attention key-value memory with PagedAttention, continuous batching, chunked prefill, prefix caching, streaming generation, and disaggregated prefill, decode, and encode. It also supports speculative decoding, quantization, optimized attention and GEMM/MoE kernels, automatic kernel generation and graph-level transformations, structured outputs, tool calling, multiple decoding algorithms, and distributed inference through tensor, pipeline, data, expert, and context parallelism. The project integrates with Hugging Face models and supports decoder-only, mixture-of-experts, hybrid state-space, multimodal, embedding, retrieval, reward, and classification models. It can run across NVIDIA, AMD, Intel, and other supported accelerators, as well as x86, ARM, and PowerPC CPUs, and can be installed with uv or pip or built from source.
Searchable transcript of Your KV Cache Can Spot Attacks on Your AI Agents - Ari Azarafrooz, Intrinsec AI — AI Engineer (07:12). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:00 Hi, I'm Ari Azarfar, founder of Intrinsic AI. Um this talk is for people who build agents and run their own inference stack, or plan to. And if that's you, you associate typically the KB cache, the bread and butter of uh modern inference engines, usually as a throughput optimization. The keys and values computed per token, kept so the model doesn't compute them again.
00:22 Today, I'll like to show KB cache in a different light. The way agents agent AI agent application could go wrong or keep growing, right? Prompt injection, jailbreaks, broken tool calls, sensitive tool calls, sensitive data exposures, hidden backdoors, and drifts, and so forth, right? So, the way right now we typically guard against against this numerous ways that an AI agent application could go wrong is by guarding the edge, by perhaps passing a judge model that checks the prompts, the inputs, and the response outputs
00:52 against your policy. All right? Or you build a classifier for each case. But here's a different light. The model turns every input into a computation before it acts, before it decodes, before it generates the response, before that response becomes another tool call. The KB cache is your engine's record of that computation. The modern inference engine, for free, are using it and reading it.
01:20 And therefore, on this talk, I would like to suggest as as a complement to the way you're already using your own guardrails, to use this data and integrated with your KB cache infrastructure. I refer to this as a KB first security signals, and basically three reasons that matters, right? So, the data is already there. You pick the gems from the pile your inference stack would already were throwing away.
01:47 It covers the guardrails blind spots, because whatever it slips past those guardrails, it still becomes computation. And it stays in place, so it's perfect for data privacy application, for cases where you do not have the luxury to read the inputs and outputs. And the recipe is pretty straightforward, right? Has three steps. You learn a baseline for your AI workload, for your AI inference traffic, how your KB page typically look like, and then find the KB pages that are rarer against your baseline, typically your
02:20 anomalies. And then you can even classify and explain those. So, first let's start with where in the stack this integration could happen. The interesting thing about the KB cache is that in a way existence of those had made the inference in a way stateful, right? So, for example, things like KB cache management layers such as LM cache stores them from GPU down to a remote storage.
02:44 Or LLMD and NVIDIA Dynamo route each new request towards the worker that already holds perhaps that cached KB page. So, there are two integration points you you you you can integrate those anomaly detection or those classifiers into. Either you can implement it directly inside of your inference engine via LMs, you name so forth. The beauty of this approach is that it's real time.
03:11 And and the the the alert that it would perhaps generate could ride along your future tool calls or your future policies. So, you can harden basically your policies or guardrails but just generating the computational anomalies that you observe within your inference engines. And you or perhaps you can integrate it with as a as a way of threat discovery in a sense.
03:35 So, you can integrate with your LM cache. So, it reads the stored KB pages and find things that don't belong with the rest, right? So, now I want to show you some numbers and just a very simple hello world use case if if this is actually working and this is a valid approach. And for that, let me use the use case that has no text to filter at all, a voice agent.
04:01 So, here's a case of study of a voice agent. In this case, it was a Gemini 404B. Our baseline was 72 ordinary clips and spoken four voices. We're looking at 16 token pages. Obviously, this granularity could change. But here was just use this shorter pages so we could have more data in the baseline. And then on the test, we have different voices one clean voice and the other one has a hidden instruction within it.
04:28 As you can as shown here, not only you can detect these cases just using simple anomaly detection, but also you would know which part of um which page which part of your request had this anomalous event. We even went further and we trained a model that uses the KD pages as features. And we didn't we're trying to explain it or tag it with things that are useful.
04:56 In this case, although it wasn't even trained on audio, it's able to generate correct information about personal information, home address, location data, travel information, destination. So, now back to text agents. In this case, it was a client 314 billion served live on bill 110 cash. The one on the left shows you an idea of how attacks and normal traffic would typically separate from each other and then the one on the right, which is perhaps more meaningful, it was using the AT bench and hugging face.
05:27 It has its own baseline of benign agent traces and eight kind of agent failure from injected tools description to misinformation and so forth. We look at the top 10 alerts that anomalous anomaly detection would generate and about seven to eight times of out of 10 of these alerts are actually anomalous meaningful and useful anomalous activities. We even went further and we looked at whether there's any kind of anomalous event that is stand out more than the rest.
06:03 Um to make things hard in production. First, models change. Fine-tunes and new releases, but also agents that each one on a different model. And routers that pick a specialist model per request. That means each model writes its pages its own coordinates. So, either you learn every baseline, classifier, and explainer, which is uh computationally prohibitive, or you have to find a way to map them all into one shared representation space.
06:37 Second, real time means moving big objects. For example, a 1 256 token chunk of 1.34 billion is about 40 megabytes of KB. Compression and streaming like cache gen could help, but moving that for security analysis and acting in real times is still a slow. And that's what we solve at Interesting AI. We are looking for teams building their own inference stacks who treat security as a foundational design decision. I hope this was helpful for you. Thank you so much for listening. Bye.