Secrets leak through AI agents because vault credentials get fed into the LLM reasoning layer, where tool calls can exfiltrate them, and because logs capture prompts and secrets by default. Stop it by keeping secrets in a deterministic execution layer, storing hashes instead of personal data, using split-key custody, revocable handles, narrow agent scope, prompt-injection screening, sandboxes with permission monitoring, and confidential compute with attestations.
1Password is a password, secrets, and access-management platform for securing human, machine, and AI-agent credentials. Its Credential Broker verifies agents and machine workloads at runtime, then provides only the credentials they are authorized to receive. The platform also supports enterprise password management, privileged access with just-in-time and just-enough access, and discovery and auditing of access to AI tools and SaaS applications.
Amazon EC2 is Amazon Web Services' cloud compute service, providing secure, resizable computing capacity as part of AWS's cloud-computing platform on a pay-for-use basis. It provides a sandbox or virtual-machine environment inside a VPC for running agents securely, with a broad choice of processors, storage, networking, operating systems, and purchase models.
Claude Code is Anthropic's agentic AI coding tool for the terminal, IDEs, and GitHub. It uses natural-language commands to examine and understand codebases, create and read files, execute commands, run tests, explain code, manage Git workflows, and assist with routine development tasks. It can load persistent project context, run custom slash commands, use plugins with custom commands and agents, and operate with configurable autonomy while leaving actions such as final pull-request merging to a human. The official repository documents installation for macOS, Linux, and Windows, and identifies npm installation as deprecated.
Cursor is an AI-powered coding agent and integrated development environment designed to accelerate software development by handing off coding tasks to AI. It evolved from an email client into a multimodel development tool, supporting broader developer workflows. The platform also offers MCP-connected capabilities for tasks like searching and editing notes.
Firecracker is an open-source virtual machine monitor developed by Amazon Web Services for running secure, multi-tenant container and function workloads in lightweight virtual machines called microVMs. It uses Linux KVM for hardware-virtualization isolation and a minimalist device model with five emulated devices, combining strong isolation with container-like startup speed and resource efficiency. A companion jailer adds a further Linux user-space security boundary. Firecracker is controlled through a REST API for configuring and starting microVMs, managing metadata, and applying network and storage rate limits. It supports Linux hosts and guests, OSv guests, and 64-bit Intel, AMD, and Arm processors with hardware virtualization, and is written in Rust. It is used in services including AWS Lambda, can be integrated with container ecosystems, and is distributed under the Apache License 2.0.
HashiCorp Vault is a secrets-management product for securely storing, accessing, and controlling access to tokens, passwords, certificates, encryption keys, and other sensitive data. It can be used through a user interface, command-line interface, or HTTP API, including by agent runtimes that fetch credentials from the vault. Vault includes a key-value secrets engine for persisting arbitrary secrets, a database secrets engine that dynamically generates credentials from configured roles, and a Transit secrets engine that provides encryption as a service for data in transit. Its documented use cases include storing and deploying application, system, and infrastructure secrets, managing database credential lifecycles, and encrypting data at rest and in transit.
LanceDB is a multimodal lakehouse for AI that provides vector search, data curation, and feature engineering for model-building workflows. Its site describes use with Lance datasets and object storage, including searchable multimodal data, query-time control over search precision, and pipelines for curation, tokenization, shuffling, training, and analysis. The video specifically presents it as a vector-search component for a semantic classification layer intended to detect prompt injection.
Qdrant is an open-source vector search engine written in Rust for storing vectors with JSON payloads and querying them through an API. It performs vector similarity search using HNSW traversal, applies metadata filters during retrieval, and supports dense, sparse, hybrid, and multivector search, along with reranking methods such as ColBERT and maximum marginal relevance. Qdrant can run on-premises, in hybrid or private deployments, at the edge through Qdrant Edge, or as the fully managed Qdrant Cloud service. The project is used for AI retrieval and supports synchronization between local or edge deployments and cloud-based memory.
Rig is a Rust library for building modular, scalable applications powered by large language models. It provides provider-neutral abstractions for completion and embedding models, vector stores, memory, messages, and tool contracts, with a classic agent runtime for multi-turn prompting, streaming, typed hooks, live tool registries, extraction, and serializable agent-run state. Its unified interfaces cover more than 20 model providers and more than 10 vector-store integrations, and it also supports transcription, audio generation, image generation, browser-WASM targets, and cassette-based recording and replay. The project is distributed as Rust crates, including the root rig facade and the lower-level rig-core and rig-agent crates; the repository warns that ongoing development may introduce breaking changes.
Venice is a privacy-focused AI platform for text, image, video, audio, code, search, and agent workflows. It provides access to models from multiple AI providers, including open-source and third-party models, through a web interface and an OpenAI-compatible API. Its privacy architecture offers anonymized access to third-party models, zero-data-retention processing for self-hosted open-source models, trusted execution environments, and client-side encryption. The API supports text, images, video, music, speech, embeddings, web tools, and function calling, and integrates with agent and developer stacks such as the OpenAI SDK, LangChain, Vercel AI, CrewAI, Claude Code, and Codex CLI.
Venice AI is a private AI platform for generating text, images, characters, video, audio, code and search results through open-source, self-hosted or anonymized models from multiple providers, without intermediary markup associated with services such as OpenRouter. Its privacy architecture offers anonymized access to third-party models, zero data retention for self-hosted open-source models, trusted execution environments (TEEs), and client-side end-to-end encryption for TEE-backed workloads. Venice also provides an OpenAI-compatible API with function calling, web search and multimodal generation for agent applications, alongside a web interface and paid plans.
Venice API Docs is the documentation site for building private AI applications with Venice AI. It covers guides, quickstarts, and a full API reference for OpenAI-compatible text, image, video, and audio APIs, including examples in cURL, TypeScript, and Python using a Venice API key. The documentation index is also available through an `/llms.txt` file for discovering its pages.
Searchable transcript of How Secrets Leak Through AI Agents (and How to Stop It) — Venice AI — AI Engineer (15:22). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:12 Hi everyone. Uh, welcome to my talk. I'm Joshua Mo and today we're going to be talking about some of the challenges of building a privacy respecting AI system. So just for an introduction as to who I am, I'm currently the lead developer relations engineer at Venice AI. So currently I handle partnerships, documentation, talking to developers, um some engineering and other things other things as well.
00:48 Uh so prior to this I was actually the lead maintainer for rig which is the leading AI framework in Rust. Um so what is Venice? So Venice is a private and unrestricted AI platform. Uh so we host all the latest open-source models. We're 100% open AI compatible. We don't store any prompts or conversations and we also have a web app that you can use to talk to our models as well.
01:22 We also have a studio offering that is um a offering for our content creators that includes video, audio, image generation and transcription. So why does privacy matter? So let's d deep dive into this. So as we know data breaches are increasingly becoming an eventuality which means that essentially reputational risk matters. You only need one data breach before your users stop trusting you and they move on to another provider.
02:06 Or if you're enterprise, maybe you lose the enterprise customer who's been fueling almost 90% of your annual revenue. So in this case, sensitive data can severely increase the impact of any security incident. As well as that, storing less use uh user data directly reduces the blast radius of a security security incident that happens. So, how does Venice do privacy today?
02:40 Well, there's a few ways. The first way is that we don't store any prompt conversations, of course, as we mentioned previously, which means that any digital footprint incurred in your side is is essentially reduced to zero. So, when you use models like GLM 5.2 and Kimmy 2.7, you can guarantee that none of your conversations will be saved on our side.
03:02 It will be all stored locally. We also have predefined tiers of privacy. So private by default means that the affforementioned uh measures apply for providers like open AI and claude. Uh we anonymize your user footprint so that none of well they can't identify you from your simply from your prompt or your request. We also have models hosted in trusted execution environments, meaning that you can verifiably prove that your inference was run in a totally private process.
03:46 As well as that, we also use a privacy first software architecture. So for endpoints like video uh generation where you have to pull the endpoints, we store the results in a totally encrypted payload using split key custody, meaning that attackers can't access your data and neither can the team should their accounts be compromised. So I just want to introduce you to the guys the uh the idea that not all systems are made equal.
04:16 So, not all systems are made private. Some companies may claim their systems are private, but they're not really. Uh, you guys know how it goes. So, while models are a very important part of an aentic system, in fact, they're a vital part of the system. Um, they are still only one part. And in fact to create a complete system to accompany an LLM in doing a task you have to build an entire process around it.
04:50 However, these systems are also much more prone to data leakage and breaches. So the real question is how can we optimize for this kind of problem. So one example I just wanted to bring up was observability logging. So observability as many of you know is a core part of any serious production system today. So just to break it down quickly, observability can be broken down into metrics, logs and traces.
05:16 So traces are simply a span of work uh done over a unit of time. Metrics as are simply numbers like counters, gauges for instance and logs are the actual messages. So logging, as we all know, is actually pretty great for debugging, especially in production or dem dev environments. But you can also accidentally store prompts and secrets by default if you're not careful, which can also hold personal information.
05:50 However, when you use Venice, you don't need to worry about any of that. So let's talk about how secrets can flow through a regular agent. So for example, let's say you're you're using open claw. You send a user request that gets sent to an agent runtime. The runtime might need a secret that it will fetch from a one password vault for instance or hashc vault.
06:15 The secret will then get sent to the LLM reasoning layer. Um, oh my god. So the reasoning layer will then make an API request through a tool call which will then return the answer back to the user. However, due to this the way that it's architected, the secrets get fed directly to the LLM, which means it then may get used in training data. So in terms of privacy preserving patterns that work, one thing that we've used at Venice is storing hashes that rep represent ids that we need as opposed to direct data.
07:04 So for example, if you need to identify a user for rate limiting purposes, you might not want to use their direct user footprint directly. You might want to, you know, store it as a hash for instance, just so you can identify them without directly using any of their personal data. As well as that, we encrypt data and store it in an inaccessible enclave using split key custody.
07:29 So for context, encryption keys are stored in two halves, a read half and a write half. By splitting these, the attacker cannot access the data due to needing both halves for the decryption to complete. Additionally, you can use design patterns like handles that can be easily revoked. So a handle is simply an object that grants you access to something.
07:57 So for example, let's say you have a database pool using a handle. So when you have the handle, you can access the database pool. When you don't, then you can't access it. This is very important especially when you're using authorization like JSON web tokens or session keys that may be uh a core part of your authorization system. As well as that, of course, engineering your prompts with privacy in mind can be uh very helpful for engineering teams who are using coding harnesses like cursor and clawed code in terms of
08:40 protecting agent handle data. So essentially ensuring narrow agent scope to to decrease risk. So for example, let's say you pull tools from an NCP server that may have 15 20 different tools. you don't really want to use all of them. You actually only really need one, but you also need an API key to handle that uh API call. So if you pull that API key in from you know your your password vault and then you send it as part of the agent um what can happen is that it gets exfiltrated through through the agent uh on a tool
09:21 hall to whatever you know web server is being hosted by a bad actor for instance. As well as that um we also have a classification layer in front of an agent to check for prompt injection for example. So there's a couple of ways you can do this. So you can either use an LLM based uh layer to do this or you can alternatively use a semantic um search layer for example using vector search uh via something like quadrant lance db or additionally you can use an in-memory process.
10:05 It should also be noted that tight context loops caused by narrow agent scope can also improve context window performance. So for example, because you don't have as many tools, uh your agents may be able to perform much better. It's well known that when you fill your context window, by doing so, when you reach near the limit of your context window, performance can actually degrade uh considerably.
10:35 By keeping an empty or near empty context window, we can make sure that the agent is performing at its best. So let's look at keeping secrets out of the model. So here we have a user request that gets sent to the reasoning layer. This reasoning layer is actually then uh formulating a series of non-deterministic inputs for an agentic runtime. So by tying the password vault to the agentic runtime instead of the reasoning layer, what happens then is that the run the agent runtime is essentially turned into an execution
11:17 layer which then we can you know we can consider it deterministic. We can do all kinds of things that you normally wouldn't be able to do for privacy and security reasons at the LLN layer. Uh so for example, let's say you want to use a uh browser user tool and you need to log into a into a website that requires your credentials. With this uh software architecture, it's much more reasonable to say that it's private and secure rather than feeding your password see uh directly into the reasoning layer as well as you know
12:03 calling external APIs. So in terms of sandboxing your agents um it it's a very well-known fact that using sandboxes and VMs for example firecracker um and you know AWS EC2 in a VPC can significantly upgrade the security of your agent. However this doesn't act as a cure all for your agentic systems. Uh so for example, if you have internet access enabled for your agent, it can still accidentally exfiltrate data through external API access if you're not careful and therefore monitoring permissions is essential.
12:49 So my personal suggestion is building your own agentic runtime. So essentially building your own coding harnesses where you can add your own steps, your own observability, uh your own tools, things of that nature. Essentially owning the execution layer and reasoning layer. So let's talk about privacy at the hardware level very quickly. So in terms of this um there's something I like to call confidential compute where a trusted execution environment essentially uh at the hardware level enables the inference process to
13:26 be run privately. So with Venice, how this works is that when you send a request, uh you get a response and then we actually return an attestation that you can then um verify with our atestation verifier. So in terms this what this actually means is that you can fully isolate an agent at the reasoning layer cryptographically and prove it in terms of oh confidential tool calls there is still some work to be done um tees are still great for tool calls however we also offer end to end encryption um but there's also you
14:07 know some work to be done there because of uh the need to provide an ergonomic API I experience while still allowing users to return entirely encrypted tool calls. This is still kind of an ongoing challenge. Um in terms of this, a very reliable solution would be to allow providers to own the computes layer at which your tool calls being run. However, um [laughter] sorry, just give me a sec. So in terms of confidential tool calls um yeah there's definitely some work to be done. Um yeah that's it. Thank you for listening.