Stop giving your agent tools — give it a sandbox instead: a sandboxed code environment where the agent searches for, composes, and executes tools cuts context bloat, turns, tokens, and cost (43% fewer tokens, 31% lower cost in Arize's Harbor evals).
Arize AI is an AI-agent observability, evaluation, tracing, and experimentation platform for diagnosing production behavior, identifying issues, and testing changes. It stores production traces, evaluations, annotations, datasets, and experiments, with instrumentation adapters for multiple frameworks and LLM providers. It combines audio, transcripts, and traces in a session view for investigating voice-agent problems such as latency, interruptions, transcription errors, and incorrect tool actions. Its audio workflows use OpenInference semantic conventions to trace real-time audio across providers, run evaluations for sentiment, latency, interruptions, and task success, and attach evaluation results to spans. Its Phoenix tooling includes code-mode capabilities such as Bash, a Python sandbox, SQL, and user-interface control, allowing agents to search for and compose tools in code rather than placing complete tool catalogs and raw outputs in their context. Its CLI and skills can bring spans and evaluation results into a coding agent's workspace, where the agent can inspect supervisor, specialist-agent, model, and tool calls, and create or use evaluators.
Arize Phoenix is an open-source AI observability and evaluation platform developed by Arize AI for tracing, evaluating, and debugging AI agents and LLM applications. It collects traces of LLM requests and the surrounding application execution, displays them in a trace UI, and supports evaluation storage, datasets, experiments, annotations, and result comparisons. Phoenix can use OpenInference telemetry to record execution paths across multiple application steps, helping identify failures in retrieval, tool execution, or model responses; it also supports code evaluations and LLM-as-a-judge evaluations. It can run locally or be self-hosted in another environment.
Bash (Bourne Again SHell) is a Unix shell and command language developed for the GNU Project. It functions as a command-line interpreter and scripting language for POSIX-like operating systems; Brian Fox originally wrote it and Chet Ramey has been a long-time maintainer. It is distributed under the GNU General Public License and is the default shell on many Linux distributions.
Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing AI agents and language models in container environments. It can run arbitrary agents, including Claude Code, OpenHands, and Codex CLI, against shared or custom benchmarks and environments such as Terminal-Bench, SWE-Bench, and Aider Polyglot. Harbor launches benchmark runs locally with Docker or in parallel through cloud and sandbox providers, and can generate rollouts for reinforcement-learning optimization. It is distributed as a Python package installable with uv or pip and provides command-line tools for running datasets, selecting agents and models, and listing supported benchmarks.
Harbor is a framework for evaluating agents in controlled sandbox environments using a reinforcement-learning framework. In the cited discussion, Harbor is used to evaluate code-mode agents and compare their token usage and cost with classic tool-calling agents.
Phoenix is an open-source AI observability platform developed by Arize for experimenting with, evaluating, and troubleshooting LLM applications and agents. It traces application runtime with OpenTelemetry-based instrumentation, supports response and retrieval evaluations, and provides versioned datasets, experiment tracking, prompt management, and a playground for comparing models, adjusting parameters, and replaying traced calls. Phoenix also includes PXI, an AI engineering agent for debugging traces and iterating on prompts, plus a remote MCP server for querying traces, datasets, and experiments from MCP clients. It can run locally, in containers, through Kubernetes and Helm, or in the cloud, and is distributed through the `arize-phoenix` Python package, Conda, Docker images, and related TypeScript and Python packages.
Searchable transcript of Stop Giving Your Agent Tools. Give It a Sandbox - Mikyo King, Arize AI — AI Engineer (14:38). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:01 [music] >> Thanks for everyone for coming. So, my name is Mikail. I head up the open source group at Arise AI and we work on a fully open source observability and evaluation framework and we also work on our own agent. So, today we want I want to kind of walk you through what Code Mode is and our journey embracing Code Mode. So, if you look back in the last 30 years of software development, you could kind of segment software into two different categories.
00:44 You could think of there's software built for software engineers and software for non-engineers. Take for example, you have you know, you're asked to get all the invoices out of your email inbox. If you're a programmer, you could write a script, [snorts] you can go in there, you can dynamically like see if there's an attachment, download it, put it into a folder.
01:02 It might be a you know, 50 lines of code and you get this tool that you can use versus if you were a non-technical, you'd have to go into these inboxes, you have to manually download it. You're restricted by the interface of the software and you had to kind of buy that software. And the thing that's really interesting about things like Cloud Code is that that boundary is completely dissolving, right?
01:24 Basically, anything you need, the coding agent can actually build that software for you dynamically. And the labs and the foundation model providers know this, right? If you look at all the benchmarks like SWE-bench, that is what these models are being trained to do. And so, doesn't it make sense basically to align the agents you're building to be able to kind of build the software you need dynamically to dissolve that boundary.
01:52 So, before I get to started in Code Mode, let's look at kind of what an agent looks like. So, hopefully you've built this uh uh kind of agent before. You give an agent kind of a huge catalog of tools, and then iteratively a user question comes in, it calls the tools that get pushed in the context window, it reasons about it, and then it calls another tool that get pushed in the context window.
02:17 And hopefully most of you have built an agent like this. Uh certainly if you have a certain set of tools, this can work pretty well. But, we're now in that age where we want a lot more. We want much more emergent behavior. We want to put a lot more context into the agent. We want to have longer horizons. And so, this is where you kind of get this bloat inside of the context.
02:39 You'll see there's a lot of tools that get added into the context window that might not be relevant to the task. Um maybe the tool call is not responding with a lot of junk, or maybe needs to paginate through data. And so, you deal with this kind of context rot. You also deal with multi-turn, a lot of turns, as a result, and you're going to have to deal with compaction.
03:02 So, luckily CodeMon solves a lot of these things. It's kind of an alternative architecture. Um instead of, let's say, just giving it a huge index or a menu of tools, you give it the ability to search for the specific tool, and then you also give it the ability to execute the tool. But, you don't give it just the ability to execute the tool, you actually give it a programming environment to execute those tools in different types of flows.
03:29 So, they can do it sequentially, they can do them in parallel, um can put conditionals around it. Um and the nice thing about this is you can call multiple tools in one turn, and also return only what's necessary to push back into the context window of the model. And so, to kind of summarize it, put those these two things hypotheticals side by side, traditional tool calling, you know, you accumulate context window blow every time you call a tool.
03:58 That's obviously going to, you know, add to latency and cost. It's also going to increase increase turns. So, the time your user takes to get to value increases, right? So, has a lot of overhead. Um, on on the flip side, code mode potentially, when done well, can unlock a lot of abilities because it can call as many as it wants in a single turn and it only has to put in exactly what's necessary into the context window to solve its problem and you can also get a reduction in turn.
04:31 So, that's sort of the the goal of of code mode. And so, let's just kind of go over some of the reasons why code mode works. Code mode works because one, you don't have to give it the giant catalog of tools up front. It can just pick and choose the tools that it needs and understand those tools and then execute just those. So, there's a progressive disclosure of tools.
04:54 Uh, certainly there's things like tool search now also built into the foundation model providers. There's also the composability. There's a huge amount of benefit to not being able to being able to call tools in different sequences, either in sequence or parallel, fan out, chaining them together. You can create much more complicated and efficient things.
05:11 If you've heard of obviously Claude's dynamic workflows is built on this composability but composability of sub agents. I already mentioned sort of token efficiency. You don't have to spit out all the tools outputs and worry about the tools outputs being shoved into the context window. The model can dynamically kind of trim and search for certain areas and only put what it needs back into its context window.
05:38 And the fourth one, which is the kind of the one of the major reasons we've embraced code mode, is this idea of emergent behavior, being able to build its own tools, right? So, for example, traditional with traditional tool calling, if I said for example like take all these records and remove personally identifiable information, it wouldn't be able to do that because it doesn't have a tool, or it might try to do it inside of its its context window, but then you're leaking credit card numbers in the context window, and
06:01 that's not great. So, the emerging behavior of being able to create its own tools has a huge amount of benefit. So, here are some ways in which we've embraced Code mod. Uh so, we build our own agent uh and harness inside of Phoenix, our open source uh our observability platform, and we started with Bash. You know, Bash is a great tool of being able to compose together things like uh you know, said and grep to be able to uh compose together, you know, built-in Linux primitives with also our own CLI and be able to
06:38 offload those into files for later use. We also added um you know, a Python's sandbox, and we hoisted all our rest CRUD endpoints. Those are like perform getting traces or uh manipulating data sets, and we put that into this coding environment so that it can compose those create, read, update, delete operations, which unlocks like a whole category of things.
07:04 And we also added one additional one, which is the ability to write SQL. And you know, text of SQL, you've obviously like heard like that probably many times before, but this also unlocks a lot of capabilities. The reason why is not everything fits into the tool category of just creating, updating, and deleting. Maybe you need to be able to do aggregate queries like tell me all the LM spans that have P95 latency of X, Y, and Z.
07:28 You don't want to have to paginate through all your traces to do that. You want it the LM to be able to pinpoint find exactly what you're looking for. This is kind of a a neat and creative way we've also used code mode. So, we've actually also created a sort of a programming paradigm in our UI that's bound to our UI. So, we've been able to kind of be able to give it the ability to change the time range or be able to read data from the data store inside of the front end.
07:56 And what's really cool about this is you kind of almost get this computer loose use like emergent behavior because we're able to put that programming paradigm in a web worker that has a two-way data binding directly into our browser and it can kind of do complex UI interactions for you and it's really pretty fun to to watch it uh cook. So, code mode is not uh obviously for free.
08:22 Um you're dealing with LLM generated code. If you've been at to the the Anthropic talks, you certainly probably seen some of these. You're dealing with resources. So, if you're working with hosted sandboxes, you have to worry about CPU, memory, how long those sandboxes stay around. Networking is a huge one. Maybe you want it to be able to go out to the web, but you never know what's out in the web.
08:44 And so, you have to carefully think about network egress. Uh secrets is a hot topic. I think every sandbox provider is trying to come up with a different vaulting strategy. But, let's say you want to use Jev inside of your sandbox like Pi uh durable just shipped, you know, that's a crazy cool idea uh to be able to make fast decisions inside of a sandbox, but you have to make sure that those secrets don't come back into the context window.
09:12 And then the other one is permissions. Uh there's some very interesting research around this as well, but typically permissions are overbroad. And so, maybe in the first five sets the permissions are correct, but in the sixth operation it is not. Or maybe in composition uh the permissions are not. And so, you have to kind of think of guard railing some of these things if that is necessary.
09:35 There's also operational complexity. Uh certainly sandboxes don't boot uh all the time perfectly. And so you have to you know be able to deal with failures. They're harder to debug because now you're debugging code and you're not debug debugging individual tools. And tool parameter accuracy becomes a much bigger issue because the tool parameter now is code.
09:56 Uh so what if something fails in the middle? Uh tool discovery is also still an issue. Previously if you did have a tool, the LM would up front know it, but now it might have to go through a handshake to be able to understand which tools are in its catalog of things to be able to do. And then this is sort of a big one, you know, you see code mode being injected into MCP, but then you see a lot more code mode being pushed also into harnesses.
10:23 And now you have a double sandboxing uh problem. So sandboxing, you know, is going to return stuff out as serialized JSON and then it needs to be parsed and then put into another code mode. And so there's this like double duty that ends up happening that also can be on operational complexity. These are sort of open problems. Uh they're not like blockers, but durability is a problem.
10:46 So let's say you build a giant piece of code that needs to execute and it breaks in the middle. You can't resume from that step. Um if determinism and mathematics are important to you, for example, it might actually force you to think about which programming language you're using. So JavaScript is something LLMs are really good at doing, but maybe not great for math.
11:09 Um For example, uh code mode can be great way to do multimodal and manipulating images or creating bytes, but getting that out of the sandbox is quite tricky, right? Because you don't want to serialize the whole binary, the whole image. That's going to end up in the the context window. So you might have to do some sort of signed image handoff. And then the other one which is like the Frontier models are amazing at code mode, but if you're trying to do cost saving and you want to swap out a smaller model, its
11:39 capabilities might not just quite um get there. So, how do we at Arize evaluate code mode? Um we use our own We dogfood our own platform, which is called Arize Phoenix, um and we run our agents through various permutations through something called the Harbor framework, which basically puts the agents into uh sandbox environments and runs them on clear instructions with network access turned off, and we run this multiple times in trials, so we eliminate a lot of the flake.
12:12 And we build evals in the form of Harbor verifiers to make sure that the agent's doing the correct thing. And then, Harbor's sort of rooted in this RL framework. We give it a a one or a zero to indicate whether or not performs the task, and we test it across many different permutations. So, this is kind of why you want to test like this, because as I mentioned, you can get a lot of flake.
12:37 Um sometimes it might work where you're dealing with an indeterminate system. It might take different trajectories uh depending how you do it, and we uh track pass at K of each individual task that we think it should pass. So, you know, how does this work? Uh this is kind of our results from uh running this Harbor run. You can see pre uh code mode and post code mode, we, you know, decreased the amount of tokens by 43%.
13:07 The cost went down by 31%, and you know, there was some overhead with the sandbox, but the latency stayed pretty constant. So, we knew that across these multiple surfaces, whether or not it was cloud code or MCP or the CLI, we had accomplished something that uh was definitely better than it was before. How do we use this data? It's really uh a huge unlock if you have you both capture the trajectory and the evals.
13:27 As you've seen with a lot of the other talks, this gives you a very great environment to reinforce and look at the behavior of these agents and be able to tell it how to hill climb. So, maybe, you know, first time we introduce a new tool, sometimes actually the efficiency goes down. And so, we can tell a coding agent why in these specific scenarios, what parts of the instruction are are guiding it in the wrong direction and we're able to hill climb these evals to be able to make our agents better.
13:58 So, if you don't take anything other than this away, hopefully this will sum it up. We really like code mode at RI because of the fewer turns you get out of code mode. We want you to be able to bring any model for us. So, we want it to be cheaper and we're really excited about the emergent behavior and the long horizon tasks that our agents can do. And so, I hope this talk convinces you to give your agent a sandbox and then use one of our tools to evaluate it. And that's it. Thank you.