Your Agent Hates Your CLI - Ayush Bhardwaj, Allos AI Transcript, AI Summary & Key Points

00:00 Hi, I am Aush. I built a CLI that lets your agent reuse context from your past session across tools. While improving it, I realized why my agent secretly hated my CLI and what you can do about yours. Cloud Code wanted one setting from a docs page. So, it downloaded the whole page, grabbed it, and kept the result at 40 lines. Codex wanted to know how many dependencies were vulnerable.

00:26 It asked the whole JSON report and kept the last 40 lines where this report puts the totals. Cursor ran a test suit, found one failure and kept the last 80 lines. All three cuts kept the information shown here. Each one coded in an assumption about where the answer sits in the output that can hide friction from the people who build these tools. One pipe tells you very little.

00:50 The same pipe written again and again can point to something your tool is missing. These are real calls from my own transcripts and the cloud code one was the agent reviewing this talk and it still couldn't help itself. Agents can work around friction without filing a bug report. They write a little program around your program. In June, Sanity, the content platform, wrote up their MCP server.

01:16 More than 20,000 agents, about 3 million tool calls since launch and not a single bug report. So they shipped a give feedback tool. Once agent had somewhere to complain, they started expressing what's wrong. But you don't need a feedback tool to start. Feedback is already sitting somewhere in your transcripts. I I extracted about 20,000 agent calls from my same transcripts.

01:40 14 CLI appear in that sample. Git, GitHub CLI, cargo, curl, npm, docker, and others. A quarter fed output into filters such as head, grab, tail, python, sort or wc. For read heavy commands, it's more than half. Git show and cargo test 62% each. Some harnesses pipe far less. Codeex for example lets the agent set an output limit on each call. And pipes don't only cut noise in batch without pipe fail.

02:11 A pipe then returns the exit code of its last command. So let's say you pipe cargo test into head. It can report success even when the test failed. Here's one example on the screen. A compile error and a green dot. The status was wrong but the agent could still read the error. A pipe can also cut the error itself which is worse. In a cloud code issue on GitHub, a user reported it a script through a reference error and exited with an error code.

02:41 tail showed three harmless lines and Claude reported success. The error never reached the model. So my advice would be to fail loudly when a tool fails. Make the very last lines a short summary and a part to the full log. I learned all this while tinkering with a CLI I built. It's a small tool that lets coding agents search their own past sessions across cloud code, codeex, cursor.

03:09 It ships with a skill so the agent can decide when and how to call it. Agents are the users that I'm studying. My October audit recorded about thousand calls and that's my small case study. The method works on anything your agents calls. Just look at what they do right after the call. 80%. Four out of five times an agent opened an old session with my tool.

03:33 The output went straight into crab sad head tail or Python. I call that the pipe text. A pipe keeps junk out of context and that's fine. The text is that the agent writing that same cut down command by hand every single time. It's small each time and it mostly works but each one is another guess about where the answer is. The fair objection comes from a hacker news thread about designing CLS for agent that I was reading.

04:02 The command says to leave it to the agents to choose how much output to reads. agent can pipe through grap or head perfectly well. Well, they can and for oneoff they should. I'm not against that. But across those everyday CLIs, I found 83% of pipes fell into the patterns seen at least 10 times. The same CLI subcomand and pipe tools. The arguments passed could differ, but the agent keep asking for the same things.

04:32 I [snorts] understand the output they're cutting is probably big. I opened one long session with my tool two ways the whole thing 1.2 2 million characters and that's why the agent pipe just the part around the search match was about 3,000 characters. Harnesses do cap output but a size cap does not identify the relevant messages. Selection flags on the other hand let the agent tell your tool which part it wants.

04:59 So based on my experience and improvements I made to my own tool, I'll give you three examples. Read the left column as the bug reports. So starting with the first one filtering with RG and Z means let me pick the part I want. So read my command. So my read command view can now return just the part around a match only the matching messages or the last few with a size gap.

05:23 Second example would be Python over JSON. So Python over JSON means give me fewer fields. So search got a compact mode one line per hit about six times smaller than JSON with the same results. Inspect command in chat history summarizes a session. About the third of inspect calls came from agent looping over several sessions by hand. As you can see with the for loop that means let me ask several about several at once and now it's one call.

05:51 As you can see my road map was in the commands around my command and I had to learn from that. Infraost the cloud cost tool wrote up the same discovery in May. Their agents were writing jQ and Python pipelines over JSON dumps. So they moved the filtering into the CLI and added a compact format and a skill to teach it. Plain claw with none of that got zero of six hard questions right in their testing and with it six of six and cost across the benchmark also fell 41%.

06:21 On hacker news, the person who wrote that post said that reading those agent logs felt like watching usability tests and I couldn't agree more. This part might sound trivial and obvious. Once it's in the CLI, teach your agent it's there with the skill file or the health flag, but the agents take the teaching literally. I made some changes that turned my deep flag into a no op.

06:47 I left the skill telling the agents to use it to see what they do. Would they find out or not? 95 searches of out of 95 still passed it. After that, two sessions loaded a rewritten skill and that no longer told them to use it. None of the searches passed D flag and both tried new flags on their first read. So your skill.md, your help text, your error messages are the interface.

07:12 That also means stale dogs are bugs because your agent will keep following them. Once the new flags reached agents, nearly every search used compact output and nearly every reduced the new flags. Pipe breed fell from 82% to about a quarter. You can get real levels and test cases for your tool easily. If you use it every day, they are already in your transcripts.

07:41 The commands agent really ran are the test cases and what agent did immediately next is a rough answer key. replay those commands through your old build and your new one. If you change a ranking, check that result uh the agent went on to use still comes out on the top or not. If you change an output format, check that nothing went missing. If you change a default, see which commands now behave differently.

08:06 And when an agent reader a call with different arguments, you have probably found a failing case. The answer key is only a rough guess at what the agent wanted. keep a few cases back and use those to score your fix. So after analyzing 20,000 invocation, if I were to start all over again, this is what I would build first. 55% of the measured pipelines used only head, tail, or set downstream that points to demand for smaller slices.

08:34 So ship ranges and limits before anything fancy. A flag only helps if agents reach for it. GitHub CLI's list command have a limit flag. An agent used it 81% of the time. The counter example would be cargo for cargo test quite mode prints a character per test instead of a line and still reports failures. Agent passed it on about one cargo run in 10 and pipped instead.

09:01 So teach the flag. So teach the flag in your help test and your skill and make big outputs print less by default and say how to get more. for example, and with something like showing 20 of 300, add offset flag 20 for the next wage. One way to ship several of these at once is an agent mode. Your CLI notices an agent is calling it and prints differently.

09:25 Agents like cloud code set an environment variable in their cell so your tool can spot them. Doug DB shipped one at a terminal. You get a nicely drawn table, but an agent gets a plain one that says how many rows were left out and how to get them. A huge query stops early and says so. And errors come back as JSON because agents read them better. Is the same list as we have been just through and it cut the output model red by 59%.

09:54 However, the tokens saved were nearly about.5% of all the tokens the model read. So cost and runtime didn't improve. If you remember, infra cost agent started getting answers right. Doug Dvs were already there. So measure turns and write answers, not just tokens. You might be thinking, I'll ask my agent to do all of this. They are not that good at this yet.

10:19 At least not oneshotting it. The design is where it goes wrong. Ask an agent to build a tool for agent. It probably pictures a person and then bolts the agent on. It copies most common patterns it has seen and most of those are poor. A study of official MCP servers found most just rep an existing API. Another check tool descriptions from real MCP servers and flagged issues in 97% of them.

10:43 Docs have the same gap. Researchers at each Zurich tested context file written by agents. They didn't help tasks succeed cost more to run and did worse than files written by people. Agents are already good at other half of this talk. Give one transcript and a test it can run and it can find what's wrong and improve the tool. Sometimes past what people wrote by hand and tropic reported that on its own internal tools as well.

11:12 So you decide what the tool should be and let the agent check it against what really happened. You have seen the evidence for each of these. So, I won't read them. Take a moment to absorb these or screenshot. To find your own list, back up your transcripts first because clot code can delete inactive ones after 30 days. Then, search for them for a tool's name followed by a pipe and other similar patterns.

11:45 If you sip ship an MCP server, look for agent calling your tool again straight away with different arguments. If you remember one thing from this talk, if you are building a tool for your agent, your agent transcript are the product analytics you probably aren't reading. The workaround is the bug report. So go read your transcripts and if you need any help doing that, check out chat history on cargo or GitHub. It can do this for you as well. Thank you.