Every correction you give your agent is data: labeled pushback sessions capture engineering judgment the agent lacks and can be turned into skills, evals or post-training for a custom reviewer.
Capture every coding-agent session your team runs, label the moments where an experienced engineer pushes back on the agent, and turn those labeled corrections into a skill, eval set, or fine-tuned model that reviews new pull requests as if it were that senior teammate.
Braintrust is an AI observability and evaluation platform for agents and other AI systems. It traces production prompts, responses, tool calls, latency, cost, and quality; supports searching logs, real-time trace inspection, dashboards, and custom views; and runs experiments against versioned datasets with scoring by LLMs, code, or humans. Teams can turn production traces into evaluation datasets, compare prompts and models, investigate recurring failures through natural-language queries, and use the results to identify regressions and improve agents. Braintrust provides native SDKs for Python, TypeScript, Go, Ruby, C#, and other environments, an MCP server for querying logs and running evaluations from an IDE, and Brainstore, its database and query engine for AI data. The platform states that it supports framework-agnostic integrations, SOC 2 Type II certification, GDPR and HIPAA compliance, SSO, role-based access control, and hybrid deployment options.
Brev.dev is an NVIDIA GPU service with an instances console, used in the videos to run direct preference optimization experiments.
Daytona is cloud sandbox infrastructure for running AI-generated code. Its website describes secure, stateful environments with fast environment creation and enterprise-grade security. It was used as the sandbox provider in early WolfBench evaluations.
Fireworks is an AI platform that provides training and inference tools to transform open models into domain-specific or specialized intelligence. The service offers capabilities for model training, fine-tuning and serving to support customized AI deployments.
Paper Console is a centralized service from Paper Compute for capturing and labeling coding-agent sessions across a team. It preserves agent traces and supports labels such as pushback, successful sessions, and regressions, including moments when engineers correct an agent's output. The labeled records can be used as training material for skills, evaluations, or model-adaptation experiments. The service is currently waitlisted.
Stereos is a runtime for agents that provides reproducible, sandboxed environments. It is used with Tapes to capture agent-session signals for building self-healing infrastructure.
Sweeper Agent is a project for running parallel agents to perform defined cleanup tasks in vibe-coded software, including documentation, test writing, and code cleanup. The videos describe configurations of up to 10 agents or approximately 200 Daytona sandboxes.
tapes is an open-source Apache-2.0 agent-session capture tool for coding agents. It records the LLM calls and session traces produced by coding agents into a local Postgres database that the user owns, without requiring an SDK. The tapes website says sessions can be searched, costs tracked, and runs replayed; the captured records can also be labeled and used to create datasets, skills, or evaluations.
Searchable transcript of Every Correction You Give Your Agent Is Data - Brian Douglas, Paper Compute — AI Engineer (12:48). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by AI Engineer. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:01 Hey, folks. Thanks for coming to my talk. I'm BDougie, Brian Douglas, and I'm going to talk about if you are going to verbally abuse your agents, at least label the outcome. Now, this is a talk I'm going to quickly just run through one example, and this will be a short talk, mainly because there's actually not too much to this. But outside of that, I want to talk about some of the work I've been doing for the past six to eight months.
00:25 And it actually came out with this idea early this year. Everyone definitely got AI-pilled when it comes to agents. So obviously, cloud codes have got to be one of the most popular ones out there. We've got quite a few other ones that you can leverage, like the Codex or ChatGPT. And there was a book I read years ago, well, quite a handful of years ago, from Rick Rubin about the creative act, a way of being.
00:48 And this was actually around the idea that Rick Rubin produced the Beastie Boys and a number of other acts. But he's actually never played music. And a lot of folks have made this correlation with vibe coding, where Rick Rubin can produce the best music without actually playing the piano. We're in a world where folks can vibe code a bunch of Rust code without never writing Rust.
01:10 And that's exactly where I'm at. There was another article months ago from Addy Osmani, who's now currently at a the Anthropic team, the Quadco team specifically, and was talking about the opportunity for juniors are basically, it's gone. When it comes to junior engineering, it was all about growing from junior to senior was about reps. So the more reps you got, the more engagement you got, opportunity to basically just level up as an engineer.
01:40 And without those reps, we're handing those reps specifically to agents, we're losing the opportunity to level up. So this comes out of something I was doing earlier this year, which built this open source project called Tapes. Tapes.dev is open source Apache 2. There's a lot of other similar stuff out there, but what I'm getting at is it's not about Tapes, it's about the idea of collecting your agent session data, and that's what this does.
02:03 So a lot of OTEL collectors, a lot of session capture services, A lot of the harnesses already do this for you out of the box. Just wanted to mention that this is what I use. You can use anything else you would like. I've also, in addition to tapes, me and my team, we've developed this thing called stereos, and this is the runtime for your agent. Again, sandboxes, not a concept that everyone needs to hear about today, but just knowing that in all the work I've been doing, I've been capturing these sessions and
02:21 recording them by having a reproducible environment. And the environment themselves, they have these signals. And the one thing I've been doing with this little setup of tapes and stereos is I've been building self-healing infrastructure. So this is a little project called Sweeper Agent. And Sweeper Agent itself is a way for me to build parallel agents.
02:51 I can scale up to 10 of these at this point. I've done quite a few Daytona sandboxes as well. I've done about 200 at the same time. That will just solve a specific problem. So documentation, test writing, code cleanup, stuff that you want to scale that's like very, very defined work that you want to spin off to multiple agents in tandem. Now, a lot of this has been cleaning up most of my vibe coding software that I built in the last 18 months at this point.
03:20 Just pointing out that there's a lot of work we can do to basically solve these problems, but it doesn't change the fact that I still don't know how to write Rust code. I've never written Rust code specifically myself, professionally rather. Ten years ago, I did write Rust code, but it was mostly for little side projects to learn Rust. But we use Rust day-to-day at the company I work for, which is Paper Compute.
03:43 And what we've been doing is because of this technology underneath this called tapes we've been capturing traces and tapes is our trace capture service telemetry capture and paper we have a console so for the moment I'll move away from the slides and show you the paper console now paper console itself this showing that I can capture sessions the difference of doing this local in my own sessions is that I can have a centralized data store if I not just me but also my teammate and what's cool about this is I can also
03:47 label sessions so this is a session that oh interesting yeah sorry i'm gonna zoom out to see the labels but yeah so here i can see i've got a session here which uh working with some paper cloud export memory limits and we've got this word this label called pushback now i actually went to specifically a session that had pushback in it and it's specifically every time an engineer pushes back on the agent now There's a thing that I think recently Anthropic or I guess Rumors, I don't know if this is actually grounded by
04:40 any facts, but Anthropic was pushing back on if you're being rude to the agent, you could potentially lose your account or get blocked. But what I find specifically when pushback, when I'm specifically looking for any Rust code or Go code that Matt has written. And whenever he pushes back, it's a good signal that he has knowledge that the agent does not.
05:08 And this usually comes in the form of we have decided our API is structured a certain way. Despite the fact that the Rust code compiles, we have made decisions on how that code should be set up and how that code should interact with different parts of the code base. Also pushing back on things like duplicating services that maybe the agent didn't have context around.
05:28 There's a bunch of reasons, but what I'm getting at, me as a quote-unquote junior Rust engineer, I don't have enough context to be able to push back to the agent to say, hey, you're doing this wrong or you're making weird choices. I usually take everything that the agent writes in Rust for face value, and what that ends up turning into is when I open up PR, I get a lot of pushback from the engineers, the human engineers, at review time.
05:52 This is mainly because we know I don't write Rust code. There's not a lot of pushback I can provide to the code base. or even at the time of review. So by the time it gets to greptile, it's a five out of five review. I just let it slide. So how do I prevent that pushback? What I've done is actually started collecting every single one of these labels and the traces.
06:11 And then one thing I'll point out is that The console itself, I capture every single session. I've got all the specific details. This is actually just a command line version of that. What I've been doing is I've been capturing all these sessions and then dropping into a data set that's been labeled. So now I can just run a little command line tool called Twin Review.
06:31 And then I'll grab a PR number, 381, and I'll run that. And I'll actually run through and... essentially review the code as if it was Matt. Now, the way I'm doing this is I'm capturing traces, I created the data set, I've also done some post-training, and this is like where, you know, choose your own adventure, but I've done some post-training specifically on every time Matt pushes back within Go or Rust code, and I asked for it to provide me comments as if you were Matt on my GitHub PR.
07:01 So this is all data that's only specifically within the traces. So this is not taking any GitHub PR review data, which I think you potentially could also follow that pattern. But what I find is like the majority of our team have started reviewing code on the command line. So you'll have Claude review the code or the PI hardest review code, or you review this PR, but you don't do a lot of pushback or a lot of commentary on GitHub itself.
07:24 So What we're seeing right now, we're shifting from code being reviewed specifically on GitHub by agents, well, previously humans and now agents, to now code is specifically just getting reviewed in your harness without any sort of oversight or commentary or anything that sort of navigates directly to your shared code base. So with that being said, what I've navigated is I've created this thing called a twin reviewer.
07:51 And this as well is open sourced. So... It is, but I did not have this open. Oh, actually, TwinReviewer is not open source. I realized I never open sourced it because I got sensitive data. But what I've done is actually I set up that same pattern using Fireworks and using Fireworks to post-train specifically. And what's cool about this is I've looked for any sort of pushbacks, apologies, any sort of golden sessions, sessions that actually have been marked as you should probably pay attention to this, and also
08:12 regressions for any things that Usually agents will just fall over. A lot of times it turns into like an agent will start reinventing Go libraries that already exist. That usually is granted to be pushed back. Ideally, you should push back on that. But again, if you don't know these libraries exist, you probably take it for face value and then you're reinventing things like Kubernetes, which sounds like an overstatement, but we've seen it reinvent things like mock libraries for testing and et cetera.
08:51 And very popular code base or very popular open source projects. So that's essentially it. This is basically, if you're going to push back and you're going to have some really, really valuable insight under reviews and your code base, my recommendation is take a look at opportunities within the traces to maybe train or post-train a model or create your own little harness for reviews locally.
09:15 Choose your own adventure. I think you could actually do a lot of this by just building your own harness. But if you did want to use a, which I've done specifically in this situation, you'd also approach that. A lot of the stuff is available to you, a lot of tools. What I did actually to start is I started taking the paper traces and creating skills, skills that I can then specifically do things like check the tapes.
09:40 So if you have a database, which tapes specifically is a Postgres database local on my machine, So you don't have to use a paper console. Actually, paper console is currently waitlisted. So probably best use case is going to use something like tapes or a brain trust or something similar to capture your traces. And then what's cool about this is once you have skills, you can create evals.
10:00 Once you have evals, you can start post-training on a Quinn model and for specific reasons, for identifying if Rust code is up to speed to how you're writing code across your organization. So I did write a blog post around training a model instead of writing a skill. My recommendation is probably write a skill first. You can probably get most mileage out of a skill today.
10:22 And I specifically across my team, we had about almost 80,000 sessions that we could create skills from which is pretty cool uh cool thing about this is also you do specialized fine tuning or direct preference optimization the latter is going to take a lot more compute or a larger some large hardware to do that so probably specialized by tuning uh if you know about quick q or if that's how i'm doing this i'm quantizing it on smaller gpus like a 50 90.
10:48 And if you don't know about specialized fine tuning, this is like writing notes in the margin of a book. If your book is your model, the notes in the margin are your weights, or your adapters, rather. So the weight you generate on top of the model that's already been pre-trained. Cool. And I did this, as you saw, on a Quinton 4B model. It works pretty good.
11:09 I did try to do this on some larger models. So specifically for direct preference optimization, it took a lot of compute. The only time I could make this work is by using NVIDIA GPUs on the Brev. service or actually I never got access to some of the stuff on Fireworks, so I only did smaller 6000s. But if you care to know direct preference optimization, if the margin notes are your specialized fine tuning, DPO is gonna be like your cliff notes.
11:40 Not possible doing a small language model or a small 4B model. You would need like 70B+, which is going to take some hardware. Me specifically, I didn't have any as hardware local, so I had to use the services that I mentioned, Fireworks and, well, not even Fireworks. Brev was the only one I could get access to at the time. Good luck if you can get access to larger, well, you got to prepay a lot for this stuff, so.
12:05 Best of luck on that one. But my suggestion is create skills. Try out tapes.dev if you're interested or leverage the tool that you have at your ready. And reminder, you should probably read this book. This is a great book if you think about, you know, how we're looking forward into... how we're approaching this world when it comes to AI and vibe coding and leveraging agents to basically build your software factory.
12:28 And I do recommend, please check out Adi Asmani's blog post. He has a series of these on the X. I think they're all great. Definitely take a look at them. He's got some clear-minded dots to check out. I'm B. Dougie, and I do appreciate you watching this video. I try to keep it brief because it was pretty straightforward. Hopefully it was. Cheers. Cheers.