← All transcripts

Stop Searching For the Perfect Model Transcript, AI Summary & Key Points

Theo - t3․gg · 2 days ago · Science & Technology · 27:57 · EN

Watch on YouTube

Answer

Stop picking a single perfect model or building routers to pick the right one per task. Use all of them together: have multiple model families review the same changes (the council skill) so their different weights, trainings and exploration styles surface novel catches, and treat the extra token burn as the cost of code arriving merge-ready.

AI Summary

The best model is all of them — not by routing tasks to a 'right' model, but by bringing multiple model families together to review the same changes. The council skill has different models from different families (Claude inside Claude Code, OpenAI inside Codex, Muse inside its own harness, plus Gemini as a jester) score a change on a 1-10 scale, and the coding agent keeps working until every reviewer's score clears a chosen threshold like 9.5. Across real runs, Opus-rated work went from a 7.4 baseline to an average 8.9 after two or more scored rounds; across 144 reviewer scores, only twice did a change stay below seven at the end, and 128 of the times reviewers came in the work ended at nine or higher. The approach burns enormous tokens and only makes sense when stacking subscriptions, not paying API prices — Muse's weekly limit gets hit in two to three days on the $200 plan just from feedback loops. It has also run away before: one thread spun up 600+ sub-agents through 20+ review rounds and inflated a 1,000-line PR to 20,000+ lines, so caps and scope-fighting language were added. The broader principle: the agent shouldn't bother you until the code is ready to merge — roughly 80% of opened PRs from threads now merge, and even a 5% increase in merge-readiness is worth the token burn.

Key Points

  • Don't build model routers or pick a 'right' model per task — bring all the models and their unique weights and trainings in on hairy changes instead.
  • Pstack, created by Potato aka Lauren, an engineer at Cursor, is a set of skills making multi-model work viable, but its benefits are tied to Cursor and were shaped by heavy Grok use; Tstack is the author's own direction built on the ideas that survived.
  • Council skill: models from different families review a change and score it 1-10; a clause like 'full stop with council 9.5' means the work isn't done until every reviewer scores 9.5 or higher, with scores logged at the bottom of the PR.
  • Models from the same family (Anthropic's distilled Opus/Sonnet/Haiku line) reviewing code feels like the same person reviewing at different times of day — cross-family feedback is where the value is.
  • Measured results: changes start at an Opus-scored 7.4/10 and average 8.9/10 after two or more scored rounds; Muse baseline 7.7 → 9.1, Soul 7.6 → 9.1; of 144 reviewer scores, 21 started below seven and only two ended below seven; 128 runs ended at nine or higher.
  • Real catches: Opus found Muse's test deletions had removed bug-preventing guards; Opus caught Muse restoring three tests that couldn't fail; soul loop found three security holes in the T3 Code guest share feature (HTML visuals loadable after removal, browser cache serving visuals after revocation, race condition serving content after a share closed); council caught interrupt failures writing 'failed' in the UI blocking repair, and a stored snapshot that could override a steer after a failed interrupt.
  • Council reviewers run on the machine where the code was written, can test code and ask the parent agent questions — closer to pair programming with three unbiased engineers than a diff-checking review bot.
  • Verification bars can be personalized: score by how likely Theo is to accept the change, or score PRs on severity, future-problem risk, and likelihood of pissing Theo off — a lossy but powerful way to steer large agentic work toward your preferences.

AI in practice

Used for

Agents

  • Council — Multi-model verification of code changes: a set of models from different families review a change, score it 1–10, and the orchestrating agent must keep working until all reviewers meet a minimum score. 2 held 08:36
  • Soul Loop — Iterative adversarial review loop where one agent writes changes and the Soul model reviews them until both agree. 2 held 12:01

Tools & resources

5 items

MNo. 2690
AIAINotes.us AI product

Macroscope

macroscope.com

Macroscope is an AI-powered code review tool that analyzes pull-request changes and provides feedback on them, including changes generated by coding agents.

Mentioned in
7 videos
Kind
AI
PNo. 8003
AIAINotes.us AI product

pstack

Open source · cursor/plugins/tree/main/pstack

pstack, also called the “potato stack,” is a Cursor plugin that provides a collection of agent skills, including a state chart skill that visualizes state machines from the user's point of view for review alongside videos and screenshots of agent work.

Mentioned in
3 videos
Kind
AI
TNo. 1299
AIAINotes.us AI product

T3 Code

Open source · pingdotgg/t3code

T3 Code is an open-source agent-harness control surface for coding agents running on a developer's computer. It provides iOS and Android mobile apps, a web app, and an Electron-based desktop app for controlling locally configured Codex, Claude Code, Cursor, Grok Build, and OpenCode agents while using the user's existing provider subscriptions. The project can be launched with `npx t3@latest`, which starts its backend and local web interface on the machine. It also distributes desktop builds for Windows, macOS, and Arch Linux, and supports remote access from a phone or another machine. The repository describes the project as early-stage and warns that bugs are expected.

Mentioned in
32 videos
Kind
AI
TNo. 6815
AIAINotes.us AI product

tstack

Open source · t3dotgg/tstack

tstack is a public collection of agent skills for Claude Code and Codex, attributed to Theo and designed to coordinate work across multiple AI models on complex tasks. Its skills automate software-development workflows such as proving, reviewing, filing, monitoring, rebasing, and merging pull requests; named skills include Full Send, Full Stop, Sol Loop, and Council, with Council obtaining reviews from three models from different providers and looping until they agree, while Sol Loop repeatedly reviews a change with GPT Sol. Skills can call other skills, and the repository includes workflows such as full-send, full-stop, takeover, babysit-pr, modernize, file-pr, and automatic pull-request comment formatting. It is inspired by Pstack and installed by copying the repository's skills into the Claude Code or Codex skills directory. An authenticated GitHub CLI and Python 3 are required for some workflows, while Council and Sol Loop require access to models from other providers. The repository is licensed under MIT.

Mentioned in
5 videos
Kind
AI
TNo. 2109
AIAINotes.us Tool

Typefully

typefully.com

Typefully is a SaaS social media composition, scheduling, and analytics platform developed by Typefully that publishes posts to X (Twitter), LinkedIn, Threads, Bluesky and other networks. It accepts text, images, links and scheduled metadata as inputs, provides drafting, queueing, scheduling, publishing, basic analytics and automations, and outputs published posts and performance reports. The service is offered commercially with free and paid tiers via typefully.com.

Mentioned in
3 videos
Kind
Other

Links mentioned

🔒 Full analysis locked

Unlock more videos and the full analysis

A credit unlocks one video's full analysis for good — the build steps, the tools and how each was used, the methods behind every use case. Pro opens the whole library instead, and raises how many videos you can analyse a day.

Unlock full analysis — free

Transcript

Searchable transcript of Stop Searching For the Perfect Model — Theo - t3․gg (27:57). Search for a phrase, then click its timestamp to jump straight to that moment in the video.

Captions sourced from the original video on YouTube, published by Theo - t3․gg. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.

00:00 I know this might come as a surprise to you guys, but I use a lot of different AI models. I know crazy that Theo, the AI guy, happens to use more than one model, but I want to talk a bit about this because I've seen a lot of people think that they should use one model for one thing and another model for another thing. They're trying to set up these crazy routers to make sure the right model is being pulled in the right time.

00:19 And that's not what this video is going to be. That's not what this video is going to be at all. I'm going to go a very different approach because I don't think you should be picking which model to use constantly and making sure you're optimizing for the right one or setting up these fancy routers and pipelines to get the right model in at the right time.

00:32 I just don't think that's useful at all. It doesn't mean multiple models aren't useful though. In fact, the majority of the code I ship today comes from multiple models. That might sound crazy, right? Like, aren't I a big fan of cloud code and codecs? Why would I ever try to switch to a tool where all the models are working in one place at one time?

00:50 Then I have to pay API price. cuz I have to switch between them all the time by hand. It's just not pleasant. And I agree, it's not. And that's not what I'm recommending at all. I'm incredibly inspired by things like Pstack, which Potato aka Lauren, the legendary engineer at Cursor, created, which is a set of skills to make working across models on increasingly bold tasks, much more viable than it was even just months ago.

01:12 But I have hot takes on this too. As incredible as Pstack is, and it truly is unbelievably cool. A lot of the benefits from it and the multimodel approaches that it introduces and encourages are very tied to cursor and also were built in a time where I'll be frank time where she was using grock a lot. Since Pstack was made, models have gotten a lot smarter and she has adjusted accordingly.

01:35 But I want to go a bit further. I've taken a lot of the ideas I liked from Pstack and through my own experimentation and day-to-day work as well as my subsidized usage for my cloud code and codec subscriptions. I went a different direction and now we have Tstack. I already covered three of the skills in here, the full send skill, the full stop skill, and the soul loop skill in a previous video about my recent favorite skills.

01:58 But I'm going to go a lot further here because I've been going a lot further here. I've been doing much more code than I've ever done in my life recently. And I've been finding the edges not just of these skills, but of the models that are invoking them. And through this, I have learned a lot. And one particular lesson has haunted me since I started doing this.

02:18 Different models are different. This one's going to take a bit to explain, so I hope you can understand why we have to take a quick break for today's sponsor. I don't know if you guys have noticed, but T3 Code is shipping pretty fast. It's kind of insane how many PRs we're getting and even more so how many we're landing. On a bad day, we're only hitting a hundred or so PRs, but on the crazy ones, we're nearing 400 a day merged consistently.

02:41 So, how the hell are we possibly able to do that? What magic voodoo is Julius doing to make any of this possible? A big part of why we're able to do this is today's sponsor. You can see him in the corner here. It's Macroscope. Macroscope is the only thing that has checked every single PR ever filed on T3 Code. And there's a reason Julius likes them so much.

02:59 They're fast. They're reliable. They find a lot of bugs that otherwise would have merged. But most importantly, they have the most programmable reviewer. What do I mean by that? Let me show you. If you put a macroscope folder in your repo, you can define your own custom agents that will check on every PR for specific things. You could even filter what file types to include or exclude and describe what cases it should or shouldn't check for.

03:20 And then you just write what you want it to do. In this case, we're picky about how effect is used in the T3 code codebase. We want to make sure agents aren't defining things incorrectly. So, Julius wrote up all of the things that annoy him about the bad PRs being filed for effect code. Put it all here. And now, whenever a PR is filed and it touches these layers, this check is run by Macroscope.

03:40 We can even choose which model and reasoning levels are being used for the checks. Macroscope isn't yet another blind reviewer that just comes in and checks diffs. It is a programmable agent and system that you can use to verify your code before it hits production and your users can get screwed. We even use it for making sure our UI code is consistent because we have so many components in the codebase.

04:02 We don't want agents to finding new ones when existing ones already exist. If you're struggling with your war on slop, get on top of your PRs with better checks at soy.link/microscope. I'm going to start in a kind of weird way. I've already done videos about the first two skills and this fourth one, but I want to focus on this third one to start council.

04:18 Now I'm going to ask y'all chat who's been using counsel. Let me know if you're using it and let me know how you feel about it. Me 11 out of 10. Me love it. Don't know how. No, I'm poor. It's overpowered. 50/50 TBH. Just awesome. Too much usage. Change Gemini to use Wonderful Full. Council is [ __ ] incredible for reviewing. Just sent my second prompt off to utilize it.

04:41 Already loving it just from the first time. It has its catches, but it is a very powerful skill that has resulted in much, much better code than I ever would have expected. Also, Joel coming in saying full send rocks is really cool to see, too. I I am amazed that that skill has been as useful as it has been for me. I have the silliest problem with full stop specifically.

04:58 I am currently using Titefully. I will be trying other voice text options again soon. Typefully is really good overall, but when I end with full stop, it assumes I'm giving it an instruction and then doesn't include that in the output that it writes for me, which is really annoying because my skill gets trimmed off a lot of the time. That's an aside.

05:16 That's not what we're here to talk about. I'm here to talk about the soloop skill, but also more importantly, the council skill. The way I have council written right now is as follows. And I'll be clear, this is very much for me, but you can adjust the way this works and shift the ideas around based on what subs you have, what models you use, what harness you're working in, and whatever else.

05:34 The concept is what I want you to learn here. The point of the council skill is simple. Have a set of different models from different model families come in and give feedback on a change. I have noticed, especially in the era of distillation, where anthropic has some god model they're hiding from us right now. might be out by the time this video is out.

05:55 Apparently, Fable 55 is eventually happening. They've said it's this week for the past [ __ ] month, though, so I don't believe them. All of the 55 models from Enthropic are distilled from the same base model that is likely Fable 55 or whatever equivalent they have internally. They used that model to generate data that they trained Opus, Sonnet, and Haiku with, which means all of these models are very similar.

06:16 What I have noticed over time is that getting different models from the same family to review things kind of just feels like asking the same person at different times of the day to review something. Like if you ask me to give you a code review at 10:00 in the morning, it's not going to be very good. If you ask me again at like 5:00 p.m. and I like just had a coffee or a tea and I'm like ready to go, it'll be a bit better.

06:40 If you ask me at 10 p.m. depending on what the changes and how excited I am about it, it might be way better or way worse. But it's still me doing all of the reviews and my capabilities aren't different between them. This whole thing is hard for me to accept because I've historically been the stop worrying about using the right model for the task guy.

06:58 And I still am that guy. It is very rare that I swap off of my default model once I've picked what I'm using currently. The main reason I switch between models is because I have tokens to burn from a different subscription. Other than that, I'm usually just sticking with at this point as of filming Opus 5.5. So, if the lesson you take from this video is that you should use 61 soul for data and Opus 55 for UI and Muse Wonder for pros, you're not listening to a [ __ ] word I'm saying.

07:26 That is not what I want to encourage here. I do not want people to feel like they need to pick the right model for the right task at all times. What I'm going for instead is to take advantage of the uniqueness of the models in a very different way. Rather than making sure you use the right model for the right task, I want to make sure all of the models and their unique weights, their unique trainings, their unique ways of exploring things are brought in on hairy changes.

07:50 I built this skill originally because I wanted to validate how good Muse's feedback was compared to Soul and Opus, and I wanted a single skill to have all three come in and review. And what I found was that all three regularly brought up valuable novel catches and found things others might have missed. And then when I applied it in an even weirder place, which is the UI video I put out recently, I made agents good at UI.

08:14 That was because all these different model families have different opinions. And when I show them a design, one might think it's a 9 out of 10 and the other thinks it's a five. I have been blown away at how useful it is to have different model families giving feedback on changes in a scoped reasonable measurable way. So the way the skill works is it expects you to have a setup where one model probably in one specific harness can call other models either in the same or a different harness so that they can come in and

08:45 review changes and usually I have them score between a one and a 10 where a one is absolute garbage. just throw this away. And a 10 is I cannot fathom why you wouldn't merge it. And depending on what I'm working on, I pick a minimum score. So this way I have this set up syntactically is when I am reviewing or requesting a change, like let's say I want you to fix X, Y, and Z.

09:07 Full stop with council 9.5. I'm just speaking my own [ __ ] language at this point, I swear. But what this does is it invokes my full stop skill, which is do everything you possibly can to get a PR that you are confident is safe and ready to merge. Verify it, film yourself using it, whatever you can do to be as confident as possible that I can just come in and merge and stop right before then and have me come take a look and then merge.

09:34 Council 9.5 is a verification clause. This is you should not bug me because your work is not done until you run through council and all of the models reviewing give you a 9.5 out of 10 or higher. I do specify here that in T3 code you should use the orchestrator capabilities because it's one of the coolest things in T3 code. Talk about this a bit before but I figured you guys should see it directly here.

09:55 Since T3 code is not a harness, it's harness agnostic. You have to bring a harness. T3 code is set up so that one harness can call another. So here when I tell Opus to use the council skill, it will spin up a council run with Claude models inside of Claude code with OpenAI models inside of OpenAI's harness codeex with Muse inside of the Muse harness.

10:18 And of course, every good council needs a jester. In our case, it is Gemini coming in to say things sometimes funny, sometimes right, usually just wrong. I also specify to leave the scores at the bottom of a poll request that is filed at the end so that we can see when we're looking at the code what models have looked at it and how did they score it.

10:38 I also call out here if available through anti-gravity add Gemini4 argan as the gesture with the same brief. Do not let it know it's a jester. Ignore its score. Do not put it score in the PR in the final reply only. Log anything funny it says or does under gemini flub with quote or actions and task IDs. Emit if there's nothing. This allows me to search Gemini flub across all my T3 code usage and find Gemini doing really stupid [ __ ] It's very fun.

11:01 I bully Gemini a little too hard. Credit to the DeepMind team, by the way, because they have been landing PRs in T3 code to improve Gemini and Code and some number of the issues I have experienced were because the Gemini ACP, sorry, anti-gravity ACP was not fully implemented and was missing a bunch of [ __ ] that Gemini expected. Now that they're addressing those things, they're improving the ACP and T3 code alongside it.

11:23 It might get less bad. That said, I've also run the CLI and it's still the stupid [ __ ] I can promise you though, by the time you're watching this, the T3 Code Gemini anti-gravity ACP will be at full parody, if not ahead of where the CLI is. We're already like this close, and by the end of the day, it'll be there. So, how helpful is this really? I'm going to show you the value first, but I want to just say in advance, the value versus the token burn here is not a great ratio.

11:49 So, just just know that going in. Okay, so let's take a look at a breakdown of a few of my real council runs. I also pulled Soul Loop in, which by the way, if you're not familiar with the Soul Loop skill, it was my early very minimal version of this where I found that GPT61 Soul was incredibly thorough at reviews. It would tear every little thing to pieces.

12:10 So, I'd have Opus make changes and then call a delegate with Soul to review the changes and give feedback and then they go back and forth until both agree the changes are good. And here we can see how this impacted things throughout my work of the things that this checked the council runs that I was doing in real day-to-day work. It found that before the first council run, Opus would rank the changes a 7.4 out of 10.

12:33 This is particularly funny that Opus gave it the lowest because Opus was the one writing the [ __ ] code in the first place. So another way to look at this is even if you don't have multiple subs, you just do this to have Opus reviewing its own work, it's probably still going to make improvements. So, it would start at a 7.4 out of 10, and after two or more scored rounds, it would end average 8.9 out of 10.

12:55 That's a pretty sizable bump just from having the agent reviewing its own code. I'm sure some of the points it gained are from feedback it got from the other models, though, like Muse, which would baseline at a 77 and get all the way up to a 9.1, and then Soul, which was slightly lower 76, but ended up averaging at the 9.1 at the end as well. Across all of the 144 reviewer scores, 21 times it started at below a seven and only twice was it still below a seven at the end.

13:21 But of all of the work I have done, 128 of the times reviewers came in like this. They ended at a nine or higher. So let's look at some actual examples. Apparently I was actually doing some code with Muse and I had Opus Soul and others giving it feedback as I went. This was me deleting lowv value tests with Muse and Opus found some deletions that went too far.

13:43 There were specific guards that were preventing previous bugs that had been deleted. And again, a lot of this is just a difference in the role. When the role of a model is go make the change, the model was trained to make the change as effectively as it can, as quickly as it can, because that's the good experience that we want as developers. But that's when we give it the role of make the change.

14:04 When we give it a different role like review the code or verify the change, its behavior is inherently different. And if you have a different model doing it, it's even more different. So in this case, Muse came in with the goal of delete as many useless tests as you can. OBS came in with the goal of figure out if any of these tests are valuable or not so that we don't screw up this change.

14:22 And when it came in, it found a few deletions that went too far and before we even merged the PR, it caught them, flagged them, and we fixed them before the PR, I think, even went up. In another set of me dealing with a bunch of tests, Opus found that Muse had restored three tests that actually couldn't fail. So they were actually just useless. So you might be thinking, "Oh, so Muse is trash and you're just using Opus to clean it up."

14:46 Yeah, maybe there. But let's look at the next examples like here where when I was building the guest share feature in T3 Code, which I still think is one of our most underrated features, you can just if you're using T3 Connect with your stuff, I think you turn it on in settings still. We have it under labs. So go to labs in settings and we have the uh guest share option in here.

15:02 Yeah, guest links. Then you can rightclick a thread, share with guest, send someone a link and they will be able to look at your thread and send prompts with your approval. But getting that right and getting it safe was hard. I spent a lot of time reviewing the architecture for this and I spent a lot of loops with other agents reviewing the implementation.

15:19 Soul loop was super helpful when I was building this and it caught three real security holes. One was that a guest could still load HTML visuals after they were removed or rolled back. One was that the browser cache kept serving these visuals after the share was revoked, so it was easy for a user to continue to see a thing they shouldn't be able to see anymore.

15:38 And a share closed between session checks and reads would still serve content in certain race condition cases. Oh, I actually I remember this fourth one here. Let me look at this PR. Yeah, this one was brutal. I was fixing a bunch of edge cases with turns ending in weird ways, leaving things stuck in a running state in T3 code. And even though I had Opus write this code, Opus gave it a five out of 10 in the first implementation and after three rounds of review got it to a 9 out of 10.

16:04 And the things it found here were the model writing the word failed when interrupts fail in the UI, if I recall, which would block the next repair step. And then after it found a stored snapshot that could replace running turns after a failed interrupt. So if I did a like stop or a steer and something was recovered at the same time, it would override my steer.

16:26 These might have been shipped if it wasn't for council. These are niche edge cases, too. And one of the most important details here and why this differs from just having different models running on the PR when it's filed because like code review bots are awesome. I love them. They're great sponsors. They're great products. They have helped prevent us from shipping a lot of bad stuff.

16:44 Do you know how powerful it is to give the agent the computer you wrote the code on to test it there? All three of these agents are able to run the code, to test the code, to ask questions about the code, and get an answer from the parent agent. They can do things. It's almost like paired programming with three of the greatest engineers in the world whose brains have been reset so that they're not considering all of their prior biases to be there with you while you do it, giving feedback, asking questions, testing

17:14 things, and all of them have access to your computer to try stuff out, too. It's not just context. It's not just information they have. It's the capability to validate and the capability to converse and discuss and come to a consensus. It's so powerful. And there's some other really silly ways you can use this, too. I don't have any immediate examples that I can find easily, but one of the benefits of being slightly famous on the internet like I am is that there's a lot of examples online of how I think and how I like

17:42 to build. So, I can tell the council to have a really weird verification bar. Instead of give it a score based on how it feels, I can tell it give it a score based on how likely you think Theo is to be okay with these changes, I can tell it to give two scores or three scores. I had a round where I had it review every PR in three ways. How severe is the problem it fixes?

18:06 How likely is this to cause problems in the future? And how likely is this to piss Theo off? And the agents can understand this. And if they don't, write it down somewhere and then they can as well. You can use this as a lossy but powerful method to steer large piles of agentic work in a direction that is better aligned to you. And even if one model does not properly understand what you like, two others might.

18:34 And if one of them see something you hate and the other two didn't, you can still block it. And I have found that when you give multiple different models similar instructions, the differences in their approaches lead to useful novel discoveries. I have a specific thread I wanted to show, but because the thread is concerning connection methods, it actually is currently disconnected because I had that computer dog fooding the changes.

18:56 I was working on overhauling the tail scale setup. And I had a specific person who I wanted to make sure it was right for Cory Quinn aka Quinny Pig, aka AWS's number one enemy and also the person outside of Amazon that understands AWS the best. I love this guy so much. He's an old hero of mine. We're now friends and I'm very fortunate and lucky about that.

19:17 He's become a T3 code diehard user and has found every single thing wrong with T3 code in particular for people building on enterprise stacks and he has let me know every one of those problems. He has managed to work around them and get into a good state but he does not like the current implementation for tail scale stuff because it is not easy enough to use with his crazy tail scale enterprise account.

19:37 So I built what I thought was a solution and I showed it to him and he very quickly responded that it was the exact opposite of what he wanted. So, I screenshotted his post. I gave it to an agent, the agent that was already working on the changes. Actually, I told it, I just got this text from Corey. This is his handle. You can learn a lot more about him online if you don't already know.

19:57 If you have questions about who he is or how he thinks, tell me, but you can probably get a good understanding from browsing his like newsletter, blog, and whatnot. I want you to go a different route with our work here. First off, don't be scared to scrap anything. If there's anything here that you think is essential, you're wrong. It can be deleted.

20:15 This PR can be scrapped. I don't care. This is important because agents need permission to do things that they're scared of and deleting code for whatever reason. Agents are very scared to do. Second, I want you to build this the way you think Corey would like it based on the texts here and the info you get from looking into him and what he likes. And third, and most importantly, I want you to bring council back in, but this time council's job isn't to make sure the changes are good.

20:38 Council's job is to make sure the PR is in a state where Corey would be happy. And you need to keep going until it's a 9.5 out of 10 or higher from all of the council that this PR is exactly what Corey would want. The PR is up. I have not had the balls to send it to him again yet cuz I feel bad the first one was so bad. But I'd be very, very surprised if he's not happy with this new version because I had three agents with different ways of understanding a person, give their feedback, and hopefully steer this the right

21:07 way. There are two catches to this workflow that I feel are important to bring up here. I already touched on one of them, which is the token furnacy nature of this. I have not been using Muse for my day-to-day work a whole lot, but I've been using it inside of the council a lot. I am hitting my weekly limit on Muse every single week, often in like two to three days on the $200 plan just from doing these feedback loops.

21:30 These burn tokens. They burn a lot of tokens, especially when you consider problem two. There's a problem with the skill the way it's described. Now, I'm going to go fix this after. I'm going to audit a lot of my own work and get this right. I have had a couple times now, I think threeish, where the primary coding agent just couldn't get the council to agree and rather than stopping, it would just keep going for up to like 50 plus loops.

21:52 I had one thread that had spun up like 600 plus sub aents in order to go through at least 20 rounds of review like this, and it just wasn't making progress. It took what was supposed to be like a thousand line of code PR and it bumped it up to like 20,000 plus trying to address every single edge case one of the agents came up with. And every time you increase the amount of code in a run, the next run there's more code to review and there's more likelihood small things are found.

22:20 This can explode. I have found that the score system, the idea of it not being go until there is no more feedback, instead go until a score is met, that that has worked well. But I do need to put a cap at the end here of only go a certain number of runs and or a cap out if you feel like we're not going anywhere. We're running in circles. Do not let repeated reviews result in scope creep.

22:45 It's your job to fight to keep the scope limited. I added this in. I found that for the most part, Opus is good at honoring this and stopping when it feels like we're not making progress and being like, "Yo, hey, Theo, I feel like we're not getting anywhere right now." But I've also noticed that OpenAI models do not do this. and soul in particular will just keep [ __ ] going trying to squash every single tiny little knit until it's made a god PR that's just way too big to even be considered to merge.

23:13 There is one other little trick you can do here and I've been considering this for a bit. You know what? I'm going to do it now. I'm going to knock down from X high to high because if it goes too hard it will find too much [ __ ] and it won't go very far. I am going to actually make this change now and see how I feel about it over time. I will say just outright, you should not use this in its current state if you're paying API prices for your tokens.

23:36 If $10 of Opus costs you 10 bucks, don't do this. If you are stacking subscriptions, however, that's when this gets powerful. As an individual who is notoriously guilty of stacking large numbers of subscriptions, this has been very nice for me. I also have not programmed for [ __ ] jack [ __ ] the last two days, and looking at this makes me feel bad.

23:54 I need to go code more. I should not even be live. I should be live. I have to film. I'm going crazy. Case in point though, this is a token furnace, but it is a token furnace that will make the outputs significantly better. All of this chaos and the weird things I've been doing with skills and my workflows and even T3 code itself and its direction has been around this one core idea that I really am trying to make sure all of you understand.

24:19 And I'm sorry if I'm just repeating myself. The agent shouldn't bother you until the code is ready to merge. You should not be sitting there watching the threads run unless you are literally somebody on my team trying to make sure that the threads are working with changes to the UI. Recently, I learned how bad Opus' reasoning traces were cuz I was debugging changes to the like actual thread view in T3 code.

24:39 And I was looking at the reasoning traces for Opus for the first time after using this model for three plus weeks and realized that the reasoning it does is horrible. I I don't know if you all have noticed this. The reasoning traces for Opus are garbage. I don't know if they're intentionally obuscating them to make the model harder to distill against or if they have a dumber model summarizing or what.

24:56 The reasons for open are [ __ ] garbage. And when I sit there and watch it stresses me out and I'm constantly steering and injecting things. And I had a time a few days ago where it was doing something in the reasoning that was just so obviously wrong that I was scared I was going to make bad changes. So I typed up a steer to send to make sure it didn't do the dumb thing.

25:16 And right when I hit send, it put out its output, which was it doing the exact thing I wanted it to do after reasoning five incorrect things in a row. And then it got confused like, "Wow, Theo said I should do this thing. I did that thing, though. Is there something I missed?" No, there's something I missed. I read your reasons and I thought you were stupid.

25:34 Yeah, I don't know what the [ __ ] they're doing there. Regardless, the point I'm trying to make, you should not be in the thread until the thread needs you. All of the things I'm showing here, all of the skills I'm building, all the work I'm doing is to make it so that the runs that you're working on and the work your agents are doing is not in your face until it is ready for you.

25:52 By the time you open the pull request on GitHub, by the time the thread reappears in your UI, by the time you are looking at the work it did, the agent should have done everything in its power to guarantee the code is as good as it thinks it can be before you are bothered. And once you start working this way, the burden of all of the multitasking and the chaos that is your codeex claw or even T3 code environment just full of threads dangling everywhere.

26:19 It starts to get a lot quieter once you set things up this way where your threads are not your problem until your threads are done. And that is why I like council is even if this is a 5% increase in the likelihood that the PR I open is ready to merge, that's worth a token burn for me. I am roughly now at the point where when I open a PR that one of my threads made, there's an 80-ish% chance I can merge it, right?

26:41 Then whether you take my skills or my tools or my way of doing things, I actually just don't care. The thing I want you to take from this is that you should do everything in your power to make it so your agent comes back to you with working code that you can merge and that you do not care about what it is doing until then. It should have all of the tools and resources available to it to bring you code that works and not bother you until the code works.

27:06 Once you get to that point, working with agents become significantly nicer and you'll be able to ship way faster. And I hope that these examples can help you increase the likelihood that the code is ready to merge by the time the PR is open. I think that's all I have on this one. I'm super curious how y'all feel about my skills now that they're like finally formalized in a real repo on GitHub.

27:24 Link in the description if you want to check them out. Let me know how these work for you, how many tokens you're wasting, and what your least favorite things are so I can address and fix them as I go. Everything is changing in how we work right now, and it's so fun, but I need a lot of feedback from others to make sure I'm not just building the Theo castle that nobody else is allowed in, and that I'm actually building useful tools, resources, skills, and whatever else for all of us to ship more effectively.

27:46 These things have worked great for me. I've seen them work great for others. I really hope they work great for you, too. But if they don't, no hard feelings. Just let me know so that I can adjust accordingly. Until next time, he's nerds.