How I Manage AI Agents at Scale (Prompting Is No Longer the Problem) Transcript, AI Summary & Key Points

00:07 Agents manage agents, and those agents manage workers of their own. That's how most of my work gets done these days, and the whole team is made entirely of disposable context windows. Now, that last part changes everything. With one agent, I write the prompt. With dozens, prompts stop being the problem. The problem becomes what each agent is allowed to know, who keeps the plan, who sees the logs, who starts with a clean slate.

00:37 That is the same problem every every manager has, and it turns out to be most of what makes a large agent team work. That is what this video is about, how the way we instruct agents changes as we give them more autonomy, more work, and more responsibility. By the end, we will get to where I think this is going next, systems that decide how much organization each task actually deserves.

01:05 A team made of disposable context windows has a few weak spots. When a session ends, everything it learned goes with it, and nobody nobody checks the work except the agent that did it. That's where the sponsor of this video comes in. Verity is a pair programmer for coding agents. Every time Claude Code finishes a task, Verity hooks in and checks it for quality, security, and whether the code actually does what was asked.

01:35 It mixes independent AI review with secret independency scans and runtime analysis without touching your token budget and hands the fixes straight back to your agent. With their own benchmark, it finds 2.3 times more real bugs than the most popular PR review tools, including race conditions, deeper logic errors, and vulnerabilities. Now, favorite part is what happens after the fix.

02:02 Every decision lands in a markdown knowledge base that future sessions learn from, so your agents stop repeating old mistakes. Verity is live for CloudCode right now with a free plan and a two-minute setup. The link is in the description, so big thanks to Verity for sponsoring this video. And now, let's go back to the beginning with one agent and me sitting right beside it.

02:32 I started with one agent. I had to approve at every step before I could add more. I had to trust that one enough to leave it alone. Instructions and skills made it reliable, and the pipeline caught whatever slipped through. But trusting agents does not mean letting them decide everything. Leaving them alone forces a different question. Which decisions belong to agents and which belong to everything around them?

03:02 If I want the agents to keep going without me, they have to own the decisions I used to make every time something went wrong. Why did this test fail? Is the problem in the code or in the test? What should they try next? If those questions, if those questions keep coming back to me, I am not delegating. I'm micromanaging. So agents should decide why a check failed and how to make it green.

03:29 What they don't get to decide is whether required check matters. That's the one decision that would let them declare victory on broken code. A red check means the work isn't done, no matter what the agents think. Now, agents can follow a strict workflow or they can find their own path through the problem. Usually, it's a mix of both. And just like with people, both extremes fail.

03:54 If the workflow is too rigid, agents follow it whether it makes sense or not. If there's no workflow at all, they do whatever they want. So, I keep the rigid parts for outcomes that are non-negotiable, like required checks, for example, and leave agents free wherever their judgment adds value. Once my agents could own those decisions, I stopped stepping in after every attempt.

04:21 It decides what the change needs. A code change might mean implementing, testing, inspecting failures, and trying again as many times as it takes. A fix to a typo in the docs doesn't need any of that. What it cannot skip is the process around the change. It doesn't push directly to main. It opens a pull request and the checks there run whether it thinks they are needed or not.

04:46 But the same context was still doing everything. It contained the original request, repository exploration, implementation decisions, failed hypotheses, logs, test output, review comments, and every repair attempt. Eventually, the context fills up and the agent compacts it, summarizing what happened so far and dropping the details. Now, that's not a disaster.

05:12 Modern agents recover from it surprisingly well, but stuffing every detail into one conversation is still a wasteful way to use context. So, the next step was not a better prompt. It was separating the work. Think about what a human manager does. They keep the objective and the constraints. They break a project into tasks and decide what can happen in parallel.

05:39 They give each person the context needed for their assignment and they get summaries back instead of sitting beside every employee for every keystroke. The manager does not need the entire debugging history from every developer. They need to know what was decided, what changed, what evidence supports it, what remains unresolved, and what should happen next.

06:02 Everything Everything in this video is about principles and they apply whether you use the tool I'm about to show you, another one or a handful of agents or when your own scripts. Doesn't matter, right? What you're about to see is Agent Deck. It's an open source application I built for running and managing many coding agents and at once. If you want to try it, the link is in the description and if you find it useful, start it.

06:27 Start it. Put a star over there on GitHub. It helps the more people find it, right? Now, an agent orchestrator should keep the same things in its context as a manager does. The goal, the plan, the current state, decisions, delegations in flight, results, evidence, open questions, not 50,000 tokens of compiler output from a worker that already fixed the problem.

06:52 So, the orchestrator stays for the whole task. The detailed work goes into disposable contexts. The orchestrator delegates one bounded operation. The worker starts fresh and it receives only the relevant brief, performs the work and returns a concise result. The next delegation can start from another clean context. That is why I call those agents specialized, let's say.

07:19 The important specialization is not the job title, it's the context. Coder receives implementation context. The tester receives the behavior that must be demonstrated. A reviewer receives the requirements and the resulting diff, but not the long conversation that convinced the coder its approach was brilliant, let's say. Separating contexts changes the quality of the result.

07:45 A single agent usually implements the task once and then evaluates its own work from inside the same reasoning trajectory. If it misunderstood the requirement, its tests and self-review can easily preserve the same misunderstanding. With a team, the coder implements the change and reports back to the orchestrator which brings in a reviewer with a fresh context to try to break the argument.

08:11 The reviewer reports finding to the orchestrator. The orchestrator sends accepted findings back to a fresh coder context. The code changes, the reviewer uh looks again and the loop continues over and over and over again. Eventually tester provides a different kind of independence. In my current workflow the coder owns production code and its unit tests.

08:35 A separate tester owns functional and end-to-end behavior. The tester knows what the feature should do but does not inherit the coder's explanation of why the implementation must already be correct. Either agent can be wrong. Two agents can be wrong as well. They can be wrong together. They can even waste time arguing about style. A disagreement becomes valuable only when it tends in stronger evidence.

09:03 A producer, an executable test, a requirement, a repository invariant, or a product decision from a human. A reviewer label is not evidence either. A passing check is and even that proves only what the check already covers. That is why the orchestrator is not just a message router. It decides whether a finding is valid, asks for evidence when it's not clear, and escalates when the disagreement is really an unresolved product decision.

09:37 I also prefer different model families for roles that should challenge each other. A coder and a reviewer running the same model tend to share the same blind spots. Agents on the same model are closer to clones than colleagues. Different models now, to be honest, are not truly independent either. They learned from much the same material and they read the same repository but they raise the odds of um generally different interpretation.

10:07 And that matters most for architecture, security, concurrency, and anything else, where several answers can look reasonable. Now, all of this, all of this is slower than asking one agent to take one pass, and it is more expensive. Research comparing a team of agents with a single agent at the same inference cost mostly doesn't show teams winning. So be skeptical of anyone claiming a swarm is inherently more intelligent.

10:38 But equal cost is not the question I'm asking. Cost and time matter. They matter to me, but quality matters more. I'm deliberately spending more on additional attempts, fresh perspectives, and independent verification because in my experience at least, that's what produces better software off on at least on complex work. So could one agent spend the same extra budget just as well?

11:08 Maybe, but it would spend it inside one context reviewing its own reasoning. And since since I'm not the one watching, the extra time does not really matter. It's just happening in the background. The manager analogy stops here, right? Human managers can rely on tacit knowledge, shared culture, relationships, and people remembering what happened last month.

11:33 Agent orchestrations cannot. They need artifacts. Plans, task files, commits, tests, pull requests, CI results, and decision records are the organizational memory. If an orchestrator compacts, it can read them. If every process crashes, I can just start another orchestration in the same work tree. The agents can resume their sessions, but they don't depend on them.

12:01 They reconcile what Git pull requests and CI actually show and just carry on. Some runtime bookkeeping might need to be rebuilt in those cases, but the the work is not trapped inside one irreplaceable process. A human organization survives because its knowledge is not stored entirely in one manager's head. An agent organization should work the same way.

12:31 An orchestrator manages one task. It has nothing, nothing to say about what happens between tasks. Someone still has to choose the next one, prepare an isolated environment, decide whether it needs one agent or a team, start it, monitor it, and keep the machine busy. If that someone is me, I'm still the scheduler. That is where the dispatcher agent comes in.

12:54 It is a manager of managers. It does not implement the issues, it manages the portfolio. With agents, a zero issues policy becomes possible, at least in theory. The moment a problem is found and filed, it just gets fixed. The moment I write down an idea, agents start working on it. What stops me isn't the agents, it's cost, capacity, and myself, me.

13:21 I have neither unlimited hardware nor unlimited tokens, and even if I would have both, I couldn't review everything they would produce. So, the dispatcher works within those limits. It runs as many issues in parallel as the machine can handle and fills each free slot as soon as one finishes. Which issue goes next matters less than it seems since all of them should get done eventually.

13:49 What matters more is whether an issue is ready to be worked on at all. A skill in the repository defines what that means for this project. The dispatcher fetches the current backlog, removes PRDs and issues that are already in flight, checks for duplicates and coupled work, verifies that an issue's premise is still valid, and claims each issue before dispatching it.

14:10 That selection work goes only as deep as the decision needs. The dispatcher reads the issue and inspects the relevant source code to judge how complex the work is and whether it divides cleanly. That alone is enough to decide how the issue should be executed. It does not design the fix or fill its context with implementation details. It does none of those things.

14:38 Those belong to the issue unit. So for each selected issue it chooses between one agent and a team. The criterion is not simply size, it is divisibility. A change concentrated in one tightly coupled area usually gets one agent even when it's large. Splitting it would create coordination without creating useful parallelism. A complex change that separates cleanly into implementation, behavioral testing and review gets a team.

15:07 When the decision is close one agent wins. Every issue gets its own branch and work tree. That is the actual actual unit of parallelism. Different issue units cannot all write each other's files and one failed experiment does not dirty the checkout used by another. So a simple unit starts one general agent with a self-contained task. A team unit starts its own issue level orchestration and workers.

15:37 The orchestrator can hand work to any available worker including several at once. Parallelism inside one work tree still takes judgment. A reviewer and an auditor can look at the same change at the same time. Neither of them writes anything. A coder and a tester can also work side by side as long as they own separate parts of the plan and don't step on each other.

16:05 And separate means more than different files, right? They still share git state, build output, services and text fixtures. Parallel work needs effects that don't collide, not just I don't know different file names. And then when a unit completes only its report returns to the dispatcher. The key hierarchy keeps each context focused. The day level dispatcher knows the queue and the outcomes.

16:32 Each issue orchestrator knows one issue. Each worker knows one operation. The details move downward only when needed and concise evidence moves upwards when the work is done. And at that point I am no longer managing organizations. I'm managing capacity, priorities, exceptions, and outcomes. So what should agents follow and what should they decide for themselves?

16:59 The question, right? I think about it in in in three layers and they apply to agents almost as much as they apply to people. First, there are a few things that must happen no matter what. In my projects, for example, there are two of those. Changes go through a pull request and a pull request doesn't merge until the required checks are green. Yours might differ, right?

17:23 Maybe a security scan or a human sign sign of for anything that touches payments, let's say. What they have in common is that agents shouldn't be the ones deciding them about their own work. And that list has to stay short. A good test is whether a rule needs unless this and that, right? Always run the tests. Sounds reasonable until the only change is in the docs.

17:53 Then it becomes unless it's in docs, then unless it's a config change. And soon you're trying to list every permutation. If a rule needs exceptions, it probably isn't a rule. And if it's something we keep finding ways around it's worth asking why, right? Sometimes it's it's just inconvenient and it should stay. More often than we would like to admit it's it's a stupid rule.

18:23 Second, there are best practices. Plan before you code, get a review, show the behavior the user will see. valuable most of the time, but not all the time. And whoever knows them well also knows when to skip them. They also depend on who does the work. Nobody tells an experienced engineer to plan before coding, right? No, nobody. But we might want to see a plan from a junior before they touch anything.

18:52 And those practices don't last forever. For example, I followed most of the Gang of Four design patterns at one point in the past. Today I rarely reach for them. Planning went through both changes with agents. I used to force them to plan all the time, all the time. Today they plan by default, and I I rarely bother even to switch on planning. A practice that had to be spelled out last year can be noise today.

19:21 So be prepared to review, measure, and update best practices continuously, or they might they might do more damage than good. And third, there is local experience. In this project, we do it this way for this reason. Models already have plenty of general experience. They have seen more code bases than any of us. What they don't have is knowledge of this repository, this team, and what went wrong last month.

19:54 The trouble is that agents don't read instructions the way people do. A person, for example, a human person, reads a rule book and just quietly skips the step that, you know, the one that obviously doesn't fit. To an agent, an instruction looks like a rule. I learned that with a step that produced a demo video for every task a team worked on. And that made sense for features and none, no sense for a change with nothing to show.

20:25 The workflow handled that with an exception like skip the demo when no end-to-end tests changed. And that over there is the unless trap again. What works better is stating why a practice exists. A demo exists so a human can see a change in behavior before merging, for example, right? That's an explanation. An agent that knows that can decide that a research task doesn't need one without anyone listing that case in advance.

20:57 Reasons cover the cases nobody thought of. Exceptions cover only the ones someone did. Agents fail in the other direction as well. They cut so many corners when a rule gets in their way. The answer to that isn't more rules. It's more than one agent. When a coder cuts a corner, a reviewer with a fresh context is likely to object, especially one running on a different model.

21:22 And when they all miss the same thing, the required checks are still there. So local experience has its own dangers. It's tempting to copy whatever an organization already does. Some of those practices exist to compensate for human limits and make no sense, no sense for agents. Others were how to say, they were never good. Really, they were And people simply worked around them without saying so.

21:54 Agents cannot work around what nobody wrote down as so they follow a bad rule faithfully and at scale. At the same time, agents need practices no human organization never had, like fresh context for every review or plans and decisions written down because nothing, nothing is remembered between sessions. And local experience piles up. Every incident adds a rule and nothing gets removed.

22:24 Writing down the reason is what makes removal possible. When the reason no longer holds, the practice just walks away. So that's why agent deck, in my case, itself is deliberately not very prescriptive. It provides mechanisms. Start a process, clear this context, delegate a task, run several roles, isolate work in a work tree, observe status, and route results.

22:50 It does not know how your project tests software, that reuse pull requests, or which decisions require a human. Project specific skills supply that policy. A command line tool, Kubernetes controller, and the documentation site needs very different tests, different reviews, and a different definition of done. So each repository defines its own organization without changing the application.

23:15 So and that over there is the balance I'm after. Not autonomous agents escaping process, but adaptive agents operating inside the process whose hard edges they cannot negotiate. What comes next is it's a system that decides how much organization each task deserves. Not one autonomous agent with an enormous context and not a permanent fake company where six personas attend every meeting, right?

23:47 Neither of those two. Start with one agent and uncertain concurrency bug might need two independent investigators. A security change might need a second auditor on a different model. A typo needs nothing more than the first one. Add a context only when it brings something the first can't. A clean perspective, independent evidence, parallel capacity, or meaningful different judgment.

24:15 My job becomes designing that organization. The few rules that cannot be bent, the practices and the reasons behind them, and the points where a human has to decide. So here's my question to you. Which decisions are you still making for your agents that they could make themselves? And which ones should they never, never, ever, ever make at all? Let me know in the comments. Once agents are doing more work than one person can personally supervise, prompting is no longer the main problem. Management is.