AI Agents Are Non-Deterministic. So Are You. Deal with It. Transcript, AI Summary & Key Points

00:07 Let me start with a question. Do you like games? Like video games, board games, whatever it is. And would you play them all day if you actually could? I mean, I know I would. But here's my problem. I can't. I like games, but I also like money. Money for rent, money for food, money for more games. And to get that money, I have to get some done for the company that pays my bills.

00:33 That's the deal. So, my real dream was never more games. I have enough games. It's getting the work done without me having to do it, so I can get back to the controller. There is all kinds of work I would happily hand off, but I want to zoom in on one slice of it. The ops work. Provisioning the infrastructure, wiring the databases together, deploying ops, keeping the whole thing running.

00:58 That's what I'm trying to get AI agents to do for me today. And that that dream is not crazy. It's not just hype. An agent can really easily do that work. It provisions the clusters, it fires up the databases, it fixes the broken pipeline, and you get to lean back and pick up the controller. But there's a catch. And right now, right now, you're probably yelling at the screen.

01:24 In in your head, you're yelling at me. AI is not deterministic, right? Is is that what you're saying? Well, I I cannot hear you, so please stop yelling at your monitor. And second, you are right. That agent working for you is non-deterministic. Give it the same task twice, and you get two completely different answers. And it will do risky things with total confidence whether it's right or whether it's dead wrong.

01:53 But does Does familiar? Because that's that's you. That's you as well. >> Who, me? >> Give the same task to two engineers or the same engineer on two different days and you get two different answers. Each delivered with the exact same confidence. You do that as well. But there's a difference. There is a real difference. So, here it goes. You you're annoyingly slow.

02:19 You're so so slow. You make the occasional mistake. An agent makes a thousand mistakes an hour and it never never never gets tired. Now, here's something worth pausing on before we go any further. Everything we just talked about an agent provisioning clusters, firing up databases, deploying apps, all of it depends on one thing. The agent actually being able to do stuff.

02:43 And that's the part everyone underestimates. A model can reason all day long, but reasoning isn't action. The moment an agent has to reach into a real system and actually take an action, that's where most agent projects stall. Permissions, flaky tools, and no record of what actually happened. People call it the action gap. And that's where the sponsor of this video comes in.

03:07 Arcade. Arcade is the runtime for MCP, the model context protocol. It's built for exactly this moment. Giving agent a secure reliable way to take actions inside real systems. It handles authentication and permissions, so an agent only does what it's actually allowed to do. It makes the tools it calls dependable instead of flaky and it gives you an audit trail of every action after the fact.

03:34 So, whether you're building on LangChain, CrewAI, OpenAI's agent, or anything you wire together yourself, you can check out Arcade at arcade.dev or click the link in the description. It's somewhere there. Big thanks to Arcade for sponsoring this video. And now, let's get back to putting those agents on a leash. So, if non-determinism is the problem, here's the here's the good news.

03:59 We already solved it. Not recently and not for AI, we solved it for ourselves years ago. Back when the unpredictable overconfident actors, you, you know, actors in the room, were human beings. Think about it. We don't let an engineer SSH straight into production at 3:00 in the morning and run whatever they feel like. You open a pull request, someone reviews it, a policy checks it, a pipeline applies it.

04:24 We built an entire discipline around one simple fact. People are unreliable, fallible, and people are non-deterministic. And here's the part that matters. AI agents are just more of those actors. They are not some brand new kind of threat that needs a brand new kind of defense. The same guardrails we built for fallible humans work on agents as well.

04:50 They're just knocking on a door we locked years ago. Now, will we eventually come up with better ways to handle agents specifically? Almost certainly. For sure. We usually do, but that's that's tomorrow's problem. Today, the solution already exists. It's battle-tested, it's sitting right over there, and it's the one we're going to use until something better comes along.

05:21 Now, let's build that leash, one piece at a time. And to build it, we have to start where all of this began, long, long before any agent showed up. Just you and the cloud. You want a cluster, so maybe you click through the console, maybe you write some Terraform scripts, HCL, or just maybe maybe if your platform team got really fancy, you click a button in a self-service portal like backstage which just runs those same tools for you under the hood.

05:51 Either way, it all funnels into the same place, an API, the giant massive one that AWS or Google or Azure puts in front of everything. And whichever path you took, it was you in that seat driving it yourself. For years, that was the job. And now we hand that seat over to a swarm of agents, a bunch of them. And they do exactly what you did through the exactly same means.

06:22 A bit of curl here, the provider CLI over there, some Terraform, and so on and so forth. They talk straight to the cloud, except they never sleep and they do it a thousand times an hour. Now, I want to be precise about what the problem actually is because it it matters. I'll explain in a minute. It's It's not that the agents are calling an API. Agents are fantastic at calling APIs.

06:44 The problem is which API. They're holding your cloud credentials pointed straight to the cloud's own API, the giant unconstrained one that follows nobody's rules. Nothing narrows what they ask for, nothing reviews what they send, and there is no undo. That's the loaded gun, a non-deterministic swarm firing straight at production, full credentials, zero constraints.

07:12 And that that that's the part that should scare you. Honestly, that should scare you. So, let's fix it one problem at a time. So, the first fix is not how to take an API away from agents, it's to give them a better one, to give them yours. We put the control plane in the middle, and that's crossplane. So, this is what you get with it. Your platform team gets to define what the cluster even means at your company.

07:44 And a cluster is just is just an example here. It could be a database, a message queue, an entire application environment, whatever your teams keep asking for. Whatever it is, you define it. Not the cloud's 200 knobs, but the 10 you actually care about. And just as important, now that thing complies with your rules. Only the approved regions, private networking, sane node sizes, the right tags, so that finance doesn't come knocking, and so on and so forth.

08:14 All of it is baked into one blessed abstraction. And in a cloud like AWS, one thing is rarely just one API. A cluster isn't only a cluster, it's the cluster plus the node groups, plus the VPC, plus the BAM rules, and so on and so forth. Each of all of those has its own API with its own pile of knobs. So, the agent no longer has to pick the right 200 settings out of thousands spread across all of them and pray and pray that it landed on a compliant combination.

08:46 And as as far as I know, models aren't religious, so praying wouldn't help them anyway. It just asks for a cluster, and what it gets back is compliant by construction. The agent can only ever ask for what your API exposes, which means the blast radius just shrunk to exactly what you decided to allow. And because that API is yours, you decide what sits behind it.

09:11 This particular composition I'm showing, for example, can turn that one request into an EKS or GKE or AKS cluster, probably a few more. That is a choice the platform team baked in, not something the agent picks. And and there's a bonus, and it takes us right back where we started, with nondeterminism. What the agent hands the control plane isn't a fire and forget command.

09:37 It's a declaration of the state it wants. And once that desired state exists in the control plane, getting the world to match it is a deterministic mechanical job. The control plane reconciles reality to the declaration again and again and again, and it corrects any drift. That machinery doesn't improvise, and it has nothing nothing to do with agents or with humans.

10:02 So, no. We haven't made the agent deterministic. We can't. But we've taken all of that non-determinism and squeezed squeezed it down to a single point, what gets declared. Everything downstream of that declaration is now boringly reliably deterministic. Now, hold onto that thought because in a minute or two we're going to put guardrails around the fun unpredictable thing that's left, the declaration itself.

10:32 But before we get there, notice what hasn't changed yet. The agents are still applying straight to the control plane. Whatever they declare goes live the instant they say so. Still no review, still no history, still no undo. So, we take that access away from everyone and everything. Nobody nobody n- nothing applies to the control plane directly anymore.

11:03 Not the agents, not you. The only way in is to commit to Git, and then Argo CD or or Flux pulls from Git and reconcile the cluster to match. Now, look at what that does to this form. It It cannot touch production even if it wanted to. There is no door left except except Git. And now you get a choice. You can require review on every single change, and if you do, this is the fun part.

11:29 The reviewer doesn't even have to be human. One agent can review another agent's pull requests. Or if you trust the swarm more, you let it merge to main and just apply automatically. Either way, you get the thing that actually matters. Now, remember the declaration, the one point we squeezed all the determinism into, right? It's now version controlled.

11:51 You get the full audit trail. The complete history of every single desired state you ever had. And there's Git revert as your undo button if you really need it. Now, whether you reviewed every single change or you let the swarm merge on its own, there is one more layer you want. One that doesn't depend on anyone anyone or anything catching anything.

12:19 Policies. And the tool enforcing them here is in this case Kyverno. Now, those are not like, "Hmm, should we do this?" They're, "This is never allowed." Full stop. And remember those guardrails I promised, the ones around the declaration itself? Th- These policies are them. Your policies run automatically on every single change. Actually, before the change.

12:41 But it doesn't matter whether it comes from an agent, from a pull request, or from you at 3:00 in the morning. They don't get tired. They cannot be argued with. They cannot be talked into into an exception. So, whether a bad change got reviewed and slipped through or was never reviewed at all, your policies stop it deterministically. Accidents and bad actors, both.

13:05 They both hit the same wall. This is the leash. And look at what we actually built it out of. GitOps policies, a control plane. None of it is new and none of it is some AI vendor's shiny new product. Every piece already exists. It's open source. It's sitting in the CNCF and it's there for the taking whether you're already using it or well, and that's a different conversation, different video.

13:39 There's one last piece. Everything we built so far governs your infrastructure, but your agents depend on something else. They depend on models and if you're shipping your prompts and your data off to somebody else's API, then that right there is the one part this whole loop you do not control. Does that sound familiar? It's the exact same problem we started with an API that isn't yours.

14:05 So, we do the exact same thing we did before. We make it ours. We put inference in the same control plane. This is model plane in this case and under the hood, it's essentially crossplane compositions. Now, is this worth doing for everyone? Well, no and I will not pretend saying that it is. If you're a small shop, sending your prompts to a hosted API is perfectly fine and almost certainly cheaper.

14:32 Running your own inference only starts to pay off when you're big enough that the math works out or when your data simply isn't allowed to leave the building. If that's you, this is how you keep it yours. So, serve this model on my GPU fleet becomes just another declarative resource. It sits right next to your cluster. It flows through the same gate, through the same policies, through the same reconciliation, the same leash.

14:59 And now the whole loop, what the agents do and the models they run on lives on infrastructure you own. So, look where we ended up. That swarm of agents is still doing all the work, still not deterministic, still tireless, still a thousand actions an hour. nothing about that changed and nothing needed to. What changed is everything around them. Every single thing they do now runs through a leash you built.

15:30 Desired state instead of blind commands, Git so nothing reaches production without a trace, policies so the rules hold whether anyone is watching or not, and the same control plane running your models as well. And here's the kicker, you didn't buy a new AI platform to get any of this. It's Crossplane, it's Argo CD, it's Caverno, it's all CNCF. Not one of them was built for AI and every one of them is either already running in your cluster or one install away. Agents not need a new platform, they need yours.