← All transcripts

AI Agents in Kubernetes Without Laptop Credentials (feat. KAOP) Transcript, AI Summary & Key Points

DevOps & AI Toolkit · yesterday · Science & Technology · 28:01 · EN

Watch on YouTube

Answer

You run AI agents in Kubernetes without laptop credentials by putting a platform like KAOP between agents and clusters: agents run inside clusters with their own identities, approvals and audit live in the platform, and the laptop holds no cluster access at all. But a platform doesn't make agents smarter, and KAOP's rough edges (blind approvals, audit gaps, no Kubernetes event triggers) need watching.

AI Summary

Komodor's Agentic Operations Platform (KAOP) gets tested by fixing the same Kubernetes incident four ways: Claude Code alone, Claude Code through KAOP, a scheduled agent with nobody at a keyboard, and a Slack mention. Claude Code already diagnoses and fixes a broken Kubernetes service on its own — a flaky HTTP 500 on a website (Agent Deck) — and a platform does not make agents smarter. What KAOP adds is the infrastructure that matters for a team: agents with their own cluster identities instead of borrowed laptop admin credentials, Slack-based human approvals for mutating commands, shared run history reachable from any agent, and cost tracking with budgets that cancel runs. The friction is real: the starting agent sees only a spinner when waiting on approvals, Slack approval requests show a command and one sentence with no reasoning, run reports arrive as raw JSON, audit records can't distinguish a person from an agent acting as them, there is no native reaction to Kubernetes events, guardrails skip Bash-executed commands, and inputs, tool calls and outputs leave your environment (self-hosted control plane is only a beta). Splitting work across multiple agents caused a preventable mistake — the docs team's deployment got deleted and recreated with a different label — that Claude Code alone avoided. The core lesson: splitting tasks across agents is harder than it looks, because what one agent knows doesn't travel to the next; rules that matter must be enforced outside the agents with permissions and approvals.

Key Points

  • Claude Code alone on a laptop with admin credentials diagnoses and fixes the flaky HTTP 500 on the Agent Deck website, changing only what it should and verifying the fix.
  • Laptop-based agents only work because the laptop holds admin credentials — in most companies a developer's laptop does not and should not hold production admin access.
  • KAOP runs agents as services inside clusters with their own identities and permissions; agents connect out to Komodor, so nothing accepts incoming connections and the laptop has no cluster access.
  • In the KAOP run, Claude Code (with no cluster access, KAOP connected over MCP) discovered KAOP's agents unprompted, sent Kubernetes RCA to investigate and the Kubernetes remediator to fix, and backgrounded the wait.
  • Every mutating kubectl command the remediator ran waited for Slack approval; two requests were approved one by one, including one containing kubectl apply that the rule caught.
  • The multi-agent fix went wrong: it deleted the docs team's deployment and recreated it with a different label, while Claude Code alone had chosen the better fix of touching the website itself.
  • Root cause of the mistake: three agents worked without knowing about each other — RCA recommended relabeling the preview without knowing the remediator existed, Claude Code turned that into orders without seeing the remediator's instructions, and the remediator followed the orders.
  • Rules that matter must be enforced outside the agents with permissions and approvals; a rule inside an agent's prompt is invisible to everyone calling it.

AI in practice

Used for

Agents

  • Kubernetes RCA — Investigate root cause of the failing website in the cluster; read-only. 2 held 05:20
  • Kubernetes remediator — Fix Kubernetes issues by changing resources, limited to a specific namespace. 2 held

Tools & resources

4 items

HNo. 2520
AIAINotes.us Tool

Helm

Open source · helm/helm

Helm is a Kubernetes package manager for creating, sharing, installing, and managing charts. A chart is a package of preconfigured Kubernetes resources containing a package description in Chart.yaml and one or more templates for Kubernetes manifests; charts can be stored locally or fetched from remote repositories. Helm renders chart templates and communicates with the Kubernetes API to install and manage releases, and can run on a developer machine or in CI/CD. The videos also describe using Helm to package Crossplane compositions with little or no templating and distribute them as versioned OCI artifacts.

Mentioned in
4 videos
Kind
Other
KNo. 8074
AIAINotes.us AI product

Komodor

komodor.com

Komodor is an agentic operations platform for production.

Mentioned in
1 video
Kind
AI
KNo. 8078
AIAINotes.us AI product

Komodor Agentic Operations Platform

komodor.com

Komodor Agentic Operations Platform is a control plane for running AI agents in production Kubernetes environments. It gives agents their own cluster identities and permissions instead of requiring borrowed laptop credentials, routes risky commands through Slack-based human approval workflows, and records shared run history, audit information, cost data, and budget enforcement. The platform supports scheduled agents, catalog agents, and an MCP gateway for external agents such as Claude Code. It adds governance and operational controls rather than improving an agent's diagnostic ability; approval waits can hide context from agents, while approval messages may show commands without their reasoning.

Mentioned in
2 videos
Kind
AI
MNo. 0415
AIAINotes.us AI product

Model Context Protocol (MCP)

Open source · modelcontextprotocol/modelcontextprotocol

Model Context Protocol (MCP) is an open, standardized protocol layer for connecting large language models and AI agents with external data sources, hosted infrastructure tools, and other contextual data. It defines formats, metadata, protocol schemas, and APIs for sharing information and tools, attaching, referencing, and validating context such as documents, embeddings, and provenance, while supporting scoped authentication and permissions. MCP translates JSON requests from an agent into calls to service APIs, including CRM, container-management, and GKE capabilities, and can provide design-system context and related tools for generating consistent applications. The official project publishes its specification, documentation, and protocol schema; the schema is defined first in TypeScript and also provided as JSON Schema for broader compatibility. The protocol was created by David Soria Parra and Justin Spahr-Summers, is hosted at modelcontextprotocol.io, and is licensed under the MIT License.

Mentioned in
33 videos
Kind
AI

Links mentioned

🔒 Full analysis locked

Unlock more videos and the full analysis

A credit unlocks one video's full analysis for good — the build steps, the tools and how each was used, the methods behind every use case. Pro opens the whole library instead, and raises how many videos you can analyse a day.

Unlock full analysis — free

Transcript

Searchable transcript of AI Agents in Kubernetes Without Laptop Credentials (feat. KAOP) — DevOps & AI Toolkit (28:01). Search for a phrase, then click its timestamp to jump straight to that moment in the video.

Captions sourced from the original video on YouTube, published by DevOps & AI Toolkit. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.

00:07 Why would anyone put a platform between an AI agent and a Kubernetes cluster? Cloud code can fix clusters on its own. It finds the cause, rules out the red herrings, changes only what it should, and verifies the fix. No platform, no special setup, just a terminal, a prompt, and my credentials. That works for one person. Now, does it work for a company where it means every laptop holding the keys to production, nobody else approving what the agent changed, no shared record of what they did or what it cost, and nothing

00:45 happening unless someone sits at the keyboard? To find out, I tried a platform built for exactly that. Commodore's agentic operations platform or Calp for short. It lets you define agents and run them inside your clusters instead of on someone else's laptop as services with their own identities. It sends the risky actions to people for approval. And it keeps a record of what they did and what it cost.

01:14 Now it does more than that like a catalog of readym made agents guardrails on the tools agents call and sidebyside testing of a new model or prompt against the current one and I'll come back to those at the end. So do we need a platform at all? What changes when there is one? And is cowp the one to pick? To answer that, we will fix the same incident four times with clo code alone, with clo code going through cop and twice with nobody at the laptop.

01:52 One of those fixes goes wrong in a way that says something about agents in general and not only about cop, but we'll get there. Let's start with the incident. There's a website running in a Kubernetes cluster and visitors say it sometimes fails. I will reload it a few times. What you're looking at is the website of agent deck, a project of mine. Agent Deck runs your coding agents under one background demon and shows you what each of them is doing in a terminal or in a desktop app.

02:27 So you stop babysitting one agent and start running a team of them. Right now back to the site. Most reloads show its landing page. Every few though, the page is blank except for preview build. And that's an HTTP 500, right? It's broken. But only sometimes, which is the worst kind of broken, right? So, let's try to fix that first. The way most of us would do it today, unless you still live in La La Land and you're convinced that LM and agents are a passing fed, right?

03:01 I will start load code on my laptop which holds the cluster's credentials. We know how the issue manifests but not what's causing it. So that's all I will tell you. The website sometimes fails. That's what matters. Now it goes to work. Watching an agent run cube control command after command isn't exactly gripping. So I will fast forward through that part and through every other weight like in this video.

03:29 After a minute or two, this is what it came back with. It fixed it. And I'm not here to tell you to use agents. You already do. So, an agent finding and fixing a problem like this is magic you're used to by now. And there's nothing nothing for me to explain. Now, this worked only because my laptop is an admin on that cluster. In most companies, a developer's laptop does not and should not hold admin credentials to production.

03:58 And that's the gap cop claims to feel. I broke the website again for the next round. COP is a control plane for agents. The agents themselves running your clusters as workers that connect out to Commodore. So nothing nothing in those clusters has to accept incoming connections. Each agent has its own identity and its own permissions. Calp has its own console with its own chat and you could do everything everything from there and I won't I will not do that.

04:35 I already have an agent I like and you have yours be it cloud code, open code, codex or something else. A platform that makes me use its interface to reach its agents is like um it's like a shop that only lets me in through its own browser instead of Chrome. Cub doesn't do that. Pretty much everything it offers is also available over MCP. So I will stay in cloud code.

04:58 I already set up the Kubernetes remediator, its permissions, the approval rule, and the slack channel. And I think that's it. In a company, that would be the platform team's job. A developer only adds Scalp's MCP server to their agent. There are two agents for this cluster. Kubernetes RCA comes from Commodore's catalog and it can only read. The Kubernetes remediator is one I built on Commodore's agent base and it can change things but only in a specific name space.

05:33 Building it was a Helm chart with a few values mostly instructions telling it what it's for plus a regular Kubernetes role that limits what it can touch. The approval rule is a short policy. Any cube control command that changes something waits until a person approves it in Slack. None of that is worth watching. So I'm I'm just going to skip it. You might say cloud code can do the same.

05:57 Its permission rules can ask before any cube control command that changes something and a company can enforce those rules for everyone. And that's true. But that only decides what the agent may do. The credentials are still on the laptop. So whoever drives the agent can run cube control themselves. The approver is always whomever sits at that terminal.

06:19 The approval is a key press nobody else sees and the rules apply only while someone is there to drive. With Calp the access belongs to the agent running in the cluster and my laptop has none. So there's nothing for me to go around. The person who says yes can be someone else entirely. And every answer is recorded in one place. Now let's do it again.

06:47 This time from a cloud code session that has Cal connected over MCP and no access to the cluster at all. First prove that this clo code has no way no way into the cluster. Cube control has nowhere to go and cloud code isn't allowed to look for credentials either. Now the same request as before word for word. This time there's more going on behind that spinner.

07:12 So the fast forward skips a few minutes. And I never mentioned culp in that prompt. Cloud code found that its own cube control was blocked. Looked at the tools it had and found cop's agents. It sent Kubernetes LCA to investigate and the remediator to fix it. and then it moved the weight into the background and said it would come back. Meanwhile, every cube control command the Kubernetes remediator wanted to run that match the ask rule waited in slack to in this run approved one by one.

07:47 I read each request before approving it. The first is mostly harmless you know a backup and a dry run but it contains cube control apply. So the rule caught it. The second is the actual fix. When the run finished, cloud code came back on its own, fixed and verified. And this time nothing, nothing on my laptop could touch the cluster. Kubernetes RCA did the reading, the remediator did the writing, and every write baited for me.

08:26 Getting there though was a mess. a real mess. When the remediator stopped to wait for my approval, Clo had no idea. All I saw was a spinner as if the run were just slow. Slug did notify me, but nothing in Clo where I actually was, said that the run was waiting for an approval, mine or anyone else's. I had to leave the agent I was working in, find the request in Slack, approve it there, and then go back to Cloud Code myself to see what happened.

08:55 And it's not that Calb doesn't know. It knows exactly which approvals are pending. It just doesn't tell the agent that started the run. If I'm the one who approves, the request should come to me in cloud code where I already have the context. If it's someone else's, cloud code should at least tell me what's waiting and for whom. Then there's the request itself.

09:23 Slack shows the command and a one line. oneline description while the reasoning behind it stayed in cloth code. I had read the reasoning so I could carry it across in in my head. If someone else were approving, they would never have seen it and all they would get is a command and a sentence. Sending the request to clot code would fix it for me. But for anyone else, the request has to bring its reasoning with it.

09:52 But look at how it was fixed. It didn't touch the website. It deleted the docs team's deployment and recreated it with a different label. Cloud code on its own chose a better solution earlier. What happened here is three agents working without knowing about each other. Kubernetes RCA recommended reabeling the preview without knowing a remediator existed, let alone that its instructions say to prefer changing the failing application resources over other teams.

10:26 Clot code turned that recommendation into orders without being able to see those instructions and the remediator follow the orders. A rule that lives in an agent's prompt is invisible to everyone calling it. So the rules that matter have to be enforced outside the agents with permissions and approvals. Here in this case the preview was in a name space the remediator could write to.

10:50 So the approval was the only thing standing between that plan and the docs team's deployment. And you know I'm lazy. I just clicked approve. Part of that is on me. I gave the remediator right access to the whole namespace and let another team's preview live in it. So permissions did not stop anything. Its instruction to prefer the failing applications resources was a preference, not a rule.

11:14 And they never told it to check who owns what before changing it. Although the preview carries an annotation saying it belongs to the doc team. Part of that mishap over there is on commodor. agents built on their base image and configured through their Helm chart can't even have a description or at least I I I couldn't figure out how. So there's nowhere to publish an agent's rules for callers to read.

11:40 Their own Kubernetes RCA put the other team's deployment first. Approval rules match the text of a command, not whose resource it touches. And the request in slug said what would be deleted not that it belonged to someone else. Now there's a bigger lesson here and it goes beyond commodor. Calp is built around many narrow agents each doing one job calling and being called by others.

12:10 Splitting work that way sounds clean. In practice every handoff loses something. What one agent knows, assumes, or is not supposed to do doesn't travel to the next. Every time a single agent held the whole problem here and in the runs coming up, it chose the gentle fix. The one time the problem passed through three agents, it didn't. Splitting tasks across agents is harder than it looks, and sometimes it's not a good idea at all.

12:50 So far, someone still had to sit at a laptop and ask, but the agents live in the cluster and their access lives in COP. So, they don't need anyone's laptop at all. Cal can start agents on a schedule from Slack or from a web hook, but it cannot react to Kubernetes events on its own, which is silly. For that, you would have to build a controller that watches the cluster and calls a web hook.

13:15 that honestly that surprised me in a platform built around Kubernetes. I would expect events first and schedules as a fallback. So I turned on a schedule that asked the Kubernetes remediator every 5 minutes whether requests to the website are succeeding. If they are, it replies healthy and stops. If they aren't, it investigates, fixes, verifies, and reports.

13:42 Then I broke the website again since the previous run fixed it, right? It fires on the next 5 minute mark and its first approval request appears in Slack a minute or so later. Each of its two rights waited for approval in Slack. Nobody asked it anything. It found the failure and it chose on its own the fix that leaves the docs preview alone. This is where the missing reasoning hurts.

14:08 I didn't start this run. So all I get is a command and one sentence and that's that's not enough to judge it. Approving it anyway is blind trust. And blind trust is no better than having no approvals at all. Now when it finished its report reached the channel. The report is good. The way slack shows it is not. is the row JSON output escape new lines and all you know folded all of that folded behind show more come on the console shows the same text properly so this is a formatting problem nothing else but it's the first

14:48 place a person sees why the agent did what he did we can talk to agents from Slack as well I broke the website again and anyone in the channel can mention cub's bot and describe the problem in plain words which sounds awesome or not. Now the bot replies in a thread within seconds and the rest is fast forwarded until the first approval shows up. Each step waited for approval in the thread.

15:15 Its answer in the thread, same agent, same fix in a thread and nobody's laptop was involved. Now would a software engineer talk to agents through Slack about production issues? I mean, I wouldn't. I talk to my agents from my phone about my personal stuff all the time, but production is a different story. Still, it's there. If you insist, use it if you must.

15:44 Just don't tell me that you do or or I would have to fire you if I could. Say it was someone else on the team who started that run from slack. I wasn't there and know nothing about it. But later I want to know whether the fix is still holding and how to roll it back. So I ask from where I actually work from cloud code. It digs through cow's run history for a bit before answering four.

16:11 Is it still holding though it offered cube control commands and his clo code has no access to the cluster. So skipping ahead again right fast forward pass the agent it sends to look. Now two things I like here. It found the slack run on its own and read its transcript. So the history is shared no matter where a run was started. And for a check that shouldn't change anything, it picked the agent that can't change anything.

16:44 With agents changing uh things on their own, the next questions are what they did, who allowed it, and what it cost. The run page of the scheduled run lists each command it asked about who answered uh through what and when. But there's a gap though. Every run code started is attributed to me. So the audit can't tell me from an agent acting as me and the run started from slack and by the schedule have no creation entry at all.

17:19 There's a console's cost page per agent and per run. And everything there is estimated from usage, not taken from provider's bill. Budgets can be set from cloud code as well. So I'll start it again. That one takes a few seconds. And it had already spent more than the cup for that day. So the budget was exhausted the moment it was created. So, let's see what that does to the schedule.

17:48 There we go. The next scheduled runs were all cancelled before they started. And each one posted a run cancelled card to Slack. The card doesn't say why, which is silly. A separate budget threshold crossed message does, and so does the run record. So, the budget works. You save money only to lose your time. instead staring at the run cancelled card every 5 minutes and wondering why the hell was it canceled and what's broken now?

18:21 Silly. Here's the short version. Companies need something like c a place to define agents and run them away from anyone's laptop. Companies need that and it has a lot of potential but right now it's rough around the edges sometimes very rough. So let's start with those. The first one the agent that starts the run is kept in the dark. When Calp holds a command for approval it doesn't tell that agent that the run is waiting for someone.

18:58 So all you see is a spinner forever and ever. If you're the approver, you have to notice that on your own. Switch to Slack and switch back. If someone else is, you don't even know what you're waiting for or for whom. A slack approval shows no reasoning, only a command and one sentence. Now, that's enough when you started the run yourself and blind trust when a schedule or a colleague did.

19:25 Now, approvals see commands, not consequences. A rule matches the text of a command. The same allow and deny lists we've always used. And the request in Slack says what will be deleted or changed, not whose it is. That's a lost opportunity. With agents in the loop, approvals could be based on context. What the run is trying to achieve, whose resources a command touches, and what depends on them.

19:56 A model could judge that first. let the obvious cases through and send the rest to a person with that context attached. Cloud codes auto mode already works a bit like that. It can stop and ask before modifying shared resources whatever the command looks like. Cal to be honest has policies judged by a model as well but only in guardrails not in approvals.

20:23 Then there is false success which we haven't seen on the screen but I will explain it. When nobody answers an approval in time, it expires. And the run's card in Slack still says run succeeded even though nothing nothing was applied. The run completed. The incident wasn't fixed. Messy slack makes all of that harder to follow. Run reports arrive as row JSON escape new lines and all of that.

20:51 Are we supposed to read JSON now? The console renders the same text properly. So Slack could do it as well. And budget blocked runs show up as castellations with no reason on the card. Agents built on Commodore's base image and configured through its Helm chart have no descriptions. Or at least I couldn't figure out how to give them one. So an agent like Cloud Code choosing among cops agents sees only names.

21:21 It can easily pick the wrong one for the job. Then there are audit gaps. Runs started from clo code are attributed to me with no way to tell me from an agent acting as me. And runs started from slack or by a schedule have no creation entry at all. There are no even triggers. Cal cannot react to Kubernetes events on its own. So you either write schedules or build and run your own controller that watches the cluster and calls a web hook.

21:49 for a platform built around Kubernetes. That's an odd gap. Guardrails skip Bash. I mentioned guardrails at the start, but they only see tool calls that go through COP's own MCP gateway. An agent that runs cube control through Bash like the remediator does is limited by approval rules and Kubernetes hardback, not by guardrails. Finally, another one we haven't seen on screen.

22:19 The data leaves your environment. The agents run in your clusters. That's true. But their inputs, tool calls, reasoning, and outputs all go to Commodore's control plane, even when you bring your own model gateway. Commodore offers a self-hosted control plane, but only as a beta and under agreement. Okay, now the good parts, right? Agents get their own identity.

22:46 Nobody's laptop needs admin access to the clusters and the agents run inside them connecting out with nothing listening for incoming connections. It works from your agent. Nearly everything is available over MCP. So you rarely have to leave cloud code or whichever agent you use and that agent can find cops agents and use them without being told. The exception is approvals which still take you to slack which is silly.

23:14 So people approve anything risky in Slack where they already are and every answer is recorded against the person who gave it and we are heading towards agents that act on their own with fewer and fewer people in the loop and that's where this should end up but most companies aren't ready to let agents lose on production and until they are this is exactly what they should ask for.

23:40 The agents need no laptop. A schedule or a slack mention can start them with nobody at the keyboard. And the history of every run is available later no matter where it was started. Then there is audit and cost. Who approved what through which channel and when, whatever run cost, and budgets that stop automatic runs once they're spent. If you work in a company where auditors are breathing breathing down your neck, that alone might be the reason to have it.

24:11 Gaps and all, it works with any agent, not only the ones built on Commodore's base. An agent in any language, one that knows nothing about cop can be brought in with a thin wrapper. And the rapper can ask for approval of the whole plan, root cause included, in one go. The catch and there is always a catch is that culp only controls what passes through it.

24:34 An agent with its own tools and its own model key gets visibility and an approval step but no cost tracking and only as much enforcement as it agrees to. There's more to explore that I haven't tested yet. Guard rails you can check against past runs before turning them on. sidebyside testing of a new model or prompt with a blinded judge and credentials that agents use through commodor's gateway without ever seeing them.

25:09 So do we need a platform not to make agents do more? Cloud code already does the work, right? What a platform like Cub gives you is a place to define agents and run them somewhere other than someone's laptop. That's that's useful even for one person. I run my own agents remotely all the time. So they don't stop when my laptop goes to sleep and so they don't sit on a machine full of things they have no business seeing.

25:35 But for one person an agent on a server is enough and a platform would be overhead. Once there are many agents, many teams or someone who will ask who approved what that's when you need more than that. So what changes when there is there is one one platform the agents run next to what they operate with their own identities and their own access. They work when nobody's at the keyboard and anyone on the team can start them and see what they did.

26:09 That's where the approvals and audit and budgets come from. It's also where most of the cons come from. You're no longer in the loop by default. So, you wait on spinners, approve in another app, and read reports in Slack. And splitting the work across agents makes it easier to lose what matters on the way. What doesn't change is how smart the agents are.

26:33 With the same model, you know, same LM. Commodore's RCA agent did not diagnose any better than cloud code alone. And here it recommended the worst fix. So is scalp the one to pick? It has the right pieces. It works with the agent you already use. And nothing I hit was a blocker. Most of what bothered me is annoying rather than broken. And that's um that's the easier kind of a problem to fix.

27:04 What I would keep an eye on is design approvals that cannot tell whose resource a command touched. and agents that cannot see each other rules. Until that changes, keep each agent's permissions tight. So, an approval isn't the only thing standing between an agent and other team's work. The one real blocker is for companies whose data cannot leave their environment, at least until the self-hosted control plane is more than a beta.

27:34 So tell me in the comments, would you trust the model to approve your agents changes or do you still want the person to click the button? Thank you for watching. See you in the next one. Cheers.