DevOps AMA: Crossplane XR Deletion, AI Impact, and Chainsaw Testing Transcript, AI Summary & Key Points

00:00 Heat. Heat. [music] [music] [music] [music] >> [music] [music] >> Heat. Heat. [music] Heat. Heat. N. [music] Heat. Heat. [music] [music] >> [music] [music] >> Heat. Heat. Heat. Heat. [music] [music] Heat. Heat. [music] Heat. Heat. [music] [music] Heat. Heat. [music] [music] Heat [music] [music] up [music] here. >> [music] >> Heat. [music] Heat. [music] Heat.

03:26 [music] [music] Heat. >> [music] >> Heat. Heat. [music] [music] Heat. Heat. [music] [music] [music] Heat. [music] Heat. [music] [music] >> [music] [music] >> Hey, hey, we have yet another ask ask me anything. This time Scott is not with us. So it's only me. You know the drill. I hope. Unless this is your first time. If this is your first time, the drill is complicated.

05:21 And complicate by complicated I mean you ask questions. I or me and my guests answer them. That's all there is to it. Right? Ask anything you want. I'll do my best to help out, to answer, to do what I can, right? To help you out. So without uh let me just uh do this to move the uh text so that I can see it. Okay, let's go with the first question. Where is it?

05:52 Here we go. Okay, so how does uh crossplane handle uh I hope you can see me well. Uh where where is this? Okay, there. Oh, there we go. Uh how does crossplane handle deletion of XRS? I notice XR gets deleted from CD. Uh even when MR seat manages haven't finished deletion. Uh yeah that was a kind of semi issue. So here's how it works worked in the past and still if you choose to do so there are uh cluster level uh namespace level uh composite comp compositions that create cluster level compositions that create cluster

06:35 level MRS it's a mess and kubernetes cannot handle different scopes kind of oh this is cluster this is um namespace level So uh when you delete one then it doesn't delete another right away or it doesn't wait for another anyway it's a mess. That was before crossman introduced uh namespace scoped everything. So if you switch to have it namespace scope you don't need claims anymore right because uh to begin with uh compositions are not deleted um uh until XRS sorry XRS are not deleted um uh before MRS are deleted claims

07:19 are deleted before MRS but we don't use claims anymore right that's crossplane v2 so make it space scoped then you will not have the trouble if you keep it as is then you will see that the claim gets deleted but the cluster scoped XR is not right and it's still there and it cannot be deleted because Kubernetes itself doesn't allow it to to be deleted until all the related resources are deleted right so I'm guessing and correct me in the comments that you're referring to claims um use claims for let's call it legacy uh

07:53 switch to namespace scoped everything and then then you don't need claims anymore uh compositions are now in the name space. They don't get deleted until I mean you can specify that you want that but by default they don't get deleted until MRS are deleted. Okay, let's see next. So type your questions. Uh what is the recommended way to ensure the XR only gets deleted when all MRC managers have been deleted?

08:21 Um yeah that's my previous uh previous answer. Uh and I see that you say that you use crossplane v 2.2 too. But uh are you using uh cluster scoped or namespace scoped resources? Uh that's that's really the question. Uh I'm guessing you're not using namespace scoped resources. You can know that they're namespace scoped because in the API they have M.

08:46 So if it's I don't know u something EC2. AWS something. Uh now there is M like EC2.m something right? Okay. You say NES scoped X XRS then I'm not sure uh they should not be deleted until until MRS are deleted. So I'm not sure uh I would need to take a look at it. If you can ping me privately, we can have a chat or kind of zoom call or something like that and and go through your setup to take a look why that's happening.

09:25 It shouldn't be happening. So uh and the reason why it shouldn't be happening is because uh that's a dependency tree in Kubernetes, right? You have one resource like XR uh and you have MRS that depend on that resource and Kubernetes itself does not allow you deletion unless you make them orphaned. Uh, and I hope you didn't put because that's not by default orphaned.

09:49 But if you can um if you can uh ping me privately, we can we can do a zoom call and then go through your setup and and I can give you better answer based on what you're really running if that works for you. Uh, okay. Next one. Will devops die in the next few years due to an AI impact? Look, I don't think so. it will greatly transform. So I will first and correct me if I'm wrong.

10:19 I will change the word DevOps to platform engineering meaning that you're creating platform for your fellow developers. Uh correct me in the comment if that's wrong. And if that's the case then uh AI uh uh platform engineering will not die uh but it will change. And it will change in terms that your the interface and the tool for your fellow developers is not anymore CLI or some web UI or backstage.

10:51 It's uh cloud code or codeex or whatever they're using. And then your job is to enable their agents to do the right thing through skills through MCPS through custom agents through many different things. Right? So, but think of the how do you extend their tool which could be cloud code or cursor or whatever they're using to do the right thing. Just as before you were extending their terminal or their whatever they were using uh to enable them to do the right thing.

11:25 Now you're extending their agents and that's the crucial difference, right? Uh now that's the near-term well relatively near near-term you know if you say years oh my god I I have no idea what's coming up in in years kind of nobody knows uh especially if you say years as in five years nobody knows what will happen with five years but uh I do strongly believe that at least as it is now AI enhances us AI augments us your fellow colleagues should be more productive with AI.

12:03 You should be more productive with with AI uh and the market will change drastically. So uh you know the stories about AI will kill this profession these jobs are all somehow assuming that we will be doing the same what we were doing before but this time with AI and if AI can do the same the job that we were doing before hence they don't need us anymore but the market will change drastically what will be expected of us will be very different it will be much bigger much faster much many of different things, right?

12:39 You're not supposed to deliver one feature a week. You're supposed to deliver 10, 100 features a week. You're not supposed to have open issues lingering there more than uh a week. Uh you're supposed to be building hardness for your agent that will make you even more productive than just firing up cloud code and say, "What's up?" Right? So I I'm very optimistic that that will not happen.

13:11 In some companies, it certainly will, but those will be the companies that will say our business stays the same, but now it's done with AI. And I think that those companies will die. Those that would be the same thing as saying uh will computers uh result in accountants disappearing and the answer was prevailing no just that they're more they were they became much more efficient with with computers or will internet make um um somebody else some other professions disappear uh and the answer was also No uh most of the

13:54 things historically transformed drastically and I think that that will happen with AI the way you work what is expected from you will change drastically if it doesn't if it doesn't I think you should be looking to change the company because that company is going to go extinct right if the company is the same output but now with AI that's not the place you want to be because that that company has zero future.

14:23 No future at all. Uh okay. Uh more I mean uh cloud-based devops uh and all related things uh still no. Uh the answer is still no. Uh your job will change. You're now be be ready to become a technical manager. That's my suggestion. Uh you are now managing agents. That's your job. uh instead of uh doing whatever you were doing before because if you stay if you think that you will be doing what you were doing before then yeah you will be replaced you're now a manager uh I would be more afraid about traditional managers

15:03 than you you're going into the management position because you have the knowledge you have the experience right you know what to ask you hopefully know how to deal with a agents uh and hardnesses and what's not So I'm I'm very optimistic about it. I I know that my work changed drastically drastically and I don't see it slowing down. If anything actually I have much more work than I spend more hours than I was spending before, not less.

15:36 Um okay. Uh what is the recommended practice? Should provider configs be created via XR compositions or the provider config gets created outside uh XR compositions? Uh that depends that depends. Uh and the answer is very often both right. Uh so normally if you have provider config for AWS, right? You would create it yourself. Uh give it credentials meaning point it to a secret.

16:08 Hopefully that secret was not created directly but was created through external secrets operator or something like that things like that right? So you create it yourself let's say for AWS. uh but then what happens very often is that >> [snorts] >> uh let's say that I have a composition that creates a Kubernetes cluster from a control plane cluster and then that same composition uh would create um uh would create a provider config so that some other compositions can use that provider config to do something in that

16:42 cluster. Right? So imagine you have a composition I create a Kubernetes cluster and then you have another composition. Oh I'm going to uh I don't know uh apply an application over there in that cluster. Uh and that's all happening in control plane and then the one that created the cluster that one would create a provider config or should create a provider config and then the other one says oh I'm connected to this provider config to do something over there.

17:08 Right? So the answer is both uh depends on the case. Um in my case at least provider configs created through compositions are are are actually providing authentication to the resources they created. Uh okay. Uh what else? This is hands yellow heart thread. Uh I'm not sure how what is that but I'm I'm guessing it's good. I hope it's not bad. I'm not sure what it is.

17:39 That's probably related to my previous comments on your questions. Uh, okay. [snorts] Uh, what else? There are too many tools and products marketing with AI sur domain. Yeah, it's BS. It's complete BS what's happening right now. Mainly handling troubleshooting of the application. What is your thoughts on this? I think that there are most of them are really bad, right?

18:02 Uh because here's what's happening. You have let's call them AI native companies that would be entropics of the world right and open AIs of the world and they know what they're doing right that's that's that's their bread and butter they know how what they're doing and I'm fully confident in them and then I have you have many of the companies in cloud native world that have nothing to do with AI right uh nothing to do with it and they re they they all realized or they're realizing while we're speaking that their

18:34 business is going away slowly uh their investors are pulling out because uh the growth is not happening or not at the speed that they expected. The expectations changed uh last couple of years like 2x growth is not good enough for investors. So most of cloud native companies including big guys like uh Google, Azure of the world and AWS's they're all depend on VC money and on investors and investors saying no bye-bye we're not investing in this anymore uh this is not going to be profitable and they're all pivoting

19:13 towards AI and then that pivot towards AI is uh has two problems one is that very often they don't understand AI by agents and all the other things that are happening. They don't really understand it. But bigger problem in my opinion is that those companies are not really thinking what would be the right solution uh given AI and given this domain let's say SR can be any domain they don't think in those terms they don't uh they think more from the perspective of how can I transform my thing whatever the thing is into AI

19:57 I something something. So basically they're wrapping their existing IP, their projects, their services, whatever they are with AI and they're making the same mistake that uh I feel that we were making when back in the day a long time ago when I worked with Jenkins thinking that okay so this is a great product or project uh the world changed uh how can we transform transform it into the new world?

20:27 How can we make it scalable? How can we make it configuration as code etc etc? How can we make it work in Kubernetes as well? And we failed because we were trying to wrap what we had in my case Jenkins into the new world instead of thinking from the scratch. Okay, what would be what would be a workflow like solution CI like solution that is actually designed for cloud that is working well in Kubernetes that is this that is that right that is good for SAS for example we were not thinking in those terms we were trying to

21:04 protect what we have uh just by extending it slightly and then we were always hitting into architectural limitations that really got the best of us uh so having existing IP IP or existing project is what usually makes it crappy. It makes it horrible now with AI because you're not thinking what's the right solution. you're thinking uh how can I uh wrap my solution uh into AI doesn't work when you combine those two existing IP being a problem and um not being really experienced with AI uh you get crappy solutions and

21:45 many of them are horrible really uh and not only that but very often what is happening also is that anthropic and similar are moving so fast that whatever somebody was building gets killed in the next iteration of cloud code say okay they just released this new feature we don't need this anymore right they're just killing competition left and right uh I'm not saying that all AI s solutions are bad it's just that significant number of them are really are bad uh for now that will change uh eventually uh some new players

22:23 will um will come along hopefully Uh, hey, hi Victor. Can you look into my resume and review it when you have spare time? Uh, sure. Drop me a note on LinkedIn or something like that. I must warn you in advance that I haven't been in a job market at least not in that way for like 30 years. So, I don't might not be the best candidate. But if you send me your resume x blue sky, LinkedIn, wherever you want, uh, I will gladly review it.

22:58 Uh, okay. Actually, you know what? Send me two resumes. Send me send me CV that you're planning to send to companies and send me the real one as well. And I know that it might sound strange what I'm asking now, but I know that everybody's inventing and everybody's um claiming things that are not fully true. Oh, I'm very experiences though in all those things and in reality or not.

23:28 So, send me the real resume, an honest one. I will not share it with anybody. I promise. And then send me the one that you are trying to push out. Uh that would help me analyze it. I promise. I promise, promise, promise. I can sign NDA if you want. I will not share it with anybody. So send me both resumes, the real one and the one you're actually using.

23:48 Uh I'm a bit confused with compositions developing delivery workflow. Is there recommended process inline compositions easy to deliver but hard to develop? Uh I actually do a combination of both. And I think that not many people do it like this. So it might not be the best recommendation but I developed my uh the the functions themselves as completely separate source code.

24:16 uh it was KCL in the past now I'm mostly on ponic and that's self-executable uh completely there is nothing related directly related there actually almost nothing directly related to compositions I can test it I can run it I can do whatever I want and then when it comes time I embed it I have a script that actually takes that source code right that is completely independent of YAML and just embeds it into into YAML and publishes it right so I personally use inline compositions but I don't develop them in line if that

24:55 makes sense right so I just at build time I just embed them over there it's like literally copy this put it there right uh into that yaml and then kind of go um so that's what I do and so my answer is both the end result is inline but development is definitely not inline that that's that's horrible horrifying don't do And also I'm I heavily this is not directly related to your question but I heavily um rely on uh chainsaw uh for testing.

25:28 It's just absolutely awesome. I replaced my c setup with the chainsaw. Uh you should try it out. Uh okay. Does it make sense to become cubern or golden cubster out? Uh most likely yes. uh if you're young, right? So, I'm too old for that stuff. Uh I already have plenty of experience and so on and so forth. So, if you have real experience in in good companies like say 20 years or something, uh probably it's there are no real benefits to it kind of you already know Kubernetes, you use it in production, you're good with

26:06 it, you can prove it, uh you work for real on it, then no real advantage. But if you're relatively new, young, uh, then it does make sense because it's it's another paper that says, "Oh, I know what I know and then I'm cub out." Yes. Right. And they have a nice program and so on and so forth. I'm personally not in um just I don't have time for it, but uh if I would be starting, if I would be in my early 20s or something like that, uh I would probably join.

26:39 I would most likely do that. The world belong belongs to young people. You should do all of it. Uh what else? Also, if I'm developing composition with different functions, crossplane render doesn't work. How can I debug compositions locally? So, I personally and again I I do many of the things that are not kind of common and normal. So, don't take me for granted.

27:03 Uh I don't do crossplane render myself. Uh what I do is what I mentioned earlier. um embeddment functions into compositions and then the same script would uh install it in a kind cluster applied directly composition without packaging it into into container image none of those things. So, uh, put it in line, apply it to the cluster, run your tests, right?

27:30 Uh, I I spend more time actually writing tests than composition themselves. Uh, and then, uh, if the test pass kind of like I'm golden, right? I'm good. Uh, and uh, once I've finished, I usually do TDD style. We need a type test. Uh, I apply composition to my kind cluster. I run those tests, they fail kind of I have confirmation that test is correct, that it detects something that I want to have and I don't have it yet.

27:58 [snorts] Do the development, run test again, apply to the cluster, run test again, uh confirm that it's green, and then repeat, repeat, repeat, repeat, repeat until I'm finished. And then the workflow, in my case, GitHub actions kick in. You know, I create a pull request. Uh it runs all the tests again. um merge domain create a release and all the address right so I I don't use uh crossplman renderer myself to be honest I don't want to look at it I want to test that it works uh and I test it with chainsaw uh mostly

28:35 right some some things you need to test really because my setup of chainsaw does not really create actual resources in AWS azure google other kubernetes clusters it just validates that um the resources themselves created by cross min are correct. Uh and then the last round of validation after I cut the release or sometimes before uh but yeah not using render.

29:00 Sorry for that. Uh I know that many people do. I just don't I I do things in a weird way. I'm not sure what's wrong with me. Uh okay we have capabilities in all the terms but why don't we have any proactive solutions in AI s domain? All the tools are reactive. what's your thought on that one? Um so the reason why it's reactive uh usually it's a natural progression I I believe uh meaning that uh it starts with I don't trust so if you have two extremes it starts with I I I have zero trust in AI and agents and all the

29:39 stuff and then over time you improve and you you do better you you learn how to deal with it and how to work with it and what is required and so on and so forth. And then the other extreme is fully autonomous system. Right. Uh I don't think that many if anybody is at the fully autonomous but somewhere in between. Yes. Right. Uh so you start with being completely reactive kind of like AI does not do anything without you saying do this and then reviewing everything and stuff like that.

30:10 Uh we're in like I'm I'm I'm somewhere in the middle personally. Um I have a Kubernetes controller for example uh that watches for events classifies the events and let's say if if this event if an event is repeated if a warning sorry warning event because there are no errors error events in Kubernetes if warning happens the same warning more than three times let's say or three times or more then fire request to an agent and then agent analyzes it uh and figures out what's wrong and depending on the rules I set it

30:55 either sends me a message on Slack email or whatever uh hey uh this is the problem this is the solution deal with it yourself but kind of here's the whole solution including commands you should execute to solve it uh but if the confidence threshold is above certain number then it would skip notifying me and just kind of like okay analyze this this is the problem this is the solution this is the fix I'm going to apply the fix create a PR uh sometimes even merge it itself uh or wait for me to to merge it uh and that's

31:30 very uh proactive right uh I mean it's still reactive in terms that it's not fixing problems before they happen I'm I'm not sure whether that's what you mean by proactive uh so not fixing problems before they happen but it does fix problems reactively in a way uh when they happen right kind of without my intervention without me coming uh and checking it out and so on and so forth right so that's happening not necessarily I don't know of many solutions that really do it well uh and that apply to everybody kind of out of

32:06 the box but it's something relatively easy to build yourself right you can also check the the project I have it I I call itai that was toolkit AI Okay. Um, check it out. Use it as inspiration. It's very remediation itself is very proactive or reactive in terms like we still waits for for an error to happen but then it's just um does [snorts] the stuff.

32:34 Okay. Uh what next? Uh hey Victor, we're starting to use chainsaw for end to end testing during EKS GK cluster upgrades. any recommendations for automatically triggering and managing tests across 20 plus clusters after upgrades. Um, [sighs and gasps] I approach it slightly differently. I don't test that actual upgrades work. I test that the service I'm providing works.

33:11 So, think of it this way. Imagine that you're AWS, right? And if you're AWS, you're not testing whether all of your EC2 instances that anybody ever created or will create are working correctly, right? That's not your job. Uh your job is to test that EC2 as a service works correctly, right? And you test it to that with all the permutations and and everything that you can imagine.

33:39 new test and say EC2 works kind of like we are confident that it works 99.9% of the cases right uh and users can create as many EC2 instances as they want that's not for us to test we're testing it as a service not individual instances of that service right so I'm not testing that upgrades myself upgrades across across Xclusters work I just make sure that upgrades of my compositions work in general, not on specific clusters kind of like I tested this to that.

34:16 I make a new release and I'm confident that now you can upgrade your clusters and you'll be fine, right? It's never 100% fine, but most likely fine, right? Um so, um I'm not saying that you shouldn't be doing what you're doing. Uh that's not what I'm saying. Just I'm not working that way. Um okay more I'm doing the same developing compositions a separate file.

34:40 Yeah and then rendering them in YAML with go plate uh script and put it in GitHub 3. Yeah that's that's what I'm doing. Yeah. Uh the only maybe difference is that I'm packaging the end result in my case is a container image that is packaged there in the marketplace. Uh and then uh my GitHub repo just changes the version. Right? So I packaged my container image with the whole composition everything.

35:10 Now it was version 1.1.1 now it's version 1.2.0. Uh [snorts] and then I changing it to 3 just just the tag nothing else. Uh but yours works as well. That's for example how Scott does it as well. Uh oh, we got uh five bucks from uh Akivalu which is on Serbian. No, maybe Croatian promise a new deos which means thank you for you helping improve DevOps knowledge or something along that that way.

35:44 Uh I'm half Croatian, half Serbian, but I'm never 100% sure which one of those two languages which are essentially the same it is. Anyways, Akali, thank you so much. So, so much. [snorts] Um if you would hire dev sec ops what see first uh that in my case it always depends on the role I'm hiring and that's whether that role is supposed to be very specialized or not right um and I think that's very specialized is maybe slightly less common now be AI anyways if it's very specialized kind of like oh I'm a networking person

36:32 kind of like I I do network that's usually very large organizations to begin with kind of I I know network there is no person on earth on this planet doesn't exist a person who knows networking better than me I know nothing else then I know nothing else I know networking I'm extremely specialized and I'm the best of the best right uh and then you have more wider people I I include myself in that group uh who are not really um the best in any single area but have a breath of knowledge.

37:03 Um and in that case um uh in my case what I'm looking for really is first experience and when I say experience not necessarily experience in exactly the things we use uh I don't think that that's that important but more experience in general right uh because hey uh if you have real experience development in development let's say dev part and I believe that dev part is always overlooked uh if you're very experienced in development and you have real experience in real amount of time and you know how to write you have

37:42 working experience with Java and with Python with JavaScript and we work in Go okay maybe it will be better if if if you if you have a lot of experience with Go but it's fine because you know how to write code that's what I really care about uh and because switching from one language to another is not that that big of a deal. Same thing like let's say cloud hey if you have real experience uh let's like where I where I am we primarily use Google cloud but you have real experience with AWS and Azure kind of um that

38:16 that's great that that's enough right kind of like and I if if if I'm confident you know what you're doing [snorts] switching to Google cloud uh is not a problem right so uh experience more than anything uh that's what I'm looking for I I repeated the story before. So I'm but I'm going to repeat again. Uh once that was long time ago uh I gave people everybody a test to do something silly.

38:43 Um and whomever open the browser got the job uh in that case because I was really looking for people who are proactive who are going to figure out how to solve the problem no matter whether they know it or not because nobody knows everything. Actually I would even say that if you look if you work in a company where actually you know what you're doing all the time uh it's a wrong company because that means that it's a company that is not solving new problems.

39:09 Um so experience productivity the ability to solve real problems uh that they never saw before. Uh okay what's next? Having the agent to warm up a ticket issue is a really nice feature. Yes it is. even if it can figure out the cause of the issue. Uh yeah, you know, people look at AI usually in very extreme ways kind of oh it's incapable of doing anything or it can do absolutely everything and and I think that both approaches are wrong.

39:44 Uh if it can help me 10%. It can reduce my work by 10% it's great. It's absolutely great. If it can uh do 50, it's even better. If it's 90, oh my god. Um, so yeah, warm up. Uh, I use it all the time with my pull requests with my issues. Uh, it always first pass with AI. Sometimes the end result is me taking a look at it and say, "Yeah, yeah, just do it."

40:12 Kind of that that's the right thing. Just do it. Sometimes uh we have a conversation sometimes even very long conversation about all the things and some other times it's kind of okay this is wrong right this simply you're wrong try again but try again now with the input I'm going to give you right um so yeah it's it's it's it's a really nice feature right doesn't matter how how how warm it makes Okay.

40:46 Uh from Shamil, maybe I should just use Helm instead of complate. Uh go template. I guess it's go template, right? Uh like Scott suggested. That way I can deliver not just composition itself but uh related manifests as well. Uh no, first of all, uh I doubt that Scott ever suggested Helm. Let's start with that. >> [snorts] >> uh because uh there are many things that where me and Scott disagree but one of the things that we do agree is that Helm is horrible.

41:18 uh neither of us really likes it and the main reason and this is important the main reason why it's horrible uh is go templating right uh and um I think that go templating is really really bad and it's bad simply because it is not data aware it does go templating does not understand it's not designed to work with data and yaml is data it's structured data that's what it is yaml json all this stuff right it's simply uh free text templating engine and and it's really really bad really really bad as a language just to be

41:54 100% clear uh and that's our problem with go templating that's our problem with helm itself if you would uh remove go templating from it I think that both me and Scott would like helm actually because there are many other good things but that that's the biggest one and and we really dislike it so neither me nor host Scott will give you a thumbs up forgo templating directly or through Helm.

42:21 Um um so yeah. No. Um what next? Any tips for the current job market? I honestly don't know. I'm not I'm I'm rarely in a market myself and I'm kind of fortunate lucky let's say lucky that uh when I do plan to switch a job the nature of my job is is that that we all know each other more or less so so I never go on the market really it's usually kind of um joining straightaway u a company uh so I I I'm not good at that I read uh I I mean simply I have no experience in the job market that that that's the problem.

43:07 So I cannot answer that question. I did read statistic that actually the number of hires at least in the US. I'm not sure about the rest of the world. I'm not in US by the way but uh I read about US that it's actually up it increases which is very strange given that many companies are hiring like crazy and many companies are firing like crazy. It's it's insane.

43:30 But I have no experience on the job market. Sorry for that. Also can you tell us a bit more about how chainsaw works? Uh so J Jensen started like so first we had KL KUTL and that was a testing framework for Kubernetes and that project end up not being maintained really or not much and then um what's his name I forgot the name doesn't matter a person from chainsaw company said uh Eddie Eddie uh said okay uh heck I'm going to fork and uh fix the problems I'm facing with it and then he realized that actually it's not good

44:15 either it can be so much better and that became more mature version of chainsaw uh what it essentially does is that it uh it can apply you know apply delete update all this jazz resources in your Kubernetes cluster just as you would do uh when you know executing cube control apply uh and then uh it can assert right so it basically you specify okay so I have this deployment the spec should have this and metadata should have that uh it only validates that what you are asserting are there because kubernetes resources are

44:58 very noisy you will have a lot of runtime information so it's not saying those two need to be exactly the same but the fields and values specified in the assert should be the same as uh in the cluster. If it's not, it waits and it tries again and again and again because in Kubernetes it's a eventual consistency. There is no guarantee that it will not become what it should be a moment later.

45:24 So it asserts it uh repeats repeats repeats if it fails and if it succeeds says okay if it fails on a timeout then you have an issue. I mean you have an issue, you have a failure and you get a diff says okay you expected this resource in Kubernetes to have this value and this is the value it has instead right just comparing resources essentially um so it's relatively simple um the way it does it is absolutely awesome but it's really asserting comparing that what you specified is what what's in a cluster um that's

46:01 that's the gist of it uh you can use uh what's the chainsaw JSON something something language to do some really funky stuff some kind of complex uh but at the very beginning you can just specify pure aml then you can go into templating and and all the stuff that you can normally do with I think it's chain so I think it's converson or something like that so it can get really complicated you want to go very funky but at the beginning just compare yamo with yaml um do you plan to buy SpaceX stocks and invest in a data

46:36 center in space. I have zero uh zero plans. I I am buying stocks sometimes when I have spare money. Uh I have zero intention to buy SpaceX stocks. Uh and this has nothing to do with my opinion about Elon Musk. uh no matter whether it's positive or negative, it's simply um that's one of the that that is one of the IPOs that is led by emotions and not really uh data, right?

47:08 Uh if you look at it from the data perspective, earnings are non-existent. uh the plans that they have for earnings are impossible and the realistic plans of their earnings is way below of what what they value their stocks. Uh so uh all the data you would normally uh use if you use data like you would normally do for other stocks almost nobody would ever say bite.

47:41 Nobody uh most of the I think that most of the purchases of their stocks when it goes IPO will be emotional. I believe in it. I believe in the vision. I believe in that man in Alen Mask and so on and so forth. Uh I'm too old to to go emotionally into those things. Uh so I'm not I think that uh if that if he did not combine companies into one uh SpaceX company if um whatever was called the one that uh was doing the oh my god the the one of the companies that was doing uh satellites that stock I would buy immediately.

48:32 ely if the price is right um but uh now that it's I think a combination of three different companies XAI I don't believe that that's going anywhere uh and um and actually XAI is supposed to supposed to be the major driver of uh of the growth of that company I don't believe it so the short answer is no uh have had real production experience and unable to pass any interview I never never had real production experience and unable to pass an interview mean studying Kubernetes uh 6 months and looking for an advice and on

49:11 maybe a platform or mentor of of course that would help me with that gap um I think that I feel again I I don't have my experience with that is long time ago uh but I feel that the Best advice is to uh two things join a community become really active in if you chose Kubernetes uh be become really active in CNCF uh there is slack channel there are local meetups and so on and so forth um when you can I think that that's where I would spend my time and also uh start contributing to uh to one to any project you choose,

50:08 whichever you like the most. Um because uh it's a chicken and egg problem, right? Everybody wants production experience and then if you don't have production experience, nobody wants you. Um that's a really tricky one. Uh but then at least until you land that job, uh you can say, "Hey, I've been working those projects. We are hiring for example where I am."

50:30 uh if you have demonstratable experience um and that's more than a few months just to be clear uh working on a project opensource project that actually in our case trumps any other experience right oh if you contribute to kubernetes oh my god you're hired immediately um but again there that's not quick fix you start small and that takes years but it's something you should start investing right away uh I believe and also uh I think you should be targeting bigger companies like where I work with 60 70 of us something

51:08 like that um [snorts] it's too small of a company to hire juniors and we are all remote and there's nothing wrong with it just to be 100% clear uh but we're too small to to hire too unorganized so it's usually you're hired go what do you mean go go just go and do stuff like that. That's the strategy. Um and many startups are like that. It's chaos. Uh it's very often remote and you're assumed to know what you're doing.

51:44 uh if you go for bigger companies like you know Microsofts, Google's, Redheads, uh Susa and so on and so forth, they had they're big enough that they can have a pool of uh entry- level jobs. Uh so that's probably where I would be I would be going, right? Uh and until that happens, uh contributions that's that's the closest to production experience without being production experience.

52:15 I think again it's been 30 years. I don't know. Uh is Argo workflow still an option in 2026? It it is an option. Uh I'm not using it. Uh I'm sick of managing. I'm fortunate first of all that uh uh all the all the projects I'm working on are in GitHub. uh and I use GitHub actions and I would consider an alternative only and exclusively after GitHub actions shows the limitation and even then I would think twice about hey is this really limitation of GitHub actions or I'm doing something stupid and crazy.

52:54 Uh so I haven't used Argo workflows for a couple of years now. Uh it just got tired tired of it tired of managing it. Um but I'm very fortunate that I don't work in with organiz in organization that is restricted you know everything needs to be on prem and so on and so forth. Now if the reason for using Argo workflow is that hey everything needs to be self-managed everything needs to be inside your data center then I would almost certainly choose Tecton over Argo workflows I think that Tecton is a better project for

53:33 that for in that category than Argo workflow um we can debate whether it should be Argo CD or plugs right or uh what's flagger or whatever is the Argo seed Argo rollouts. Uh but when it comes to workflow, yeah, I'm confident that Tekkon is a better choice. Okay. Which hardness seems most promising to use in Kubernetes? Cent tohive? Uh I don't use any of them to be honest.

54:07 I honestly don't. uh uh cage first of all I'm not sure what the value they provide so actually um let's start with this uh it depends on whether you want a quick fix like kind of oh I want a custom agent uh the where I define system prompt and MCPS uh in that case I'm not sure why do you want that because cloud old code is so much better. Uh but if you want to go beyond that, I think you should be looking into Versella SDK uh or one of the many others and just write your hardness then you get real value.

54:59 It takes more time but you get real value. Um, so if the question is for creating agents, I don't like it. If the question is how to run agents, create it any way you like. I don't like it either because I don't see what the value is. It's literally just a deployment service in English. If I have my custom agent, it's just like any other application.

55:25 Um and cloud code talks to that agent uh through MCP 82A but A2A is really not well supported. So uh no I I don't like any of those. I I used all of them. I had high hopes. I I don't like it to be honest. I don't see the value really beyond maybe first day kind of like oh I want to create an agent and kind of like here's the system prompt and connect to those three MCPS but uh just use cloud code then if that's all you need or colleagues uh okay uh from uh that's probably reference to some of the previous questions let

56:08 me try to figure it out is referring more to orchestrate chainsaw tests across multiple clusters and providers. We're starting with validation of core cluster components and are looking for ideas and scalable approach. Uh unfortunately, I don't have many ideas because I'm not doing that uh myself, right? Uh I'm not trying to test uh I'm not trying to test individual composite resources.

56:38 I'm trying to test that composition itself. is working correctly and then users including when that's me just kind of create instances of that. So just my workflow is different. Sorry. [snorts] What else? If you would hire dev sec ops uh no did they do that? I I did I think I No, I I jumped uh before. Uh now I need to find out where I am. Okay, here I think that this is the next one.

57:14 Where do you personally see software engineering in the next 5 to 10 years? I have no idea. 5 to 10 years. I honestly don't know what will happen next year. 5 to 10 years, nobody knows. I my brain cannot comprehend. Uh just look at the changes happening last year kind of like next year maybe I can predict but five to 10 years impossible. I think that whomever I who know I don't know really it's moving so fast I cannot even follow anthropics release notes anymore that's how fast it's going do software engines become

57:49 empowered in other fields like robotics healthcare etc uh I mean software engineers are already in other fields like robotics healthcare and so on and so forth just to be clear that has nothing to do with AI uh I have friends who work like software engineers in healthcare Um I have friends who work with robotics, software engineers, just different areas of software engineering.

58:14 Or does everyone become a software engineer? Uh so do software engineers become empowered in other fields? Do you mean by that question you you need to write a comment kind of do we switch career from software engineers to become healthcare? But if that's the case, I'm not sure kind of you still be a software engineer for healthcare. And that's that's independent of AI or you mean kind of like really become a nurse.

58:41 So I'm not I'm not sure what you mean by that. Uh what else? What actions do you take to shift the mindset operationally from ECS to EKS? Can you list some of the things EKS introduces in production? Uh, first of all, it's been years and I'm literally mean years since I gave up on ECS and said I don't want to see it ever again. Uh, I don't even remember all the reasons I had.

59:10 Uh, I know it's a piece of, you know, uh, There you go. I said it. Uh, and no, and BKS is is now much more widely used within AWS. Not to mention that Kubernetes is much more widely used everywhere. Uh don't even think about it on a list. Simply you're talking about legacy stuff and if it works for you and you have zero investment in it, keep it. That's my strategy.

59:37 The moment you you need to invest in it to do something new extra, just forget about it. God for EKS. Uh even AWS believes in EKS now. So um and apart from simply case being so much infinitely better uh it's the ecosystem there is selium for networking there is argod for for for flags for okay so here's here if you want one thing and that's the ecosystem right CNCF is the biggest ecosystem in software engineering on this planet you're missing out on all of you're missing out on narosidium flux on selium on uh onto

01:00:29 search manager on crossplane on crow or so on and so forth kind of you're missing on hundreds of the cool projects and innovations or thousands that happened since emergence of kubernetes and you're locked at largely abandoned [clears throat] service from AWS that that that's the list. Uh he said in the last stream that uh packaging component files to Kubernetes manifest with Helm or I got it wrong.

01:01:02 Maybe I got it wrong but I would be very surprised that I got it wrong because I know that Scott believes in Helm being useful only for third parties and I share that opinion fully. third parties. Yes, because that's the most widely accepted distribution mechanism. Everybody publishes their Helm chart for your own stuff. Whatever the stuff is, no helm.

01:01:26 Uh I'm pretty sure uh he couldn't come unfortunately this week. Uh Shamil, please please please come next week, next Thursday. he will be here hopefully and uh ask him because now I'm curious that I misunderstood his words but I doubt is it normal that as a as product team can only do basic stuff regarding Kubernetes our cluster is lacking controllers and this really blocks us uh depends what do you mean by normal is it common let's use that word that people only do basic stuff.

01:02:05 The answer is yes. And this is nothing to do with Kubernetes. In general, people do basic stuff. That's I mean by definition advanced stuff is something that uh the bar what is called what is qualified as advanced is reserved for a majority of the top right. Uh so by definition majority of people are actually using sorry minority at the top. So by definition majority of people are doing normal common uh stuff right in general and that includes Kubernetes.

01:02:38 So yes is it normal? Is it common? Yes. Should it be? No. But when everybody starts doing something more advanced then what is considered advanced will change again right? Because let's put it this way. Whatever normal common things you do with Kubernetes that's infinitely more advanced than what I was doing 30 years ago. not me but the industry as a whole right even the most advanced in in in the industry uh plate is ah goo plate I you know I thought it's go template complate is a tool that allows you render files

01:03:14 from templates using go template engine just like helm but without help okay so I was not very aware about about complate still I think and I want you to ask this question next week to for Scott to confirm that God templating is the worst option of them all without a doubt. Uh as as as the way you write things simply because it's not designed to deal with data.

01:03:41 It's as simple as that. KCL better YT better. Uh anything is better really anything. U what whatever you choose better than free text templating engine because think about description of God templating. What is it? It's free text templating engine. No. Okay. I got to thank you. Thank you for thanking me. Uh what is devil's direction? Uh what is devil's direction is the worst?

01:04:14 I'm not sure I understand. What is the worst? Can you can you post a comment? I'm not sure I understand the question. Sorry for that. Do you use loops in cloud code? What is it good for? Oh. Uh, I don't I don't use loops, but that's mainly because uh I built my own um hardness for agents. Uh I run them on a remote server. I connect them to them from wherever I am.

01:04:47 Um I have an application I wrote it in Rust that orchestrates different agents for each feature and then creates multiple features. I normally have like 20 30 50 agents running in parallel typically five or six working on each of the tasks. Uh so I'm not using loops myself. Uh so [snorts] uh um no no deep experience and I try to avoid answering if I'm not really um proficient with something.

01:05:23 Uh Flux is using true helm engine to deliver charts. Correct. Recently run into an issue when it can't deploy envoy gateway chart because cds are too big to them in the secret. Is is is it legitimate to deploy CRD separately in which way is the best way? Uh unfortunately it is legitimate unfortunately uh because CRDs are too big sometimes they are interdependent and so on and so forth.

01:05:50 Uh so yeah it is legitimate to deploy them separately. I would still wrap them into customize um because customize should be able to reference external uh file, right? Uh so that you don't copy the whole CRD to your repo. That would be insane. So probably use customize to reference those files and then point flux or whatever you're using to your customize for the CRS and [clears throat] we are out of time and out of questions at the same time.

01:06:28 This never happens. Most of the time we have more questions than the time. This time just the right number of questions just right amount of time. Probably because I was going faster than when when when I was. Okay, one more last one. Uh I mean I develop composition body separately then rendering it in the Kubernetes manifest with go template. So ignoring now go plate go template kind of that's separate discussion.

01:06:51 Yes, that's the way to do it fully. Yes, on on the process. And that's uh that's a wrap for today. Um hope to see you all next week. Uh and I hope that Scott will be able to join us. So see you all next week. Cheers.