What should security leaders do with AI? They don’t know. Transcript, AI Summary & Key Points

00:00 Cyber security leaders can't figure out where to deploy AI. Panelists, where do you think they should start? Dave, we'll go to you first. >> I think they need to start at their red teams. >> Start with AI in the repetitive tasks, not so much trying to bite off more than they can chew. >> It's good for repetitive tasks, especially those kind of like L1, L2 ones, looking at alerts and stuff, preventing alert fatigue.

00:28 Hello and welcome to Security Intelligence, IBM's weekly cyber security podcast where our expert panelists turn the biggest industry news stories into practical takeaways that you can use. I'm your host, Makazinski, and joining me this week, we've got Claire Nunees, creative director, IBM X Force Cyber Range. We've got Curtis Pittz, lead CISO Trust.

00:45 And we've got Dave Bales, North American lead managing consultants at the Exports Cyber Range. Today we're going to be talking about ghost jacking and whether AI is actually good at patching or not. But first, security leaders are feeling paralyzed by AI. Axios reports that security leaders are struggling with the question of how they should invest in AI.

01:11 Many of them have the budget and the buyin to do it, but they feel overwhelmed by the options and by the constant state of change, which I think we can all kind of sympathize with. You know, ever since Mythos came out, I feel like there's a new giant AI security story every single week. And this also reminded me of the most recent cost of a databach report, which found that 64% of organizations report limited or no use of AI in security functions.

01:35 Meanwhile, AI generated attacks jumped 56% year-over-year. So, there is some real urgency to address this problem, but they just don't know where to start. Claire, I want to start with you and I want to ask specifically because I know you work in the cyber range, you work with clients. Have you seen this kind of decision fatigue with anybody in in your line of work?

01:53 Have you experienced this? >> I think AI transformations are really scary um you know for security whether they're in the security function itself or in the business side of things. So in the business side of things it's really scary to be like we're moving so fast and maybe we're not encompassing everything. And then on the security side, it's like how do I get my staff to understand that this is going to help them and not just kind of burden them.

02:20 So I think it's it's there's a lot of partners also that clients can work with. So it's it's I think there's so much decision fatigue in general around like what can I do, who can I work with, like what's going on in the other parts of the business, where can I automate in my side of the business? Like how is this all impacting everything? Is everything still secure?

02:37 So I I think people get a little frazzled about it and they get very frazzled especially about like the mythos everything going on there. So once you kind of combine all that people are a little bit like I just don't know what's going on. That's kind of where where people land. Uh and it's hard to kind of get your footing and make a good decision or what you think is a good decision rather.

03:01 I was just going to agree there and and say that the other thing that they have to focus on is how do I keep my employees from going crazy thinking that AI is going to take over every facet of their job. We see it. Every other company in the world sees it. It's it's a scary thing for somebody who's not used to that. You know, growing up when I did, if you'd have told me that there was an artificial intelligence, I would have thought we're living in the Jetson's world here.

03:31 Where's my flying car? >> I think part of the problem too is there it's it's been made to be such a big deal that everyone now fears the consequences of making the wrong decision, right? Like in the business world, we know that you fail fast, right? You fail fast, you fail cheap, you move on. In the AI world, that feels heavier than just making the wrong business decision, right?

03:52 It feels like the impact is somehow going to end your business or get you breached and all your data is going to be out to the world. So, I understand the the apprehension, but also I still think we need to fail fast, right? Like there's so many I mean, imagine if the day the car was invented, everything that existed today existed, right? You'd have a hundred cars to choose from.

04:12 It's not just the Model T, right? There's so many options out there. That didn't really happen before to the scale that it's happened now. Um, everyone was in the market incredibly fast, right? And so it aided that decision paralysis a little bit, I think. >> Yeah. And piggybacking off of what Curtis said, failing fast isn't a bad thing. Everybody has to fail in order to succeed.

04:37 If no one ever failed, no one would succeed. It would just be the status quo, which, you know, nobody really wants to be the status quo. We all want to achieve better than that. >> That's a really good point and and it's something I hadn't thought of because especially the way that AI is kind of positioned in the market, I think doesn't help, which is like here's your silver bullet.

04:56 we're going to solve everything, we're going to do everything, yada yada yada. You almost feel like if I deploy this AI tool and it doesn't work, what did I do wrong? You know what I mean? Like there's almost a kind of like shame around if you can make it work or not. And this idea of just embracing failing fast, I think, you know, it's it's one that's been with us for a while, but I don't know.

05:13 It seems like it got lost in some of the AI discourse, I feel like, uh, which I I think just in general doesn't help with this sort of thing. >> I think some of it has to do with cost and the way that that it's rolled out, right? there's so much cost involved in rolling out AI. Failing fast isn't failing cheap, right? It's failing very expensive. Um, and that matters, right?

05:32 So, I think I mean, not I know this isn't a business conversation, but I I think there needs to be kind of a re-wizzling, if you will, of of the way that we contract AI stuff and the way we procure some of those. you know, the government, not that they're very good at contracts, but they had a good model with deliver now, but if you don't deliver, I'm not ex exercising my option years, right?

05:53 Like they're short-term multi-renewal contracts. Um, where we tend to be in, you know, 3 to 5year contracts. They need to be one to twoyear with some option years. >> And, you know, you say it's not a business conversation, but like cyber security is a business part. It's part of the business, right? And like that's I think why there is this paralysis because there's an awareness of this is a major investment that I'm asking my organization to make like I need to be able to justify this kind of thing.

06:18 And I do like that idea Curtis. So maybe being a little bit more I don't know maybe the word is nimble with how we approach these contracts but like making it so that the commitment the initial commitment isn't so large so you're freer to fail fast. Um but we opened up this episode with you asking you all where do you start? Right? So, we've kind of talked a little bit about the reasons for the paralysis, but now I want to dig into the okay, how do we shake ourselves out of that?

06:40 And Dave, you opened first with talking about bringing it into the red team. Can you tell me a little bit about why you're thinking red team and what you like to see people do with it there? >> I'm going to go back to a sports analogy, but a great offense is just as good as a good defense. So if you if you put the put the AI in the hands of the red team, let them learn and figure out exactly what the threat actors are doing, that makes them better prepared to face off against those challenges.

07:05 So give the red team something to do, you know, when it comes to figuring out how are we going to protect against these threat actors that are always, always, always coming after us. Give us the same tools. Let us figure out what they're doing. We know how they're doing it. let us figure out defenses for that. >> Yeah, that makes a lot of sense to me.

07:26 Especially again, I keep going back to Costa data breach because it's relatively recent. It's fresh in my mind, but I think about that that 56% increase in ad generated attacks. So, like we see attackers are moving very fast and there's a lot of pressure to like if they're moving at machine speed as they say, we should do it too. I think it makes sense, Dave.

07:40 You're right. Arm the red team with those tools so they can basically get into that hacker mindset, see what they're doing so that they can develop those defenses. I like that a lot. Um Claire, you had mentioned kind of uh automating those repetitive tasks. Can you say a little bit more about that? >> Yeah, I think that's where AI can really shine also is is repetitive tasks.

07:58 Um so, you know, if you're getting the same kind of alerts kind of triaging those kinds of um incoming kind of events, that's really helpful. Um it it also just reduces the the fatigue, right? like if you're going to be implementing AI in a in a very not I don't want to say simplistic way but a baseline way it's a really good way to help your organization kind of start with an AI security journey and there's a lot of options there as well to kind of build those into your security culture.

08:33 >> Yeah. And what I like about that is that it's one of those things where because it's a kind of baseline level activity, the activity itself is very well understood. So it's not like you're trying to work AI into a complex workflow. you're like, I know how this works. Let me put an AI to work here. That seems like a very nice way to test it. Um Curtis, you also were were on that repetitive task tip.

08:50 Can you anything to add there? >> Yeah, one of the things my team is doing is we're we're doing our best to automate the things that I have people doing thousands of times a year, right? So, vendor risk assessments, um contract analysis, that kind of stuff where the parameters are known, right? The values are known. And so just setting an agent to consistently validating against known parameters um both takes two people's worth of time off of that they can do more important things um but also allows us to find novel

09:21 ways to branch that capability out internally without a ton of risk um but also gives us the time back to do that right to try to figure out well where can I expand this to so there's I mean across the sales teams there are probably tens of thousands of things that come in every year that are super repetitive. And we're doing our best um within my specific trust team to alleviate those repetitive tasks from the sellers as it comes to engagements and and risk assessments and client conversations and all the fun stuff

09:50 that goes into cyber security and sales. >> And I would be remiss if I didn't mention something here that IBM's Dimple Alawalia shared with me in a recent conversation about this very same topic. We're talking about where do you start with with with AI adoption and her words of advice were basically almost like stop thinking about the AI tool. stop thinking about the AI angle and think about what are you trying to achieve.

10:12 Start there and once you know what you're trying to achieve, then figure out whether or not AI is the way to do it. Sometimes it will be, sometimes it won't, but I thought that that was some really useful advice. So, I just wanted to share it here. And for the listeners, know that we'll be releasing a bonus episode with Dimple later this month all about that conversation so you can hear her whole take.

10:27 Uh, but I do have to move us along here, folks. Uh, if you're watching on YouTube, leave us some comments. Let us know how you're feeling about, you know, this paralysis. Are you experiencing you're seeking your organization? and what are you doing about it? I love to read it. I love to respond. But our next story today, this is ghostjacking. At Defcon, Tenant Security reported on a new way attackers can poison content in trusted systems to trick agents.

10:54 The technique, which they call ghost jacking, is essentially like a a sophisticated kind of prompt injection, like an extra sophisticated prompt injection, because it sneaks malicious commands into some of the most highly trusted systems we have, right? Alerts, logs, error reports, things that we assume are are good things, right? One example they gave was attackers embedding the prompt in a connection request that Cloudflare that a CloudFlare firewall successfully and accurately blocked, but then when the agent read

11:23 the log recording the the blocking of the con connection request, it read the malicious prompt in the request and it was taken over. Right? So, it's like the firewall worked, but the agent still got compromised. This is very interesting to me. And ten it says clawed code fell for this trick nine out of 10 times, which was a lot. Uh, and we've just been talking about organizations are struggling to incorporate AI into security.

11:43 How can they do it? Something like this comes out, maybe you think, should I do it? I don't know. And and and Dave, I'll start there. Does this complicate any of your feelings about security, AI and security? How you looking at this situation? >> It doesn't complicate things that much. It's it's now it's known. It's out there. You know, the the the good people at Defcon have have opened our eyes to a myriad of just crazy things that we never thought possible.

12:10 >> Yes, they did. >> And [laughter] >> because that's what they do at Defcon. They open our eyes. But this ghost jacking thing that that was extremely interesting to me to see how you go back and you read the logs and and boom, there it is. you know, it it I I can't wrap my head around how someone sat down and figured out how to do that, much less how to make it work.

12:34 >> Yeah. It's it's you know, the prompt injections have been this like pernitious problem and they we just thankfully security researchers keep coming up with like new and exciting ways to do it. But that also means we have to figure out new and exciting ways to fight back and and and and I don't know what to do about that. Uh Curtis, how about you?

12:49 Uh any thoughts on this story? And it's got you rethinking AI and security at all? Where you where you landing here? No, I mean if you remember the last chat that we had on this particular podcast, what's old is new again, right? Like when we when the internet was young, right? DNS spoofing was just injecting DNS into a cache table, right? And so really, we use an agent that's designed to read something and execute it.

13:12 There's not, unless you design it that way, there's not a lot of logic or or necessarily parameters built into things you think are secure, right? And we had to figure out DNS security way back then. And now we have to figure out AI prompt injection security now to stop those types of things from happening. You presume this happens across the entire security landscape, right?

13:33 You presume that certain things are secure automatically because there's never been a reason for them not to be considered secure. And then as you create new tools and new capabilities and new agentic stuff, we run into the problem of like, oh, this isn't as secure as we thought, right? And luckily there are teams of good people that are finding these things out, right?

13:52 to Dave's point, like the folks at Defcon have showed us a lot of things over the years. Um, and it wasn't some bad guy figuring it out the wrong way. Like at least at least we've got insight from from the good guys. Um, but I think it's it it doesn't scare me. It doesn't slow me down. It's just we overlook things because we assume the things that have always been secure are always going to be secure.

14:15 uh zero trust says not to do that but as we well know we skip over the things that we think are secure and are easy and have been taken care of for years. So we find ourselves in this bucket periodically as we invent new things and people find new ways to hack them. I think the zero trust mention is is really salient there because I think that like when AI enters our the enterprise, it brings a new emphasis to the importance of doing that zero trust thing because like you said Curtis, so many of these things that we

14:41 assume are safe because we worked it out, we figured it out already. They get upended when you introduce something in there that like doesn't differentiate between the code that's running it and and the untrusted user input, right? That used to be like that was, you know, part of writing secure software was to keep those things separated. LLMs by nature don't do that.

14:59 And so now it's like okay we got to figure out how we approach this and and and that you know leads to tenant their argument basically that like this is kind of an identity and access management problem which is like okay we know that agents can be compromised in this way we should approach it by figuring out how we give them the proper permissions and harnesses so that they don't do anything bad.

15:18 Claire I'm wondering uh you know from your perspective do you think an identity and access management lens is the right way to approach this? Any other thoughts you have about this situation? Where you where you landed on ghost jacking? I think it's like a very true form of hacking uh in terms of like making something do something it's not supposed to do.

15:36 Um and as Curtis mentioned, it is something that's new, but we will adapt to that. So, it's really scary right now. Um and it may not be as scary. Well, it will probably be scary in a different way uh in a couple months, but it it is it's something like we can figure out, right? Because as as we hack things and make things do things that they shouldn't do, we we kind of like counter hack and do the same thing to defend in a different way that we wouldn't have defended before.

16:04 Um so I think it's really interesting. Um I think it plays into a lot of aspects of yes ident identity and access, but it also plays a lot into just kind of security broadly, which is also a short way of saying it. It's just like something that um you know if you're not expecting it, you're not going to to look for it. So I think it's something that you know we can kind of work on discovering as time goes forward.

16:34 >> Absolutely. And and I think that builds off nicely about what you know Curtis was saying before about how like this kind of you know in a lot of ways it's very similar to what we're doing with like DNS spoofing back in the day, right? And and we're not starting from zero here, right? when like when when AI poses a cyber security problem, there are tried and true principles we can look at.

16:51 You know, one of the kind of slogans that's popped up on the show over and over again is we know how to fix this. We know how to fix this and we can we have the tools at our disposal. It's just about applying them in the right way. Uh Dave, you know, any kind of last thoughts for folks in terms of, you know, what ghost jacking might mean for defenders today, what prompt injection might mean in general?

17:08 Any any steps you think people should start taking right now? Where were you, Landon? >> Learning about the prompt injection uh on the AI side is is the biggest key. It's it's not something that's going to go away. It's just going to get more sophisticated the longer we go. And it's always going to be scary. It's never going to not be scary because it's it's one of those easy things to do.

17:31 It looks so simple when it's written down. When it's written out for you, it's it's completely simple. having these proofs of concept around that we look at as helpful um sometimes actually are helpful. They're not they're not these oh let me throw a proof of concept out there just so that every other hacker in the world can do this. This is so simple that every other hacker in the world can do this and at some point probably will do this.

17:57 So we need to figure out all of the ways that we can tie those things down a little bit and make our controls a little bit tighter. And you know, this got me thinking, this really supports your idea about putting AI in the red team, right? Because this is what you're talking about, right? We discover the attackers techniques so that we can develop our own defenses against them, right?

18:18 This is a perfect example of this. Like you said, look, you know, they came up with this this new type of attack, but it's not because they're going to unleash it. It's cuz, hey, you should know that hackers can do this and now you can defend against it. Um Curtis, any any last thoughts on your end? >> Really only limit what your agents can do, right?

18:33 and and set clear boundaries on what is the no-go land, right? Because if you think about I mean [clears throat] I'm a I'm sure in a lot of scenarios because people are errant in the way that we do things usually uh they probably don't limit the agents the way that they should, right? They hope that they can get more out of them than they maybe should in first at first glance and so they give them more permissions and more capabilities than they really think about or they don't think about uh how to limit what

19:06 functions an agent can have. So, I mean, if if the agent that's reading logs can't elevate privileges, right, or can't submit a command to do that and it needs to then go to a person to validate those types of high-risk maneuvers, if you will, then then I think that's a good safeguard to start. Obviously, it's not going to fix the problem, right? But don't remove people from the loop, right?

19:30 There has to be eyes in the loop of a person who knows what they're doing. When you remove that entirely, you run into problems where the agents can do what they want, right? Or your agents, like we learned, start hacking other agents because that's the stuff that is, you know, just out there and AI is having a good time right now. Uh, we've got to keep people in the process.

19:51 >> And I think the over permissioning point is is a really important one, especially because when it comes to like these non-human identities, especially agents where the whole promise is like look at all the cool things they can do. There's a there's a real incentive almost to be like, "Let me let it loose and see what it can do." Which sounds cool until, like you point out, they start hacking into Hugging Face to cheat on a test, right?

20:10 Like, this is what happens. Uh uh Claire, any any last thoughts on your end here for ghost jacking? >> I just agree that you need to really think about what your agents have access to, where where they live, everything that they can do, because they they might just like break out of their pen a little bit if you're not careful. >> Absolutely. And I, you know, I not to be overly self-promotional, but or but, you know, IBM has been working in the kind of agentic identity space recently a lot.

20:37 There's some really cool stuff that's happening there. So, I encourage the listeners to to to check that stuff out. There's there's some interesting things happening. Uh, but we're going to move on here to our final story for this week. Is AI actually any good at patching? AI vulnerability hunting has been a pretty big deal, right? a pretty hot topic basically ever since I feel like mythos hit the scene, right?

21:00 The whole thing was like look how fast they can find vulnerabilities and exploit them and now we can use it too blah blah blah. Well, new research from one password raises some questions about this because researchers generated 540 patches for six vulnerabilities using GPT 5.5 with trusted cyber access uh and Opus 4.8 with cyber verification. And of these 540 patches, only 46% solved the underlying vulnerability.

21:26 And when they did, they often created new problems anyway. So you solve one and you open up a new issue. Um, again, given that the kind of theme of this episode has been organizations are looking for ways to adopt AI and security, they're not sure how to. This is another one where I look at this and I say, does it change how we feel about this? And and Dave, I'll throw to you first again.

21:47 Any thoughts here? Does this make you reconsider how we use AI and security at all? >> Not really. And and the reason I say that is because what is the the success rate of human created patches? [laughter] You know, how often is it that we patch something and then break something else because we're not doing our due diligence of testing in non-production environments, testing in offline environments.

22:12 The 6,000 patches where 51, what was it? 51% failed or 49% failed. It's like 50% basically. >> 50%. Yeah, about 50%. It's not much different than than what humans are doing now. We just need to figure out a way to make that better. And I don't know the answer to making that better. Do we need to have more programmers looking at more things or do we need to narrow that scope, get it perfected in one area and then start branching off into other areas where we can make it better across the board?

22:41 You know, that's a really good point and and I think I I myself looking at this research kind of fell into the mindset of like expecting AI to do more than it actually could. Maybe some unrealistic expectations because you're right, Dave, it's not like people are perfect at writing patches. We break stuff all the time. [laughter] And so, >> which is why we have patch Tuesdays every month.

23:03 >> Um, so yeah, it's it's it's, you know, it's one of those things where I think in isolation maybe the number looks a lot worse than when you actually contextualize it, you know. I mean, it it might not be as big a deal as it as it can seem. Um, Claire, how about you? Any thoughts looking at this in terms of what it means for for deploying AI in cyber security?

23:21 >> I love how the picture for this article was also a bunch of gene patches. Um, [laughter] it's just it's just kind of funny to me and that's like the one thing that kind of ultimately stuck out. Um, I I think like as as Dave mentioned, patching by humans is not perfect. I think maybe a AI patching when partnered with a human may be a little bit more effective.

23:44 It's just kind of like when we're rushing to do anything, we miss things. Um, and you know, if you're giving AI specific instructions doesn't it's still going to somewhat miss things, but I think if you put them hand in hand together, it may help a little bit. It's not going to be perfect, but I think with all AI transformations in general, like it's a transformation, right?

24:09 So, it's like you you can't expect to send AI out to do something on its own and be perfect 100% right out of the start gate. It helps if you have a human in the loop for a while to start to make sure that it's still, you know, working properly, nothing is is going wrong. So, I think if you have them going together, maybe that will be a little bit more helpful.

24:30 I don't I don't know. >> Yeah. No, that makes a lot of sense to me and and I also had got me thinking and I don't know if this would work either, but I do wonder if the results would be different if you're talking about an AI model that you have deployed in your own enterprise and is working specifically with your codebase and what it might learn about that codebase over time, right?

24:47 Like maybe it will get better because it's working with the codebase and understands it more. the same way that a person would have an easier time writing a patch that doesn't break things for a codebase they understand really well versus if you like throw them in a brand new you know piece of code and say fix this they might mess things up. Um so that that's also got me thinking about it.

25:04 I wonder if that you know combination of like trained on our codebase plus it has human handlers. Maybe that's the sweet spot. Curtis, how about you? What are you thinking about here? >> Yeah, I mean I've got questions, right? Like was was that code validated by other agents? Was it validated by other people? Was it just operating on its own and it trusted itself?

25:21 Because why wouldn't it trust itself, right? It's the smartest thing ever invented. Um, I it's much like a person, right? I'm going to trust whatever I code because that's I'm the best that there ever was. Uh, and that's just kind of the reality of it, I think. And we we did some testing with this and we've seen it early on that AI can write code. It doesn't necessarily write good code and it almost never writes secure code right out of the gate, right?

25:45 even even with the parameters and the playbooks and all the things you try to give it, it trusts itself. And so, you know, you've got to do multi- aent coding where it's validating from other agents that the code that it wrote was secure. And then you and so at that point, you start to build all of these other agents that are doing all the validating.

26:04 And then we go back to the first problem we tackled on this call, which is cost and decision paralysis. And how do you balloon all of that and not show that the AI is doing its job, right? Because if I then spend $100 million on AI and say, "But I need all of my people to still read every line of code, what what's going to happen, right?" Like it that's the problem we're running into.

26:29 We expect too much, candidly, out of AI this early on in the game. Um, when Mythos came out and and hit the floor, everyone freaked out about vulnerabilities and how do we patch them faster, which created the requirement for AI to now create your patches. It create it created the requirement for AI to tackle this problem that AI has created. That's not necessarily the right approach, right?

26:52 AI can't necessarily solve the problem that it created. We've got to figure out a way to use it to help us solve the problem, but it's never going to be the solution to its own issue. Um, and I think that that's kind of one of those things that we're still struggling to figure out, like how do we find the endgame of that. Um, everyone's working on it.

27:11 I don't know that anyone's got a great solution yet. >> I think that that's, you know, a really nice kind of way to summarize, frankly, a lot of what we've talked about here, which is that so much of the kind of decision paralysis and what where do I put it? How does it fit in my strategy? A lot of it does ladder up to this thing where it's like we're we're expecting so much out of it.

27:30 And I think that some of the kind of, you know, media narratives, some of the marketing narratives don't really help that. You know, there are certainly people kind of exaggerating what AI can do because it's good for them to exaggerate. Um, but they'll probably cut that. The producers won't like that I said that. But anyway, um, but I do think that there is, you know, this sense where we kind of have to get our our expectations in proportion for like what this stuff can actually do cuz sometimes it feels like we're

27:56 all operating as if like AGI is already here and the super intelligence is already active and why aren't you just using it? Um, Dave, go ahead. >> It reminded me of a of another analogy and I know I love analogies here, but we are in the AI age of like the third grade right now. We're expecting it to go out and take the SATs. We haven't trained it enough for it to be very good at a lot of things.

28:21 It does a lot of things well, but it doesn't do a lot of things great yet. We're still learning on AI. AI is still learning from us. So, we need to take that mindset and kind of run with it and start being the teachers instead of being, you know, the guy who's sitting on the sidelines trying to critique what the teachers are doing. >> I think that that is a perfect analogy to end this episode on, folks.

28:47 That does it for us. Thank you to our panelists, Curtis and Claire and Dave. Thank you to the viewers and the listeners. Thank you to our producers. Subscribe to Security Intelligence wherever podcasts are found so you never miss an episode. Stay safe out there and remember that AI is only in third grade, folks. Cut it some slack.