Can you social engineer an AI? Plus: AI worms and the nonhuman identity problem Transcript, AI Summary & Key Points

00:00 Folks. Who do you think is more likely to fall for a social engineering scheme, a human being, or an AI agent? I feel like an AI. Right now, the answer is AI. AI is going to fall for it. I do think we're going to have to make AI a lot smarter, not just in the book sense, but in the street sense as well. Hello and welcome to Security Intelligence, folks, IBM's weekly cybersecurity podcast, where our expert panelists turn the biggest industry news stories into practical takeaways you can use.

00:26 I'm your host, Matt Kozinski, broadcasting live from some kind of dungeon. And joining me this week, we've got a real classic Security Intelligence lineup. It's Claire Nuñez, creative director, IBM X-Force Cyber Range, Jeff Crume, distinguished engineer, master inventor, data and AI security. And Nick Bradley, manager, X-Force Threat Intelligence. And on the docket today, we've got an AI-powered worm designed by University of Toronto researchers and the Sophos State of Identity Security 2026 report.

00:57 But first, we're going to keep talking about AI falling for social engineering schemes, because some hackers tricked customer support agents into handing over Instagram account passwords. Now, 404 Media broke this story last week. It's about a spate of attacks where hackers just posed as the legitimate Instagram account owners, messaged Meta's AI customer support agent and asked it to update their accounts with new emails, emails they controlled.

01:26 And the agent just did it. It just handed the accounts right over to these people, put their emails in there, and they were able to use that to recover the passwords and take over the accounts. Now, the vulnerability has since been resolved, but I think there's something to learn here. And, Claire, I want to start with you, and I want to ask, are you surprised by just how easy it seems like it was to pull off this attack?

01:44 Does it give you any concerns about AI in customer-facing roles? How are you feeling? I feel like it's surprising on one aspect because like the whole premise of Instagram is like your account and you being able to access your account. So you would think Meta would have that, you know, under lock. But it's not surprising at the same time that something so seemingly simple has been overlooked.

02:11 I mean, as Jeff mentioned, agents don't understand nuance, and they are very naive. So, if you they're not going to ask why are you changing this email? So it's not that surprising to me, but you would think that a company that kind of like, revolves around your account and your ability to access that account for, like, everything, would have looked at that a little more closely.

02:35 Absolutely. And, you know, I know I think it reminds me of, like, it's very easy to trick a person. Right. Maybe I shouldn't say very easy, but it can be quite easy to trick a person. And one of the promises of AI is supposed to be that it's it's, you know, it's got certain contexts that we don't have that maybe it won't fall for some of these things, but we're seeing the opposite happen here.

02:54 And, Jeff, I wanted to ask you, you know, similar question to Claire. Like looking at an AI agent fall for something like this. Does it give you concerns about putting these things in customer-facing roles right now? How do you feel? AI is like this, this person that's got a thousand PhDs but has never spent one day above ground, you know, out of their parents' basement.

03:14 So a lot of intelligence in one level, but not a lot of common sense, not a lot of the kind of lived experience that allows us to realize when somebody is trying to con us or not, when somebody is trying to ask for something that's unreasonable or not. I mean, if you put in the system prompt, you're a helpful, customer support assistant. Well, then it's going to try to be helpful, and it's going to be very eager and work as hard as it can to satisfy the request.

03:41 And just like a person, if you haven't taught them: Yeah, but every person that asks for your wallet, you don't hand it to them. You know, you want to be polite, but you know, there's certain limitations to that kind of politeness and what people ask you to do and reasonableness checks. This one was way too easy though. I mean, the the attackers just basically put their VPN so that it looked like they were in the area of where the the person would have been and then said, you know, hey, send me a reset code for my

04:10 account to this new email address. And we know you shouldn't be able to do that without some other form of authentication, but unless you explicitly say, no, don't do this, it's going to probably try to do it. And that's that's the thing we all had to learn as people. You know, we were growing up and we got fooled a lot of times. Well, AI is really young, too.

04:34 It might be really smart, but that doesn't mean it's not very young and naive. And this is a classic example of that. Got to be a lot, lot more specific in terms of what we tell them, what they can do. I think about my grandson, he's four years old, and we're having to constantly tell him the kinds of things, you know, you don't play in the street. And here's why.

04:56 Okay. That sort of stuff that we take for granted. But everybody had to be taught that. And these AIs have to be taught this as well. That's a very good point. And I like this comparison you draw actually with age. You know, young folks, I think my son. Right. He is just about three years old. And it's a similar thing. I got to tell him, no, you can't hang out with the cars in the streets.

05:17 It's actually not. I know you like the cars, but you can't go out there. It's not safe. And it goes back to this idea that you you brought up, which is that, like, we have to be very specific with directions here. And so in ways that I was like slightly surprised by this attack, I think maybe I shouldn't have been surprised because like you said, Jeff, when you tell an agent you're here to help customers access, do things they need to do unless you specifically tell them not to do that.

05:39 They want to help customers. They they they think they're doing the right thing and they're just going to try to do it. So I think that's a very good point. Nick, I wanted to ask you, do you think there's any kind of meaningful difference between tricking a person with a social engineering attack, or tricking an AI agent with a social engineering attack?

05:55 Like this is basically the same kind of thing? Are they slightly different? How are you feeling about it? So there is a key word that I was looking for that I was hoping one of the three of you were going to drop and you didn't. So I'm kind of glad because then I wouldn't have had anything to say. And the word I'm, the word I'm looking for is is wisdom, right?

06:14 Because AI is AI. It's not AW. It's artificial intelligence, not artificial wisdom. And that's the difference between those of us sitting on this call and an agent. Right? We all have intelligence, at least allegedly, and we all have wisdom. Some of us show it more than others and the color of our hair at this point. But that's the point, is, it's the wisdom that develops over time and learning behavior on do I trust someone or do I not trust someone?

06:45 It's that gut feeling or just that experience that comes with time that AI just doesn't have that? I mean, every one of us has probably been in a situation where we went, this seems really sus, right? AI is not going to do that. And it's not because of a failing of AI. It's just because it's not in its wheelhouse. It doesn't have that capability. And so this is a case of of, it's kind of a case of us being our own worst enemy because we're going to shove AI into everything, because AI will solve world hunger, world

07:14 peace, and everything else it possibly can save. We're just not asking ourselves how it's going to come about, how it's going to do it, how is it going to come about solving those problems, because it's going to do things that you would never do as a human being, with the wisdom to know better. And so the question for me then becomes, you know, we're circling around this issue of guardrails and needing to explain to AI what it does but how do you teach something wisdom?

07:36 Like, can you? I don't know. I mean, Claire, I'm going to ask you first any thoughts on like, what we do to maybe help AI get a little bit better against social engineering attacks? And any thoughts there? Well, I think the first thing you have to do is recognize that AI doesn't have, like, a gut instinct. It doesn't have a gut to follow. So you need to remember that AI doesn't.

07:59 Doesn't have context like you and I do of looking and feeling. It just has the context that we give it. So you have to remember that. You have to. You have to tell AI things that you don't necessarily think about. So even if it you kind of have to think a little bit more about everything. Like you wouldn't think as an adult, like to tell another adult to go, not go in the street when the cross sign is is off or on.

08:23 But you need to tell that to a child, right? Like, you need to think about almost as if you're explaining it to somebody that is like completely new in the industry or completely new on Earth. Like, you have to just kind of think about giving it those contexts, which is a little more work. And I guess, like you could in theory say, yeah, I'm going to ask an AI to do this for me.

08:48 But I feel like you also need to like reason as a human of what would I be doing and provide those guardrails. Yeah, I think that's really good advice for AI use in general, right. No matter what you're using it for. Like, sure, you can give it the one sentence prompt, but if you really want it to do something like write good code for you, or maybe not give away account passwords, you do need to think about that extra context and give that to them.

09:14 So I think you can apply that in general. Jeff, anything to add there to Claire's thoughts on on kind of strengthening our AI against this? Sure, sure. If you think about what AI is, the simplest definition I can think of is it's basically trying to match or exceed human intelligence in a computer. Okay, so we're using ourselves as the model, and we're saying we want this system to be as smart or smarter than we are.

09:33 And so we train it in the ways that we think. Well, look, your initial question about will humans or AI be more likely to fall for this? It has implied in it that people, in fact, still fall for these things. And we've had thousands and thousands of years of of developing common sense and developing a tradition where we teach each other, you know, don't fall for this, don't fall for that.

10:00 And yet still, phishing attacks work. So it's not like this is a solved problem for people. So it's definitely not a solved problem for AI. I do think we're going to need to spend more time than we have on trying to teach AI what's right and what's wrong, because right now we've focused, I think, most of our efforts on telling it, having it understand what's true and what isn't true.

10:24 You know, basic information and facts and things like that. Like you know, Nick mentioned wisdom and you can think of this as kind of a pyramid. You know, you've got data, you've got information, you've got knowledge. Wisdom is on top of all of that pyramid where we process each one of those things to a greater degree. I didn't mention wisdom before because it's not a word I'm familiar with.

10:50 I don't I don't deal in this in this area. As for those who know, but but the. I do think we're going to have to make AI a lot smarter, not just in the book sense, but in the street sense as well. And we haven't spent a whole lot of time on that, because honestly, even though we call it common sense, it's actually pretty hard to define. And, and what's common for one person might not be common for the next, but this one clearly was a common sense case.

11:15 And there should have been, and I'm sure there are at these companies rules, procedures that say, if you're going to reset somebody's account, here are the things that must be met, the conditions that must be met. And you have to make sure the AI follows those conditions just to the letter, you know, just like anyone else would. It can be courteous, but it can say courteously, no, we're not going to, you know, reset your account to this random email address.

11:43 Yeah. And I like that you point out and I think it's important to keep in mind, you know, we haven't solved this problem for people either. Right. So like the takeaway here isn't like, oh, the AI is not good. Like, you know, it's not like a gotcha kind of thing, right? It's like we are also not great at this. And so like as we're learning and trying to learn, we also need to be helping the AI try to learn.

12:01 And like you said, Jeff, develop some of those street smarts we haven't focused on yet. Nick, to close this out, I want to ask you a slightly different question, which is that so much of the conversation recently around AI in cybersecurity has revolved around these big, powerful models. You know, your Mythoses, your GPT-5.5s. And here's a story of a really, really simple, basic attack working.

12:19 Is there a lesson here for us? Yeah, we sometimes just miss the forest for the trees, right? We're trying to solve the big problems when some of the small problems are still lingering about. Right. And then the question that I have in general is, can this be learned? And I don't have an answer. Can can wisdom or at least the the facsimile of wisdom can that be learned, right?

12:44 Can it use certain clues and logic and whatnot to determine if, okay, this person is asking to do said thing, but here are the things that should be the red flags on why I shouldn't do it right. So we have what we're calling that that gut feeling. Right. But AI is not going to be able to do that. It's going to have to use logic. And is it possible? I don't know.

13:07 Somebody smarter than me is going to have to figure that out. And we can look at this particular case and say, this was really stupid, you know, why did this happen? And therefore AI and we jumped to the conclusion, AI is not ready for prime time. This stuff is junk, blah blah blah. Look, if if a single failure case was was enough to disqualify a technology, then humans would have been disqualified a long time ago.

13:36 I guarantee you there are helpdesk agents who have made similar errors, and yet we don't summarily dismiss all of humankind because of it. So. And there are and I do have to move this along, but I have to mention it now that you said it, Jeff, is that I've had many conversations with folks in X-Force who do like, you know, social engineering testing.

13:52 And they've always told me that, like, the most successful attack we do is we just call the help desk, pretend we're the person, and need to reset a password. And it works almost every time. And so you're literally right. It can work against people too. But I have to move this along to the next story. Before I do. I'm going to open it up. You know, viewers on YouTube, if you have thoughts about whether we can teach an AI wisdom, let us know.

14:16 Maybe you have the answer, because I don't think any of us here do. But to move on. Our second story for the week, University of Toronto researchers design a new AI worm. Using an open source LLM, the researchers created what they call a self-replicating agent. It can spread from device to device, like any worm, using device resources to run a local model that can supposedly reason its way through attacks, choosing different vulnerabilities and exploits for each device it encounters.

14:47 Now, I've been following the kind of AI malware story for a few years now, and virtually every time I talk to people about it, they say it's not really a thing. Like, yeah, attackers use AI to generate malware code, but it looks a lot like regular malware code. And I'm wondering, have we finally reached a point now where AI malware is actually here?

15:02 Like, is this something different or more of the same? And Jeff, I'm going to ask you first, what do you think? Is this a genuine advancement or just more of the same? Well, my first reaction was, what took you so long? I actually, I actually figured this was coming. To me, it was obvious. In fact, I've covered this in some of the YouTube videos I've done on the IBM channel in the past as predictions as to where malware would go.

15:27 Malware. I mean, I remember when we first started seeing the first samples of polymorphic viruses, these viruses that would change themselves as they, you know, propagated. And everyone thought that was going to be the end of the world. Well, you know, the world is is still spinning around on its axis and we're all still here. And so people that look at this probably will jump to the same conclusion.

15:49 And it's definitely a bigger risk. It's not surprising to me that this happened. And as large language models become small language models, become even smaller language models, these things will be more containable and more portable and more able to be sent around. So I fully expect to see more of the same, which means the good guys just have to use a similar but better technology to do the detection and repelling and prevention and all that kind of stuff.

16:19 But it's this is the arms race. And like I said, I'm just surprised that it took so long for this to happen. I think that's a really fair point. And I didn't even think about the bigger trend. Like you said, the models are getting smaller and the smaller they get, the more it becomes possible to do an attack like this that literally spreads a model from computer to computer.

16:38 Claire, any thoughts on your end? Looking at the story of the AI worm, what's it got you thinking about? Anything come to mind in terms of risks? Things you're concerned about? My first thought was, I'm sure this isn't the first AI malware that we've seen out there that's been partially developed. I also don't think a hacker group is going to be like, I used AI to develop this malware that I'm sending to all these people.

17:04 It's kind of like you're not going to give away your, like, secret operating sauce. It's kind of like quantum too, where it's like hackers are not going to, you know, most likely come out and say, I've used quantum computing like to decrypt all this data. I think it's probably similar where you know, that they're they're using AI, but, you know, we'll start to see it more.

17:19 I mean, at the end of the day, I've said this a million times on this podcast, but these are cybercrime businesses. They are trying to just make their products and services a little more profitable for themselves. So they're just trying to improve ROI. And if AI-related malware does that, then they're going to keep using it. It depends how successful it is, too, in terms of, you know, how we see it propagate.

17:51 Yeah, I mean, it ties back into what Jeff said about this being an arms race, and it's an arms race in a pretty literal sense that, like, we're trying to develop these technologies, they're trying to develop these technologies. We're subject to almost the same kind of market pressures. And so I like that you brought it back to that, Claire. Reminded us that, like, these are businesses.

18:07 They're not just doing this for fun. They're trying to get some kind of return on it. Nick, how about you looking at this? Do you feel like this is a genuine leap forward? More of the same where you land in here? I think it is a logical expectation in the evolution of what we've been watching. Between the smaller models, you know, more compact, less of a footprint on top of the of the things we're seeing with the frontier models.

18:28 This was to, to use the same word again. This was inevitable. It was going to happen. And I agree with Jeff on why did it take so long. But I'm going to I'm going to add on to that is I don't think it did take so long. I think it's been there. And this is just the first time we've got good guys that have done it and just said, here, look what we've done, and then you've got the bad guys going, "Rats — shut up!"

18:56 Yep. Yep. Kicking them under the table. Right. My my my thought on this though is it's going to happen. So take advantage of it. We could use the same the same methodology to let this run and find those same vulnerabilities that it's going to reason its way through, but instead of taking advantage of them, give me a report, tell me how to fix it. Tell me what I need to go do to tighten up my environment so the same tool could be used in for for good instead of evil.

19:25 I'm glad you brought up that that take on it, Nick, because I saw people saying something similar, right. That like, hey, if we can use it to spread an AI to like look for vulnerabilities that it then exploits, why not use it to spread an AI that looks for vulnerabilities, that it then patches and like that seems like a really. A lot of times when we talk about, you know, AI attacks, I end the segment feeling kind of down.

19:42 I feel pretty okay about this one, you know, because this is one where the development maybe can be readily used for, for good ends. But but before we move on, I do want to talk a little bit more about, you know, the so what, the how do we protect ourselves and and Jeff, I want to I want to ask you this question because you were on the episode we did a little while ago about protecting open source AI infrastructure.

20:01 And one of the interesting things about this worm is that they used an open source model. They didn't say which one, but they said they use an open source model. And part of the reason they did that was because that means there's no, you know, OpenAI or Anthropic watching what they do with the model and being able to kick them off the platform. Do you think this is a problem we're going to have to contend with, people using open source AI models for for bad purposes?

20:23 And what do we do about it? Any thoughts there, Jeff? Oh, no doubt it's an issue, but this is one where you know, the there's, all the animals have escaped. There's no point trying to lock the barn door at this point. By that I mean, we've got Hugging Face, which is an, an open AI model repository. Think of it as GitHub but for AI models. There's more than 2 million AI models on there already.

20:55 Okay, so there's no way you're going to make that stop. There's no way to unring that bell. Or toothpaste back in the tube, genie back in the bottle. If I can think of any more analogies, I'll, I'll run them all down into the ground. But. So this is this is our new reality, and we have to accept it. I think I think there's a larger trend going on with regards to AI, because the change has occurred so fast that we've seen these kinds of of new attack types, or it seems new to people.

21:25 It's really variations on a theme done at greater velocity and greater volume. So therefore it feels new. But that, that we, we've got there's no point in saying, stop everybody, just stop it, you know, don't do this anymore. And people are saying that when it comes to using AI in the classroom, they're saying, don't use it with music, don't use it, you know, in this area, that area.

21:50 All these other different areas. Look, it's already out there. So now we have to accept the reality that it's out there, or use an AI model that's too dangerous for everyone to have because you know it, it uncovers vulnerabilities. Well, actually, that could be a good thing if it's in the hands of the right people. So all of these things have the potential equal potential to do harm as they do to do, you know, good things.

22:13 And it all depends on whose hands it's in and what their motivations are. I was just going to say I loved all the analogies, but the one I want to stick with, well, it's not even an analogy, but it was something you said on the don't do that. You know what I want to do the second you tell me not to do something, I want to do it. Right. And that's that's human nature.

22:36 Don't push that button. Don't open that door, don't touch that. I'm going to do all the things, right? And and that's just. The best way to to get me to do something is tell me don't do it. 100%. And so we need to understand that Pandora's box is open. Everything's out. Now we just have to deal with it. Nick, that's what I wanted to stress, too, that part of the of Jeff's response, because I think you're really right to point out.

23:02 Look, people can sit there and say, I'm not going to use AI for XYZ reason. Okay, fine. The attackers are going to use it, though, like they're going to stop using it. So sure, you can willingly take yourself out of that race, but it just means you're going to get left behind. You know what I mean? At a certain point, you kind of got to get in there.

23:15 Claire, to round out this segment, though, for us, any last thoughts on, you know, what this might mean for security? Any advice you would give organizations? What's your take here? I think a lot of organizations know by now that security is moving so much faster and evolving every day. I think something that you said, Matt, that or maybe Nick, you said it, that if you're not, like, in the race with it, you're going to get left behind.

23:43 So, I mean, if you're not looking at AI security solutions, if you're not following AI and security at all. You've kind of been left behind already. You can catch up and you should catch up. You can stick your head in the sand if you're afraid of the rain, but all you're going to do is drown. Yep. On that note, folks, I'm going to move this along to our final story for the week.

24:05 This is the State of Identity Security 2026 report. This is Sophos' survey of 5000 IT and cybersecurity leaders. And it found that 71% of them suffered at least one identity-related breach in the last year. I don't think this comes as a surprise to any of us. You know, IBM, we do the X-Force Threat Intelligence Index every year, and identity-based attacks are a number one attack vector.

24:29 They were number two this year, but still like 32% of attacks involved them. So like we know it's a big thing, right? What was especially interesting to me about this report though, and the reason I included it here, was some of the stats around non-human identity specifically. Sophos kind of broke out which attacks involved people stealing user identities and which involved them stealing non-human identities.

24:48 You know, AI agents. You know, APIs, service accounts, those kinds of things. And they found that non-human identities were involved in 41% of successful identity breaches, and only about a third of organizations regularly audit or rotate their NHIs or their credentials. Nick, I want to ask you, because this is something that's come up over and over again on the show.

25:09 What is it about non-human identities, non-human credentials that makes them such a weak spot? Why are we overlooking them still? Any thoughts there? Because no one's watching. If I steal your password and log into your account, your machine, or whatever it is you do. It's only a matter of time before you figure out that somebody has gotten ahold of your credentials.

25:30 But if I get a hold of a service account, are you ever going to find out if you're not watching it or auditing the activities of that account? So it's it's brilliant, to be honest, because I'm stealing something that no one was even watching in the first place. So it's going to continue to happen unless we put better mechanisms in place to monitor these things.

25:53 Because as it was said in the article, some of these, these service accounts come into play and they're used forever. They're not rotated, they're not monitored. They're just they're there until they fail. And then someone tries to figure out, wait, what's broken? Who set up this account? The person that set up the account is not even here anymore. So I think that's an extremely good point.

26:11 And it's a very simple one. And it's one that again, it didn't cross my mind. But but you're right. Like somebody is looking at a user account every day. Someone goes in there. Most of them are not looking at service accounts every day. Nobody's like you said, nobody looks until they fail. Jeff, any thoughts on how we start to maybe get a little more visibility into this kind of thing, or start protecting these accounts better?

26:32 What's your take there? Sure. Well, first of all, I'll make the the master of the obvious statement. Identity is hard. It always has been. It's seemingly always going to be. I mean, I've been working in the identity and access management space for more than a quarter of a century, and we still haven't solved all the problems. We're still solving the same problems that we've had for, you know, over and over and over again.

26:54 And, and I look at it this way. When we first started off, you know, if I wanted to give you an account, well, you filled out a paper form and that got bucked around and somebody signed it, and then an administrator went and created your account. And then we put that in a file cabinet. I mean, it was all a very manual process, very error prone, very inconsistent.

27:14 And then we started moving. And yet still some organizations have not fully done this, moving to more automated identity management systems, where we can do provisioning and deprovisioning automatically based upon your job role and things like that. So that sped things up and give us more insight and accountability into the systems. There's a new evolution that has to occur as well, that even this report I don't think has anticipated, and it's related to non-human identities.

27:43 But the fact that we're going to need way more than we think we do. And creating some of these non-human identities. I mean, they're not going to go fill out a form. They're not going to go get hired by HR and then have a job role that we can map them to and so forth. They're going to need privileges for maybe a few seconds and then that's it. They pop up.

27:59 They go away. When we're talking about agents. A lot of these kinds of capabilities, these are ephemeral IDs. These are things I need to go do this right now. Okay. We'll provision you to do that. But principle of least privilege says we're not going to let you do anything more than just what is absolutely necessary. Even though you're an agent, I don't trust you any further than I can throw you.

28:26 And so I'm only going to I'm going to bound you to this and only for this period of time. And then your ID goes away. This is the kind of systems that we're needing to build now. And for the most part, people don't have those. And things like OpenClaw that allow anybody to run an agentic framework on their laptop, I guarantee you they haven't thought about all of this.

28:43 So they're running these things under the main user account in many cases. You know, the essentially the root user on the system, you know, the sysadmin, the the the, the superuser account on that, you know, Windows or Mac system. And therefore if the agent makes a mistake, well, it's got the full privileges of that user and it can make a real big mess in a hurry.

29:07 A friend of mine on, on LinkedIn just recently sent me an article where it referred to, what was it? It, it basically, AI is a fast fool, and that's what it is. It can do these things really fast and miss a point that we would have been able to have, have gated and, and paced more if a human was doing it. So that's another aspect of identity management that is only going to get more complicated as we move forward.

29:40 Absolutely. And I'm glad you bring up this idea of like, especially spinning up ephemeral IDs and ephemeral permissions and how we manage that. And it reminds me a lot of this idea I see coming out of the kind of HashiCorp wing of IBM right now around security lifecycle management and how you automate things like provisioning these accounts, making sure they have just the right privileges for just the right time, just in time access.

30:02 A lot of people are starting to think about this thing. But like you said, Jeff, you know, identity is hard. And so we're working our way through it. Claire, I want to bring you in here. Looking at the report or things we've talked about so far, what's what's coming up for you? What's sticking out? What are you thinking about? I think a common misconception with non-human identities is that there's, like, no human attached, like at any point.

30:19 But at the end of the day, a human did provision this at some point or it provisioned it at some point in the chain. So like whatever mistakes the the person made at some point are then moving around and going down the chain, it's kind of like if you think about when a fish eats plastic and then a bigger fish eats that, and like the plastic just builds, it's like something it's like the mistakes just kind of keep rolling.

30:48 And we're seeing a lot of clients, like, consider, non-human identities and their experiences, and sometimes their executive teams are a little bit like, oh, wait, what does this even mean? So it is something too, that's like a little bit meta where it's like, what do you mean? We have people and we don't have people at the same time. Like we just have all these identities.

31:08 But it's interesting because at the end of the day, they do always tie back to a human somewhere far back in the line. Yeah, that makes a lot of sense to me. You know, and it is funny to point out, like non-human identities, this thing we talk about a lot now, but like, it feels kind of far out, you know what I mean? Like, am I talking about aliens?

31:26 Like, what do you mean by a non-human identity? And it's no, these, these machines that are active. And I think focusing on that person who's there at some point, like I think that's a that's a good place to start looking, right. Like what are they giving. How are they setting these things up? It's like Jeff said before, people setting up OpenClaw without knowing what they're getting into.

31:43 If they know what they're getting into, maybe we can head off more things at the pass, lest that plastic gets through. Like you said. Claire. Nick, I have to close this out. But before I do, last thoughts on this issue of non-human identity security? What we do to better going forward? Any last words for us? I think we're going to have to figure out some type of dynamic behavior analysis that we haven't conceived of yet.

32:07 And sadly, it's probably going to take AI to solve it. So we've just come full circle. The AI solves the problems that the AI makes for us folks. That is the story of security today in many ways, but that does it for this episode. I want to thank our panelists, Claire and Nick. And Jeff. Thank you to the viewers and the listeners and our producers. Subscribe to Security Intelligence wherever podcasts are found, so that you never miss an episode.

32:28 Stay safe out there and remember, when it comes to their susceptibility to social engineering and credential theft, non-humans are basically humans, too. Folks. Who do you think is more likely to fall for a social engineering scheme, a human being or an AI agent? Feel like an. AI right now? The answer is AI is going to fall for it. I do think we're going to have to make AI a lot smarter, not just in the book sense, but in the street sense as well.

00:22 Hello, and welcome to Security Intelligence, Folks, IBM's weekly cybersecurity podcast, where our expert panelists turn the biggest industry news stories into practical takeaways you can use. I'm your host, Matt Kozinski, broadcasting live from some kind of dungeon. And joining me this week. We've got a real classic security intelligence line up. It's Claire Nuñez, creative director, IBM X-Force Cyber Range, Jeff Crume, distinguished engineer, master inventor, data and AI security.

00:47 And Nick Bradley, manager, X-Force Threat Intelligence. And on the docket today, we've got an AI powered worm designed by University of Toronto researchers and the Sophos State of Identity Security 2026 report. But first, we're going to keep talking about AI falling for social engineering schemes, because some hackers tricked customer support agents into handing over Instagram account passwords.

01:14 Now, 404 Media broke this story last week. It's about a spate of attacks. Were hackers just posed as the legitimate Instagram account owners messaged Meta's AI customer support agent and asked it to update their accounts with new emails, emails they controlled and the agent just did it. It just handed the accounts right over to these people, put their emails in there, and they were able to use that to recover the passwords and take over the accounts.

01:38 Now, the vulnerability has since been resolved, but I think there's something to learn here. And, Claire, I want to start with you, and I want to ask, are you surprised by just how easy it seems like it was to pull off this attack? Does it give you any concerns about AI in customer facing roles? How are you feeling? I feel like it's surprising on one aspect because like the whole premise of Instagram is like your account and you being able to access your account.

02:03 So you would think Meta would have that, you know, under lock. But it's not surprising at the same time that something so seemingly simple has been overlooked. I mean, as Jeff mentioned, agents don't understand nuance, and they are very naive. So, if you they're not going to ask why are you changing this email? So it's not that surprising to me, but you would think that a company that kind of like, revolves around your account and your ability to access that account for, like, everything, would have looked at that a

02:35 little more closely. Absolutely. And, you know, I know I think it reminds me of, like, it's very easy to trick a person, right? Maybe I shouldn't say very easy, but it can be quite easy to trick a person. And one of the promises of AI is supposed to be that it's it's, you know, it's got certain context that we don't have that maybe won't fall for some of these things, but we're seeing the opposite happen here.

02:56 And, Jeff, I wanted to ask you, you know, similar question to Claire, like, looking at an AI agent fall for something like this. Does it give you concerns about putting these things in customer facing roles right now? How do you feel? AI is like this? This person that's got, a thousand PhDs but has never spent one day above ground, you know, out of their parents basement.

03:18 So a lot of intelligence in one level, but not a lot of common sense, not a lot of the kind of lived experience that allows us to realize when somebody is trying to con us or not, when somebody's trying to ask for something that's unreasonable or not. I mean, if you put in the system prompt, you're a helpful, customer support assistant. Well, then it's going to try to be helpful, and it's going to be very eager and work as hard as it can to satisfy the request.

03:44 And just like a person, if you haven't taught them. Yeah, but every person that asks for your wallet, you don't hand it to them. You know, you want to be polite, but you know, there's certain limitations to that kind of politeness. And what people ask you to do. And reasonable is text. This one was way too easy, though. I mean, the attackers just basically put their VPN so that it looked like they were in the area of where the the person would have been and then said, you know, hey, send me a reset code for my

04:13 account to this new email address. And we know you shouldn't be able to do that without some other form of authentication, but unless you explicitly say, no, don't do this, it's going to probably try to do it. And that's that's the thing we all had to learn as people. You know, we were growing up and we got fooled a lot of times. Well, AI is really young, too.

04:38 It might be really smart, but that doesn't mean it's not very young and naive. And this is a classic example of that, got to be a lot, lot more specific in terms of what we tell them, what they can do. I think about my grandson, he's four years old, and we're having to constantly tell him the kinds of things, you know, you don't play in the street. And here's why.

04:58 Okay, that sort of stuff that we take for granted. But everybody had to be taught that. And these AIs have to be taught this as well. That's a very good point. And I like this comparison. You draw actually with, an age, you know, young folks, I think of my son. Right. He is just about three years old. And it's a similar thing. I got to tell him, you know, you can't hang out with the cars in the streets.

05:19 It's actually not. I know you like the cars, but you can't go out there. It's not safe. And it goes back to this idea that you you brought up, which is that like we have to be very specific with directions here. And so in, in ways that I was like slightly surprised by this attack. I think maybe I shouldn't have been surprised because like you said, Jeff, when you tell an agent you're here to help customers, you know, do things they need to do, unless you specifically tell them not to do that.

05:41 They want to help customers. They they think they're doing the right thing and they're just going to try to do it. So I think it's a very good point. Nick, I wanted to ask you, do you think there's any kind of meaningful difference between tricking a person with a social engineering attack, or tricking an AI agent with a social engineering attack like this is basically the same kind of thing.

05:57 Are they slightly different? How are you feeling about it? So there is a key word that I was looking for that I was hoping one of the three of you were going to drop and you didn't. So I'm kind of glad because then I wouldn't have had anything to say. And the word I'm the word I'm looking for is is wisdom, right? Because AI is AI. It's not AW, it's artificial intelligence, not artificial wisdom.

06:21 And that's the difference between those of us sitting on this call and an agent. Right? We all have intelligence, at least allegedly, and we all have wisdom. Some of us show it more than others and the color of our hair at this point. But that's the point, is, it's the wisdom that develops over time and learning behavior on do I trust someone or do I not trust someone?

06:45 It's that gut feeling, or just that experience that comes with time that AI just doesn't have that I mean, every one of us has probably been in a situation where we went, this seems really sus, right? AI is not going to do that. And it's not because of a failing of AI, it's just because it's not in its wheelhouse. It doesn't have that capability. And so this is a case of it's kind of a case of us being our own worst enemy because we're going to shove AI into everything because AI will solve world hunger, world

07:16 peace, and everything else that possibly can save. We're just not asking ourselves how it's going to come about, how it's going to do it, how is it going to come about solving those problems, because it's going to do things that you would never do as a human being with the wisdom to know better. And so the question for me then becomes, you know, we're circling around this issue of guardrails and needing to explain to AI what it does, but how do you teach something wisdom like, can you?

07:41 I don't know, I mean, Claire, I'm going to ask you first any thoughts on like, what we do to maybe help AI get a little bit better against social engineering attacks and any thoughts there? Well, I think the first thing you have to do is recognize that AI doesn't have, like, a gut instinct. It doesn't have a gut to follow. So you need to remember that AI doesn't doesn't have context like you and I do of looking and feeling it.

08:07 It just has the context that we give it. So you have to remember that. You have to you have to tell AI things like, you don't necessarily think about. So even if it you kind of have to think a little bit more about everything. Like you wouldn't think as an adult, like to tell another adult to go, not go in the street when the cross sign is is off or on.

08:30 But you need to tell that to a child, right? Like you need to think about almost as if you're explaining it to somebody that is like completely new in the industry or completely new on earth. Like you have to just kind of think about giving it those contexts, which is a little more work. And I guess, like you could in theory say, yeah, I'm going to ask an AI to do this for me, but I feel like you also need to like reason as a human of what would I be doing and provide those guardrails.

08:59 Yeah, I think that's really good. Advice for AI use in general, right? No matter what you're using it for. Like, sure, you can give it the one sentence prompt, but if you really want it to do something like write good code for you, or maybe not give away account passwords, you do need to think about that extra context and give that to them. So I think you can apply that in general.

09:18 Jeff, anything to add there? To Claire's thoughts on on kind of strengthening our AI against this. Sure, sure. If you think about what AI is, the simplest definition I can think of is it's basically trying to match or exceed human intelligence in a computer. Okay, so we're using ourselves as the model, and we're saying we want this system to be as smart or smarter than we are.

09:39 And so we train it in the ways that we think. Well, look, your initial question about will humans or AI be more likely to fall for this? It has implied in it that people, in fact, still fall for these things. And we've had thousands and thousands of years of of developing common sense and developing a tradition where we teach each other, you know, don't fall for this, don't fall for that.

10:04 And yet still, phishing attacks work. So it's not like this is a solved problem for people. So it's definitely not a solved problem for AI. I do think we're going to need to spend more time than we have on trying to teach AI what's right and what's wrong, because right now we focused, I think most of our efforts on telling it, having it understand what's true and what isn't true.

10:29 You know, basic information and facts and things like that. Like you know, Nick mentioned wisdom and you can think of this kind of a pyramid. You know, you've got data, you've got information, you've got knowledge. Wisdom is on top of all of that pyramid where we process each one of those things to a greater degree. I didn't mention wisdom before because it's not a word I'm familiar with.

10:51 I don't I don't deal in this in this area. As for those who know, but but the I do think we're going to have to make AI a lot smarter, not just in the book sense, but in the street sense as well. And we haven't spent a whole lot of time on that, because honestly, even though we call it common sense, it's actually pretty hard to define. And, and what's common for one person might not be common for the next, but this one clearly was a common sense case.

11:20 And there should have been, and I'm sure there are at these companies rules, procedures that say if you're going to reset somebody's account, here are the things that must be met, the conditions that must be met, and you have to make sure the AI follows those conditions just to the letter, you know, just like anyone else would. It can be courteous, but it can say courteously, no, we're not going to, you know, reset your account to this random email address.

11:46 Yeah. And I like that. You point out and I think it's important to keep in mind, you know, we haven't solved this problem for people either. Right. So like the takeaway here isn't like, oh, the AI is not good. Like, you know, it's not like a gotcha kind of thing. Right? It's like we are also not great at this. And so like as we're learning and trying to learn, we also need to be helping the AI try to learn.

12:04 And like you said, Jeff, developed some of those street smarts we haven't focused on yet. Nick, to close this out, I want to ask you a slightly different question, which is that so much of the conversation recently around AI in cybersecurity has revolved around these big, powerful models. You know, your Mythos, your GPT-5.5s. And here's a story of a really, really simple, basic attack working.

12:24 Is there a lesson in here for us? Yeah. We sometimes just miss the forest for the trees. Right? We were trying to solve the big problems when some of the small problems are still lingering about. Right. And then the question that I have in general is can this be learned? And I don't have an answer. Can can wisdom or at least, the facsimile of wisdom, can that be learned.

12:50 Right. Can it use certain, clues and logic and whatnot to determine if, okay, this person is asking to do said thing, but here are the things that should be the red flags on why I shouldn't do it right. So we have what we're calling that that gut feeling. Right. But AI is not going to be able to do that. It's going to have to use logic and is it possible?

13:11 I don't know, somebody smarter than me is going to have to figure that out. And we can look at this particular case and say, this was really stupid, you know, why did this happen? And therefore AI and we jumped to the conclusion, AI is not ready for prime time. This stuff is junk, blah blah blah. Look, if if a single failure case was was enough to disqualify a technology, then humans would have been disqualified a long time ago.

13:36 I guarantee you there are helpdesk agents who have made similar errors, and yet we don't, summarily dismiss all of humankind, because of it. So. And there are and I do have to move this along, but I have to mention it now that you said it, Jeff, is that I've had many conversations with folks in X-Force who do like, you know, social engineering testing.

13:55 And they've always told me that, like, the most successful attack we do is we just call the helpdesk, pretend we're the person, and need to reset a password. And it works. Almost every time. And so you're literally right. It can work against people, too. But I have to move this along to the next story. Before I do. I'm going to open it up, you know, viewers on YouTube, if you have thoughts about whether we can teach an AI wisdom, let us know.

14:16 Maybe you have the answer, because I don't think any of us here do. But to move on, our second story for the week, University of Toronto researchers design a new AI worm. Using an open source LLM, the researchers created what they call a self-replicating agent. It can spread from device to device. Like any worm, using device resources to run a local model that can supposedly reason its way through attacks, choosing different vulnerabilities and exploits for each device it encounters.

14:49 Now, I've been following the kind of AI malware story for a few years now, and virtually every time I talk to people about it, they say it's not really a thing. Like, yeah, attackers use AI to generate malware code, but it looks a lot like regular malware code. And I'm wondering,