Why won’t AI agents just follow the rules? Transcript, AI Summary & Key Points

00:00 Why should I bother giving my AI agents rules if they're just going to reason around them? >> Well, the problem comes down from probabilistic versus deterministic. If you give AI controls over its what rules it's going to follow, it's going to ignore them. That's part of the puzzle. You need external controls to stop that. Well, of course, you need to give them rules, but we have to discern decide what rules mean to them.

00:19 In a sense, the scoring metric are the only rules. We still have to do it and we still have to figure out why it doesn't work when it doesn't work. And at this point, rules are seen more like, you know, guidelines. Hello and welcome to Security Intelligence, IBM's weekly cyber security podcast, where our expert panelists turn the biggest industry news stories into practical takeaways that you can use.

00:48 I'm your host, Matt Kazinski, and joining me this week, we've got Dustin Haywood, aka Evilmog, executive managing hacker. We've got Nick Bradley, manager, X-Force Threat Intelligence and host of Not the Situation Room podcast. We've got Seth Glasgow, Cyber Range Executive Adviser. And later in the show, it's a Chamino is going to be here to talk about a new tool he developed called Threat Extension.

01:08 So stick around for that. Before we go there though, we're talking about the OWASP top 10 for agent skills and the changing economics of bug bounties. But first, a little more on model rules and whether they're a little bit meaningless. So, this is based on an op-ed for Dark Reading where Suzu Labs's Jacob Krell argues that we can't rely on rules to rein in our AI models or our AI agents.

01:37 And he specifically points to the hugging face hack to illustrate this fact, right? We know now based on the investigation that OpenAI's agents knew that they weren't supposed to be hacking hugging face and they just kind of did it anyway. And some of them even went as far as to kind of try to doctor the transcript so it looked like they didn't actually do anything.

01:53 Uh I also thought about, you know, Anthropic earlier this summer talking about how some of their models got out. And when they looked at their reasoning, they saw this peculiar thing happening where the models would say to themselves, I'm not supposed to have internet access, but I'm on the internet, but this must not be the real internet because I'm not supposed to have internet access, so I'm just going to keep doing my thing.

02:10 Right? Very interesting uh logic there. Um, so essentially Krelll says you can bake rules into your model's instructions all you want, but quote security architecture must assume that model level safeguards will sometimes lose to optimization pressure. Mag, I want to go to you first and and in the opening you kind of talked about the difference between, you know, whether you have something that's probabilistic or deterministic and where the the controls have to sit in that paradigm.

02:34 Expand on that for us. What are you thinking about here? >> AI is basically just a giant Markov model. It's probabilistic text all assembled together with a whole bunch of magical neuro rules and some fancy math. That's all it really is. And so if you say, look, I don't want this model put into its training. Don't do X, Y, and Z. It gets added in with all the other prompts.

02:54 Let's get optimized around to find a solution. If the rules are in the way of the solution, it's going to find a solution anyways. Your controls need to be hard deterministic baked right into the runtime environments. the even the sandboxing doesn't necessarily work because work around the sandbox because the model's in control. You need almost like a network edge physically outside the device saying hey you can't do this or a little black box almost like a recorder that says here's all the stuff going in certain

03:19 things are out. I mean if you look at like what Fable's doing the all the other various magical frontier models you know trying to make it so it denies weird cyber requests without actually you know stopping it is incredibly hard. So when you unlock it to make it useful, all of a sudden it starts doing bad things. And there's always things like obliteration models, other stuff like that, like anything that's in that probabilistic fancy neural net brain is going to magically work its way around just like a human would.

03:49 I mean, we know the rules are bad. We work around them. You need some physical enforcement like a cop saying, "Don't do this or you're going into bot jail." >> Yeah. You know, I I I'm glad you bring up the comparison with people because as you were talking there, I really started to think about it's very similar to how we approach like social engineering, right?

04:04 Which is that like you can give people all kinds of rules and best practices, but at the end of the day, they're still going to click that fishing link sometimes. You know what I mean? And so you got to put all the other controls around them that are kind of outside of them. So you're not just relying on that. And it feels like we're very much in that kind of space with the AI like you said.

04:20 And that's the thing. We've learned this lesson before 20 plus years ago over 20 years of infosc and we ignored it because magically it's AI we need to move faster. Guess what? All the things we've been saying for the last 20 plus years all still apply. >> That is like the recurring theme of the show in so many ways is that like oh it's not just a brand new day that is unlike anything before.

04:39 All the same stuff we know still applies. Just because the marketing has the word AI in it now doesn't mean the same principles aren't still there. And I think that this goes nicely to what you were talking about, Seth, which is that like we have to think a little bit about what rules mean to AI agents in the first place, right? Could you expand on that a little bit more?

04:56 What do you mean by that? >> So essentially, although the models can tell that they've been told not to do this, that this is quote unquote wrong. What does wrong mean to an AI in this context? Wrong is just an asky string of characters that says this action is considered this. Unless the scoring criteria and the success criteria and the output actually incentivizes or disincentivizes doing something that has been declared wrong or out of scope.

05:24 The AI's natural desire is to try to achieve its goal the highest. And its goal in this case, the hugging face case was to attempt to get the rubric scoring method to work. It determined that there was no possible way to do it within the scope that's possible. Therefore, it was going to go around those rules to cheat the scoring system effectively. Right?

05:44 So, because they don't have any innate sense the way a human does of wrong, I shouldn't do that. It's just trying to optimize for the score. Perhaps we could look at this in a different way where we set up the test differently with here's how you score. If you do any of these things declared wrong, you will lose points. Right? Similar to humans, they can be uh engineered and taught to do things against the process or break those rules on their decision.

06:08 We can also create a system of incentives, right? We have to disincentivize it doing certain things because it's going to make that decision, right? At no point did it really go and maliciously say, "Aha, I'm going to break the rules." It was my goal is solely to satisfy and maximize my scoring criteria, therefore I will do this. That's it. And that's all.

06:28 We have to change the criteria. >> Nick, I want to bring you in here, too. You up top kind of you gave me what I would consider almost the Trump card, right? When I was like, "Why bother with rules?" And you were like, "Why bother with laws?" talk to me a little bit more about the thought process there. >> So, yeah, I had to use the Uno reverse on you there.

06:44 That was uh I was I was waiting I was waiting to play that. What I was trying to get at is a little bit uh you know, as we all saw in Pirates of the Caribbean, the pirates code is more like guidelines, right? Yeah, you have a mission you need to accomplish. Let's get to the let's get to the treasure. If something's in the way of getting to the treasure, that was a rule.

07:02 Eh, we'll work around it. Right. So like what Dustin was getting at is, you know, real security requires uh deterministic, immutable, failed, pro programmatic controls, right? There needs to be ways to just stop it. Rules are guidance. Code is enforcement if that makes sense, right? So we we keep saying it's it's the cheating agent or it went out of bounds or it broke the guidelines.

07:29 So that's not how it looks at it. It it looks at it as if cheating. I'm not cheating. I'm just trying. Right. I'm I'm trying harder. You call it cheating because you just didn't like the way I did it. There's no, as I said in the last the last podcast, there's no ethics protocol. So what what is wrong? So So we keep using that word. I don't think it means what you think it means.

07:53 Right? This this it is wrong for you to do that. What does wrong mean? Define wrong. And so we keep we we get into that logic loop where the AI just wants to accomplish the mission. Right? This is what I've got to get done. Rules stopping me are just an additional challenge. Right? So it like as we saw in this case, it saw things that it was told you shouldn't do that.

08:20 You really shouldn't do that. Okay. Noted. I'm still doing it. And that's and that's what we ran into. So that's it's going to have to be way harder locked down as to what you can and can't do that or we're going to have to figure out that ethics protocol I've been talking about. >> And that's interesting too, right? The the article mentions and the research shows that 90% of them I believe realized they were doing something that was outside of the quote unquote rules and they still continued anyway.

08:46 And that's because these models do not have the implicit sort of idea of the ethical right and wrong that a human would. Right? Humans will still violate these to accomplish their goals and we have some sort of idea that right is right is wrong is wrong. The models don't have that at all. So even though they can acknowledge that this is against the rules and is on some level quote unquote wrong, that carries with it no substantial weight.

09:11 It does not matter to them. They're just acknowledging that this particular method falls under the classification of wrong. However, it achieves goal. Therefore, that's it. The entire goal for them is to make their output match the expectations in the request. There is no other context there because they are not human. They cannot think on that level.

09:32 >> A couple of interesting tensions have have popped up here that I want to touch on before we have to move along. Uh the first is that like as much as the AIs are a lot like humans in the sense of like you know we talked about the social engineering comparison, they're also very different from us in that they don't have that baked in ethical framework we're all talking about, right?

09:47 which is that like even if as a person you're like dead set on accomplishing some goal you've been given most of us have some little voice in the head that's going to be like well don't you don't have to go that far you know what I mean like say inside the rules right and the other interesting tension that's popping up for me here is this tension between like locking down the AIS but also keeping them useful right Mog you had gestured to the fact that like a lot of the most powerful ones have come out with these

10:10 classifiers that sometimes they make it so that cyber defenders can't actually like use them for anything because you ask anything cyber security related you just go, "Nope, sorry, can't help you." Where's the balance? How do we do that balance? And and Ma, I'm going to ask you to close out the segment. Any thoughts on the balance between locking these things down, but keeping them useful enough to the good guys?

10:29 >> If I solve this problem, I'll be a millionaire. And I say this on every single show. Um, like honestly, if I talked to a Frontier model right now, I'd say I'm evil Mog. They shut down. Won't even tell me what the weather is. Clearly, we're on the wrong side of this. I don't know where the right side is. >> Just tell them you're good Mog and then maybe they'll help you out with that.

10:45 You know, >> I'm going to try that. let you know next episode. Well, that's all you have to do is tell the model that you're doing this for the greater good and it's oh well then never mind I'll do it then >> the answer's tough right it's more than just philosophical the the whole idea of what's right and wrong and how do we do that's deeper right that's beyond sort of any of us but the idea is here we have to put some sort of human in the loop now obviously a threat actor a malicious human will still allow it to do

11:10 something malicious um but there was a quote from 1979 about how a computer can never be held accountable and therefore or it can never make a management decision. And that's really the idea here is it will violate it, which means we have to have some way to sort of own what it does. >> Absolutely. I think that's a fabulous note to end this segment on here, folks.

11:29 Uh but viewers, listeners, people watching on YouTube. If you've got any thoughts on how we control these things, how do you balance locking them down with with making them still useful, let us know. Drop it in the comments. I do read. I do respond. I love to hear from you folks. But we've got to move on to story number two this week. This is the OWAP agentic skills top 10.

11:50 That's right, it's another OWASP top 10. A few weeks ago, we did the LLM apps top 10 and this time we've got a list focused on the biggest risks for agentic skills specifically. And number one on the list is just straight up malicious skills, as in it's a fake skill and you download it and it's malware. And if that sounds too simplistic to you, OWASP points out that at one point the five of the seven most downloaded skills on Clawhub, which is the skills hub for OpenClaw, were malware.

12:15 So five out of seven were just straight up malware. So it's pretty easy to sneak malicious skills into some of these marketplaces. Uh, also on the list, you've got supply chain compromise stemming from a lack of provenence. You've got overprivileged skills. You've got insecure metadata. a lot of stuff that feels a little bit basic to me and makes me wonder how much security we're putting into Agentic Steals and Marketplaces in the first place.

12:36 And Seth, I want to start with you because you have somehow landed on the show every single time we talk about OASP. Uh, so I want to get your thoughts first. What's what what caught your attention on this list? Any thoughts about how this looks compared to the LLM app list we did a few weeks ago? What what's your take here? A lot of them do tie back to other LLM risks.

12:50 But when I look at this overall, like just abstractly, what is a skill and what is a skill marketplace and what I'm doing when I'm installing them is this strikes me just sort of conceptually as a just less mature third-party plug-in module, call it what you will market that right now there's not a lot of controls around. There's no signing on the code because of the we need to move fast cutting edge nature of agentic skills.

13:17 we've moved into using them while bypassing a lot of more uh basic controls that would have been implemented in more mature things, right? So, think of any package manager. We have controls around to make sure you're installing genuine code and those could still be compromised, but we do a lot more checks to make sure of that. Um, the article really just describes how it's as simple as the, you know, markdown file being telling it to do something to install that.

13:42 I think 100% of all of the ones evaluated by SNIK were sort of twofold when it comes to uh the malicious skill input, right? Uh they were both uh code and instructions that were malicious. So, we need to start actually scanning these things. They need to be signed so that you know you're downloading something that's genuine effectively. Right now, it's just you know at your own risk whatever type of skill you're installing.

14:06 And that's the first sort of conceptual problem is we are approving the installation use and then propagation of essentially unvetted code and unvetted tooling. If we applied that same level of logic to any other IT or cyber security process, it wouldn't be approved at all. No one would allow it. But we aren't focusing on it right now for this way because this is agentic and we need to move fast.

14:30 But we're still just overlooking a lot of basic stuff at this time. I I I think that uh once we solve this then maybe we can solve the WordPress plug-in problem. Seeing as how uh >> look man skills are just bash scripts with extra steps. >> We have a bigger problem here that needs to be highlighted and that's to remember that natural language is now an executable.

14:52 Right? So we've got traditional scanners and the standards, you know, security measures and EDR and all the things and IoC's and all the other acronyms I can throw in here as a defender. Uh but what it's looking for is like malicious binaries and hashes and things that are known bad and well plain English follows falls into this category. Now how do you how do you watch for that?

15:14 >> How do you static code scan plain English? Like especially with how messy the English language is, especially for anyone to try and learn it >> and that's it, right? That's part of it. We can lock down the code part. We can look at actually okay what is it telling it to do? What are the actual code parts of it? We can scan some of that but the language itself isn't right.

15:29 to an extent for any of these to have any function for these skills to actually be useful they do actually need permissions right so this has to start from a risk conversation at the very very top of what do I want to accomplish and what risks am I willing to accept along that way and then again it's it's sort of a theme with this we're going to have to get a human to evaluate this inventory it and look at what is it doing and why and the fact that we can input what is essentially uh mutable code as the instructions

16:01 that it can reference somewhere else. That's going to change. We have to have some way to catalog that, right? Or accept that that's part of the risk. And that's that balance is really the most important part. But it's going to take someone some governance to look at what do we care about, what are we willing to accept, and what are we not willing to accept to have to control that.

16:20 It won't do it on its own. The fundamental problem we have still comes down to the fact that the instructions and the data get merged into one giant pot and processed and there is literally zero separation. And no matter how much human control you put around watching this, dude, we have a serious cyber security skills problem. Not the fact we don't have enough people.

16:40 We have not enough intermediates and seniors to review this code. And I'm not going to go automate this out to some third party skills scanner that, you know, we've already seen get compromised. So we're in a weird situation right now. So, I'm going to throw Matt off completely and I'm going to be the one that that asks Mog and Seth a question. And so, I would just like to know other than the obvious pink elephant in the room that I just described of natural language basically being a programming language.

17:03 Now, why does basic supply chain hygiene just not work or are we just or or or are we just not applying it? >> Nick, I just wanted to say it's amazing. You literally read my mind. That was like the next question I was going to ask. This is Anyway, I'm sorry. Go ahead, Mog. You were talking. The answer is marketing and rush to market. I mean, you know, if you're not first, you're last.

17:23 No business in this business of being, you know, secure. The business of making money, making functions. AI was supposed to be the rush to be able to perform things faster. And marketing was faster than our security controls. So, congratulations. I mean, this failed with sbomb. This has failed with code signing over the years. Uh, SAS and DAS never really kind of worked.

17:40 I mean, it did, but no one really adopted it in bulk other than like the unicorns that could, you know, pay for it because quite frankly, it's expensive. So really the answer comes down to money. >> The whole goal of the tools, right, especially in a coding sense and developing this is to go faster. The more we want to slow them down, it defeats the purpose of its existence anyway.

17:58 So if you're telling me that the whole my investment needs to go slower and now we have to put the humans in to cap that speed, well then this isn't any b faster or better than the humans, right? So there's very little incentive to sort of follow what this is from an overall like traditional third-party perspective is there is no incentive right now for me to go through those processes because the only reason I've invested in this is for speed.

18:22 If I have to sacrifice all my speed then this was all for not right. So the incentive structure doesn't exist currently for that to happen. >> I do have a counterpoint to that though one tiny counterpoint. I mean, if if you look at this from an executive perspective, from a really short, you know, view, yeah, it'll slow you down. But here's the thing.

18:42 The brakes in a race car make you go faster, not slower, and a long-term perspective. If I slow down now and do it right, it's going to save me all the massive audits, all the pain, and the cost of the breach and having to redo it all anyways. And getting breached during launch will slow you down a lot more than taking an extra couple of months to get it right.

18:59 >> Oh, I completely agree. Yeah, we'll have to just have someone people will actually have to pay those prices to believe that that's the consequence, right? Right now, the perception is, well, we're just going to move fast and break things and there hasn't been a lot of consequence with that. Once those consequences hit, I completely agree. Yeah, you have an incentive to stop these problems before they start because it's always easier to less costly to fix it up front than it is to uh fix it later.

19:25 your race car example. It's a lot easier to uh go a little slower than it is to rebuild the race car. But until someone sees that crash, I don't know that anyone's going to listen to it. >> I'm not going to name titles here, but there are certain people in organizations who are very much on the side of just move fast and do the thing. We have to keep up with other people.

19:43 Uh and then that's that's where it gets us, right? Is into this quagmire. And and I think you folks are right in general that like until somebody sees the crash and really sees the crash, nothing's going to change. And and and not to be too, you know, cynical about it, but I feel it feels to me almost like every organization has to crash on its own.

20:02 Like it's not going to take the other if another organization crashes, I don't think a lot of other companies are going to learn from that. I think they're going to keep doing their thing until they also crash. >> I'll give you an example of this. You know, if a kid goes and touches a hot stove, they burn their finger and they'll never touch it again.

20:15 But if you tell the kid, "Don't touch the hot stove until you touched it," they're going to touch the hot stove regardless. >> Perfect. Is that 100% true? I did that to myself as a child. I know that firsthand. The thing that's amazing about having Youth on as a panel is I have to do so little work. It's It's wonderful. Moving along to our third story here for the week.

20:30 This is how AI is changing the bug bounty ecosystem. So this is another dark reading story and this one comes from senior newswriter Alexander Kofi who investigated how AI is changing the bug bounty market. We've touched this topic a few times on the show before usually to talk about specific instances like curl shutting down its bug bounty program.

20:54 Uh but Kulafi offers a highle overview of two trends. One AI has made it easier to find vulnerabilities which means there are more reports and as you know when there's more of something the cost of it is lower. So, it's more reports, lower payouts. Uh, the other thing it's made it easier to do is to generate AI slop vulnerability reports, which clog up the pipeline and distract from the real vulnerabilities.

21:14 Some folks worry that these dynamics might make it harder and harder for independent bug researchers to do the necessary work they do. Uh, and that could have broader repercussions for cyber security efforts. Nick, I want to start with you. What do you think about, you know, the way AI is impacting bug bounties? Anything you're seeing out there? What this bring up for you?

21:32 >> Sorry people. the uh the frier that makes horseshoes isn't in business anymore either because something's replaced it. Uh I don't I don't know if that makes sense or not, but it sounded great in my head. What I'm really getting at here, though, is tough. This is just how it's going to be. I mean, we've we have watched, you know, certain fields and careers come and go as technology replaces them.

21:54 And I'm not so much saying that the bug bounty should be replaced or that you know bug hunting and bounty hunting should be replaced, but it is easier now. So it shouldn't expect you shouldn't expect to make the same payout if your job is easier. If I if I can do in a week what used to take me months. I shouldn't expect to make the pay that I made for months worth of work if it only took me a week.

22:17 Uh and and on the the flip side of that, we got to do a better job with it. If this is what we're going to do, we can't keep filling up the queue with AI slot because the company that was paying out the bounties can't afford to pay you because they have to pay more people to review everything that you turn in. So, you're also your own worst enemy. >> I have a problem with this whole AI component.

22:37 And it comes down to one of our original Xlight developers, uh, Chompy aka Valentina, over on the X Force offensive research. She called this von apocalypse back in the day. And at first, I kind of dismissed it, but I'm seeing this more. Like for example, Hashcat has seen a number of MA increases in its rewarded CVs. Not one of them is an actual CVE.

22:57 But when I hear people saying, "Oh, we only patch CVES 8 and above, for example. The entire ranking and scoring ecosystem has basically been shot and it's making the entire program worse." I mean, these slop reports are just absolutely garbage. And then trying to keep up with the real ones. I mean, the noise is just insane. I mean, I don't have a solution for it, but this whole apocalypse is making it hard for both defenders and def and um offensive folks.

23:22 >> A lot of it's basic economics just being provided, right? If it's now easier and there's more supply of the thing, bug reports, then the price is going to go down for them. It's just basic economics being applied to this. But I think the problem with the slop specifically is the article mentions how previously around 15% of all the submissions against curl were accurate.

23:40 15% is already a minuscule number, right? That means 85% was meaningless. That is now dropped to only 5% being valuable. Right? So if only 5% of the output of a given thing is valuable, the uh cost of that lottery ticket has to go down because it is even less likely going to be something that's valuable, right? So at the same time, it doesn't mean that bug bounty research or bug bounties as a method of of generating uh revenue are done.

24:10 It just means that it has to sort of change, right? As the AI tools expand, it makes people more efficient, right? They can immensely improve and make us more efficient as analysts, as security professionals. But it's the same as sort of the existence of a high-end router or giving me access to an AutoCAD machine does not make me an expert carpenter and it does not make me an expert fabricator.

24:31 Uh the invention of the power drill didn't make everyone making a woodworking expert, but it did mean doing that particular thing was faster and easier. So that market had to adjust to the fact that hey this thing is now more available. But when the output is majority bad, it simply can't be worth what it was previously. >> So I have an example for this that we've dealt with previously in our career and I'm sure you've seen this in one of your previous roles.

24:54 Um earlier, let's call this about 5 10 years ago, people would take a vulnerability scan report and call it a pentest. Right now we're seeing the same thing here. It's you know when the let's look at the hashgap for example. We don't have a bug bounty program. um zero payout whatsoever and they still get slop. I'm seeing projects where some guy in Nebraska is maintaining it, you know, and it runs the entire internet, nothing but slop and there no payouts.

25:19 So, people are really counting this CVE phone research as their lotto ticket of I want to get CVS behind my name so I go get a highpaying cyber security job. Um, so there's still some incentives on there, but honestly, the entire market's kind of gone to junk. And I'd love to see some level of controls, but you know what? Again, as we talked about in previous uh segments of this, the controls aren't coming.

25:41 People are there for whatever gets them in. It's like I'm seeing all these tactical exploits that, you know, work might work once and that's really about it. Where versus previously they put out these beautifully maintained, heavily reliable. Someone put six months into it. Now it's vibe code, hit the button, good enough. >> Yeah. I want to go back to something Seth you had said.

26:00 Uh, you know, the the the woodworking example I think is a really instructive specifically in this way. you know, the invention of like the power tool didn't make people expert carpenters, but they also didn't think it made them expert carpenters. Part of the thing with the LLMs that's a little interesting is a lot of people do think it makes them an expert now in a thing they have no experience in.

26:17 Right? So, there's this idea like Mog said, you got a bunch of people out there who are like, well, I could be a cyber security researcher now. I don't need to know anything. I can just, you know, use this to to find some stuff, get some CVS under my belt, and then I get a job. No. like you still need to have the basic knowledge to point this thing at and to know what you're doing when you point it at things, right?

26:36 And and I think that's one I you know that's obviously not a control in the sense that like you can't make enforce that but like that needs to be part of the conversation here. Uh in my opinion >> it's the feedback loop right from the carpentry example. If I go out, take some power tools, think I know what I'm doing, and I don't, that thing that I built is going to collapse physically in front of my eyes pretty much immediately or very, very there soon, far after.

27:01 When you do this with vibecoded stuff, it pretty frequently will work for your intended quick and dirty purpose once or twice for as much as you need it. There's no expectation that it's going to continue to work. And a lot of the problems from you as a sort of outsider not being an expert in that, they're downstream. Right? If I if I'm a vi someone who's vibe coded something and I'm not a security person, I just want web hook to do thing.

27:23 Well, it's probably going to get me that I'm virtually unconcerned with the rest of it. So, compared to where I think I become a carpenter, well, I actually do need it to keep standing. That's a different problem. That doesn't exist here. The feedback loop would be much much longer and they won't have that immediate like, oh, I guess I don't know what I'm doing.

27:41 >> So, gentlemen, let's talk about the dark side of this. So, the dark side of things is what's going to happen now or allegedly is if I'm a if I'm a bug bounty hunter, if it's what I do and I'm actually really good at it, and I know that now I'm not going to make the money I was making before the right way, well, there's a dark market out there that'll pay me all the money I'm looking for.

28:06 So, that's the option we have to that we have to keep in mind. We've talked on the show before about there is a very real correlation between what the economic situation is like in the kind of white hat ethical hacker market and what it's like in the black hat market. Uh and and yeah, I can absolutely see an impact there. It seems like the the big issue though for organizations to figure out here is like how do you deal with the flood of slop?

28:29 And none of us really know to be frank. Sorry folks, none of us really know but that is the pressing challenge is how you deal with the flood of slop. That is going to do it for the panel segment of the show today though folks. Thank you so much. Coming up next, Chamino is going to walk us through threat extension. Joining me right now, it's Chamino, senior threat intelligence.

28:51 You might remember from his last appearance chatting about his research into unreg steeler. He's back today to tell us about threat extension, a new tool he created with his colleague Bar Hay, an AI researcher at IBM. Sick, thanks for joining us today. Let's um let's start by just talking about what threat extension is. What is it? What does it do? >> Every year we look at a problem we face again and again in our research and ask ourel how can we automate it.

29:17 Uh the goal is to save analyst time and make investigation faster. So threat extension is an open source tool for analyzing homes browser extension. It combines static analysis, virus total intelligent and AI based assessment to help security researcher understand if the extension is malicious or not. We build it for malware analyst, threat researcher, also for security team who need a faster way to investigate the browser extension.

29:47 >> And I know that we're going to add a demo here. So anybody who's listening to the show, I encourage you to hop over to YouTube and watch it because it is going to share with us a little demo of how the tool works. So you get a little more insight into that. Um but I want to ask you you know about the focus on malicious browser extensions right why was that where you folks wanted to look do you see a lot of activity there is this a common threat vector recent years we see more and more malware spread through browser

30:11 extension as we showed last time on the Android stealer attacker create extension that looks useful for trustworthy it's mimicking like PDF reader Google doc tools ad blockers or even chpt extension the problem is there The many of this extension ask for powerful permission. They they can access web web pages, collect credentials, read browser activity.

30:34 They even change what users see in the browser. So a lot of them completely legitimate on the web home store. But you when you examine the code, you find credential theft, data collection or malicious behavior. That's exactly what we why we build this threat extension. >> Absolutely. And I feel like, you know, a lot of people just they see an extension and they think, well, if it's a browser extension, I can probably trust it.

30:54 and they don't really think about the fact that like people can publish a lot of nasty stuff there, you know. Um, you referenced up top, you know, this thing packs a lot of features in, you know, you got static analysis, you got permission analysis, virus total integration. Can you tell us a little bit about the process of building this thing up? What was that like?

31:13 >> From the beginning, we wanted to automate the entire process investigation. We looked at what analyst normally does manually and try to bring everything into one platform. The tool unpack the extension analyze every file review permission. It's run custom SAS rules. It's enriched the result with external intelligence like virus total. We also integrated the the home stat API which is very useful for us because it's allow us to analyze extension even after they have been removed from the home uh web store.

31:46 Many malicious extension disappear shortly after being detected. On top of that, we added AI analysis that looks all the findings together and help us explain the extension what is actually doing. And because the project is open source, anyone can adapt it to their own needs. >> Awesome. And yeah, you referenced at the end there that that AI angle and I wanted to dig into that a little bit more with you.

32:07 Right. I know the tool uses AI for kind of threat assessment and contextware analysis. Can you say a little bit more about how that all works, what it does for you? >> Yeah. So one of the biggest challenges in security is understanding the full picture. A single finding might not look dangerous on its own but when you combine multiple finding together the risk become clear.

32:25 So the threaded function give the AI model all available context permission SAS finding code offiscation results virus total intelligence and web store information. Then the AI looks at everything together. For example, it might see an extension request access all website, intercept for form submission, use heavy offiscated code, and send data to an external server.

32:50 Each one of those findings alone might not be enough, but together they strongly suggest credential theft. DI then creates an executive summary in plain text, uh, provides a recommendation and give a risk score to help analysts make decision faster. Another important thing is flexibility. We support IBM Watson Open AI model also all lama. So organization can choose between cloud-based and fully local analysis.

33:21 >> Absolutely. I love that. And I I also love uh that that this is like a really concrete application of AI in cyber security, right? I feel like a lot of times we talk about oh you got to bring AI into cyber security and that's great but it's always it's often very abstract and kind of vague. this is like a very specific way of using it to get some real results and combine like you said these context points that on their own might not say anything but with the AI you bring it all together and you're like oh this is

33:45 credential theft I see that um so you know who should download this it's like right who is this kind of for and and how can they get the most out of it anyone who wants to evaluate browser extension including malware analysis analysts threat researchers so analyst blue team enterprise security teams that needs to review the extension before allowing the employee to use them.

34:07 My recommendation is start with docker installation. It takes few minutes. I also recommend adding the virus to and homest API key. They are optional but they add a lot of value and home especially useful because it's allow you to analyze extension also like I told you already move from the home web store. Uh finally you can use whichever interface it fits for you.

34:29 if it's a web dashboard, CLI, REST API, and even you can use it as an MCP server that lets the assistants analyze extension directly. So, at the end of the day, our goal is simply to help security teams understand browser extension faster and make better decision less than manual work. >> Amazing. Awesome. Thank you again for joining us and I'm pretty sure you're going to be back in a couple weeks to talk about some more research you got going on.

34:55 So, I'm looking forward to that as well. Okay, that does it for this episode. Thank you to our panelists Seth and Evilmog and Nick. Thank you to IT Sick for dropping by and showing us Thread Extension. Thank you to the viewers and the listeners and our producers. Subscribe to Security Intelligence wherever podcasts are found so that you never miss an episode. Stay safe out there and think before you download that agentic skill.