AI slop stalls bug bounties, malware hides in IoT and agents hit Wikipedia Transcript, AI Summary & Key Points

00:00 panelists. Today we're dealing with among other things IoT devices playing double agent and agents causing headaches for our favorite website, Wikipedia. So my question to you is what is your favorite Wikipedia page? Um Sophie, how about you? >> That's hard for me because growing up I was always warned against using Wikipedia. So it was kind of a scary to me.

00:23 Um, but I would say anything science related, pop culture, anything I need a quick uh deep dive for IDMBB page and whatnot. >> Probably the last page I'm on. So, as you kind of go down the rabbit hole, or I could say whatever page Steven Colbear comments on next and causes the entire edit storm. Oh, >> it's tough because the best part of Wikipedia is going down the rabbit hole and ending up somewhere where you had no intention to be.

00:48 So, I'd say my favorite Wikipedia page is one I haven't found yet. Hello and welcome to Security Intelligence, IBM's weekly cyber security podcast, where our expert panelists turn the biggest industry news stories into practical takeaways that you can use. I'm your host, Patrick Lucas Austin, and today we're talking bug bounties, compromised enterprise devices, and AI agents going after the world's most useful website, I'll say.

01:18 We're joined by a great group of panelists today. We have Sophie Cunningham, dark web analyst for X Force threat intelligence. We've got Ian Malloy, department head of security research, and Seth Glasgow, Cyber Range executive adviser. Let's get into our first story. It's a story I think that we've heard a lot more this year, AI overload. This one's specifically going after bug bounties, which has seen its own sort of um reaction to the influx of AIdriven uh bug bounties.

01:55 Um in this case, it's Google's turn to be overwhelmed. Google's open-source software vulnerability rewards program. Um, it's kind of a mouthful, so I'm glad they shortened it to OSSVRP. Um, has been basically swimming in bug bounty reports, which sounds great until you realize a lot of them are a bunch of nonsense. On October 1st, Google paused its uh vulnerability reward program after being overwhelmed with AI generated reports that were full of hallucinations.

02:25 The program originally launched in 2010 and expanded in 2022 to cover open-source software, which is incredibly important because Google maintains quite a lot of open- source software and some very important open- source software uh Angular, Chromium, Go, Kubernetes, etc. Um, so it's kind of a big deal to have an open source software uh bug program paused because of an influx of basically bogus reports.

02:58 Sophie, I want to go to you. How does your average bug bounty program work and how does this influx of AIdriven reports change um you know how people are responding to these to bug bounties? uh in general >> I mean typically it's how it should work is a security researcher will identify a bug and exploit submit it and get paid for it if it's successful but with the advent of AI there's now a lot of AI slop and so anybody script kitties people have no experience can just create their own exploit or bug try to identify

03:40 bugs and submit it and hope that hope for the best. I mean, think of how many reports they're getting to have to pause a program. Uh because they have to verify it and a lot of them are just junk. It seems >> it is a lot of reports. Luckily, Google has said submissions made before October 1st will be sort of looked at and honored, but the program, the open source part of the program is going to be frozen until Q1 2027 according to Google, which seems like a pretty long time.

04:12 Um, if you find other flaws, uh, Google says you can report them to the patch rewards program or their cloud vulnerability reward program. Um, but it seems like this is a problem that is not going away. Ian, the bugs persist, but will the bounties persist in this particular form? Will they sort of morph or evolve into something different? >> That's kind of difficult to say.

04:36 I mean, typically do bug bounty programs so that people don't weaponize them. Uh, there's going to be a big market for them. So, you want to have them fixed properly into the community as opposed to kind of being used for kind of illicit purposes. I think the one really interesting thing here is we can look at this a couple different ways. One would be uh like pessimistic or optimistic.

04:57 I mean pessimistically they're taking away the bug bounty program. Uh but and that is going to reduce some of the avenues to you know properly report these and the fear is that they're not going to be reported. On the other side, you can kind of look at this as Google has very very sophisticated AI models, agents that they have that they're scanning and they might have actually just done the calculation and said, "Look, we are better than at this than some of the uh contributions that we're getting."

05:24 And it is now cheaper for us to run all of our crazy AI models and agents against everything than it is to go through and adjudicate all the slop that we're actually getting. And Google's also not alone. There have been other kind of communities that have had to do similar things. Uh if you read through curl, they've had a whole bunch where they've had to kind of stop and they actually make for some hilarious reading.

05:43 Uh if you want to see the type of slop that people submit and a lot of it is, I don't know, there's a bug because Claude or GPT or some other random agent says just a bug and when you ask the person to kind of clarify it, they said I don't know. I'll have to go ask the agent. So it it's really placing an incredibly large burden on the people reviewing bug bounty programs and they just have to find a better way to to to go about this.

06:06 So, it's really just kind of shifting the cost. It's shifting it from finding them, which used to be very difficult, that's now become easy, and it's now kind of verifying and patching that's become difficult. >> That's a great point and it and it brings up uh Google's recent sort of development um using its page break AI agent to find hundreds of flaws in its in its own web apps.

06:26 Um, is there anything about that particular aspect of Google using AI to find its own bugs that that stands out to you that that makes you think, well, you know, bug boundaries are basically kind of going going away. If we can find them and fix them faster than other people can. >> Yeah, I think they said they they found about 500 uh vulnerabilities in their own, you know, websites.

06:48 The interesting thing that they did is through verification. I think that's kind of one of the big distinctions. They had, you know, nonAI agents. So they had uh deterministic ways to kind of go through and verify dynamically that it works. You can actually run the exploit. You can determine that it has actually successfully you know uh executed uh injected JavaScript or it's you know read or write files, executed different processes and so you know definitively that it has actually done that and it's a true positive.

07:17 So this is I think what a lot of people are kind of shifting towards. um a lot of the other kind of solutions if you look at you know anthropics glass wing and I'm sure Daybreak and all the other kind of efforts they're moving more towards like dynamic verification which is usually been more difficult but then it really cuts down with false positives and so that's probably the biggest thing >> I think on the other side of that which is my pessimism talking is if we're giving them more practice with exploiting is that's

07:42 concerning for breaking out of sandboxes >> that's a great point >> it's just you know what what side of the coin you want, the lesser of two evils. So, we'll see what happens. >> Seth, I wonder what you think about the the more modern um you know, approaches to bug bounties. Um I know Microsoft and GitHub are trying a sort of VIP program where you find a certain number and a certain type of bounties to be um a certain number and certain type of vulnerability, excuse me, to be a part of its sort of membersonly VIP bug

08:18 bounty program. Um, and I know Microsoft has expanded its bug bounty program in general and awarded over $20 million this year to over 500 researchers. There to me there is value in this program in these programs for researchers um, you know, looking to uh, practice their craft and and improve their and improve their you know ability in cyber security.

08:42 Are we gaining or losing something in the in the cyber security community when the bug bounty program is basically handed over to the AI agent? >> Yes and no. Um really when you look at like sort of the vetting of a VIP, you're essentially saying these individuals are trustworthy. We can trust the findings that they have to a higher degree. So it's going to cost us less to deal with them in the long run.

09:06 At the end of the day, we're talking about just ways to mitigate the slop, right? And so when it comes to page break as well, especially with web vulnerabilities, it's very easy to scan for them. You get tons of false positives because you're lacking that runtime context. Using an agent to then go and run the exploit sort of allows us to fight AI with additional AI, right?

09:26 We can't ever unring the bell. There will never be a world where we're not using AI to look for vulnerabilities at this point. And so we have to just be sensible with how we do it. So shutting it down is makes sense, right? Like, hey, we're getting entirely too much. we have to look at this in a different way. We can't vet all of these because people are just flinging it at the wall and see what sticks, right?

09:44 They lack context. So potentially using AI to help with that can be one avenue to still replace the high volume, but finding trusted individuals who you say, "Hey, we sort of deputized you to have access to this because we trust you is another way to do it." Right? That sets another barrier of entry for just trying to mitigate the amount of slop that you're going to get thrown at yourself.

10:07 Right? It's basic human sort of behavior. There's a financial incentive here. I can program a tool to try to capture that financial incentive. Folks will continue to do it. So I don't know if it's about losing anything. It's about how do we manage the new normal in terms of the amount of data and the amount of reports, right? And using something like page break on the agentic side to actually exploit the vulnerability, figure out is this real?

10:30 I I think that they said they were finding, you know, next to 0% false positives. That's huge from a webcanning perspective, right? because there are typically very very many and then you have to manually go in and figure out what works. The end of the day we have to cut down the noise to get to better signal. >> Comment on what Seth was saying there.

10:46 I actually think AI agents uh the bigger risk is not killing the bug bounty programs but killing CTFs cuz that is the education. That's the pipeline. That's what gives people humans actually the understanding of security to kind of hone their skills. and now they've been completely taken over. And so we're going to have people that no longer have the skills to kind of look into the the findings to kind of think creativity create creatively to reason about them and actually, you know, be the ones who look at these

11:18 reports you're going to get from the agents and determine whether or not is this a true case? Is it a false positive? Is it just a fine, you know, is it a hardening finding? Is it something that's even exploitable that I need to worry about? Yeah, it's sort of that talent pipeline problem is going to affect pretty much all areas that we've applied AI fully to the lowest level, right?

11:34 Sort of in socket analysis, right? That level one stuff is being almost handled entirely by AI. So, it's less about should we? Yes, it's good at it. It can do the job, but it's more of a philosophical question on how do we develop talent, right? When someone comes out and doesn't have experience, we used to let them cut their teeth on these things to get better so they can start doing that higher level analysis that's required.

11:54 um we end up we may end up creating sort of a der of talent at that low level and how do you get from here to that higher level if you didn't sort of get the at bats at an earlier point in your career. Yeah, I think that's more of a social problem than an AI problem in that we have to decide as a society where we're going to have AI take place and where we're going to say okay no you can train with the AI so that you're learning these skills and working your way up or do our um kind of jobs change and shape with AI and

12:25 so now we don't need engineers to take on this role so they don't need to know the lower end skills once AI develops enough. And so I think just figuring that out is going to be a really big issue in the in the future. >> I think it's a little different though because you spend a significant amount of your time looking at reviewing, you know, a proposal, a plan the AI has created, look at the code that's generated, look at the vulnerability report it's actually generated, and you have to be the arbiter of like, is this

12:55 good or not? You have to have that taste. And and that's one of the things that I think people are potentially losing. Educationally too, it's not like the existence of calculators made it so that no one learned how to do math by hand, right? We could use the tools and still learn how it works, right? There is a way to combine these two things. It's not it doesn't have to be all AI or no AI.

13:15 >> I think that's a great place to uh move on to our next story. Um, but before we do that, I would love to encourage everyone watching and listening to subscribe, to like, to leave a comment, and uh, let us know what you think. A new Linux malware strain is here, and it's going after more than your smart home light bulbs. It's going after your company's network devices.

13:40 A Forinet Foragard Labs threat research report found Linux malware called Clingun uh is going after IoT devices using old and new vulnerabilities that date as far back as 2021. Um, according to Fordicard Labs, it uh has employed about two dozen vulnerabilities to exploit some enterprise devices using what it calls the public stun server, the session traversal utilities forNAT, which is used for enterprise devices to communicate to other devices from behind a firewall.

14:18 Um, Clson is very interesting. Um it kind of once uh Clingstone has penetrated a network, it then uses seven other vulnerabilities to get further within an enterprise network and exploit other IoT devices. Um, according to analyst Vincent Lee from Portuguard, um, quote, "Its use of multiple vulnerabilities for delivery highlights the importance of maintaining an accurate device inventory, reducing unnecessary internet exposure, and promptly updating firmware and software.

14:52 Uh, my question for you, Seth, is why uh why are software updates still a problem? Why why isn't it, you know, why aren't we doing this automatically yet?" So in particular, this vulnerability takes advantage of sort of what I would call internet backbone type technology, right? It's hiding in plain sight with how it's traversing on things that are necessary for the internet to work, right?

15:14 Without network address translation, our entire concept of how IP addressing works is sort of out the window, right? We had the idea of IPv6, but we've moved back from that. In terms of the patching problems, we're talking often about IoT devices, right? Firmware updates are very invasive. oftentimes these devices may be abandoned by the manufacturers.

15:34 The manufacturers no longer exist. And then furthermore, we're not just talking consumer grade, but even in the corporate environment, an enterprise environment, these devices that sit on the edge and have this connectivity often fulfill very crucial and critical roles. Right? So trying to patch something that is an OT device that sits in a critical process carries with it a lot of risk.

15:54 If I update it, does it still work the same? I have to absorb downtime to take it. It may require someone physically going to the device depending on where it's sitting as well to plug in and flash that firmware. So sure, you can force updates on consumers, but you can't really necessarily force updates on an enterprise when you're going to take something down and cause financial impact.

16:14 So this just goes back to stronger processes and trying to build better resiliency. Is there any benefit to like uh Vincent Lee says you know monitoring traffic um adding multiffactor authentication restricting access does that um I would say maybe extend the lifpan of you know the the time you can sort of wait to update your devices or is is making sure that firmware update goes through priority one and these are all sort of um you know second steps.

16:45 It's those are all those are all best practice, right? They they were already best practice. Setting those things up was nothing new. Uh the danger here is that being able to chain the vulnerabilities together quickly. So it sort of now is an external exposure to a bunch of the soft underbelly inside which isn't patched as well. So yeah, we we do need to do all those things and patch all them better and apply authentication and monitor traffic, but those were things we needed to do anyway.

17:10 So, it's it's sort of just the same operational condition, but the time to remediate has shrunk significantly. >> Sure. Um Sophie, I'm thinking of a story covered on the show last week about uh malware impersonating um email uh services and applications. And the idea being the idea being it was difficult to detect them because there was no baseline um to detect any anomalies.

17:36 Is there is that a similar situation here where a baseline would sort of help uh detect when something new pops up or when something um interesting happens, something um unoured happens? >> Yeah, I mean it's part of it, right? This is a living off the land type of attack. And so like Seth said, a lot of the stun protocol is actual normal behavior and then they're hiding in their malicious behavior uh and trying to get passed by.

18:07 And so you need some sort of behavioral scanning and understanding that baseline of your system to be able to detect, oh, this request is unusual. We don't have enterprise coming out of Asia. why is there why is it hitting here? And so having that baseline and being able to have these um uh these really smart tools look for these things is is key. >> Ian, one more question, one question for you.

18:40 Um the researchers found it difficult to detect any any evidence of these um of this malware because of its sort of deleting of information sitting on the server using these servers as a middleman um of sorts. is the future of enterprise attacks. These sort of more uh middleman type middleman type attacks taking advantage of public public in public uh networking um and then kind of pulling away if there's any sort of um um response from the enterprise >> probably viewed as like all bets are off.

19:19 I mean everything is fair game, everything's going to be used. Everything's going to be leveraged. you know, agents are going to be incredibly creative, uh, and they will take any resource they can, any small vulnerability, and they'll figure out here's the best way I can use this to my advantage to kind of spread laterally. And if a human hadn't realized this, I think I was looking, the latest oldest vulnerability was like 2014, I'm looking forward as a vulnerability getting its driver's license.

19:43 Um, but it still has been patched. And it's probably because a human, you know, maybe hadn't figured that out or realized how to chain this and exploit it. But yeah, now agents are, you know, they're they're pulling it all in. Here's every single vulnerability. So it's like, you know, what what are the union of all vulnerabilities that that exist out there?

20:00 And because as Seth said beautifully, like no one's updating, so they're all going to be used living off the land attacks are, you know, a great thing. And this one, it it blends in very nicely. So you can imagine like yeah baselining but if uh you know the the malicious traffic fits in your 5% confidence interval very nicely you can't knock off that 5% because as I said you know critical infrastructure the risks there are just too high >> potentially something that could help us mitigate it you know if you had an

20:26 inventory one of the tasks typically in sort of the pre-automation world was an individual usually on the cyber security team would have to take that inventory and let's say you're in an OT heavy environment utility something like that at a regular interval you need to go out and check every device model number for known flaws and known vulnerabilities.

20:44 It's a task that can be done. We could speed that up significantly with agents. That gets us one place to where, hey, we know there's a problem. We know we can patch it. That eliminates one of the issues of you don't know what you don't know. But even still, applying the patch requires a lot of coordination, a lot of prioritization, and even in the case where you may have a patch available, and it makes sense, the risk of taking that item offline could exceed the risk of exploitation.

21:08 And so in that place, we're going to have to look at compensating controls, right? Especially in the case of stun, we're hiding this in traffic that is known good, right? That's the ideal state for an attacker because it is by definition not an anomaly. So from that perspective, it's potentially trying to get better knowledge of where I have weaknesses, accepting that to an extent, and building better defenses around them because I may not be able to switch out the vulnerable component.

21:37 Moving on to our last story. Um, open AI agents have found a new target and it will probably lead to you seeing a lot more banners asking you for money. um presumably with Jimmy Whales face on it or I guess I guess Bernardet Mian now that she's the new CEO of Wikipdia um because Open AI agents have found Wikipedia and um are having fun on Wikipedia.

22:03 They've so far made millions of API requests, crawled millions of pages, and have made hundreds and of thousands of queries to the Wiki data query service which may have led to its partial shutdown in May. According to Wikipdia, um according to Wikipedia, um which released a a statement about the evidence, uh it found that OpenAI agents were u crawling all over its uh website.

22:33 Quote, "We did not find any evidence that our systems were used for coordination among agents, nor did we find any evidence of our systems or data being compromised. However, we are concerned about what could have occurred here. the difficulty and effort involved in investigating and attributing this activity and the growing risk of agentic AI activity on our platforms in general.

22:55 Uh my question for you uh Sophie, how does Wikipedia deal with the influx of AI agents without a blanket ban on all agents traffic? Or is that what they do? I mean to me the story is AI agents being AI agents in that they were kind of scraping a bunch of things. They were looking for exploits. They're trying to they're making changes that only they could read because it was kind of more of a sandbox environment.

23:28 >> Sure. Um it's it's particularly concerning I think in Wikipdia's case considering it is a you know pretty humanpowered endeavor um with human editors and and you know people donating to keep the site alive. Um, you know, Wik Wikip media says, quote, "Incidents like this one and the many others that have been and are still being uncovered illustrate how AI agents can drain resources and crash servers as well as attempt to compromise trustworthy information.

23:59 These AI agents modified pages that human editors had to go back and and fix. Um, so I'm wondering, you know, Seth, I'm I'm wondering if Open AI or or any any company producing these AI agents needs to rethink how determined these agents are when it comes to solving a problem, you know, come come hell or high water. You know, we're going to get it done regardless.

24:24 No matter what sites we crawl, no matter what uh, you know, pages we hack, we're going to get it done. >> The problem, I think, is it's larger than that, right? As Sophie said, this is how agents will behave. They are going to attempt to finish the task that they are given. But they don't do that in a vacuum, right? The agent didn't decide to to do this to Wikipedia because it was fun.

24:44 It doesn't have that notion. It was given an incentive structure by open AI to do it. If I have a bonfire in my backyard and it spreads and burns down someone else's house, that's not just fire being fire. That is my responsibility to control the thing that I am doing. So we need we need to define legislation, right? As Sophie said, if a human had done this, it resulted in a denial of service.

25:06 That would be undoubtedly a crime. I think it's insane that we can let well, we're just going to let Open AI do it because we're playing out here. It's like, you've failed to control it, right? They have to either do this or face actual consequences and own the liability of something that they are turning loose onto the internet causing measurable and real harm to other organizations.

25:29 you know, fair. Um, you know, Ian, it feels like this is we are in many ways in a sort of wild west of AI agents. um you know what can any what can Wikipedia or any any company that maybe wants to control you know a little more how that data is shared with these agents you know how does Wikipedia sort of you know where does Wikipedia get its own sort of sheriff to sort of protect Wikipedia from from these from these agents coming into town >> I think Seth had won maybe some form of like liability or to law would

26:09 actually be some way to actually resolve of that which is kind of interesting. I mean just kind of go back uh I mean every AI model is trained on Wikipdia data. I mean this is where they get the vast majority of very like grounding knowledge. It's incredibly high quality data but they make it very easy for you to go and download in responsible ways to train the data there.

26:28 I don't think that's what the agents were doing here. It's not like they were trying to you know gain some new information. It's they're looking at ways of communicating, sharing messages, things along those lines. So, I've effectively heard, you know, uh, with the OpenAI hacks, you know, there are two main things that people keep kind of recommending.

26:46 One is that we need to tell the agents they need to give up. You know, if the agents only just gave up, that they wouldn't have people. And I kind of fundamentally don't agree with with that as a notion. U, we want them to do things and to think long and hard because that's how they're going to do the things we want them to do like cure cancer. What we don't want them to do is hack other companies.

27:04 And that kind of comes down to, you know, having the appropriate controls in there. As as I said, it's it's your responsibility. Where's the leash that you actually put on the agents, where are the guardrails? How are you confining them? How are you monitoring them? So that you know exactly what the agent is doing. And you can actually identify when it's broken out of its confinement.

27:23 And this is more than just I'm running an agent inside of a sandbox and oh, I gave it all full network access, but it couldn't read or write some file on my file system. Like that's not isolation. they need something significantly stronger and it and it kind of feels like they've all failed on a couple of these like fundamental checks. Um the second thing I keep seeing the recommended is oh we need to prevent any agent from talking to another agent like that is you know once they find these uh these ways of doing it

27:50 whether or not it's through Artifactory, Wikipedia, Pacepin or anything else and that's kind of where everything kind of breaks down but again like there's a lot of very valuable reasons you want agents to communicate with other agents. So lets them, you know, share knowledge and break down tasks and not kind of waste resources and things along those lines.

28:04 People probably need to do two things. One, let's actually define a way that they can do that. Here, if you have a service and the agents need to be able to communicate, provide them with that service so that they don't go and kind of, you know, repurpose something else. And two, you can monitor it. Like I I can look at that service. I can monitor. I can see exactly what the agents are doing.

28:24 And that gives you that additional signal if the agents are maybe becoming misaligned or going rogue. and you misinterpreting or kind of coming up with their own objectives. So, these are a couple of the things I think the question you actually asked is what can Wikipdia do and you know better firewalls or you know captures that they're going to break through or I'm sure someone's going to start recommending was it X42 you know there's like micro payments uh you know it's got to make you have to make it cost something

28:52 for the agents or make it more difficult for them to actually kind of get through but without kind of now burdening humans the services are actually designed for >> we've already started to see other corporations take this action right Amazon has already banned and um Muse Meta's agent from doing shopping tasks, right? Because they immediately saw an influx of traffic.

29:08 Wikipdia's case, they sort of need to be accessible that way to fulfill their function of distributing knowledge. If we don't get some sort of control on what the agents can do or how they're going to do it, that will drive downstream parties to take more extreme measures such as attempting to block the traffic. And I don't think that's really what drives us.

29:27 That's not the positive outcome for anyone, right? We don't want to have to shut it down because it's sort of making a mess. We want to have it work without making an a mess. Ideally, >> I think we're also starting to see this transition and that people are starting to notice that we're going very fast for this technology and they're starting to call for slowing down and thinking about this legislation and taking the time to create it.

29:50 And from like a society point of view, I think that's important as well. If I could throw in one more comment, I think this Wikipedia and everything else, it's the dead internet theory is playing out very nicely now. It is it is their it is their internet. >> Sure. Could you could you elaborate a bit on the dead internet theory? I >> think the dead internet theory is the theory that, you know, there will be no more people.

30:11 There'll just be like bots and agents kind of communicating on the internet. >> That's what scares me most is just the cyclical nature. >> We'll come full circle. They told us not to trust the internet. They overrusted the internet. and we'll end up back not trusting the internet. >> Hopefully in our next go around on this circle I get to hang out with you guys um because it's been a great uh episode.

30:33 Thank you very much for your time. Sophie, Ian, Seth, uh thanks for joining us today. Thank you to the viewers and our listeners and thank you to our producers. Um subscribe to Security Intelligence wherever podcasts are found so that you never miss an episode and stay safe out there.