Future Cybersecurity: Hardware Memory Safety, Automated Governance and Post-Quantum Cryptography Transcript, AI Summary & Key Points

00:01 The decisions you're making right now about AI adoption, architecture trade-offs, and how your team works together will shape your systems for years. Getting those calls right when the landscape is shifting this fast is hard. QCon San Francisco has spent 20 years connecting senior engineers with practitioners who are a few steps ahead on the same problems.

00:16 This November 16th through the 20th, 60 plus speakers across 12 tracks will share what's actually working in production and what isn't. No hidden product pitches, just senior practitioners helping senior practitioners. Learn more at qconsf.com. Hello everybody. I'm Olympop infoq editor and I have in front of me today Chris Swan. I'll let him explain what he does for the living but I know him as a track host for the infoq event QCon mainly in London but he's part of the program committee for a long period of time this

00:55 year particularly he had the security track on his hand and given that we had so many things to learn from we said that we'll just look into what happened and what he predicts for the future Chris can you please introduce yourself >> sure thanks so hi I'm Chris Swan my main job is as an engineer at where we're building a platform that makes it easy for people to build postquantum security into their agentic applications.

01:18 So I've been doing that for a little over 5 years now. It involves me in open source and I also like to spend time in the broader community with things like uh QCON. It was my real pleasure sort of end of last year to be invited back in order to host the uh security track. I think at the time I didn't have a particular overarching theme or thread running right the way through it, but on reflection I think you could say it became about understanding the foundations that we're standing on and the fact that a lot of our

01:53 vulnerability comes from the leaky abstractions beneath us, whether that's dependencies or underlying platforms and things like that and that we can't rely upon human vigilance to keep track of all of that stuff. We need to systematize those things. We need to automate them in order to have the machines constantly pay attention to what's happening in those layers and where the vulnerabilities might be emerging.

02:18 And and I think that on reflection stitches together all of the different talks that were in the track. >> Yeah, it was nice to see all those topics coming together because uh I looked into this in the last couple of years and um one topic that people usually left behind. Well, we all know that security in terms of software development is always left behind for somebody else or for a later stage for the next version.

02:43 But what happened is that you had governance as points of the discussion. I think Sarah Wells was talking about it and then you had um Victor Anderson talking about test bombs and especially in Europe that's or should be an important topic given the CRA that is coming into force and putting the limelight on um the way how open source is built and how is maintained.

03:07 But what was nice is to see also points about hardware security, how the lower levels are being treated. And funny enough, in the last couple of months, so in the 6 months since QCON became history, it just came out that these things are even more important, especially with AI being introduced into the AI space. Any thoughts on how things evolved, if you would have had any predictions in April?

03:32 Which of them um came to fruition and which of them are still a question mark? So part of putting cherry onto the track which was David's talk you know focusing on how we can make security better in the hardware was I think we're still at a point where there's an opportunity to make safety better in hardware which will benefit you know the entire software ecosystem and that opportunity comes about because the profile for risk 5 on Android hasn't yet been fully baked and so it's still possible that Cherry could become

04:08 part of that. And if we then had hardware memory safety baked into the system on chips that that were in, you know, popular handsets and the stacks using that, I think that would be a massive boon for our security. Now, we've already seen a glimpse of that. If you look at the latest iPhone system on chip and the latest kind of flagship Android system on chip for the Google Pixel and some of the Samsung Galaxy devices, they've been what I could kind of call cherry-picking from Cherry.

04:42 So, they've got some hardware, memory, safety features in them which are sort of activated with their particular versions of iOS and Android. And it would just be great for that to become a more ubiquitous thing. And so part of my reasoning behind picking David's talk was to raise awareness of the possibility that's offered by Cherry and things like that.

05:06 And so in terms of looking back and how how we could have been looking forward at the time, getting better hardware protection in place seemed like something worth advocating for. You know, it kind of runs through Alex's talk about, you know, the layers that that we stand on. You know, Alex didn't talk a great deal about their day job at Adira in terms of what they're doing there to protect containers, but I think being able to anchor what we're doing in software down to the hardware, which is what they do at Adira, is

05:39 going to become all the more important. And what we've seen since with Mythos and Glass Wing and, you know, now kind of what we could almost refer to as the AI lab leaks, you know, these supposedly agents run a muk starting to break into services like hugging face is really an escalation in the exploitation landscape. But I think there's also potentially a very helpful side of that because there's ultimately a finite number of vulnerabilities and so we have the opportunity now to use these things to help us find them

06:18 all and rid ourselves of them all. So if I look back 20 years, there was a brilliant paper presented by uh Microsoft research use security about does software age like milk or wine and it was building statistical models of the latent vulnerabilities in software and how these got discovered and potentially exploited and you know from that you could actually build a model of how many vulnerabilities do I think would be in a large codebase that I've probably not found yet.

06:51 And I think we're actually potentially on the cusp of being able to take that kind of knowledge and say, "Okay, let's grind my way through." And if I look at real world projects like curl, it feels like they've been on this journey for a while. And we've seen that journey accelerate for a bit in terms of the AI tools have helped them find a lot more vulnerabilities a lot faster than they might have otherwise done.

07:18 But this can't keep going forever. And so at some point, you know, we'll find ourselves in a situation where there'll just be, you know, fewer vulnerabilities left to discover and that should be a happier place for all of us to exist in because we won't be worrying as much about, you know, zero days and their exploitation. >> Okay. Well, you mentioned Sher and the fact that both Android and iPhones are looking into adopting this model.

07:41 Maybe it's uh we can allocate a couple of minutes to get more details to our listeners and understand what Sherry actually is. Maybe we inspire other folks to adopt it. >> So Cher's a project that's kind of emerged out of research at Cambridge on providing hardware memory safety. And so if we we look at a whole kind of class of memory safety bugs, there's been a big push over recent years to to start using more memory safe languages.

08:10 And and so you know, one thing we hear is, "Oh, just rewrite it all in Rust." But if you look at the size of the problem, there's something like 5 a half billion lines of open- source C++ and something like 70 million lines of Rust at the moment. And so if we were to plot our course of how long it would take humanity to rewrite all of that C and C++ into Rust, even with AI assistance, it's almost an impossibly long duration.

08:42 And we've also seen projects where rewrites in Rust have introduced new logical flaws because, okay, you get memory safety, but it doesn't mean that everything's invulnerable. You still have to think through the other ways that systems can be attacked. you know, the rewrite of some of the core utils into Rust has unfortunately thrown away, you know, decades of hard one experience about what can go wrong with applications.

09:09 You know, you end up with functionally equivalent replacements, but that have some new bugs in them. They might not memory safety bugs, but they're still potentially security problems. What Cherry does is say, you know, what if I could catch out of bounds errors? you know, the kind of things that kind of cause buffer overflows, etc. in my hardware and raise an exception out of the hardware.

09:33 And then I can take all of my existing C and C++, do a recompile to make it cherry aware, and that should be pretty much all I need to do. Some cases there's a small amount of, you know, rework involved, but you know, the research so far suggests that that's a fairly light load. And so we can now have applications that are unsafe in terms of the language that they're using, but the hardware is ensuring that the safety is there.

10:00 So we we don't have to do this, you know, potentially huge scale rewrite into Rust. And so getting memory safety from the hardware is ultimately going to be a lot cheaper than redoing all of the software. Given the points that you you had having it in uh Android and iOS will have a very wide span given the the UB code nature of the mobile phone. If it it would be to have a magic wand and you would should pick another platform that would have it tomorrow which one would that be?

10:29 So it would be risk 5 and I say that because risk 5 is still nent as a instruction set architecture in terms of its adoption. It's still standards wise a little bit more malleable than the x86 and the ARM stuff that that we've all become accustomed to. And I think having the cherry instructions in the Android profile for risk 5 would be something where we would see the the emergence of an entire new class of devices and it wouldn't be the high-end ones like the flagships that presently have these memory safety features.

11:07 uh it would be entry level and mid-level phones and tablets and stuff like that that would come with memory safety and I think that would then drive memory safety ubiquity. you know, Cherry kind of just missed the boat to be able to do that with ARM on Android, you know, first time around and there was potentially an opportunity in the past to push it in, but uh that opportunity wasn't taken and we've kind of got another swing at it now with risk 5, but I think if we did that with risk 5, that approach would very

11:38 quickly find its way into the mainstream for everything. >> Yeah, probably given the open source nature of risk five, we'll have maybe hopefully a better blast radius but fingers crossed that we get closer to having a more memory safe ecosystem. >> Yeah. >> Based on the things that happened in the last time and the points that we heard during CUCON, what do you think will be the minimal requirements or the recommendation for people responsible for security in companies these days?

12:14 Because things change a lot. This touches very well upon Victor's talk and Sarah's talk. Victor was talking about software bill of materials. As you mentioned, you know, that's due to become a part of the European Union Cyber Resilience Act. That act, the CRA is now in force in terms of breach reporting requirements. So that that took effect just a few days ago as we're recording this.

12:36 The requirement for software bill of materials takes effect next year, next December. And what that means is anybody selling software or things that contain software in the European Union is going to have to have a software bill of materials that goes along with that. And that really I think achieves two things. One, it provides evidence that an organization making stuff you know with software is paying attention to their security.

13:05 they you know they they have to pay a certain amount of attention just to generate the software bill of materials and the software bill of materials becomes in sort of evidence for that. The other part of it is that once you've got that software bill of materials in hand you can use that with other tools to determine your exposure to vulnerability. It was really Josh Corman who started this off a few years back where his idea was to make software bill of materials mandatory for the federal government in the US.

13:33 And the notion was nobody would want to sell known vulnerable software to Uncle Sam because obviously if you're saying hey I've got my thing and here's the software bill of materials but you can see it's got a whole bunch of vulnerable dependencies in it then the buyer of that is going to say I don't think I'm happy with that. So you know either you're going to fix that or I'm going to pay you a lot less money.

13:58 And if that applied to federal government, federal government's a large enough buyer of things that it would become a kind of community benefit that that we would all benefit from that. And that sort of happened with the executive order around software bill materials in the US and some of the work that was done by scissor after that. But I feel like the EU has really picked this up and run with it now.

14:19 And CRA is the thing that's actually going to make it happen on a wide scale. That sort of dovetales into how Sarah was talking about, you know, how we do governance and and how we can build into our delivery process the mechanisms by which we're evaluating security and producing evidence of security. You know, if we look at a continuous delivery pipeline, one of the things I can do in a continuous delivery pipeline is I can create my sbomb.

14:51 I can have a bunch of salsa attestations about how the software has been built and I can do sort of automated processes like open source security foundation scorecards. But if I use a kitchen analogy of that, it's kind of I can show the customer of that software that we've used the right ingredients that we've prepared the meal properly and that the kitchen has been hygienic throughout the process.

15:15 And so providing that evidence of security is I think a you know good thing in the overall software supply chain. But the point that Sarah put across there is so much easier if you build this stuff in. So you talked about security being an afterthought and I think it often has been but if we do the whole shift left thing and build security into how we make our applications it's not actually a massive amount of effort and once it's built in there then it's a continuous thing that that we can get evidence out of.

15:54 That sounds quite quite right. And looking at it, it's just a matter of choice these days because things got a lot cheaper with AI. But do you see LMS as a opportunity or or other threat in this ecosystem? >> It has to be both. A bad guy with an LLM is a threat because they can do damage quicker and a larger scale than they would have done without. But LLMs used defensively is also an enormous opportunity.

16:23 And so, you know, what we're seeing with things like Project Glass Wing is the opportunity for defenders to be able to use the models to evaluate their software and discover vulnerabilities hopefully ahead of the attackers. And I think you know more broadly we're seeing LLMs being deployed to do whitebox testing. So internal uh source code analysis where in the past a project might have waited until it's just about to deliver and then done a pentest and maybe somebody would have done some source code analysis at that

17:00 point and now people are just asking their programming assistant can you run a security evaluation of this thing and I'm seeing it becoming more common for that to be a standard part of people's process and delivery practice and I think that's going to be ultimately very helpful. >> One of the points that just thinking about how the track looked like one thing that missed that it's still seen as futuristic is post quantum cryptography.

17:27 That's more or less your day-to-day business. What is that in terms of evolution at this point? >> I didn't pick a talk on postquantum cryptography for for QCon this year and I've been asked to to do the track again for next year. Almost certainly there will be at least one talk on postquantum cryptography and I think it's sort of like a timing thing.

17:52 ESBOM was clearly something that uh we needed to start talking about this year because of the CRA mandate becoming so close. PQ is something that we need to talk about next year because Qday is getting close and actually if we look at the forecasts on on Qday which is this notion of the date by which we might have quantum compute capability emerge that would be able to implement Shaw's algorithm and hence break the asymmetric crypto that we've been using with classical algorithms such as RSA and elliptic curves.

18:24 This stuff's stepping up closer and closer. I did a talk myself about this just um a few weeks back and planning another one and the title for that other one is sort of you know we're trying to build the house at speed on wobbly foundations and so the thing that we've been finding with postquantum implementations is the standards for the cryptography itself the algorithms are settled so there was a n process uh winners emerged and and all of that's been standardized but the implementation landscape is very immature and

18:57 So things like library availability, OpenSSL now has the implementations for MLDDSA and ML Chem, but you will only have that OpenSSL if you're on the bleeding edge distributions. And so if we look at an enterprise landscape, they're probably not on those distributions yet. And so you're talking about having to migrate stuff to the latest distros just in order to implement your postquantum initiative or or potentially backporting stuff which again kind of then gets a little bit messy.

19:30 But I think one of the things people are grappling with at the moment is simply knowing where all of the cryptography in their organization is deployed and all of the things that they they have to touch. So, I can see this being a bit like a rerun of Y2K where first of all, we need to track down all of the places where we've got these problems and then we need to grind through the process of updating our software to be crypto agile and putting the new algorithms in.

19:58 And our focus has been very much on crypto agility because there's a lot of distrust associated with the new postquantum algorithms and I think that comes about by the fact that so many of them failed during the evaluation process and then there's one of the latises has succumbed to a sort of brute force attack using anthropics AI tools and so everybody's kind of sideeying the algorithms that have emerged from this process.

20:28 and going, "Well, are they going to hold up in the long term?" And I think we have to imagine, "No, they probably won't." Uh, and so we could be doing all of this again in a few years. And so, we don't want to be baking the new algorithms too tightly into anything that we do now because we're probably going to have to replace them again. >> Yeah, you you touch on on a very sensible point from my perspective, running multiple distros or multiple versions in parallel.

20:56 And one of the things that uh I wrote I think in the last week a researcher found out that the way playing out with GPD6 cyber allowed him to just circumvent his whole virtual machines infrastructure and then use zero days or the fact that some of the known bugs were not backported to LTS's older LTSS even though they were supported. What would be I don't know short list of uh not the regular Joe but people that are not on the on the bleeding edge of technology.

21:30 What should be the things that we should care about in terms of I don't know the Linux distributions patching or using the latest uh security releases. >> I I think most fundamentally we need to think about how we establish trust and where our trust anchors are coming from. You know, one of the doomsday scenarios that's been sketched out for this is the private keys for the certificate authorities that we use as kind of the basis of much of the trust, especially in the sort of public internet sphere could be reverse

22:04 engineered. And with those private keys in hand, then I could start pretending to be, you know, let's encrypt or Google or whoever else and issuing certificates that nobody would be able to tell were the wrong certificates. Now, actually, people would be able to tell because we've got a certain amount of certificate observatory stuff going on and Moxy Mullins's done some amazing work in in that area.

22:28 But one of the things we're waiting for here is for the certificate authorities and those roots of trust to actually move to postquantum so that they're ready. I think this has been part of the move that's going on at the moment to reduce certificate lifespans. So we've seen certificate lifespans kind of going down from a year and heading towards uh 90 days and then 45 days.

22:50 You know, in terms of organizations thinking about this, some of this is we're being dragged along anyway. You know, whether you think about it or not, your certificates are going to need to go down to these new issue deadlines, and you probably need to have automation in place to to cope with that. How that then relates to things like being on more recent distributions and achieving patching and stuff like that, it's all important.

23:22 And I think this comes back to to Sarah's talk and and the governance around it. It's so much easier to do this if you've got straight through processes where you can achieve continuous delivery versus the more kind of handc cranked ticket driven processes that many organizations unfortunately are still stuck with. And so achieving that shift is a benefit that's going to keep on paying back.

23:49 This also relates to some of the AI stuff we've been talking about in terms of we're seeing massive patch bundles at the moment. So the last Microsoft patch Tuesday I think was somewhere around a thousand patches in a single bundle. If organizations haven't already been able to catch up with simply being able to get the Microsoft patch Tuesday out as soon as possible once it hits then they're potentially in trouble because that as soon as that thousand bugs is out there then you know people are using AI etc to be able

24:24 to come up with exploits to to go after those. And so one of the things we're dealing with is the shortening of that window between, you know, knowledge or even rumor of a vulnerability and the exploitation of that vulnerability. >> Yeah, it's probably it will be it's worth mentioning another another point. We looked into AI being a threat. We looked into AI that can be a help into patching quicker, but also obviously as you mentioned this acceleration means a lot more bugs out there that can be used for other stuff.

24:56 But the other day security researcher I miss his name now he found a zero day in Muse the meta application and what was interesting about it it's again the the human in the loop is the problem in terms of security because what happened is that you have Mac OS that built a quite decent way of putting privacy in place in in Mac applications in Apple applications and now we are just giving them access we give agents access to our file system to our privacy settings to just be able to help us in terms of doing chores for

25:32 us and that's that's a whole different uh perspective into we have bulletproof security but we are just giving the key away for our comfort into I don't know small stuff. >> So this has been a festering problem long before AI but AI is kind of rubbing our faces in it. As I prepare for next year's security track, I reached out to all of this year's speakers and said, "Who would you love to see talking on next year's security track?"

26:02 Victor got back to me almost straight away. And he was like, "I want to see a talk about AI identity." and it's something that um he's he's actually started writing a little open-source project about. But we've always talked about a notion of least privilege that we should give people just the privileges that they need to do their job. And actually, you know, this notion of non-human identity has existed since way before AI.

26:31 And if you look in most organizations, they've probably got something like 10x more non-human identities than they have people working in the organization. And of course, that's exploding now because of agents. It's not a new problem, but it's something that I think has often been overlooked and, you know, a little bit brushed under the carpet. And so absolutely we should not be giving our AI agents our keys to the kingdom and saying you know off you go do some stuff on my behalf.

27:04 We should be very carefully controlling taskbased fine grain permissions to do the thing that needs to be done and no more with all of the appropriate um auditing and surveillance that should go along with that as well. And I think what's actually happening here is stuff that has been, you know, an air quotes enterprise problem and solved to a certain extent by enterprise identity management and the associated platforms that go with that is now becoming an individual problem.

27:37 It's a hairy one to manage and we don't want to be kind of manually thinking about this stuff. And so we now kind of get into the sort of aura burough thing of we need an AI to help us manage the permissions that we're giving to our agents because otherwise it just becomes unmanageable. >> I don't know if it's fair to ask you to provide more teasers about next year but for me two points uh uh well you mentioned one and that's postquantum computing or post quantum cryptography that's something that you have on your mind.

28:11 you did mention um some kind of security net for agents. Is there anything else that you would like to tease? >> I think there will be some more low-level talks. So, there was a talk that we had lined up where the speaker couldn't actually make it this year. So I'm hoping that they're going to be able to return and we will get the newer better version of their talk which will be examining some of the lower level things that we can do to build security in and I think that applies in terms of you know humans and agents

28:41 the working title at the moment is about you know how do we defend systems from humans and agents and I think actually there's a common convergence there that the attack surface area is the same whether it's a human or an agent. The nature of the attack and the scale of the attack is probably very different between human and agent. If you look at the postmortem of the hugging face incident, it was described by their size as I think like having 10,000 drunk house robbers show up all at once.

29:17 And you know that sounds like a very chaotic scene, but I think that might kind of perfectly encapsulate where we are at the moment. And even if each individual attacker in that case isn't particularly capable, just the the sheer scale brings a whole bunch of problems on its own. So I I think I'd like to get a talk that sort of looks at that aspect of what we're confronting, but I want to kind of ground it in the reality of if you're doing a good job of defense, then that's going to defend against the human attacker,

29:57 the AI agent attacker, and the sort of human AI hybrid. And so getting that stuff right is always going to be you know the the sort of helping yourself out way to go. Okay, just to summarize the the points that I think I heard going to lower level might help us as as an industry given that that will provide a better foundation and hence the ripple effect will be a lot um broader probably and help different points regardless of if we are discussing about agents or humans and best practices still hold if we are doing our

30:33 homework most often than not we are closer to safety than uh than not. >> Yes. Okay. Is there anything else? >> I I feel like we didn't talk much about Andy's talk. Andy's talk kind of sat in the middle between, you know, something like Sarah's talk about the the high level governance or something like David's talk about the low-level hardware implementation.

30:57 And yet Andy's talk was emblematic of we need to pay attention to how we're building things and the abstraction layers that sit beneath us. And so I'll just kind of return to it because that was the overriding sort of lesson that we have here with security that it's not just the software that we write. It's the entire stack that we have to concern ourselves with.

31:26 You know, there is absolutely stuff that you can do to make sure that you're on a good stack. You know, some of that is about making wise choices in terms of languages and frameworks. Some of that actually can be by using modern hardware assistance. And a lot of that comes to ultimately how do I pay attention to this? There's not enough human bandwidth to pay attention to it all the time.

31:54 And so I have to systematize that attention. And that kind of takes us back to what Sarah and Victor were talking about. But Andy's lesson was don't be just blinkered about the software that you're writing. Look more broadly at the ecosystem that you're writing in and make sure that everything has been properly considered. >> So governance, we shouldn't run away from it.

32:21 the sbombs we need to take care of um maintenance and who's responsible for that and at the points we need to take care of that if not for our sake because it's we are legally obliged especially if you're doing business in the European Union nowadays and then it's a lot of things happening behind the scenes uh on the hardware level in the abstraction levels so we have to pay a lot of attention these days on security because it's getting closer and closer to us and as you said even if the agents are not particularly

32:52 skillful. There are a lot of them and that might create uh new breaches that we weren't aware of before this. >> Yeah. And of course, today's agents are the worst they'll ever be. >> Yeah. >> The next iterations of those attacks will be smarter, less drunk agents and potentially even at larger scale as well. So, um we need to prepare ourselves for that.

33:20 >> Thank you, Chris. Good luck with preparations for the next QCon. You did mention and I have the same feeling that this year's QCon London was one of the best. Good luck in making it even better for next year. >> We're always trying to make it better than it was last time around. I think 26 was one of my favorite QCons ever. Some amazing speakers who absolutely smashed it.

33:41 It was just wonderful to um be able to spend some time with them. But uh, of course, we're always trying to raise the bar, so hopefully 27's even better. >> No pressure. Good luck with that. Thank you. >> Thank you.