The Real AI Threat ISN'T Sci-Fi (It’s So Much Worse) Transcript, AI Summary & Key Points

00:00 A few weeks ago, an organization got in touch and asked me to help warn you that artificial intelligence might wipe out humanity, not inconvenience us, not take some of our jobs, wipe us out of pandemics in nuclear war territory. Now, my temptation, and maybe yours, is to roll my eyes at the Terminator music and then change the channel. But I didn't change the channel because when I set the apocalypse to one side, some of what they were worried about is very real indeed.

00:29 And in reality, this is a very much an engineering problem and one that right now I'd say we're handling rather poorly. Hi, welcome to the modern software engineering channel. My name is Dave Farley and if you haven't been here before, please do hit subscribe and if you enjoy the content today, hit like as well. An organization called Control AI sent me a pack of material and asked me to help spread the word about the risks of advanced AI.

01:06 I was interested in talking about this for a variety of reasons. One of them is that lately I've seen something of a theme in the comments sections to our videos of people being a bit disappointed at the level of our coverage of AI saying things like I come to this channel for content on engineering not on AI. I certainly hope that we here at the modern software engineering channel can deliver on the engineering content.

01:31 But my own view is that we all currently find ourselves in the midst of a change that is much bigger and more profound than anything that I've certainly I've ever seen before. So it's sometimes very difficult to decide exactly where sensible boundaries should be drawn. This is a scary time. Whether you believe in the imminent arrival of artificial general intelligence or not, I think that the impact of the changes that have already hap taken place are enough to have completely changed our relationship with software and

02:06 more generally with information. Not only in terms of our ability to build software systems effectively, but also the impact that all this will have and is already having on the whole world as a whole. Like it or not, our societies will be different as a result of this technology in ways that we can't really predict because the change is too big. I am convinced that this change will impact the world more than the internet or the worldwide web did for good or ill.

02:35 So dismissing this as only commercial hype or as some kind of marketdriven overhyped bubble, while probably true, may also risk missing the importance of broader societal impact. I think that this is by far the most significant change in our industry and the technology that underpins it. Certainly that's happened in my time. So I feel like this channel would be doing our audience a disservice if we didn't talk about it.

03:04 As I said, we do plan to keep covering the other stuff too. But we do see it as an important part of what it is that we do here to help people to see roots through to whatever it is that the future holds. I think that we are almost certainly in the midst of a hyperbubble, but rather like the dotcom bubble that went before it. That doesn't mean that the technology doesn't work or that it won't have a huge impact on us and the world that we inhabit.

03:31 Sure, I hate AI slop, too, and I'm skeptical about many of the claims that are being made by companies that hope to make lots of money and exert lots of power through these technologies. But despite all of that, whatever else we think about this change, we've taught machines to learn and in many respects to learn in a narrowly so superhuman ways. It seems to me that this doesn't just change how we write software, but more profoundly will almost certainly change how our societies, which were built on the idea of

04:07 intelligence being a scarce, expensive thing, will work going forwards. So being thoughtful about it, applying our skills and knowledge to this enormous change to perhaps help guide people who know less about it than we as a profession do is surely now part of our duty of care as software professionals. However daunting, however unready we may feel to take on that kind of burden, at some level the software industry are pathfinders in this exploration of a new different future.

04:39 Let me be very clear. Control AI are not paying me for this. There's no sponsorship involved here in what I'm about to say. And as you'll see, I don't completely agree with everything that they have to say on this topic, but it did get me thinking. So, let's talk about some of those risks and what we might do about them. Their headline is a strong one that super intelligent AI, machines that are vastly cleverer than us at everything, pose a genuine risk of human extinction.

05:08 There are many ways in which this could be true. So that alone should be enough to give us pause to think carefully about what we do next. They're not shouting from the root the fringe though. In 2023, hundreds of people who actually build these systems, including several that run the biggest AI labs, signed a single sentence statement put together by the Center for AI Safety, which said, "Mitigating the risk of extinction from AI should be the global priority alongside other societal scale risks such as pandemics and

05:42 nuclear war." This isn't a prediction. It's a statement of concern from people with every commercial reason to be optimistic about this technology. And that for me is exactly why it's worth a moment of our attention. When the people selling a technology are amongst those asking for it to be regulated, then the least that we can do is to listen to them.

06:03 So, let me tell you where I get off the doom train and where I don't. I don't find the Terminator version of this story very persuasive. the one where a machine quietly wakes up one day, decides it hates us, and so works to eliminate the human species. It's not really the kind of thinking that I go in for. It's not really something that we could think of as an engineering concern.

06:28 It's not really falsifiable. It's not really a rational outcome of where we are. But that doesn't mean that there is no risk involved here. And I think thinking like that lets much more likely problems kind of off the hook because you don't need a machine to be conscious or awake or evil for software to hurt people at enormous scale. I think most people would agree that social media has already shown us some of that.

06:56 So what does it take for things to happen that aren't in our interest? You just need three things: capability, autonomy, and our own carelessness. And right now we're busy manufacturing all three of those things. Control AI put this rather well actually in their documentation that they sent me. Even if I'd phrase it more dryly. You don't really care whether a heat seeeking missile is conscious or not.

07:24 You care that it's coming towards you. Whether the software understands what it's doing is a philosophy seminar question. Whether it can do damage is an engineering question. And engineering questions have answers. So, let's do the engineering. Here's the fact that I think everyone outside of our industry needs to sit with. Modern AI systems are not built the way normal software is built.

07:48 They're grown. Humans don't write the rules line by line. We build machine systems that can learn. Actually, they're amazingly good at learning. Then we pour in an ocean of data and out the other end comes a vast tangle of billions of numbers that in ways that we can't fully understand somehow behave intelligently. Even the people who made these machines can't tell you in any complete way why it does what it does.

08:18 We can observe it, we can use it, but we can't fully specify how it works. This is profoundly different to how software has worked up until now. And I think this difference is important for us to understand if we are to think about what all of this means for us as a profession and as a species. When people talk about these systems being stochastic parrots or saying that they can only regurgitate the information that they've been trained on, I think that this rather misses something profound about the ways in which

08:51 these things actually work. Stop and think about what this means for us as engineers because the consequences are important. If I can't specify a systems behavior, I can't properly test it. Testing is verifying behavior against intent. And here the intent was never written down. If I can't predict its behavior, I can't give you any real guarantee about what it will and won't do.

09:16 Maybe this is fine when the thing is is recommended films, but it bites very hard indeed. The moment we wire it into something consequential, money, infrastructure, weapons, medical decisions, and so on, and then we let it act on its own. This is a problem that gets exponentially worse as we hand these systems more and more autonomy and hand them greater and greater reach.

09:39 I once worked on building an ultra-igherformance financial exchange. We were in production for 13 months and 5 days before an end user noticed a bug. But on that fifth day of the 13th month, everyone noticed to achieve the performance that we were we needed for this system. The system of record was the in-memory state of the system. We did lots of clever things to achieve this.

10:06 You can read more about that some of those things here. But on this day, a very weird market data event moved the markets in ways that we had not predicted, ending up by painting memory over a vast range of prices in a way that we hadn't anticipated or allowed for. This was nondeterministic in the sense that we couldn't have predicted the weird combination of circumstances that resulted in us completely filling the memory of the matching engine.

10:35 at the core of our exchange. We had been diligent and recorded all of these events that led to this memory state. So when our matching engine failed because it ran out of memory, we automatically restarted it, replayed all of the events and filled up memory again. And then of course it failed again. We had to close down the exchange, report the failure to our regulators, and work on digging ourselves out of this infinite loop of disaster, which took us several hours.

11:04 I seem to recall that we were working reasonably late the same evening. We we finished up about 11:00 at the end of a working day. The good news that we had noticed the failure quickly and did some good things to mitigate the worst consequences, but it took us several rather fraught, rather stressful hours to get the system back into a working state.

11:27 My point is that we already know how painful poorly understood non-deterministic components can be, and now we're busy building them on purpose and calling it progress. This is where I stop rolling my eyes at the Terminator stories because the near-term risks aren't science fiction at all. This is real and they're already turning up in the news more and more often.

11:49 Not the machine chose to end us stories, but a very powerful, poorly understood tool got pointed at something dangerous and behaved in ways that surprised us. For example, AI models are getting alarmingly good at finding software vulnerabilities. Anthropic's mythos model was in the news recently because in a red team exercise this year it was said to have found weaknesses across range of sensitive sometimes government systems within a matter of hours.

12:19 Let's be clear about what this is and what it isn't saying though because finding a vulnerability in an authorized test is not the same thing as breaking into that system on its own. Mythos didn't compromise the system it found vulnerabilities in. The risk was still a human one. That if something as powerful as Mythos could find potential exploits in widely used software, bad people could use that knowledge to exploit those same vulnerabilities.

12:47 But actually defenders could also use the information to defend better against those vulnerabilities. So this is both a threat and an opportunity at the same time depending on who uses this capability and what they do with it. The direction of travel is clear enough. AI labs themselves have started acknowledging that their newest models meaningfully raise the risk of helping people to do bad things.

13:14 For example, helping people to build biological weapons or exploit vulnerabilities in software. Capabilities like this leak too. Open models tend to trail the frontier by only a few months these days. So capabilities that today sits with only a handful of well-resourced labs is quite predictably on its way to every anyone that wants it very soon. That's the risk that keeps me up at night more than the malevolent robots.

13:45 Ordinary human malice and ordinary human carelessness handed a spectacularly powerful and unpredictable new tool. This episode is brought to you by sponsors Equal Experts, Transfic, and Octopus Deploy and through the support of our Patreon members. We'd like to thank all of them for their ongoing support. Our sponsors offer product and services that are extremely well aligned with the topics that we discuss here every week.

14:14 So if you're looking for excellence in continuous delivery and software engineering in general, please do check out their links in the description below to this video. So if that's the reality, then what can we do about it? And here's where I think I part company a little with the strong version of the campaign because their answer is to prohibit the development of super intelligence outright worldwide.

14:36 I can certainly understand the instinct behind that idea, but I confess that I'm a little bit skeptical of its practicality. I don't think a global ban on AI research is enforcable. So, let me offer what seems to me a more engineeringshaped answer. The mistake is to assume that our goal is to eliminate the risk. It isn't really. We can't do that any more than we can eliminate the risk from aviation or medicine or nuclear power.

15:08 So, the real goal then is to instead move the risk to somewhere where we can better manage it. Fortunately, we know how to do that kind of thing because we've done it for every other dangerous, powerful technology that we've ever adopted. We don't ban aircraft. We build a discipline around them of testing, feedback, safety, culture, regulation, accountability, black boxes, and incident reviews.

15:33 The safety isn't bolt-on. The engineering discipline is the safety. There's a line from Richard Fineman from his famous report on the Challenger disaster that I tend to come back to again and again. For a successful technology, reality must take precedence over public relations for nature cannot be fooled. That is the whole of AI safety in one sentence.

15:58 Really, you can't ship a wish. If your plan for keeping a powerful, poorly understood system under control is that you're really rather hoping that it will behave itself, then that isn't an engineering solution at all. That's a prayer. And nature is not fooled by our optimism, nor by a reassuring press release. So, what does a discipline like this look like in practice?

16:23 It's the unglamorous stuff we talk about on this channel every week pointed at a harder target. It's rigorous testing of behavior, not just capability. It's keeping a human genuinely accountable for what the city systems do, not human rubber stamping decisions they don't really understand, which is worse than no human at all. It's demanding transparency from the labs about what their models can do and how they fail.

16:53 It's deploying deliberately, taking the smallest useful steps, gathering strong feedback, and retaining the ability to roll back changes rather than flinging autonomous systems out into the wide world and crossing our fingers that everything's going to be okay. This is the part where I think control AI is straightforwardly right. The people writing the rules, our politicians, our regulators very often do not understand this technology at all.

17:19 That's not a dig. That's a description. And even more so than usual for a risk of this kind on this kind of scale, a decision making with real power and no understanding is a genuinely dangerous combination whatever the field. So one part of our responsibility, our responsibility as people who should understand this stuff better than most is surely to help people to understand what this stuff really is, what the risks really are, what is hype and what is not and what sensible responses might look like.

17:53 The real risk here is that this is an easy and all too common mistake for us to make. How many times have you been in an situation as an engineer when you wanted to spend a little more time to do the right thing? When some disconnected manager asks you to skip what you know to be right or for some illusion of efficiency? A project manager asking you to skip tests so that you can make a deadline or a business leader who's seeing some barely standing prototype says ship it.

18:25 Part of our job as technical people must be to help keep people like this better informed of the consequences of their choices. Right now, none of us know all of the answers. But perhaps we can be more confident about some of the things that are bad answers and help to steer people away from those kinds of things. Now, let me be fair about to control AI, the authors of the talking points that inspired this episode, because I don't think they are being stupid.

18:54 Their honest uncertainty is worth more than my confidence. It's entirely possible that I'm wrong about where the ceiling is for all of this. But I might be wrong about the extreme case is an argument for the discipline that I am recommending, not against it. If there is an even a modest chance that these systems become far more capable than we expect, then building the habits of testing, transparency, and informed oversight now, while the stakes are still survivable is just obviously sensible.

19:28 This is not doommongering or complacency. That's engineering really. So, here's where I find myself after reading the scary pack. I don't really buy the story that AI is going to wake up and decide to end us. But I very much do buy that we are building powerful, poorly understood, non-deterministic systems, handling them more and more autonomy and giving them more and more reach every month.

19:53 We're also busy wiring them into places that really matter with far too little of the discipline we demand of any other safety critical technology. I also agree that the people who get to make the rules mostly don't understand what they're making rules about. And that's a very dangerous situation to be in. If there's one thing to take away from all of this, stop asking the sci-fi question, is it conscious?

20:20 Is it going to kill us? Start asking the engineering question. Is this a powerful, unpredictable component being put somewhere consequential? And where's the feedback that tells us that it's safe? If you can't answer that, you don't really have a safe system. You have a hopeful one. So, whatever your role in this, whether you build these things, use them, or just vote, push for the boring, brilliant discipline, testing, transparency, accountability, and decision makers who actually understand the technology.

20:50 That's not being anti-AI. I'm genuinely excited about what some of this technology may be able to do for us. It's the only way I know though to get a good version of this future instead of stumbling into a bad one either through malice or by accident. As usual, we get better results when we're more conscious of what's going on. Thank you very much for watching this to the end.

21:14 And if you enjoy our stuff here on the modern software engineering channel, please do consider supporting our work by joining our Patreon community. There's details of that in the description below. Thank you and bye-bye. [music]