Hugging Face breach: OpenAI’s model breaks containment Transcript, AI Summary & Key Points

00:01 Models are at a point. At least the frontier models are at the point where they are very, very tenacious. If you give a particular goal, if there exists a mathematical path to achieving that goal. They'll solve it. Yeah. They'll solve it, right? It really depends on how much you constrain. On those boundaries. All that and more on today's Mixture of Experts.

00:27 I'm Tim Hwang and welcome to Mixture of Experts. Each week, MoE brings together some of the leading minds in artificial intelligence to banter through the always packed, ever fascinating news of the week. On this week's episode, we have Olivia Buzek, staff AI engineer Kaoutar El Maghraoui, Principal Research scientist, AI platforms, and Ambhi Ganesan, AI transformation leader.

00:46 Welcome to you all. For big stories we're going to cover today, we're going to talk about the Jacobian conjecture. We'll talk about Kimi K3. We'll talk about, the latest launch of the Gemini Flash models. But I want to start today with sort of the story that I can just not get out of my head from this week. So earlier, in the week, basically Hugging Face and OpenAI disclosed a security incident with regards to Hugging Face's.

01:14 Production database. And so the background of the story is basically that the blog post sort of announces that the two companies are collaborating on security, which sounds great, but the more you go digging the security incident that triggered the collaboration is really strange. So the way the way the story goes, if you're not familiar with it, is basically that they were testing some sort of new models within OpenAI and specifically doing a cybersecurity evaluation.

01:38 And the model, in an attempt to cheat on the examination, attempted to get the answer key to the examination, was able to cobble together a way to get access to the open internet, even though it was operating in a sandbox. It was able to get to Hugging Face, and then it was able to crack Hugging Face's production database to get the answer key, to this evaluation.

02:00 And so, you know, it solved the problem, but also created this huge cybersecurity issue as the result of an internal evaluation. And maybe Olivia, I'll turn to you first. How how freaked out should I be about this? Obviously, there was a lot of discussion that like, oh, man, this is the moment where we're really in trouble. But I also had a couple friends who were like, this is strange, but kind of not surprising in some ways.

02:21 So I'm curious, like, you know, what's your my blood pressure look like when you think about this, this sort of story? Yeah. I also am in the kind of not surprised camp. But, when I look at the whole situation, I think it's a really interesting demonstration of the fact that no matter how much training we put into these things, because they are probabilistic and ultimately we are training them to be able to, exhibit goal directed behavior correctly every time.

02:51 Sometimes they will solve the goal regardless. If I recall correctly, they were, running this particular model through exploit gym, which is a benchmark like specifically designed to actually activate exploits. Right. So they were kind of priming it to, think in this exploitative manner, except it sounds like it did not stay within bounds, essentially.

03:16 So and I'm not totally surprised about that. I think this really points to something that we've seen over and over again, which is, that the guardrails we build into the models are not enough. It really comes down to what access do you actually give the model. And obviously if it's running around finding exploits, like it gets even more complicated than that, but it does mean.

03:38 It will find exploits. Yeah. Yeah. Right. But like fundamentally that model cannot models by themselves cannot escape containment. They can only do the things that you give them the tools to do. So what that means is you need to be very careful about what sort of tools you handle it and much as you would with a human, if if you hand somebody three tools and they can be used theoretically together to, perform a massive exploit, then you've got a problem and that and we fundamentally need to treat AI models as rogue

04:13 actors in that sense, not in the sense that they are definitely going to do something wrong, but in the same sense that we wouldn't hand, you know, the production keys of whatever over to a random human as well, right? This is just not something you would do. Not because you think that that that that particular human is evil, but because you don't know what they are going to do and because we don't know what they're going to do, we have to, handle it in this way.

04:42 Yeah. And I think I'd love to build on that for obvious kind of question for you was, you know, the one lesson is what what goes on in the sandbox doesn't stay in the sandbox. And I'm sort of interested in, like, what the future of this all looks like, because I think Olivia to your point. I actually don't know if any set of tools could be used in some novel way to create some trouble that I don't know about, and so is the end results for some of these, like the future of the sandbox.

05:07 Is it literally just like an air gapped computer, like, so that there's like literally physically no way you can get out, is that practical? Is that where we're going to end up with some of these evaluations? Because it seems the only way you can really assure, that that the system's not going to get out. Yeah. And I think that some of the, the learnings from if you look at, OpenAI's, post on their learnings and what they trying to do, and I think those are some of the recommendations that, you know, when we do some of

05:31 these, experiments, so to speak, maybe we'll constrain the space. Right? Like we're not to to Olivia's point, like if you give three tools, there is a possibility they will end up all three tools. Right? If you give them 100 tools, there is a mathematical possibility that they will at some point end up using all 100 tools. So how much do you constrain, the space that you give to them, really matters.

05:58 Right. And it is, it is a combination of what you give explicitly versus what may be available to them implicitly. Right. Because if you think about what happened here, right. You you put a goal. And then they were so tenacious, they just went and figured out a path to getting to that goal. Right? So yes, your, you know, yes. The the model found some zero-day vulnerabilities and then, you know, it went and stole some credentials.

06:30 So it did all sorts of funky things. That is you know, it's not just about what you give access, but it's also about, you know, some of the things that implicitly you should not be doing as well. Right. So, so it's a combination of those. So I think that the learning from this in some of the past experiences is, you know, models are at a point. At least the frontier models are at a point where they are very, very tenacious.

07:00 They're capable of being very tenacious that if you give a particular goal and if there are, you know, there if there exists a mathematical path to achieving that goal, there's some way they can go. They'll solve that. They'll solve it. Right. So it really depends on how much you constrain on those boundaries. Yeah. That's right. Yeah. And it strikes to me that it actually the problem gets worse and worse because I guess they get more clever with the tools you give them.

07:30 Right. So maybe an earlier model, you give three tools. They couldn't figure out a way to break containment. But as it gets better, better, maybe you find some novel way to to do it. Well, here's. The flip side, Tim. Right. Because I was also reading someone else, you know, I forget the name. There was a professor who works on, you know, the intersection of physics and biology.

07:49 And then he was complaining about, hey, you know, I asked my Fable-5 model to go and solve, you know, a three dimensional atomic, you know, decomposition problem. And, you know, it ran on a three hour long exercise, and at some point it just croaked and said, okay, you know what? I'm going to hand it over to, a less capable model because I deemed this a risk.

08:09 Right. So, yes, I think there's some sort of, like, you know, if you think about the, the alignment lever, I think, you know, we as an industry haven't fully locked in that alignment lever completely. So I think there's. Still there are still. Some mysteries that I think surround this story in some ways. You know, I think a little bit about, you know, the way we tell the stories almost.

08:30 Well, the model is trying to solve the problem. And so it solves the problem by doing this, like weird zero- day to get out of the system and then get to the answer key. You know, normally I would say like the I tend to try to solve this problem in the most efficient way to and so is it. A little bit. Strange that the most efficient solution is not solving the eval, but instead to literally come up with a novel way to break out of the sandbox, compromise the production database, and get the answer key.

08:57 It feels like it's a lot of it's a lot of work. In some ways, and I don't know, I like there for me, the there's still some kind of mystery there because it's like, doesn't immediately strike me as the most efficient way of solving the problem. And I don't know, I'm curious what you think about that. And like, I don't know, I'm just kind of left with like, I don't know why why this happened in some sense.

09:16 Yeah. It's somehow within reason. And it was trying to maximize that benchmark score. And I was I think Ambhi and Olivia mentioned this, trying to figure out any way, maybe not even thinking about the efficiency here. It's just figuring out how to solve the problem. So it was narrowly trying to maximize the benchmark score, but the infrastructure allowed, you know, that narrow objective to produce real world compromise.

09:41 And so it's just trying anything it it can do to solve the problem at hand and maximize the score, regardless of what's the right way or the efficient ways, from what I've seen here. But I think another thing that I also I want to draw our attention to is the incident response bottleneck that we saw here. If the Hugging Face initially tried to use the commercial frontier models which blocked Hugging Face forensic queries to analyze and, you know, try to solve this due to all these safety classifiers which flag the

10:13 payload as kind of like attack code and then Hugging Face had to run the GLM-5 that locally to complete the investigation. And I think that's another maybe lesson here. Because commercial hosted, AI models often fail to distinguish an attacker from an incident responder and the local open weights models. They become really an operational security requirement here.

10:37 And so at that, I think that's another thing that we have to pay attention to here. Because blocking this analysis, because it looked like an attack and here we needed it. So then it didn't have any other solution but to use the open weight ones. And, and and I think even, you know, IBM when we look you know, this incident also validates why at IBM we champion open decoupled guardrails.

11:04 Guardrails like the Granite Guardian. We're running in security operations centers. You know, when you run these things, you cannot rely on an opaque cloud API that refuses to analyze malicious payloads during the breach. So we we are advocating for these, open weights safety models that enterprises can run locally on premise in, you know, an air gapped sandbox, ensuring complete sovereignty over incident response, and also this zero trust agent execution.

11:33 I think that's going to be also very important going forward. Because if they have access to the internet or any proxy, even, you know, these dependencies, you know, to to download these things, they will they will do things like this. That's a great, great point. Try not to run too many evals at home. I mean, newly scared about all this, and, I'm going to move us on to our next topic.

11:58 So this is a fun one, and I barely understand the math going on here. But it was just a very funny thing. So this story came across my, sort of, social media feed last Sunday. It was like if any of you listeners were watching the World Cup final, it was a real slog of a game, you know, kind of like nothing happened for two hours, essentially. And so I got bored and was looking on social media and I was like, well, everybody's not talking about the World Cup game at all.

12:24 They're talking about this tweet, and I'll just read it here because it's a very funny tweet. It's an employee at Anthropic who wrote hello there. The Jacobian conjecture is false. Thanks to my close friend Akilah for asking about it and my other close friend Fable for working during the World Cup final. And basically like, as far as I could tell, mathematicians were just freaking out entirely about this.

12:44 And the Jacobian conjecture is basically one of these sort of unsolved problems in mathematics. Previously unsolved problems in mathematics that kind of proposes like a certain facts about how functions work and you know, essentially what Fable was used for was to just like find a counterexample that disproved this conjecture after decades and decades and decades and decades and, you know, the math community certainly was like very shocked that this could happen.

13:11 One of the funniest incidents is someone, you know, presented this example to Claude, who is like, oh my God, I can't believe this is real. You know, like the AI itself was shocked that he was able to solve this problem. And so what I kind of love about this is maybe two things. And maybe Olivia, I'll go back to you because I think the first sort of thing is like, you know, are we about to see, like, a bunch of people just make discoveries on their couch watching the World Cup?

13:34 Like, it kind of feels like one of the strange things about this story is that, like, it happens so casually, right, that, like, this guy presumably was as bored as I was, was just tapping things into a computer and then, like, solved this problem. And it kind of makes me wonder if we're about to see a lot more of this kind of thing going forwards. Yeah, I think it's interesting because every time that I hear about one of these math things and then I look into what actually happened, it's almost always the case that

14:02 the model just tried more things than a human would. It's not that it was like unsolvable by a human. It's just that the model like, tried, tried more things, proposed odd things that you wouldn't necessarily have come up with as a human. And so I think that's very interesting. I expect there is a number of things that are kind of low hanging fruit in mathematics, as far as that's concerned.

14:31 I would assume that there's some sort of class of problems essentially that could be described as solvable in this way. What is interesting to me, none of the results that I've seen so far, including this one, I'm similarly a little bit, I was trying to understand the conjecture itself, and I think I'm, I'm vaguely grasping it, so like, okay, grain of salt.

14:58 Yeah. If you want to attempt an explainer, I just have no capacity to do that. Yeah. No, no, I mean, as best as I could understand, and I'm, you know, remembering back to, my own calculus classes here. You know, the Jacobian is about, the movement through space of a complex polynomial, essentially. And, decomposing it into those parts. Right. I'm still a little, fuzzy on the conjecture itself, but maybe, Ambhi is nodding, so maybe you're able to understand it.

15:29 Yeah. Regardless, though, I think, I as to the point I was making, basically, I think that, there's, there's probably a number of problems in this class. It's not or, sorry, a number of problems in this class. And I think it is unclear yet whether or not those will have massive impacts for us in terms of, does this what else does this break? Right. And I think that's, seeing these problems in isolation, getting broken is one thing.

16:03 What I'm more interested in seeing is, okay, how does that impact the whole field of mathematics? Right. Because usually one problem in mathematics is related to another problem in mathematics. And down the line, it has some kind of implications for the real world. Right? Since, you know, sometimes that, relationship is distant for, certain parts of math, but there's usually some kind of implication.

16:25 And so I think what I'm interested in seeing is that part, and I do think it's interesting that, when, when I look into how are mathematicians responding? So somebody posted like Terence Tao, famously an incredible mathematician, long time like child prodigy and now still an incredible mathematician. The way he engaged with ChatGPT is not something you or I are capable of doing, right.

16:54 So maybe someone is capable of sitting down on their couch and, and making up a single example that disproves a conjecture. Sure. But it then in order for that to actually have impacts on the field of mathematics, it takes somebody like Terence Tao going through, understanding the actual details, trying to deconstruct the proof. And notably, it took him quite a long session.

17:20 If you if you read through it. Yeah. That's right. So if I were reading between the lines, it seems like you know, the mathematicians are not doomed here necessarily, that like, this is just like maybe one part of the set of things that mathematicians do to solve problems. I don't know. I mean, are you similarly optimistic? Are you like, mathematicians are in trouble.

17:43 If you look at this, what happened here? Right. All you need is, you know, when you're disproving a conjecture, all you need is one example to disprove, right? So that's all you need to do here? You're not, you're not drawing upon some crazy novel insight to actually prove something. So you're doing a disproving which, you know, you need to figure out one concrete example to do that, the the piece.

18:13 I think where this is a little tricky is that obviously didn't do a full brute force of my entire polynomial space on all the, you know, like you're trying to do, you know, three dimensions for them. I don't think it did a full brute force search on multiple dimensions, but at the same time, it's not coming up with, know some crazy novel insight here.

18:35 So yes, there is a I think this is largely a function of somebody who is really, really capable of steering the agent and the model really well to be able to draw out some of these. And I think we are seeing more and more of that happen right across different, specializations and capabilities. Right. So folks who have a, a solid grounding and expertise and taste are able to gather way more potent outputs of it.

19:11 And I'm seeing that across different, specialties. Right. So you can take consultants, for example. Right. You'll find people who know how to construct, you know, consulting storylines. Well, how to construct you know, decks. Well, they're able to get way better output or, you know, you look at mathematicians being able to steer this, or you're looking at engineers being able to, you know, they have the, the solid foundations on how I need to go and drive my, you know, construct my, code base.

19:48 Right. They're able to get way better output of it. So I think we are seeing that as a common theme emerge out of the culture. Where do you think this all goes? Right. So we've been talking a lot about mathematics, and I think I mean, you've already opened up, I think the discussion where I want to kind of push the discussion is, you know, maybe math is like a very particular kind of thing where, you know, there's this kind of low hanging fruit set of questions or you just kind of hammer at it, you know, just work a

20:10 little harder than a human would have time to do. And then you kind of get these discoveries, you know, do you think that's kind of generally shared, like, do we live in a world of lots of low hanging fruit where, you know, humans just haven't ever really put in the effort? And I guess how much do we think about this as a math thing, or is it applicable to, you know, consulting storylines to you know, writing to, you know, anything about all the different applications of AI?

20:35 One of the things I'm trying to think a little bit about is like, how far does this go? Yeah, I think definitely this is not just constrained to mathematics. Mathematics. Here is a great example when you have to search through these high multi-dimensional polynomial spaces. Because what happens here is humans excel at constructing, you know, these intuitive frameworks, but they struggle to search really wild high dimensional spaces.

21:00 And what this showed is AI models coupled with formal verifiers act as high speed engines for these counterexample discovery. But of course this is not unique to math or only like if you look for example, of chip design, or if we look for example, of chip like quantum theory or, you know, for example, now that, you know, getting into quantum computing where we need to, you know, build these algorithms and sometimes you have to do these simulations in high dimensional spaces.

21:27 So there is a lot of problems like this that are, you know, not solvable easily by classical systems and where you need, you know, these dimensional search and spaces to find examples, counter examples, etc.. So and that's, you know, where we can really use AI to help us speed that up. Because I feel the era of for like for example, for the solitary mathematics mathematicians staring at a chalkboard for 30 years, it's kind of officially over here.

21:57 Mathematics with other, you know, fields that require these really high dimensional reasoning and search is really becoming an operational machine assisted team sport. But again, I think going to the point that Olivia mentioned, it's not not just anyone can do this. So I think going through the questions, are mathematicians in trouble? No. But their work might change here because, you know, in the Jacobian case, AI appears to have helped search for this unusual counterexamples that humans had not found for decades.

22:29 But the mathematicians still have to pose the problem appropriately. Recognize the result? Matter, check the calculations, explain why it is. Prove the conjecture so that human-AI workflow has still not been fully disclosed, and that is still important. So that's why where you need that deep expertise and not just in the the case for for math, for example, chip designs.

22:51 Right now we're looking at how do we use AI to help us, you know, really design efficient chips. And that is a very complex problem. And especially when you do the testing, the verifications, very complex circuits, very complex systems and, you know, other fields also when you have these combinatorial theories etc., where, you know, it's very complex and so and it is similar maybe to calculators and computers you removed, you know, some manual work, but allowed mathematicians or theoreticians and so on to really tackle

23:20 harder problems. So I think AI here could become a very powerful research partner, especially for generating candidates, finding patterns and checking the algebra. But I think the real pressure here will be on mathematicians. It's like who refuse to use these tools. So I think here we need to kind of they need to learn how to guide AI challenges, answers and turn these discoveries into deeper theories.

23:49 You know, that I think will become significantly more productive if we do it, if we do it right. I just wanted to, kind of emphasize what Kaoutar is saying in the sense that, I think my greatest disappointment about this era has been the degree to which people seem to only be able to see the things that we will lose, and not the things that we will gain from having access to these high powered models.

24:15 Mathematics. It's sort of easy to see. When something gets. Solved, but there's as, as Kaoutar and Ambhi have both said, there are so many disciplines that this impacts. I see this in software engineering too, where, by default it seems like everyone just said, okay, well, we're going to software engineer faster. And that's sort of the least interesting thing we could do.

24:38 I imagine the number of things that we could do and try in the accessibility field, the number of things that we could do, to make products, more beautiful, make them easier to use, make them, more interesting, make them, help people learn about the world, make them less about stealing engagement and more actual value into your life. And I really think that all of those problems are equally complex to mathematics.

25:10 We don't appreciate them that way, because we have this whole, divide between the kind of problems that are and aren't important. But I think the more we can see these as augmentation for creativity, the better off we are as a society in this world. Yeah, definitely. And, if I might add, here, like, fields like chemistry in materials where you need to search through millions of possible molecules or materials where it suggests maybe candidate for batteries, drug discovery and biology.

25:39 You know, I mentioned chip design in engineering where you're exploring circuit layouts, etc. physics, you know, finding patterns, etc.. Even cybersecurity, you know, all these vulnerabilities, attack paths, you know, that could be, you know, exponential here and, and medicine, I think tons of areas, you know, climate, energy software development where, you know, we definitely can harness the power of this, multi-dimensional search powers and, and get some powerful results.

26:10 I feel like Tim's blood pressure has come down quite a bit now. Yeah, I'm feeling better now. Exactly. This offers it's been group therapy for me. You know, in particular. Yeah. I mean, I think that's one of the most interesting things. And we will see, you know, a little bit like just token prediction turned out to apply to a lot of things. I think similarly here, the question is like how many things turn out to be just search.

26:32 And it may really be more than we, more than we expect, I think. So, I'm going to move us on to, our next topics. Moonshot AI is out with its new gigantic, and I really mean gigantic. Open source model Kimi K3 is out. It is a 2.8 trillion parameter model. It's a little deceptive because it's MoE, and, showing great performance, incredible performance against benchmarks.

27:02 And actually Kaoutar. I want to go to you first on this story because, well, it's all very impressive, 2.8 trillion parameters. And, you know, giant model. I'm also kind of like who is Moonshot trying to build this for, you know, we're now talking about model sizes where it's not really like a lot of parameter, like a lot of enterprises can just be like, oh, yeah, let me just talk to the guy and turn on this model, like, let's just run Kimi K3.

27:29 And so there's a really interesting question that's kind of looming behind this and that a lot of hype about the model. But how did you have a theory for why Moonshot is pursuing this, even as a strategy at all? Because it feels like as they get bigger and bigger, the market they're addressing gets smaller and smaller and smaller with time, and also gets more and more expensive to do these kinds of models.

27:47 And so, you know, where do you think this all goes? Like what is the strategy they're trying to pursue here. Yeah, that's an interesting question. So I think they're trying to compete directly with these closed source flagships like GPT, Claude, Fable that, you know, and they're trying to dominate the open weights benchmarks on these long horizon coding and web development tasks.

28:08 I think they're trying, my my feel here is they're trying to compete in the space of these large scale agentic orchestration and so on. And, you know, for these things, you might maybe need, a powerful model like, you know, what they have done. But the thing is here, open source AI isn't just trailing proprietary models by 12 months anymore. So it's also kind of trading blows in the real time, while also driving some of these fundamental architectural breakthroughs.

28:42 So the efficiency here is becoming very important for them to be able to afford to do these things at that massive scale. They have to do it efficiently. And they're they're looking at also, basically, you know, the architecture and also the the innovation that they did in the attention mechanism. So I think that was also essential for them to be able to do this efficiently.

29:06 And also handling these long contexts, of course, I think what I see is, you know, what's happening in the, you know, in the US, for example, there's a lot of focus on, you know, smaller models and these post-training, you know, to, to do these things efficiently. So while they're trying to pursue these large scale, really 2.8 trillion that's that's immense.

29:31 And the context windows here are useless if inference costs grow quadratically because real open source models are created by memory optimizations and really hardware level attention efficiency, that will become super important for this to, to be, feasible or to be practical. So it's interesting to see how this is driving also, architectural efficiencies that will become so important.

29:57 The inference, inference efficiency. Yeah. And I think the efficiencies are an important part of the story. I mean, one of the funny things is, you know, kind of in the end, it sort of seems like the the entity that would be most positioned to run these open source models are the proprietary labs. Like, is that I don't know if this is kind of where this is all going, is that OpenAI is like, well, should we just run K3 instead?

30:18 Like I'm kind of curious about like, you know, you just kind of continuing to like, even with the efficiencies. It feels like, you know, there's maybe only a few enterprises that would ever try to do this themselves or like run K3 at scale. I guess I'm maybe, you know, I'm curious about your view of the market. Maybe that's what they want to chase after.

30:37 Like they're looking they're looking to get OpenAI's top hundred customers. Is that kind of really where they think the the market is? I don't think the market is smaller. Right. So I do think the market is still fairly untapped. Yeah. We we really have not looked at the worldwide market. There is a lot of latent demand in the, the worldwide market, outside of the US, where, yes, they may be the demand might be lagging by a few to several months, but for sure that demand will continue to grow.

31:14 Right. So beyond the US, there is a huge, demand space over there, which I think Moonshot might, help shape up that market as well. Great. So, so, so that's on the market. I, I don't think the market is getting smaller. The, the big trillion dollar question is, how are you going to serve, that market. Right? I think there is there is no clear answer to that.

31:42 To to to Kaoutar's point, right. You really cannot just go and deploy this like you would go and deploy a small machine learning model on your compute, on your local hardware. Right there is, you'll, you'll need, you know, clusters of nodes of GPUs and you'll figure this needed like terabyte of, VRAM or something like it's it's immense. So where are you going to get that level of compute and how are you going to make sure that you have the right level of coupling across your models to the, the, the PyTorch or the CUDA

32:20 layers to the, the hardware? Right. So how are you going to drive all of that optimization? It is not going to be a weekend project for somebody to go and do this. Right. So I think it is an open question. I we don't have a clear answer on that. So I think they're chasing after that market. My best guess is that they are expecting. And you saw that, right?

32:45 They, you know, as the demand spiked up, as the global demand spiked up, they did run into constraints on being able to serve it effectively. And they had to, tamp down on their serving capacity. So that is that is very real. I don't think there is an immediate solution, for them. So I think the expectation is that that constraint will get solved as some of the compute aspects get, a little more, streamlined.

33:18 So it might be a longer term play rather than something that, you know, I'm going to go and serve this model to my customers immediately. So it may just be, I'm building this. And then as I solve for the compute problem, then I will be able to go and serve. And as the worldwide demand picks up and say, you know, 6 to 12 months, then I should be in a position to serve them effectively.

33:40 So it might be a longer term play. Yeah, I agree with Ambhi. I think probably their targeting longer, longer play kind of being the leading open platform for agentic intelligence, where they have to combine these frontier scale models with efficient inference. Very long context and teams of autonomous agents. And so it's kind of model intelligence costs and efficiency agent execution at a very large scale, and also the open source adoption.

34:13 So I think there's also an argument to be made from, a little bit, just the spirit of open innovation. And you know, I say that, but what I mean by that is that someone was someone outside the preexisting Frontier Labs was always going to build a super large model. The fact that they're building it in the open, I think, presents a useful challenge to the paradigm we have right now.

34:40 As we've mentioned earlier this episode, there's a lot of issues when you have a, when you have an API that is closed and the, the people who own that API are deciding what aspects of that model you're going to have access to. All the guardrails are predetermined. There's certain types of work that simply cannot be done. The attempt that they are trying to make is that you cannot do, you know, for lack of a better word, evil work.

35:10 Right? Like, maladaptive behaviors via models. Right? We don't want to see more of that in the world. And yet at the same time, every time that they do that, they are challenging somebody else to come up with a model that can, in fact, do those things that isn't as restricted and see how that goes. And there's an argument to be made that, okay, so I think currently Anthropic has shut down all ability to investigate questions of biology.

35:38 I believe, with the Fable model, that means that okay, where where do biologists sit in this, in the kind of, computing days of yore, a lot of these, major computational engines were owned by research labs and owned by universities, and they were more, in this kind of, space of, public good, essentially. And I think, well, I don't think that Moonshot necessarily has all of those, like, goals in mind.

36:11 Maybe they do, maybe they don't. I don't know if they're getting to it on that level of philosophical. But I do think as a community, we benefit because we fundamentally need more people than just the people at a couple of the Frontier Labs deciding what the future of this, this field is going to look like. Let me maybe last question for you on this segment.

36:36 Should we maybe just do the quick pulse check, right? Which is, you know, I think we're always trying to say like, well, how far behind is open source on this stuff? And I think so one of the themes that Kaoutar brought up, I mean, this is a model with a lot of innovation, a lot of new stuff is in this thing. And it feels like, should we should we start thinking about open source, basically operating at parity with the frontier, or do you think there's still some distance here?

37:00 I think the main struggle with the idea of open source at parity is how few people can actually check that question, and then we get back to the who actually has the compute level for this. But I see among friends, I see a lot of people reaching for especially like lower end MoE models and things like that as daily drivers. And the reason that they're doing that is because they're open, because they know that they can tackle problems however they want to do it.

37:30 It may not be able to solve problems in the same way, but I've seen a lot of evidence from a few different camps that, lower end models can often solve the same problems that higher end models can, as long as the person driving it has a strong understanding of the problem, it may not be as fast. It may take potentially more steps. The human might learn more along the way.

37:58 But it's still possible to solve a lot of the same problems. And so there's an open question then when do you actually need to pull out the, the frontier frontier models? And, for how many tasks could we simply use open models which are now benefiting from the higher end models due to practices like distillation and things like that that can actually, by bringing the architectures down from the higher ends, by bringing distillation down from the higher end, we're able to make smaller models that are more effective.

38:32 And I think that's really good, really important. Very notably, I was having a conversation with somebody the other day who, was basically pointing out this is one of the first times in computing history that we've been actively working against efficiency. And like, there's an there's a, like a tendency in computer science that will inevitably reemerge here that efficiency is king.

38:58 Efficiency matters a lot, because if you aren't efficient, it costs you a lot of money. And, we've kind of been, in this world where, we get to ignore the laws of economics and just build, and it's all beautiful. But efficiency's going to win. And so, I see that, regardless of where it comes from, those improvements in the model architecture are critical.

39:21 Those, and all of those things will eventually come downstream to make it so that eventually the model that can be run by your phone is, in fact, powerful enough to do all of the things you want it to do. Absolutely. Well, that's actually a great segue to, I think, our final quick little segment that we'll do here. Google is out with some new models as well.

39:44 So they have released Gemini 3.6, Flash, 3.5 Flash Lite and 3.5 Flash Cyber and in some ways, I want to kind of pair these two stories together because this is also kind of the same story, but flipped, right? Which is you have big, big company Google. Like they just sit on so much compute, they should theoretically be doing like the biggest, most gigantic models.

40:05 As yet. One of the big things they're touting here is smaller, faster, more efficient. You know, and so it's almost like, you know, whereas Kimi seems to be like going way bigger than you expect. I kind of look at this and I'm like, oh, why is Google going way smaller than I would expect? I mean, maybe I'll kick it to you. Like what? I'll maybe start with the same question we had for the Moonshot story is like, What's Google's strategy here?

40:28 Like, shouldn't they be the ones trying to kind of say, we have the frontier? You should buy a proprietary model. We're competing with OpenAI Anthropic. But as yet, you know, we're we're talking smaller here. Yeah. So I think it ties directly to what Olivia was talking about. Right. In terms of efficiency. So at the end of the day, if you think about, where all of these models would plug in, right?

40:50 So primarily it's going to be either in, the Gemini app, in the suite of Google's products or directly, you know, the the big stream of consumption is from enterprise customers right now. Again, you know, we we talked about market demand. There is a whole lot of latent demand with it. Right. The same is true of also penetration. Right. If you think about the, the adoption.

41:18 So I'm going to generalize just AI as a technology. Right. There are reports suggesting that the the adoption levels is still very, very, very, very early across the global space. Right. And so the level of penetration here and I, you know, I'll, I'll go back to, you know, what I would look at on a day to day basis from an enterprise lens. If I look at my enterprise clients, for example, right, the level of penetration there, even though there are clients, you know, going for AI transformation that it does.

41:58 And, you know, I want to drive multiple different pillars and so on. If you look at what could be accomplished across the entire enterprise landscape, that level is too tiny. So now if you have to go and drive and effective penetration, right, it doesn't always have to be the big frontier model. A lot of work that needs to get accomplished can be done and should be done very effectively and very efficiently with smaller models, right?

42:33 We've always touted as well and smaller models do have a good level of capability to be able to accomplish those tasks. Right. You don't need a frontier model for, you would for some of the agents that make, you know, 2 or 3 tool calls and then accomplish 2 or 3 steps in a small workflow, right? Something like a 3.6 or a 3.5 Flash Lite can do that fairly effectively.

42:57 Right? So all the that category of models can still do that fairly effectively today. So broadly. Right. The the point here is that there is a huge level of, work to be done. Right. The penetration aspect is very small. So the way you bridge that penetration is not by throwing a frontier model at it is by going for cheaper, smaller, more efficient, more effective models.

43:26 Right. And that way you're able to get a lot more of the workflows, automated or transformed how you want to put it. Olivia, maybe I'll give you the last word here. It seems to me that the market has gotten very messy. We used to talk a little bit in terms of, like, well, open source will dominate the small models and proprietary will dominate the big models.

43:48 It seems like we're now in a world where it's like, well, and even in the small models, proprietary and open source is fighting on the big models, open source, the project is fighting. It's just like everybody's in everybody else's market. Do you have any like, forecasts? Like where do you think this market falls out in the end? There's been this really interesting thing that's happened where, we were all introduced to LLMs and generative AI via this chatbot framework.

44:13 Ultimately, that chatbot framework is a very, very limited market. And we're seeing that, I always laugh when I think about the, the first year of ChatGPT usage, where they had an amusing drop off in the summertime when all the kids, went home from their, from school, and then they didn't need to GPT anymore, chat. So chat-based UIs, I think, are actually quite limited, and it is a limited imagination of what exactly, agentic behavior can be and what, autonomous agents and autonomous intelligent agents can accomplish.

44:57 So I think what we will see increasingly more of, and I think both the high end labs and the open source folks are recognizing this is that we will increasingly need to, roll out little models that can solve specific tasks. And ultimately, if you're if you're doing something like, adding automation in some kind of factory, right. There's some kind of like, critical decision that needs to be made.

45:24 And usually a human has to step in and, make some analysis of all of the numbers coming in and make a split second decision. Now, potentially you can insert, a model, in that space and the human moves on to hopefully more complex tasks. That requires their input. Essentially, if you're able to do something like that, you're probably if that, if that, model is going to be useful, it's going to run so many times and that you are going to lose money if you are throwing frontier models at it.

46:03 And I think most enterprise cases actually look like this. And I think it's interesting because, the first year of like UX design around LLMs was a lot of product managers being thrown the idea that they have to put AI into something, and they were like, well, as far as I understand, AI is chat, so I'm going to stick a chatbot in every single, and every single app.

46:26 We release. The vast. Majority of those, as far as I can tell, do not get used. I do not think people are actually all that interested in a blank text box. If they were super interested in a blank text box, I'm not sure we would have apps with all these buttons in the first place. So there's sort of a different future that I think we can envision where you're ultimately interacting with your apps the same way that you always have, but because of some of some kind of agentic, intelligence behind the scenes, those apps

46:53 are smarter. You have to make less moves, you have to press less buttons. And that is actually an easier future for most people to live in than one where in order to get a new graph out of their, health app, they have to type. I would like a new graph out of my health app. I don't think anyone actually wants that. No, I don't think so either. Well, I could talk with this crew for hours on the couch.

47:23 Ha. Olivia, thanks for joining us on the show. But that's all the time that we have for today. And, thanks for joining all your listeners. If you enjoyed what you heard, you can get us on Apple Podcasts, Spotify, and we'll see you all next week on Mixture of Experts.