Reddit cracks down on AI slop & the future of AI compute Transcript, AI Summary & Key Points

00:00 I thought it was really interesting that AI fits into the patterns of daily life, and most people use it within the bounds of their regular daily cadence. And I think that's, um, probably good for the mental health of the world. All that and more on today's Mixture of Experts. I'm Tim Hwang, reporting from a new office. And welcome to Mixture of Experts.

00:22 Each week, MoE brings together a group of brilliant minds working in artificial intelligence to walk you through the welter of the week's technology news. On this week's episode, we've got Mihai Criveti, Distinguished Engineer, Agentic AI; Rynne Whitnah, Technical Lead, AI Ecosystem; and Gabe Goodhart, Chief Architect, AI Open Innovation. Welcome to you all.

00:41 As always, we've got a lot of things to cover. Today we're going to talk about interesting results coming out of the Anthropic Economic Index, a really interesting new proposal on how we might buy and sell compute in the future, and some interesting news about Anthropic getting into the chip business. But first, I wanted to start because I am a, um.

01:00 Some of you may know a long time Reddit user, and so this blog post from the company caught my eye. First story is that they are kind of announcing a very aggressive new strategy to lock down on sort of spam and AI slop, uh, on the platform. Um, and I guess maybe I'll start with you. Uh, I thought was just really interesting that they kind of went out of their way to announce that they were fighting this, um, and, uh, I don't know, I guess.

01:29 Do you think this is, like, a bigger and bigger problem? I suppose, for, for platforms going forwards that they have to start dealing with, like, more and more sophisticated AI spam. I think it's more a question of quantity than anything else. Uh, where, you know, there's always been spam, there's always been garbage on all of these platforms, right?

01:50 Um, and with the advent of generative AI, like the sophistication at the speed at which one person or one actor can generate more spam or more content has dramatically increased. And more importantly, it's not just the ability to spam the same message over and over. You can make a lot of variants of that same message and make it more difficult to find those.

02:16 So, you know, I think this is one of those things where we're starting to see, uh, the ouroboros of AI, right? Where, where when you have actors that are using AI for malicious purposes, the only tool we have in our toolbox to combat that is, frankly, AI, right? Where where we we have the ability to do that. But like, you know, as we start discovering more and more vulnerabilities and things using AI models, we also need to be using AI to remediate those.

02:46 That's kind of the only way we can keep up with the speed of that. So I think this is interesting and it's applying some of those same techniques to, um, communication. Right. And we've used AI models for that for a very, very long time, just different types of models. And I think combining this feels like a natural evolution. But it's also really interesting to see it applied in this way.

03:10 Mihai, I think a little bit about, uh, offensive defense balance here. You know, we normally think about that in the context of cybersecurity, right? Like our hacker is able to get in and are we able to defend against them? This is almost like a slightly different thing. Right. Which is kind of fake or false accounts being on platforms not really to compromise someone's security, but really to kind of like influence the discussion online.

03:31 Um, and I guess I'm kind of curious about where you think that balance is going to lie as sort of AI gets deployed on both sides more extensively. So I basically see it as a war of proliferation of weaponized AI in the form of spam bots and inauthentic accounts. If you look at it from a, I would say, platform perspective, the reason I would read something on Reddit, as opposed to just having a conversation with ChatGPT is because I get to see the opinions or understand the points of view of real humans, right?

04:00 If I'm looking for a review of a specific item, I want to make sure a real human has actually used that item and is giving me their genuine opinion, as opposed to, for example, some kind of a bot that sources information from marketing material and giving a very dry, inauthentic view. So for a platform such as Reddit, it's important to differentiate themselves from the likes of ChatGPT or just talking to Claude by providing that real human connection.

04:31 So I think for them it's existential. It also gives more value to the data they can resell later on to the same AI companies which are causing the problem for them, right, to the AI companies. So it's like a circular economy where AI is part of the problem because it's used by spammers to attack the platform. AI is also the solution because it's used by the defenders to defend against the use of AI.

05:03 It's also part of the financial revenue stream of these platforms which sell that authentic data to the AI. So in a way, we all work for AI, and Reddit is just giving a better curated data set to our AI overlords. Yay! Uh, Gabe, any comments on that somewhat grim, grim cyberpunk diagnosis? Well, I think, you know, my take from this was actually kind of the opposite of we all work for AI, which was that ultimately, you know, these platforms exist for the humans, even if they're selling it back to the AI, it's so that

05:45 the AI can serve the humans better. Right. So as you said, it's an ouroboros. And humans sit somewhere on that loop. Um, but what I was particularly interesting about this article was asking myself the question of why did they need to publish this article in the first place? I mean, putting AI guardrails in the moderation context is not new at all. Right?

06:03 This has been done for decades, but long before generative AI was part of either the problem or the solution. Um, and what occurred to me was, you know, this is fundamentally about building trust. And I appreciate that there was a little nuance painting AI on both sides, because I think especially for an audience that's consuming social data, AI is, generally speaking, just bad.

06:29 Like if you read through a thread, you look at posts, and everybody at this point has an inbuilt filter that kind of assumes AI by default and, you know, searches for a genuine signal in the noise. It's become a needle in the haystack problem in these social platforms. So I think the general consensus is that AI is just ruining these platforms. And I think they needed to publish this to basically demonstrate that.

06:53 Well, there's also a real benefit to this technology, um, that we are leveraging on your behalf, you users that would like to trust us. So I thought that was kind of interesting. The other part of it too, was, you know, as someone who sits in the ecosystem of building out AI, sometimes it's easy to sort of look at abstract patterns. You know, how to build an agent, how to build a chatbot, how to build RAG, um, and think less about how these things actually get deployed in real, very practical settings.

07:21 And I thought the end of this article was interesting because it actually sort of painted this interesting, uh, tiered approach to how the concrete tools that they're building are extremely practical in nature and then get scaled throughout their, you know, human ecosystem. So they have the admins, which is a very small paid group of people who get the most powerful AI tools to actually combat this.

07:47 And then they have the volunteer mods that get access to some of that. And then they have, you know, the sort of not AI enabled at all masses, they can simply apply their their human scale to upvote and downvote and sort of suppress the noise that way. So it was just an interesting look at a very practical AI solution to an AI problem. Um, that I thought, you know, it's nice to see a company trying to paint with some nuance here and not just pick a side in all bad or all good, right?

08:16 Maybe the last question on this is, um, you know, I also kind of wonder about in the future how open communities will be online, right. Like, I think one of the really exciting things about Reddit, why I became kind of a long time user, was that it was just anyone could join, right? It was like really easy to just create an account and start going. And it seems to me, you know, one of the things that they mentioned, they're going to try to do to kind of enforce authenticity is also to just increase the verification

08:38 requirements. Right? If they think you are a fishy account or suspicious account, they might very well ask you to prove your humanity in some ways. Um, and I guess I don't know. I'm curious about where you think that all goes in terms of, like, sort of online communities in the future, like, because we're dealing with more and more sophisticated bots.

08:57 Does that mean in the future, you know, I'm going to have to scan my retina to start posting online? Um, in this effort to kind of, you know, defend the walls as much as possible. And I guess maybe I'm being paranoid there, but, I mean, it has gotten me in a little bit of a cyberpunk mode. You know, uh, I think it's fundamentally a question of curation, right?

09:15 Where, um, you know, as Gabe was talking about, you need to figure out what that web of trust is going to look like. You know, is this a trusted person because Reddit vouched for them? Is this a trusted person? Because someone invited them to this Discord server. And I know who that person is. You know, depending on the community and what you're looking for, there's a lot of different versions of that, right?

09:39 I find a lot of my online engagement in social communities is pretty high trust. Um, like smaller communities in Discord, for instance, I was a pretty, pretty active user of Discord over the past couple of years. Um, but like, you know, it's always one of those trade offs, right? Where the more open you make it, the more your stuff's going to leak in.

10:03 And it's interesting that they're trying to tackle that. I, I don't know how much that can be completely handled at the platform layer. Um, but um, but it is it is interesting. Right. And, you know, a defense in depth approach. Right. Just like we would have with almost any other model of threat actor is probably the only way to mitigate that. Maybe just another thought on this.

10:27 Just to jump back in here. I think it helps to also look at Reddit, not just like a community, because it's not just a community, it's a business. It's actually a massive business. 2500 plus employees are publicly traded. It's got a number of investments, which apparently include some from OpenAI with 9%. Or at least this is what Wikipedia is telling me.

10:51 Right. So they are in a business and they are protecting the revenue stream. If the quality of the data goes down. If users stop trusting Reddit. If Reddit no longer becomes a source of authentic user generated content, then their $2.2 billion in revenue stream is going to suffer as a result of it. So I think it's become existential for a lot of these companies to ensure that their business model is protected from potentially weaponized use of AI and spam bots and all these other kind of things.

11:29 I think we actually saw that with Stack Overflow, right where Stack Overflow was the place that we all went to get developer questions answered. And I'm not sure it is anymore. Right. We've it's kind of been subsumed by the ability to ask a question to your AI assistant. A great discussion. We're going to have to keep an eye on this, and I'm sure what Reddit's doing here is going to be imitated in other places around the web.

11:52 So we'll have to see what all the downstream effects are. I'm going to move us on to our next topic. Anthropic Economic Index, which I think we've talked about in the past, this sort of Anthropic effort to sort of capture sort of the economic effects of its technology, largely by using data on what users are actually using their products and services for.

12:15 And so they're out with their June edition of the Anthropic Economic Index. And this is a fun one. As per usual, there's lots of fun charts and kind of a welter of data that they've kind of pulled. So there's a lot of different sort of ways we could get into this story. I guess maybe. Gabe, I'll start with you. You know, the first one is I think there's been a vicious rumor around for a while, which is, well, you know, people just use chatbots mostly for cheating on college homework.

12:41 Um, and, uh, I think one of the fun parts about this study was they said, okay, well, let's try to figure out what people are actually using, um, the technology for during the day. Um, and there really is kind of like a vast set of use cases, right? Like, I guess maybe, you know, a question for you is like, whether or not this kind of, you know, settles once and for all the idea that this technology is actually pretty narrow use in terms of what its people are putting, it's used for.

13:06 Um, and there actually are indeed a lot of diversity in, like, what people are actually going to put AI to. Narrow scope of this tool is a narrative that I frankly think is a couple of years old at this point, and has pretty well been put to bed already based on what we can accomplish with more complex agent capabilities. Um, I think the bigger question is not capability breadth, but adoption breadth.

13:34 Right. And and so like, are there enough people now and has the UX gotten simple enough to access this broader capability set that it's actually diffusing out into the overall economy. And I think this report pretty clearly shows the answer, at least for Claude users, is yes. Yes it has. I think that's the biggest mitigating factor in this whole report, is that it is a hugely skewed sampling of the general population, and it cannot be taken as a representative sample of humanity.

14:05 It is people that are opting in to use Claude, and Claude is already a more technically minded audience than general purpose AI users, which is already a more technically minded audience than the general population. So I think we can certainly conclude that, yes, like for people that are opting in, this has become a broadly useful tool. Um, and there's some really interesting generalizable study that happens in this article, um, that doesn't really need the qualifier because it really only applies to how is AI being

14:37 used by people that are using AI? But I think some of the broader conclusions that they try to draw towards the end, especially around AI sentiment and AI fear, um, I don't think are really worth a hill of beans because it's not a representative sample. Um, so that that was sort of my cynical take on this, is that like, that should just be in very big, bold letters at the top that like this is only sampling our users.

15:05 They do mention it in the article, but you have to read pretty carefully to catch that. Um, the part that I actually thought was most heartening about this, many of the conclusions that their data delivered were not terribly surprising. Um, but I thought it was really interesting and I guess reassuring to me that AI fits into the patterns of daily life.

15:29 I think there's been a lot of stories about AI vampires, and people where they're sort of rebuilding their life around the cadence of their token budgets. Uh, and it seems like, uh, this, I mean, I I'm looking at you here. Uh, but but but I think this report shows that that's actually not the majority of users. And most people use it within the bounds of their regular daily cadence.

15:53 And I think that's, um, probably good for the mental health of the world. Um, and, you know, it's nice to see that on the weekend, the topics turn to weekend topics, and on the weekdays the topics turned back to professional topics. And, you know, they align throughout the daily, uh, hourly cycle with things you might imagine aligning up with a daily, you know, look for news in the morning, recipes at night, that kind of thing.

16:16 So it's nice to see that sort of the patterns play out in a very human way. Uh, and hopefully that persists and sort of fosters the story of AI as a complement to humans' natural, uh, usage and cycles, rather than sort of forcing the humans to adapt to the AI. Back to your point, Mihai. That we are all working our AI overlords at this point, I think.

16:40 I think we still have a hand on the AI overlords ourselves. We'll see. Yeah, I think the selection effect observation is a really good one. You know, there's this chart that I'm looking at that they have, which is you know what share of work can your AI do today. What share of work do you think it'll do in 12 months. But I like the idea that, like, those are the people who are already so enthusiastic because they are already self-selected Claude users, so they actually may be relatively more optimistic about this than

17:01 than others. Um, I guess Mihai, as someone who raised his hand as someone who's rebuilding his life over his token budget, um, do you see yourself in this report? Do you feel it's reflective, or do you feel like you looked at this in your, like, amateurs? No. Not necessarily, not necessarily. I mean, uh, speaking of rewiring my life, I actually owe no sleep.

17:23 I haven't slept in the last two days. Almost. Um, because if, you know, um, Anthropic has restored access to Fable. And if you're a max subscriber, they've only restored it until July 7th as part of your usage limits. And it's about, at least for me, it's about 90 bucks a question. So that thing is expensive, right? So I spent all of my nights right as the limit was about to go off, spending all of my tokens, uh, on on Mythos or Fable.

17:55 You know, that version, um, just before they announced, oh, we're moving the deadline instead of July 7th, you get another weekend. I was like, oh, I get some sleep, please get some sleep, sleep. But now I'm looking at it and my weekly sorry, my monthly, weekly weekly quota has expired. So I've hit the second quota, not just my Mythos quota because it blows through your quota very fast.

18:19 So I definitely see the rewiring your life around AI as an important aspect that's not necessarily represented here. But to your first question, which is like, oh, it's just a bunch of kids doing it, using it for homework. I've actually seen that in the report. The number one use case globally is homework. That's something like, you know, double the next use case.

18:41 So I still think homework is one of the use cases where we're seeing, uh, AI use more and more and more. So maybe we're going to see some restrictions. Maybe we're going to see similar guardrails being put in place to restrict AI from using it for homework. So this field is going to evolve, especially when it comes to restrictions, guardrails, controls, policies, global policies.

19:04 Um, I'm already starting to see as I just walk in my neighborhood, there's a school next to next to my house, and there's a big, big banner on the school that says, this is a mobile phone and device free school. No use of technology allowed once you enter the campus. One of my hobbies is video game development. Uh, so I've been building my own video games and things like that in my spare time, and it's a joy.

19:29 Um, and it's been really interesting to see, uh, that was, uh, I think number three or number four on the list of what people do on the weekends with these tools, which is fascinating. Um, you know, uh, I would say, you know, probably the part that is most interesting, uh, in this entire report to me was actually further down, right where they talked about, uh, okay, this is the usage right now, and this is where we think the usage could get to.

20:01 And I didn't really see that backed up with a lot of numbers. It was it was it was numbers. But I, I'm not quite sure where those numbers came from. And I think the interesting part about that is, you know, I've been thinking a lot about, you know, the nature of LLMs and how they, you know, they repeat things they've seen before. And there's this interesting thing where, uh, I think as humans, we tend to think that our use case is more unique than it actually is a lot of the time.

20:31 Uh, so like when I'm programming a game, I'm using patterns that have been written before by many, many people. And so that sort of stuff works really, really well these days. Right. It's it's it's shocking how good that is. And also I think, um, you know, one of the things that's been really heartening about that has been this idea that, you know, none of the other stuff goes away.

20:56 Right. There will always be a need for the unique stuff, too. Um, and that really led into the messages of, you know, what people are hoping for in the future at the end. So, um, you know, I am definitely somewhere in the middle in terms of, you know, not not not it's nuanced. Right. But I think it's, uh, it's a very, very powerful tool that can do a lot.

21:23 Yeah. For sure. Um, maybe a final thought. I mean, if you've got a response to, you know, Gabe's observation, which is this is like a very selective set of people, I guess. Do you have any thoughts on, like, if we had conducted this with, like, I don't know, we had some magic wand that we could say we're going to actually just survey the population as a whole, uh, or even on ChatGPT users.

21:42 Right. I think they have a larger consumer base. Um, how do you think it might look different in a lot of the communities I'm in, I see a lot of, uh, a lot of really negative sentiment. Right now, I think this is a, um, compared to what we would see on the broader scale. This is a very positive report. Um, and I think there is incentive to do so. But I also think, you know, there's a lot of fear, especially around like, creative and artist communities and things like that.

22:12 You know, I my partner, right, is a musician. And I had used, uh, I had tried making some gimmicky songs with the music AI and that did not go over well. So I think that's that's an interesting thing where we, we, you know, there is an interesting thing where we need to make sure that we are, um, keeping in mind what happens to the people who are doing that creative work, because the only way any of this gets any better.

22:45 Well, and just on the point of the selection bias here, like Claude, by and large, is the most AI invested community. You know, you can't really use Claude in any meaningful way without paying for it, even at a small amount. And so even as you said, ChatGPT users have a generous free tier. Um, you know, Google Gemini just kind of comes along for the ride when you type in a Google search.

23:15 there are many much more casual user communities than the Claude community. Um, even for folks that are getting benefit out of, uh, AI. So I do think, you know, I would love to see a lot of this study repeated from a third party, um, survey company, and I know they are doing that, so maybe I have to go do a little research on my own and try and find some, um, unbiased sources for some of these types of study elements.

23:41 Um, but I just think it's interesting to learn how Claude users are using Claude, but we have to take it as exactly that. I'm going to move this on to our third topic of the day. Um, this is kind of a fun story that popped up on Axios. We don't normally cover sort of specific, you know, startups, uh, on the show. But this one, I think was worth kind of covering., there's a company called Orn, um, which was backed and raised a $33 million seed round.

24:14 And the whole premise of Orn, the company is that they really want to create a much more liquid marketplace for buying and selling computing power. So the idea is that rather, in the future, you know, in the future, rather than go out there and say, well, I'm going to go work with a CoreWeave or something like that to get my own compute cluster set up, that there simply will be like a commodity market where people will be selling, no, I'm buying and selling kind of computing rights to clusters that they own.

24:39 And this was sort of an interesting idea. As someone who's been kind of involved in large kind of compute cluster build outs in the past. These things are both really expensive, really time consuming and very bespoke. And so sort of the notion that you'd be able to kind of turn it into a much more kind of liquid marketplace seems like a really interesting one, particularly in a world where you know you as a company, you're trying to maximize your use of the GPUs as much as possible.

25:04 But everybody knows that there's kind of excess capacity. And so the idea that you could rent some of that out does seem particularly attractive. Um, I guess maybe maybe I'll turn it to you. I mean, if you were the investor here, would you put money into a company like Orn? Maybe. I mean, we've seen this take off with things like carbon credits and carbon credits swaps before.

25:22 In the past, you could see this as kind of like the opposite of a carbon. Well, maybe even similar to that, right? Um, I think what's really missing from this field is regulation to incentivize the kind of trading. Um, as well, I would say. Right now there is enough AI to go around, even if it's very expensive. The moment you're going to see demand go up or availability go down.

25:54 And we are starting to see some restrictions. You know, memory prices have shot through the roof and GPUs and everything getting more and more expensive. Then we're going to see this kind of market hit its stride. So if you look, for example, like more powerful models like Mythos and Fable, there's definitely shortage. They're definitely very expensive.

26:09 Um, when I've used Claude, they gave me an estimate of how much money I would have spent using those models. And I think I spent in a day and a half of prompting about $593, solving very little but of good quality. So obviously it doesn't match the subscription. And if I were to spend the same amount of money myself, I wouldn't have. So you could see it.

26:35 For example, for the more powerful models for availability of GPUs to run, the more powerful models could be traded like commodities. So I see it as a potential future. Not sure it's one that's good for consumers, but definitely another interesting market for traders, especially with, you know, other markets like the betting markets and so on, being so attractive.

26:58 It could be another market that opens up so we could see it. And I think Sam Altman also had something to say about it, like two years ago, about creating a crypto token that allows or gives everyone their own kind of availability to trade or to manage their AI spend. So I definitely think something in this space is going to happen, whether the regulations are going to be there, whether the consumer appetite is going to be there.

27:25 We're going to have to see Gabe. Any thoughts on this? I mean, I guess one thing that underlies this is, you know, and I'm sure if there's any oil traders that are listeners of a MoE, they're going to hate what I'm about to say, but it kind of feels like a lot of oil is very similar to one another. Um, you know, there's like there's obviously grades of oil, I think, but like, I don't know, is compute like, is every cycle of compute pretty much fungible?

27:44 Gabe I guess that's kind of the question I have is like this kind of market. That was exactly where my head went as well. And of course, yeah, oil is different. There is thicker oil which can be distilled in specific countries. There is thinner oil. There are models which run on one type of GPUs. So I think it's actually very similar to oil. Right. So, so so Mihai, you yeah, you you probably picked on nuance that I wasn't going to pick on, but I think, um, oil probably has similar complexity in the extraction and

28:18 distribution phase to, to compute. Right. I mean, getting it out of the ground is hard. Moving it to where it can be refined is hard. Delivering it to be where to where it can be consumed is hard. But the consumption point is remarkably uniform. Um, you know, you have three grades at the pump. You four if you count diesels, you stick the pump in your vehicle and you press the button and you, you know, scroll your phone for a minute or two and then you drive away.

28:49 Um, I think that that consumption point is going to be the real challenge in this kind of a market. The technical challenge, because GPU consumption is not all created equal. And in, uh, you know, in there's a there's a big scheduling problem here. Different GPU workloads, um, require different amounts of compute, many of which span beyond the scope of a single card or a single node containing, you know, a maximum number of bridged cards.

29:20 Um, so if you're talking about a job that requires a huge amount of compute, like a training run for a very large model, this is a big gang scheduling problem where you have to actually wait till all the excess compute that you're going to need for a certain long period of time is all available, all with high networking available between the different nodes that are all going to execute this job.

29:41 Like that's a much there's a lot of constraints on that beyond stick the the pump in my car. Um, inference. I think I could see this happening at a much in a much more straightforward way. The challenge on the inference side is going to be the sensitivity of the job. Um, and so I think, you know, the users would need some ability to taint the sensitivity of their compute request such that it would never get routed outside of, say, geographical regions, or that it would land on a encrypted node so that the payloads that

30:12 they're sending couldn't be read. So there's a lot more sensitivity and security constraints. I mean, I think going back to some of our earlier conversations about trust, we see that little, you know, uh, trust icon on the pump that says this is, you know, premium gas, this is regular unleaded gas. And we say, cool. I'm assuming that this liquid I'm going to shove into my gas tank is what it says.

30:39 It is the end. Um, but we don't generally have company secrets sitting in that pipeline. So, um, there are a lot of logistical concerns that would have to be executed. And I think there would be a lot of technical infrastructure build out that would need to happen to make this an effective market. Um, I could definitely see a company targeting a subset of this in a meaningful way.

31:01 In fact, there already are companies out there where you can essentially, you know, loan out your GPU for short lived jobs. Um, so, you know, if people are willing to put aside the sensitivity concerns or the resilience concerns or the, uh, you know, multi-GPU scheduling concerns. You probably can go rent a GPU today for relatively, you know, market price.

31:25 But scaling this up to the the level of, you know, having the big labs buy their compute futures, I think is going to be very logistically challenging. Um, Rynne any final comments on this? I mean, I'm just thinking about a comment that you made a little bit earlier, which is that like people, people generally are kind of unhappy about AI and like sort of the notion of like, well, let's also add some like financial speculation on top also doesn't feel like we're headed in the right direction.

31:51 But curious about what you think about that or if you've got other takes on. I mean, I guess if you were an investor to ask the same question that asked Mihai, curious about if you would put money into all this? Yeah. So, um, there's a question I like to ask people when I meet them. And, you know, once they've explained what they do for a living or whatever, like, I always like to ask, what's the hardest thing about that?

32:11 And probably the most interesting answer I've ever gotten was from someone who was describing that he worked for a company that made socks. And that that was his job was to go and navigate the wool futures market in order to get them the materials they needed to make the socks. And I thought that was fascinating because, you know, it's not just a matter of, hey, I go out and I buy wool when I need it.

32:36 Once you turn it into a financial instrument, you have to plan ahead for that in the future. Right. You have to say, I'm going to make this reservation six months from now or two months from now or whatever. Right. The futures markets are all about planning for demand, right? And anticipating where you're going to end up. And it makes it less fungible, not necessarily more fungible.

33:00 Right. Once that takes over, where, um, you know, especially with the demands of training and compute, probably what that does is it raises the price of compute. Um, but it doesn't necessarily, um, you know, more compute will go to the big players, so it may get more financial utilization out of those assets, but it will not necessarily. Um, at least, you know, I'm not sure it will be some large democratization of, okay, there's this unused capacity.

33:31 Um, now, that said, you know, Gabe hit on the idea of there is a bit of a commoditization of the consumption side and that does compete a little bit with that. Um, you know, Nvidia has been doing this for a little while where they will jump in with a new, uh, AI compute provider and they'll provide some of the infrastructure and some of the, uh, like cards and networking equipment that's required to get started as a neocloud.

34:04 And then they take a cut of your profits in the future, right? So they've been doing that for a little while. Um, and that feels like a different version of this model where you can still get the reserved capacity that I think is necessary for the training market. And then that continues. Um, there's also a concept called Jevons paradox, right? That I think is really interesting here.

34:28 And that is the idea that, you know, as cost decreases, the net amount of that asset being used increases. Um, so it's not that, you know, in a way, this could actually raise the prices of some stuff and then slow the adoption, I think is the, the interesting thing there. Yeah, that'll be really interesting to see. I hadn't really thought about that that angle, but seems very possible.

34:52 Um, if the incentives aren't setting up in the right way, you might make more money reselling your infrastructure than training your latest model. Oops. Stop the training. Forget the new model. Forget the consumers. Just go rent that out. Yeah, yeah. We're too busy speculating on compute to actually train the next model. I just feel like a very funny outcome.

35:14 Final topic of the day. Another Anthropic story. I apologize, but they've obviously been in the news lately. Um, the interesting kind of story we've been tracking. You know, the wave of kind of speculation around the frontier labs getting into building their own hardware. Um, OpenAI has largely been the focus of these stories. They announced a chip called jalapeño.

35:36 Um, you know, just the other week. Um, and it looks like Anthropic is doing the same. So there's rumors swirling around since earlier in the year. It looks like there's kind of maybe preliminary discussions now happening with Samsung about building a chip, though details remain very sketchy. And, you know, I think the reason for this we've talked about on the show before is just, you know, everybody's looking to optimize against their own models and technologies that they're building.

35:59 And also, you know, in the edge case, in the best possible case, maybe they reduce some dependence on Nvidia. Um, and I guess maybe Gabe I'll kick this one over to you. You just start the discussion is uh, I guess what I want to know is like, you know, are these. Are these for real? Like, it's really, really hard to do your own chip well and to do it at scale and, and so kind of just interested in your thought about whether or not, you know, these efforts really should be considered kind of like genuine efforts to

36:30 eventually when they just kind of like replace, say, Nvidia chips with, um, you know, in-house chips, or if that's not really kind of how we should be thinking about this. I think this is just an economic story. Um, and I, I don't know the people behind, uh, these decisions, how deeply they've researched the economics of it, but I'm assuming pretty deeply.

36:49 Um, so to our previous story, compute is in high demand. It's hard to come by. There's a supply demand imbalance. One of the natural ways to fix that is to increase supply. And if you happen to be one of the companies with a lot of leverage in this ecosystem, you can do that in a proprietary way that gives you the advantage of the increased supply without everybody else.

37:12 So now we're seeing all the people with all the leverage try to do that, increase that supply side. Um, I think so in some ways I think, you know, not surprising. Probably see it happen, how successful it'll be. I don't think it will replace Nvidia. I think it will just augment the existing dependency, because it's trying to fundamentally make up for a gap between the demand and the supply.

37:34 Um, I think, you know, the the story around sovereignty and resilience to disruption is probably a little hedging against the the geopolitical political element of AI and the, you know, corporate political element of AI. Um, I think, you know, one big player starts to say, we're going to take all of our compute, uh, in-house to a chip that only we can use.

37:59 Okay. So now the other big players say, well, we better be able to do the same thing. We know we're going to get squeezed out of this market. Um, so I think, you know, there's there's a whole lot of politics floating around this and a whole lot of economics to see whether this actually makes financial sense. I think the other side of this, that's actually in some ways more interesting to me is the, the shoe that all of us are worried about dropping, which is the cost of tokens for consumers.

38:25 So Mihai mentioned that he managed to spend the equivalent of $500 in a day on Fable pretty easily. Um, and I'm sure Mihai is not alone there. I think we have all seen this fear of the actual cost of what we're using, uh, eventually catching up to us. Right. And so I'm sure the big companies themselves are worried about this. Not because they're worried about, oh, great.

38:51 We're going to have a huge, you know, like we're finally going to actually have our balance sheets even out. That's going to be bad for us, but it's going to feel really bad to their consumers. And a company fundamentally has to serve its consumers. Um, so I suspect that these power efficient chips are specifically targeted at at least partially mitigating that eventual shoe dropping.

39:17 Which is to say, if we can make the actual tokens themselves less expensive for us, we have to pass less cost on to you as consumers, and we make you less angry and less likely to leave us. Um, so I think there's like a self-preservation element there, because I suspect all these big labs are kind of freaking out about what are we going to do when we actually have to start paying our bills?

39:34 Um, and we can't just subsidize our users anymore. Yeah. Well, I guess, Mihai, your natural person to respond to this is, uh, you've been you've been getting one over Dario. how you've been maxing your, uh, your Fable usage. Uh, when is the music going to stop? I mean, do the companies have the time to kind of get to a cost efficient world? And even in that cost efficient world, I actually wonder whether or not it will be inexpensive.

39:59 Like, I think it still might potentially be really expensive. Um, so yeah, I guess, Mihai, if you want to respond to that. Sure. So if you look at the history of how AI is being trained on Nvidia chips and all these things. It's a combination of the availability of the hardware, the ecosystem, SDKs that they've produced which make that easy. That's why it's harder for folks like AMD to enter in this space.

40:20 But recently Google has made quite substantial use of their TPUs. And in fact, I think Anthropic is consuming very, very large numbers of of TPUs. Unlike the Nvidia chips, they're optimized or initially started to have a stack designed for AI, not just for gaming. So you could argue that more efficient use of the memory of the components of the platform.

40:45 You also look at vendors like Cerebras and Groq and all these other things on the market, which are producing specialized endpoints and accelerators that they run the same models, but they run them ten times or 20 times faster. So for use cases like voice, for example, or, you know, in customer support where you're calling in and say, hey, can you help me with my problem?

41:09 And then there is this awkward silence while the AI and the agent is processing tokens. Is that time to first token optimization? It's the response time optimization. So I definitely see a need for specialized accelerators that are either more cost efficient or tailored to specific use cases. It's you're still going to need platforms like Nvidia for that general purpose compute.

41:34 You can use it for training. You can use it for inference. You can use it for a bunch of things. You can run any models of any size, and it's going to perform okay for most of those use cases. But with these dedicated accelerators, you can really tailor to build, you know, it runs just this one specific model, but it runs it really, really well. It's very efficient for the inference.

41:55 You get your time to first token much faster. Um, so I think of it from a consumer perspective, it's needed going in this direction in the right direction. It also gives Anthropic some independence from Google, who, while being one of their competitors, they're also one of their main suppliers for compute, and it lets them diversify their supply chain.

42:14 So I see it as a good move, and I think we're going to see more and more of this from the likes of OpenAI, from other vendors, for building and training their own models. Rynne, final word on this if you want to bring us home, I guess the question I have after this sort of discussion is there's obviously been a lot of speculation about the model companies sort of going further down the stack to build their own compute.

42:39 I guess the question I have is why doesn't Nvidia do the other way? Right? Why doesn't Nvidia say, well, we're going to do our own frontier models. Um, you know, they've they've announced some open source stuff they've done, but certainly nothing sort of like Fable-like or Mythos-like that they've actually tried to do, you know, is that a smart move for Nvidia to kind of stay away from that, or should they, should they also be playing in the same waters in the same way that they're playing in Nvidia's waters.

43:00 Well, that's an interesting question. Um, I'm going to say, you know, traditionally there are two ways to expand a business, right? And that is either horizontal integration where you take over more of the market, or vertical integration where you take over more of your supply chain. And both of those are valid approaches, right? Where we can see, you know, Nvidia has done a lot of horizontal scaling where they've gotten into pretty much everything that anybody is doing with AI.

43:27 They've got to they've got a hand in it. Um, and I think they've really focused on this selling shovels to gold miners. Uh, perspective so far. Because right now. Right. Building a model is a loss leader, right? It is not a profitable market. And it may become one at some point, but like, uh, it's a loss leader. Um, and, you know, for OpenAI, for Anthropic, going down stack is a matter of showing that they can have independence as they go into their IPOs.

44:00 You know just cynically. Right. Like they are both looking at, uh, going for an initial public offering in the next months to years. Uh, and that will be a important part of their story is how do you keep from having your competitors like Google ratchet up the costs on you? Um, so they need that. Um, and Nvidia, I'm not sure they need it, and I'm not sure that they have the data where it is not just a matter of having compute, it is a matter of having so very much data.

44:32 Um, in order to build one of these. And that is a pretty big moat and it's becoming bigger by the day. Um, you know, there was a period where people just scraped the entire internet and pulled it in and did that, and that's harder now. Right. Like, I, I don't know. I don't know how feasible that is for those who are just exploring and they've built models.

44:57 They've built some pretty cool models, honestly, some cool architectures that have been treated as you know. Here's how you use the tech. And I think that's valuable. I'm just not sure that, um, you know, consumer facing inference provider is where Nvidia wants to land. Why doesn't Nvidia build their own games? They also don't do that. But they did run their own cloud gaming service.

45:22 I do still write GeForce Now. So that is a hosting for other people's games provider. And I think that is much more aligned with where they'd like to be. Rynne. Mihai, Gabe I always learn something when you all are on the show, and it's always a pleasure to have you here, so hopefully we'll have you back soon. And thanks for joining all your listeners. If you enjoyed what you heard, you can get us on Apple Podcasts, Spotify and podcast platforms everywhere, and we'll see you all next week on Mixture of Experts.