OpenAI cancels Astra release, Sonnet 5.5 & what Meta Muse means for work Transcript, AI Summary & Key Points

00:01 you know, when I want to get a piece of work done, I set off my swarms and, you know, I'll maybe have a hundred agents go off and perform a task and come back to me and then Madison shouts at me for uh what my token bill is going to be. But but the reality is you're going to you're going to hand off those tasks and it's going to it's going to get done.

00:16 [snorts] >> All that and more on this week's Mixture of Experts. [music] >> Hi, I'm David Sax and welcome to Mixture of Experts. Each week, we bring together a panel of IBM experts to work out what actually mattered in this week's AI related news. Regular [music] listeners or viewers will notice a new voice/face here. Tim Hong has sadly departed the show.

00:38 He has handed me the mic. I asked my boss Elliot Tricket if we could just [music] use a deep fake of Tim moving forward, but it turned out not to be legal, so you are stuck with me for the time being. My guests today are [music] Chris Haye, distinguished engineer, Madison Gu, vice president of Watson X for Americas, and Ash Manhass, practice manager, technical [music] content.

00:55 Today, we're going to cover three stories. We're going to look at two models and an agent. We're going to look at Anthropics Sonnet 5.5 model. We're going to talk about Meta's agentic muse. And why don't we start off with uh some news out of OpenAI which is about a a model that isn't it's a a story about self-restraint and a model that is not coming out.

01:17 [music] And so the Wall Street Journal uh this week reported that OpenAI would be scrapping its plans to release Astra 6.1 I believe is the model uh in October. And the reasons for this were that it underperformed visav a previous model on some key metrics including deception which I think most of us don't want to have in our models as well as something called scope authorization or sort of a tendency to go beyond what it was sort of permitted or asked to do.

01:49 I think I'll start with you, Ash. Reading about this uh this self-restraint of OpenAI, what was your reaction here? Um and also, were you at all disappointed? Were you looking forward to this model? I think the timing given the fact that they had their dev day the day afterwards and they did announce another model and a bunch of other things tells me again that there's like some narrative here which raises awareness of the company and raises awareness of what they're doing right at the right opportune moment for them

02:22 to get as much engagement and like sort of interested in in what they're up to for their for their dev day. So I think there's an element of that. Um I think not publicly releasing any data around this apart from saying oh yeah it kind of failed our testing so we didn't release it. Um, actually we're not going to give you like the eval data. Just trust us.

02:42 We tested it. Like why are you telling us, right? How many models have all these model labs got that they've got under development? Like why did they decide to tell us that? You know, it's like not enough information to do something with. Yes. Which isn't a a problem that we already acknowledge and are aware. Like it's like you're just marketing it.

03:02 >> I I was sort of looking at the circumstances under which the news came out, right? So far as I know, it wasn't it wasn't like OpenAI trumpeted a press release. It was, you know, it was as though the Wall Street Journal were were breaking news that this was reported, but you know, presumably I used to be a journalist, you know, at Fast Company and places like the I wrote for the Journal and so one knows that sort of there's often negotiations happening behind the scenes of how a story comes out.

03:26 Um Chris, I'll pass the mic to you. What, you know, what was your reaction when you saw this journal report and do you share Ash's skepticism about the sincerity of it all? >> No, I I think it's fair enough. I'm going to go the opposite cuz I I you know, first of all, I mean, I desperately wanted a model cuz I'm like I'm like addicted to models. You give me Oh, here's a new model.

03:45 I want to go play with it. Here's another model. I want to go and play with it. You know, 6.1. No, I need 6.2. I need 6.5. I want that new Bell internal model. Why have I not got that to play with just now? So, you know, you're you're never going to satisfy me by saying I'm I'm not releasing a model cuz I'm I'm addicted to models. Um I know I shouldn't say that, but you know, hey, I love my models.

04:04 However, I think personally personally that if you've just been washed over the coals by uh you know the Australian government for hacking their sites with your models, I don't think you can then turn around in the same week and go and yeah, no, this this model didn't pass our safety requirements, but we're going to release it anyway. What what's the worst that can happen?

04:25 I I think I think they I think they've got to say something in that case. And I I do think there was a lot of push, you know, everybody was expecting the Astro models. We've seen the amount of leaks that are happening on on there, the latest one, you know, on on on, you know, for dev day and and with Opus 5.5 being just released and it being so good.

04:47 I I think they they wanted to be able to respond, you know, but I I think they they had to address it somewhere because they did release um they did release the new soul model, 6.1 soul, right? And then everybody's like, "Where's Astro?" You know, so I think I think they had to. Um, so I I I'm slightly against Ashen that one. I I I don't think it's cynical.

05:08 I think if they had a model that was safe that could be up against 55, I think they would be uh doing it. >> Model addicts, what what are their behaviors? Like I'm just curious like when a new model comes out, how do you put it through the paces? What are you what what crisp benchmarks are you running it through? uh why why is why is a new given that there are so many models coming out why do you choose to you know break break in a new one rather than go deeper on the ones that you're already playing with >> I think I

05:34 mean I hate to say this I spend way too much money on models to begin with so there's but there's certain things that I'm looking for like I still like the front-end capabilities of Astra I still slightly prefer that overus 55 for example so if I'm going to do a front-end task I'm going to tend to lean towards the Astra models so the sort of as I will do on this one which Ash will appreciate will be like crap an you know an iOS app and you know and then I'll try and give it a design system that I wanted to follow.

06:00 So I'll sort of really try and push how far I can get on that whereas the uh maybe the Opus 55 for example the more backend research I tend to push towards the Opus models but then if I need to get something out really quickly then I'm looking for a fast model like sonet so that will be everyday coding. So, so I have a set of stuff that I constantly run like like a lot of my open source projects for example and then I will just be running them against those various projects and then and then you can see it now for the

06:28 earlier Opus models if it if it you know if it says a whole bunch of stuff that I literally can't understand because it's speaking you know u researcher gibberish then um it starts to put me off that model and um but yeah that's that's how I kind of do I'd love to say I I run some amazing benchmark but I just kind of vibe it with the the projects that I work on.

06:48 >> Uh Madison, let's turn to you. Uh you are first of all, welcome to the show. It's it's uh your first time here. Um you're vice president of Watson X for Americas. I have played around Watson X and um it is a platform in which you can encounter various models and make choices about what models you are going to throw at certain tasks. So, I'm curious how you're reacting to Chris's um uh kind of segmentation of how he applies certain models to certain tasks and how do you think this affects users of Watson Xplatforms

07:15 that now there's one fewer model than maybe there might have been on a certain timeline. >> Yeah, great question and thanks for having me um on the show. I think some interesting things about what Chris said, right? I I think about a lot of this from a business lens or an enterprise lens and what it means to apply these technologies and to drive transformation as a whole.

07:33 And we were having a bit of chatter this morning before we started right about you know consumers eventually become your business principles making many of these decisions inside of an organization. And so I think largely what we see in the market today I think it's pretty interesting because so many companies and so many businesses are talking about how do we move from just this massive state of experimentation which like it or not there's still a ton of experimentation healthy experimentation that's going on inside

08:05 enterprises and and businesses today. But how do we really move from experimentation into the application of this AI technology to transform processes and to ultimately deliver value to the bottom line? But if you backtrack into some of the elements that Chris was talking about, right, if if Chris is out each and every day playing with new models that you know come out from whether it's OpenAI or Anthropic or Google or on Hugging Face, take your pick, right?

08:34 Chris has a flavor of models that he likes in order to do different tasks. And I think that transcends then into the business realm of whatever Chris does, right? From a 9 to5 every day and working. And we see this a lot with developers in the enterprise. If they're given an option, it doesn't matter if there's a cheaper model or there's a faster model.

08:57 They have a bias toward a model that they've played with and so they naturally go there. And most of the time, you know what that bias is? It's the largest model and it's the model that cost a business the most money. So I think one thing that's interesting about Watson X and what IBM is really trying to do is saying take IBM Bob, right? Our agentic IDE for example.

09:18 You know, we built something that was kind of quite unpopular at first if we're honest with ourselves, right? We said, "Hey, we're going to build this engine that understands the intent, then assesses the context, and then actually chooses the best model on the back end, whether that's a model from Anthropic or whatever model we're plumbing it with, right?

09:39 Maybe it's an open source model, maybe it's an IBM granite model." Um, but we're going to do the assessment and then we're going to choose the best way to execute that task. We're going to assess it from a performance perspective and a price perspective. and the developer doesn't get the opportunity to just choose and say, "I'm going to go run everything through Sonnet or I'm going to go run everything, you know, through this new Astro model that I'm I'm crazed about, right, for what I'm doing dayto-day."

10:05 And so, I think it's very interesting when you start to consider, you know, the example that Chris gave of like I play with these models. I'm into these models. Like he I think he called him I'm a model addict, right? In terms of, hey, he probably has a priority order, right? first, second and third of the models um you know [clears throat] that that you know that are used to execute certain tasks and that that are played with day-to-day right but when you get from a business perspective right and looking at what

10:33 enterprises really need to use from a technology point of view it becomes less about the model it becomes more I think about having a framework an architectural framework that embraces many models that can be used for many different tasks but we have to take it up a level to say how are we optimizing for the right level of price and the right level of performance to get those things done.

10:55 Otherwise, you end up in what I call an Uber situation of millions and millions and millions and millions and millions of tokens being spent by those developers who preferred a, you know, one model or another, preferred to use one vendor or another. And at the top of the food chain, the person having to pay the bill is saying, "What did this do for us again?

11:15 someone help me understand. Right? So, that's a little bit of the lens I think that that I take to this day-to-day because I I think in my mind I spend so much time on what businesses struggle with in order to get many of these newfound capabilities that we're all really excited about. I'm really excited about them, too. Like, these models are cool.

11:35 They do cool things. But what does it mean, right? And what value does it actually deliver um to a business? [music] Anthropics uh release of its latest version of its midtier model, Sonnet. So, so let's get mid, guys. Um I was surprised to see news about Sonnet at all. And and first of all, I think I I think I'm probably speaking for a lot of people when I say that it's getting really confusing to track all the different model names and sort of brand families.

12:08 And you know, in the OpenAI universe, uh it would seem that they're going for like an astronomy theme. They've got soul and Astra uh in the in the uh anthropic universe. They've uh they're going with poems and types of narratives, right? So they have haiku. For a while it was haiku versus sonnet, right? And then I feel like not so very long ago they said but if you really want to do overkill we have this thing called opus.

12:33 And then you started hearing about mythos and then mythos became fable and now there's you know there's sort of brands within brands and numbers within within those brands. Um, from my point of view as a consumer, and this really speaks to what Madison was saying, I find there to be, um, I find that the cost of throwing less intelligence at something with something I'm just vibe coding at home and then the the process breaking feels to me like it exceeds the cost of throw the overkill, right, of like throwing too much

13:02 intelligence at something. So, I'm regularly throwing fable at things that probably Haiku could handle for all I know, but I don't want to hit that dead end and have wasted time. Uh what's interesting about sonnet here is again it's the second lowest in my understanding of the in the sort of anthropic universe but uh it's performing really really well and I think they ran some benchmarks and on on certain tasks I think set even outperformed opus and I could be wrong about that but um it's it's certainly approaching a

13:32 sort of asmtoic you know equality. So, first of all, like let's start with Chris. What does it mean when the mid-tier models are beginning to catch up to the top tier models on certain tasks or in some cases outperformed? Is that is that like almost like a a branding error on some level? Are we are we reaching an era where this sort of tiering doesn't even quite make sense?

13:56 >> No, I I think it still makes sense. Like I mean I tend to see the middle models as kind of your workhorse models, you know what I mean? So like let's take the Sonic 55. It's really good at agenda coding. It's it's going to get through. So and know or if I want to do a lot of orchestration, it's going to do a pretty good job on that. And again, I would argue the same with um you know, open AAIS ones.

14:20 Although I tend never to go beyond below soul um which is which is a really interesting piece. So I I don't know. or soul is somehow kind of turned more into a mid-tier model in that sense. Um but you know that's that's kind of fine but you know if you want to let's say if the tasks are a little bit simpler you want to burn quite a few tokens cuz you you know you're going to be hitting a lot of files or whatever I think they're turning into that.

14:46 So whereas probably up until Opus 55 I would have said you know whereas when you're on the higher tier models you're you're burning through your tokens really quickly right. So I mean if I use a mythos model for example or if I'm using Astra I'm I'm really burning through my tokens just like what Madison was saying there right? So I might want to do some task but I know that if I run a mythos or an Astra style model at that I'm I'm going to burn my tokens so quickly for something that probably a sonic could have done.

15:15 So you know and and this is why I do like the example Madison's given about things like Bob where it's automatically uh choosing models for you as long as that algorithm is good um then you are going to get the the right level of compute for the task task that you're performing whereas I mean today if I'm using cloud code if I'm using codeex etc I am doing mental gymnastics right I am saying well you know I I need to get a lot of coding here but there's not a lot of intelligence I need for this so I'm I'm going to burn

15:44 some sauna on that and then whereas if it's a really high-end task, it's like I really want an amazing design. I want it to come together. I want a story, then I'm I'm now burning on Astra. So, so I I don't think it's a mislabeling. I think it's a reaction to the industry saying, you know what, I' I've just got a lot of work that I've got to do and I can't run out of my token limits or I don't want to spend that cost of tokens.

16:07 So they they need those mid-tier models to be running at a better uh cost ratio and and I think that's what what you're seeing and you can kind of see that in some of the stats earlier. So if you look at um some of the fable releases right so if you look at enterprise I think it was like less than 5% of usage I can't remember the exact stat maybe it was 5% 6% of usage um for the models and most of the enterprise tier customers were sticking with the opus class models or the sonic class models and it's exactly to

16:37 Madison's point which is people don't want to be burning or paying a cost for a high-end model for a small task so I I I think this is where you know the industry is focus focusing a lot at the moment, which is how can I maximize my intelligence with the lowest cost? >> Uh, Madison, maybe I'll um behave like IBM Bob and and route [laughter] uh route a question back to you.

16:59 Um hopefully I've chosen the right expert module here. Um my bias again is more like as a consumer just in terms of what I experience. Um, so in the Chris is talking about burning through tokens and I actually was a UX researcher for many years and I found the UX the user experience of managing one's tokens budget to be one of the most underdeveloped features of at least these desktop models.

17:25 Um, it's interesting. I from recollection at least in the desktop version of Anthropics Claude, uh, you can do the model picker. Um, but it doesn't tell you much about token burning. And if you want to see how many tokens are burning, you have to do three clicks to go into your settings and look at your usage. The one exception is that if you choose max thinking, right?

17:48 Because yeah, there's four tiers of models like a vertical sort of axis and then there's this horizontal axis of how long it's going to kind of chain of thought itself. And the only like conspicuous UI um you know user interface that I've seen is um if you choose max, which is I think the the fifth most extreme tier of of chain of thinking I I if I understand then it says warning 3.5x usage.

18:10 I find that really interesting and it's also accurate. It does seem like that is the thing more than the vertical axis, the horizontal axis is what's going to burn your tokens. I'm curious from an enterprise standpoint from a you know if you're the cloud fin or the AI you know ops person who's looking at the bill at the end of the month what is the UI and what is IBM and and even its collaborators competitors what are they doing to kind of help people have a dashboard to monitor token burning as they as their

18:40 enterprise goes through through the month's kind of token bill. >> I think that's a that's a great thought a great question. Let's start kind of at the beginning. Um, we just talked about how usable, right? I'm going to use the word usable, right? Usability, um, of open AAI and Anthropic, right, that they've made their products. And then you just gave a contrast and a comparison of how many clicks that it took you to figure out how to manage your token consumption.

19:06 Let's make no mistake, these companies know what they're doing from a productled growth perspective. They're some of the best in the industry on it. They have consumers across the world leveraging their models. if they wanted to create an easier way for you to manage your token consumption or have visibility to your token consumption, I'm quite confident that they could figure out a way to do that.

19:28 So, when you think about the types of technologies that are good for business, right, again, like the single model provider doesn't really want you looking at your token consumption all the time. They just want you to keep consuming. So, take Bob for example, right? this concept of Bob coins and true token rationalization, you get an opportunity not to say, I'm going to give Ash 20,000 tokens and David 20,000 tokens and Chris 20,000 tokens and everybody has to have the same amount of tokens across the board.

19:55 All of a sudden, we have a pulled model where I can say the work that Ash is doing, we actually know is going to consume a lot more from AI. So, I'm giving Ash 40,000 tokens, but David, for your job and the project that you're on, 10,000 tokens, and if you need more, we can pull from an overarching enterprise pool of tokens in order to give people what they need to get their jobs done.

20:18 But frankly, regardless of the models that you're using, regardless of the agents that you're using, people should have an AI governance framework in place in real time to see where the token consumption is coming from. What models do you have that are powering the task in your business that are consuming the most? And moreover, [clears throat] a good AI governance framework is also going to connect that to value, right?

20:40 And mapping to the target value that you're expected to get from those projects, from those use cases, from where the AI is being used, right? And there is technology out there today, right, that IBM has that it gives a very clear pictorial on this, right, from a a PHOps perspective, whether you're using things from the IBM platform or whether you're using OpenAI or Anthropic or Microsoft or Google or Amazon, take your pick, right?

21:05 Um, of those types of models and agents coming through. So again, I would just summarize that by saying, you know, I think AI governance is underrated in terms of the speed, the scalability, and the level of visibility to managing some of these things that are really hard from a consumer and a business point of view. >> Wonderful. Sorry, just unmuting there.

21:25 Um Ash, is there anything you want to chime in just with a final note on on 5.5 sign of 5.5? Anything that stood out to you? >> I do have a uh a a benchmark that I've run. I at one point I went down this road of like getting hold of prompt fu and setting up my eval suite and whatever and I was like this is too serious and it's not fun enough and I want me playing with models to be fun.

21:46 So my benchmark is to say generate me an SVG image of a ninja cat in Jersey City cuz that's where I live. Okay. And I run that. And what I normally do is when the models come out, I'll run it on Opus Extra and Opus Medium because the uh as you mentioned earlier in the uh in in the chat, David, when they announced Sonic 55, they were like, "Oh, you know, the performance sometimes will exceed Opus and so forth, right?"

22:12 And that's down to uh the effort levels, right? And the effort levels obviously influence sort of like how well the model's going to perform. And so I'll run my little benchmark against various permutations, you know, like Opus 5.5, high effort. What does my ninja cat look like? Sonic 5.5 medium, what does my ninja cat look like? And um yeah, I would I would say that yeah, Opus 5.5 extra, best looking ninja cat I've ever seen.

22:38 Um Sonic 5.5 medium, it's kind of okay. Um I I would say that as Chris said, it's definitely a workhorse model. it responds very quickly. Um, and it's uh very good for like doing sort of agentic coding tasks. I mean, I played with it from that perspective as well. Um, but yeah, it's like it it it's it's it's a good incremental improvement on what Sonic 5 was before.

23:07 [music] Let's talk about Meta's Muse. Um, I think this is in some ways like an off-brand story for IBM because it's really quite consumer focused, but I've been around the block for a couple decades now. And I remember reporting about how when the iPhone came out, suddenly that transformed enterprise. Like I don't think in 2008 um Big Blue was anticipating that they would have to accommodate my addiction to this thing, right?

23:34 And like let me have slack on it and so on. And so my contention is that um Meta's Muse, which is Meta's uh um a agent, which topped the apps app charts last week, uh which no one was quite expecting, um could in theory in the long tale have a similar impact on enterprise. So my my thesis my hypothesis let us say is that if this kind of cuddly form of agentic AI that already kind of exists in places where you hang out like Instagram and Facebook if that kind of low friction um doesn't even require the technical savvy

24:15 required of vibe coding kind style of agentic AI if that becomes a dominant paradigm especially for a young generation coming up or going through college um if college remains a thing and entering the enterprise soon that that could create norms that in turn are felt in enterprise. It's it's it's kind of a long horizon and a big you know a hypothesis and it may be proven wrong.

24:39 The thing that made it special or or made it interesting for me was then why I think this makes a big difference versus what some of the other uh AI providers are doing is is that Meta's making it very very accessible for you to have an agent on demand that can do computer use seamlessly for you. Okay. Like there's like it's like taking like open claw and what was open claw and making it even more accessible and even more like a lot of people were getting open claw and then going installing it on their Mac mini and

25:13 then like connecting it up to WhatsApp and all the rest of it. It's like meta just went you know what we'll just like take the whole thing and package it up really nicely and we'll put a cuddly person on top of it and we'll just open that up to the world. And yeah I I honestly I really really am super impressed with Muse. been using it lots for other things as well like planning trips and anything that I can do asynchronously to increase my productivity that needs like computer use and browsing the internet I just give

25:40 it to Muse and say go do it in a world where 20 year olds start using Muse as their preferred way to pursue vigilante justice as Ash did last week and become uh you know self-deputed as a cop uh with palunteer-like powers or surveillance. What happens when that 21-year-old grows accustomed to this kind of packaged frictionless agentic experience and then comes to work at IBM or any or any IBM client?

26:08 Um, what do you think, Chris? >> I don't think this is just a consumer experience. By the way, I think this this the news experience is really the experience we've been waiting for a while. And um I mean I'm just jealous that unlike Ash I'm still in the UK so therefore I don't have the pleasure of using Muse cuz I'm happen to be in the wrong country to go and play with the cool toys.

26:31 But you know um I'm very aware of the experience and I think this is a general direction where agents are going to go. they you want your agents to go off and do things um on your behalf whilst you're going off and doing other things as opposed to you know sitting like Homer Simpson and just having your finger do that on the keyboard while you wait.

26:49 So I think I think that's great and then I think you're going to see I mean I think OpenAI has just released their dots as well which is a very similar thing and then Ash mentioned OpenClaw you know actually underneath the hood of all of these things is open claw anyway so even Meta said that was the case so I I think these agentic swarms these um offline agents that are going to go and process things in your behalf are the key how does that translate for these uh 20 year olds at the move in enterprise well guess what

27:16 we're all going to have to do the same thing in enterprise as Oh, right. I mean, I've got my agentic swarms that I run. You know, there's a there's a a big conversation piece on that one, but you know, when I want to get a piece of work done, I set off my swarms and, you know, I'll maybe have 100 agents go off and perform a task and come back to me and then Madison shouts at me for uh what my token bill is going to be.

27:36 But but the reality is you're going to you're going to hand off those tasks and it's going to it's going to get done. And and so I rather than saying how are they going to feel when they come into the enterprise actually the consumerf facing experience is where the enterprise is going to go. Of course you're going to get an enterprise version of that but but this reality is pushing forward and and therefore things like the wearables and uh you know and being able to have those things in your pocket and you know always

28:03 own capabilities. I I think this is the future. We're just getting the early glimpses of that just now. So finally, Madison, in an era of agents everywhere in everyone's pocket, what is that going to feel like in the enterprise? And how does how does that what does that make you think about as someone who's, you know, very senior in the Watson X universe?

28:22 >> Yeah. Well, I go back to one of your first questions of, you know, how are these 20-year-olds that come into IBM going to feel about this if they have that experience in their consumer day-to-day? I think they're going to expect a similar experience in the enterprise in terms of ways to solve problems, right? and what it looks like in terms of the experience that they can go to to get help or to complete tasks.

28:44 It's got to be a familiar place for them. But I think one interesting thing if we talk about the makeup of Muse and what Meta did before they launched this, right? There's everything from things like Agentic Payments with Plaid. There's Shopify integration that's there, right? All things that are very key and core to making my life easier to uh well frankly to acquire material things, right?

29:09 Less clicks, less click of a button for me to do that. Um but a lot more intelligence about the products that I'm acquiring and things that might fit within my portfolio, right, of what uh you know what an AI app like like Muse knows about me, right? Um so you know again I think those are actually all very relevant things for business right think about IBM right we still process 99% of the world's transactions that go on out there right we have deep expertise and technology in the payment space so all of this

29:46 eventually is going to move to agentic and it's not all going to be agentic but a lot of this will be somewhere in between right something that is highly deterministic and highly guarded to things that will be fully identified and then many somewhere in between right of what I would call an agentass assisted workflow or an agent assisted task. So I think it would behoove us to actually embrace it more than we uh push against it because I think that's how people are going to come into a company like IBM or the

30:20 businesses that we serve and that's how they're going to expect to interact and solve problems. And I think those capabilities as we talked you know about earlier on in the podcast what people learn as a consumer and what they're doing with these tools they do transcend into the business. people develop natural biases toward ways that they solve problems where they go right when they have a question uh to their favorite model to their favorite experience or or one or the other.

30:46 And so I think all that transcends into how they do their day-to-day work. And we need to embrace that. We just need to figure out from a business perspective how we do that in a well-governed way, how we do it in a safe way, how it's secure, how we're compliant. um in terms of how we do that. And I think again I'll go back to I think things like AI governance is one of the most underrated things today that can produce a ton of value for a business and a ton of value for the enterprise.

31:13 And so instead of just rejecting these types of experiences or rejecting these types of tools, let's figure out how to embrace them and and the responsible practice thereof. >> Well, um Madison, Chris, Ash, thank you so much for joining me today. Um I look forward to seeing you all on the show again soon and thank you all for listening. [music] You can find us on Apple Podcasts, Spotify, YouTube, wherever you get your podcasts. We [music] will see you next week on Mixture of Experts. >> [music]