So I've been testing Gemini 4... Transcript, AI Summary & Key Points

00:00 It finally happened. Gemini 4 argan is here. Before we get started, I got a quick question. How long do you think it has been since we got a normalsized large Gemini model release like the Pro line? How many months do you think it has been? The gap between a model release like 56 Soul and 6 Astra was 1 to two months. The gap between 31 Pro and Gemini 4 Argan is 8 months.

00:18 It's been since February. We have not gotten anything other than flash models from Google. And now, after teasing Gemini 4 Argan over a week ago, they have finally put it out for all of us to use. With a drop like this, there's a lot we have to cover. From how it's priced to how good it is at front end to all the 3D capabilities, the weird things that make Gemini models Gemini models, the unique capabilities in those models, and of course, the comparison with recent releases like Opus 5.5, GBT61 Soul, and Muse

00:49 Wonderful Full, which if you haven't seen my video on that, I am quite enjoying. A handful of my friends at Google actually stuck their necks out a bit to get me early access to this model. I'm very thankful for them and I sincerely hope they don't get in trouble for what I'm going to say because there are things that are good about this model and there are things that are less good about this model.

01:05 Normally, this is where I'd make a joke about how I'm not a paid chill and the company isn't paying me. But in this particular case, I can't because Google does pay me quite a bit because I'm a YouTuber as you might know watching this video likely on YouTube. All of that said, they are not paying me for my coverage for anti-gravity and Gemini stuff.

01:18 In fact, my previous Gemini videos almost got me demonetized on YouTube. So, uh, this comes at a cost to me if anything, which is why I hope you can forgive us for a quick break for today's sponsor. I got an interesting question for you. You want people like me to use your product because if you do, you probably shouldn't do a normal sign-in flow because I'm not going to be clicking those buttons myself nowadays.

01:37 I do most of my work through my agents now, which means any platform that my agents choose needs to be usable by the agent. In particular, it needs to be able to be signed up for by your agents. They sponsors work OS and that might confuse you because they are a platform for user signing in right well they are. They are the best way to have humans and businesses signing into your app.

01:55 So if you're trying to get enterprise customers then you should probably be using work OS. That's why everyone from OpenAI and Enthropic to FA Cursor Bolt Vanta Carta and more are all using them. But I want to talk about OMD because this is a super super cool standard. The goal of OMD is to make it easier for agents to sign up for your apps. It's a new standard that's been designed by Work OS in partnership with companies like Firecrawl and Cloudflare.

02:16 And the results speak for themselves. There's already a ton of companies adopting these standards like Neon, Monday, Parallel, and more. What makes it so cool is that agents can now register on behalf of a user. So, if you're trying to get business customers and you want their agents signing up, this is a great place to start. Get offright for all your users at soy.link/workos.

02:35 Now that that's out of the way, let's dive in to the newest model from Google, Gemini 4. Flashbang warning for their website because it is not dark mode for some reason. Argan is now globally available today for developers and enterprises via the API and studio and Gemini platform. Google AI Ultra subscribers in anti-gravity and Google AI Ultra and Pro subs in the Gemini app and AI mode in search.

02:54 This is God I did not realize how bad Google's like product line had gotten until this. It's a mess. model excels at deep work in complex real world tasks across domains including coding, finance, legal, cyber security and science. Its depth and precision across multi-step tasks lead argan to outperform other frontier models on benchmarks across domains ranking number one on the VL index which measures model performance across critical economic sectors like finance, coding, legal and tax work.

03:23 And yes indeed Gemini 4 argan does appear to be one of the best values here. Somehow they have a cost for Wonderful though, which is strange cuz I was never given token pricing. Okay, they're saying it's $4 in and $20 out. That's wrong. We don't have a cost for Wonderful and it won't be the same price as standard Wonder obviously. So, uh, that makes me trust this a lot less.

03:46 I'll be real. Regardless, I have my own benches. I've actually run a lot of benches on this model. I'll start with some benches that aren't mine with Slopolytics. I already shared the numbers for Gemini here. You can see that its cost is not actually that great when you factor in how many tokens it uses on something like the artificial analysis intelligence index, but its score is putting it above Astra and also above 61 soul, but meaningfully lower than Opus 55 on high as well as obviously X high and max.

04:14 And it is also more expensive than Opus 55 on high despite getting a lower score. Of course, artificial analysis intelligence index is not a perfect measure of model capabilities. It is not the best index for things like code, but it is a good general idea of where intelligence sits. I chose to show you these numbers first because I'm scared people's brains fall out whenever I show them the model price.

04:36 Because if you see this $2 per mill in and $10 per mill out, your immediate assumption is going to be this model is cheap. When you compare it to things like the 525 rates for Opus 5.5, this seems like a really good deal. It's not. It's a combination of reasons. The first one is that this model is it's token hungry. It's not particularly efficient, which is part of what puts it in the higher price range like we just saw in artificial analysis.

04:58 The other really, really egregious problem is that it sucks at hitting cash. I have had the worst rate of cash reads from any model I've used recently with this release. I'm talking like 70 to 80% cash hit versus other models where I see 99% plus. It means that that 95% discount on cache reads is not going to happen as often and you're going to actually eat a lot of input cost on this model.

05:24 This might be a bug in anti-gravity either in how it's actually sending things or how it's reporting data to me. It is slightly possible this is a bug in the MCP implementation that we use in T3 code because you will never catch me dead in their awful CLI. But I am not particularly confident in any of this and I am not interested in doing more work to figure it out because Google should be the ones doing this, not me.

05:45 They have a section about using it to solve complex engineering tasks, saying that uh anti-gravity has helped teams increase agentic code submissions by more than 35% while reducing code rollbacks by nearly a third. Uh those are not great numbers considering that what they were using before was also uh not particularly good. The more interesting thing, and I won't air out who this was, but I was talking to a Google employee and not one of the ones that Google knows I talked to, so don't go after the people who normally

06:12 work with me. The way they described Argan to me wasn't it's a revolution or this model has changed how we build. The way it was described to me is I find myself reaching for Opus 5 less. Not 5.5, Opus 5. The one that I notoriously panned after realizing I was wrong in my first video. It is what it is. Let's see what this can do. At least according to Google.

06:34 Apparently, they use it to untangle browser feature code in Chrome, piping multiple clean code changes at a time. Argan helped the team finish an architectural refactor across hundreds of files, completing months of work in weeks. This feels like it was written in 2024. It optimized a production Python codebase in just a few hours. Here they have some fun demos of the cool 3D stuff they've been doing with it.

06:58 God, why is it making sounds? That's horrible. This isn't a generated video. This is generated 3D models that they then went through with some type of like software rotoscopes type thing to show the models in video form. Very cool. Animations are still a bit cringe. This YouTube player embed is broken and the overlay won't disappear. Classic Google isms.

07:21 I I've joked for a bit now. You'll know when Google has a good model when their software finally starts working. They got a quote from someone at Deote. Early impressions from Gemini 4 are very strong for long-running tasks across these workflows completed successfully 100% of the time in our evals maintaining context and running through multiple tools across impressive spans.

07:43 In depth of reasoning delivers significantly higher clear yada yada the text clips Google being Google I shouldn't be as annoyed as I am by this completed successfully 100% of the time in our evals thing because this depends entirely on what's being evaluated. For example, I have an eval that Gemini completed 100% of the time. It's skatebench. I still actually can't believe this.

08:05 I think it's cool as hell. Gemini 4 argan got a perfect score on my private version of my benchmark for seeing how well models name skateboard tricks. For reference, Opus 55 on Max gets a 79%. This bench has proven to be weirdly difficult for most models to do. In particular, some of the tricks that are like switch and fakey stance cuz skateboarding has a whole grammar around the way you're standing on the board.

08:28 Google has always had a weird lead here with models like 38 Flash getting a 96%. But seeing the first perfect score was kind of trippy for me, especially when I looked into it more and saw just how many errors it was hitting when it ran because the API was quite unstable during my early testing. I don't know if that's improved since, but I was getting pretty intense error rates and had to overhaul a bunch of my benchmark just to get around the fact that the endpoints errored as often as they did.

08:53 It was also relatively impressive in terms of its token utilization in the bench. Like it got answers fast and didn't overthink, which is rare for a Gemini model and nice to see. So yeah, if your task is somehow reminiscent to naming skateboard tricks, there's a decent chance you're going to have a good experience with Argan, both in its like speed, token efficiency, but also its accuracy and doing the thing.

09:13 But again, when you say a model has 100% success in our evals, the eval and its relation to the work you're doing matters. And I don't think a lot of people are making a living naming skateboard tricks. I wish they were. I wish I could have that be my job, but uh sadly it is not. They also say that the model has advanced reasoning and multimmodal capabilities while maintaining strict grounding, yada yada yada.

09:34 It's good at looking at documents, PDFs, videos, audio, things like that. I have historically seen a meaningful lead in Gemini models ability to like do things with vision, like giving it a picture of a motherboard and asking it to label the parts and draw that in like Python on top of it. The Google models have historically been very impressive in this space.

09:50 I did not benchmark any of that because I was just using it for code. But if you have complex vision tasks, it is definitely worth running them against Gemini 4 just to see if you have a real benefit. You might be surprised. Do note though as we start to complain more about GoogleMS that the current price of $2 per mill in and $10 per mill out is an introductory price.

10:11 This is Google's new favorite thing to do when they drop a model. They charge a reasonable price up until a certain date and then they double the price after. So, this model's actually going to be $4 per mill in and $20 per mill out starting December 1st. It's almost like they don't want you to start building heavily on top of the model. It's It's rough.

10:29 They've been doing this for most of their model drops recently and it makes pricing weird. I I don't think it's as bad as what Meta did by not putting out a price at all for Wonderful and only having it in the sub, but I still hate this. Just give us a price that you can actually serve it at. lower the price in the future. Don't raise it. The only lab that regularly raises prices for models is our friends over at Google.

10:49 And then they tell us to get started. So, I guess we should. First, we'll start with one of the things Google models have historically been pretty strong at. The thing is, of course, front end. I have used Google models for front-end stuff pretty often when they are the new thing. Specifically, when 3 Pro and 31 Pro dropped, I found a lot of good uses.

11:04 I built a lot of the UI and UX around Lawn, my alternative to Frame.io for video stuff. It's what we actually use for all of our video management for my company. Every one of my YouTube videos gets uploaded on this platform and managed by my team online. And a lot of the UI and UX here was built by Gemini 31 Pro. I obviously did a lot more work on top of that using the Opus models at the time, but I found that the back and forth between those models allowed me to make actually uniqueish homepages at the time.

11:35 So, let's see how that capability has carried through. I think this is quite good. I don't necessarily think a homepage should have this much text on it, but it is very nice looking. I would not have guessed that this was AI generated if I just stumbled on a page like this, like browsing the web. It also made a nice little interactive section here where you can do things that is actually functional and not cringe.

11:58 Like this is decent. Believe me, I would talk a lot of crap if I could. This is fine. Hop into the other designs. That one had a bit of like font popping. That might just be the way that Witch AI is set up. By the way, whichai created by our friend Dra, really cool kid. Check it out at whichi.dev. Very useful to see how the front end capabilities vary across different models.

12:16 All of the versions I'm showing now are using the official anthropic design skill. So, if you aren't getting good designs out of the model, definitely try turning that on and see if you have better results. If I turn it off, you get a lot more cringe, a lot more of this like unnecessary text everywhere. A lot of fonts that are kind of gross, just not the best quality.

12:39 Like all of them have that like awful topnav. Like every single one of them does. God, this retro one is so cringe. And this is without any skills on. So if you are using this for design, highly recommend Anthropics design skill. I don't recommend the taste skill though. that gets it back to where it was with the weird cringe top navs and as well as like the really consistent gross layout.

13:00 The design skill seems to be a good balance that actually forces the model to make things that look decent and unique. So for front end seems pretty good. Is it state-of-the-art? I still think Fable 5.1 is the best I've seen. Opus has its moments, but Fable overall makes the coolest, most unique front ends and like homepages, marketing sites, that type of thing.

13:18 I still don't think any model is good at just like doing things right the first try and I prefer ones where I can go back and forth. For example, with Slopolytics, I used OpenAI models for this entire site. I did everything with Astra on ultraast. Not because ultraast is magically better at design. All open models kind of suck at design. But since I was using ultraast, I could tell it what I wanted and ask for different versions and go back and forth until I got it just how I wanted without having to wait constantly

13:45 throughout. So, I think that speed is really important when you're iterating on real UX, not like a marketing site, but like the buttons people press on a website and how they interface with it. Google models are fast, but that doesn't necessarily mean they get work done quickly. Anecdotally, having tested this a bunch myself, it's not a super fast model.

14:03 It might generate tokens quickly, but it doesn't solve problems fast. Next, we should talk a bit about the unique capabilities of Google models. This is strange because every lab kind of has something like this. For example, with OpenAI models, you can generate images and it will use their API to call the image genen tools and models that they have.

14:23 Where with cloud models, you can't. It would have to call some other external service. Grock models have the ability to do cool things with the Twitter API. Like if I want to get good data out of Twitter, I can't do it through the API. It's way too expensive. So, I'll just tell Codeex or Claude to call Grock. And by calling grock, it gets access to the XAI APIs.

14:39 A lot of different models have these types of capabilities. If they have access to unique data, you can get it through that model. So why I bring this up? Gemini models have some unique capabilities, many of which are very useful to me. Why are they useful to me? I'm a YouTuber. That means I have to do a lot of stuff on YouTube. And Google gives very generous access to the YouTube API and capabilities to the Gemini models.

15:00 If you want to get a transcript from a YouTube video, the Gemini API is one of the best ways to do it right now. And we've been using the Google models heavily for all of those types of things throughout my businesses. So I will be using this model a lot. Not because I necessarily want to or I'm choosing to because it's so uniquely intelligent and such a good value just because Google has unique capabilities that they are providing their models through their APIs, through their data centers, through the things that

15:25 they uniquely have access to. That said, I've actually found the Gemini models not that good at most Google services. I have a much better experience dealing with my email inbox through tools like Grockbot and Chatg GPT than I do with anything Gemini. In fact, every time the Gemini button appears in some app, it scares me and is often not functional or just outright cringe.

15:48 I think if you're a YouTuber or you work with YouTubers, Gemini will be a very useful thing to have in your pipeline. If you're doing a lot of unique vision work, historically, the Gemini model has been good at that as well. So, worth benchmarking. I don't think there is as much here as there used to be though cuz a lot of the things that Google's models were uniquely good at have been maintained and achieved by others externally.

16:10 There is some secret sauce in this model. There's something that makes it different. You can see it in Skatebench. It's the only model to get a perfect score and the only models to break the 90 range are Gemini and somehow Muse Wonderful. Everything else is in the 80s7s even 50s and 60s. So there might be more secret hidden capabilities I'm not seeing in the model.

16:33 I've heard people say that they're really good for internationalization, localization, multilingual stuff. I haven't seen too much of this unique capability in my own testing. I just wanted to call it out because it's not that common nowadays. Usually since all the models are just throw all the data in pre-training and then RL it on code, we don't get as much unique capability in things anymore, but Google still has some.

16:53 And I thought it was worth noting. Now, we get into the meat. The things that y'all are most hyped about. Things like the real day-to-day code work, the things I hate about Gemini itself and the Gemini ecosystem, Opus, Soul, and Muse, and how they compare, and of course, the thing everyone has hyped about, weirdly so, Fishlop, my game that I make with every model to see how good they are at 3D.

17:17 So, uh, without further ado, I think it's time for fish slop. And no, the video is not encoding incorrectly. It did actually screw up the top of the water that badly. The UI is uh it's Gemini to say the least. It did make the whole game loop where there's five tanks to get through which is solid. Submarine models not particularly great. It's not as bad as things like Muse and Grock, but it's not great considering how crazy all the 3D demos I saw other people posting were.

17:46 I expected a lot more. I gave it the exact same prompt, exact same setup, exact same tools and everything as the others had. It does animation decently well. Like the way the fish moves is reasonable. The eyes are pointed in the right direction, which might not sound like much, but just a few months ago that was a rare skill. But there are some uniquely Googleesque things here.

18:07 Let me show you one of the funnier examples. So, games have to have physics and mechanics. My game has a mechanic where the fish eat the food. There's a problem. The food can fall to the bottom of the tank and when it does, it doesn't expire. The fish can go to the bottom of the tank to eat it, but they can't figure out how. So, the fish will come over cuz they're hungry.

18:32 They'll try to eat the food that's on the floor, so they can't. And then they'll just sit there staring at the food trying to get to it until they die. This wouldn't be too bad if I could just like feed them more. But the game has a mechanic where you can only put so much food in the tank at a time. And it forgot to make these expire, so it will actually just sit here forever.

18:50 I also just realized the sound is not routing. So, let me fix that quick. There we go. Sorry for the lack of sound for the first bit. As I said, you weren't missing out on too much. It's not particularly great sounding game, but this version was bad enough and broken enough that I wanted to give it one more shot. So, I did. I told about some of the things that were broken, like the shimmering on the ceiling of the tank, the food mechanic being broken where the food wouldn't expire and they couldn't get it at the

19:17 bottom, and that I was disappointed in the graphics, and I wanted it to improve. So, it ran for a while. It came back already looking a hell of a lot better. Didn't touch the model for the sub, sadly, but it did fix a lot of the things involving like the feeding of the fish. And I was able to actually like play this version for a bit, but then you start to notice some of the the Geminiisms.

19:44 And the best I can put it is like it doesn't have big model smell. Big model smell is really hard to describe. It's a a type of thoughtfulness and generalization that models do when they have like a better understanding of the world that comes from the model being bigger. I talked about this a bit in terms of 3D in my previous video about Muse where I talked about how the models 3D capabilities do not reflect how smart or capable the model is.

20:11 3D is a thing that they train in separately like they add it on. This model clearly got more time with that 3D stuff, but it also has some stupid mistakes that I haven't seen other models do. Monsters spawn every minute. When you finish the wave, they drop a power up and the power up lasts for 30 seconds. Why is there a power up that helps with my attack?

20:34 I can't shoot anything after I get it and they expire before I can use it. So, one of the powerups is legitimately useless because there will never be a monster in the tank when you have it on because it will disappear before the monsters come in. is just like the type of silly thing that a human would not have done that a smart model also would not have done that a model that's goal is to just write code that seems good without thinking about it would absolutely do.

21:03 Imagine that you were a part-time contributor to a game like this and there was three power-ups made. You might throw the idea of a power up that does more damage and not even think about it and then add it. A smart model should have a good enough understanding of the system to know that that is stupid. Gemini doesn't. I think now is a better time than ever to start talking about the way this model behaves in real world tasks.

21:22 In particular, hallucination. The thing I just showed you wasn't hallucination. It's not like the model imagined this thing that wasn't the case. It saw a powerup system. It came up with an idea for the powerup system and it put it in the game. It might be a stupid powerup. Doesn't make sense, but it didn't hallucinate anything. According to artificial analysis, the model actually doesn't hallucinate much.

21:41 On their AI omniscience benchmark, which is their attempt to measure hallucination, Gemini 4 argan had a 15% hallucination rate, the lowest of any model scoring over a 45 on the intelligence index. And yes, indeed, you can see here in the official charts, Gemini 4 argan has an impressively low hallucination rate. But the AI omnitions bench is not just hallucinations.

22:03 It's both incorrect answers as well as partial answers and refusals. So, if it gets the questions correctly, it gets points. If it gets them incorrectly, it loses points. If it says, "I can't answer this," it's neutral. When you look at the overall score, it's no longer first place. It's now in fourth because it gets few enough things right that it's refusal to get things wrong does not really matter as much.

22:26 It's just like, "Sorry, can't help more often than others." Cool. Nice to see Google making their models hallucinate less. Still not a particularly good score. Sadly, from my experience, this benchmark is not that realistic because I have seen this model just be wrong a lot. Here is where we get into the real code work and where my actual breakdown of the model begins.

22:50 I started by throwing it at TS Rust, a project I built I talked about in other videos where I rewrote the entire Tyson compiler in Rust using Opus 5.5. I had a bunch of different models audit the codebase and describe it things that could be improved. What is it worth knowing in this codebase? I would take all the claims it made, validate them, and then put them through a panel of judges to rate how good or bad the model did.

23:15 GBD61 Soul came out on the top, making 38 claims that were all true. It had zero wrong claims, and the claims were informative, accurate, and useful. So, all the judges gave it a pretty good score, averaged an 8.84. It was 5'5, slightly lower, 878. It took way longer, 49 minutes versus 28 minutes, and it cost $24 instead of $4. And it got one of the claims wrong.

23:35 Still a very good score. It just made some mistakes and took too long and was too expensive. Six Astra also had no wrong claims, was way faster at 20 minutes, still a bit too expensive. 61 Soul on X high, things improve. Also funny, uh, Six Astra on max scored worse than six Astra on XHigh. Once again, as I've been saying for a while, you should probably avoid the max reasoning levels.

23:57 They do more damage than good. You might notice something's missing, though. Where is Gemini? Well, we hadn't scrolled down far enough. Yeah, Gemini for Argan in my audit the codebase bench scored a 4.38, less than half of Opus' score and 61 Soul's score. It also took 43 minutes long, almost double the time 61 Soul took and more than double it than it took on X High.

24:24 On the bright side, it would have only cost $3. But on the less bright side, of its 36 claims, 24 were false. The majority of the things it claimed about the codebase were wrong. Do you understand how egregious that is? It means that when this model is doing real work in a real codebase, it's as likely, if not more likely, to be wrong than right. And this has lined up with my experience using it.

24:48 I I hate to be that guy, especially because I've likely burned some credibility with a lot of y'all with my Muse Wonder video. I don't care. Muse Wonder is a unique balance here where it is getting similar scores to Fable way faster and gets things wrong, but 8 out of 33 is comparable to Fable 51's 11 of 33 and a whole different world than 24 of 36.

25:09 I'm going to be so real and you'll you'll see what I mean as we go deeper. I don't think anybody who finds this model on the same frontier level capability as things like Opus 55, Fable 51, 6 Astra, and 61 Soul, or even Muse Wonder. I don't think anyone who finds these models comparable is actually working on that reel of stuff. Show me demos if you disagree.

25:29 I would like to be wrong here. I want to have more faith in Google as a competitor, but the numbers aren't going to get better as we keep going. The next round was one where I audited a real bug in T3 code related to how the bulk remote update workflows were functioning. Opus 55 got the best score with an 822. It had five claims that were wrong if it's 46.

25:50 It was $26 to run. It took 20 minutes, but these are like good useful numbers. 61 Soul got a lower score. It missed some of the high severity things OP 55 found, but it didn't say anything incorrect. It took 10 minutes and it cost a dollar. Insane. This is why I like 61 soul so much as a review model. News Wonder scored quite a bit lower. I think it missed a handful of claims and it still had the same roughly error rate as Opus did, but it only took six minutes.

26:14 And most importantly, in my opinion, it was super readable. So again, I like Muse Wonder cuz it's a different flavor and finds useful stuff. Gemini 4 Argon is another flavor. It's not just a claw distill like a lot of the open weight labs are doing. So like if you talk to something like GLM53 Flash, Kimmy K3, or any of those other models, they all talk and act relatively similarly.

26:34 All four of these models behave entirely differently. Gemini is too dumb to justify the flavor change. Muse Wonderful Full is probably too expensive to justify it. We don't know cuz again, we don't have the price, but at least it's not lying 14 out of 40 times. Yeah, Muse has taken whatever I was hoping Gemini would do. Hopefully, you're starting to see why I was nicer to Muse than people expected.

26:58 Testing both the Muse model and the Gemini model at the same time really deeply reminded me of how much I don't like working with Gemini. Sadly, it gets worse. I didn't do a fancy visual for this one. This was an audit of my cloud product, Lakebed 61. Soul got a 9 out of 10. Son of 55 got an 8 and a half out of 10. Mostly cuz Sonet 5 on max will just burn and burn and [ __ ] burn.

27:21 So, it finds everything. Muse took a pretty big hit here going from the 825 from Opus 55 down to 5.75. It's not as big of a hit as Gemini gets. 2.25. These are scores out of 10. Muse gave its findings a three. Opus Fable and Astra gave its findings a two. Of its 78 findings, 56 were wrong. That is a 70% failure rate. This model is more likely to gaslight your agents than it is to give useful feedback in reviews.

27:51 If you do decide to use this model for real work for some reason, for the love of God, please put the things it does through Soul or Opus or some model that doesn't have this egregious rate of lying. I can't fathom how any bench measured this model as not hallucinating a lot when I don't h I've never seen it not hallucinate in my day-to-day realorld code work.

28:16 Then my final run where I had it go through the T3 code architecture, it got a 3.95 out of 10. And you can also see again Muse Wonderful being not like as good if not better than things like Fable and Astra. But neck andneck with Astra while also being much more pleasant to work with is worth noting. Again, to compare to Muse, it got 29 true claims, eight partially true, four false.

28:39 Argan got 31 true, 14 partially true, 47 false. Gemini 4 argan had more false findings than Muse Wonderful had true findings. What the [ __ ] It's not just like kind of bad or a little more likely to do something wrong. No, it's like an order of magnitude gap here. It is lying 10 times more than any of the other models I'm benching other than Muse Wonder, the non-fall version.

29:06 That one's just garbage. Throw it out. Don't touch it. But like the only bad models in this bench were Muse Wonder Standard and Gemini 4 Argan. And they're both garbage. As you saw in my Muse video, I'm a little bit more forgiving if the model's pleasant to interact with. No points here either, Google. Walls upon walls of text full of things that don't really matter that much.

29:28 Not very pleasant to read. But if the code works, that's fine then, right? Well, I had to do some work around the SQL like connection layer in T3 code in hopes of stabilizing it for heavy users on platforms like Windows where we're having some connectivity issues. I already knew a lot of the fixes. I was just curious if Gemini could find them. Said there's lots of user reporting issues with instability and connections.

29:52 A lot appear to be SQLite related. What should we prioritize to fix this issue? I even gave it a hint with async SQLite. This is not the right solution. The right solution is pooling and having read pools at a single write. It did a hunt. It did a lot of the classic Gemini isms of just reading every single file synchronously. It just kind of went on and on with that.

30:17 It does correctly identify the split read pool in a single write connection piece. I tell it to make a PR for priority one, which is the first section it had in here. Uh 30 minutes later, way longer than it should have taken. I've been told by some people that apparently Gemini is way slower in T3 code. You might think that's our fault. It literally cannot be.

30:38 We're using their ACP bindings that they provide on their servers that we embed in our app. We actually did a bunch of weird [ __ ] to make it work at all because Google's a special snowflake. We'll talk more about the Google special snowflake [ __ ] Regardless, 30 minutes is an unreasonable amount of time to make this PR, which uh when we open, we will see is closed because this change should not have been 1,200 lines of [ __ ] code.

31:03 This change should have been 30. At this point, I was done, especially when I read the PR, which by the way, we have skills in the repo for how to format the PR that are really simple. It's supposed to open with a simple two to three sentence description of the exact problem the user had that they're trying to fix. We don't get that. We get a summary.

31:20 We split the SQLite read access and write access in this particular file in the orchestration projection and query services that read snapshots and standalone select queries. Is this a run-on sentence? It is unreadable slop that didn't follow any of the repo rules. Just garbage. So, I was already pretty ready to close it. Then I saw how big it was. Then I saw my favorite thing.

31:41 CI failed. It just didn't write the code right. the check and the lint both failed and it never bothered to fix it or take care of any of the high or higher level comments that it got from our review bots. So, I gave up. I opened up Opus 5.5. I pasted the thread ID in the PR and said, "This is opened by Gemini for Argan. I think the PR is garbage and doesn't really fix the problems that it barely stated.

32:06 Go through the T3 code thread. Figure out where it went wrong. Give me your honest thoughts on the work that Gemini did. You agree slop? Close the PR." I agree it's slop and I closed it with a comment that gives the reasons where it went wrong. Turn one, the diagnostic was mostly correct, but it had no evidence. Gemini found the real mechanism. The synchronous database sync runs on the main thread.

32:25 yada yada yada. Turn two, it fixed the wrong half of its own diagnosis. The reader pool still opens database sync on the main thread. A slow read on a reader connection blocks heartbeats the same way it does on the writer. The only effect is less waiting on this one semaphore call and that matters only when a fiber yields inside a transaction. What it's saying is it didn't work.

32:45 It also called out that it didn't check prior art. There was already two PRs that fixed these problems. You might remember the previous video with Muse where I pointed out how much I liked that Muse said, "I'm not going to write code. Here's a PR that fixes it. You should merge this instead." This model doesn't do that. In fact, this model will hallucinate its own incorrect solution instead.

33:03 The correct answer was in the PR tab on our GitHub. And not only did it not find that, it invented its own wrong answer. It also solved it with a reax to route requests. I wish I was kidding. It's so bad. It was so garbage. According to Opus, Gemini can read code and explain how it works. It does not test its own theory. It never asked whether a reader pool on the main thread stops the event loop from blocking.

33:29 It answered your loose question with a confident road map and a percentage that it invented. When I told it to do priority 1, it built that literally and at maximum size without checking whether priority 1 was even correct. So I told it to go make the changes and it was 21 lines of code. They are now merged. They'll be in the next nightly and people will have a better experience with SQLite.

33:48 Real quick crash out about the instability of this model and the anti-gravity stuff. If I get my Anthropic account banned, that sucks. But I can open a new one with a new email address and I don't lose too much. If my OpenAI account gets banned, slightly more annoying, but again, I open a new one, I'm good. I might sign into both of those with my Google account, but if I'm banned, it's just my sign in to anthropic or open that's banned.

34:10 If my Gemini account gets banned, I might lose my YouTube channel because my Gemini account is my Google account. So, I could lose my email. I could lose my billing system for my credit cards. I could lose my saved passwords in Chrome. I could lose my access to my YouTube channel. I will lose my access to everything else I sign into with Google. There's a real risk.

34:27 They have promised that they will only ban you from Gemini and anti-gravity. They won't ban you from Google anymore. But I've seen people getting their Google accounts banned still. So, it's a risk. As such, integrating their official ACP into T3 code felt risky. We chose to do it because I was told it should be fine. But I can also tell they're not putting a whole lot of effort into this, which you can see in places like here where the model just failed outright to connect to the gateway in the middle of a run after

34:55 reading endless files for an hour and 2 minutes and then it just failed and refused to continue and I had to restart this entire thread with none of its existing history. Horrible. And then there's a bunch of silly little Googleisms like they don't report the model and then let you pick effort in the ACP. All of the model slugs are paired with the effort levels.

35:18 So you have to pick for argan high, medium or low as a model. Every other lab gets these things right. ACP is a pretty simple standard to implement and integrate. Google is Google and they got all of this [ __ ] wrong. It is embarrassing how miserable it was to integrate anti-gravity and how high a percentage of the bugs in T3 code are just for anti-gravity.

35:44 It has consistently been a thorn in my side since we first integrated it. And even having Gemini 4 did not make it better. It was a relief to have a reasonably usable UI for a Gemini model for the first time ever. I honestly thought it would make me like it more, but now that I see it next to models that are competent like Muse, Opus, even like Sonnet, honestly, it just hurts more.

36:07 And now we get to the end comparing it to the other models. This feels unfair. Feels unfair to the other models. This is just garbage. Don't ask me about how good the usage is on the $200 plan or anything. I don't care. I'm not going to push the limits enough to know. It's probably practically unlimited because you're not going to do a whole lot with it cuz you're going to get frustrated and give up.

36:25 It's just it's just bad. I'm going to put my money where my mouth is here and show you just how committed I am to this. I don't trust any Gemini models nor Muse Wonder. To be clear, Muse Wonder full is fine, but the standard Muse Wonder as well as Gemini 3, 4, etc., they do not contribute useful code to the T3 codebase. I would like to find some way to autoc close all poll requests that are filed using models like Gemini 4 argan or the normal Muse Wonder when they are filed to T3 code.

37:04 I am not joking. I will find a way to get this in later this week. I have wasted more time in tokens trying to vet code that was written by Gemini 4 than I would have spent writing the code with the smarter model in the first place. Gemini is not a useful model for anything in the code world other than maybe as like a first or second pass on a nicel looking marketing page.

37:23 As soon as I'm done with this video, I'm just not going to use Gemini for anything other than like fetching data for YouTube. This is one of the worst releases of the year. I don't care what the benchmarks say. I have been testing it constantly in real world work and I have not merged a thing that this model has shipped. This feels like a throwback to 2024 with some of the hallucinations I see this model making.

37:45 I was polite up to this point because I'm thankful for the Google employees I've talked to. They have generally been one of the more receptive labs to my feedback and I tried to share this feedback with them early. I've also heard rumors that there is a better snapshot like Carbon or something that's coming soon. I don't know how much better it's going to be, but it needs to be a lot because this model is garbage and I have no intention to ever use it again other than for fetching data on YouTube.

38:07 Tell me what y'all think. Am I overreacting massively or is this model actually just garbage? They told me it was good at code and it's not good at code. So, I'm I'm pretty done with it. Maybe you can find some cool uses. Let me know in the comments if you do. Until next time, I'm sticking with Muse. Okay, seriously, I'm probably just going to stick with Opus 5.5. You get the idea, though. I just And I'm out. Bye.