← All transcripts

I was wrong... Transcript, AI Summary & Key Points

Theo - t3․gg · 4 days ago · Science & Technology · 19:39 · EN

Watch on YouTube

Answer

The author was wrong about Gemini 4 Carbon: after auditing real-world usage, it is consistently five times slower than any other model and unusable for real work, worse than his earlier video suggested.

AI Summary

Gemini 4 Carbon is unusably slow — roughly five times slower than every other model in like-for-like agentic coding work, a gap the author admits underreporting in an earlier video. An audit of real-world usage shows the model takes over 11 seconds after a request hits Google's server before it starts responding, spends more time waiting to start responding than Opus and Soul spend responding combined, and reads a file 8 to 16 times on average, all compounded by unreliable caching (roughly a 30% chance a read misses a cache just written). The speed gap cannot be explained by slow tokens-per-second or sequential tool calls — those would only add 30-50% — and it gets dramatically worse in multi-turn agentic work than benchmark timings suggest. On top of the speed, the Vertex hosting endpoint is unusable, reasoning effort is embedded in the model slug instead of a separate field, the subscription price doubles on December 1st, and the only real advantage is that it's slightly less likely to flag security work. DeepMind staff have been responsive and are working on the speed, but the author refuses to respect the engineering judgment of anyone publicly calling the model usable, directs personal criticism at a DeepMind leader over his handling of Windsurf and his dismissive public replies, and recommends getting a $20 Claude subscription instead.

Key Points

  • In like-for-like agentic work, Gemini 4 Carbon is consistently five times slower than every other model — far worse than the benchmark data showed.
  • Gemini 4 Carbon takes over 11 seconds after a request hits Google's server before it starts responding, even for a single step of a turn.
  • The model spends more time waiting to start responding than Opus and Soul spend responding almost combined; it also takes more time to start than 61 Soul takes to finish responding in most scenarios.
  • Slow tokens-per-second and imperfect tool calls can't explain the gap — those would only make it 30-50% slower; the problem is before the request lands.
  • The model reads a file 8 to 16 times on average and has broken caching: there is roughly a 30% chance a read misses the cache just written, sometimes reading a 30-minute-old cache or failing entirely.
  • Benchmark timing is misleading because it measures a single accepted request; once tasks become multi-turn agentic — how every coding agent works — the overhead compounds per step.
  • The author updated his agents' skill to kill stuck Carbon tasks rather than wait, which actually flattered Gemini in the measurements.
  • Gemini 4 Carbon's subscription price doubles on December 1st, three weeks after the video.

Tools & resources

1 item

Links mentioned

🔒 Full analysis locked

Unlock more videos and the full analysis

A credit unlocks one video's full analysis for good — the build steps, the tools and how each was used, the methods behind every use case. Pro opens the whole library instead, and raises how many videos you can analyse a day.

Unlock full analysis — free

Transcript

Searchable transcript of I was wrong... — Theo - t3․gg (19:39). Search for a phrase, then click its timestamp to jump straight to that moment in the video.

Captions sourced from the original video on YouTube, published by Theo - t3․gg. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.

00:00 As many of you may have noticed, I just published a video about Gemini 4 Carbon. And despite my plans to be really negative towards the model, I found myself strangely okay with it. Not because it writes great code or that it's good at 3D or anything other than the fact that it didn't seem like it was going to restrict me too much. And that was a real valuable aspect of it.

00:16 And I really just didn't find it worth using otherwise. I did my best to cover the facts of the model, how much it costs, how fast it really is, what it looks like to use in real world projects. and I was not harsh enough. This hurts me a bit because I, as I've mentioned, like working with the Deep Mind guys for the most part. They're really upfront.

00:37 They're really awesome and they try hard to fix problems. And that's why we're making this video because I was wrong about the speed gap. I'd mentioned the fact that four carbon low was slower than pretty much every other model in history. I was not aware of the fact that it was five times slower. As many of you guys know, I don't watch my threads when they run very often.

00:56 I fire them off and then I look when they're done. So, I took the time to audit my real world usage of all of these models. And in like for like work, Gemini was consistently five times slower than anything [ __ ] else. However bad you think this is, it's worse. And I am going to crash out so much about this because it shows on a fundamental level that Google does not give a single flying [ __ ] about anyone using their inference.

01:26 And until these things are fixed, you should not take them seriously as a company. And I hate to be this strict about it, but it is my responsibility to because I feel like I screwed up here. And I also, to be fair, feel like everybody else is not being real about this either. People sincerely think the Gemini models are a reasonable alternative to other things.

01:46 I can't respect you if you pick Gemini right now because it is that bad. And I am sorry to my friends there. I hope you use this video as leverage both to get these things fixed and also in interviews for jobs at companies that actually give a [ __ ] because I am so goddamn done. And I'm going to explain in detail the levels of incompetence specifically on the hosting side for Gemini after a quick word for today's sponsor.

02:10 Back in my day, building a mobile app required a whole dedicated team or two even, one for iOS and one for Android. Getting all of this right was really hard. And even with awesome tools like React Native, there were still so many holes you could fall into that made it nearly impossible to get right. Believe me, I have experience here. I've built a lot of apps with and without React Native.

02:29 The number of pitfalls you can run into, even as an expert, is obnoxious. And your agents might be great, but they're going to hit these even faster and cause even more problems. I've been there. It was rough. And I'm very thankful that today's sponsor is here to help out. Infinite Red are some of the industry leaders in React Native expertise, but they go further than that.

02:48 They're more than happy to dive into Swift, Cotlin, Objective C, or even Java if necessary, if that's what gets the best performance for the thing they're working on. They've been helping companies set up their code bases for success in React Native forever. And it turns out those same things are really good for agents, too. If you make your codebase stupid proof so that engineers who are less experienced with Native can make good code, turns out the agents can handle it even better.

03:09 And that's what they are here to do, to set you up for success. You go hire a bunch of expensive dedicated mobile teams like I'm sure lots of companies have or you can do the thing that these small businesses like Microsoft or Dominoes or Realtor.com has done. Wait, those are big companies. Yeah, turns out making good mobile apps is hard enough that even big businesses are relying on Infinite Red to get things right.

03:32 Whether you're trying to outsource the development of your mobile app or you're trying to level up your web team to make a better experience on mobile or you just want to make sure your agents are writing good code, make sure your apps are built right at soy. Infinite. I I will apologize in advance for how blunt I am being and also for how blunt I have been.

03:48 I cannot respect the engineering opinions of anyone who thinks this model is usable for real world engineering work outside of the fact that when you do cyber security stuff with it, it's slightly less likely to flag you. There is literally no reason otherwise to use this model right now. And I have lost respect for a handful of engineers that I thought knew better that are publicly claiming it's good for day-to-day work.

04:10 I am partially guilty here myself. I did not think the gap was this big. I did not think using it was this miserable. And of course the subscription subsidy side. It is so disgustingly expensive too, especially starting December 1st when they double the price. Yeah, this model is going to double in price in 3 weeks. What am I crashing out about right now?

04:27 Let's go through this. As I mentioned before, have a lot of friends at Google that have been doing a very good job handling my crashs, getting things fixed where they can, and really making things better. I know for a fact that they are currently all hands on deck for the speed stuff that I'm complaining about, but I need you guys to understand how bad it is.

04:45 In my previous video, I showed that when you don't use max effort for anything, because you shouldn't, max effort is bad. Gemini 4 carbon is the slowest modern model by a lot. There is no way you can use Opus where it takes longer than Gemini 4 carbon. And Fable 5 on XH high is slightly slower than Gemini 4 carbon high, but on high, it is faster than every single thing Gemini 4 carbon has completed.

05:11 This is wrong and misleading because this measures from the time the request officially is hit and accepted by the server on Google's side to when the response comes in. And a lot of the tasks for artificial analysis are not agentic. So they're not multi-turn. Once you get into multi-step turns, which is guess what how every single agent coding tool works, this [ __ ] crumbles so fast.

05:34 As I mentioned before, once I started analyzing my data, Gemini 4 Carbon was consistently five times slower in day-to-day work. But this number is actually worse than it seems. I had nine separate times that Carbon was brought in as a judge for work. And you'll notice after the first three, something changed. Started to say stopped by the parent a whole bunch.

05:56 Do you know why that happened? It was so slow that I went and updated my skill to tell my agents, "By the way, it looks like carbon's pretty slow. If you're stuck waiting for it, but you have work to fix, don't bother. Just stop it and get back to work." If I hadn't done that, the numbers would have been worse. I accidentally gave them an advantage here.

06:19 So, not only is it way worse than I thought it was in my first video, it's way worse than I'm showing you now. I don't know how much worse it is because I'm not willing to slow down my day-to-day work enough to properly measure it. I will never learn how much time it takes to hand feed a turtle a full head of lettuce because I am a busy human being that's not going to sit there and [ __ ] do it.

06:36 It's insane. It's It's so bad. It's so [ __ ] bad. And it gets so much worse when you see the reasons why. You might think, "Oh, it's the TPS that's slow. Oh, they're not giving it enough compute. Oh, maybe it's not calling tool calls in parallel properly." All those things are true. And all of those would make it like 30 to 50% slower. You don't get 5x slower by doing tool calls.

06:56 Okay, you can get 5x lower by doing tool calls wrong. They're not doing them that wrong. They're close. The problem comes before the request even lands. This model consistently takes over 11 seconds from when the request hits their server to when it starts doing something. 11 seconds. Let's go through this together in real time. I'm just going to sit here and we can experience the zen of Gemini 4 Carbon together.

07:32 Still going. Okay, cool. Two of three are gone. Fable ran a slow tool call in this demo, but it's still finished. And now Gemini is done as well. This was for one step, by the way. This was not for a proper full turn. This is one step. When you combine this with the fact that Gemini takes more steps to complete tasks and you have to compound this additional 11 seconds per 20 seconds on every single request.

08:04 The result isn't just embarrassing, it's actually unusable. And as I tried to use Gemini more for real work, when I realized I had underounted its intelligence, I realized I had over accounted my own because it made me feel so [ __ ] stupid to ever pick a model that is this miserable to use. So you might be thinking as my chat already is, did nobody notice this?

08:26 How did this happen? I have seen this happen once before. Is that the small startup that does a little bit of AI stuff? You might have heard of them. Their name Microsoft. Azure had a similar problem with OpenAI models where most of the time they would get you the first token in 10 to 15 seconds, but occasionally it would take 100 to 200 seconds, which meant it was the most unreliable, unusable garbage in the [ __ ] world.

08:48 I crashed out at them so hard that I had execs hitting me up and they ended up fixing it. Yeah, the million-dollar incident. I have a whole video about that one. I was proud as [ __ ] of that one. And that's why I'm here now. The difference is I have nothing to lose because nobody likes Gemini. That time I had a million dollars on the line and this time I'm doing this for the love of the [ __ ] game.

09:06 So why am I talking about this now? Why am I bothering with another video about Gemini? As I mentioned before, there's a lot of people at DeepMind that I actually preferred almost every other lab. Like I would say the lab I feel like listens to me the best and is the most likely to actually care and try to fix things is Deep Mind. They have been pretty good overall at responding to my crashs.

09:27 As such, I am sorry for this video to my friends there. This hopefully will be a useful tool either as leverage you can use in the internal meetings to get these [ __ ] things fixed cuz they're embarrassing. or if that doesn't work, which I'll be real, it very well might not, you can use this video as the greatest letter of recommendation possible when you go to get a job at a real lab.

09:49 Because at this point, I will not refer to DeepMind as a real lab because they are not making technology that you can really use. And as much as I respect most of the people I interact with and talk to at DeepMind, I'm going to get personal a bit because I do not respect Verun. This is the guy that stole a billion dollars from his own team in order to quit the company he started and go to DeepMind with all of the tech.

10:12 I've never seen like and it's somebody who cares a lot about their team who has lost a lot of money in order to do the right thing for their team who could have been bought out personally for a large amount of money but chose not to because I want my team to win as well. Mark and I are 50/50 split on equity for T3 tools and code and all of that and Julius is not far behind us.

10:33 The only people who made any money from Windsurf are him and like two of his friends. They left a 130 person company they started. The three of them left it just out to die. And all three of them took their money. I'm going to be so real with y'all. I'm choosing to get more personal here because I want to see if Arun can handle pressure. I want him to know that I think he's a spineless prick.

10:51 I want him to know that I think he's a bad leader. I want him to know that the way he responded to me in this scenario is unacceptable. First off, like saying this is high praise instead of acknowledging the issues that I described, which mind you, like every other person I talked to at DeepMind has been awesome about this. They've been DMing me all day for the last two days, begging me for feedback, trying their [ __ ] hardest to make the models better and the tools more usable.

11:12 He drops a laugh emoji and some annoying but useful advice cuz I crashed out about my email always being visible in their CLI. Cool. Awesome. Fine. I asked him to make it a config. Said, "Awesome." Around this point, I had started digging deeper into the speed issues because I was using it more and it was annoying me. I had not dug deep enough to know exactly what was causing the problems, but I assumed cuz I had just done an analysis of how tool calls were working with Gemini and it I learned it was doing things in

11:40 parallel. I didn't think it was and it was. So, I was confused how it was still so slow. So, I recommended a video I had done in the past which is about OpenAI moving to websockets for inference in codecs. The reason they did that, and I've talked about this before, watch that video if you're curious, the TLDDR is that the model needs your whole history.

11:57 So when you make a request for the next turn or the next step in a turn, it would send the whole history up, which is just so much latency sending the whole thing up waiting for it to be parsed, matched to a catch, prefilled, and then loaded in so it could start responding every time a tool call happened. It was just a ton of overhead. And one of the reasons Codex used to feel so much faster than Claude was because they put this layer in front of the inference to connect over a websocket and send less data back and

12:23 forth and keep it warm. It was awesome. It was so much faster. I don't know how Enthropic caught up without doing the websocket thing, but they did and it's great. I think a lot of it is cash changes regardless. Ready for the response. Time to first token is also just high because prefill is not cheap. Frown face. It's not just network transport. He's right here.

12:41 That's correct. let me quantify, but I'm not sure that would immediately solve this. And to his credit, he's right. This probably wouldn't immediately solve it. But I was annoyed enough about this that I chose to dig in more. And that's when I found these much, much crazier numbers. Gemini 4 carbon takes more time to start responding than 61 soul takes to finish responding in most scenarios.

13:05 Insanity. And then there's the funny Gemini here, like the fact that it on average reads a file 8 to 16 times. Hilarious. I don't think you guys properly understand how far behind they are right now. It's not like first and second and third. It's first and second then like out in like a different school somewhere. It's like the two best students in college and then somebody who was held back four times in kindergarten and then dropped out.

13:32 It's so bad. And it's already [ __ ] cached. I've been saying this for a bit and I did bring this up in my previous Gemini video. They are so goddamn bad at caching. It's insane. It's insane. They are the only lab where you write to cache and then read from cache and it's a 30% chance it doesn't read from the cache you just wrote. It'll read from something.

13:53 It might be a cache from 30 minutes ago. It might just fail to read the cache entirely. It's so bad. It's so [ __ ] bad. This is what I get for actually trying to use the model because I found something it could do which gave me a reason for the first time in a year and a half plus to actually try a Gemini model for day-to-day work. And I feel like I've been going insane for the past 24 hours as I've tried my hardest to use this [ __ ] model.

14:18 The problem with Google isn't just that Gemini models are bad to work with. It's that everything around them is crumbling under the weight of this company that doesn't know what the [ __ ] they are doing. It's insane. And you'll never guess who replied to this tweet. Give you a hint. I don't like them. Not what we're seeing on our workloads. I got some advice for you, Verun.

14:38 DM me in the future. I do not like how you engage with me publicly and it makes both of us look bad. The best thing for both of us here is for you to accept that I know what the [ __ ] I'm doing because I know you do. You are clearly very smart. You know your [ __ ] You're just a bad leader. Just DM me in the future, please. ideally with at least one other person in the group chat so you'll feel a little embarrassment when you say things that are dumb.

15:00 Maybe any of the other Gemini folks here. Like here's Philip. He's awesome. I've had a lot of really good conversations with this guy. There's a bunch of others that are like in these convos throughout that have been pretty awesome to work with that genuinely care a lot. I wish those people were the leaders. But yeah, DM me in the future cuz you're not good at engaging with me publicly.

15:16 And I'm more than happy to give you guys every single thing I've ran, every single prompt, every single environment, every single thing I've done if that's what it takes for y'all to fix this because I want you to fix this. I want another player in the race. I want to be able to use Gemini 4 Carbon in order to get these things that it's good at done.

15:34 And I want the next model to be good, too. But right now, the model's not good enough to justify using. And even if it was, the cost isn't good enough to justify spending. And even if it was, the speed is so slow that you'll be miserable using it. Every single thing about using this model is hell. All of them. Okay, there there's one that's slightly better.

15:53 It speaks less in GPTisms than GPT models do. That's about it. I actually just cannot take anyone seriously who prefers Gemini right now. It is so obviously bad that even when I tried my [ __ ] hardest, it has just made me more angry. I saw somebody mentioning that this might be because the subscriptions are slower and the API is faster. Funny enough, someone else in my replies just point out the opposite.

16:17 The vertex hosting for Gemini 4 Carbon is unusable. You basically have to go against the terms of service and use the anti-gravity endpoints everywhere in the reasoning part of the model slug, which by the way, they put the reasoning effort as part of the model name, not as a separate field like literally everybody [ __ ] else does, which [ __ ] up so much stuff.

16:34 It ruins our analytics. It just it it feels like Google does not understand where anyone else is right now. I have it's jarring that like Muse comes out a new meta model and it just fits in perfectly with everything else I do. It's just another model that has its own behavioral quirks that has its own behavioral benefits. It's annoying that Muse Wonderful isn't over API and that you have to use it in the subscription.

16:56 You can kind of just plug it into your tools and switch to it and it feels fine. I have more weird [ __ ] happen when I try to use Gemini in a day than I've had with all things Meta has ever produced in history. And I've been using Meta [ __ ] since I was in high school on Facebook. I am really tired of this. I'm going to be real. I even feel bad filming this video, but I feel worse that I definitely misled some people with my previous one into thinking that a Gemini model might finally be usable.

17:21 It's not. And even if they made the model two times smarter, the only reason it would be usable is if the model gets smart enough that they can set it on their internal [ __ ] and fix all of their broken stuff. But until they do that, I've said this for a bit, I'll say it again, you'll know that Google has finally made a great model when their products and infrastructure stop falling apart because it is a dumpster fire right now.

17:43 And I I knew it was bad. I had not internalized how bad it was until I learned that the amount of time carbon spends waiting to start responding in realworld tasks, not just one step, but a whole turn. This model spends more time waiting to start responding than Opus and Soul spend responding almost combined. Don't use this model. And maybe someday they'll finally deal with this and stop pretending it's just some small prefill thing.

18:15 One last verunism before we wrap here. We're looking into how we can optimize this more. There's always a trade-off of latency and cost, but we should do better here. This is the type of response that sounds good until you realize that the model in the screenshot is the most expensive in the screenshot, the worst in the screenshot, the slowest in the screenshot, and the most miserable to work with.

18:35 This is effectively admitting that they cannot keep up on any axis. This no longer is, well, Gemini is slightly worse at some things. It's like 5 to 10% worse. It is an order of magnitude worse in effectively every single scenario. And I'm tired of pretending otherwise. I'm going to end this with a great comment from chat here. Thank you a lack of hygiene for dropping a banger.

18:58 Friends, don't let friends use Gemini. I'm done with this. I've said all I have to. I'm going to go back to not using Gemini models unless I'm reviewing them for content and maybe someday they will function properly. But until then, seriously, just just go get a $20 Claude sub. You're going to get more inference out of that, more value out of that, more work done.

19:18 There is no reason to use a Gemini model willingly for anything right now. You will just make yourself sad. And if somehow it doesn't make you sad, you should start job hunting because this world is not made out for you in the dev world anymore. Uh, I'm going insane. I'm going to go do literally anything else now. I hope you enjoyed this crash out.