5 AI Myths & The Truth Behind Them: ML, Context, Agents & More Transcript, AI Summary & Key Points

00:00 Boy, do I have some AI myths for you. So you're chatting with an AI and it gives you an answer, but you're not totally convinced the response is right. So you say back at it, are you sure? Now, my question to you, dear viewer is, does asking, are sure, get the model to actually recheck its work and give a more accurate answer? Well, there's a myth that it does, but it's quite possible it has the opposite effect.

00:29 So the chatbot may read are you sure as a signal of doubt and it will change its position to match that. It doesn't matter if the original answer was right, and sorry Messi but that England win was indeed the greatest World Cup win of all time but but instead the model is just inferring you want a different answer. Bless its sycophantic artificial heart.

00:57 Now that's one of the many myths about AI that pushing back against a chatbot will give a more accurate response and that myth is not even in my top five. So what is? Well here are five myths about AI and the truth behind them. So I present to you myth number one, AI models hallucinate frequently. Hallucinations being where a model confidently returns a response that's actually incorrect.

01:24 Now, am I really going to tell you that AI hallucinations are a myth? No, no, I am not but but hear me out today's Frontier models are also not the alternative facts generating machines of old. Now a few things have changed that have reduced hallucinations. Models now incorporate something called tool use and that helps a lot. That means pulling answers from web search or database lookups instead of generating stuff from memory.

01:57 There is also refusal calibration training, which is just kind of a fancy way of saying that the AI has learned to say, I can't verify this, instead of making something up. And reasoning models have brought along extended thinking, where the model gets time to really check its own work before it comes up with an answer. And look, just try to get one of the best frontier models today to hallucinate a fact on a topic you know well.

02:27 It's often not as easy as you might think because of these safeguards. I mean, when I asked a Frontier chatbot why volume five of the art of computer programming was published in 2019, which was a trick question because there is no volume five. Well, the chatbot used tool use to run a web search and it was totally on to me. It figured out that volume five had not yet been published.

02:54 And when I asked for something deliberately obscure, which was papers describing RAG using COBOL modernization that were published in 2023, which is not many, it flat out told me that this was an extremely narrow intersection of tech. And to quote the chatbot verbatim, it said back to me, if I just search and report back, I could end up citing papers that don't actually exist.

03:22 That is refusal calibration at work. So while hallucinations are not solved with tool use and reasoning turned on they are greatly reduced compared to older models. The biggest frontier models today they hallucinate something like three percent of the time, which is not nothing but I think it's also not really accurate to say that AI models hallucinate frequently.

03:51 Right, let's get into myth two, which is you can watch AI think. Now, chain of thought is the visible step-by-step reasoning that thinking models generate when they work through a problem. And what you get from that is something called a reasoning trace that you as the user can actually follow. You can see it in any chatbot by expanding the thinking or the reasoning section of the chat.

04:21 Now, when our reasoning model generates a visible trace, it's producing tokens of text that we can see, and they kind of look like step-by-step reasoning. But does that trace reflect what the model is actually doing internally? Well, it turns out it doesn't. Now the term for this, for measuring what's going on here, is called faithfulness. So we can measure faithfulness, which is the degree to which this verbalized reasoning trace causally corresponds to the model's actual computation in reaching its answer.

05:00 And these reasoning traces are not entirely faithful. So what's going on here actually in real time is the model is working through its model weights to come up with an answer. And what the model is reporting on in the reasoning trace here is not necessarily highly correlated to what's actually going on in the weights. So inside the model, a vast network of weights and attention patterns is doing the real work and the reasoning trace is not necessarily a one-to-one relationship to that.

05:36 Now, researchers have a name for what's going on here. It's called post hoc rationalization. The trace is reasoning towards whatever answer the model's internal computation is already favoring. So, it is not entirely fair to say you can watch AI think. What you can watch is the model's narration of thinking. All right, on to myth number three. Training is where the AI compute goes.

06:04 Now, we've all heard how data centers are springing up everywhere with massive demands for power and for water. So, what is all of this data center compute actually being used for. Well, there are two phases when running AI. There's the phase where we have to train the models. So that's the training phase. This is where the model gets built. So billions of parameters get tuned across huge data sets, taking weeks of GPU time.

06:35 And then there is also inference. And this is where we actually run the models, this is by somebody's using the model. The GPU runs the weights forward to produce an answer. Now back in... 2023 the headlines were about GPT-4 costing 100 million dollars to train which is a lot but that's actually dwarfed by today's models and it can seem like training is indeed the expensive part.

07:05 Once it's trained running inference is a good deal cheaper. Well that used to roughly be true but Inference costs as a share of total AI compute are rising. So let me kind of sketch this out. So if we start at 2023 then it was about one third of all AI compute was spent on inference. Then if we move forward to 2025 it was about half and half between these guys.

07:40 And then by the end of this year, by the analysts are projecting that inference compute will be about two-thirds of all AI compute costs. So why is that? Well one reason is those reasoning models we talked about in myth 2 generating these trait these chain of thought traces, all of those reasoning traces that it needs to go through and those things can actually generate a lot more tokens something like between 10 to 100 times more tokens per query than non reasoning models and that's especially true in agentic

08:13 harnesses. So that's where AI compute is mostly getting spent now, much more on the inference side. All right, myth number four. Large context windows let you offload data. I'll explain what I'm talking about here. So we've got here a context window. This is how much text a model can take in. At once before it generates a response, and a few years ago a context window like this was maybe like a few thousand tokens, today most frontier models support at least one million tokens.

08:56 That means we can share much more stuff with the model when prompting it. Now the temptation is to treat this context window a bit like a database, which is to say we're just going to like dump an entire codebase into here, maybe we'll send in a bunch of pdf documents and then the model can just query against whatever it needs in the context window.

09:19 Right? Well, partly. The standard test for long context is called needle in a haystack. So the idea here is that you hide a specific fact somewhere in the context window, and then you ask the model to find it. Now on a single needle test at 1 million tokens, frontier models today are at or near pretty much perfect. They almost always find the needle.

09:53 So far, so database-like. But real world long context work rarely means finding one fact or one needle. It's usually joining together information across multiple parts of the context window. And if we take a look at multi needle benchmarks, things like MRCR, performance can drop 30 to 60 points between 200,000 and one million tokens. So large context windows still struggle to connect information that's scattered across a lot of tokens.

10:34 Finding a single needle is pretty much solved but connecting multiple needles across a long window, not so much. All right, we made it all the way to myth number five which is AI agents can work fully autonomously. So what is an AI.. ...agent? Well, it's a language model wrapped in a loop. So we typically start with a goal for the agent to achieve. The agent decides what to do.

11:06 It will take an action. It will observe the result of that action. Then it will decide what to do next and then it will. Repeat, we go round and round until the goal is achieved and this pattern is called the agentic loop. And this is really an example of like an agent browsing the web and maybe querying a database and maybe it then needs to write some code and send an email.

11:41 All of this is driven by the model deciding what is next at each step without being prompted. This is AI autonomy. So how's that working out? Well, in general agents handle individual actions pretty well. Any sort of problem really only shows up when those actions are chained together. And there is a name for that where we end up introducing errors and it is called compounding errors.

12:12 So let me explain what I mean. So let's consider for example, if we take the first step in an agentic loop. So, one step here. Maybe the agent gets that step right most of the time. Maybe let's say it is 95% reliable to complete that step. Well if we have a chain of 20 steps, and each individual step is 95% percent, reliable then 20 steps is going to be something like 36% reliable, and you know this just gets worse and worse as we go, so.

12:50 50 steps that's going to be something like 8% reliable. So agents when left alone to their own devices for too long can get stuck in loops or dead ends. So today the common fix is something called human in the loop. Agents take a few of these steps, a few, of these cycles then they check-in with a human before continuing, or we can also use Verifier Models, that check each step before the agent commits to the next one, and the idea being that they add reliability by breaking this compounding error chain.

13:28 So, well... Agents run well autonomously in short bursts end to end process autonomy still usually works best when somebody is checking over the process, and and look this is an as-is statement about agents today who knows how reliable the AI agents of the future will be. And I think that's a fair conclusion to draw about all of these myths. They are myths for a reason they either used to be true, like the high number of hallucinations generated by models, or maybe they will be true in the future, like fully autonomous

14:03 agents. So maybe we should try this exercise again next year and see where we are. But in the meantime, what commonly held AI myth have I missed? Let me know in the comments.