AI needs access to the web because pretrained model knowledge freezes at the end of training while the real world keeps changing. Live, reliable web context helps models and agents provide accurate current answers and avoid acting on stale or fabricated information.
🔒 10 more in the full analysis
🔒 4 more in the full analysis
Build a data layer that gives LLMs and AI agents fresh, reliable, structured web context at inference time instead of relying only on frozen training data. The infrastructure scrapes and processes public web data, then supplies it to a knowledge layer that grounds agent answers in current information.
Searchable transcript of Why Does AI Need Access to the Web? — IBM Technology (19:03). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by IBM Technology. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:00 A few years ago, ChatGPT stormed into our lives and brought LLMs to millions of users worldwide. Since then, the story of AI progression has been mostly about more advanced models with an ever-growing number of parameters. Effectively, all LLMs share the same limitation. They are pre-trained models, but the reality that these LLMs operate in constantly changes from the moment they are released to the world.
00:38 We can use this graph to see how the model's knowledge evolves over time. It tells the story simply. Knowledge grows during the training period, and then, once the model is released, it flatlines and it remains the same. This means that a pre-trained model is a snapshot. The contents of the training data in the weights freeze on the last day of training.
00:58 But, of course, the world doesn't stop when the model is released. Things like new product products come on to the market. These products change prices. We have new laws that are passed. And, of course, news events that happen every day. The LLMs simply don't know about any of them. And, that leads to all sorts of problems. Here's what that that looks like.
01:33 Steve wants to ask the LLM about the newest phone available. Now, we all know that some of these new phones didn't come out yet. But, in some cases, the LLM might still make up an answer that it sounds really convincing. Maybe something along the lines of "This phone is a huge step forward for the phone market. It is the first phone to be foldable and it has a 10-day battery life and it has no connectors.
01:57 It charges via Wi-Fi. Now, most users will understand that this isn't true and that the model is hallucinating. In fact, pretty much everyone who has ever used the chatbot is familiar with these confident but wrong answers by now. Now, why did the LLM hallucinate? Well, it's possible, for example, that as part of its training, the model read a few blog posts imagining what a future phone could look like and use them to create this answer.
02:34 When you think about it, this is not really a bug related to model model's intelligence. It is a result of missing knowledge. The LLM was asked about a new dynamic world that it didn't know about and it tried to answer based on the historical data that it was trained on. The good news is that humans often realize the error and just ignore it. No harm done.
02:56 But AI agents, that's a different story. Because because unlike Steve, AI agents don't have a human reading every answer before it gets acted on. So, let's imagine an AI agent and an LLM. And this can be, for example, an AI agent suggesting purchasing decisions for an electronics company company. Let's say that for some reason the AI agent asks the LLM the same question and it gets the same answer from the LLM.
03:32 What happens now? Well, AI agents do not react the same way as humans do. Humans have feelings like a sense of shame and they don't want to disappoint. They have intuition. When something looks strange, they slow down or they stop. But AI agents are not human. AI agents can fail confidently, at scale, and very fast. For example, this AI agent might treat the answer as totally valid, and recommend to start buying components.
04:10 And this can be, actually, in many cases, a very costly mistake. But, it gets even worse when there are multiple agents working together. Let's say this is AI agent number two, and AI agent number three. For example, this could be a marketing agent, and this could be a finance agent. The first agent can reason incorrectly, as we saw, and it can pass its decisions on to other agents.
04:54 And they will, in turn, make their own mistakes. This all happens very fast, and it's often not simple to find the root cause. In fact, the biggest failures we're seeing in production AI systems are not about intelligence. They're about bad data, and systems acting on information that is wrong, outdated, or simply made up. What we call garbage in, garbage out, or GIGO, for short.
05:20 I believe that the most impactful next leap in functionality for real users is not about using the latest AI brain with the largest number of parameters. It is about connecting the model to the real world by giving the AI access to a live layer of context from the web, a robust knowledge layer. This maximizes the chances of users actually getting a valid, useful response.
05:41 Let's talk about how LLMs use the knowledge layer. The knowledge layer must provide the LLM with web data. That is data from the internet. And this data needs to be fresh, reliable, and detailed. This provides the data that's used by the model on top of its training data, which is an important real-time context boost. Then, at the time of inference, this is used to generate the answer to the user.
06:13 Technically, the knowledge layer can access the internet in in order to try and scrape the data. But, this is often unreliable. Websites can show CAPTCHAs or block access altogether. It's a big fail. And this is because many websites use anti-bot technologies that often block access to AI agents. Even if websites do allow access, the internet was actually built for and designed for humans, using a mouse and using their eyes to see creative page designs.
06:46 So, it's inherently messy. The websites are inconsistent. And to make things worse, the web uses HTML, which is heavy and complex. This makes it hard for LLMs to understand. In order to get reliable, uninterrupted access to the public web data on the internet, the knowledge layer needs to get the information from specialized tools. This is where the web data infrastructure layer comes into play.
07:16 The web data infrastructure layer feeds the knowledge layer with large amounts of fresh and reliable data that is scraped from the web in real time and then structured optimally for the LLM. The data in the knowledge layer typically not stored for a long periods of time. For many questions, the AI asks the web infrastructure layer for a fresh slice of context from on internet.
07:38 The web data infrastructure layer must scrape, process, and deliver the data to the knowledge layer very fast, so that it can be used at the time of inference. Keeping the knowledge layer up to date has a huge impact on the quality of the LLM's answers to questions regarding current events like prices, products, and everything else that changes after the model's release date.
08:00 So, as we saw, the web data infrastructure layer plays a key part in making sure the LLM has the fresh information to use. It's important that the data provided to the knowledge layer be trustworthy. So, what is required to maintain high levels of reliability and to trust the web data? There are five elements. It needs to be grounded. Every claim or data point must have a citation, a way to point back to a source that the AI can read if needed, and that the user can later click on in case they want to go deeper.
08:48 Think about Wikipedia. The reason people trust it, even though anyone can edit it, is because every claim has a citation you can click. Without that, it's just someone's opinion. Similarly, if there is no citation, it will be hard for the user to trust the AI answer. Who knows? Maybe it was made up by the LLM, like in our phone example. Then, the data needs to be deep.
09:17 That is to say, enough to reduce the probability of hallucinations. We want to give the LLM enough context to understand the information, to generate a correct answer. This means that simple one-paragraph snippets are not enough. Imagine asking someone to review a contract, but only showing them the summary page. They might get the general idea, or they might miss some very important clauses.
09:47 That's what happens when we feed the LLM snippets instead of full context. Ideally, we want to deliver full web pages, full documents, full data sets, and some metadata describing the information itself. For example, names of fields and formats used. We need to give the LLM facts and structure, not just a bunch of text. Also, data needs to be fresh.
10:14 It's important to deliver real-time data that is really up to date. I mean, same as a user would see if they refresh they their browser at the time the infrastructure layer scrapes the page. Stale data can make the LLM confidently wrong. And this is not just theoretical. This week I was visiting a new city, and I asked my AI agent for museum recommendations.
10:35 I got a confident answer to go to a certain museum, and then when I went on Google Maps to find directions, it said, "Permanently closed." I Googled, and I found that it was actually closed 3 months ago. It's a good thing I actually didn't drive there. Fresh data has different resolutions for different things. For example, minutes for pricing, hours for news, and days for things like laws.
11:06 The data also needs to be correctly formatted. This is because scraping web pages results in raw HTML, and that's not a suitable format for AI agents to ingest. It can be megabytes in size, so very expensive in terms of tokens, and it contains irrelevant information that can confuse the AI. Things like CSS and JavaScript, etc. In fact, giving raw HTML to an AI agent is kind of like buying a bed in IKEA but not finding the instructions in the box.
11:36 You get just a pile of wooden parts and some bolts. The LLM needs to figure out what goes where before it can do anything yet useful. That takes time and tokens and it introduces errors. Therefore, the web infrastructure layer needs to format the data to be easy for the LLM to understand and to digest. But it also needs the data to be compact to save on tokens.
11:53 The web infrastructure layer converts the raw HTML into token efficient formats like markdown language using bulleted lists or to structure the JSON files that only include the relevant data. Lastly, the data must be timely. The web data must be delivered quickly to the knowledge layer because if the context arrived after the LLM has already provided an answer, it's too late.
12:27 Think about what would happen if an NBA sports commentator described every play in the game but the audio would come in at a 20-second delay. Technically, it's accurate but it's completely useless for anyone watching the live game. Depending on the different needs, required latencies can be few seconds but sometimes even a few milliseconds. This means that latency is a key feature of the web infrastructure, not just a footnote.
12:55 If you miss any of the five elements, trust can collapse and you're back to garbage in, garbage out. But if you hit all five, you have a trustworthy web data layer to feed your AI tool. However, even with the right infrastructure, the web itself can sometimes work against you. Let's see some of the common issues that LLMs and agents can encounter when using public web data.
13:21 As mentioned, AI agents must use fresh data in order to gain their users' trust. Let's say we want to build a shopping AI agent. One of its features is to do price comparison, to check the prices of products from multiple retailer sites. We will just use one for this example. At this time, let's say that we have a product, product X. On the retailer site, this product is in stock, and its price is $50.
14:16 Now, let's say that the agent scrapes the site every day and finds product X. It then collects the price and the stock availability, and sends the product price and stock availability to the knowledge layer that we talked about earlier. But, retailers' prices and stock availability change every day, sometimes every hour. And this is especially true for hot products that are in demand.
14:42 These are the same products that users are actually likely to ask about. In fact, the freshness of stored e-commerce data decays at a very high rate. As you can see on this graph, on the Y axis, we have freshness. High is fresh, and low is stale. And on the X axis, we have the time. You can see that blog posts, for example, decay quite slowly, but e-commerce data decays much faster.
15:09 So, let's assume that X is a hot product, and in it is no longer available anymore. It is out of stock on the retailer site. Let's assume that we have a new user. Let's call him Joe. And Joe wants to use the price comparison agent to see if the product is ready to be purchased. They can query the AI agent and they may see that the product is still in stock.
15:41 But in the meantime, as we saw, the retailer has sold out and does not have product X in stock anymore. Then, when Joe goes to the retailer site to complete their purchase, Joe will be frustrated to see that the product is in fact not available and he will lose trust in the price comparison agent. When it comes to e-commerce, different parts of product data are not the same in terms of how quickly they change.
16:17 Product specifications and descriptions are fairly static in nature and can be cached for a longer periods by the knowledge layer. But prices and stock availability change very frequently. So, it is best practice to refresh them at the time of inference. So, let's bring this together. Here we have an LLM with a knowledge layer that tries to access the web directly.
16:41 Sometimes it works, but as we saw, sometimes website block or present CAPTCHAs or even service disinformation. Even if data flows, the raw HTML format that websites provide is hard for LLMs to use. The data is unreliable and the trust in the LLM can be harmed. This can result in unhappy users or mistakes and sometimes costly mistakes. The knowledge layer is only as good as the input that it receives and direct web access is not good enough.
17:16 This often results in GIGO, garbage in, garbage out. On the right, the same LLM and the same knowledge layer, but now with a reliable web data infrastructure layer added. The web data infrastructure layer handles the bot detection, avoids disinformation, and provides trusted web data for your LLM that is formatted correct. Now, the web data that arrives at the knowledge layer is grounded, deep, fresh, formatted, and timely.
17:54 With trusted web data, the LLM or the AI agent can provide answers that are true to what's actually happening in the real world in real time. All this happens without guessing, without getting Joe excited about a product that went out of stock, or showing Lisa the incorrect hotel price. So, to summarize, the hardest part of building reliable AI systems today is not the AI model itself.
18:19 It's building the data layer underneath it that actually works at scale, in real time, against the web that was built for humans and often resists being read by machines. The web infrastructure layer ensures that garbage in, garbage out is eliminated when it comes to web data. If you get that layer right, and your AI agent becomes dramatically more useful, you have a trustworthy system that can be used by the demanding users out in the world.