Searchable transcript of What Is Chunkless RAG? How Docling & AI Agents Navigate Documents — IBM Technology (07:00). Search for a phrase, then click its timestamp to jump straight to that moment in the video.
Captions sourced from the original video on YouTube, published by IBM Technology. The video, its captions and all related intellectual property remain the property of their respective owners; AINotes claims no ownership. Provided for research, accessibility and search — see the Transcript Notice and Copyright Policy.
00:00 Say you hand a model a 200-page annual report and ask it one specific question, something like, what changed in the revenue recognition policy this year, and where does the report explain why? That's a normal question. A person who knows the document could answer it in about two minutes. They'd flip to the right section and read it. But watch what the machine does to answer the same thing, because the usual approach throws away the one thing that we would use to make the question easy.
00:25 The usual way to do this is retrieval augmented generation, or RAG. You take the document and you cut it into chunks. Maybe every 500 words, maybe by paragraph. You then turn each chunk into a vector. Store all of them, and when a question comes in, you turn that question into a vector as well, and you pull back the handful of chunks that look the most similar.
00:46 Those go into the model's context, and it answers from them. This is useful, and it's cheap, and for a lot of questions, it's all you really need. The strain shows up when one big structured document shows up. The moment you cut it into chunks, you've discarded how it was put together. A heading gets separated from the paragraphs it was introducing.
01:05 A table gets split from the sentences that say what the table means. And if the answer lives across multiple different sections, similarity search has no idea those sections belong together because it was only ever comparing small blobs of text. You get fragments back and the model has to guess how they can relate. But that document was never a pile of text to begin with.
01:25 Somebody wrote it as a structure. There's a title, headers, sections, subsections, there's paragraphs, tables, images, all of which are underneath this tree structure. The author already organized the information for you. Chunking takes that tree and flattens it so that we can do a similarity search over those flat sections, but we end up destroying the structure and then spend a lot more effort trying to recover the relationships that were sitting right there the whole time.
01:57 So let's think about another way to find the right material. Don't flatten the document, keep the tree. And let the model reason its way to the right part instead of matching by similarity. Think about how you'd actually answer that revenue question for yourself. You wouldn't read all 200 papers of the document. You'd open the table of contents, maybe find the section on accounting policies, flip to it, and then read just that section.
02:22 If it pointed to a footnote, you may follow the reference. You'd navigate through the document, and that's the model that an agent can do as well. It starts with an outline of the tree where each section comes with a short summary so that it can see the shape of the document without reading the body. It reasons about which section is most likely to hold the answer, opens just that one and reads it.
02:49 If that's enough, it answers the question. If not, it can pick the next section and it keeps going a few steps at a time until it has what it needs. Two things fall out of this that are hard to get with just flat chunks. First, context comes along for free. When the agent is reading a paragraph, it still knows which section that paragraph lives in and which subsection because it walked the entire tree.
03:13 The headings above it are part of that path. A chunk pulled by similarity search has no idea where it came from. Second, you can answer questions that span the whole document. If the policy is defined in one section and reasoning is three sections later, the agent can still hold its place, go read the other branch and come back. It's moving around the map.
03:33 Similarity search just hand you the pieces that happen to look similar to your query. There's another benefit as well. The agent isn't doing less total work. Walking the tree means several passes and that adds up. What changes is what ends up in front of the model before it finally answers. Instead of the entire document or just a handful of chunks that have lost their place in the document, the model gets one relevant section with its heading still on it.
03:58 It's reasoning over the right material and its right shape. And tends to mean cleaner answers with fewer of those moments where it invents something from disconnected fragments. Everything I just described depends on actually having the tree. And that's the hard part because most documents show up as PDFs, which are basically just instructions for placing characters and figures onto a page.
04:22 There's no clean structure in there to walk. This is the job that Docling does. You give it a PDF. And you get back a Docling document. Which is gonna be that structured tree, real sections and headings, reading order being preserved, tables that are still tables. The PDF format buries the hierarchy that the author has put in and Docling reconstructs it.
04:56 Once you have that object, the Docling agent. It can work on that structure directly. It can write, edit, extract fields, and enrich sections. It includes an agent that does exactly the navigation that I just walked through. The project calls this Chunkless RAG. Which is a good description, same goal as ordinary retrieval, find the right material and ground the answer in it.
05:33 But it gets there by reasoning over the document structure instead of chopping it up into chunks and matching by similarity. This is still retrieval. It just keeps the document whole while you're doing it. It isn't free though. You need a good tree before any of it works and parsing real world documents into clean structure is a hard problem. That's most of the engineering.
05:55 Reasoning through the structure also takes more back and forth with the model than a single vector lookup does. So there's a lot more latency and more calls. The chunk-based retrieval is still the right tool a lot of the time. If you've got millions of documents and a fuzzy find me anything about X kind of question, similarity search is hard to beat.
06:15 The structure approach earns its keep on long organized documents where precision matters and the connections between parts are what you're after. In a lot of real world system, you'd use both, similarity search to find the right document, structure to navigate inside of it. So that's the real choice. Both of these are retrieval. The question is whether you chop the document into pieces and match by similarity, or keep it whole and reason through its structure.
06:43 The author already drew the map, document gives it back to you, and an agent can now follow it.
Chunkless RAG keeps documents whole and uses their structure for agent-based navigation instead of relying only on chunking and similarity matching. It is most useful for long, organized documents where context and cross-section relationships matter; chunk-based retrieval remains preferable for many broad searches and very large document collections.