Extract Insights from Videos with Docling + OpenRAG Transcript, AI Summary & Key Points

Have you ever watched a podcast or conference talk packed with insights you know you'll wanna reference later? In this video, I'll show how Docling and OpenRAG can help you extract the underlying data, so you don't have to re-watch the same video twice. Now, video is great when we're watching and experiencing it, but it's not great at all when we need to mine that data later.

I'm David Jones Gilardi, Developer Relations Engineer at watsonx Data, IBM. Let's get into it. For those of you who are not familiar with these tools, Docling is an intelligent document parser. It has support for tons of different file types. It has advanced PDF parsing and converts documents into formats that LLMs more efficiently and better understand.

And OpenRAG is an agentic RAG platform that allows you to spin it up very quickly for a full RAG pipeline, both on ingestion and being able to then intelligently search on your documents. And both Docling and OpenRAG, they work together extremely well. Both are open source projects, and both allow you to use local models if you so choose. So I mentioned a moment ago that being able to get at the underlying information.

Those like nuggets of information, is the challenge. And if we just go to the end, right? We're going to the end result here. You can see I'm having this conversation about MCP apps. And it was a topic that I talked about with one of the guests I had on my podcast. And what I was really looking for were the exact timestamps various spots in the video.

If you compare this to having to scrub through the video, an hour-long video at that, and find exactly all those points, it's very clear that this saves me, potentially hours of time to be able to do this. Now, I'll pop over to a repo from my colleague, Tejas Kumar, who built a reference implementation that we'll use as a starting point. He has a video on this channel, so feel free to check it out.

You can see his TejasQ example Docling media and transcribe.py. And this is such an easy example to follow. You can see we're setting up the Docling packages that we need. I have an audio path to some file. In this case, it's an MP3. In our case, it will be an MP4 video. We set up some pipeline options. In particular, we're gonna use Whisper Turbo, thank you, OpenAI, to do the transcription for us, a converter, and then at the end, we just converter.convert whatever that file was and take the results of that and export

it to Markdown. Now, something that's really cool here is that Docling is doing all the work. Now, Docling does in fact wrap, Whisper and FFmpeg, right? So what's really neat about that is I don't need to worry about the implementation details of those. Docling is handling it. Even going so far on my M1 Pro laptop to recognize that I have an MLX architecture and is automatically using the right model for me, right?

So it's really just kind of making all of that easy, so I don't have to worry about those particular details. So what I wanna do now is I wanna take this code, and I'm just gonna pop over to my code base. You'll notice I also have transcribe.py. Now, this one is essentially an implementation of Tejas' code. So you can see I've got that same audio path.

This is where as we, as the application is iterating through and it's grabbing, say, video files like you see here on the left, it will then inject those into the audio path. We're gonna come down here. Right, we have the same exact setup for the packages from Docling. We have the pipeline options, the same thing with ASR pipeline, Whisper Turbo. I'm creating my converter.

All right. Finally, here's where it deviates just a little bit. We still have converter.convert. We're still transcribing it in the same way, but there's a key piece of information that I need in order to make this app and the intelligent agent really useful, which are those timestamps, right? Docling does in fact contain that information, so this is another place where Docling makes this really easy for me.

So in this case, instead of passing MP3s, I'm actually passing MP4s. I'm using Docling to essentially kick off Whisper and FFmpeg to transcribe that, but it pulls the timestamp data out, and can see in this for loop right here, that's exactly what we're doing. So I'm going down to the source, I'm getting the start time, I'm pulling that data out, and I'm gonna inject that back into our document.

And then finally, we have, just like we did in the original code, document export to Markdown. The difference here is that now the Markdown has those timestamps. And if we actually take a look, I'll pull one of those up, and you can see, boom, I have these nice granular timestamps. Now, something to kind of point out here, if I was doing this myself, if I was doing this directly for my own kind of content, maybe I was creating chapters in a video, right?

This is absolutely something I would do. I wouldn't want information so granular in this case. It's not gonna be as useful for me when I'm trying to pull out my chapter time codes or something like that. But for the AI and the RAG and everything, for this pipeline, it actually works extremely well and gives it very accurate results when it responds.

If we take a look on the right-hand side, you'll see where I have done a previous run. I just did a previous run of the app. These videos, by the way, I decided to ingest all of my podcasts. That's actually something particularly useful for me. But those are an hour long apiece, so each one of them takes a good five, six minutes to transcribe. I'm not gonna have everybody wait through that.

So what I did is I just did a run, and we're gonna run through it real quick. Now, you notice here on the command line, this is something you just run from the command line. I have my ingest command. I am, in this case, passing a playlist. You could use a playlist. You could use a channel. You could use an individual video. Up to you. And then there are some options.

Dash, dash force. Because I ran this previously, I don't wanna re-download all of the videos. That would be an absolute waste. I've already got them, right? So I'm gonna use this dash, dash force to make sure that it just kind sees that and skips over those. But this key one here is dash, dash filter podcast. What is that about? Well, if we come back over to OpenRAG.

Now, OpenRAG is an open source platform. It actually contains three components. It actually has Docling itself, OpenSearch as its main vector search, vector store, and then LangFlow is the orchestration AI layer. And it brings all these things together to make it so I can just stand up a fully realized RAG platform, and then I can just ingest data and go.

Okay, so where do those filters come into play? Well, if you notice on the left-hand side here, I have a set of filters, Podcast, Compare and OpenRAG Docs. I have other things in the system. This is not a RAG pipeline only for my podcasts. And so I may have a larger corpus of knowledge that I'm looking at. And so when I do my searches, I wanna be able to search just for the information that I'm looking for.

So when I pass that command line option here, this dash dash filter, I'm saying, "I want you to automatically create a podcast filter." Now, something about OpenRAG is it does have an SDK, and the SDK, other than ingest and delete and such like that, does allow me to create filters on the fly, and that's exactly what I'm doing here. So if I then click on that podcast filter, I'm only looking at the results for the podcast.

And this actually not only helps me narrow the results, it actually makes my searches faster. It's more efficient because I'm not having to look through that whole corpus of knowledge. Okay, great. So I have all of the podcasts in place. So we take a look, come back over here to the right-hand side. So I run this and you can see initially it's gonna download from YouTube.

I'm using, yt-dlp, old tried and true. That library's been around for years. This is really what I'm after right here, right? You can see that I've got all of my episodes and the titles and everything, and it's downloaded all those. Each of those are the MP4s that you see over here on the left-hand side, and now we're gonna go through and we're gonna transcribe them.

And again, this is where Docling comes into play. And because I was actually learning in this as well, right? I was not aware before I made this that Docling could handle both audio files and video files. So I was being kinda chatty in the output logging because I wanted to see what was happening, and this is really interesting to see. So you can see Docling as it's going through, it's initializing its document converter, the ASR pipeline, Whisper Turbo, the track source, which is super important.

That's the one that's getting those timestamps for us, and then its chunker. And notice as we come down here again, Docling automatically detected my MLX architecture and then used the correct model on its own, right? I didn't have to do anything for this. So this is where Docling acting as this nice wrapper really simplifies the process of this. I did not have to worry about FFmpeg or Whisper directly.

I let Docling handle it, and it handled all the whole pipeline there. Then once Docling finishes with what it's doing, that's where OpenRag comes into play, right? So you can see here it's starting this OpenRAG ingest. It's gonna take the result Markdown file from Docling with the timestamps, like the one I showed you earlier. It's then going to ingest that into OpenRAG, and that's what you see at the end result.

So if I pick one of these, as you can see the time codes, you can see the text from that particular time code, right? And this is something that the agent will do a very good job of being able to peruse through. And then finally, if we come back to our chat, you can actually see this happen. Matter of fact, let's just go ahead and do this in real time so you can get the full effect, right?

Okay. So I'm gonna say, "What was that video that talked about MCP apps?" And don't forget, I wanna put my filter on just to filter on that particular data. See what it does. Just takes a second. There we go. Okay, great. So notice that it says, "The video you are referring to discussing MCP apps appears to be titled Postman AI and the Rise of MCP Apps."

Wonderful. Okay. So, let's see. Okay. "How about the links to the exact moments in the Postman YouTube video?" Okay, let's see. Okay, let's see. It was very early, and actually there are two stories here, how we got into MCP and also how we got into MCP apps, which is the other thing that one. There is, right? So that's the part where he's talking about how we get into MCP apps, right?

And so again, imagine that you're trying to scrub through this material, and now you're able to just get a direct link to this, and this was in seconds. It would've taken me most likely hours of work to do the same thing if I was trying to scrub through that and get all of those particular moments. And that's how you can use local models and open source libraries like Docling and OpenRAG to pull data like time codes from YouTube videos and feed it into an intelligent agent for fast search.

Thanks for watching and happy coding.