Crawl4AI

Crawl4AI is an open-source Python web crawler and scraper that converts web pages into structured, LLM-ready Markdown for retrieval-augmented generation, agents, and data pipelines. Its asynchronous Playwright-based crawler supports Chromium, Firefox, and WebKit, dynamic JavaScript pages, sessions, persistent browser profiles, cookies, headers, proxies, screenshots, media, iframes, lazy loading, full-page scanning, caching, and deep crawling with BFS, DFS, and best-first strategies.

For extraction, it provides heuristic Markdown filtering including BM25-based relevance filtering, CSS- and XPath-based schema extraction, chunking and cosine-similarity strategies, and optional LLM-driven structured JSON extraction. It also includes adaptive crawling, link analysis, URL seeding, virtual-scroll handling, anti-bot and proxy escalation features, and customizable hooks.

Crawl4AI can be installed with pip and used through Python or its command-line interface. It is also distributed as a Dockerized FastAPI server with JWT authentication, browser pooling, monitoring dashboards, a playground, and endpoints for crawling, HTML extraction, screenshots, PDF generation, and JavaScript execution. The repository states that it is licensed under Apache License 2.0.