Firecrawl

Crawl4AI is an open-source Python web crawler and scraper that converts web pages into structured, LLM-ready Markdown for retrieval-augmented generation, agents, and data pipelines. Its asynchronous Playwright-based crawler supports Chromium, Firefox, and WebKit, dynamic JavaScript pages, sessions, persistent browser profiles, cookies, headers, proxies, screenshots, media, iframes, lazy loading, full-page scanning, caching, and deep crawling with BFS, DFS, and best-first strategies. For extraction, it provides heuristic Markdown filtering including BM25-based relevance filtering, CSS- and XPath-based schema extraction, chunking and cosine-similarity strategies, and optional LLM-driven structured JSON extraction. It also includes adaptive crawling, link analysis, URL seeding, virtual-scroll handling, anti-bot and proxy escalation features, and customizable hooks. Crawl4AI can be installed with pip and used through Python or its command-line interface. It is also distributed as a Dockerized FastAPI server with JWT authentication, browser pooling, monitoring dashboards, a playground, and endpoints for crawling, HTML extraction, screenshots, PDF generation, and JavaScript execution. The repository states that it is licensed under Apache License 2.0.

CrawlSEO is an open-source, self-hosted SEO monitoring dashboard for founders. It combines Google Search Console analytics for keywords, pages, clicks, impressions, click-through rate, and position changes with 28-day comparisons; a concurrent site crawler that handles up to 2,000 pages, assigns a health score, identifies 16 issue types, scores content, and provides per-page audit and remediation data; Core Web Vitals monitoring through PageSpeed Insights; historical rank tracking; SEO opportunity detection; alerts; and CSV export. Optional keyword research and backlink analysis use DataForSEO with a bring-your-own-key setup, while Google Autocomplete provides a free fallback for keyword suggestions. The included Model Context Protocol server exposes 10 tools for Claude Code, Claude Desktop, and Cursor. Connected AI agents can query SEO data, run crawls, inspect crawl issues and performance, and find opportunities. The application is built with Next.js, TypeScript, PostgreSQL, Prisma, and NextAuth.js, and is distributed under the MIT license for local or Docker Compose deployment with Google OAuth and a database. It is described as free and open source, with self-hosting intended to keep site data under the operator's control.

Camofox Browser is an open-source anti-detection headless browser server for AI agents, built by the team behind jo and powered by Camoufox, a Firefox fork that applies fingerprint spoofing at the C++ level. It exposes browser automation through a REST API with accessibility snapshots, stable element references for clicking and typing, navigation and search macros, screenshots, downloads, file uploads, structured extraction, YouTube transcript extraction, and OpenAPI documentation. Browser contexts provide isolated cookies and storage, with optional persistence, cookie import, proxy and GeoIP configuration, VNC-based interactive login, and Playwright session tracing. It can run from npm, Docker, Fly.io, or Railway, and supports lazy browser startup and idle shutdown. The repository is licensed under MIT.

Apify is a marketplace and platform for web-data extraction and AI automation tools called Actors. Actors collect data from sources such as TikTok, Instagram, Google Maps, and websites; their results can be exported, accessed through an API, scheduled, monitored, or integrated with applications, workflows, and AI agents. Apify’s Website Content Crawler cleans HTML, extracts website text in Markdown and other rich formats, downloads files, and supplies content to AI models, LLM applications, vector databases, and retrieval-augmented generation pipelines, with integrations including LangChain and LlamaIndex.

Nimble Web Search Agents are domain-specific web research agents from Nimble that perform self-learning search, crawling, structured extraction, dataset enrichment, and web monitoring. They use auditable Search Plans, a proprietary index, and memory that adapts to a use case after each run to retrieve targeted data from the live web, including JavaScript-rendered pages, without requiring an LLM to parse raw pages. Nimble exposes these capabilities through APIs for search, complex research workflows, dataset building, monitoring, extraction, and crawling.