Cloudflare AI Crawl Control

Cloudflare Web Application Firewall (WAF) is Cloudflare's edge security feature that protects web applications by filtering, monitoring, and blocking HTTP traffic to mitigate common exploits such as SQL injection, cross-site scripting, and other OWASP threats. It is provided by Cloudflare as part of its CDN and edge platform and includes configurable firewall rules, managed rulesets, and integrations with bot management and DDoS protection.

camelAI is an AI coding assistant platform built on Cloudflare Workers and Durable Objects. Each chat thread runs a coding agent in its own persistent workspace, retaining chat state and project files while providing workspace-aware tools. The platform connects to APIs, databases, email, Slack, Discord, and other services; supports application builds, notebook analysis, and SQL execution in isolated short-lived sandboxes; and provides previews and publishing through Workers for Platforms. Its agent harness is built on pi's lower-level agent-loop and state-management libraries. The agent uses native file tools and writes JavaScript rather than shell commands; Code Mode executes that JavaScript in fresh V8 isolates with explicit platform and connection methods. Project files are stored in WorkspaceFilesystem Durable Objects, using Durable Object SQLite for small files and R2 for larger ones, while Cloudflare Artifacts provides Git history. Linux sandbox containers are reserved for jobs requiring them, including builds, notebook analysis, and database queries, and credentials remain outside the execution sandbox. The platform can use Anthropic, OpenAI, OpenRouter, Bedrock, and custom model endpoints. Its repository is released under the MIT License and includes local development setup using Node.js, Bun, and a Cloudflare account.

Cloudflare is an internet infrastructure and security platform that provides hosting and delivery infrastructure for applications, agents, websites, and data. Its network combines compute, connectivity, and security at edge locations, with Workers running code close to users and backend data; the company states that its network operates in more than 335 cities and reaches 95% of the world's Internet-connected population within 50 milliseconds. Cloudflare also provides AI crawler controls that let site owners inspect, allow, or block crawler activity, along with pay-per-crawl and a monetization gateway for pages, data sets, APIs, MCP tool calls, files, and search indexes. The described x402 flow uses an HTTP 402 response to request payment, after which an agent pays, retries with proof, and Cloudflare verifies the payment at the edge.

Firecrawl is an open-source web context API and hosted service for AI agents and applications. It searches the web, scrapes individual pages, crawls websites, maps site URLs, and batch-processes large URL sets, returning content as clean Markdown, HTML, screenshots, structured JSON, and other extracted data. It handles JavaScript-heavy pages, rotating proxies, orchestration, rate limits, and blocked content, and can parse web-hosted PDFs and DOCX files. Its interaction endpoint lets users or agents click, scroll, write, wait, and press on a page before extracting content; its agent endpoint gathers web data from a natural-language request without requiring URLs. Firecrawl provides Python and Node.js SDKs, cURL and CLI interfaces, and an MCP connection for AI agents and applications.

Crawl4AI is an open-source Python web crawler and scraper that converts web pages into structured, LLM-ready Markdown for retrieval-augmented generation, agents, and data pipelines. Its asynchronous Playwright-based crawler supports Chromium, Firefox, and WebKit, dynamic JavaScript pages, sessions, persistent browser profiles, cookies, headers, proxies, screenshots, media, iframes, lazy loading, full-page scanning, caching, and deep crawling with BFS, DFS, and best-first strategies. For extraction, it provides heuristic Markdown filtering including BM25-based relevance filtering, CSS- and XPath-based schema extraction, chunking and cosine-similarity strategies, and optional LLM-driven structured JSON extraction. It also includes adaptive crawling, link analysis, URL seeding, virtual-scroll handling, anti-bot and proxy escalation features, and customizable hooks. Crawl4AI can be installed with pip and used through Python or its command-line interface. It is also distributed as a Dockerized FastAPI server with JWT authentication, browser pooling, monitoring dashboards, a playground, and endpoints for crawling, HTML extraction, screenshots, PDF generation, and JavaScript execution. The repository states that it is licensed under Apache License 2.0.