Tool Open source · BSD-3-Clause
Scrapling is a Python web-scraping framework for fetching, parsing, and crawling websites from a single request to a full-scale crawl. It combines fast HTTP requests, browser-based JavaScript rendering through Playwright, stealth fetching, persistent sessions, proxy rotation, retries, concurrent spiders, pause and resume, and background capture of API responses. Its parser supports CSS and XPath selectors, text and regular-expression search, adaptive element tracking, similar-element matching, DOM navigation, and Markdown output.
Scrapling provides a Scrapy-like spider API with async parsing, throttling, blocked-request detection, optional robots.txt compliance, ready-made crawl templates, link extraction, and JSON, JSONL, CSV, and XML exports. It also includes a CLI, interactive shell, MCP server for AI agents, an Agent Skill, and tools for producing sanitized Markdown suitable for retrieval-augmented generation. The core package requires Python 3.10 or later; browser, AI, RAG, shell, and combined features are available through optional dependencies. The project is distributed under the BSD-3-Clause License and provides Docker images with browser dependencies.
1 use taken from transcripts — each links to the moment in the video.
A Python web scraping framework combining fast HTTP requests, stealth browsers, parsing and concurrent crawling. Fetches static or JavaScript-heavy pages, selects data with CSS or XPath, supports proxy rotation, retries and pause/resume, and exports to JSON, CSV or markdown.
1 in the library.