Crawl4AI

Crawl4AI produces two types of Markdown output as described on its official site. Clean Markdown preserves accurate page formatting with headings, tables, code blocks, and citation hints. Fit Markdown applies heuristic based filtering through a pruning algorithm or BM25 relevance scoring to strip boilerplate, navigation, and footer noise.

This dual output is specifically designed for RAG pipelines and direct LLM ingestion. Users can also build custom Markdown generation strategies to match their exact pipeline requirements.

The tool provides two distinct extraction paths. For pages with predictable layouts, the CSS and XPath based JsonCssExtractionStrategy pulls structured JSON using schema definitions and requires zero LLM calls.

For complex or unpredictable pages, the LLMExtractionStrategy connects to any LLM provider (OpenAI, Ollama, DeepSeek, and others) and uses Pydantic schemas to return perfectly structured data. Chunking strategies including topic based, regex, and sentence level processing handle large pages efficiently.

Announced on crawl4ai.com as a flagship capability, adaptive crawling uses information foraging algorithms with a three layer scoring system that measures coverage, consistency, and saturation. Rather than crawling every page on a site, it evaluates content relevance at each step and stops automatically when confidence thresholds are met.

It supports both a statistical strategy (fast, free, term based) and an embedding strategy (semantic understanding with query expansion). This prevents over crawling and saves significant compute resources.

Introduced in v0.8.5, the three tier anti bot detection system checks known vendor signatures, generic block indicators, and structural integrity of returned pages. When a block is detected, the system automatically retries through a configurable proxy chain with fallback fetch functions. Combined with stealth mode that mimics real user behaviour and the undetected browser mode from v0.7.3, this gives Crawl4AI a strong toolkit for accessing protected sites.

For large scale jobs that span thousands of pages, deep crawl strategies (BFS, DFS, Best First) include built-in crash recovery as released in v0.8.0. An on_state_change callback persists state after each URL, and the resume_state parameter lets you continue from the exact checkpoint after a failure.

The prefetch mode skips Markdown generation and extraction entirely, enabling URL discovery at 5 to 10 times normal speed for two phase crawling workflows.

Crawl4AI ships an optimised Docker image featuring a FastAPI server, JWT token authentication, a real time monitoring dashboard with live system metrics, and a three tier browser pool (permanent, hot, cold) with page pre-warming. The interactive playground lets teams test crawl configurations and generate request code without writing scripts.

MCP integration connects directly to AI tools like Claude Code. Multi architecture support with automatic AMD64 and ARM64 detection ensures it runs on any cloud provider.