GitHub - unclecode/crawl4ai: 🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8K

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.

🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon)

Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.

👉 Apply here for early access

We’ll be onboarding in phases and working closely with early users. Limited slots.

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.

✨ Check out latest update v0.9.2

✨ New in v0.9.2 : Maintenance patch release.

Fixes a MemoryAdaptiveDispatcher task/page leak when a streaming crawl is closed, Docker Playground "Advanced Config" and Monitor WebSocket auth, Playwright headless-shell packaging, and GPU ( ENABLE_GPU=true ) Docker builds. Release notes →

✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Release notes →

✨ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. Release notes →

✨ Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with resume_state and on_state_change callbacks for long-running crawls. New prefetch=True mode for 5-10x faster URL discovery. Release notes →

✨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. Release notes →

🤓 My Personal Story I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.

In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.

I made it open source for availability , anyone can use it without a gate. Now I’m building the platform for affordability , anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.

Why developers pick Crawl4AI

• LLM ready output , smart Markdown with headings, tables, code, citation hints

• Fast in practice , async browser pool, caching, minimal hops

• Full control , sessions, proxies, cookies, user scripts, hooks

• Adaptive intelligence , learns site patterns, explores only what matters

• Deploy anywhere , zero keys, CLI and Docker, cloud friendly

🚀 Quick Start

• Install Crawl4AI:

Install the package

pip install -U crawl4ai

For pre release versions

pip install crawl4ai --pre

Run post-installation setup

crawl4ai-setup

Verify your installation

crawl4ai-doctor

If you encounter any browser-related issues, you can install them manually:

python -m playwright install --with-deps chromium

• Run a simple web crawl with Python:

import asyncio from crawl4ai import *

async def main (): async with AsyncWebCrawler () as crawler : result = await crawler . arun ( url = "https://www.nbcnews.com/business" , ) print ( result . markdown )

if __name__ == "__main__" : asyncio . run ( main ())

• Or use the new command-line interface:

Basic crawl with markdown output

crwl https://www.nbcnews.com/business -o markdown

Deep crawl with BFS strategy, max 10 pages

crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10

Use LLM extraction with a specific question

crwl https://www.example.com/products -q " Extract all product prices "

💖 Support Crawl4AI

🎉 Sponsorship Program Now Open! After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for startups and enterprises . Be among the first 50 Founding Sponsors for permanent recognition in our Hall of Fame.

Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits.

🤝 Sponsorship Tiers

• 🌱 Believer ($5/mo) — Join the movement for data democratization

• 🚀 Builder ($50/mo) — Priority support & early access to features

• 💼 Growing Team ($500/mo) — Bi-weekly syncs & optimization help

• 🏢 Data Infrastructure Partner ($2000/mo) — Full partnership with dedicated support

Custom arrangements available - see SPONSORS.md for details & contact

Why sponsor?

No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI.

See All Tiers & Benefits →

✨ Features

📝 Markdown Generation

• 🧹 Clean Markdown : Generates clean, structured Markdown with accurate formatting.

• 🎯 Fit Markdown : Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.

• 🔗 Citations and References : Converts page links into a numbered reference list with clean citations.

• 🛠️ Custom Strategies : Users can create their own Markdown generation strategies tailored to specific needs.

• 📚 BM25 Algorithm : Employs BM25-based filtering for extracting core information and removing irrelevant content.

📊 Structured Data Extraction

• 🤖 LLM-Driven Extraction : Supports all LLMs (open-source and proprietary) for structured data extraction.

• 🧱 Chunking Strategies : Implements chunking (topic-based, regex, sentence-level) for targeted content processing.

• 🌌 Cosine Similarity : Find relevant content chunks based on user queries for semantic extraction.

• 🔎 CSS-Based Extraction : Fast schema-based data extraction using XPath and CSS selectors.

• 🔧 Schema Definition : Define custom schemas for extracting structured JSON from repetitive patterns.

🌐 Browser Integration

• 🖥️ Managed Browser : Use user-owned browsers with full control, avoiding bot detection.

• 🔄 Remote Browser Control : Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.

• 👤 Browser Profiler : Create and manage persistent profiles with saved authentication states, cookies, and settings.

• 🔒 Session Management : Preserve browser states and reuse them for multi-step crawling.

• 🧩 Proxy Support : Seamlessly connect to proxies with authentication for secure access.

• ⚙️ Full Browser Control : Modify headers, cookies, user agents, and more for tailored crawling setups.

• 🌍 Multi-Browser Support : Compatible with Chromium, Firefox, and WebKit.

• 📐 Dynamic Viewport Adjustment : Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.

🔎 Crawling & Scraping

• 🖼️ Media Support : Extract images, audio, videos, and responsive image formats like srcset and picture .

• 🚀 Dynamic Crawling : Execute JS and wait for async or sync for dynamic content extraction.

• 📸 Screenshots : Capture page screenshots during crawling for debugging or analysis.

• 📂 Raw Data Crawling : Directly process raw HTML ( raw: ) or local files ( file:// ).

• 🔗 Comprehensive Link Extraction : Extracts internal, external links, and embedded iframe content.

• 🛠️ Customizable Hooks : Define hooks at every step to customize crawling behavior (supports both string and function-based APIs).

• 💾 Caching : Cache data for improved speed and to avoid redundant fetches.

• 📄 Metadata Extraction : Retrieve structured metadata from web pages.

• 📡 IFrame Content Extraction : Seamless extraction from embedded iframe content.

• 🕵️ Lazy Load Handling : Waits for images to fully load, ensuring no content is missed due to lazy loading.

• 🔄 Full-Page Scanning : Simulates scrolling to load and capture all dynamic content, perfect for infinite scroll pages.

🚀 Deployment

• 🐳 Dockerized Setup : Optimized Docker image with FastAPI server for easy deployment.

• 🔑 Secure Authentication : Built-in JWT token authentication for API security.

• 🔄 API Gateway : One-click deployment with secure token authentication for API-based workflows.

• 🌐 Scalable Architecture : Designed for mass-scale production and optimized server performance.

• ☁️ Cloud Deployment : Ready-to-deploy configurations for major cloud platforms.

🎯 Additional Features

• 🕶️ Stealth Mode : Avoid bot detection by mimicking real users.

• 🏷️ Tag-Based Content Extraction : Refine crawling based on custom tags, headers, or metadata.

• 🔗 Link Analysis : Extract and analyze all links for detailed data exploration.

• 🛡️ Error Handling : Robust error management for seamless execution.

• 🔐 CORS & Static Serving : Supports filesystem-based caching and cross-origin requests.

• 📖 Clear Documentation : Simplified and updated guides for onboarding and advanced usage.

• 🙌 Community Recognition : Acknowledges contributors and pull requests for transparency.

Try it Now!

✨ Play around with this

✨ Visit our Documentation Website

Installation 🛠️

Crawl4AI offers flexible installation options to suit various use cases. You can install it as a Python package or use Docker.

🐍 Using pip Choose the installation option that best fits your needs:

Basic Installation

For basic web crawling and scraping tasks:

pip install crawl4ai

crawl4ai-setup # Setup the browser

By default, this will install the asynchronous version of Crawl4AI, using Playwright for web crawling.

👉 Note : When you install Crawl4AI, the crawl4ai-setup should automatically install and set up Playwright. However, if you encounter any Playwright-related errors, you can manually install it using one of these methods:

- Through the command line:

playwright install

- If the above doesn't work, try this more specific command:

python -m playwright install chromium

This second method has proven to be more reliable in some cases.

Installation with Synchronous Version

The sync version is deprecated and will be removed in future versions. If you need the synchronous version using Selenium:

pip install crawl4ai[sync]

Development Installation

For contributors who plan to modify the source code:

git clone https://github.com/unclecode/crawl4ai.git

cd crawl4ai

pip install -e . # Basic installation in editable mode

Install optional features:

pip install -e " .[torch] " # With PyTorch features

pip install -e " .[transformer] " # With Transformer features

pip install -e " .[cosine] " # With cosine similarity features

pip install -e " .[sync] " # With synchronous crawling (Selenium)

pip install -e " .[all] " # Install all optional features

🐳 Docker Deployment

🚀 Now Available! Our completely redesigned Docker implementation is here! This new solution makes deployment more efficient and seamless than ever.

New Docker Features

The new Docker implementation includes:

• Real-time Monitoring Dashboard with live system metrics and browser pool visibility

• Browser pooling with page pre-warming for faster response times

• Interactive playground to test and generate request code

• MCP integration for direct connection to AI tools like Claude Code

• Comprehensive API endpoints including HTML extraction, screenshots, PDF generation, and JavaScript execution

• Multi-architecture support with automatic detection (AMD64/ARM64)

• Optimized resources with improved memory management

Getting Started

Pull and run the latest release

docker pull unclecode/crawl4ai:latest

docker run -d -p 11235:11235 --name crawl4ai --shm-size=1g unclecode/crawl4ai:latest

Visit the monitoring dashboard at http://localhost:11235/dashboard

Or the playground at http://localhost:11235/playground

Quick Test

Run a quick test (works for both Docker options):

import requests

Submit a crawl job

response = requests . post ( "http://localhost:11235/crawl" , json = { "urls" : [ "https://example.com" ], "priority" : 10 } ) if response . status_code == 200 : print ( "Crawl job submitted successfully." )

if "results" in response . json (): results = response . json ()[ "results" ] print ( "Crawl job completed. Results:" ) for result in results : print ( result ) else : task_id = response . json ()[ "task_id" ] print ( f"Crawl job submitted.

Task ID:: { task_id } " ) result = requests . get ( f"http://localhost:11235/task/ { task_id } " )

For more examples, see our Docker Examples . For advanced configuration, monitoring features, and production deployment, see our Self-Hosting Guide .

🔬 Advanced Usage Examples 🔬

You can check the project structure in the directory docs/examples . Over there, you can find a variety of examples; here, some popular examples are shared.

📝 Heuristic Markdown Generation with Clean and Fit Markdown import asyncio from crawl4ai import AsyncWebCrawler , BrowserConfig , CrawlerRunConfig , CacheMode from crawl4ai .

content_filter_strategy import PruningContentFilter , BM25ContentFilter from crawl4ai . markdown_generation_strategy import DefaultMarkdownGenerator

async def main (): browser_config = BrowserConfig ( headless = True , verbose = True , ) run_config = CrawlerRunConfig ( cache_mode = CacheMode . ENABLED , markdown_generator = DefaultMarkdownGenerator ( content_filter = PruningContentFilter ( threshold = 0.

48 , threshold_type = "fixed" , min_word_threshold = 0 ) ),

markdown_generator=DefaultMarkdownGenerator(

content_filter=BM25ContentFilter(user_query="WHEN_WE_FOCUS_BASED_ON_A_USER_QUERY", bm25_threshold=1.0)

),

)

async with AsyncWebCrawler ( config = browser_config ) as crawler : result = await crawler . arun ( url = "https://docs.micronaut.io/4.9.9/guide/" , config = run_config ) print ( len ( result . markdown . raw_markdown )) print ( len ( result . markdown . fit_markdown ))

if __name__ == "__main__" : asyncio . run ( main ())

🖥️ Executing JavaScript & Extract Structured Data without LLMs import asyncio from crawl4ai import AsyncWebCrawler , BrowserConfig , CrawlerRunConfig , CacheMode from crawl4ai import JsonCssExtractionStrategy import json

async def main (): schema = { "name" : "KidoCode Courses" , "baseSelector" : "section.charge-methodology .w-tab-content > div" , "fields" : [ { "name" : "section_title" , "selector" : "h3.

heading-50" , "type" : "text" , }, { "name" : "section_description" , "selector" : ".charge-content" , "type" : "text" , }, { "name" : "course_name" , "selector" : ".text-block-93" , "type" : "text" , }, { "name" : "course_description" , "selector" : ".

course-content-text" , "type" : "text" , }, { "name" : "course_icon" , "selector" : ".image-92" , "type" : "attribute" , "attribute" : "src" } ] }

extraction_strategy = JsonCssExtractionStrategy ( schema , verbose = True )

browser_config = BrowserConfig ( headless = False , verbose = True ) run_config = CrawlerRunConfig ( extraction_strategy = extraction_strategy , js_code = [ """(async () => {const tabs = document.querySelectorAll("section.charge-methodology .

tabs-menu-3 > div");for(let tab of tabs) {tab.scrollIntoView();tab.click();await new Promise(r => setTimeout(r, 500));}})();""" ], cache_mode = CacheMode . BYPASS )

async with AsyncWebCrawler ( config = browser_config ) as crawler :

result = await crawler . arun ( url = "https://www.kidocode.com/degrees/technology" , config = run_config )

companies = json . loads ( result . extracted_content ) print ( f"Successfully extracted { len ( companies ) } companies" ) print ( json . dumps ( companies [ 0 ], indent = 2 ))

if __name__ == "__main__" : asyncio . run ( main ())

📚 Extracting Structured Data with LLMs import os import asyncio from crawl4ai import AsyncWebCrawler , BrowserConfig , CrawlerRunConfig , CacheMode , LLMConfig from crawl4ai import LLMExtractionStrategy from pydantic import BaseModel , Field

class OpenAIModelFee ( BaseModel ): model_name : str = Field (., description = "Name of the OpenAI model." ) input_fee : str = Field (., description = "Fee for input token for the OpenAI model." ) output_fee : str = Field (.

, description = "Fee for output token for the OpenAI model." )

async def main (): browser_config = BrowserConfig ( verbose = True ) run_config = CrawlerRunConfig ( word_count_threshold = 1 , extraction_strategy = LLMExtractionStrategy (

Here you can use any provider that Litellm library supports, for instance: ollama/qwen2

provider="ollama/qwen2", api_token="no-token",

llm_config = LLMConfig ( provider = "openai/gpt-4o" , api_token = os . getenv ( 'OPENAI_API_KEY' )), schema = OpenAIModelFee .

schema (), extraction_type = "schema" , instruction = """From the crawled content, extract all mentioned model names along with their fees for input and output tokens. Do not miss any models in the entire content.

One extracted model JSON format should look like this: {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}.""" ), cache_mode = CacheMode . BYPASS , )

async with AsyncWebCrawler ( config = browser_config ) as crawler : result = await crawler . arun ( url = 'https://openai.com/api/pricing/' , config = run_config ) print ( result . extracted_content )

if __name__ == "__main__" : asyncio . run ( main ())

🤖 Using Your own Browser with Custom User Profile import os , sys from pathlib import Path import asyncio , time from crawl4ai import AsyncWebCrawler , BrowserConfig , CrawlerRunConfig , CacheMode

async def test_news_crawl ():

Create a persistent user data directory

user_data_dir = os . path . join ( Path . home (), ".crawl4ai" , "browser_profile" ) os . makedirs ( user_data_dir , exist_ok = True )

browser_config = BrowserConfig ( verbose = True , headless = True , user_data_dir = user_data_dir , use_persistent_context = True , ) run_config = CrawlerRunConfig ( cache_mode = CacheMode . BYPASS )

async with AsyncWebCrawler ( config = browser_config ) as crawler : url = "ADDRESS_OF_A_CHALLENGING_WEBSITE"

result = await crawler . arun ( url , config = run_config , magic = True , )

print ( f"Successfully crawled { url } " ) print ( f"Content length: { len ( result . markdown ) } " )

✨ Recent Updates

Version 0.9.2 Release Highlights - Maintenance Bug Fixes A maintenance patch release with bug fixes across the dispatcher, Docker, and GPU builds. MemoryAdaptiveDispatcher no longer leaks crawl tasks and browser pages when a streaming crawl is closed.

Docker fixes cover the Playground "Advanced Config" 400, the Monitor WebSocket 500 under JWT auth, and Playwright headless-shell packaging. ENABLE_GPU=true Docker builds no longer fail on the CUDA toolkit.

No new features, no breaking changes.

pip install -U crawl4ai

Full v0.9.2 Release Notes →

Version 0.9.1 Release Highlights - Bug Fixes & PruningContentFilter Whitelist A patch release with 12 bug fixes and one new feature.

The new preserve_classes / preserve_tags parameters for PruningContentFilter let you whitelist CSS classes or HTML tags that should never be pruned — useful for protecting short metadata elements like author names and timestamps.

Bug fixes span Docker (auth gate UI, supervisord/redis dirs, FastAPI compatibility, redis auth), browser (Windows channel crash, context snapshot leak), core (HTTP timeout unit mismatch, best-first ordering), and extraction (html2text table attributes).

pip install -U crawl4ai

Full v0.9.1 Release Notes →

Version 0.9.0 Release Highlights - Secure-by-Default Docker Server A major, secure-by-default release of the Docker API server.

The out-of-the-box deployment is hardened with defense in depth: authentication is on by default, the server binds loopback unless you give it a token, and the network request body is treated as an untrusted trust boundary.

pip install -U crawl4ai

Migration Guide → · Full v0.9.0 Release Notes →

Version 0.8.7 Release Highlights - Security Hardening, DomainMapper & Community Fixes A security-hardening release.

Fixes critical Docker API vulnerabilities (AST sandbox escape RCE, hook sandbox RCE, hardcoded JWT secret, SSRF on webhook and crawl endpoints, arbitrary file write, monitor auth bypass, stored XSS, and unauthenticated JS execution), adds the DomainMapper feature, and ships a batch of scraping, deep-crawl, and LLM fixes.

If you self-host the Docker API, upgrade immediately.

pip install -U crawl4ai

Full v0.8.7 Release Notes →

Version 0.8.6 - Security Hotfix: litellm Supply Chain Fix Replaced litellm dependency with unclecode-litellm due to a PyPI supply chain compromise affecting the original package. If you're on v0.8.5 or earlier, upgrade immediately.

pip install -U crawl4ai

Version 0.8.5 Release Highlights - Anti-Bot Detection, Shadow DOM & 60+ Bug Fixes Our biggest release since v0.8.0. Anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes.

- 🛡️ Anti-Bot Detection & Proxy Escalation :

• 3-tier detection: known vendors, generic block indicators, structural integrity checks

• Automatic retry with proxy chain and fallback fetch function

from crawl4ai import CrawlerRunConfig from crawl4ai . async_configs import ProxyConfig

config = CrawlerRunConfig ( proxy_config = [ ProxyConfig . DIRECT , ProxyConfig ( server = "http://my-proxy:8080" )], max_retries = 2 , fallback_fetch_function = my_web_unlocker , )

- 🌑 Shadow DOM Flattening :

• Extract content hidden inside shadow DOM components

config = CrawlerRunConfig ( flatten_shadow_dom = True )

- 🛑 Deep Crawl Cancellation :

• Stop long crawls gracefully with cancel() or should_cancel callback

• Works with BFS, DFS, and BestFirst strategies

- ⚙️ Config Defaults API :

• set_defaults() / get_defaults() / reset_defaults() on BrowserConfig and CrawlerRunConfig

- 🔒 Critical Security Fixes :

• RCE via deserialization in Docker /crawl endpoint — removed eval() , added allowlist

• Redis CVE-2025-49844 (CVSS 10.0) — upgraded to 7.2.7

- 60+ Bug Fixes across browser management, proxy, deep crawling, extraction, CLI, and Docker

Full v0.8.5 Release Notes →

Version 0.8.0 Release Highlights - Crash Recovery & Prefetch Mode This release introduces crash recovery for deep crawls, a new prefetch mode for fast URL discovery, and critical security fixes for Docker deployments.

- 🔄 Deep Crawl Crash Recovery :

• on_state_change callback fires after each URL for real-time state persistence

• resume_state parameter to continue from a saved checkpoint

• JSON-serializable state for Redis/database storage

• Works with BFS, DFS, and Best-First strategies

from crawl4ai . deep_crawling import BFSDeepCrawlStrategy

strategy = BFSDeepCrawlStrategy ( max_depth = 3 , resume_state = saved_state , # Continue from checkpoint on_state_change = save_to_redis , # Called after each URL )

- ⚡ Prefetch Mode for Fast URL Discovery :

prefetch=True skips markdown, extraction, and media processing

• 5-10x faster than full processing

• Perfect for two-phase crawling: discover first, process selectively

config = CrawlerRunConfig ( prefetch = True ) result = await crawler . arun ( "https://example.com" , config = config )

Returns HTML and links only - no markdown generation

- 🔒 Security Fixes (Docker API) :

• Hooks disabled by default ( CRAWL4AI_HOOKS_ENABLED=false )

• file:// URLs blocked on API endpoints to prevent LFI

• __import__ removed from hook execution sandbox

Full v0.8.0 Release Notes →

Version 0.7.8 Release Highlights - Stability & Bug Fix Release This release focuses on stability with 11 bug fixes addressing issues reported by the community. No new features, but significant improvements to reliability.

- 🐳 Docker API Fixes :

• Fixed ContentRelevanceFilter deserialization in deep crawl requests (#1642)

• Fixed ProxyConfig JSON serialization in BrowserConfig.to_dict() (#1629)

• Fixed .cache folder permissions in Docker image (#1638)

- 🤖 LLM Extraction Improvements :

from crawl4ai import LLMConfig

• Configurable rate limiter backoff with new LLMConfig parameters (#1269):

config = LLMConfig ( provider = "openai/gpt-4o-mini" , backoff_base_delay = 5 , # Wait 5s on first retry backoff_max_attempts = 5 , # Try up to 5 times backoff_exponential_factor = 3 # Multiply delay by 3 each attempt )

from crawl4ai import LLMExtractionStrategy

• HTML input format support for LLMExtractionStrategy (#1178):

strategy = LLMExtractionStrategy ( llm_config = config , instruction = "Extract table data" , input_format = "html" # Now supports: "html", "markdown", "fit_markdown" )

• Fixed raw HTML URL variable - extraction strategies now receive "Raw HTML" instead of HTML blob (#1116)

- 🔗 URL Handling :

• Fixed relative URL resolution after JavaScript redirects (#1268)

• Fixed import statement formatting in extracted code (#1181)

- 📦 Dependency Updates :

• Replaced deprecated PyPDF2 with pypdf (#1412)

• Pydantic v2 ConfigDict compatibility - no more deprecation warnings (#678)

- 🧠 AdaptiveCrawler :

• Fixed query expansion to actually use LLM instead of hardcoded mock data (#1621)

Full v0.7.8 Release Notes →

Version 0.7.7 Release Highlights - The Self-Hosting & Monitoring Update

- 📊 Real-time Monitoring Dashboard : Interactive web UI with live system metrics and browser pool visibility

Access the monitoring dashboard

Visit: http://localhost:11235/dashboard

Real-time metrics include:

- System health (CPU, memory, network, uptime)

- Active and completed request tracking

- Browser pool management (permanent/hot/cold)

- Janitor cleanup events

- Error monitoring with full context

- 🔌 Comprehensive Monitor API : Complete REST API for programmatic access to all monitoring data

import httpx

async with httpx . AsyncClient () as client :

System health

health = await client . get ( "http://localhost:11235/monitor/health" )

Request tracking

requests = await client . get ( "http://localhost:11235/monitor/requests" )

Browser pool status

browsers = await client . get ( "http://localhost:11235/monitor/browsers" )

Endpoint statistics

stats = await client . get ( "http://localhost:11235/monitor/endpoints/stats" )

- ⚡ WebSocket Streaming : Real-time updates every 2 seconds for custom dashboards

- 🔥 Smart Browser Pool : 3-tier architecture (permanent/hot/cold) with automatic promotion and cleanup

- 🧹 Janitor System : Automatic resource management with event logging

- 🎮 Control Actions : Manual browser management (kill, restart, cleanup) via API

- 📈 Production Metrics : 6 critical metrics for operational excellence with Prometheus integration

- 🐛 Critical Bug Fixes :

• Fixed async LLM extraction blocking issue (#1055)

• Enhanced DFS deep crawl strategy (#1607)

• Fixed sitemap parsing in AsyncUrlSeeder (#1598)

• Resolved browser viewport configuration (#1495)

• Fixed CDP timing with exponential backoff (#1528)

• Security update for pyOpenSSL (>=25.3.0)

Full v0.7.7 Release Notes →

Version 0.7.5 Release Highlights - The Docker Hooks & Security Update

- 🔧 Docker Hooks System : Complete pipeline customization with user-provided Python functions at 8 key points

- ✨ Function-Based Hooks API (NEW) : Write hooks as regular Python functions with full IDE support:

from crawl4ai import hooks_to_string from crawl4ai . docker_client import Crawl4aiDockerClient

Define hooks as regular Python functions

async def on_page_context_created ( page , context , ** kwargs ): """Block images to speed up crawling""" await context . route ( "**/*.{png,jpg,jpeg,gif,webp}" , lambda route : route . abort ()) await page . set_viewport_size ({ "width" : 1920 , "height" : 1080 }) return page

async def before_goto ( page , context , url , ** kwargs ): """Add custom headers""" await page . set_extra_http_headers ({ 'X-Crawl4AI' : 'v0.7.5' }) return page

Option 1: Use hooks_to_string() utility for REST API

hooks_code = hooks_to_string ({ "on_page_context_created" : on_page_context_created , "before_goto" : before_goto })

Option 2: Docker client with automatic conversion (Recommended)

client = Crawl4aiDockerClient ( base_url = "http://l

— 本文由 AI 根据公开来源辅助整理，命令、版本与许可证请在使用前到原始页面复核。