unclecode / crawl4ai

查看原文 / View Original 编辑 / Edit

unclecode / crawl4ai

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. 🚀🤖 Crawl4AI：开源且对大模型（LLM）友好的网页爬虫与抓取工具。

Crawl4AI Cloud API — Closed Beta (Launching Soon) Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions. 👉 Apply here for early access We’ll be onboarding in phases and working closely with early users. Limited slots. Crawl4AI 云端 API — 封闭测试版（即将推出）。提供可靠的大规模网页提取服务，其构建目标是比现有任何解决方案都更具成本效益。👉 点击此处申请抢先体验。我们将分阶段进行内测，并与早期用户密切合作。名额有限。

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community. Crawl4AI 能将网页转换为干净、适配大模型的 Markdown 格式，适用于 RAG（检索增强生成）、智能体和数据流水线。它速度快、可控性强，并经过了 5 万多名社区用户的实战检验。

✨ Check out latest update v0.9.1 ✨ 查看最新更新 v0.9.1

New in v0.9.1: Patch release with 12 bug fixes across Docker, browser, and core. Adds preserve_classes/preserve_tags whitelist for PruningContentFilter, fixes Windows browser crash, Docker auth gate UI, HTTP timeout unit mismatch, and more. Release notes → v0.9.1 新特性：补丁版本，修复了 Docker、浏览器和核心代码中的 12 个 Bug。为 PruningContentFilter 添加了 preserve_classes/preserve_tags 白名单，修复了 Windows 浏览器崩溃、Docker 认证界面 UI、HTTP 超时单位不匹配等问题。查看发布说明 →

✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Release notes → ✨ 近期 v0.9.0：Docker API 服务器的重大安全默认版本。 默认开启身份验证，除非提供令牌，否则服务器仅绑定本地回环地址，且请求体现在被视为不可信边界。查看发布说明 →

✨ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. Release notes → ✨ 近期 v0.8.7：安全加固版本。 修复了关键的 Docker API 漏洞（RCE、SSRF、身份验证绕过、文件写入、XSS、硬编码 JWT 密钥），添加了 DomainMapper，并发布了关于抓取、深度爬取和 LLM 的修复程序。查看发布说明 →

✨ Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with resume_state and on_state_change callbacks for long-running crawls. New prefetch=True mode for 5-10x faster URL discovery. Release notes → ✨ 此前 v0.8.0：崩溃恢复与预取模式！ 针对长时间爬取任务，通过 resume_state 和 on_state_change 回调实现深度爬取崩溃恢复。新增 prefetch=True 模式，URL 发现速度提升 5-10 倍。查看发布说明 →

✨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. Release notes → ✨ 此前 v0.7.8：稳定性与 Bug 修复版本！ 修复了 11 个 Bug，涵盖 Docker API 问题、LLM 提取改进、URL 处理修复及依赖项更新。查看发布说明 →

🤓 My Personal Story 🤓 我的个人故事

I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters. In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub. 多亏了我父亲，我是在 Amstrad 电脑上长大的，从此从未停止过开发。在研究生期间，我专注于自然语言处理（NLP）并为研究构建了爬虫。正是在那时，我意识到数据提取有多么重要。2023 年，我需要将网页转换为 Markdown。当时的“开源”选项要求注册账号、提供 API 令牌并支付 16 美元，结果效果还很差。我愤怒之下，在几天内构建了 Crawl4AI，随后它迅速走红。现在，它是 GitHub 上星标最多的爬虫项目。

I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing. 我将其开源是为了让所有人都能使用，无需任何门槛。现在，我正在构建一个经济实惠的平台，让任何人都能进行严肃的爬取任务而无需花费巨资。如果你有共鸣，欢迎加入、发送反馈，或者直接去爬取一些精彩的内容。

Why developers pick Crawl4AI 为什么开发者选择 Crawl4AI

  • LLM ready output, smart Markdown with headings, tables, code, citation hints
  • 适配大模型的输出，包含标题、表格、代码和引用提示的智能 Markdown
  • Fast in practice, async browser pool, caching, minimal hops
  • 实战速度快，异步浏览器池，缓存机制，最少跳转
  • Full control, sessions, proxies, cookies, user scripts, hooks
  • 完全掌控，支持会话、代理、Cookie、用户脚本和钩子
  • Adaptive intelligence, learns site patterns, explores only what matters
  • 自适应智能，学习网站模式，只探索有价值的内容
  • Deploy anywhere, zero keys, CLI and Docker, cloud friendly
  • 随处部署，无需密钥，支持 CLI 和 Docker，云原生友好

🚀 Quick Start 🚀 快速开始

Install Crawl4AI: 安装 Crawl4AI：

# Install the package
pip install -U crawl4ai
# For pre release versions
pip install crawl4ai --pre
# Run post-installation setup
crawl4ai-setup
# Verify your installation
crawl4ai-doctor

If you encounter any browser-related issues, you can install them manually: 如果你遇到任何与浏览器相关的问题，可以手动安装：

python -m playwright install --with-deps chromium

Run a simple web crawl with Python: 使用 Python 运行简单的网页爬取：

import asyncio
from crawl4ai import *

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://www.nbcnews.com/business",
        )
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

Or use the new command-line interface: 或者使用新的命令行界面：

# Basic crawl with markdown output
crwl https://www.nbcnews.com/business -o markdown

# Deep crawl with BFS strategy, max 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10

# Use LLM extraction with a specific question
crwl https://www.example.com/products -q "Extract all product prices"

💖 Support Crawl4AI 💖 支持 Crawl4AI

🎉 Sponsorship Program Now Open! After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for startups and enterprises. Be among the first 50 Founding Sponsors for permanent recognition in our Hall of Fame. Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits. 🎉 赞助计划现已开启！ 在服务了超过 5.1 万名开发者并经历了一年的成长后，Crawl4AI 现为初创公司和企业推出专属支持服务。成为首批 50 位创始赞助商，即可在我们的名人堂中获得永久署名。Crawl4AI 是 GitHub 上排名第一的开源爬虫项目。您的支持将使其保持独立、创新并对社区免费，同时您还能直接获得高级权益。

🤝 Sponsorship Tiers 🤝 赞助等级

  • 🌱 Believer ($5/mo) — Join the movement for data democratization
  • 🌱 信徒 ($5/月) — 加入数据民主化运动
  • 🚀 Builder ($50/mo) — Priority support & early access to features
  • 🚀 构建者 ($50/月) — 优先支持与功能抢先体验
  • 💼 Growing Team ($500/mo) — Bi-weekly syncs & optimization help
  • 💼 成长团队 ($500/月) — 双周同步会议与优化帮助
  • 🏢 Data Infrastructure Partner ($2000/mo) — Full partnership with dedicated support
  • 🏢 数据基础设施合作伙伴 ($2000/月) — 全面合作伙伴关系及专属支持

Custom arrangements available - see SPONSORS.md for details & contact. Why sponsor? No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI. See All Tiers & Benefits → 可提供定制方案 - 详情及联系方式请参阅 SPONSORS.md。为什么要赞助？没有 API 速率限制，没有厂商锁定。在 Crawl4AI 创建者的直接指导下，构建并拥有属于你自己的数据流水线。查看所有等级与权益 →

✨ Features ✨ 功能特性

📝 Markdown Generation 📝 Markdown 生成

  • 🧹 Clean Markdown: Generates clean, structured Markdown with accurate formatting.
  • 🧹 干净的 Markdown： 生成格式准确、结构清晰的 Markdown。
  • 🎯 Fit Markdown: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.
  • 🎯 适配 Markdown： 基于启发式过滤，去除噪音和无关部分，便于 AI 处理。
  • 🔗 Citations and References: Converts page links into a numbered reference list with clean citations.
  • 🔗 引用与参考： 将页面链接转换为带清晰引用的编号参考列表。
  • 🛠️ Custom Strategies: Users can create their own Markdown generation strategies tailored to specific needs.
  • 🛠️ 自定义策略： 用户可以根据特定需求创建自己的 Markdown 生成策略。
  • 📚 BM25 Algorithm: Employs BM25-based filtering for extracting core information and removing irrelevant content.
  • 📚 BM25 算法： 采用基于 BM25 的过滤算法来提取核心信息并剔除无关内容。

📊 Structured Data Extraction 📊 结构化数据提取

  • 🤖 LLM-Driven Extraction: Supports all LLMs (open-source and proprietary) for structured data extraction.
  • 🤖 LLM 驱动提取： 支持所有大模型（开源及闭源）进行结构化数据提取。
  • 🧱 Chunking Strategies: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.
  • 🧱 分块策略： 实现多种分块方式（基于主题、正则、句子级别），用于针对性的内容处理。
  • 🌌 Cosine Similarity: Find relevant content chunks based on user queries for semantic extraction.
  • 🌌 余弦相似度： 根据用户查询查找相关内容块，实现语义提取。
  • 🔎 CSS-Based Extraction: Fast schema-based data extraction using XPath and CSS selectors.
  • 🔎 基于 CSS 的提取： 使用 XPath 和 CSS 选择器进行快速的模式化数据提取。
  • 🔧 Schema Definition: Define custom schemas for extracting structured JSON from repetitive patterns.
  • 🔧 模式定义： 定义自定义模式，从重复模式中提取结构化 JSON。

🌐 Browser Integration 🌐 浏览器集成

  • 🖥️ Managed Browser: Use user-owned browsers with full control, avoiding bot detection.
  • 🖥️ 托管浏览器： 使用用户自有的浏览器并拥有完全控制权，避免被机器人检测。
  • 🔄 Remote Browser Control: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
  • 🔄 远程浏览器控制： 连接到 Chrome 开发者工具协议，进行远程、大规模数据提取。
  • 👤 Browser Profiler: …
  • 👤 浏览器分析器： …
← 返回新闻列表 / Back to news list