How to Build a Competitor Intelligence Agent with CrewAI and ZenRows

查看原文 / View Original 编辑 / Edit

How to Build a Competitor Intelligence Agent with CrewAI and ZenRows

如何使用 CrewAI 和 ZenRows 构建竞争对手情报智能体

A competitor intelligence agent enables real-time pricing visibility, automated positioning tracking, and structured competitor briefs for brands without manual research. At the center of these workflows are a researcher agent responsible for gathering data and an analysis agent responsible for generating a summary and a downstream report. 竞争对手情报智能体能够为品牌提供实时价格可见性、自动化定位跟踪以及结构化的竞争对手简报，无需人工研究。这些工作流的核心是一个负责收集数据的研究员智能体，以及一个负责生成摘要和后续报告的分析智能体。

However, one of the things that makes the researcher agent’s output trustworthy is its retrieval layer, since a poor retrieval-layer output can make the analysis agent’s recommendation questionable. If the researcher agent searches the webpage and retrieves a challenge page, an empty response, or blocked content, every downstream conclusion becomes less trustworthy. 然而，研究员智能体输出的可信度很大程度上取决于其检索层，因为糟糕的检索层输出会使分析智能体的建议变得不可靠。如果研究员智能体在搜索网页时获取到的是验证码页面、空响应或被拦截的内容，那么后续的所有结论都将失去可信度。

With a 99.93% success rate on protected websites, ZenRows provides the reliable retrieval layer that makes these workflows practical in production. This tutorial shows why a CrewAI researcher agent can fail on protected competitor pages. It starts with a custom ZenRows-based tool setup, then later shows the MCP server as an alternative approach. ZenRows 在受保护网站上的成功率高达 99.93%，它提供了可靠的检索层，使这些工作流在生产环境中变得切实可行。本教程将展示为什么 CrewAI 研究员智能体在面对受保护的竞争对手页面时会失败。教程将从基于 ZenRows 的自定义工具设置开始，随后介绍作为替代方案的 MCP 服务器。

Prerequisites

前置要求

  • Python 3.10 or newer. If you are on an older Python version, create a dedicated virtual environment with a current Python installation to avoid dependency conflicts. Python 3.10 to 3.13. CrewAI requires this range. Python 3.10 或更高版本。 如果你使用的是旧版 Python，请创建一个使用当前 Python 版本的专用虚拟环境，以避免依赖冲突。CrewAI 要求 Python 版本在 3.10 到 3.13 之间。
  • ZenRows API key. Create an account at zenrows.com and copy your key from the dashboard. This key authenticates every scrape the researcher agent runs for data extraction. ZenRows API 密钥。 在 zenrows.com 创建账户并从仪表板复制你的密钥。该密钥用于验证研究员智能体在数据提取过程中进行的每一次抓取。
  • Anthropic API key. The crew uses Claude to drive both agents. Generate a key in the Anthropic Console. Anthropic API 密钥。 该团队使用 Claude 来驱动这两个智能体。请在 Anthropic 控制台中生成密钥。
  • Install dependencies and the required packages using pip install "crewai[anthropic]" crewai-tools zenrows python-dotenv. Create a .env file and save your API keys there. 使用 pip install "crewai[anthropic]" crewai-tools zenrows python-dotenv 安装依赖项和所需包。创建一个 .env 文件并将你的 API 密钥保存在其中。

Why the built-in ScrapeWebsiteTool fails on competitor pages

为什么内置的 ScrapeWebsiteTool 在竞争对手页面上会失败

The problem, as established earlier, starts before the analysis. A CrewAI workflow depends primarily on the information collected, because a competitor intelligence agent relies on a chain of steps. The researcher AI agent collects information, the analysis agent interprets it, and the final output is built from what the researcher agent retrieved. This means the scraping layer becomes the foundation of the entire workflow. If the researcher agent receives incomplete content, every other agent working with it will work with the wrong data. 正如前面所提到的，问题在分析之前就已经存在。CrewAI 工作流主要依赖于所收集的信息，因为竞争对手情报智能体依赖于一系列步骤。研究员 AI 智能体收集信息，分析智能体进行解读，最终的输出建立在研究员智能体检索到的内容之上。这意味着抓取层成为了整个工作流的基础。如果研究员智能体接收到的是不完整的内容，那么与之协作的其他所有智能体都将基于错误的数据进行工作。

CrewAI is an open-source framework that enables faster prototyping of AI-driven data extraction pipelines by orchestrating autonomous multi-agent systems. One of its features is CrewAI’s built-in ScrapeWebsiteTool, which, though designed to extract and read website URL content, makes standard HTTP requests without any JavaScript rendering. This is a problem because many competitor pages use Cloudflare, DataDome, or similar anti-bot systems that can block standard HTML requests. CrewAI 是一个开源框架，通过编排自主多智能体系统，能够更快地构建 AI 驱动的数据提取流水线原型。其功能之一是内置的 ScrapeWebsiteTool，虽然它旨在提取和读取网站 URL 内容，但它仅进行标准的 HTTP 请求，而不进行任何 JavaScript 渲染。这是一个问题，因为许多竞争对手页面使用 Cloudflare、DataDome 或类似的防爬系统，这些系统会拦截标准的 HTML 请求。

Browsers load the page structure first, then JavaScript fetches and inserts pricing tables and other dynamic content. A standard request using just CrewAI’s built-in ScrapeWebsiteTool would receive the page shell but not the data your researcher agent needs. Likewise, it would return a challenge page or a Cloudflare 1020 error instead of the competitor page if that specified website has an anti-bot protection layer. The agent, however, doesn’t validate the output. It just takes the incomplete output as the actual content. 浏览器会先加载页面结构，然后通过 JavaScript 获取并插入价格表及其他动态内容。仅使用 CrewAI 内置的 ScrapeWebsiteTool 进行标准请求，只会收到页面框架，而无法获取研究员智能体所需的数据。同样，如果目标网站有防爬保护层，它会返回验证码页面或 Cloudflare 1020 错误，而不是竞争对手的页面。然而，智能体并不会验证输出，它只是将不完整的输出视为实际内容。

Let’s set up the ScrapeWebsiteTool for making HTTP requests and target a protected site. Run it yourself and see the challenge it faces. 让我们设置 ScrapeWebsiteTool 来进行 HTTP 请求并针对一个受保护的网站进行测试。你可以亲自运行它，看看它面临的挑战。

from crewai_tools import ScrapeWebsiteTool

# Cloudflare-protected site
tool = ScrapeWebsiteTool(website_url="https://www.yelp.com")
text = tool.run()
print(text)

The output below shows that ScrapeWebsiteTool returned an exception and could not retrieve the relevant results from Yelp because it was behind Cloudflare’s protection. The analyst agent will work with bad data. This is one example of the limitation of CrewAI’s ScrapeWebsiteTool, not proof that it always returns an empty page. 下面的输出显示 ScrapeWebsiteTool 返回了一个异常，并且由于 Yelp 处于 Cloudflare 的保护之下，它无法检索到相关结果。分析智能体将不得不使用错误的数据进行工作。这只是 CrewAI ScrapeWebsiteTool 局限性的一个例子，并不代表它总是返回空页面。

((venv) ) benny@Mac competitor_intelligence % python scrapertool.py
The following text is scraped website content:
yelp.com
Please enable JS and disable any ad blocker
((venv) ) benny@Mac competitor_intelligence %

ZenRows solves the retrieval-layer issue by successfully fetching pages from protected sites.

ZenRows 通过成功从受保护站点获取页面，解决了检索层的问题。

ZenRows is an AI web data infrastructure that combines JavaScript rendering and premium proxy rotation to retrieve data from protected sites. Other recommendations, according to CrewAI’s docs, are Scrapfly and Firecrawl. ZenRows is the right fit for this workflow because it is purpose-built for retrieving data from heavily protected targets and maintains a 99.93% success rate across supported domains. ZenRows 是一个 AI 网络数据基础设施，结合了 JavaScript 渲染和高级代理轮换功能，用于从受保护的站点检索数据。根据 CrewAI 的文档，其他推荐方案还包括 Scrapfly 和 Firecrawl。ZenRows 非常适合此工作流，因为它是专门为从高强度保护的目标中检索数据而构建的，并在支持的域名上保持着 99.93% 的成功率。

Below is a script using ZenRows on the same target (Yelp) to retrieve data. 以下是使用 ZenRows 对同一目标（Yelp）进行数据检索的脚本。

import os
import requests
from dotenv import load_dotenv

load_dotenv()
ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]

# Same protected competitor URL used with ScrapeWebsiteTool
url = "https://www.yelp.com"
params = {
    "url": url,
    "apikey": ZENROWS_API_KEY,
    "js_render": "true",
    "premium_proxy": "true"
}

response = requests.get(
    "https://api.zenrows.com/v1/", 
    params=params, 
    timeout=180
)

print("status code:", response.status_code)
print("bytes returned:", len(response.text))
print(response.text[:1000])

ZenRows was able to reach the site and successfully retrieved the actual page HTML. ZenRows 成功访问了该站点并获取到了实际的页面 HTML。

((venv) ) benny@Mac competitor_intelligence % python zenrowss.py
status code: 200
bytes returned: 501630
<html lang="en-US" prefix="og: http://ogp.me/ns#" style="margin: 0; padding: 0; border: 0; font-size: 100%; font: inherit; vertical-align: baseline;" data-cookbook-theme="light" class="">
<head>
<script>document.documentElement.className=document.documentElement.className.replace(/no-js/,"js");</script>
<meta http-equiv="Content-Language" content="en-US">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="mask-icon" sizes="any" href="https://s3-media0.fl
← 返回新闻列表 / Back to news list