Get Notified When You Gain or Lose a Backlink: A Python Monitor Without an SEO Subscription

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Losing a good backlink is silent. Nobody emails you when a partner redesigns their site and drops your link, or when a competitor gets a mention in a big publication. The large SEO suites will alert you, but they’re priced as monthly subscriptions, and often you only need the alert for a handful of domains. This post covers: where backlink data comes from (and why scraping an SEO tool’s pages is a bad idea), how to turn “all backlinks” into “what changed since last time”, what it costs, with numbers from a real run.

失去一个优质的反向链接往往是悄无声息的。当合作伙伴改版网站删除了你的链接，或者竞争对手在大型出版物中获得提及，没人会发邮件通知你。大型 SEO 套件虽然能提供提醒，但它们通常按月订阅收费，而你往往只需要监控少数几个域名。本文将介绍：反向链接数据的来源（以及为什么抓取 SEO 工具页面不是个好主意）、如何将“所有反向链接”转化为“自上次检查以来的变化”，以及实际运行的成本数据。

Where backlink data comes from: You can’t check backlinks by visiting your own site. A backlink lives on someone else’s page, so you need a crawler that has seen a large part of the web and kept the link graph. Building that yourself isn’t realistic, and scraping a vendor’s web UI breaks often and goes against their terms. What’s left is to license an index through an API. We use DataForSEO’s Backlinks API. For each link it returns the linking page, anchor text, dofollow/nofollow, first and last seen dates, and 0–100 ranks for the linking domain and page.

反向链接数据的来源：你无法通过访问自己的网站来检查反向链接。反向链接存在于他人的页面上，因此你需要一个已经抓取过互联网大部分内容并保留了链接图谱的爬虫。自己构建这样的系统并不现实，而抓取供应商的网页界面不仅容易失效，还违反了它们的服务条款。剩下的选择是通过 API 获取索引授权。我们使用的是 DataForSEO 的 Backlinks API。对于每个链接，它会返回来源页面、锚文本、dofollow/nofollow 状态、首次和最后发现日期，以及来源域名和页面的 0-100 排名。

The diff is the product: A full list of a big site’s backlinks is interesting once. What you want every week is: new: links whose first_seen is after your last check, lost: links the crawler found missing since your last check. The core of a DIY version is small: keep a hash of every change you’ve already reported, and only report what’s new.

差异即产品：大型网站的反向链接完整列表看一次就够了。你每周真正需要的是：新增链接（首次发现日期在你上次检查之后）和丢失链接（爬虫在上次检查后发现已消失的链接）。DIY 版本的核心很简单：保存你已经报告过的每个变化的哈希值，只报告新的变化。

Three gotchas we hit while building this: Index lag. A link first seen yesterday may only show up in the index today. If you query strictly “since my last run”, you miss it for good. Query with an overlap window instead (we use 48 hours) and drop changes you already reported, using the hashes above. That way each change shows up exactly once. Big domains are noisy. Large sites gain and lose links all the time, many of them sitewide or scraper links. For big sites, monitor referring domains instead of individual links, and sort the changes by the linking domain’s rank. Filters don’t behave like SQL. On the referring-domains endpoint, an OR filter that includes lost_date silently drops every row where lost_date is null, which means all new domains. We ended up running “new” and “lost” as two separate queries.

我们在构建过程中遇到的三个陷阱：索引延迟。昨天首次发现的链接可能今天才出现在索引中。如果你严格查询“自上次运行以来”的数据，就会永久错过它。建议使用重叠窗口进行查询（我们使用 48 小时），并利用上述哈希值剔除已报告过的变化。这样，每个变化都只会显示一次。大型域名噪音很大。大型网站时刻都在增加和丢失链接，其中许多是全站链接或抓取链接。对于大型网站，建议监控引用域名而非单个链接，并按来源域名的排名对变化进行排序。过滤器不像 SQL 那样工作。在引用域名接口上，包含 lost_date 的 OR 过滤器会静默丢弃所有 lost_date 为空的行，这意味着所有新域名都会被过滤掉。我们最终将“新增”和“丢失”作为两个独立的查询来运行。

What the data looks like: Here’s a real run: the 100 strongest backlinks each for backlinko.com and plausible.io, sorted by the linking domain’s rank. A few things in that output are worth knowing before you build on it: The top 100 links by rank came from only 13 linking domains for backlinko.com and 4 for plausible.io. Strong sites often link many times from many pages. If you want breadth, deduplicate to one link per domain. Only 34 of the 200 links were dofollow. High-rank sites like GitHub mark most outbound links nofollow, so filter on dofollow if that’s what you care about.

数据呈现：以下是一次实际运行的结果：backlinko.com 和 plausible.io 各自最强的 100 个反向链接，按来源域名排名排序。在基于此构建之前，有几点值得注意：按排名计算的前 100 个链接仅来自 backlinko.com 的 13 个域名和 plausible.io 的 4 个域名。强大的网站通常会从多个页面多次链接。如果你追求广度，请按每个域名去重。200 个链接中只有 34 个是 dofollow。像 GitHub 这样高排名的网站会将大多数出站链接标记为 nofollow，因此如果你只关心 dofollow，请务必进行过滤。

The tool we use: Disclosure: we (SteadyScrape) built the tool below and get paid when it’s used. We packaged all of the above as an Apify Actor: Backlink Checker API: New and Lost Link Monitor (steady-scrape/backlink-explorer). It has four modes: backlinks, referring_domains, summary (up to 10,000 domains in one run) and monitor. For the weekly alert, schedule it with an input like this: Each run returns only the links gained or lost since the previous run, each with anchor, dofollow and the linking domain’s rank. A domain with no changes skips the per-link lookup.

我们使用的工具：披露：我们（SteadyScrape）构建了以下工具，并在其被使用时获得报酬。我们将上述所有功能打包为一个 Apify Actor：Backlink Checker API: New and Lost Link Monitor (steady-scrape/backlink-explorer)。它有四种模式：backlinks（反向链接）、referring_domains（引用域名）、summary（单次运行最多 10,000 个域名）和 monitor（监控）。对于每周提醒，请按此输入进行调度：每次运行仅返回自上次运行以来新增或丢失的链接，并包含锚文本、dofollow 状态和来源域名排名。没有变化的域名将跳过单链接查询。

What it costs: Pricing is pay per event, with no subscription. List prices (Apify Free plan): Event Price. Run start (only if at least one domain returns data) $0.03. Domain looked up in backlinks or referring-domains mode (summary row included) $0.05. Backlink row $0.0008 ($0.80 per 1,000). Referring-domain row $0.001. Domain in summary mode $0.004. Domain checked in monitor mode $0.05. New or lost link reported $0.002. Apify also charges its standard $0.00005 start event. Row prices drop on paid Apify plans. Invalid inputs and domains with no data are free. The run above (2 domains × 100 backlinks) works out to $0.03 + 2 × $0.05 + 200 × $0.0008 = $0.29 at list price, and it took about 6 seconds.

成本说明：按事件付费，无需订阅。标价（Apify 免费计划）：运行启动（仅当至少有一个域名返回数据时）$0.03；在反向链接或引用域名模式下查询域名（包含摘要行）$0.05；反向链接行 $0.0008（每 1,000 行 $0.80）；引用域名行 $0.001；摘要模式下的域名 $0.004；监控模式下检查的域名 $0.05；报告新增或丢失的链接 $0.002。Apify 还会收取标准的 $0.00005 启动事件费用。付费 Apify 计划的行价格会降低。无效输入和无数据的域名免费。上述运行（2 个域名 × 100 个反向链接）按标价计算为 $0.29，耗时约 6 秒。