Comparing prices across retailers is a unit-normalization problem, not a scraping problem

查看原文 / View Original 编辑 / Edit

Comparing prices across retailers is a unit-normalization problem, not a scraping problem

跨零售商比价是一个单位标准化问题，而非抓取问题

Disclosure: I’m the founder of Popgot, which I use as the example below. The problem and the approach apply regardless of what you build on. Every price comparison project I’ve seen starts the same way: scrape a bunch of retailers, store the prices, sort ascending. And then it produces garbage rankings, because price is not a comparable field. 披露：我是 Popgot 的创始人，下文将以它为例。无论你基于什么平台构建，这个问题和解决方法都适用。我见过的每一个比价项目起步方式都一样：抓取一堆零售商的数据，存储价格，然后升序排列。结果却产生了一堆垃圾排名，因为“价格”本身并不是一个可直接比较的字段。

Here’s the classic failure. Three listings for AA batteries: 这是一个典型的失败案例。以下是三款 AA 电池的商品列表：

ListingPriceCount
Brand A$5.9916
Brand B$6.9920
Brand C$11.9440

Sort by price and Brand A “wins” at $5.99. Sort by cost per battery and the order flips completely: Brand C is ~29.9c per cell, Brand A is ~37.4c. The cheapest listing is the worst deal on the page. 按价格排序，品牌 A 以 $5.99 “胜出”。但如果按单节电池成本排序，顺序会完全颠倒：品牌 C 约为每节 29.9 美分，而品牌 A 为 37.4 美分。最便宜的列表反而是页面上最不划算的交易。

Why this is hard

为什么这很难

The naive fix is “just divide price by quantity.” The problem is that quantity almost never exists as a clean number. It’s buried in the title, and the title is written by whoever uploaded the listing: 天真的解决方法是“直接用价格除以数量”。问题在于，数量几乎从来不是一个规整的数字。它隐藏在标题中，而标题是由上传商品的人随意编写的：

  • AA Batteries 24 Pack
  • AA Alkaline Batteries, 1.5 Volts, 24 Count
  • 48-Pack (2 x 24) Double A

So you end up writing a title parser. Then you discover the same product needs a different unit depending on the category: per fluid ounce for detergent, per serving for protein powder, per 100g for coffee, per sheet for paper towels. 因此，你最终不得不编写一个标题解析器。接着你会发现，同一产品根据类别需要不同的单位：洗涤剂按液量盎司计算，蛋白粉按份量计算，咖啡按 100 克计算，纸巾按张数计算。

Then you discover that some categories need a spec filter before unit price is even meaningful. A fish oil at 20c per serving isn’t cheaper than one at 34c per serving if the first one has half the EPA+DHA. You’re comparing two different products. That last part is the piece people underestimate. Normalization is only valid within a set of products that actually satisfy the same requirement, which means something has to read the label, not just the title. 然后你还会发现，有些类别在计算单价前需要先进行规格过滤。如果一款鱼油每份 20 美分，另一款每份 34 美分，但前者的 EPA+DHA 含量只有后者的一半，那么它并不算更便宜。你比较的是两种不同的产品。最后这一点往往被人们低估。标准化只有在真正满足相同需求的产品集合内才有效，这意味着系统必须读取标签信息，而不仅仅是标题。

What a normalized record looks like

标准化后的记录长什么样

This is the problem I ended up building Popgot around, so rather than describe it abstractly, here’s the shape of the data. The developer API returns listings with the unit math already done: 这就是我构建 Popgot 要解决的问题，所以与其抽象地描述，不如直接看数据结构。开发者 API 返回的列表已经完成了单位换算：

GET /api/developer-api/products?query=aa+batteries&limit=10
{
  "products": [
    {
      "display_title": "ACDelco 40-Count AA Batteries",
      "source_type": "amazon",
      "price_cents": 1194,
      "unit_count": 40,
      "price_cents_per_unit": 29.85,
      "rating_average": 4.7,
      "review_count": 54696,
      "value_score": 27.413,
      "rank": 5
    }
  ]
}

The fields that matter for this problem are unit_count and price_cents_per_unit. source_type tells you which retailer the listing came from, so cross-retailer comparison is a single sort instead of a reconciliation job. 对于这个问题，关键字段是 unit_count 和 price_cents_per_unit。source_type 告诉你列表来自哪个零售商，因此跨零售商比价只需一次排序，而无需进行繁琐的对账工作。

A minimal client:

一个极简的客户端示例：

const res = await fetch(
  "https://popgot.com/api/developer-api/products?" + 
  new URLSearchParams({ query: "aa batteries", limit: 20 })
);
const { products } = await res.json();

const byUnitPrice = products
  .filter((p) => p.unit_count > 0)
  .sort((a, b) => a.price_cents_per_unit - b.price_cents_per_unit);

for (const p of byUnitPrice.slice(0, 5)) {
  console.log(
    `${(p.price_cents_per_unit / 100).toFixed(3)}/unit`,
    `${p.source_type.padEnd(8)}`,
    p.display_title
  );
}

Note the unit_count > 0 guard. Any dataset like this will have listings where the count couldn’t be resolved, and you want those excluded from a unit-price sort rather than silently ranked at zero. 注意 unit_count > 0 的保护性判断。任何此类数据集都会包含无法解析数量的列表，你应当将它们从单价排序中排除，而不是让它们被静默地排在零位。

Things worth knowing before you build on it

在此基础上构建前值得了解的事项

I’d rather you hit these in a blog post than in production, so here are the sharp edges — including the ones in my own API. 我宁愿你在博客文章中看到这些坑，而不是在生产环境中踩到它们，所以这里列出了一些“尖锐的边缘”——包括我自己的 API 中存在的问题。

  • value_score is opinionated. It blends unit price with rating signals, so it is not the same as “cheapest.” In the sample above the top-ranked-by-value item is not the lowest price_cents_per_unit. If your product promises “cheapest,” sort on the raw unit price yourself and ignore rank.

  • value_score 是主观的。 它将单价与评分信号混合在一起，因此它不等同于“最便宜”。在上面的示例中，按价值排名第一的商品并不是 price_cents_per_unit 最低的。如果你的产品承诺“最便宜”，请自行按原始单价排序，忽略排名。

  • Cache aggressively, but treat cached prices as hints. Prices move, and the retailer’s price at checkout is the one that actually applies. Never present a stored price as a guarantee.

  • 积极缓存，但将缓存价格视为参考。 价格会变动，结账时的零售商价格才是最终生效的价格。永远不要将存储的价格作为保证。

  • Units are category-specific. Don’t build UI copy that hardcodes “per item.” Render whatever unit the category actually uses, or your detergent page will say “$0.06 per item” and mean nothing.

  • 单位是特定于类别的。 不要编写硬编码为“每件”的 UI 文案。渲染该类别实际使用的单位，否则你的洗涤剂页面显示“每件 $0.06”将毫无意义。

  • Spec filters belong upstream. If a user needs “at least 1000mg EPA+DHA,” express that in the query rather than post-filtering on the title string. Title-based filtering will drop valid products and keep invalid ones.

  • 规格过滤应在前端处理。 如果用户需要“至少 1000mg EPA+DHA”，请在查询中表达出来，而不是在标题字符串上进行后置过滤。基于标题的过滤会丢弃有效产品并保留无效产品。

The takeaway

总结

If you’re building anything that ranks products — a deals site, a budgeting tool, an internal procurement dashboard — the interesting engineering isn’t collecting prices. It’s deciding what the denominator is, and making sure the things you’re dividing are genuinely substitutable. Get that wrong and you ship a sorted list that confidently recommends the worst option. 如果你正在构建任何涉及产品排名的系统——比如优惠网站、预算工具或内部采购仪表板——核心工程挑战不在于收集价格，而在于确定分母是什么，并确保你进行除法运算的对象是真正可替代的。如果这一点做错了，你发布出来的排序列表只会自信地推荐最差的选项。

If you want to eyeball the output before writing any code, the search side of the same engine is at popgot.com — useful for sanity-checking your own unit math against ours, and for finding the cases where we get it wrong. 如果你想在写代码前先看看输出效果，该引擎的搜索端位于 popgot.com——这对于对照我们的单位换算逻辑来验证你自己的逻辑，以及找出我们出错的情况非常有用。

How are you handling this? I’m especially curious whether anyone has found a clean way to normalize multi-pack listings (2 x 24) without a pile of regex. 你是如何处理这个问题的？我特别好奇是否有人找到了一种无需大量正则表达式就能标准化多包装列表（如 2 x 24）的简洁方法。

← 返回新闻列表 / Back to news list