Extracting a Bibliography Into Structured Citation Records

查看原文 / View Original 编辑 / Edit

Extracting a Bibliography Into Structured Citation Records

将参考文献提取为结构化引用记录

The instinct is to hand the whole reference list to a model and ask for an array of citation objects. On a list of eighty entries that produces seventy-three, with two merged and five hallucinated into tidiness. The fix is to make segmentation a separate, deterministic step. 人们的直觉通常是把整个参考文献列表交给模型，并要求其输出一个引用对象数组。但对于一个包含 80 条条目的列表，这种做法往往只能得到 73 条结果，其中还伴随着两条合并错误和五条因模型“幻觉”而产生的虚假条目。解决办法是将分段（segmentation）作为一个独立的、确定性的步骤来处理。

Two stages, and why the first one is harder. Parsing one reference string into author, year, title and venue is a task current models do well. Deciding where one reference ends and the next begins is a task they do badly, because the boundary is typographic rather than semantic: a hanging indent, a numeric label, a line break that is either a wrap or a separator depending on the column width. 分为两个阶段，以及为什么第一阶段更难。将一个引用字符串解析为作者、年份、标题和出版来源，是当前模型擅长的任务。但判断一个引用在哪里结束、下一个在哪里开始，却是它们的弱项，因为这种边界是排版上的而非语义上的：悬挂缩进、数字标签，或者取决于列宽的换行符（可能是自动换行，也可能是分隔符）。

Splitting the work also gives you a count to assert against. If the list is numbered 1 to 84 and you segmented 81 entries, you know the parse is wrong before you have looked at a single field. A single-call extraction gives you no such handle — a merged pair looks identical to a list that was three shorter. 将工作拆分还能提供一个用于校验的计数。如果列表编号从 1 到 84，而你只分出了 81 个条目，那么在查看任何具体字段之前，你就已经知道解析出错了。而“一次性提取”则无法提供这种控制手段——合并后的条目看起来与缺失了三条的列表完全一样。

Step 1: segment the list. Three reference-list styles cover almost everything, and each has a different boundary signal: 第一步：分段列表。三种参考文献格式涵盖了绝大多数情况，每种格式都有不同的边界信号：

  • Numbered (Vancouver, IEEE). Each entry begins with 1. or [1]. Boundary detection is a regex, the sequence is monotonic, and you get the assertion for free. 数字编号式（Vancouver, IEEE）。 每个条目以“1.”或“[1]”开头。边界检测可以使用正则表达式，序列是单调递增的，你可以免费获得校验依据。
  • Author-date (APA, Harvard, Chicago author-date). No labels. Entries are separated by a hanging indent — the first line starts at the margin and continuations are indented — which is invisible in a flat text stream and obvious in the layout. 作者-日期式（APA, Harvard, Chicago author-date）。 没有标签。条目通过悬挂缩进分隔——第一行从页边距开始，后续行缩进——这在纯文本流中不可见，但在排版布局中非常明显。
  • Note-bibliography (Chicago notes). Also unlabelled, also hanging-indented, and additionally uses a three-em dash for a repeated first author, which is the case discussed below. 注释-参考文献式（Chicago notes）。 同样没有标签，同样使用悬挂缩进，此外还会对重复出现的同一作者使用三字线（three-em dash），这种情况将在下文讨论。

For the unlabelled styles, segment on the indent rather than on the text. If you have coordinates from the PDF, an entry starts at every line whose left edge is at the block minimum and continues through every line indented further. If you do not have coordinates, a reasonable proxy is a line that begins with a capital letter followed by a comma-and-initial pattern, but it is a proxy and it will miss entries beginning with an institutional author or a title. 对于无标签格式，应根据缩进而非文本内容进行分段。如果你有 PDF 的坐标信息，那么每个左边缘处于块最小值的行都是一个条目的开始，并持续到所有缩进更深的行。如果你没有坐标，一个合理的替代方案是寻找以大写字母开头，后跟“逗号+首字母”模式的行，但这只是一个近似值，会漏掉以机构作者或标题开头的条目。

function segmentNumbered(text) {
  // Split before a line-initial "1." / "[1]" / "1)"
  const parts = text.split(/\n(?=\s*(?:\[\d+\]|\d+[.)])\s)/);
  return parts.map((p) => p.trim()).filter(Boolean);
}

function segmentByIndent(lines) {
  // lines: [{ text, x }] from the PDF text layer, one entry per visual line.
  const margin = Math.min(...lines.map((l) => l.x));
  const out = [];
  for (const line of lines) {
    if (Math.abs(line.x - margin) < 1.5) out.push(line.text);
    else out[out.length - 1] += " " + line.text;
  }
  return out;
}

The tolerance of 1.5 points matters. Text-layer x-coordinates are not exact — kerning and the width of an opening quotation mark shift the reported origin — and a strict equality test puts every reference beginning with a quotation mark into the previous entry. 1.5 点的容差非常重要。文本层的 X 坐标并不精确——字距调整和开引号的宽度会改变报告的起始位置——如果使用严格相等测试，会将所有以引号开头的引用错误地归入前一个条目中。

The conventions that destroy authorship. Two style rules silently delete the field you most want, and neither is recoverable from the entry in isolation. 破坏作者信息的惯例。有两种格式规则会悄无声息地删除你最需要的字段，且两者都无法通过孤立的条目恢复。

  • The repeated-author dash. In Chicago-style bibliographies, consecutive works by the same author replace the name with a three-em dash: ———. 2019. A parser that reads each entry independently records the author as a dash, or as empty, for every entry after the first. The entry is only parseable in the context of its predecessor, which means segmentation order is load-bearing and you cannot parallelise the parse across a shuffled list. Carry the previous entry’s author forward, and note that the dash can appear in several widths — em dash repeated three times, a single three-em dash character, or a run of hyphens in a plain-text rendering. 重复作者破折号。 在芝加哥格式的参考文献中，同一作者的连续作品会用三字线（———）代替名字。如果解析器独立读取每个条目，那么第一条之后的所有条目，作者字段都会被记录为破折号或空值。该条目只有在上下文关联中才能被解析，这意味着分段顺序至关重要，你无法对打乱顺序的列表进行并行解析。你需要将前一条目的作者信息传递下去，并注意破折号可能有多种宽度——重复三次的 em dash、单个三字线字符，或纯文本渲染中的连字符序列。

  • Et al. truncation. Most styles abbreviate long author lists after a threshold. The information is gone from the page; no amount of prompting recovers it. The right response is to record what is there and mark the list as truncated, so that a downstream match on “same author list” does not fail against the full record from a lookup. “等”（Et al.）截断。 大多数格式在作者列表超过一定阈值后会进行缩写。这些信息在页面上已经丢失，无论如何提示模型都无法恢复。正确的做法是记录现有内容并标记列表为“已截断”，这样下游在进行“相同作者列表”匹配时，就不会因为与查找出的完整记录不符而失败。

Two further traps are worth pre-empting. Page ranges use an en dash and often an elided upper bound — 1123–31 means 1123 to 1131, not 1123 to 31 — so expansion is a rule, not a parse. And a trailing period is part of the sentence, not part of the DOI, which is the single most common way a greedy identifier regex captures a character that makes the DOI unresolvable. 还有两个陷阱值得预先防范。页码范围使用 en dash 且通常会省略上限（例如 1123–31 表示 1123 到 1131，而非 1123 到 31），因此扩展页码是一个规则问题，而非解析问题。此外，末尾的句号属于句子而非 DOI 的一部分，这是贪婪匹配的标识符正则表达式最常捕获到的字符，会导致 DOI 无法解析。

Step 2: parse each entry. With entries isolated, per-entry parsing is a constrained structured output task. Use a strict schema so the model cannot invent a field, and give it an explicit entry_type enum — a book chapter, a conference paper and a preprint have genuinely different fields, and forcing all three into a journal-article shape loses the container title. 第二步：解析每个条目。在条目被隔离后，单条解析就变成了一个受限的结构化输出任务。使用严格的 Schema 以防止模型凭空捏造字段，并提供明确的 entry_type 枚举——书的章节、会议论文和预印本确实拥有不同的字段，强行将三者塞入期刊文章的格式会导致丢失容器标题（如书名或会议名）。

const CITATION_SCHEMA = {
  type: "object",
  additionalProperties: false,
  required: ["entry_type", "authors", "title", "raw"],
  properties: {
    entry_type: {
      enum: ["journal_article", "book", "book_chapter", "conference_paper", "preprint", "thesis", "report", "webpage", "other"],
    },
    authors: {
      type: "array",
      items: {
        type: "object",
        additionalProperties: false,
        required: ["family"],
        properties: {
          family: { type: "string" },
          given: { type: "string" }
        },
      },
    },
    authors_truncated: { type: "boolean" },
    title: { type: "string" },
    container_title: { type: "string" }, // journal, book or proceedings
    year: { type: "integer" },
    volume: { type: "string" },
    issue: { type: "string" },
    pages: { type: "string" },
    doi: { type: "string" },
    raw: { type: "string" }, // the entry exactly as segmented
  },
};

Keeping raw is not optional. It is what makes every later disagreement resolvable without going back to the PDF, and it is what you diff against when you change the prompt. 保留 raw 字段是必须的。它使得后续出现的任何分歧都无需回溯 PDF 即可解决，也是你在修改提示词（prompt）时进行差异对比的依据。

← 返回新闻列表 / Back to news list