Parsing Expression Grammar vs. regexes: Building Org parser in Lisp that exports to HTML (via SXML)

解析表达式语法 (PEG) 与正则表达式：用 Lisp 构建 Org 解析器并导出为 HTML (通过 SXML)

Hi everyone. In this blog post I want to take you in an adventure of parsing Org mode with Parsing Expression Grammars (PEG) in Guile Scheme (ice-9 peg) and converting to HTML (via SXML): OrgWebAlchemy. 大家好。在这篇博文中，我想带大家踏上一段冒险之旅，使用 Guile Scheme 中的解析表达式语法 (PEG) (ice-9 peg) 来解析 Org mode，并将其转换为 HTML (通过 SXML)：这就是 OrgWebAlchemy。

I wanted to share something with you all that I’ve been working on for a while. It all started with some naive regular expressions to parse Org mode content, but I pretty quickly realized I needed something smarter than that to get to where I want to. 我想和大家分享一些我研究了一段时间的项目。最初，我只是用一些简单的正则表达式来解析 Org mode 内容，但我很快意识到，要达到我的目标，我需要比这更智能的方法。

It’s taken a while but I am finally more knowledgeable of what Parsing Expression Grammars can do, thanks to GNU’s great (ice-9 peg) module and tutorials. I thought it might be interesting to people here who enjoy Lisp, Scheme, parsing, Org mode, or the general idea of meta-meta-meta-programming as I like to call it. 虽然花了一些时间，但多亏了 GNU 出色的 (ice-9 peg) 模块和教程，我终于对解析表达式语法的功能有了更深入的了解。我想，对于那些喜欢 Lisp、Scheme、解析、Org mode，或者喜欢我所称的“元-元-元编程”概念的人来说，这可能会很有趣。

Disclosure, AI has helped me get a grip of PEG and debug some things, but development of OrgWebAlchemy is “my own spaghetti” and the unit tests and manual verification (and lots of pretty printing the AST) has guided me towards quite a nice implementation (if I may say so myself). 披露一下，人工智能确实帮助我掌握了 PEG 并调试了一些问题，但 OrgWebAlchemy 的开发完全是我自己的“意大利面条式代码”，单元测试、手动验证（以及大量的 AST 格式化打印）引导我完成了一个相当不错的实现（如果我可以这么说的话）。

Project’s source code @ Codeberg: https://codeberg.org/jjba23/orgwebalchemy 项目源代码 @ Codeberg: https://codeberg.org/jjba23/orgwebalchemy

OrgWebAlchemy is a Guile Scheme library for parsing Org-mode documents into an AST and rendering them to HTML. My main use-case is to export Org to HTML without needing Emacs, and to integrate this feature into some projects of mine, allowing me to write Org mode and have it pretty rendered. OrgWebAlchemy 是一个 Guile Scheme 库，用于将 Org-mode 文档解析为 AST 并将其渲染为 HTML。我的主要用例是在不需要 Emacs 的情况下将 Org 导出为 HTML，并将此功能集成到我的一些项目中，从而让我能够编写 Org mode 并将其美观地渲染出来。

The basic idea is pretty simple: 基本思路非常简单：

(use-modules (orgwebalchemy html))
(org->html "This is ~test~ code.")

becomes something like: 会变成类似这样的内容：

This is <code>test</code> code.

But the interesting part is what happens in between. 但有趣的部分在于中间发生了什么。

Org document -> Parsing Expression Grammar -> AST -> SXML -> HTML Org 文档 -> 解析表达式语法 -> AST -> SXML -> HTML

See here an example showing how OrgWebAlchemy enables the LucidPlan project to render pretty Org mode to HTML. 点击此处查看一个示例，展示了 OrgWebAlchemy 如何使 LucidPlan 项目能够将美观的 Org mode 渲染为 HTML。

PEG vs. a mountain of regexes?

PEG 对抗堆积如山的正则表达式？

Org-mode looks simple until you actually try to parse it. Headings are easy. A paragraph is easy. A list is easy (wait actually no, this has made me sweat). And then suddenly you have: nested lists, ordered, unordered and description lists, different indentation levels, inline markup, links containing descriptions, source blocks, example blocks, quote blocks, tables, escaping constructs which must stop consuming input at exactly the right place. Org-mode 看起来很简单，直到你真正尝试去解析它。标题很容易，段落很容易，列表也很容易（等等，其实不然，这让我出了一身冷汗）。然后你突然发现：嵌套列表、有序/无序/描述列表、不同的缩进级别、行内标记、包含描述的链接、源代码块、示例块、引用块、表格，以及必须在恰当位置停止消耗输入的转义结构。

At this point, the usual approach of adding another regular expression starts to become somewhat… adventurous. :-) You end up with things like: match this, unless that follows it, except inside this block, unless it is a description, but don’t consume the newline, unless the previous line was a list item… That is not really describing a language anymore. It is describing the history of your parser’s bugs. 到了这一步，通常采用的“再加一个正则表达式”的方法就开始变得有些……冒险了。:-) 你最终会得到类似这样的逻辑：匹配这个，除非后面跟着那个，除非在某个块内，除非它是一个描述，但不要消耗换行符，除非上一行是一个列表项……这已经不再是在描述一种语言了，而是在描述你解析器 Bug 的历史。

So OrgWebAlchemy uses Parsing Expression Grammars (PEGs) through Guile’s excellent (ice-9 peg) module. 因此，OrgWebAlchemy 通过 Guile 出色的 (ice-9 peg) 模块使用了解析表达式语法 (PEG)。

(define-peg-pattern element body
  (or empty-line heading separator table src-block quote-block
      example-block export-html-block description-list
      unordered-list ordered-list paragraph))

This is rather nice because the grammar itself starts looking like documentation for the language. And Guile lets us express PEGs directly as S-expressions (alternatively you can also use the more traditional syntax if you don’t like it), which makes the Lisper in me very happy. 这非常棒，因为语法本身看起来就像是该语言的文档。而且 Guile 允许我们直接用 S-表达式来表达 PEG（如果你不喜欢，也可以使用更传统的语法），这让身为 Lisp 爱好者的我感到非常高兴。

One thing I particularly like about this approach is that we have loose coupling and the detail of generating SXML and then rendering HTML is a “presentation concern”. This opens possibilities to later exporting to Markdown or other formats. 我特别喜欢这种方法的一点是，我们实现了松耦合，生成 SXML 然后渲染 HTML 的细节属于“表现层关注点”。这为以后导出为 Markdown 或其他格式提供了可能。

I’m still busy with the exact representation and getting it all right. But as of now v1.0 has some stability. 我仍在忙于完善具体的表示方式并确保一切正确。但目前 v1.0 版本已经具备了一定的稳定性。

YAY recursive lists

耶，递归列表

One of the fun parts has been getting nested Org lists right. Something like: 有趣的部分之一是正确处理嵌套的 Org 列表。像这样：

• Item 1
  • Item 1.1
  • Item 1.2
• Item 2

should become a quasi-tree. The parser initially produces the flat sequence of list items, and the AST processing phase turns indentation into nested structure. 应该变成一个准树结构。解析器最初生成列表项的扁平序列，而 AST 处理阶段将缩进转换为嵌套结构。

The HTML renderer can then naturally produce: HTML 渲染器随后可以自然地生成：

<ul>
  <li>Item 1
    <ul>
      <li>Item 1.1</li>
      <li>Item 1.2</li>
    </ul>
  </li>
  <li>Item 2</li>
</ul>

I do still have a small issue here, and that is about the mixing of different list types in nested way. Hopefully it’s a subtle bug to fix. The HTML side uses SXML, because if we’re already writing Lisp, we might as well represent our HTML as Lisp data too. :-) That really helps a lot and makes building the markup tree so much nicer. 我在这里还有一个小问题，那就是在嵌套方式中混合不同列表类型的情况。希望这是一个容易修复的细微 Bug。HTML 端使用了 SXML，因为既然我们已经在写 Lisp 了，不妨也将 HTML 表示为 Lisp 数据。:-) 这真的很有帮助，让构建标记树变得更加顺畅。