Perspec 1.0: A Haskell desktop app for perspective correction of document photos

Perspec 1.0：一款用于文档照片透视校正的 Haskell 桌面应用

I’m very excited to announce the 1.0 release of Perspec! Perspec is a desktop app for correcting the perspective of images. This is primarily useful for photos of documents and receipts, but it can be used for any kind of image. This has finally become the app I envisioned when I started working on the project 9 years ago. I didn’t think it would take me this long to get here, but I’m very happy with the result and I hope you’ll like it too!

我非常激动地宣布 Perspec 1.0 版本正式发布！Perspec 是一款用于校正图像透视的桌面应用。它主要用于处理文档和收据的照片，但也可以用于任何类型的图像。这终于成为了我 9 年前刚开始这个项目时所设想的那款应用。我没想到会花这么长时间才走到这一步，但我对结果非常满意，希望你们也会喜欢它！

Initial Motivation

最初的动机

You’re probably familiar with the scanner apps available for mobile phones, like Adobe Scan, vFlat, SwiftScan, … and numerous others. Scanning functionality is also integrated into Dropbox, and these days even natively into iOS itself. However, I don’t like working on my phone and I’d rather just take photos of the documents and receipts and deal with cleaning them up and organizing them on my computer another day. There, I have a big screen, a keyboard, and a precise mouse, which makes editing faster and more accurate.

你可能对手机上的扫描应用很熟悉，比如 Adobe Scan、vFlat、SwiftScan 等等。扫描功能也被集成到了 Dropbox 中，如今甚至在 iOS 系统中也已原生支持。然而，我不喜欢在手机上操作，我更倾向于直接拍下文档和收据，改天再在电脑上进行清理和整理。在电脑上，我有大屏幕、键盘和精准的鼠标，这使得编辑过程更快、更准确。

Also, the mobile apps make some annoying technical decisions in the name of giving users something they’re familiar with. For example: If you store a document as a grayscale PNG, you can get small file sizes without introducing any compression artifacts. However, all the popular apps will give you a grayscale JPEG image with a much bigger file size and worse image quality, just because JPEG is what people are familiar with. Or maybe I’m giving them too much credit and they actually don’t know that PNGs can be smaller than JPEGs if the image contains large areas of uniform color, whereas for normal photos, JPEGs are smaller than PNGs. And no, converting it to PNG afterwards is not an option, as by then the image already contains all the JPEG compression artifacts.

此外，移动应用为了让用户感到“熟悉”，往往会做出一些令人恼火的技术决策。例如：如果你将文档存储为灰度 PNG，可以在不引入任何压缩伪影的情况下获得较小的文件大小。然而，所有主流应用都会给你一个文件更大、画质更差的灰度 JPEG 图像，仅仅因为 JPEG 是人们所熟悉的格式。或者也许是我高估了他们，他们实际上并不知道如果图像包含大面积的纯色区域，PNG 可以比 JPEG 更小（尽管对于普通照片，JPEG 通常比 PNG 小）。而且，事后再将其转换为 PNG 是行不通的，因为那时图像已经包含了所有的 JPEG 压缩伪影。

For example, let’s compare the results of scanning the following document: The other apps produce bigger files, and you can clearly see the compression artifacts that degrade the result.

例如，让我们比较一下扫描同一份文档的结果：其他应用生成的文件更大，并且你可以清楚地看到降低了画质的压缩伪影。

App	Result	Preview	Notes
Perspec	~110 kB, PNG	View result
Scanner Pro	~190 kB, JPEG	View result	Extracted JPEG from exported PDF
iOS	~300 kB, JPEG	View result	Extracted JPEG from exported PDF

Another thing that annoys me more than it should is the ridiculous detection previews that seemingly every app includes these days: While you’re taking a photo, the app shows you a live overlay of where it detects the document. This, however, doesn’t help you at all. Just because it can detect the document correctly in the preview video feed doesn’t mean it will detect it correctly in the final photo. Due to the higher resolution, different lighting (exposure times, flash, …), and different contrast, the detection will often be quite different in the final photo. So all the preview is telling you is that there is indeed a document in front of your camera, which you already know since you placed it there. 🤦‍♂️

另一件让我感到格外恼火的事情是，如今似乎每个应用都包含那种荒谬的检测预览：当你拍照时，应用会实时显示它检测到的文档位置。然而，这根本没有任何帮助。仅仅因为它能在预览视频流中正确检测到文档，并不意味着它能在最终照片中正确检测到。由于更高的分辨率、不同的光照（曝光时间、闪光灯等）以及不同的对比度，最终照片中的检测结果往往会有很大差异。所以，预览唯一告诉你的就是你的相机前确实有一个文档，而这一点你早就知道，因为是你把它放在那里的。🤦‍♂️

Lastly, and most importantly, I knew I could build a better document detection algorithm for the kind of photos I was taking. The detection in existing apps would often be slightly off, even if you had a good picture with good contrast between the document and the background. Most apps use some kind of edge detection step in their pipeline, as Dropbox explains here. But I knew that documents and receipts often don’t have straight edges but rather wrinkled or curved ones. When you try to match even just a slight curve with a straight line, the endpoints will be quite far off. So instead, the app should try to detect the corners and build up the document from there. There is a detailed explanation of the computer vision techniques later in the post.

最后，也是最重要的一点，我知道我可以为我拍摄的那类照片构建更好的文档检测算法。现有应用中的检测往往会有轻微偏差，即使你拍的照片很好，文档和背景之间的对比度也很高。大多数应用在流程中都会使用某种边缘检测步骤，正如 Dropbox 在此所解释的那样。但我知道，文档和收据往往没有笔直的边缘，而是褶皱或弯曲的。当你试图用直线去匹配哪怕是一点点曲线时，端点也会产生很大的偏差。因此，应用应该尝试检测角点并从那里构建文档。本文稍后会对计算机视觉技术进行详细解释。

The Long Road to 1.0

通往 1.0 的漫漫长路

I was still a student when I started working on Perspec and had to scan a lot of stuff for my studies, so I had plenty of motivation to build something like this. Sure, you could also fix the perspective with Photoshop, Affinity Photo, or GIMP. But the overhead is substantial: Open each photo, find the perspective tool, drag the corners, pick the right export settings, repeat for the next photo, and so on. These tools are built to do everything with any image and not to churn through 50 receipts as quickly as possible. I wanted an app that’s focused on this one task, with a workflow that’s as streamlined as possible.

当我开始开发 Perspec 时，我还是个学生，需要为学业扫描大量资料，所以我有很多动力去构建这样的工具。当然，你也可以用 Photoshop、Affinity Photo 或 GIMP 来校正透视。但开销太大了：打开每张照片、找到透视工具、拖动角点、选择正确的导出设置，然后为下一张照片重复上述步骤，等等。这些工具旨在处理任何图像的各种任务，而不是为了尽可能快地处理 50 张收据。我想要的是一款专注于这一项任务的应用，并拥有尽可能精简的工作流。

My first iteration was a fully automatic CLI app called Perspectra, implemented with Python and scikit-image. You’d pass your image and it would try to detect and extract the document for you. Simple as that. Although I actually liked scikit-image — feature-rich, yet more straightforward than OpenCV — I quickly realized that I absolutely do not like Python. But more importantly, I realized that I also needed a GUI to fix incorrectly detected document boundaries, as the fully automatic CV pipeline would never get all documents 100% right. And how do you build a desktop app with a GUI? Obviously with Haskell. 😝

我的第一个版本是一个名为 Perspectra 的全自动命令行应用，使用 Python 和 scikit-image 实现。你只需传入图像，它就会尝试为你检测并提取文档。就这么简单。虽然我确实喜欢 scikit-image（功能丰富，且比 OpenCV 更直观），但我很快意识到我绝对不喜欢 Python。但更重要的是，我意识到我还需要一个图形界面（GUI）来修正检测错误的文档边界，因为全自动的计算机视觉流程永远无法 100% 正确处理所有文档。那么，如何构建一个带有 GUI 的桌面应用呢？显然是用 Haskell。😝

Joking aside, I had recently started learning Haskell and was absolutely in love with it. So naturally, I wanted to see if it could be used for building the desktop app. As I didn’t want to use Python any longer, my next instinct was to use ImageMagick for the computer vision and image manipulation tasks, as I had some experience with its features and capabilities. The existing Haskell bindings were rather lacking, so I opted to simply call magick as an external process. While this mostly worked, it was always a pain to get it installed and linked correctly across platforms, and the performance was surprisingly bad for larger images. Another obvious choice would have been OpenCV, but I had some bad memories of using it at university (maybe it was just the C++ context …), and the Haskell bindings looked rather painful. So, my next experiment was using the native Haskell image processing library Hip. With the help of its author @lehins himself and @HanStolpo, we were able to make it work at ZuriHac! (Thanks again!) However, it was still missing some features that I wanted, like binarization with Otsu.

玩笑归玩笑，我当时刚开始学习 Haskell，并且完全爱上了它。所以自然地，我想看看它是否能用于构建桌面应用。由于我不想再使用 Python，我的下一个直觉是使用 ImageMagick 来处理计算机视觉和图像操作任务，因为我对它的功能和特性有一些经验。现有的 Haskell 绑定相当匮乏，所以我选择直接将 magick 作为外部进程调用。虽然这在大多数情况下有效，但在不同平台上安装和正确链接它总是很痛苦，而且对于较大的图像，性能也出奇地差。另一个显而易见的选择是 OpenCV，但我对在大学时使用它的经历有一些糟糕的记忆（也许只是因为 C++ 环境……），而且 Haskell 绑定看起来相当痛苦。因此，我的下一个实验是使用原生的 Haskell 图像处理库 Hip。在作者 @lehins 本人和 @HanStolpo 的帮助下，我们在 ZuriHac 上成功让它运行起来了！（再次感谢！）然而，它仍然缺少一些我想要的功能，比如 Otsu 二值化。