Jigsaw Jeeves: Building a Puzzle Assistant using Computer Vision

Jigsaw Jeeves：利用计算机视觉构建拼图助手

Conceptual overview and walkthrough of a solution approach in Python Python 解决方案的概念概述与实践指南

Chinmay Kakatkar | Aug 19, 2026 | 25 min read Chinmay Kakatkar | 2026年8月19日 | 25分钟阅读

Picture this: you are at home on a quiet Sunday afternoon, with a 5,000-piece jigsaw puzzle of the English countryside spread across the living room floor. Rolling hills, hedgerows, a gray overcast sky dissolving into the horizon. After what seems like ages, the border is done. Now you are staring at a pile of roughly 4,800 pieces, most of them some shade of green or gray, and every piece you try turns out to be the wrong one.

想象一下：在一个宁静的周日下午，你正待在家里，客厅地板上铺着一幅5000片的英国乡村拼图。连绵的丘陵、树篱，还有那融入地平线的灰蒙蒙的天空。在经历了漫长的时光后，边框终于拼好了。现在，你正盯着剩下大约4800片拼图，它们大多是绿色或灰色的不同色调，而你尝试的每一片似乎都不对。

The same brain that can recognize a friend’s face from fifty feet away is completely stumped by 37 nearly identical shades of grass. At some point, you begin doubting your process, then your eyesight, then your life choices — and seriously consider sweeping the whole thing off the table, swearing off jigsaw puzzles entirely.

那个能在五十英尺外认出朋友脸庞的大脑，此刻却被37种几乎一模一样的草地色调难住了。在某个瞬间，你开始怀疑自己的方法，接着怀疑自己的视力，甚至怀疑自己的人生选择——并认真考虑把整张桌子扫空，发誓再也不碰拼图了。

The puzzle itself is not the issue. After all, you sat down to solve the puzzle because you enjoy the intellectual challenge. The frustration creeps in the moment the puzzle goes from being something that is pleasurably challenging to something that feels intractable and progress grinds to a halt. That is when a well-placed nudge in the right direction can bring back the joy.

拼图本身不是问题所在。毕竟，你坐下来拼图是因为你享受这种智力挑战。当拼图从一种令人愉悦的挑战变成一种无法解决的困境，进度陷入停滞时，挫败感便油然而生。这时，一个恰到好处的指引就能让你重拾乐趣。

If you are solving with someone more experienced, they can give you that nudge by pointing you toward a promising cluster, telling you which pieces share a color family, and helping you break the original 5,000-piece problem into a series of smaller, manageable ones. But when you are on your own, AI can help.

如果你和更有经验的人一起拼，他们可以通过指向一个有希望的区域、告诉你哪些拼图属于同一色系，并帮你把最初的5000片难题拆解成一系列更小、更易处理的任务来为你提供指引。但当你独自一人时，人工智能可以提供帮助。

The aim is to have a sort of “Jeeves for jigsaws” that provides just enough assistance to make the puzzle tractable again, rather than solving it outright, which would rob you of the joy and also be genuinely hard given the degrees of freedom involved (e.g., piece alignment, arbitrary rotations, irregular shapes, lighting variation).

我们的目标是打造一个“拼图界的管家（Jeeves）”，它提供恰到好处的辅助，让拼图重新变得可解，而不是直接帮你拼好——那样不仅会剥夺你的乐趣，而且考虑到涉及的自由度（如拼图对齐、任意旋转、不规则形状、光照变化），这本身也非常困难。

For instance, the AI could tell you which region of the board a specific piece most likely belongs to, which pieces might form a cluster, and which divide-and-conquer strategies may be most fruitful.

例如，AI可以告诉你某块拼图最可能属于画面的哪个区域，哪些拼图可能组成一个集群，以及哪些分治策略最为有效。

Interestingly, jigsaw-esque problems show up in contexts ranging from satellite image tile stitching and forensic document reconstruction to manufacturing assembly verification and art restoration. Each of these can be framed as a fragment-to-reference matching problem and the related solution techniques (e.g., feature extraction, similarity measurement, global assignment) are applicable across domains.

有趣的是，类似拼图的问题出现在各种场景中，从卫星图像拼接、法医文档重建，到制造装配验证和艺术品修复。每一个都可以被归纳为“碎片到参考图”的匹配问题，相关的解决方案（如特征提取、相似度测量、全局分配）在各个领域都是通用的。

The jigsaw puzzle is an intuitive, relatable, and verifiable use case in which to explore these ideas before applying them elsewhere. In what follows, we will build an assistant for solving jigsaw puzzles, starting by framing a simplified version of the problem, and ending with an implementation in Python using OpenCV, NumPy, and SciPy — something that you could even use yourself the next time you are confronted with a particularly tricky jigsaw puzzle.

拼图是一个直观、易懂且可验证的用例，适合在将其应用于其他领域之前探索这些想法。接下来，我们将构建一个拼图助手，从简化问题模型开始，最终使用Python中的OpenCV、NumPy和SciPy实现它——下次当你面对特别棘手的拼图时，甚至可以亲自使用它。

Framing the Jigsaw Problem

拼图问题的建模

A jigsaw puzzle consists of a fixed set of uniquely shaped, interlocking pieces that typically form a rectangular image. The goal is to reconstruct that image, starting from a scrambled pile of pieces, by placing every piece in its correct position with edges that interlock with their neighbors. A complete solution requires matching both the visual content of each piece and the geometric compatibility of its edges.

拼图由一组形状独特、相互咬合的碎片组成，通常拼成一幅矩形图像。目标是从一堆混乱的碎片开始，通过将每一块拼图放置在正确位置，并使其边缘与邻居咬合，从而还原图像。完整的解决方案需要同时匹配每块拼图的视觉内容及其边缘的几何兼容性。

This can be a difficult process to automate. Getting clean input data is the first challenge. The scrambled pieces would need to be photographed (e.g., with a smartphone), which introduces uneven lighting, shadows, glare, and perspective distortion. The solved reference is usually the puzzle box cover, which may carry overlaid text, a different color profile, and a different scale than the scrambled photo.

这个过程很难自动化。获取干净的输入数据是第一个挑战。混乱的碎片需要被拍摄下来（例如用智能手机），这会引入光照不均、阴影、反光和透视畸变。已完成的参考图通常是拼图盒盖，它可能带有覆盖的文字、不同的色彩配置，且比例与拍摄的照片不同。

Then there are the pieces themselves, with arbitrary orientations, irregular silhouettes, and large visually uniform regions (sky, grass, fur) where many pieces look nearly identical. Taken together, these degrees of freedom make jigsaw puzzles a hard problem to solve in a fully automated manner in general.

此外还有碎片本身，它们具有任意的方向、不规则的轮廓，以及大面积视觉上统一的区域（天空、草地、毛发），在这些区域中，许多碎片看起来几乎一模一样。综合来看，这些自由度使得拼图在一般情况下很难通过完全自动化的方式解决。

However, since our goal is only to build an assistant that can provide helpful nudges, we can simplify the problem scope significantly. Piece silhouettes can be ignored, allowing our algorithm to overlay a regular grid on both images and treat each grid cell as the unit of comparison; this reduces the geometric matching problem to a primarily visual one.

然而，由于我们的目标只是构建一个能提供有用指引的助手，我们可以显著简化问题范围。可以忽略碎片的轮廓，让我们的算法在两张图像上覆盖一个规则网格，并将每个网格单元视为比较单位；这会将几何匹配问题简化为主要的视觉匹配问题。

The grid lines may not follow piece edges exactly, but the resulting inaccuracies at cell boundaries tend to have an effect on the overall color and edge profile of each cell that is modest enough to produce useful localization for puzzles with varied imagery. And rather than placing every piece exactly right, the assistant need only narrow each piece down to a sufficiently small candidate region.

网格线可能无法完全贴合碎片边缘，但由此产生的单元格边界不准确性，对每个单元格整体颜色和边缘轮廓的影响通常较小，足以在图像丰富的拼图中实现有效的定位。而且，助手不需要将每块拼图放置得完全准确，只需将每块拼图缩小到一个足够小的候选区域即可。

We can state the resulting jigsaw piece assignment problem formally as follows: Given two images (a solved reference and a scrambled puzzle with tiles arranged in an R-by-C grid) find a bijective mapping from each scrambled tile position to its correct position in the solved grid.

我们可以将最终的拼图分配问题正式表述如下：给定两张图像（一张已完成的参考图和一张碎片排列在R×C网格中的混乱拼图），找到从每个混乱碎片位置到已完成网格中正确位置的双射映射。

Even with the above simplifications, three issues still make the problem non-trivial: 即便有了上述简化，仍有三个问题使得该问题并不简单：

Visual ambiguity: Large regions of similar color (sky, grass, water, fur, etc.) mean that many pieces look nearly identical when comparing small image patches. In such cases, the similarity scores between a candidate piece and any destination cell in that region will be nearly uniform, producing a flat distribution with no clear winner. The algorithm will have no principled basis for ranking one destination over another, so assignments in these regions may be essentially arbitrary.
视觉模糊性： 大面积相似颜色（天空、草地、水面、毛发等）意味着在比较小图像块时，许多碎片看起来几乎相同。在这种情况下，候选碎片与该区域内任何目标单元格之间的相似度得分将几乎一致，产生一个没有明显胜出的平坦分布。算法将没有原则性的依据来对不同的目标进行排序，因此这些区域的分配本质上可能是随机的。
Bijective mapping objective: A sequential greedy approach (without replacement) in which each piece is assigned to the best available destination…
双射映射目标： 一种顺序贪婪方法（无放回），其中每块碎片被分配到最佳可用目标……