Painting with Gaussians / 用高斯函数作画

August 3, 2026 2026年8月3日

Last year I built an edge-aware pixelation tool to turn images into pixel art by deforming a grid so that it follows image edges instead of naively using a fixed grid over the picture. Using an edge adapter grid bent to inform the color and brightness of the pixels worked well for keeping edges crisp and preserving details. 去年，我构建了一个边缘感知像素化工具，通过变形网格使其跟随图像边缘，而不是简单地在图片上使用固定网格，从而将图像转换为像素艺术。使用这种能够根据边缘调整的弯曲网格来确定像素的颜色和亮度，在保持边缘清晰和保留细节方面效果显著。

I later realized the same edge information could also be applied in a context of digital painting. A painting program has to figure out where to put brush strokes, how big should they be, and which direction should they flow in. Much of that is already encoded in the edges since they mark the boundaries between regions. These are the contours around areas of objects that a brush would trace, and their absence indicates generally flat areas where a few broad strokes should suffice. 后来我意识到，同样的边缘信息也可以应用于数字绘画。绘画程序必须确定在哪里放置笔触、笔触应该多大，以及它们应该朝哪个方向流动。这些信息大部分已经编码在边缘中，因为它们标记了区域之间的边界。这些是画笔会描绘的物体轮廓，而它们的缺失通常意味着平坦区域，只需几笔宽大的笔触即可。

And so, I set off to see if I could make a program that paints in the style of a digital painting where marks derived from the image structure would resemble brush strokes. My goal was to make an interactive tool where you drag sliders around and the painting reforms itself in front of you. The project was also a great opportunity for me to test drive Jolt and see how well it works for building a non-trivial project. 于是，我开始尝试制作一个程序，以数字绘画的风格进行创作，其中从图像结构中提取的标记类似于笔触。我的目标是制作一个交互式工具，通过拖动滑块，画作就能在你面前实时重构。这个项目也是我测试 Jolt 并观察其在构建复杂项目时表现如何的绝佳机会。

In this post, I’ll walk you through how it all came together. We’ll see what ideas worked and which ones didn’t. Most importantly, we’ll find out whether the end result actually ends up resembling anything like a painting. 在这篇文章中，我将带你了解这一切是如何实现的。我们将看到哪些想法奏效了，哪些没有。最重要的是，我们将看看最终结果是否真的看起来像一幅画。

Why Gaussians? / 为什么选择高斯函数？

The first question we need to consider is what a brush stroke is exactly in computational terms. A stroke of oil or acrylic is an elongated mark which has a center of color that fades toward its edges, and its orientation is the product of a brush being dragged across a canvas. It’s translucent at the edges, and strokes overlap, allowing a painter to lay down broad blocks of color first, then build detail on top with smaller and more translucent marks to add finer detail. 我们需要考虑的第一个问题是，从计算角度来看，笔触到底是什么。油画或丙烯画的笔触是一个细长的标记，其颜色中心向边缘逐渐淡出，其方向是画笔在画布上拖动的结果。它在边缘处是半透明的，且笔触会重叠，这使得画家可以先铺设大面积的色块，然后用更小、更透明的标记在上面构建细节，以增加更精细的层次。

It turns out that a 2D Gaussian splat maps onto this idea surprisingly well. It has a mean which is where the stroke lands, a covariance matrix representing how it’s stretched and rotated, along with a color and opacity. The covariance can be used to encode the brush direction and its elongation with the major axis pointing along the stroke, and the minor axis across it. Rendering a field of splats with standard over-compositing where each one is occluding what’s behind it by its alpha gives you a similar effect to a natural painting model that allows marks to layer and blend together. Of course, you don’t get the same fidelity of actual paint, so the effect is closer to digital painting using a tool like GIMP or Krita. 事实证明，二维高斯溅射（Gaussian splat）与这一概念非常契合。它有一个均值（代表笔触落点）、一个协方差矩阵（代表其拉伸和旋转方式），以及颜色和不透明度。协方差可用于编码画笔方向及其拉伸程度，长轴指向笔触方向，短轴则横跨笔触。使用标准的“覆盖合成”（over-compositing）渲染溅射场，即每个溅射根据其 Alpha 值遮挡其背后的内容，可以产生类似于自然绘画模型的效果，允许标记层叠和混合。当然，你无法获得真实颜料的质感，因此这种效果更接近于使用 GIMP 或 Krita 等工具进行的数字绘画。

There are already some implementations of this idea such as DrawingWithGaussians and 2d-gaussian-splatting-Art. However, both of them use a gradient descent approach where they seed random splats, and then iteratively nudge their positions, shapes, and colors until the rendered field has the appearance of a target image. That’s the well known approach which is both slow and opaque. The worst part is that the end result ends up being simply a lossy reconstruction of the input image rather than looking like any sort of a painting. 目前已经有一些实现这一想法的项目，例如 DrawingWithGaussians 和 2d-gaussian-splatting-Art。然而，它们都使用了梯度下降法，即先随机生成溅射，然后迭代调整它们的位置、形状和颜色，直到渲染出的图像看起来像目标图像。这是一种众所周知的方法，但既缓慢又不透明。最糟糕的是，最终结果仅仅是输入图像的有损重建，看起来并不像任何形式的画作。

Since I already had the solution for extracting edge information from the image, I didn’t see the point of evolving the image blindly. Instead, the extracted edges can be used to guide the painting process because they tell us where the details are along with the orientation of the strokes. Between detail density and pixel colors I had all the information that I’d need without having to resort to gradient descent. I’d basically just need to trace the existing image. How hard could it be really? 由于我已经有了从图像中提取边缘信息的解决方案，我不觉得有必要盲目地演化图像。相反，提取出的边缘可以用来引导绘画过程，因为它们告诉了我们细节在哪里以及笔触的方向。有了细节密度和像素颜色，我就拥有了所需的所有信息，而无需诉诸梯度下降。我基本上只需要描摹现有的图像。这能有多难呢？

So, I started following the reference rasterizer which uses additive blending where: pixel = background + Σ(intensity × color). Turns out, this approach works in the fitting regime because the optimizer learns colors that compensate for overlap. Unfortunately, seeding thousands of splats directly from pixel colors and rendering them additively creates a lot more overlap. With 1,200 splats on a 64×64 image, the sum hit 22.06 in some pixels, creating pure white blobs all over the image. Luckily, the problem can be solved by using the standard over-operator from alpha compositing to make each splat occlude what’s behind it by its alpha so that the summed color never exceeds 1.0. Another benefit of this approach is that it cleanly separates color sampled from the image and opacity. 因此，我开始参考使用加法混合的栅格化器，公式为：像素 = 背景 + Σ(强度 × 颜色)。事实证明，这种方法在拟合阶段是有效的，因为优化器会学习补偿重叠的颜色。不幸的是，直接从像素颜色生成数千个溅射并进行加法渲染会产生过多的重叠。在 64×64 的图像上使用 1,200 个溅射时，某些像素的总和达到了 22.06，导致图像上到处都是纯白色的斑点。幸运的是，这个问题可以通过使用 Alpha 合成中的标准“覆盖算子”（over-operator）来解决，使每个溅射根据其 Alpha 值遮挡其背后的内容，从而使总颜色值永远不会超过 1.0。这种方法的另一个好处是，它清晰地分离了从图像中采样的颜色和不透明度。

Edges Tell You Where to Paint / 边缘告诉你该在哪里作画

Every image in this section is the same photo run through the same pipeline, with one idea switched off at a time — same source, same stroke budget, same base size — so each step shows exactly what that one idea buys. 本节中的每张图片都是同一张照片通过同一流程处理的结果，每次关闭一个想法——相同的源、相同的笔触预算、相同的基准尺寸——因此每一步都准确展示了该想法带来的效果。

I started using the following source photo, and recorded the progress as I continued to improve the app to illustrate what each idea buys. Let’s see how the painting evolves as new tricks are added to the mix. 我开始使用以下源照片，并记录了随着我不断改进应用程序而取得的进展，以说明每个想法的作用。让我们看看随着新技巧的加入，画作是如何演变的。

I got a rather sad output which looked like a uniform mosaic with my initial renderer. Every splat had the same size, aspect ratio, and rotation, producing a regular grid of identical blobs. Not really looking like much of a painting so far. 在最初的渲染器中，我得到了一个相当令人沮丧的结果，看起来像是一个均匀的马赛克。每个溅射都有相同的大小、长宽比和旋转角度，产生了一个由相同斑点组成的规则网格。到目前为止，看起来并不太像一幅画。

An actual painter would vary their strokes using a few broad strokes for flat regions such as the sky or a smooth surface. Then, along edges and in textured areas like eyes or fabric, a smaller brush gets used to make numerous finer strokes that follow the contours of the objects. 真正的画家会改变他们的笔触，在天空或光滑表面等平坦区域使用几笔宽大的笔触。然后，沿着边缘和眼睛或织物等纹理区域，使用较小的画笔进行多次更精细的笔触，以跟随物体的轮廓。

One way to emulate this is by using a structure tensor to compute the image gradient, encoding how much and in which direction the color changes for each pixel. A 2×2 tensor is formed from the gradient outer product, and blurred over a neighborhood. Importantly, the tensor’s eigenvectors will tell you three key things. The major eigenvector points across the contour, providing the direction of the strongest gradient. The minor eigenvector points along the edge, giving the direction of the… 模拟这一点的一种方法是使用结构张量来计算图像梯度，编码每个像素颜色变化的程度和方向。通过梯度外积形成一个 2×2 的张量，并在邻域内进行模糊处理。重要的是，张量的特征向量会告诉你三个关键信息。主特征向量指向轮廓的横向，提供了最强梯度的方向。次特征向量指向边缘的纵向，给出了……的方向。