Three ffmpeg Filters Looked for a Person Crossing a Still Camera's View. One Found Them

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Continuous recording has an obvious cost: hours of a picture in which nothing changes, with the few seconds you care about somewhere inside. The usual answer is motion detection in the camera app, decided at recording time. This piece is about the other answer: keep the continuous recording and ask a laptop afterwards where the picture changed.

持续录制有一个显而易见的代价：数小时画面中什么都没有改变，而你关心的那几秒钟就隐藏在其中。通常的解决方法是在摄像机应用程序中进行录制时实时检测运动。本文探讨的是另一种方案：保留连续录制的内容，事后通过笔记本电脑分析画面在何处发生了变化。

To see which of ffmpeg’s built-in filters can do that, I scored three of them against a ninety-second synthetic clip with a person-sized shape crossing the room at a known time. One could not tell the person from the sensor noise at any setting. Another caught the person only inside a narrow band of settings, where a slow dimming of the room looked just as busy. The third found the crossing to within a frame or two at each end, once its thresholds were raised well past the defaults.

为了测试 ffmpeg 的内置滤镜中哪些能实现这一功能，我针对一段 90 秒的合成视频片段对其中三个滤镜进行了评分，该片段中有一个人大小的形状在已知时间穿过房间。其中一个滤镜在任何设置下都无法将人与传感器噪声区分开来。另一个滤镜仅在极窄的设置范围内捕捉到了人，且此时房间缓慢变暗的过程看起来也同样“忙碌”。第三个滤镜在将阈值调至远高于默认值后，能够将穿过过程定位在前后一两帧的范围内。

I build a camera app for old Android phones, but nothing here depends on it, and I am not describing how my app or any other app detects motion. Every result below comes from the commands shown, so you can rerun them and then try them on your own footage.

我为旧款安卓手机开发了一款摄像机应用，但本文内容并不依赖于此，我也不是在描述我的应用或其他应用如何检测运动。以下所有结果均来自所展示的命令，因此你可以重新运行它们，并在你自己的素材上进行尝试。

The test clip: The clip is synthetic on purpose: I know exactly when each event happens, so I can score the filters against the truth. It has four things in it. A grey room with a darker shape in one corner. A dark block, about the size a person would be at that distance, entering at the left edge at 20 s and leaving at the right edge by 26 s. A slow dimming of the whole picture from 40 s to 70 s, the way a room darkens at dusk. And a light switched back on, in a single frame, at 80 s. Every frame also carries temporal noise, which is what a small sensor produces in low light and what makes this hard.

测试片段：该片段是特意合成的：我确切地知道每个事件发生的时间，因此可以根据事实对滤镜进行评分。它包含四个要素：一个角落里有较暗形状的灰色房间；一个大小相当于该距离下人体的深色方块，在 20 秒时从左边缘进入，并在 26 秒时从右边缘离开；从 40 秒到 70 秒整个画面缓慢变暗，模拟黄昏时房间变暗的过程；以及在 80 秒时单帧内灯光重新开启。每一帧还带有时间噪声，这是小型传感器在弱光下产生的，也是导致检测困难的原因。

I checked the clip by pulling single frames and reading pixel values: at 23 s the block is in frame where expected, at 60 s the room is dimmer, at 85 s it is back to full brightness. A test clip needs its own check before it can check anything else: a clip in which the block never appeared would make every detector look right for reporting nothing. All runs below use ffmpeg 4.4.2 from Ubuntu 22.04. The current documentation describes options this build does not have, which I note where it matters.

我通过提取单帧并读取像素值来检查该片段：在 23 秒时方块位于预期的帧内，在 60 秒时房间变暗，在 85 秒时恢复到全亮度。测试片段在用于检测其他事物之前，自身需要先经过验证：如果片段中从未出现过方块，那么任何检测器在报告“无变化”时看起来都是正确的。以下所有运行均使用 Ubuntu 22.04 中的 ffmpeg 4.4.2 版本。当前的文档描述了一些该版本所不具备的选项，我会在相关地方进行说明。

freezedetect: built for the opposite question. freezedetect exists to find video that has stopped. The ffmpeg filter documentation says it “logs a message and sets frame metadata when it detects that the input video has no significant change in content during a specified duration.” For a fixed camera, a stretch with no significant change is an empty room, so the gaps between freezes should be where something happened. The documentation also says how it decides: “Video freeze detection calculates the mean average absolute difference of all the components of video frames and compares it to a noise floor.” That word mean is the whole story, as it turned out.

freezedetect：为相反的问题而生。freezedetect 的存在是为了寻找停止的视频。ffmpeg 滤镜文档称，当它检测到输入视频在指定持续时间内内容没有显著变化时，会“记录一条消息并设置帧元数据”。对于固定摄像机而言，没有显著变化的时段就是空房间，因此冻结之间的间隙应该是发生某些事情的地方。文档还说明了其判定方式：“视频冻结检测计算视频帧所有分量的平均绝对差，并将其与噪声基准进行比较。”事实证明，“平均”这个词就是问题的关键。

At the default noise tolerance, the whole ninety seconds is one long change, because the sensor noise alone is larger than the floor. That is a useful warning on its own: on footage from a small sensor at night, the defaults will report activity everywhere. At 0.02 the block appears, from 20.13 s to 25.80 s, which matches the stretch when any part of it is in the frame, give or take a frame or two. But the slow dimming produces three more change intervals and seven cuts, so a person and a sunset look alike in the output. At 0.03 the person disappears entirely, and only the dimming and the light switch remain. I swept the tolerance more finely: nothing froze at all up to 0.019, and the block was still caught at 0.025. On this clip the usable window is narrow, and a person-sized object is a s…

在默认的噪声容限下，整个 90 秒被视为一个漫长的变化过程，因为仅传感器噪声就大于基准值。这本身就是一个有用的警告：在夜间使用小型传感器拍摄的素材中，默认设置会报告到处都有活动。在 0.02 时，方块出现在 20.13 秒到 25.80 秒之间，这与它出现在画面中的时间段相吻合，误差仅在一两帧左右。但缓慢变暗的过程产生了另外三个变化区间和七次切断，因此在输出结果中，人和日落看起来是一样的。在 0.03 时，人完全消失了，只剩下变暗和灯光切换。我更精细地扫描了容限：在 0.019 之前没有任何冻结，而在 0.025 时仍能捕捉到方块。在这个片段上，可用的窗口很窄，而人大小的物体是一个……