TrackFish3D: Self-Supervised 3D Tracking of Schooling Fish from Multi-view Videos
TrackFish3D: Self-Supervised 3D Tracking of Schooling Fish from Multi-view Videos
TrackFish3D：基于多视角视频的鱼群自监督 3D 追踪
Abstract: Quantifying collective fish behavior requires accurate trajectories, yet multi-view 3D tracking remains challenging due to frequent occlusions, visually similar individuals, and the long-standing scarcity of identity annotations.
摘要： 量化鱼群的集体行为需要精确的轨迹，然而，由于频繁的遮挡、个体外观的高度相似性以及长期以来身份标注的匮乏，多视角 3D 追踪仍然极具挑战性。
We present TrackFish3D, a geometry-driven self-supervised framework for dense multi-camera 3D tracking of schooling fish. Instead of relying on appearance-based re-identification or manually annotated identities, TrackFish3D turns calibrated multi-view geometry into supervision: triangulation and reprojection consistency provide pseudo-associations, while a geometric encoder and global association transformer learn all-to-all cross-view correspondence within each frame.
我们提出了 TrackFish3D，这是一个用于鱼群密集多相机 3D 追踪的几何驱动自监督框架。TrackFish3D 不依赖于基于外观的重识别或人工标注的身份，而是将经过校准的多视角几何转化为监督信号：三角测量和重投影一致性提供了伪关联，同时几何编码器和全局关联 Transformer 学习了每一帧内所有视角间的对应关系。
To make these associations identity-aware, TrackFish3D introduces a self-supervised contrastive objective that separates co-visible individuals in the embedding space, together with a temporal predictor that preserves identities and bridges short occlusions across frames.
为了使这些关联具备身份感知能力，TrackFish3D 引入了一种自监督对比目标，在嵌入空间中分离同时可见的个体，并结合时间预测器来保持身份连续性，从而跨帧弥补短暂的遮挡。
The resulting model is trained once on unlabeled footage and applied directly to unseen test videos, requiring no cross-view identity labels, temporal annotations, 3D ground truth, appearance features, or test-time optimization.
该模型只需在未标注的素材上训练一次，即可直接应用于未见过的测试视频，无需跨视角身份标签、时间标注、3D 真值、外观特征或测试时优化。
On our benchmark, TrackFish3D improves 3D Multi-Object Tracking Accuracy from 87.7% for the strongest baseline to 95.8%. On the 3D-ZeF zebrafish benchmark, it achieves 81.1% MOTA, compared with 77.4% for the best geometric baseline. TrackFish3D also generalizes beyond fish, achieving strong results on real-world bird tracking.
在我们的基准测试中，TrackFish3D 将 3D 多目标追踪准确率（MOTA）从最强基准的 87.7% 提升至 95.8%。在 3D-ZeF 斑马鱼基准测试中，它达到了 81.1% 的 MOTA，而最佳几何基准为 77.4%。TrackFish3D 的泛化能力不仅限于鱼类，在现实世界的鸟类追踪中也取得了优异的成果。