Flux 3 X Mimic: The Next Generation of Video-Action Models
Flux 3 X Mimic: The Next Generation of Video-Action Models
Flux 3 与 Mimic：下一代视频-动作模型
An early version of FLUX 3, our new multimodal foundation model, is now running on robots. We gave mimic robotics early access to FLUX.3. Their strength in robot learning and deployment, combined with the model’s world knowledge and BFL’s foundation model expertise, produced FLUX-mimic: the next generation of video-action models.
我们全新的多模态基础模型 FLUX 3 的早期版本现已在机器人上运行。我们为 mimic robotics 提供了 FLUX.3 的早期访问权限。他们深厚的机器人学习与部署能力，结合该模型的世界知识以及 BFL 在基础模型方面的专业技术，共同打造了 FLUX-mimic：下一代视频-动作模型。
FLUX
FLUX
FLUX 1 and FLUX 2 generate images. FLUX 3 expands into multimodality and generates audio-visual content jointly - and, at the same time, provides the foundation of FLUX-mimic: A video-action model, developed in collaboration with mimic, running robots that have been tested and deployed at Audi.
FLUX 1 和 FLUX 2 用于生成图像。FLUX 3 则扩展至多模态领域，能够联合生成视听内容。同时，它也为 FLUX-mimic 提供了基础：这是一款与 mimic 合作开发的视频-动作模型，目前已在奥迪（Audi）的生产线上进行了测试与部署。
At first glance, producing convincing visual content and controlling robots seem to have little in common. One requires generating pixels, the other an understanding of how the physical world responds when you touch and manipulate it. If one model does both, it was never really only a content creation model. It is a model of how the world behaves, and content creation is one thing one can do with it. That is what FLUX 3 is.
乍看之下，生成逼真的视觉内容与控制机器人似乎毫无关联。前者需要生成像素，而后者需要理解当你触碰和操纵物理世界时，它会如何做出反应。如果一个模型能同时做到这两点，那么它本质上就不再仅仅是一个内容创作模型。它是一个关于“世界如何运作”的模型，而内容创作只是它能实现的功能之一。这正是 FLUX 3 的本质。
Video is the hard part
视频是难点所在
FLUX 3 is one model, jointly trained across images, video and audio from the beginning. The most demanding part of that training - accounting for over 95% of the total compute costs - is video prediction. To generate realistic videos, a model has no choice but to learn contact, motion, weight, cause and effect; get any of them wrong and it looks wrong. Learning to render the world accurately means learning how the world behaves.
FLUX 3 是一个从一开始就跨图像、视频和音频进行联合训练的单一模型。训练中最具挑战性的部分——占总计算成本的 95% 以上——是视频预测。为了生成逼真的视频，模型必须学习接触、运动、重量、因果关系；任何环节出错，画面都会显得不真实。学习如何准确地渲染世界，意味着学习世界是如何运作的。
Relatively speaking, audio is the easy modality. Low dimensional and far less detailed than video, it makes up less than 0.5% of the tokens in a 720p video with audio. Once a model has done the hard work of learning video understanding, it will learn the causal relationship between video and audio to predict speech synchronized to lip movement and audio effects synchronized to the physical events causing them.
相对而言，音频是一个简单的模态。它的维度较低，细节远少于视频，在一段带有音频的 720p 视频中，音频 token 的占比不到 0.5%。一旦模型完成了视频理解这一艰巨任务，它就能学习视频与音频之间的因果关系，从而预测与唇形同步的语音，以及与引发声音的物理事件同步的音效。
Actions follow the same shape: a low dimensional representation of a robot’s state, tightly coupled to visual observations. Actions, audio and video frames are all partial representations of a single underlying physical reality. After the model has learned about the physical processes behind video and audio, action prediction is not a new departure - it is one more view of the reality it already models.
动作遵循同样的逻辑：它是机器人状态的低维表示，与视觉观察紧密耦合。动作、音频和视频帧都是单一底层物理现实的局部表现。在模型学习了视频和音频背后的物理过程后，动作预测并非全新的课题，而是它所建模的现实的另一个维度。
A single backbone
单一主干网络
If that framing is correct, teaching FLUX 3 to predict actions should not incur lasting costs: we expect a brief phase of disturbance as the model has to learn the structure of the action space and align its internal representation of the world to it, before returning to full performance. That is exactly what we observe.
如果这一框架是正确的，那么教导 FLUX 3 预测动作不应产生持久的性能损耗：我们预计模型在学习动作空间结构并将其内部世界表示与之对齐时，会经历短暂的波动，随后便会恢复到最佳性能。这正是我们所观察到的结果。
In a large-scale training run, we added action prediction to the curriculum and observed the effect on video generation quality. Human ratings on text-to-video and image-to-video initially fell by up to 10% as the model started to incorporate the new action modality. After 3500 steps, the model had regained its full previous quality on video generation tasks while now also predicting actions.
在大规模训练中，我们将动作预测加入训练课程，并观察其对视频生成质量的影响。当模型开始整合新的动作模态时，人类对文生视频和图生视频的评分最初下降了多达 10%。但在 3500 步之后，模型在视频生成任务上恢复了此前的全部质量，同时还能进行动作预测。
The model had to integrate actions into its inputs and outputs - but doing so didn’t cost it capacity permanently. It merely had to learn how this new modality relates to its existing model of the world. Once this was figured out, the performance penalty on its existing capabilities was gone. Video generation and action prediction don’t need separate foundations. The same backbone carries both.
模型必须将动作整合到其输入和输出中，但这并没有永久性地消耗其容量。它仅仅是需要学习这一新模态如何与现有的世界模型相关联。一旦掌握了这一点，对现有能力的性能损耗便消失了。视频生成和动作预测不需要独立的基础模型，同一个主干网络即可承载两者。
This makes Physical AI a natural extension of our roadmap at Black Forest Labs rather than a change in direction. Content creation is what our multimodal FLUX 3 backbone does with image, video and audio. Physical AI is what it does with actions. One foundation model, with visual intelligence at its core, enabling two families of applications. We didn’t build a separate foundation model. We focused on the hard thing: building a model that understands the world. Acting in it is what that understanding makes possible.
这使得“物理 AI”（Physical AI）成为 Black Forest Labs 路线图中自然延伸的一部分，而非方向的转变。内容创作是我们多模态 FLUX 3 主干网络在图像、视频和音频上的应用，而物理 AI 则是它在动作上的应用。一个以视觉智能为核心的基础模型，支撑起了两大应用领域。我们没有构建独立的基础模型，而是专注于攻克难点：构建一个能够理解世界的模型。而在这个世界中采取行动，正是这种理解所带来的可能。
From lab to reality: FLUX-mimic
从实验室到现实：FLUX-mimic
What happens when we point the FLUX 3 backbone at real automation tasks on real production lines? That’s the question mimic and BFL created FLUX-mimic to answer. mimic builds their own robots and brings expertise in robot learning, dexterous manipulation and production deployment; BFL builds visual foundation models and brings multimodal training and modeling expertise. Together, we built a next-generation model for general-purpose manipulation - adapted to industry requirements and integrated into mimic’s full-stack deployment system. FLUX-mimic is a video-action model built on the FLUX 3 backbone.
当我们把 FLUX 3 主干网络应用于真实生产线上的自动化任务时，会发生什么？这就是 mimic 和 BFL 创建 FLUX-mimic 旨在回答的问题。mimic 制造自己的机器人，并拥有机器人学习、灵巧操作和生产部署方面的专业知识；BFL 构建视觉基础模型，并提供多模态训练和建模的专业能力。我们共同打造了一款面向通用操作的下一代模型——它适应工业需求，并集成到了 mimic 的全栈部署系统中。FLUX-mimic 正是基于 FLUX 3 主干构建的视频-动作模型。
Decoding the learned world model
解码已习得的世界模型
Our thesis is that FLUX 3 has to learn an internal representation of the world to be able to generate videos. FLUX-mimic follows through on this thesis and decodes actions from the learned world representation of the FLUX backbone. This approach, pioneered in mimic-video, trains a lightweight action decoder on top of intermediate features extracted from the video prediction path of FLUX.
我们的论点是：FLUX 3 必须学习世界的内部表示，才能生成视频。FLUX-mimic 遵循这一论点，从 FLUX 主干网络习得的世界表示中解码出动作。这种在 mimic-video 中首创的方法，通过在从 FLUX 视频预测路径中提取的中间特征之上，训练一个轻量级的动作解码器来实现。
The success of this approach depends on two related but different aspects: the quality of the world model learned by FLUX and the quality of the feature representation of this world model. The quality of the world model is directly related to the generation quality: if a model does not understand how the world behaves, it cannot simulate it. However, even the best world model does not help an action decoder if it is inaccessible: if the feature space keeps the causal relationships between modalities entangled nonlinearly, understanding those relationships from the feature representation remains as difficult as understanding them from the raw inputs - representation quality matters.
这种方法的成功取决于两个相关但不同的方面：FLUX 习得的世界模型质量，以及该世界模型特征表示的质量。世界模型的质量直接关系到生成质量：如果模型不理解世界的运作方式，它就无法模拟它。然而，即使是最好的世界模型，如果无法被访问也对动作解码器毫无帮助：如果特征空间将模态之间的因果关系以非线性方式纠缠在一起，那么从特征表示中理解这些关系，就像从原始输入中理解它们一样困难——因此，表示质量至关重要。