本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Explainable Causal Reinforcement Learning for bio-inspired soft robotics maintenance with embodied agent feedback loops

Introduction: When a Silicone Tentacle Taught Me About Causality Last spring, I spent three weeks trying to keep a bio-inspired soft robotic arm from tearing itself apart. The arm—a pneumatically actuated silicone tentacle modeled loosely on an octopus—was supposed to learn how to grip fragile objects. Instead, it kept developing micro-tears near its bending joints, and my standard deep reinforcement learning agent had absolutely no idea why. It would happily keep executing the same policy, racking up reward, while the hardware quietly degraded underneath it. That experience sent me down a rabbit hole that I’m still climbing out of. I started asking questions that vanilla RL simply can’t answer: Why is the robot failing? Which physical variable actually caused the degradation? And can an embodied agent—one that lives inside the robot, sensing its own body—learn a maintenance policy that’s not just effective but explainable? While exploring the intersection of causal inference and reinforcement learning, I realized that the maintenance problem for soft robotics is fundamentally a causal problem. Soft bodies deform, fatigue, and fail in ways that are highly non-linear and difficult to observe directly. A policy that maximizes task reward without a causal model of its own wear-and-tear is essentially flying blind. This article is a write-up of what I learned building an explainable causal RL pipeline for soft robotic maintenance, complete with embodied agent feedback loops.

引言：当硅胶触手教会我因果关系时 去年春天，我花了三周时间试图防止一只仿生软体机械臂自我损毁。这只机械臂是一个气动硅胶触手，大致模仿了章鱼的结构，本应学习如何抓取易碎物体。然而，它却不断在弯曲关节附近产生微裂纹，而我使用的标准深度强化学习智能体完全不知道原因。它会愉快地继续执行相同的策略并不断获得奖励，而硬件却在它下方悄然退化。那次经历让我陷入了一个至今仍未走出的深渊。我开始提出普通强化学习无法回答的问题：机器人为什么会失败？哪个物理变量真正导致了退化？一个具身智能体——即生活在机器人内部、感知自身身体的智能体——能否学习一种不仅有效而且可解释的维护策略？在探索因果推理与强化学习的交叉领域时，我意识到软体机器人的维护问题本质上是一个因果问题。软体结构的变形、疲劳和失效方式高度非线性，且难以直接观察。如果一个策略在没有自身磨损因果模型的情况下最大化任务奖励，本质上就是在盲目飞行。本文记录了我构建用于软体机器人维护的可解释因果强化学习流水线的心得，其中包含了具身智能体反馈循环。

Why Soft Robotics Maintenance Is a Different Beast In my research of rigid robotics, maintenance is often a discrete, well-understood problem: replace a joint every N cycles, monitor motor current, done. Soft robotics breaks every one of those assumptions. Bio-inspired soft robots are typically made from silicone elastomers, hydrogels, or dielectric elastomer actuators. Their failure modes include: Fatigue crack propagation in hyperelastic materials, Hysteresis drift in pneumatic chambers, Viscoelastic creep that changes the robot’s kinematics over time, and Delamination between composite layers. None of these are directly observable from the outside. You can’t put a rotary encoder on a silicone tentacle and call it a day. The robot has to feel its own degradation through distributed sensing—which is exactly where embodied agent feedback loops become essential. One interesting finding from my experimentation with embedded strain sensors was that the rate of change of sensor readings, not their absolute values, was the strongest early predictor of failure. That’s a causal signal hiding inside a noisy time series.

为什么软体机器人维护是一个完全不同的难题 在我对刚性机器人的研究中，维护通常是一个离散且易于理解的问题：每运行 N 个周期更换一个关节，监测电机电流，搞定。但软体机器人打破了所有这些假设。仿生软体机器人通常由硅橡胶、水凝胶或介电弹性体驱动器制成。它们的失效模式包括：超弹性材料中的疲劳裂纹扩展、气腔中的迟滞漂移、随时间改变机器人运动学的粘弹性蠕变，以及复合层之间的分层。这些都无法从外部直接观察到。你不能在硅胶触手上安装一个旋转编码器就万事大吉了。机器人必须通过分布式传感来感知自身的退化——这正是具身智能体反馈循环变得至关重要的原因。我在嵌入式应变传感器实验中发现的一个有趣结论是：传感器读数的变化率，而非其绝对值，是失效的最强早期预测指标。这是一个隐藏在噪声时间序列中的因果信号。

The Core Idea: Causal RL with Structural Priors Standard RL maximizes expected return: J(θ) = E_π[Σ γ^t r_t]. Causal RL augments this by learning a structural causal model (SCM) of the environment, so the agent can reason about interventions: what happens if I do X instead of Y? For soft robotics maintenance, the SCM has a natural structure. Let’s define: S_t: sensor readings (strain, pressure, temperature), A_t: action (actuation command), D_t: latent degradation state, R_t: task reward, M_t: maintenance event. The causal graph looks like this: D_t ──► S_t ──► A_t ──► D_{t+1} │ │ └────────► R_t ◄─────────┘. The key insight from my exploration of causal discovery methods: if we can identify the causal parents of D_{t+1}, we can build a maintenance policy that intervenes on the causes of degradation rather than reacting to symptoms.

核心理念：带有结构先验的因果强化学习 标准强化学习旨在最大化期望回报：J(θ) = E_π[Σ γ^t r_t]。因果强化学习通过学习环境的结构因果模型 (SCM) 来增强这一点，从而使智能体能够对干预进行推理：如果我做 X 而不是 Y 会发生什么？对于软体机器人维护，SCM 具有天然的结构。我们定义：S_t：传感器读数（应变、压力、温度），A_t：动作（驱动指令），D_t：潜在退化状态，R_t：任务奖励，M_t：维护事件。因果图如下所示：D_t ──► S_t ──► A_t ──► D_{t+1} │ │ └────────► R_t ◄─────────┘。我从因果发现方法探索中获得的关键见解是：如果我们能识别出 D_{t+1} 的因果父节点，我们就能构建一种针对退化原因进行干预，而非仅仅对症状做出反应的维护策略。

Building the Causal World Model I implemented a lightweight causal world model using a variational autoencoder for latent degradation estimation, combined with a structural equation model for the dynamics. Here’s the core of it: [Code omitted for brevity]. The latent z is our best guess at the true degradation state. Crucially, we don’t train it to predict reward—we train it to predict future sensor readings, which forces it to capture the physical dynamics rather than task-specific shortcuts.

构建因果世界模型 我使用变分自编码器进行潜在退化估计，并结合动力学的结构方程模型，实现了一个轻量级的因果世界模型。其核心代码如下：[为简洁起见省略代码]。潜在变量 z 是我们对真实退化状态的最佳猜测。至关重要的是，我们训练它不是为了预测奖励，而是为了预测未来的传感器读数，这迫使它捕捉物理动力学，而不是寻找特定任务的捷径。

Embodied Agent Feedback Loops The “embodied” part of this architecture is what makes it work in practice. Instead of a single centralized controller, I deployed a hierarchy of agents that live inside the robot’s control loop: Reflex agents at the actuator level (millisecond timescale), Tactical agents at the joint level (second timescale), Strategic agents at the whole-body level (minute timescale). Each layer shares the causal world model but runs its own policy. The feedback loop is bidirectional: high-level agents send intentions down, and low-level agents send causal evidence up.

具身智能体反馈循环 该架构的“具身”部分是其在实践中发挥作用的关键。我没有使用单一的集中式控制器，而是部署了一个存在于机器人控制循环中的智能体层级结构：执行器层面的反射智能体（毫秒级）、关节层面的战术智能体（秒级）、全身层面的战略智能体（分钟级）。每一层都共享因果世界模型，但运行各自的策略。反馈循环是双向的：高层智能体向下发送意图，低层智能体向上发送因果证据。