Dynamical System Transfer Learning with Reduced Order Models
Dynamical System Transfer Learning with Reduced Order Models
基于降阶模型的动力系统迁移学习
I have discussed previously the potential utility of applying Reinforcement Learning (RL) to analysis and control of complex physical systems. Developing RL algorithms to analyze or control complex dynamical systems faces one of the main challenges of many other RL problems — lengthy training times. This problem is complicated by the high computational cost of simulating complex physical dynamics. Compounding the thousands (perhaps millions) of RL training iterations needed for large state spaces and nonlinear systems with simulation times that may run into the hours (or even just minutes) per iteration can make problems intractable. Transfer learning has been used to reduce training times for RL problems and here we will explore application of transfer learning to RL for dynamical systems.
我之前讨论过将强化学习（RL）应用于复杂物理系统分析与控制的潜在效用。开发用于分析或控制复杂动力系统的强化学习算法，面临着许多其他强化学习问题的主要挑战之一——漫长的训练时间。模拟复杂物理动力学的高昂计算成本使这一问题变得更加复杂。对于大规模状态空间和非线性系统，强化学习通常需要数千（甚至数百万）次训练迭代，如果每次迭代的模拟时间长达数小时（甚至仅需几分钟），都会使问题变得难以处理。迁移学习已被用于减少强化学习问题的训练时间，在这里，我们将探讨迁移学习在动力系统强化学习中的应用。
Transfer learning relies on using models trained on similar problems to reduce training time with the assumption that the problem is similar enough that the model will have to make only small changes. For example, using an RL algorithm trained to drive a small RC car will probably more quickly learn how to drive a tractor trailer than one starting from scratch despite the considerable difference between the two.
迁移学习依赖于使用在类似问题上训练过的模型来减少训练时间，其前提是问题足够相似，模型只需进行微小的调整。例如，使用一个训练过驾驶小型遥控车的强化学习算法，学习驾驶半挂卡车的过程，很可能比从零开始训练要快得多，尽管两者之间存在显著差异。
Reduced Order Models
降阶模型 (Reduced Order Models, ROMs)
For physics simulations we need a problem environment to train the RL algorithm on that will be similar enough but substantially faster to simulate than the original. Reduced Order Models (ROMs) provide a way to generate such environments. A ROM, at its simplest, is a simplified model that seeks to reduce complexity while retaining as much accuracy as possible. There are many ways to generate ROMs. Any AI can be considered a ROM, from supervised learning to even large language models as ROMs of human language. Here we will apply unsupervised learning based on an approach outlined in Brunton and Kutz’s book Data Driven Science and Engineering to develop a data-driven ROM of a physical dynamical system based only on a data set of system measurements with little knowledge of the underlying system.
对于物理模拟，我们需要一个用于训练强化学习算法的问题环境，它既要足够相似，又要比原始环境模拟得快得多。降阶模型（ROM）提供了一种生成此类环境的方法。简单来说，ROM 是一种简化模型，旨在降低复杂性的同时尽可能保持准确性。生成 ROM 的方法有很多。任何人工智能都可以被视为一种 ROM，从监督学习到甚至作为人类语言“降阶模型”的大型语言模型。在这里，我们将应用基于 Brunton 和 Kutz 的著作《数据驱动科学与工程》（Data Driven Science and Engineering）中概述的方法，通过无监督学习，仅基于系统测量数据集，在对底层系统知之甚少的情况下，开发物理动力系统的数据驱动 ROM。
As a brief introduction, dynamical systems are those that can be described as evolving through time. They are often summarized as $d\mathbf{x}/dt = F(\mathbf{x})$, where $\mathbf{x}$ is the state vector (set of variables that describe the system), $d\mathbf{x}/dt$ is the change in state with the change in time, and $F(\mathbf{x})$ is some function of the state variables (sometimes including their history). The state vector $\mathbf{x}$ can be one variable or thousands, especially when state history is needed. $F(\mathbf{x})$ may be a linear or nonlinear system. Linear systems have predictable response to input—double the input, double the response. Nonlinear systems, far more common and representative of the real world, do not exhibit that behavior throughout all possible states. Doubling input may double output, multiply output by 100, or drive output to 0, and this behavior can change depending on where you are in the state space (region of all possible values the variables can take).
简要介绍一下，动力系统是指那些可以描述为随时间演化的系统。它们通常总结为 $d\mathbf{x}/dt = F(\mathbf{x})$，其中 $\mathbf{x}$ 是状态向量（描述系统的一组变量），$d\mathbf{x}/dt$ 是状态随时间的变化，而 $F(\mathbf{x})$ 是状态变量的某种函数（有时包括其历史记录）。状态向量 $\mathbf{x}$ 可以是一个变量，也可以是数千个变量，特别是在需要状态历史记录时。$F(\mathbf{x})$ 可以是线性或非线性系统。线性系统对输入有可预测的响应——输入加倍，响应也加倍。非线性系统在现实世界中更为常见且具有代表性，它们在所有可能的状态下并不表现出这种行为。输入加倍可能导致输出加倍、输出乘以 100，或将输出驱动至 0，并且这种行为可能会根据你在状态空间（变量可取的所有可能值的区域）中的位置而改变。
Real-world physical systems exhibit nonlinearity making them difficult to characterize. On top of that, realistic physical and engineering systems often lack solvable, direct, comprehensive governing equations that would support analysis of their behavior. The equations we can obtain, much less solve, often require simplification that constrict their utility to a subregion of the state space or lose track of higher order dynamics. Consider turbojet engine dynamics (which will be our example problem): the simulation we will use is based on a series of coupled equations that must be solved iteratively rather than directly solving a dynamic system of equations. Even then, it contains many simplifications of the coupled fluid, structural, thermodynamic, and chemical processes at work. Typical approaches, such as linearization around known solutions and creative use of coordinate systems, have become less helpful as complexity and scope of systems under analysis increase. However, as system complexity has exceeded traditional analytical approaches, our ability to gather, store, aggregate, and analyze measurement data at GB, TB, and even PB level has improved.
现实世界的物理系统表现出非线性，使其难以表征。此外，现实的物理和工程系统往往缺乏可求解、直接且全面的控制方程来支持对其行为的分析。我们能够获得（更不用说求解）的方程通常需要简化，这将其效用限制在状态空间的子区域内，或者丢失了高阶动力学信息。以涡轮喷气发动机动力学为例（这将是我们的示例问题）：我们将使用的模拟基于一系列必须迭代求解的耦合方程，而不是直接求解动力系统方程。即便如此，它也包含了对耦合流体、结构、热力学和化学过程的许多简化。随着分析系统复杂性和范围的增加，传统的处理方法（如围绕已知解进行线性化和创造性地使用坐标系）已变得不再那么有效。然而，随着系统复杂性超过了传统分析方法的能力，我们在 GB、TB 甚至 PB 级别上收集、存储、聚合和分析测量数据的能力得到了提升。
Brunton and Kutz’s text discusses new techniques to approach dynamical systems exploiting increased measurement, storage, and analysis capabilities. Their approaches are based on unsupervised learning—that is looking for underlying patterns without a priori knowledge of what those patterns should be. These techniques use measurements of system state over time to derive dynamic models that can be used to describe, analyze, and predict behavior. The particular method we will use here is SINDy (Sparse Identification of Nonlinear Dynamics).
Brunton 和 Kutz 的著作讨论了利用增强的测量、存储和分析能力来处理动力系统的新技术。他们的方法基于无监督学习——即在没有先验知识的情况下寻找潜在模式。这些技术利用系统随时间的测量状态来推导动力学模型，从而用于描述、分析和预测行为。我们在这里将使用的特定方法是 SINDy（非线性动力学的稀疏识别）。
Sparse Identification of Nonlinear Dynamics (SINDy)
非线性动力学的稀疏识别 (SINDy)
The goal of the SINDy algorithm is to discover system models in the form of mathematical equations using system measurement data, specifically a time series of state data that can be used to calculate an array of time derivatives of state variables over time. Essentially, it takes a candidate library of possible equation terms (such as products, quotients, and derivatives of states) and determines a parsimonious equation (that is, one that uses the fewest number of terms) that best describes the state changes over time. The basic equation describing SINDy is: $d\mathbf{x}/dt = \theta(\mathbf{x})\Xi$, where $\Theta(\mathbf{x})$ is the array of possible equation terms and $\Xi$ is a matrix of coefficients for those terms. For example, if the equation exhibits strong quadratic behavior but no sinusoidal, then the value in $\Xi$ corresponding to the quadratic vector of $\Theta(\mathbf{x})$ will have a large absolute value and the value corresponding to the sinusoid will be near 0.
SINDy 算法的目标是利用系统测量数据（特别是可用于计算状态变量随时间变化的时间导数数组的状态数据时间序列）以数学方程的形式发现系统模型。本质上，它采用一个包含可能方程项（如状态的乘积、商和导数）的候选库，并确定一个最能描述随时间状态变化的简约方程（即使用最少项数的方程）。描述 SINDy 的基本方程是：$d\mathbf{x}/dt = \theta(\mathbf{x})\Xi$，其中 $\Theta(\mathbf{x})$ 是可能方程项的数组，$\Xi$ 是这些项的系数矩阵。例如，如果方程表现出强烈的二次行为但没有正弦行为，那么 $\Xi$ 中对应于 $\Theta(\mathbf{x})$ 二次向量的值将具有较大的绝对值，而对应于正弦项的值将接近 0。