Generative Modeling: Learning the Distribution Behind the Data

生成式建模：学习数据背后的分布

Generative Modeling starts with a simple shift in perspective: instead of memorizing individual observations, learn the probabilistic structure that produced them. If a model can represent that structure, it can use the learned distribution to evaluate data and generate new samples that follow the same underlying patterns. That gives us the core mental model: Learn the distribution behind the data, then use that distribution for estimation or generation.

生成式建模始于视角上的简单转变：不再是死记硬背单个观测样本，而是学习产生这些样本的概率结构。如果一个模型能够表示这种结构，它就可以利用学到的分布来评估数据，并生成遵循相同底层模式的新样本。这为我们提供了核心思维模型：学习数据背后的分布，然后利用该分布进行评估或生成。

From Observed Data to a Model Distribution

从观测数据到模型分布

Suppose a dataset contains many dog images. Storing those images is not enough to create a new dog image that was never part of the training set. To generate new samples, the model needs to capture structure shared across the observed examples and represent that structure as a probability distribution. We can think of the training data as samples drawn from an unknown Data Distribution: $\text{training data} \sim p_{\text{data}}(x)$

假设一个数据集包含许多狗的图像。仅仅存储这些图像不足以创造出一张训练集中从未出现过的新狗图。为了生成新样本，模型需要捕捉观测示例中共享的结构，并将该结构表示为概率分布。我们可以将训练数据视为从一个未知的“数据分布”中抽取的样本： $\text{训练数据} \sim p_{\text{data}}(x)$

The exact form of $p_{\text{data}}(x)$ is not known in advance. We only observe samples from it. Generative Modeling therefore learns a Model Distribution, $p_{\text{model}}(x)$, from those observations. New samples can then be drawn from the learned distribution: $\text{generated samples} \sim p_{\text{model}}(x)$

$p_{\text{data}}(x)$ 的确切形式事先是未知的，我们只能观测到其中的样本。因此，生成式建模通过这些观测结果来学习一个“模型分布” $p_{\text{model}}(x)$。随后，可以从学到的分布中抽取新样本： $\text{生成样本} \sim p_{\text{model}}(x)$

Here, $p_{\text{data}}(x)$ represents the distribution followed by real data, while $p_{\text{model}}(x)$ represents the distribution learned by the model. The goal is for the Model Distribution to represent the Data Distribution well enough to support both probability evaluation and new sample generation.

在此，$p_{\text{data}}(x)$ 代表真实数据所遵循的分布，而 $p_{\text{model}}(x)$ 代表模型所学习到的分布。其目标是让模型分布能够充分代表数据分布，从而同时支持概率评估和新样本生成。

密度估计与样本生成共享同一基础

A learned Model Distribution can be used in two closely related ways.

学到的模型分布可以通过两种紧密相关的方式使用。

Density Estimation focuses on learning a distribution that can evaluate probabilities for data. If the model captures the structure of real data well, samples similar to real data can receive higher probability, while samples that differ from that structure can receive lower probability. The important point is that the model is not simply remembering particular training examples. It is representing the broader probabilistic structure shared by the data.

密度估计专注于学习一个能够评估数据概率的分布。如果模型能很好地捕捉真实数据的结构，那么与真实数据相似的样本将获得更高的概率，而偏离该结构的样本则获得较低的概率。关键点在于，模型并非仅仅是在记忆特定的训练示例，而是在表示数据所共享的更广泛的概率结构。

Sample Generation uses the learned Model Distribution to draw new samples. The generated samples do not need to be identical to anything in the training set. What matters is that they follow the structure represented by $p_{\text{model}}(x)$.

样本生成利用学到的模型分布来抽取新样本。生成的样本不需要与训练集中的任何内容完全相同，重要的是它们遵循 $p_{\text{model}}(x)$ 所表示的结构。

Why Random Sampling Is Not Enough

为什么随机采样是不够的

Images, text, and audio are represented in high-dimensional spaces. But not every point in those spaces corresponds to meaningful real data. Consider images. If every pixel value is selected uniformly at random, the result is unlikely to resemble a real image. The same applies to text. Randomly choosing characters rarely produces meaningful language.

图像、文本和音频都是在高维空间中表示的。但这些空间中的每一个点并不都对应于有意义的真实数据。以图像为例，如果每个像素值都是随机均匀选择的，结果很可能看起来不像一张真实的图像。文本也是如此，随机选择字符几乎无法产生有意义的语言。

The number of possible images or text sequences can be enormous, while realistic data occupies only a negligible portion of that space. So generation is not simply a matter of selecting values from a large space. The model has to represent the restricted structure where real data actually exists.

可能的图像或文本序列数量极其庞大，而真实数据仅占据了该空间中微不足道的一部分。因此，生成不仅仅是从大空间中选择数值的问题，模型必须表示真实数据实际存在的受限结构。

The Real Challenge in High Dimensions

高维空间中的真正挑战

The difficulty is not only that the data lives in a high-dimensional space. The model must represent the complex structure formed by real observations inside that space. Real data exists in limited, structured regions rather than being spread uniformly across all possible configurations. This perspective connects naturally to the kind of data structure considered in Manifold Learning.

困难不仅在于数据存在于高维空间中，还在于模型必须表示真实观测结果在该空间内形成的复杂结构。真实数据存在于有限的、结构化的区域中，而不是均匀分布在所有可能的配置中。这一视角自然地与流形学习（Manifold Learning）中所考虑的数据结构联系起来。

A naive picture of generation would be: Generation = choose values from a large space. A better mental model is: Generation = learn where realistic data is structured + sample according to that learned structure.

一种天真的生成观念是：生成 = 从大空间中选择数值。更好的思维模型是：生成 = 学习真实数据的结构位置 + 根据学到的结构进行采样。

Representing complex high-dimensional distributions requires sufficient model capacity. Simple models may not be able to capture this complexity, which leads naturally toward Deep Generative Models for representing more complex data distributions.

表示复杂的、高维的分布需要足够的模型容量。简单的模型可能无法捕捉这种复杂性，这自然地引导我们使用深度生成模型（Deep Generative Models）来表示更复杂的数据分布。

Why Distribution Learning Matters Beyond Direct Generation

为什么分布学习不仅仅是为了直接生成

Generative Modeling matters for more than producing new samples. Once a model has learned structure in the data distribution, that structure can also support other learning problems. In reinforcement learning, possible futures can be simulated and used for Planning. In Continual Learning, Generative Replay can reuse previously learned information through generated data.

生成式建模的意义远不止于产生新样本。一旦模型学习到了数据分布中的结构，该结构还可以支持其他学习问题。在强化学习中，可以模拟可能的未来并用于规划（Planning）；在持续学习（Continual Learning）中，生成式重放（Generative Replay）可以通过生成的数据重用先前学到的信息。

Distribution learning can also help when part of the data is unobserved. The learned structure can support predictions about missing parts, connecting Generative Modeling with Imputation. Another important connection is Latent Representation learning. A generative model can capture features shared across different observations in a latent form, and those representations can then serve as general features for other learning problems.

当部分数据未被观测到时，分布学习也能提供帮助。学到的结构可以支持对缺失部分的预测，从而将生成式建模与数据填补（Imputation）联系起来。另一个重要的联系是潜在表示学习（Latent Representation learning）。生成模型可以以潜在形式捕捉不同观测结果中共享的特征，这些表示随后可以作为其他学习问题的通用特征。