Linear Discriminant Analysis (LDA) in Real-Life: Dimensionality Reduction in a Real-Estate Dataset

线性判别分析 (LDA) 实战：房地产数据集中的降维应用

Linear Discriminant Analysis (LDA) is a supervised learning technique used to surface the core components, or patterns, in the data. In this article we’re going to see how LDA works with a real-life example, how it is used to reduce the dimensionality of a dataset and identify class separation boundaries.

线性判别分析 (LDA) 是一种监督学习技术，用于提取数据中的核心成分或模式。在本文中，我们将通过一个现实生活中的例子来了解 LDA 的工作原理，以及它如何用于降低数据集的维度并识别类别分离边界。

Linear Discriminant Analysis is a statistical technique, typically applied in the data preparation phase of Machine Learning Classification problems. It’s used to reduce the dimensionality of a dataset, and highlight the characteristics of the data that best ensure the separability of its different classes.

线性判别分析是一种统计技术，通常应用于机器学习分类问题的数据准备阶段。它用于降低数据集的维度，并突出显示最能确保不同类别可分性的数据特征。

One major application of LDA, for instance, is in image classification. Image classification datasets tend to have thousands of features and LDA is used to reduce the number of features that can be used to properly distinguish between classes, and then run a classification machine learning algorithm on a much smaller feature space.

例如，LDA 的一个主要应用是在图像分类中。图像分类数据集往往具有数千个特征，LDA 被用于减少特征数量，以便在正确区分类别的前提下，在更小的特征空间上运行机器学习分类算法。

Linear Discriminant Analysis: the technique

线性判别分析：技术原理

In literature, LDA can also be referred to as Normal Discriminant Analysis or Fisher Linear Discriminant Analysis, the latter being a reference to Ronald A. Fisher, the polymath who developed the criterion LDA aims to maximize.

在文献中，LDA 也被称为正态判别分析或费舍尔线性判别分析 (Fisher Linear Discriminant Analysis)，后者是为了纪念罗纳德·A·费舍尔 (Ronald A. Fisher)，这位博学家开发了 LDA 旨在最大化的准则。

The goal

目标

The idea behind LDA is to take a dataset with high-dimensionality, e.g., a dataset with hundreds or thousands of features, and represent that same dataset with a smaller number of features. After applying LDA, you don’t necessarily have features, you’ll have different linear discriminants, which still encode the original characteristics of the data.

LDA 的核心思想是获取一个高维数据集（例如具有数百或数千个特征的数据集），并用较少数量的特征来表示该数据集。应用 LDA 后，你得到的可能不再是原始特征，而是不同的线性判别式，它们仍然编码了数据的原始特征。

This latter point is crucial. This technique will reduce the amount of information that is used to describe the original dataset, because originally you had a large amount of features and afterwards you have maybe a handful of discriminant components. However, the characteristics of the original dataset are preserved, meaning, the dataset is just encoded in a much smaller feature space.

最后一点至关重要。该技术会减少用于描述原始数据集的信息量，因为最初你有大量的特征，而之后可能只有少数几个判别分量。然而，原始数据集的特征得到了保留，这意味着数据集只是被编码到了一个更小的特征空间中。

It’s almost like you have a super complex music piece, with tons of notes for different instruments and then convert it to MIDI, which is encoded and transmitted in 8-bit bytes. Nevertheless, in the MIDI version you’re still able to distinguish the melody. The bones, i.e., the core characteristics of that music piece are still there, they’re just encoded in a lower-dimensionality space.

这就像你有一首极其复杂的乐曲，包含不同乐器的海量音符，然后将其转换为 MIDI 格式，以 8 位字节进行编码和传输。尽管如此，在 MIDI 版本中你仍然能够分辨出旋律。乐曲的骨架，即核心特征依然存在，只是被编码在了一个更低维的空间中。

Additionally, LDA ensures the different classes in the dataset are as distinguishable as possible, such that you’re able to separate the data points in each class as much as possible.

此外，LDA 确保数据集中的不同类别尽可能可区分，从而使你能够尽可能地分离每个类别中的数据点。

Assumptions

假设条件

Before jumping into the mathematical definition, it’s important to mention that LDA must take into account a few assumptions:

Data is linearly separable
Data follows a Gaussian (Normal) Distribution
Shared Covariance matrix across all classes

在深入数学定义之前，必须指出 LDA 需要考虑以下几个假设：

数据是线性可分的
数据服从高斯（正态）分布
所有类别共享协方差矩阵

Data is linearly separable: This is a very strong assumption for this technique. If you attempt to apply LDA to a dataset that is not linearly separable, i.e., the boundaries between the classes in the data are curves and not lines, you may get discriminants, but they may not be the best encoding of the core characteristics of the dataset. LDA is known to fail to properly capture nonlinear relationships and manifold structures in the data.

数据是线性可分的： 对于该技术而言，这是一个非常强的假设。如果你尝试将 LDA 应用于非线性可分的数据集（即数据中类别之间的边界是曲线而非直线），你可能会得到判别式，但它们可能不是数据集核心特征的最佳编码。众所周知，LDA 无法正确捕捉数据中的非线性关系和流形结构。

Data follows a Gaussian (Normal) Distribution: This technique assumes that data from each class is drawn from a Gaussian Distribution, i.e., the Normal Distribution.

数据服从高斯（正态）分布： 该技术假设每个类别的数据都来自高斯分布，即正态分布。

Shared Covariance matrix across all classes: In order to maximize the separability between classes, LDA needs to look at the variance in the data. However, since we’re working with high-dimensionality data, i.e., matrices, instead of scalars, we can’t use the variance as the measure of spread or dispersion. We need to use a covariance matrix. Succinctly, the covariance matrix describes the amount of spread each feature has, along its diagonal, as well as, the correlation between each pair of features, on the off diagonal.

所有类别共享协方差矩阵： 为了最大化类别之间的可分性，LDA 需要查看数据中的方差。然而，由于我们处理的是高维数据（即矩阵而非标量），我们不能使用方差作为散布或离散度的度量。我们需要使用协方差矩阵。简而言之，协方差矩阵在其对角线上描述了每个特征的散布程度，并在非对角线上描述了每对特征之间的相关性。

Specifically for LDA, the assumption is there’s a shared variance across all classes, meaning that each within-class covariance matrix is the same across all classes. This is necessary to ensure the pooled, i.e., shared, within-class covariance matrix is an accurate representation of each within class covariances.

具体对于 LDA 而言，假设所有类别共享方差，这意味着每个类内协方差矩阵在所有类别中都是相同的。这对于确保合并后的（即共享的）类内协方差矩阵能够准确代表每个类内的协方差是必要的。

Mathematical Definition

数学定义

Mathematically speaking, Linear Discriminant Analysis identifies the linear combination of features that best discriminate or distinguish between classes in the data. This method approximates the Bayes Classifier, which assigns an observation to a class for which the posterior probability, i.e., the probability the observation belongs to a class k, is the greatest.

从数学上讲，线性判别分析确定了最能区分数据中类别的特征线性组合。该方法近似于贝叶斯分类器，后者将观测值分配给后验概率（即观测值属于 k 类的概率）最大的类别。