Why Transformers Need Positional Encoding For Time Series: A Visual Guide
Why Transformers Need Positional Encoding For Time Series: A Visual Guide
为什么时间序列 Transformer 需要位置编码：可视化指南
While digging into foundation models for time series, I realized that I could not really understand them without first understanding transformers. I did not want to use these models as black boxes, so I started tracing the ideas backward, from foundation models to transformers, and from transformers to self-attention. What made the transition interesting is that although transformers were originally built for language, the core idea carries naturally to time series. The two modalities are very different, but they share something fundamental: both are sequences, and in both cases, order changes meaning.
在深入研究时间序列的基础模型时，我意识到如果不先理解 Transformer，就无法真正理解它们。我不想把这些模型当作“黑盒”来使用，所以我开始回溯这些概念：从基础模型到 Transformer，再从 Transformer 到自注意力机制（Self-Attention）。有趣的是，尽管 Transformer 最初是为语言处理而构建的，但其核心思想可以自然地迁移到时间序列中。这两种模态虽然截然不同，但它们共享一个基本特征：两者都是序列，且在两种情况下，顺序都会改变含义。
In language, “dog bites man” is very different from “man bites dog.” Time series are no different. A temperature of 30° yesterday and 20° today tells a different story from 20° yesterday and 30° today. The values may be the same, but their order changes the meaning of the sequence. The question is that if self-attention looks at all observations at once, how does a transformer know which observation came first, which came later, or how far apart two observations are? That question led me to positional encoding.
在语言中，“狗咬人”和“人咬狗”有着天壤之别。时间序列也是如此。昨天 30°、今天 20° 的气温，与昨天 20°、今天 30° 所传达的信息完全不同。数值可能相同，但顺序改变了序列的含义。问题在于，如果自注意力机制同时查看所有观测值，Transformer 如何知道哪个观测值在前，哪个在后，或者两个观测值之间相隔多远？这个问题引导我走向了位置编码（Positional Encoding）。
What surprised me most was how such a simple mathematical idea could give a Transformer a sense of order. The exact techniques have evolved considerably since then, but the underlying problem remains the same. This article is my attempt to build that intuition from the ground up, starting with a simple time series and following the path from raw observations to self-attention and finally to positional encoding.
最让我惊讶的是，这样一个简单的数学思想竟然能赋予 Transformer 顺序感。虽然具体的技术手段自那时起已经有了很大演变，但其背后的核心问题始终如一。本文旨在从零开始建立这种直觉，从简单的时间序列出发，沿着从原始观测值到自注意力机制，最后到位置编码的路径进行探讨。
From scalar observations to vector representations
从标量观测值到向量表示
Consider a simple time series containing the temperature recorded over five weekdays. Each observation $x_t$ is only a scalar. A transformer, however, operates on vectors of dimensionality $d_{model}$. The scalar observations therefore need to be mapped into that representation space first. A simple way to do this is through a learned linear projection (i.e., embedding): $e_t = W_e x_t + b_e$, giving us a sequence of vector representations: $e_1, e_2, e_3, e_4, e_5$.
考虑一个包含五个工作日气温记录的简单时间序列。每个观测值 $x_t$ 只是一个标量。然而，Transformer 是在维度为 $d_{model}$ 的向量上运行的。因此，标量观测值需要首先映射到该表示空间中。实现这一点的简单方法是通过学习到的线性投影（即嵌入）：$e_t = W_e x_t + b_e$，从而得到一系列向量表示：$e_1, e_2, e_3, e_4, e_5$。
An embedding is a deep, abstract representation of the series in the form of a multidimensional numerical vector that encodes its features and that the model understands. Each $e_t$ captures information about the observed value at that timestep, but at this point, it is just a representation of the observation. The important word here is “learned.” The model is not given a predefined vector representation for a temperature such as 18°. The parameters $W_e$ and $b_e$ are learned during training so that the resulting representations become useful for the task. At this point, $e_t$ represents what was observed. It does not yet tell the model where that observation occurred in the sequence.
嵌入（Embedding）是序列的一种深层、抽象的表示，表现为多维数值向量，它编码了序列的特征并能被模型所理解。每个 $e_t$ 都捕获了该时间步观测值的信息，但此时，它仅仅是观测值的表示。这里关键词是“学习到的”。模型并没有被预先赋予像 18° 这样的温度的预定义向量表示。参数 $W_e$ 和 $b_e$ 是在训练过程中学习到的，以便生成的表示对任务有用。此时，$e_t$ 仅代表观测到的内容，它尚未告诉模型该观测值在序列中的位置。
How self-attention builds context?
自注意力机制如何构建上下文？
Self-attention allows each observation to use information from the rest of the sequence. Suppose we want to update Friday’s representation. The model first creates three learned projections from every $e_t$: $q_t = W_Q e_t, k_t = W_K e_t, v_t = W_V e_t$. The matrices $W_Q, W_K$ and $W_V$ are also learned during training. The model is not told beforehand what a useful query, key, or value should look like.
自注意力机制允许每个观测值利用序列中其余部分的信息。假设我们要更新周五的表示。模型首先从每个 $e_t$ 创建三个学习到的投影：$q_t = W_Q e_t, k_t = W_K e_t, v_t = W_V e_t$。矩阵 $W_Q, W_K$ 和 $W_V$ 也是在训练过程中学习到的。模型事先并不知道什么样的查询（Query）、键（Key）或值（Value）是有用的。
For Friday, its query $q_5$ is compared with the keys of all observations: $k_1, k_2, k_3, k_4, k_5$. Each comparison produces an attention score: $s_{5,j} = \frac{q_5^\top k_j}{\sqrt{d_k}}$, which measures how relevant observation ‘j’ is when updating Friday’s representation. The scaling factor $\sqrt{d_k}$ prevents the dot products from growing too large as the dimensionality of the query and key vectors increases. These scores are passed through a softmax function to convert them into attention weights: $\alpha_{5,j} = \frac{\exp(s_{5,j})}{\sum_{j’} \exp(s_{5,j’})}$. Finally, those weights are used to combine the value vectors: $z_5 = \sum_j \alpha_{5,j}v_j$.
对于周五，其查询向量 $q_5$ 会与所有观测值的键向量 $k_1, k_2, k_3, k_4, k_5$ 进行比较。每次比较都会产生一个注意力分数：$s_{5,j} = \frac{q_5^\top k_j}{\sqrt{d_k}}$，该分数衡量了在更新周五的表示时，观测值 ‘j’ 的相关性。缩放因子 $\sqrt{d_k}$ 防止了随着查询和键向量维度的增加，点积结果变得过大。这些分数通过 Softmax 函数转换为注意力权重：$\alpha_{5,j} = \frac{\exp(s_{5,j})}{\sum_{j’} \exp(s_{5,j’})}$。最后，这些权重被用于组合值向量：$z_5 = \sum_j \alpha_{5,j}v_j$。
So $e_5$ is Friday’s representation before incorporating information from the rest of the sequence, while $z_5$ is its context-aware representation after self-attention. In short: Queries and keys learn which observations are relevant to one another. Values carry the information that is combined to form the new representation.
因此，$e_5$ 是周五在整合序列其余部分信息之前的表示，而 $z_5$ 是其经过自注意力机制处理后具有上下文感知的表示。简而言之：查询和键学习哪些观测值彼此相关，而值则携带用于形成新表示的信息。
What happens if we shuffle the sequence?
如果我们打乱序列会发生什么？
Now comes the important question. Suppose the same five temperature observations are rearranged. The values themselves have not changed; only the order has. After the learned projection, we still have the same set of value representations, just rearranged. Self-attention can still compare each representation with all the others. The same query, key, and value projections are applied, and the same kinds of pairwise relationships can still be computed.
现在到了关键问题。假设同样的五个气温观测值被重新排列。数值本身没有改变，只有顺序变了。经过学习到的投影后，我们仍然拥有同样的一组值表示，只是顺序变了。自注意力机制仍然可以比较每个表示与其他所有表示。同样的查询、键和值投影会被应用，同样的成对关系仍然可以被计算出来。
What has disappeared is the temporal structure. Nothing inside $e(27^\circ)$ says that it originally came from Thursday. Nothing inside $e(18^\circ)$ says that it occurred after $e(27^\circ)$. Also, if Wednesday and Friday have the same temperature value, the learned projection will map them to the same embedding vector. Without positional information, the model therefore has no way to distinguish which embedding came from Wednesday and which came from Friday.
消失的是时间结构。$e(27^\circ)$ 内部没有任何信息表明它最初来自周四。$e(18^\circ)$ 内部也没有任何信息表明它发生在 $e(27^\circ)$ 之后。此外，如果周三和周五的气温值相同，学习到的投影会将它们映射到相同的嵌入向量。因此，如果没有位置信息，模型就无法区分哪个嵌入来自周三，哪个来自周五。
This is the key limitation: Self-attention can learn which observations are related, but without an additional positional signal, it has no built-in way to know where those observations occurred in the sequence.
这是核心局限性：自注意力机制可以学习哪些观测值是相关的，但如果没有额外的位置信号，它没有内置的方法来知道这些观测值在序列中发生的位置。