Linux 7.3 improves performance when running out of vRAM
Linux 7.3 improves performance when running out of vRAM
Linux 7.3 改进了显存（vRAM）不足时的性能表现
Earlier this year, I blogged about work I did to improve VRAM management for games. Now, after many months of floating around in mailing lists, the kernel patches are finally merged upstream and queued for Linux 7.3! Hooray! 今年早些时候，我写了一篇博客，介绍了我在改进游戏显存（VRAM）管理方面所做的工作。现在，经过数月在邮件列表中的讨论，这些内核补丁终于被合并到上游，并已进入 Linux 7.3 的队列！太棒了！
To celebrate, let’s look a bit deeper at one sentence I wrote in my previous post: [Games] should perform much more stable - as long as the game itself doesn’t use more VRAM than you actually have. So, one may ask: What if they do, in fact, use more VRAM than you actually have? 为了庆祝这一进展，让我们深入探讨一下我在上一篇文章中写的一句话：“只要游戏本身使用的显存不超过你实际拥有的显存，（游戏）的运行就应该更加稳定。” 那么，有人可能会问：如果游戏实际使用的显存确实超过了你拥有的显存，会发生什么呢？
Typical expectations for this seem to be that once this happens you’re pretty much screwed. Games will start crashing left and right, performance plummets to unplayable levels, a good gaming experience becomes impossible. But is that really just an unavoidable fact of life? What really makes running out of VRAM suck so hard? And, most importantly: How can we make it suck as little as possible? 人们通常认为，一旦这种情况发生，你就基本“完蛋”了。游戏会频繁崩溃，性能跌至无法游玩的水平，良好的游戏体验将变得不可能。但事实真的如此不可避免吗？究竟是什么导致显存不足时体验如此糟糕？最重要的是：我们该如何尽可能减轻这种负面影响？
Setting expectations
设定预期
In theory, running out of VRAM should exclusively be a performance issue, not a stability one. Support for overcommitting VRAM has existed for as long as GPU drivers have: If the driver overcommits VRAM, you are generally allowed to request as much VRAM as you’d like, and you’ll get as much as the kernel driver decides it can fit into the physical memory that exists on GPU. 理论上，显存不足应该仅仅是一个性能问题，而不是稳定性问题。显存超额分配（Overcommitting）的支持与 GPU 驱动程序一样古老：如果驱动程序支持超额分配，你通常可以请求任意数量的显存，而内核驱动程序会根据 GPU 实际拥有的物理内存容量，决定分配给你多少。
On the performance side, the big-picture reason for bad performance when you run out of VRAM is fairly simple. As soon as the game requests more VRAM than is physically present, some of the game’s memory will have to be moved/evicted to CPU RAM instead. For the GPU, accessing CPU RAM is much slower than VRAM: Not only is CPU RAM slower than a dedicated GPU’s VRAM in general, all memory accesses also have to go over the PCI bus. 从性能角度来看，显存不足导致性能下降的原因很简单。一旦游戏请求的显存超过了物理容量，部分游戏内存就必须被移动或置换（evicted）到 CPU 内存中。对于 GPU 而言，访问 CPU 内存比访问显存慢得多：不仅 CPU 内存本身通常比专用显存慢，而且所有内存访问都必须经过 PCI 总线。
The PCI bus adds latency and is typically also the limiting factor in bandwidth when fetching from CPU memory. Due to PCI speed limitations, there are some truly unavoidable performance constraints when overcommitting VRAM. Assuming the GPU is hooked up via a PCIe 4.0x16 connection, you get a little less than 32GiB/s of bandwidth. Each millisecond, that PCIe bus can transfer ~32.2MiB of data. PCI 总线增加了延迟，并且通常是读取 CPU 内存时带宽的瓶颈。由于 PCI 速度的限制，在显存超额分配时，存在一些真正无法避免的性能约束。假设 GPU 通过 PCIe 4.0x16 连接，你将获得略低于 32GiB/s 的带宽。每一毫秒，PCIe 总线可以传输约 32.2MiB 的数据。
For a minimum framerate of 30 frames per second (33.3ms per frame), the absolute maximum amount of data the GPU is able to access is ~1,075.5MiB, a tiny bit over 1GiB of data. In other words, if so much memory gets evicted that the GPU needs to fetch more than 1GiB from evicted memory in one single frame, it is simply impossible to still hit 30 FPS. 对于 30 帧每秒（每帧 33.3 毫秒）的最低帧率，GPU 每帧能够访问的绝对最大数据量约为 1,075.5MiB，即略高于 1GiB。换句话说，如果被置换到 CPU 内存的数据量过大，导致 GPU 在单帧内需要从这些内存中读取超过 1GiB 的数据，那么维持 30 FPS 是绝对不可能的。
Not all memory is equal
并非所有内存都生而平等
At the same time, just reading a little bit of CPU memory on the GPU is not immediately a death sentence for performance. In fact, GPU drivers sometimes decide to let things like command buffer data and related allocations live in CPU RAM even when there’s plenty of VRAM available! Whenever the GPU executes these commands, it has to access CPU memory, and yet in these cases everything runs completely fine. 与此同时，GPU 读取少量 CPU 内存并不意味着性能会立即崩溃。事实上，即使在显存充足的情况下，GPU 驱动程序有时也会决定将命令缓冲区数据等分配存放在 CPU 内存中！每当 GPU 执行这些命令时，它都必须访问 CPU 内存，但在这些情况下，一切运行都非常正常。
So what makes these accesses different - why are they fine and yet running out of VRAM seems catastrophic? One thing that influences the calculus significantly is caching. Since the access latency in case of a cache hit is the same regardless of whether the cached memory lives on CPU or GPU, the high initial cost of fetching over the PCI bus can be amortized by cache hits (to some extent). 那么，这些访问有什么不同？为什么它们没问题，而显存不足却显得如此灾难性？影响这一计算的一个重要因素是缓存。由于缓存命中时的访问延迟与缓存数据位于 CPU 还是 GPU 无关，因此通过 PCI 总线获取数据的初始高昂成本可以在一定程度上被缓存命中抵消。
We can estimate latency differences between fetching CPU RAM and VRAM by writing microbenchmarks that measure access latency for different buffer sizes (using an adversarial access pattern to minimize cache hitrates as far as possible). The result you get may look something like this (captured on RDNA3): 我们可以通过编写微基准测试来估算读取 CPU 内存和显存之间的延迟差异，该测试测量不同缓冲区大小的访问延迟（使用对抗性访问模式以尽可能降低缓存命中率）。你得到的结果可能如下所示（在 RDNA3 上捕获）：
As expected, if the buffer fits into L2 (or any higher-level cache), access latencies are exactly the same for memory backed by CPU RAM and memory backed by VRAM, because the data gets fetched directly from cache in either case. At a size of 6MB (the L2 cache size on RDNA3), CPU memory latencies go up to about 2400 cycles per access, while device memory latencies stay within the same rough ballpark. 正如预期的那样，如果缓冲区能放入 L2（或任何更高级别的缓存），那么由 CPU 内存支持的内存和由显存支持的内存，其访问延迟是完全相同的，因为数据在两种情况下都是直接从缓存中获取的。在 6MB 大小（RDNA3 的 L2 缓存大小）时，CPU 内存的访问延迟上升到每次访问约 2400 个周期，而设备内存的延迟则保持在大致相同的范围内。
Note that VRAM accesses also go through the Infinity Cache, but CPU memory accesses do not (they hit PCIe directly on an L2 miss). I suspect this is because the Infinity Cache sits directly on top of VRAM, so any access that doesn’t hit VRAM also doesn’t reach the Infinity Cache. Obviously, memory doesn’t start off with being cached anywhere, so the first access will still have considerably higher latency. 请注意，显存访问也会经过 Infinity Cache，但 CPU 内存访问不会（它们在 L2 未命中时直接访问 PCIe）。我怀疑这是因为 Infinity Cache 直接位于显存之上，因此任何未命中显存的访问也无法到达 Infinity Cache。显然，内存最初并不会被缓存在任何地方，因此第一次访问的延迟仍然会高得多。
Also, losing the Infinity Cache definitely hurts as well: PCIe fetches seem to have somewhere around 7.3x as much latency than an Infinity Cache hit, and around 4.6x as much latency as a fetch from VRAM. This increased latency needs really high cache hitrates to fully amortize the cost of going over PCIe. That means there is only a small set of use cases where using CPU memory has such minuscule slowdowns that you’d actively decide to use it in favor of VRAM when you have the choice. 此外，失去 Infinity Cache 的支持确实会造成影响：PCIe 读取的延迟大约是 Infinity Cache 命中的 7.3 倍，是显存读取的 4.6 倍左右。这种增加的延迟需要极高的缓存命中率才能完全抵消通过 PCIe 传输的成本。这意味着只有在极少数用例中，使用 CPU 内存带来的减速微乎其微，以至于在有选择的情况下，你会主动选择它而不是显存。
When you’re evicting memory from VRAM, there will almost unavoidably be at least some degree of slower performance. Still, even though slowdown is unavoidable, there is going to be memory where eviction matters more and memory where eviction has a lesser effect on overall perf. Memory that is accessed in very cache-friendly ways is not affected by the slowdown of CPU RAM as much. 当你从显存中置换内存时，几乎不可避免地会出现一定程度的性能下降。尽管减速不可避免，但有些内存的置换影响更大，而有些内存的置换对整体性能的影响较小。以缓存友好方式访问的内存受 CPU 内存减速的影响较小。
If the access patterns aren’t cache-friendly but the memory isn’t accessed very often, things may also still be fine since the GPU only rarely needs to actually fetch data from CPU RAM. There might be many memory allocations where the GPU will only access a small part of the total allocation size, and never even read the rest. If these allocations were to be evicted, you might evict multiple GiBs of data, but still remain well below the 1GiB hard limit of data that is actually accessed per frame. 如果访问模式对缓存不友好，但内存访问频率不高，情况也可能尚可，因为 GPU 很少需要真正从 CPU 内存中获取数据。可能存在许多内存分配，GPU 只会访问总分配大小的一小部分，而从不读取其余部分。如果这些分配被置换，你可能会置换出数 GiB 的数据，但每帧实际访问的数据量仍远低于 1GiB 的硬性限制。
All of these variables make it surprisingly hard to predict how performance actually pans out in practice when memory is being evicted. But in short: Depending on how much the evicted memory gets accessed… 所有这些变量使得在内存被置换时，很难预测实际的性能表现。但简而言之：取决于被置换内存的访问频率……