Architecting memory and storage in the AI era

AI 时代的内存与存储架构设计

The era of AI inference has arrived. Imagine a healthcare system analyzing millions of data points in real time to accelerate life-saving medical research, or an intelligent assistant instantly resolving thousands of complex customer needs at once. These real-world breakthroughs rely on advanced infrastructure acting as the engine of continuous intelligence, powering real-time services while also supporting an increasingly intelligent edge of IoT and consumer devices.

AI 推理时代已经到来。想象一下，一个医疗系统能够实时分析数百万个数据点，以加速挽救生命的医学研究；或者一个智能助手能够瞬间同时解决数千个复杂的客户需求。这些现实世界的突破依赖于先进的基础设施，它们充当着持续智能的引擎，在为实时服务提供动力的同时，也支持着物联网和消费设备日益智能化的边缘。

However, in this inference-driven landscape, every delay, bottleneck, or wasted watt directly affects human outcomes and operating costs. This shift changes what infrastructure must deliver. Performance, latency, memory bandwidth, storage throughput, and networking cannot be optimized in silos. Inference workloads are continuous, geographically distributed, and highly sensitive to response time, requiring systems designed for scale, resilience, and efficiency from the start.

然而，在这个由推理驱动的环境中，每一次延迟、瓶颈或能源浪费都会直接影响人类的成果和运营成本。这种转变改变了基础设施必须交付的内容。性能、延迟、内存带宽、存储吞吐量和网络连接不能再孤立地进行优化。推理工作负载是持续的、地理分布式的，并且对响应时间高度敏感，这要求系统从设计之初就具备可扩展性、弹性和高效性。

“We tend to think of AI as a single workload, and it’s not. It’s thousands, it’s millions, it’s billions of different workloads,” says Jim McGregor, founder and principal analyst, Tirias Research. AI inference changes the optimization problem from one of raw compute to coordinated infrastructure—memory, storage, and networking. For business leaders, the priority is clear: AI infrastructure decisions must balance cost, flexibility, and future readiness. The winners will be organizations that improve performance per watt, reduce environmental footprint, and remove memory and storage bottlenecks before they limit growth.

Tirias Research 创始人兼首席分析师 Jim McGregor 表示：“我们倾向于将 AI 视为单一工作负载，但事实并非如此。它是成千上万、数以亿计的不同工作负载。”AI 推理将优化问题从单纯的原始计算能力转变为协同的基础设施——即内存、存储和网络的协调。对于商业领袖来说，优先级很明确：AI 基础设施决策必须平衡成本、灵活性和未来就绪性。最终的赢家将是那些能够提高每瓦性能、减少环境足迹，并在内存和存储瓶颈限制增长之前将其消除的组织。

AI inference requires a new architectural approach

AI 推理需要全新的架构方法

Systems for AI need to be rearchitected because shoehorning modern AI systems into legacy infrastructure limits AI’s transformative potential. Purpose-built architectures are essential to realize the true value of AI, from accelerating scientific discovery to creating truly autonomous digital agents. Traditional enterprise IT has been able to rely on relatively stable infrastructure assumptions, but inference and agentic AI introduce new demands around latency, data movement, scalability, and utilization that make architecture choices far more consequential.

AI 系统需要重新架构，因为将现代 AI 系统强行塞入传统基础设施会限制 AI 的变革潜力。专用架构对于实现 AI 的真正价值至关重要，从加速科学发现到创建真正的自主数字代理。传统的企业 IT 过去能够依赖相对稳定的基础设施假设，但推理和代理式 AI（Agentic AI）在延迟、数据移动、可扩展性和利用率方面提出了新的要求，这使得架构选择的影响变得更加深远。

“Data centers must now support continuous, distributed, and increasingly real-time AI services—none of which are a single workload,” says McGregor. “They all require different requirements from a system-level perspective.” To support real-time AI, enterprises can no longer view memory and storage merely as supporting hardware, but at the heart of the system. Organizations need to architect a data pipeline that can rapidly ingest, clean, transform, store, move, and deliver data.

McGregor 说：“数据中心现在必须支持持续的、分布式的、且日益实时化的 AI 服务——这些服务都不是单一的工作负载。从系统层面来看，它们都有不同的需求。”为了支持实时 AI，企业不能再仅仅将内存和存储视为辅助硬件，而应将其视为系统的核心。组织需要构建一个能够快速摄取、清洗、转换、存储、移动和交付数据的数据流水线。

Inference workloads place sustained pressure on infrastructure in ways that look very different from earlier training-centric deployments, demanding continuous data retrieval and caching that traditional applications never required. Accordingly, performance by itself is no longer the sole benchmark that matters. Enterprises increasingly must balance performance with efficiency, cost, and scalability, especially as they try to support different AI services without overbuilding infrastructure for peak conditions.

推理工作负载对基础设施施加的持续压力，与早期以训练为中心的工作负载截然不同，它需要传统应用程序从未要求的持续数据检索和缓存。因此，性能本身不再是唯一的衡量标准。企业越来越需要在性能与效率、成本和可扩展性之间取得平衡，特别是在他们试图支持不同的 AI 服务，同时又不想为了峰值条件而过度建设基础设施时。

“You have to optimize the entire network, and that includes memory and storage, around the types of workloads you plan on running,” says McGregor. “You have to really have a detailed understanding of what those workloads are going to be.” Any AI infrastructure strategy must start with workload awareness. Inference, agentic AI, and other emerging AI use cases require organizations to treat the data center as an integrated system.

McGregor 表示：“你必须围绕你计划运行的工作负载类型来优化整个网络，这包括内存和存储。你必须真正详细了解这些工作负载将是什么。”任何 AI 基础设施战略都必须从工作负载感知开始。推理、代理式 AI 和其他新兴的 AI 用例要求组织将数据中心视为一个集成系统。

Data movement is the new bottleneck and an opportunity for competitive advantage

数据移动是新的瓶颈，也是竞争优势的机遇

As enterprises deploy advanced inference and agentic systems, the sheer volume of data being queried in real time has made data movement the most pressing constraint. Modern AI techniques like retrieval-augmented generation (RAG) require systems to constantly scan massive databases to generate accurate responses. This requires immense computing power, but more importantly, it requires immediate access to data.

随着企业部署先进的推理和代理系统，实时查询的数据量之大，使得数据移动成为最紧迫的制约因素。检索增强生成（RAG）等现代 AI 技术要求系统不断扫描海量数据库以生成准确的响应。这需要巨大的计算能力，但更重要的是，它需要对数据的即时访问。

McGregor says the focus shift to how efficiently data can be moved, cached, and delivered across the broader architecture elevates memory and storage from background infrastructure to strategic assets. “The biggest thing we’re doing right now is moving data from one place to another and making sure that we can use it effectively.” Because AI is not a single workload category, simply buying the fastest processors is insufficient. Inference depends heavily on memory bandwidth, caching, storage proximity, and the ability to retrieve relevant information quickly and consistently.

McGregor 表示，关注点转向数据如何在更广泛的架构中高效地移动、缓存和交付，将内存和存储从后台基础设施提升为战略资产。“我们现在做的最重要的事情就是将数据从一个地方移动到另一个地方，并确保我们能有效地使用它。”由于 AI 不是单一的工作负载类别，仅仅购买最快的处理器是不够的。推理在很大程度上依赖于内存带宽、缓存、存储邻近性以及快速且一致地检索相关信息的能力。

Understanding where each resource belongs in the stack and how those layers interact under real operating conditions has become a business imperative. The most effective AI infrastructure looks less like a collection of best-in-class parts and more like a balanced system of compute, memory, storage, and networking, McGregor says, because bottlenecks tend to migrate from one layer to the next. “You have to architect all four together to be efficient, and that’s the challenge.”

了解每个资源在堆栈中的位置，以及这些层在实际运行条件下如何交互，已成为一项商业要务。McGregor 说，最有效的 AI 基础设施看起来不像是一堆顶级部件的集合，而更像是一个计算、内存、存储和网络平衡的系统，因为瓶颈往往会从一层迁移到另一层。“你必须将这四者结合起来进行架构设计才能实现高效，这就是挑战所在。”

The interdependence of data-plane design and network bandwidth means AI infrastructure planning has become a business decision just as much as an engineering one: latency is now inseparable from value. In robotics, financial services, healthcare, and customer-facing AI systems, delays are not merely technical imperfections; they can undermine safety, responsiveness, or trust. AI infrastructure performance becomes a matter of reputation management. The organizations that gain the most from AI may not be those with the largest clusters, but those with the clearest understanding of how to align every infrastructure element to effectively execute AI workloads.

数据平面设计与网络带宽之间的相互依赖性意味着，AI 基础设施规划不仅是工程决策，更是商业决策：延迟现在与价值密不可分。在机器人、金融服务、医疗保健和面向客户的 AI 系统中，延迟不仅仅是技术上的瑕疵；它们可能会损害安全性、响应能力或信任度。AI 基础设施的性能已成为声誉管理的问题。从 AI 中获益最多的组织可能不是那些拥有最大集群的组织，而是那些最清楚如何协调每个基础设施元素以有效执行 AI 工作负载的组织。