Akashic: A Low-Overhead LLM Inference Service with MemAttention

Akashic：一种基于 MemAttention 的低开销大模型推理服务

Abstract: Recent LLM-based agent systems continuously accumulate context across multi-turn interactions, tool invocations, and cross-session workflows. Replaying the full history for every request quickly becomes impractical: long contexts increase prefill cost, may exceed context limits, and often bury task-relevant evidence in irrelevant content, degrading both serving efficiency and output quality.

摘要： 近期基于大语言模型（LLM）的智能体系统在多轮交互、工具调用和跨会话工作流中会持续积累上下文。针对每个请求重放完整历史记录很快变得不切实际：过长的上下文会增加预填充（prefill）成本，可能超出上下文限制，且往往会将任务相关的证据淹没在无关内容中，从而降低服务效率和输出质量。

We propose Akashic, a low-overhead memory system built around MemAttention, which organizes context into bounded chunks and models semantic relationships across chunks, preserving cross-chunk evidence without repeatedly rewriting the full history.

我们提出了 Akashic，这是一个围绕 MemAttention 构建的低开销内存系统。它将上下文组织成有界的块（chunks），并对块之间的语义关系进行建模，从而在无需重复重写完整历史记录的情况下，保留跨块的证据。

Akashic further applies hardware-software co-designed memory placement to co-locate likely co-retrieved chunks, reducing retrieval fragmentation and I/O overhead.

Akashic 进一步应用了软硬件协同设计的内存布局策略，将可能被同时检索的块放置在一起，从而减少了检索碎片并降低了 I/O 开销。

Across four representative workloads and three model sizes, Akashic improves task accuracy by up to 10.2 points, throughput by up to 1.21x, and sustainable request rate by up to 1.88x over strong prior memory baselines.

在四个代表性工作负载和三种模型规模的测试中，与现有的强力内存基准相比，Akashic 将任务准确率提高了 10.2 个百分点，吞吐量提升了 1.21 倍，可持续请求速率提升了 1.88 倍。