本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Preface

Querying data in Logfire is about to get dramatically faster. Consider >1,000 times faster in some cases. Queries that previously timed out after 30 seconds now complete in under a second. “Seeing much faster <customer_attribute> query times! Feels basically instant now”, says an early access customer. This technical blog post details how we achieved this through dynamic shredding, an optimization for querying semi-structured attributes data. We’ll begin to roll out this feature to all customers in February 2026.

前言

Logfire 的数据查询速度即将大幅提升，在某些情况下甚至能快上 1000 倍以上。原本需要 30 秒才能完成的查询，现在不到一秒即可完成。一位抢先体验的客户表示：“现在的 <customer_attribute> 查询速度快多了！感觉几乎是瞬间完成。”本篇技术博客将详细介绍我们如何通过“动态切片”（dynamic shredding）实现这一目标，这是一种针对半结构化属性数据查询的优化方案。我们计划于 2026 年 2 月向所有客户推出此功能。

Logfire Data Model

OpenTelemetry data (the data that Logfire ingests) is semi-structured data. It contains a mix of: Fixed schema fields (timestamp, span name, etc.) Semi-structured attributes (HTTP status codes, user IDs, regions, etc.) Semi-structured data is sent via attributes (per-span key-value pairs) or resource attributes (otel_resource_attributes in Logfire; these are per-resource key-value pairs where a resource is a service/k8s pod/etc, essentially an instance of your application or service).

Logfire 数据模型

OpenTelemetry 数据（即 Logfire 摄取的数据）属于半结构化数据。它包含以下混合内容：固定模式字段（时间戳、跨度名称等）以及半结构化属性（HTTP 状态码、用户 ID、区域等）。半结构化数据通过属性（每个跨度的键值对）或资源属性（Logfire 中的 otel_resource_attributes；这些是每个资源的键值对，其中资源指服务/k8s pod 等，本质上是应用程序或服务的一个实例）进行发送。

When you write code like:

app = FastAPI()
@app.get('/user/{user_id}')
async def handle_request(user_id: str) -> dict[str, str]:
    with logfire.span(f'handle_request for user {user_id}', user_id=user_id):
        return {"user_id": user_id}

This generates an export along the lines of:

{
  "resource": {
    "attributes": {
      "service.name": "my-web-service",
      "region": "us-west-2"
    }
  },
  "spans": [
    {
      "span_name": "handle_request for user user123",
      "start_timestamp": "2024-01-01T12:00:00Z",
      "end_timestamp": "2024-01-01T12:00:01Z",
      "message": "handle_request for user user123",
      "trace_id": "0194c7b1e5f4a3b2c1d0e6f7a8b9c0d1",
      "span_id": "abcdef1234567890",
      "parent_span_id": "1234567890abcdef",
      "attributes": {
        "user_id": "user123",
        "http.status_code": 200,
      },
    },
    {
      "span_name": "GET /user/{user_id}",
      "start_timestamp": "2024-01-01T12:00:00Z",
      "end_timestamp": "2024-01-01T12:00:01Z",
      "message": "GET /user/user123",
      "trace_id": "0194c7b1e5f4a3b2c1d0e6f7a8b9c0d1",
      "span_id": "1234567890abcdef",
      "parent_span_id": null,
      "attributes": {
        "http.method": "GET",
        "http.response": 200,
      },
    }
  ]
}

当你编写如下代码时： （代码块同上） 这会生成类似以下的导出数据： （JSON 数据块同上）

I’m using JSON here for illustration, although we support various wire formats (OTLP protobuf, JSON, etc.); most data is sent as protobuf over gRPC. There are also some details like timestamps being represented as integers (epoch nanos) rather than ISO strings and an extra level of nesting from scopes that I’ve omitted for clarity. A couple of things to note here: Some fields like start_timestamp, end_timestamp, span_name, and message are fixed-schema fields that exist for every span and are strongly typed. Attributes are semi-structured and can vary widely between spans. Some attributes like http.status_code and http.method are common and have semantic meaning in the OpenTelemetry spec, while others like user_id are application-specific and could be an integer in some places and a string in others.

我在这里使用 JSON 进行说明，尽管我们支持多种传输格式（OTLP protobuf、JSON 等）；大多数数据是通过 gRPC 传输的 protobuf。为了清晰起见，我省略了一些细节，例如时间戳表示为整数（纪元纳秒）而非 ISO 字符串，以及来自作用域的额外嵌套层级。这里有几点需要注意：像 start_timestamp、end_timestamp、span_name 和 message 这样的一些字段是固定模式字段，存在于每个跨度中且具有强类型。属性是半结构化的，在不同跨度之间差异很大。一些属性（如 http.status_code 和 http.method）是通用的，在 OpenTelemetry 规范中具有语义含义，而其他属性（如 user_id）则是特定于应用程序的，在某些地方可能是整数，而在另一些地方可能是字符串。

We ultimately store this data as Parquet files in object storage, with a columnar schema roughly like: start_timestamp: Timestamp(Microsecond, “+00:00”) end_timestamp: Timestamp(Microsecond, “+00:00”) span_name: Utf8View message: Utf8View trace_id: FixedSizeBinary(16) span_id: FixedSizeBinary(8) parent_span_id: FixedSizeBinary(8) attributes: Utf8View (JSON) otel_resource_attributes: Utf8View (JSON)

我们最终将这些数据作为 Parquet 文件存储在对象存储中，其列式模式大致如下： start_timestamp: Timestamp(Microsecond, “+00:00”) end_timestamp: Timestamp(Microsecond, “+00:00”) span_name: Utf8View message: Utf8View trace_id: FixedSizeBinary(16) span_id: FixedSizeBinary(8) parent_span_id: FixedSizeBinary(8) attributes: Utf8View (JSON) otel_resource_attributes: Utf8View (JSON)

Some background on columnar storage, Parquet and zone map pruning is beneficial to understand the rest of this blog post. I recommend reading MotherDuck’s excellent overview of the topic.

了解有关列式存储、Parquet 和区域映射修剪（zone map pruning）的一些背景知识，有助于理解本篇博客的后续内容。我建议阅读 MotherDuck 关于该主题的优秀概述。

How we started: fast JSON parsing

Pydantic started as a library to validate and parse JSON data efficiently. When we built Logfire, we initially stored all attributes as JSON blobs in a single column and relied on Jiter (one of the fastest JSON parsers out there in any language) to rip through the JSON data as quickly as possible. This is the layout detailed above (attributes: Utf8View (JSON)). This approach was simple but made a fundamental mistake: we were optimizing to do work faster, but we couldn’t really avoid doing the work. Compression is also suboptimal since a single varying attribute makes Parquet level dictionary compression ineffective and even zstd struggles with highly variable data, all the JSON syntax overhead, etc. The biggest problem really ends up being extra IO: since most of our data is stored on object storage, extra IO from downloading large JSON blobs, and especially the latency of starting these downloads, ends up dominating query runtimes much more so than any CPU work.

我们最初的做法：快速 JSON 解析

Pydantic 最初是一个用于高效验证和解析 JSON 数据的库。在构建 Logfire 时，我们最初将所有属性作为 JSON 二进制大对象（blob）存储在单个列中，并依赖 Jiter（目前各语言中最快的 JSON 解析器之一）尽可能快地处理 JSON 数据。这就是上述布局（attributes: Utf8View (JSON)）。这种方法很简单，但犯了一个根本性的错误：我们是在优化工作速度，却无法真正避免这些工作。压缩效果也不理想，因为单个变化的属性会使 Parquet 级别的字典压缩失效，即使是 zstd 在处理高度可变的数据、所有 JSON 语法开销等问题时也很吃力。最大的问题实际上是额外的 IO：由于我们的大部分数据存储在对象存储上，下载大型 JSON 二进制大对象带来的额外 IO，尤其是启动这些下载的延迟，最终对查询运行时间的影响远超任何 CPU 工作。

Shredding columns to save IO and compute

To make queries fast, we need to avoid unnecessary work. For normal columns DataFusion / Parquet avoid unnecessary work by using columnar storage (allowing reading only the columns you need) and zone maps / min-max statistics (allowing skipping entire row groups or pages that don’t match your filter predicates). So when you write a query like: SELECT count(*) FROM records WHERE duration < 0; The system can efficiently read just the duration column and skip row groups where the max duration is < 0 based on the stored statistics. In this case this is a valid query, duration is a f64 so in theory it could have values <0. In practice though duration is always >= 0 so this query will be very fast: based on statistics we know no rows match and we can return 0 immediately.

通过切片列来节省 IO 和计算资源

为了加快查询速度，我们需要避免不必要的工作。对于普通列，DataFusion / Parquet 通过使用列式存储（允许仅读取所需的列）和区域映射/最小-最大统计信息（允许跳过不符合过滤条件的整个行组或页面）来避免不必要的工作。因此，当你编写如下查询时： SELECT count(*) FROM records WHERE duration < 0; 系统可以高效地仅读取 duration 列，并根据存储的统计信息跳过最大持续时间小于 0 的行组。在这种情况下，这是一个有效的查询，duration 是 f64 类型，理论上可以有小于 0 的值。但在实践中，持续时间总是大于等于 0，因此该查询会非常快：根据统计信息，我们知道没有行匹配，因此可以立即返回 0。

So how can we enable this for semi-structured data like attributes stored as JSON? Our initial solution was static shredding: we manually picked a small set of commonly queried attributes (like http.status_code, http.method, etc.) and extracted these into their own strongly typed columns during ingestion. So when we receive: attributes {‘http.response.status_code’: 200, ‘url.full’: ‘https:‎//example.com?foo=bar’} {‘my_llm_response’: ‘large text’} {‘user_id’: ‘123’} Assuming our hardcoded shredding configuration extracts http.response.status_code, url

那么，我们如何为存储为 JSON 的半结构化数据（如属性）实现这一点呢？我们最初的解决方案是静态切片：我们在摄取过程中手动挑选了一小部分常用查询的属性（如 http.status_code、http.method 等），并将它们提取到各自的强类型列中。因此，当我们收到： attributes {‘http.response.status_code’: 200, ‘url.full’: ‘https:‎//example.com?foo=bar’} {‘my_llm_response’: ‘large text’} {‘user_id’: ‘123’} 假设我们硬编码的切片配置提取了 http.response.status_code, url