Parsing JSON at tens of gigabytes per second… in Python
Parsing JSON at tens of gigabytes per second… in Python
12 October 2026 · 5 min Python used to have limited parallelism, due to its global interpreter lock (GIL). Only one thread could run Python code at a time. Python 3.14 comes in a free-threaded version (python3.14t), without the global interpreter lock. Several threads can now run Python code at the same time.
2026年10月12日 · 5分钟阅读。由于全局解释器锁（GIL）的存在，Python过去在并行处理方面受到限制，同一时间只能有一个线程运行Python代码。Python 3.14引入了无GIL的自由线程版本（python3.14t），现在多个线程可以同时运行Python代码。
Let us pick a realistic problem. GitHub publishes everything that happens on its public repositories: pushes, issues, pull requests, stars, and so forth. The GH Archive project collects these events as JSON, one event per line (NDJSON), one file per hour. I took eight hours of March 13, 2024: 7.6 GB of JSON once decompressed, 2.2 million events. I want to know which programming languages get the most new pull requests. So I count the events by type, and for each pull request that was opened, I look up the language of the repository. Each event is a large JSON object, about 3.4 kB on average: I need only a few of its fields.
让我们选择一个实际的问题。GitHub会发布其公共存储库中发生的所有事件：推送、议题、拉取请求、星标等。GH Archive项目将这些事件收集为每行一个JSON对象（NDJSON）的格式，每小时一个文件。我选取了2024年3月13日的八小时数据：解压后为7.6 GB的JSON，包含220万个事件。我想知道哪些编程语言获得的新拉取请求最多。因此，我按类型统计事件，并针对每个打开的拉取请求，查找存储库的语言。每个事件都是一个大型JSON对象，平均约3.4 kB，但我只需要其中的几个字段。
The answer, for these eight hours: 65,269 pull requests were opened, mostly toward TypeScript (11,565), JavaScript (7,778), Python (7,057) and Java (4,668) repositories. About 5,857 targeted repositories without a language. With fastsimdjson, our Python binding for simdjson, the code looks as follows. The function map_many calls my function on each event, using as many threads as I ask for.
这八小时的统计结果显示：共打开了65,269个拉取请求，主要针对TypeScript（11,565）、JavaScript（7,778）、Python（7,057）和Java（4,668）存储库。约有5,857个请求针对的是没有语言标记的存储库。使用我们为simdjson编写的Python绑定fastsimdjson，代码如下所示。函数map_many会在每个事件上调用我的函数，并使用我指定的线程数。
import collections, threading
import fastsimdjson
local = threading.local()
parts = []
def one_event(e):
acc = getattr(local, "acc", None)
if acc is None:
# first event seen by this thread
acc = local.acc = (collections.Counter(), collections.Counter())
parts.append(acc)
t = e["type"]
acc[0][t] += 1
if t == "PullRequestEvent":
p = e["payload"]
if p["action"] == "opened":
acc[1][p["pull_request"]["base"]["repo"]["language"]] += 1
data = open("events.json", "rb").read()
fastsimdjson.map_many(one_event, data, format="lines", threads=64, lazy=True)
types, languages = collections.Counter(), collections.Counter()
for t, lang in parts:
types.update(t)
languages.update(lang)Each thread keeps its own counters. The first event a thread sees creates the pair and appends it to parts. Later events on that thread update the same pair. Nothing in the counters is shared, so the threads do not wait for each other. map_many returns only once every event has been counted. The main thread then walks parts, one pair per thread, and Counter.update adds the counts. A few dozen event types and about a hundred languages: the merge is tiny next to the parsing.
每个线程都维护自己的计数器。线程看到的第一个事件会创建计数对并将其添加到parts列表中。该线程后续的事件会更新同一计数对。计数器中没有任何共享内容，因此线程之间互不等待。map_many仅在所有事件统计完成后才返回。随后，主线程遍历parts列表（每个线程对应一对），并使用Counter.update累加计数。由于只有几十种事件类型和约一百种语言，合并操作相对于解析过程来说微不足道。
With lazy=True, my function receives a lazy view of the event instead of a dictionary: the event is parsed, but Python objects are created only for the values I access. The lazy approach was first developed by Tyler Kennedy in another simdjson binding, pysimdjson. How does it compare? I tried the usual suspects: the standard library (json), orjson and msgspec. msgspec deserves special mention: it can decode JSON into typed structures that declare only the fields you need, skipping the rest. It is a lot like our lazy views, except that you must declare a schema beforehand. I gave msgspec this advantage.
设置lazy=True后，我的函数接收的是事件的惰性视图而非字典：事件会被解析，但只有在我访问具体值时才会创建Python对象。这种惰性方法最初由Tyler Kennedy在另一个simdjson绑定pysimdjson中开发。它的表现如何？我测试了常见的库：标准库（json）、orjson和msgspec。msgspec值得特别提及：它可以将JSON解码为仅声明所需字段的类型化结构，从而跳过其余部分。这与我们的惰性视图非常相似，区别在于必须预先声明模式。我给予了msgspec这一优势。
In each case, I verify that we get exactly the same counts. I use a server with two Intel Xeon Gold 6548N processors (64 cores), Python 3.14.6, and the current versions of the libraries (fastsimdjson 0.6.0, orjson 3.13.0, msgspec 0.22.0). The data is decompressed in memory before the timer starts. I report the best of seven runs, and one run for the slow methods: the single-thread parsers, and the standard library on 64 threads. Let us first use a single core.
在每种情况下，我都验证了统计结果完全一致。我使用了一台配备两颗Intel Xeon Gold 6548N处理器（共64核）的服务器，运行Python 3.14.6以及当前版本的库（fastsimdjson 0.6.0, orjson 3.13.0, msgspec 0.22.0）。数据在计时开始前已在内存中解压。我报告了七次运行中的最佳结果，对于较慢的方法（单线程解析器以及在64个线程上运行的标准库），则仅报告一次运行结果。首先让我们使用单核。
| method | GB/s |
|---|---|
| json (standard library) | 0.18 |
| orjson | 0.42 |
| fastsimdjson map_many | 0.63 |
| msgspec (typed structures) | 1.29 |
| fastsimdjson map_many (lazy) | 2.13 |
The standard library needs 42 seconds to go through the eight hours. With fastsimdjson and lazy views, it takes 3.6 seconds. Notice how the methods that materialize everything (json, orjson and fastsimdjson without lazy views) are slow: most of the time goes into creating Python objects that we discard right away.
标准库需要42秒才能处理完这八小时的数据。使用fastsimdjson和惰性视图，仅需3.6秒。请注意，那些将所有内容具体化（materialize）的方法（json、orjson以及不使用惰性视图的fastsimdjson）速度较慢：大部分时间都花在了创建随后即被丢弃的Python对象上。
Now let us use all 64 cores, still in one process. On free-threaded Python, I use a thread pool for json and msgspec, and map_many(…, threads=64) for fastsimdjson. The threads share the input.
现在让我们使用全部64个核心，依然在单个进程中运行。在自由线程Python上，我为json和msgspec使用了线程池，并为fastsimdjson使用了map_many(…, threads=64)。这些线程共享输入数据。
| method | GB/s |
|---|---|
| json (standard library), 64 threads | 3.17 |
| msgspec (typed structures), 64 threads | 24.6 |
| fastsimdjson map_many, 64 threads | 29.8 |
| fastsimdjson map_many (lazy), 64 threads | 65.7 |
We go through 7.6 GB of JSON in about 0.12 seconds, at 66 GB/s, in one Python process. That is 31 times faster than with one core. It is faster than msgspec using 64 threads (25 GB/s). So, can you parse JSON at tens of gigabytes per second in Python? Yes. fastsimdjson 0.6.0 is available on PyPI. My benchmark is available.
我们在单个Python进程中，以约66 GB/s的速度在0.12秒内处理了7.6 GB的JSON数据。这比单核速度快了31倍。它也比使用64个线程的msgspec（25 GB/s）更快。所以，在Python中可以以每秒数十GB的速度解析JSON吗？答案是肯定的。fastsimdjson 0.6.0已在PyPI上发布。我的基准测试代码也已公开。