Python JSON Parsing Reaches 66 GB/s With fastsimdjson and Free Threading

What was measured

Daniel Lemire used eight hours of GitHub public repository events from March 13, 2024, collected by GH Archive as NDJSON. The decompressed sample was 7.6 GB and contained 2.2 million events, averaging about 3.4 kB each. The task was to count event types and, for pull request events with action opened, count the base repository language. For the eight hours, Lemire counted 65,269 opened pull requests. The largest groups were TypeScript with 11,565, JavaScript with 7,778, Python with 7,057 and Java with 4,668. About 5,857 opened pull requests targeted repositories without a language.

How the parser stayed lean

The work used fastsimdjson, the Python binding for simdjson, with Python 3.14.6 on a server with two Intel Xeon Gold 6548N processors and 64 cores. In free threaded Python, the GIL is gone and several threads can run Python code at the same time. The fastsimdjson map_many call distributes lines across a requested thread count while sharing the input. Each thread keeps its own Counter pair, appends it to a shared list only on its first event, then updates that pair without synchronizing with other threads. The main thread merges the per thread pairs after map_many returns. With lazy = True, each callback receives a lazy view instead of a dictionary. The JSON is still parsed, but Python objects are created only for the fields the code accesses. Lemire notes that the lazy approach was first developed by Tyler Kennedy in the pysimdjson binding.

Benchmark results

The data was decompressed in memory before timing. Lemire reports the best of seven runs, with one run for the single thread parsers and for the standard library on 64 threads. On one core, the standard library json parser ran at 0.18 GB/s, orjson at 0.42 GB/s, fastsimdjson map_many at 0.63 GB/s, msgspec typed structures at 1.29 GB/s and fastsimdjson map_many with lazy views at 2.13 GB/s. The standard library needed 42 seconds for the sample, while fastsimdjson with lazy views needed 3.6 seconds. Using 64 threads in one process, the standard library reached 3.17 GB/s, msgspec typed structures reached 24.6 GB/s, fastsimdjson map_many reached 29.8 GB/s and the lazy variant reached 65.7 GB/s. That is about 0.12 seconds for the 7.6 GB sample, around 66 GB/s, or 31 times the single core result. The lazy fastsimdjson result was also faster than msgspec with 64 threads. Lemire concluded that Python can parse JSON at tens of gigabytes per second, with fastsimdjson 0.6.0 available on PyPI.

Sources: Lobsters