OneBRC: Processing a Billion Rows

The One Billion Row Challenge in Python - from a naive loop to Polars and DuckDB - and what a 100x speedup actually comes from.
Author

Benedict Thekkel

Published

September 15, 2025

The One Billion Row Challenge is simple to state: read 1,000,000,000 lines of station;temperature, and print the min / mean / max per station. Simple to state, brutal on a naive implementation.

I worked through it in the Other repo (p_50_OneBRC), leaning on Polars and DuckDB. The lesson: the win isn’t a clever algorithm - it’s columnar, multi-threaded, out-of-core engines doing the boring work well.

The approaches

Approach Idea Runtime Peak mem LOC Notebook
Naive Python line loop + dict ~390 s low ~20 OneBRC
pandas read_csv + groupby ~130 s very high (OOM risk) ~5 OneBRC
Polars (lazy) columnar, multi-thread, streaming ~12 s bounded ~6 Polars
DuckDB SQL over the file, out-of-core ~9 s bounded ~3 DuckDB

Representative timings on a consumer multi-core laptop; your mileage varies with cores, disk, and SSD vs NVMe.

Runtime, by approach

The naive loop is the baseline; each step down is a different execution model, not a smarter formula.

Speedup vs the naive baseline

Where the time actually goes

Three things explain almost all of the gap:

flowchart LR
  A[Naive Python] -->|columnar layout| B[Vectorised ops]
  B -->|all cores| C[Multi-threaded]
  C -->|stream, don't load| D[Out-of-core]
  D --> E[~40x faster]

  • Columnar + vectorised - operate on typed columns, not Python objects per cell.
  • Multi-threaded - Polars and DuckDB saturate every core; naive Python uses one.
  • Streaming / out-of-core - never materialise a billion rows in RAM; pandas tries to, and pays for it (or OOMs).

Peak memory

The other half of the story: pandas wins on lines-of-code but loses on memory. Polars and DuckDB stay bounded because they stream.

Takeaway

For big tabular work, reach for a columnar engine first and only optimise the algorithm if it’s still slow. Same query, 40x less wall-clock, bounded memory, fewer lines of code.

Back to top