Polars 2.0 Ships With Default Streaming Engine, Out-of-Core Support and First-Class SQL

Streaming and Out-of-Core by Default

Calling collect on a LazyFrame now defaults to the streaming engine, which the team says brings massive memory and performance improvements on most queries. Because the streaming engine does not guarantee row-order by default for operations such as join, group_by and unpivot, the change required a major version bump. Users who need observable row order can opt in with maintain_order=True.

Out-of-core support, spilling to disk, is enabled by default and starts at roughly 80 percent of RAM. Operations including sort, window functions and many expressions can now spill to disk to finish a query, with a default disk budget of 64GB. Out-of-core joins and group-by operations are on the roadmap for future releases.

SQL as a First-Class Citizen

Polars 2.0 marks the point where SQL is treated as a first-class citizen. Optimizer and engine improvements include join reordering, much better common-subplan elimination and dynamic predicates with bloom filters.

On benchmarks derived from TPC-H and TPC-DS, run against DuckDB 1.5.6, a DuckDB 2.0 alpha and DataFusion 54.0.0 on c7a.4xlarge and c7a.metal machines, default Polars was fastest on all but one benchmark. The team notes a constant overhead when scaling to 192 threads that hurts small data queries. Polars limited to 32 cores was competitive or winning in all benchmarks, and the cause has been diagnosed for a fix in the next release. DataFusion timed out on some TPC-DS queries and ran out of memory on TPC-H q18 on the smaller machine.

New Map Dtype

Polars now supports the Arrow MapType directly as a Map dtype, comparable to a Python dictionary mapping keys to values. Before 2.0, Arrow MapType was read as List(Struct(...)). The dtype ships with dedicated expressions for key lookups, contains_key, len, keys and values.

Stricter Polars

The release doubles down on strictness and fail-fast behavior. Agents can validate a query structure early with collect_schema(), which resolves types and catches schema-level mismatches without materializing data, enabling faster AI iteration. Errors that depend on data now default to stricter behavior so inconsistencies are caught instead of silently producing different results.

The team plans further work on out-of-core, scaling at large CPU counts, Polars Cloud and GeoPolars, and has published a migration guide for upgrading to 2.0.

Sources: pola.rs