Instead of pandas: Polars

Why Polars is better

• Wikipedia, citing comparisons, puts Polars at 5 to 10 times faster than pandas on similar tasks, using all CPU cores where pandas mostly uses one.
• pandas needs around 5 to 10 times the dataset size in RAM, against 2 to 4 times for Polars, so larger files fit on the same machine.
• Polars plans a whole query and pushes filters and column selection down into the file scan, which pandas, being eager only, cannot do.
• Polars has no index and uses expressions, which removes a whole class of pandas surprises around index alignment and chained assignment.

When to keep pandas

Keep pandas when your plotting, statistics or machine learning libraries take pandas DataFrames, when you rely on its index for time series work, or when a codebase is large and working. pandas is still actively released, now at version 3.1.1.

Switching

This is a rewrite, not a drop-in. The syntax differs: no index, expressions instead of column assignment, and no chained indexing. Polars' migration guide maps the common pandas idioms. Convert at the edges with to_pandas() and from_pandas() so you can move one pipeline at a time.

Watch out

Some libraries still accept only pandas objects, so you will convert at boundaries. Polars is developed by a venture-funded company, though the library is MIT-licensed. Code written for pandas habits, such as row-by-row apply, gains little until it is rewritten as expressions.

Also worth a look

• DuckDB An in-process SQL database that queries files and DataFrames directly; pick it if your team thinks in SQL.

More newer, better dataframe libraries and notebooks

• Jupyter Notebook →, use marimo