Menu

Polars vs pandas vs DuckDB: 2026 Benchmark on 10 and 50 Million Rows

Polars vs pandas vs DuckDB: 2026 Benchmark on 10 and 50 Million Rows - 9
Paul Francis

Table of content

    Summary

    Key takeaways

    • Polars and DuckDB substantially outperform pandas on large analytical workloads, especially group-by operations and joins.
    • In Uvik Software’s 2026 benchmark on 10 million rows, Polars completed 10 group-by queries in 2.86 seconds versus 16.39 seconds for pandas, making it about 5.7x faster.
    • DuckDB was the fastest engine on 6 of 10 group-by queries, while Polars was fastest on the other 4.
    • Polars performed best on joins, completing all 5 benchmark joins in 3.16 seconds versus 11.85 seconds for pandas and 10.46 seconds for DuckDB.
    • Python-level group operations are particularly expensive in pandas: one apply-based group-by took 6.46 seconds compared with 0.16 seconds in Polars and 0.14 seconds in DuckDB.
    • Parquet is a better storage format than CSV for analytical workloads because it loads faster and uses significantly less disk space across all three engines.
    • Memory limits become a major differentiator at larger scale: with 50 million rows and a 5.84 GiB cap, pandas failed during loading, Polars completed 9 of 10 queries, and DuckDB completed all 10.
    • Polars is a strong fit for Python-native DataFrame workflows where speed, joins, expressions, and lazy execution matter.
    • DuckDB is especially useful when datasets approach or exceed available RAM because its memory management and disk spilling allow larger analytical queries to finish.
    • pandas remains the most widely adopted ecosystem and may still be the best choice for smaller datasets, existing codebases, and workloads where compatibility matters more than maximum performance.

    When this applies

    This applies when you are choosing between pandas, Polars, and DuckDB for analytical Python workloads involving large DataFrames, group-bys, joins, CSV or Parquet files, and memory-constrained processing. It is particularly relevant when pandas workloads are becoming slow or memory-intensive, when datasets have grown into the multi-million-row range, or when you are designing new data pipelines and want to choose an engine based on measured performance rather than familiarity alone.

    When this does not apply

    This does not apply as a universal rule that Polars or DuckDB should replace pandas everywhere. Benchmark results depend on hardware, query shape, data types, implementation style, and library versions. pandas can remain the better choice for small datasets, mature codebases, libraries that expect pandas objects, or workflows where developer familiarity and ecosystem compatibility matter more than raw execution speed. The benchmark should therefore be treated as a starting point and reproduced with your own representative data before a major migration.

    Checklist

    1. Measure the current size of your datasets and expected growth over the next 12–24 months.
    2. Identify the operations that consume the most time, such as joins, group-bys, filtering, or custom Python functions.
    3. Benchmark the current pandas workload before changing libraries.
    4. Test Polars with the same representative queries and data types.
    5. Test DuckDB when SQL is acceptable or the workload is primarily analytical.
    6. Use Parquet instead of CSV where practical for repeated analytical workloads.
    7. Measure peak memory consumption in addition to execution time.
    8. Test with realistic memory limits rather than assuming unlimited RAM.
    9. Review pandas code for apply-based group operations that could be rewritten with vectorized expressions.
    10. Evaluate whether Polars lazy execution can reduce unnecessary reads and intermediate work.
    11. Evaluate DuckDB for workloads that need to query files directly without loading everything into a DataFrame first.
    12. Check library and ecosystem compatibility before replacing pandas in an existing application.
    13. Verify row-order assumptions when moving to Polars because some operations do not preserve order unless explicitly requested.
    14. Run benchmarks on your actual deployment hardware rather than relying only on published results.
    15. Choose the engine based on the dominant workload, memory constraints, developer skills, and integration requirements rather than headline speed alone.

    Common pitfalls

    • Assuming a benchmark winner will automatically be fastest for every workload.
    • Comparing libraries with toy datasets that are too small to expose meaningful performance differences.
    • Ignoring memory usage and focusing only on execution time.
    • Continuing to use CSV for large recurring analytical workloads when Parquet would reduce both storage and load time.
    • Using pandas apply inside large group-by operations when a vectorized implementation is possible.
    • Replacing pandas throughout an application without checking downstream library compatibility.
    • Expecting Polars to spill every large operation to disk even though some joins and group-bys can still exceed memory limits.
    • Loading an entire dataset into memory before considering whether DuckDB could query the source files directly.
    • Ignoring semantic differences such as indexes, expression APIs, SQL workflows, and row-order behavior when migrating code.
    • Making a technology decision based on one published benchmark instead of reproducing the comparison with your own representative workload.

    Short answer: is Polars faster than pandas? Yes. In the Uvik Software DataFrame Benchmark 2026, Polars 2.0 ran 10 standard group-by queries on 10 million rows 5.7x faster than pandas 3.0, and 5 joins 3.75x faster. DuckDB 1.5 was 4.2x faster than pandas on group-by. It was also the only engine that finished all queries on 50 million rows within a 5.84 GiB memory cap.

    Key findings (Uvik Software DataFrame Benchmark 2026)

    1. Polars 2.0.0 ran 10 group-by queries on 10 million rows in 2.86 seconds, against 16.39 seconds for pandas 3.0.6 (5.7x faster). DuckDB 1.5.6 took 3.91 seconds (4.2x faster).
    2. DuckDB was the fastest engine on 6 of the 10 group-by queries. Polars was the fastest on the other 4. pandas was the slowest on all 10.
    3. Polars won every join query. It ran 5 joins in 3.16 seconds, against 11.85 seconds for pandas (3.75x faster) and 10.46 seconds for DuckDB.
    4. A pandas group-by that calls a Python function for each group (apply) took 6.46 seconds. Polars did the same work in 0.16 seconds (40x faster) and DuckDB in 0.14 seconds (47x faster).
    5. At 50 million rows with a 5.84 GiB memory cap, pandas ran out of memory while it loaded the data. Polars finished 9 of 10 queries. DuckDB finished all 10.
    6. Parquet loaded faster than CSV in all three engines. Polars loaded 10 million rows from Parquet in 0.93 seconds, 2.6x faster than from CSV.
    7. pandas needed 3.5 to 4.9 times the CSV file size in memory. Loading a 569 MB CSV file used up to 2.8 GB of memory.

    Get the code and the data. The full benchmark package is available as a downloadable ZIP archive with the data generator, test harness, Python scripts and raw results. Run it on your own machine and compare. Code license: MIT. Results license: CC BY 4.0, with credit to Uvik Software.

    Get the benchmark code and results (ZIP) | Download the results (CSV)

    Polars vs pandas vs DuckDB: the results at a glance

    Bar chart comparing 10 group-by queries on 10 million rows: pandas 16.39 s, Polars 2.86 s and DuckDB 3.91 s.

    Figure 1. Group-by benchmark: total time for 10 standard queries on 10 million rows. Source: Uvik Software DataFrame Benchmark 2026.

    Measure (10 million rows unless stated) pandas 3.0.6 Polars 2.0.0 DuckDB 1.5.6
    Group-by: 10 queries (sum of medians) 16.39 s 2.86 s 3.91 s
    Speed against pandas (group-by) 1.0x 5.7x 4.2x
    Group-by queries where the engine was fastest 0 of 10 4 of 10 6 of 10
    Join: 5 queries (sum of medians) 11.85 s 3.16 s 10.46 s
    Speed against pandas (join) 1.0x 3.75x 1.13x
    Load the group-by table from CSV (569 MB) 3.06 s 2.39 s 4.46 s
    Load the group-by table from Parquet (213 MB) 2.26 s 0.93 s 2.75 s
    Peak memory during group-by queries 1,910 MiB 1,754 MiB 1,852 MiB
    Peak memory during join queries 4,255 MiB 3,671 MiB 4,301 MiB
    50 million rows: group-by queries completed 0 of 10 9 of 10 10 of 10

    Peak memory is the peak resident memory of the whole process, with the loaded data, in MiB (1 MiB = 1.049 MB). All times are medians of 3 runs, except the 50-million-row test (1 run).

    Group-by benchmark: 10 queries on 10 million rows

    On 10 million rows, Polars 2.0 finished the 10 db-benchmark group-by queries in 2.86 seconds and DuckDB 1.5 in 3.91 seconds, against 16.39 seconds for pandas 3.0. DuckDB won 6 queries and Polars won 4. pandas was the slowest on every query.

    Dot plot showing median seconds for 10 group-by queries in pandas, Polars and DuckDB on a logarithmic scale.

    Figure 2. Every group-by query, engine by engine: median seconds per query on 10 million rows (log scale). Source: Uvik Software DataFrame Benchmark 2026.

    Query What it calculates pandas (s) Polars (s) DuckDB (s) Fastest
    q1 Sum of v1 by id1 (100 groups) 0.224 0.041 0.021 DuckDB
    q2 Sum of v1 by id1 and id2 (10,000 groups) 0.509 0.102 0.079 DuckDB
    q3 Sum of v1 and mean of v3 by id3 (100,000 groups) 0.600 0.168 0.326 Polars
    q4 Mean of v1, v2 and v3 by id4 (100 groups) 0.263 0.062 0.042 DuckDB
    q5 Sum of v1, v2 and v3 by id6 (100,000 groups) 0.555 0.159 0.170 Polars
    q6 Median and standard deviation of v3 by id4 and id5 (10,000 groups) 0.748 0.545 0.502 DuckDB
    q7 Maximum of v1 minus minimum of v2 by id3 (100,000 groups) 0.453 0.149 0.309 Polars
    q8 Two largest values of v3 by id6 (100,000 groups) 2.303 0.586 0.305 DuckDB
    q9 Correlation squared of v1 and v2 by id2 and id4 (10,000 groups) 6.458 0.160 0.138 DuckDB
    q10 Sum of v3 and row count by all six id columns (10 million groups) 4.275 0.889 2.023 Polars
    Total 16.39 2.86 3.91

    What the queries show:

    • Polars won the simple aggregations on keys with many groups (q3, q5, q7 and q10).
    • DuckDB won the aggregations on keys with few groups (q1, q2 and q4) and the queries with statistical or top-N functions (q6, q8 and q9).
    • pandas was slow where it runs Python code for each group. In q9, it calls a Python function for each of the 10,000 groups, as the db-benchmark pandas script does. A vectorized pandas version can be faster. We kept apply() because db-benchmark uses it and because it is a common pattern in pandas code.

    Join benchmark: 5 queries on 10 million rows

    Polars 2.0 won all 5 db-benchmark join queries. It finished them in 3.16 seconds, against 11.85 seconds for pandas 3.0 and 10.46 seconds for DuckDB 1.5. On 2 vCPUs, DuckDB was only 1.13x faster than pandas on joins.

    Bar chart comparing five join queries: pandas 11.85 s, Polars 3.16 s and DuckDB 10.46 s.

    Figure 3. Join benchmark: total time for 5 standard join queries on 10 million rows. Source: Uvik Software DataFrame Benchmark 2026.

    Query What it does pandas (s) Polars (s) DuckDB (s)
    q1 Inner join with a 10-row table on an integer key 1.456 0.217 1.442
    q2 Inner join with a 10,000-row table on an integer key 1.616 0.248 1.729
    q3 Left join with a 10,000-row table on an integer key 1.155 0.316 2.077
    q4 Inner join with a 10,000-row table on a string key 2.535 0.260 1.639
    q5 Inner join with a 10-million-row table on an integer key 5.092 2.120 3.572
    Total 11.85 3.16 10.46

    Why was DuckDB slower on joins in this test? One likely reason is the size of the result. Each join returns about 9 million to 10 million rows, and DuckDB writes each result to a new table, as db-benchmark requires. With only 2 threads, this costs time. On a 128-core machine, the gap is smaller: in the db-benchmark report of August 2026, DuckDB ran the 0.5 GB join set in 0.40 seconds and Polars in 0.30 seconds.

    Loading data: CSV vs Parquet

    Parquet loaded faster than CSV in all three engines. Polars loaded the 10-million-row table from Parquet in 0.93 seconds and from CSV in 2.39 seconds. Parquet also saves disk space: the same data used 213 MB as Parquet and 569 MB as CSV.

    Two bar charts comparing CSV and Parquet loading times for 10 million rows in pandas, Polars and DuckDB.

    Figure 4. Load 10 million rows: CSV vs Parquet, median seconds. Source: Uvik Software DataFrame Benchmark 2026.

    Load time (median of 3) pandas Polars DuckDB
    Group-by table, CSV (569 MB) 3.06 s 2.39 s 4.46 s
    Group-by table, Parquet (213 MB) 2.26 s 0.93 s 2.75 s
    Join tables, CSV (4 files, 1.04 GB) 3.03 s 3.18 s 5.05 s
    Join tables, Parquet (4 files, 494 MB) 1.75 s 0.59 s 2.62 s

    Load time includes the type changes that db-benchmark uses. pandas and Polars convert the string keys of the group-by table to categorical types. DuckDB converts two keys to ENUM types and copies the data into its own table format. DuckDB can also query CSV and Parquet files directly, with no load step.

    Memory: what happens at 50 million rows

    With a 5.84 GiB memory cap, pandas 3.0 ran out of memory while it loaded 50 million rows. Polars 2.0 loaded the data and finished 9 of 10 group-by queries, then ran out of memory on the six-key query. DuckDB 1.5 finished all 10 queries in 32.3 seconds.

    Bar chart showing queries completed under a 5.84 GiB memory cap: pandas 0, Polars 9 and DuckDB 10.

    Figure 5. 50 million rows with a 5.84 GiB memory cap: group-by queries completed out of 10. Source: Uvik Software DataFrame Benchmark 2026.

    50 million rows (1.18 GB Parquet, 2.89 GB as CSV), 1 run pandas 3.0.6 Polars 2.0.0 DuckDB 1.5.6
    Load from Parquet Out of memory (stopped at 5,952 MiB) 7.98 s 25.35 s
    Memory used by the loaded data n/a 4,486 MiB 4,886 MiB
    Group-by queries completed 0 of 10 9 of 10 10 of 10
    Time for q1 to q9 n/a 13.15 s 15.96 s
    Time for q10 (six keys, 50 million groups) n/a Out of memory 16.30 s

    Polars 2.0 spills to disk by default, but its release notes say that joins and group-bys do not spill yet. This matches our result: the six-key group-by makes 50 million groups and needs more memory than the cap. DuckDB manages memory with a buffer manager and a memory limit (we set 4.6 GiB), and it can spill large operations to disk. It finished every query under the same cap.

    How much memory does pandas need? In our test, pandas 3.0 needed 3.5 times the CSV file size during group-by queries and 4.9 times during the CSV load. If your CSV data is bigger than about one fifth of your RAM, plan for Polars or DuckDB. If the data is close to or bigger than your RAM, use DuckDB.

    What changed in pandas 3.0, Polars 2.0 and DuckDB 1.5

    pandas 3.0 (21 January 2026) made Copy-on-Write the default and added a dedicated string type. Polars 2.0 (6 October 2026) made the streaming engine the default for lazy queries and turned on spill to disk. DuckDB 1.5 (9 March 2026) is the current release line, and DuckDB 2.0 is planned for 21 October 2026.

    Library Version we tested Major release What changed License
    pandas 3.0.6 (17 September 2026) 3.0.0: 21 January 2026 Copy-on-Write is the default, so chained assignment no longer works. A dedicated string type is on by default (backed by PyArrow when it is installed). New pd.col() expressions. BSD-3-Clause
    Polars 2.0.0 (6 October 2026) 2.0.0: 6 October 2026 collect() on a LazyFrame uses the streaming engine by default. Spill to disk is on by default, but not yet for joins and group-bys. Row order is kept only with maintain_order=True. MIT
    DuckDB 1.5.6 (28 September 2026) 1.5.0: 9 March 2026 Since 1.4.0, every second release is a long-term support (LTS) release; the LTS line is 1.4.5. DuckDB 2.0 is planned for 21 October 2026 (tentative date). MIT

    Who is behind each project. pandas is a community project. Polars is backed by a company that raised an €18 million Series A led by Accel in September 2025. In August 2026, AWS agreed to acquire DuckLabs (formerly DuckDB Labs), the company that maintains DuckDB. DuckDB stays open source, and the non-profit DuckDB Foundation holds much of its intellectual property.

    Adoption. pandas is still far ahead. In the last month, pandas had 546.9 million downloads from PyPI, DuckDB 52.2 million and Polars 49.9 million (pypistats.org, 7 October 2026). For more data on the Python ecosystem, see our Python developer statistics and the Python ecosystem scorecard.

    pandas vs Polars vs DuckDB: the same code in each

    Most pandas operations have a direct Polars expression and a DuckDB SQL query. The main differences: Polars has no index, uses expressions such as pl.col() and can run lazy queries. DuckDB uses SQL and can query pandas and Polars DataFrames by their variable names.

    Task pandas 3.0 Polars 2.0 DuckDB 1.5 (SQL)
    Read Parquet pd.read_parquet("f.parquet") pl.read_parquet("f.parquet") SELECT * FROM 'f.parquet'
    Filter rows df[df["v3"] > 50] df.filter(pl.col("v3") > 50) SELECT * FROM df WHERE v3 > 50
    Add a column df.assign(r=df["v1"] - df["v2"]) df.with_columns((pl.col("v1") - pl.col("v2")).alias("r")) SELECT *, v1 - v2 AS r FROM df
    Group and sum df.groupby("id1", as_index=False).agg(v1=("v1", "sum")) df.group_by("id1").agg(pl.col("v1").sum()) SELECT id1, sum(v1) AS v1 FROM df GROUP BY id1
    Inner join x.merge(y, on="id1") x.join(y, on="id1") SELECT * FROM x JOIN y USING (id1)
    Left join x.merge(y, how="left", on="id2") x.join(y, how="left", on="id2") SELECT * FROM x LEFT JOIN y USING (id2)
    Two largest per group df.sort_values("v3", ascending=False).groupby("id6").head(2) df.group_by("id6").agg(pl.col("v3").top_k(2)).explode("v3") SELECT id6, unnest(max(v3, 2)) AS v3 FROM df GROUP BY id6
    Lazy query Not available (eager only) df.lazy().filter(...).collect() DuckDB plans every query before it runs
    Convert pl.from_pandas(df) df.to_pandas() duckdb.sql("...").df() or .pl()
    Missing values NaN or NA null (NaN is a float value, not a missing value) NULL

    Which one should you use?

    Use Polars for fast DataFrame code in Python on one machine. Use DuckDB when your team prefers SQL, when you query Parquet files directly or when the data is close to or bigger than your RAM. Keep pandas for small data and for libraries that expect pandas objects.

    Decision tree for choosing DuckDB for SQL and large datasets, Polars for fast Python pipelines, and pandas for small datasets.

    Figure 6. pandas, Polars or DuckDB: a decision guide built from the Uvik Software DataFrame Benchmark 2026 results.

    • Choose Polars for new Python data pipelines that fit in memory. It was 5.7x faster than pandas on group-by and 3.75x faster on joins, and it won every join.
    • Choose DuckDB for SQL teams, for queries on Parquet and CSV files and for data that is close to your RAM size. It won 6 of 10 group-by queries and was the only engine to finish all queries on 50 million rows.
    • Keep pandas for small data, for plotting and for libraries that expect pandas objects. pandas 3.0 brings safer defaults (Copy-on-Write), but it was the slowest engine on every group-by query.
    • Mix them. DuckDB can query pandas and Polars DataFrames by name and return the result with .df() or .pl(). For Polars, the data moves through Apache Arrow.

    For a wider view of the data stack, see our guide to data engineering tools.

    How to migrate from pandas to Polars or DuckDB

    1. Find the slow steps. Profile the pipeline. Record the time and the peak memory of each step.
    2. Convert storage to Parquet. In our test, Parquet loaded up to 5.4x faster than CSV (Polars, join tables) and used 63% less disk space (group-by table).
    3. Move one slow step at a time. Use pl.from_pandas() or a DuckDB query on the existing DataFrame. Keep the other steps in pandas until the new step passes its tests.
    4. Compare the outputs. Check the row count and the sums of the numeric columns after each migrated step, as we did in this benchmark. Check nulls and NaN values: Polars treats NaN as a value, not as a missing value. Our guide to data quality metrics lists more checks.
    5. Remove index logic. Polars has no index. Replace .loc lookups and index joins with explicit key columns.
    6. Pin the versions and test in CI. Polars 2.0 changed its default engine, and DuckDB 2.0 is due. Pin each version and run your tests again before each upgrade.

    Make your data pipelines faster and cheaper. Uvik Software data engineers profile your pandas pipelines, move the slow steps to Polars or DuckDB and add tests that prove the outputs match. You keep the code and the tests.

    Talk to a data engineer | Hire data engineers

    Methodology

    • Hardware. 2 vCPUs (Intel Xeon at 2.80 GHz) and 8 GB of RAM, with a 5.84 GiB memory cap for the test and no swap, on Linux. This is the same class as a GitHub-hosted runner for private repositories (2 CPUs and 8 GB of RAM), so many teams run their CI on a similar machine.
    • Software. Python 3.13.16, pandas 3.0.6, Polars 2.0.0, DuckDB 1.5.6 and PyArrow 25.0.1. Polars and DuckDB used 2 threads. The DuckDB memory limit was 4.6 GiB. Polars used its 2.0 default engine.
    • Data. Synthetic data with the db-benchmark schema (created by H2O.ai, now maintained by DuckDB Labs): a 10-million-row group-by table with 9 columns and 100 groups per low-cardinality key, four join tables and a 50-million-row group-by table. We generated the data in Python with NumPy (PCG64 generator, seed 108). It matches the distributions and group counts of the original R generator, but not its exact values.
    • Queries. The 10 group-by and 5 join questions of db-benchmark, written in the normal style of each engine, with the db-benchmark scripts as the template.
    • Timing. Wall-clock time of each query, with the result fully built: a pandas DataFrame, a Polars collect() and a DuckDB CREATE TABLE AS. 3 runs per query at 10 million rows, median reported. 1 run at 50 million rows. Load times are separate from query times.
    • Memory. Peak resident memory of the process (VmHWM), reset before each run, reported in MiB.
    • Correctness. For each query, we compared the row count and the sums of all numeric result columns across the three engines. All results matched within floating-point tolerance.
    • Isolation. Each engine ran in its own process, and the runner recorded each failure. Both out-of-memory failures were stops by the operating system at the 5.84 GiB memory cap.
    • Limits. One small machine, synthetic data and no GPU engines. Results on large machines are different (see the next section). We followed the rules in “Fair Benchmarking Considered Difficult” (Raasveldt et al., DBTest 2018): separate load and query times, tune each engine, check the output and report medians.

    How our results compare with other benchmarks

    Other public benchmarks agree on the main result: Polars and DuckDB are much faster than pandas. The winner between Polars and DuckDB changes with the workload and the machine size. Each of the three large benchmarks is run by a vendor, so always check who runs a benchmark.

    Benchmark Run by Machine Result
    db-benchmark (report of 17 August 2026) DuckDB Labs (now DuckLabs) AWS c6id.metal, 128 cores, 256 GB 0.5 GB group-by: DuckDB 1.5.4 0.52 s, Polars 1.42.1 1.64 s, pandas 2.2.3 12.01 s. 0.5 GB join: Polars 0.30 s, DuckDB 0.40 s, pandas 10.70 s (Uvik Software sums of the run 1 times per question)
    PDS-H, derived from TPC-H (1 June 2025) Polars AWS c7a.24xlarge, 96 vCPUs Scale factor 10: Polars streaming 3.89 s, DuckDB 5.87 s, pandas 365.71 s. Scale factor 100: DuckDB 19.65 s, Polars streaming 23.94 s
    ClickBench, 43 queries ClickHouse AWS c6a.metal Polars DataFrame 11.92 s, DuckDB 17.32 s, pandas 2,430.31 s (Uvik Software sums of the best hot runs)
    Uvik Software DataFrame Benchmark 2026 Uvik Software 2 vCPUs, 5.84 GiB memory cap 10M-row group-by: Polars 2.86 s, DuckDB 3.91 s, pandas 16.39 s

    The pandas results in db-benchmark are from pandas 2.2.3 (February 2025). To our knowledge, this report is the first public benchmark of pandas 3.0 against Polars 2.0. Uvik Software runs this benchmark as an engineering company, not as a vendor of any of the three engines.

    How to cite this report

    Uvik Software (2026). Polars vs pandas vs DuckDB: 2026 benchmark on 10 and 50 million rows (Uvik Software DataFrame Benchmark 2026). Run on 7 October 2026. https://uvik.net/blog/polars-vs-pandas/

    You can reuse the charts and the results under CC BY 4.0, and the code under MIT. Please credit “Uvik Software” with a link to this page.

    Sources

    1. Uvik Software. DataFrame Benchmark 2026: code, data generator and raw results. Download the benchmark package (ZIP) and results (CSV).
    2. DuckDB Labs. Database-like ops benchmark (db-benchmark), report generated 17 August 2026. https://duckdblabs.github.io/db-benchmark/
    3. pandas. What’s new in 3.0.0 (January 21, 2026). https://pandas.pydata.org/docs/whatsnew/v3.0.0.html
    4. Polars. Polars 2.0 release post, 6 October 2026. https://pola.rs/posts/release-polars-2
    5. Polars. Coming from pandas (migration guide). https://docs.pola.rs/user-guide/migration/pandas/
    6. Polars. PDS-H benchmark results, 1 June 2025. https://pola.rs/posts/benchmarks
    7. DuckDB. Release calendar. https://duckdb.org/release_calendar
    8. DuckDB. Integration with Polars. https://duckdb.org/docs/current/guides/python/polars.html
    9. DuckDB Foundation. https://duckdb.org/foundation/
    10. Amazon. AWS to acquire DuckLabs, August 2026. https://www.aboutamazon.com/news/company-news/aws-ducklabs
    11. GeekWire. Amazon acquires DuckLabs, adding the team behind DuckDB, August 2026. https://geekwire.com/2026/amazon-acquires-ducklabs-adding-the-team-behind-duckdb-amid-broader-shakeup-in-cloud-data/
    12. TechCrunch. The startup behind open source tool Polars raises $21M from Accel, 29 September 2025. https://techcrunch.com/2025/09/29/the-startup-behind-open-source-tool-polars-raises-21m-from-accel/
    13. ClickHouse. ClickBench. https://benchmark.clickhouse.com/
    14. Raasveldt, M., Holanda, P., Gubner, T. and Mühleisen, H. Fair Benchmarking Considered Difficult: Common Pitfalls in Database Performance Testing. DBTest 2018. https://ir.cwi.nl/pub/27862/27862.pdf
    15. pypistats.org. Download statistics for pandas, polars and duckdb, viewed 7 October 2026. https://pypistats.org/packages/polars
    16. GitHub Docs. GitHub-hosted runners reference, viewed 7 October 2026. https://docs.github.com/en/actions/reference/runners/github-hosted-runners

    Get senior Python and data engineers. Uvik Software provides senior Python, data and AI engineers: more than 50 engineers, no juniors, and at least 7 years of experience each. You get matched profiles 48 hours after the SOW and a 30-day no-cost replacement. Rates: $50 to $99 per hour. Uvik Software is a Databricks Bronze partner and a member of the Python Software Foundation.

    Request matched profiles | Hire Python developers

    Frequently asked questions

    Is Polars faster than pandas?

    Yes. In our 2026 benchmark on 10 million rows, Polars 2.0 was 5.7x faster than pandas 3.0 on 10 group-by queries and 3.75x faster on 5 joins.

    Is DuckDB faster than Polars?

    It depends on the query. DuckDB won 6 of 10 group-by queries and Polars won 4. Polars won all 5 joins. DuckDB was the only engine that finished all queries on 50 million rows within a 5.84 GiB memory cap.

    Is Polars better than pandas?

    For query speed and memory, yes in our tests: Polars was faster on every query and used less peak memory. pandas is still a good choice for small data and for libraries that expect pandas objects.

    Why is Polars faster than pandas?

    Polars runs many operations in parallel on all CPU cores, uses the Apache Arrow memory format and optimizes lazy queries before it runs them. The core of pandas is single-threaded.

    Will Polars replace pandas?

    Not soon. In the last month, pandas had about 547 million PyPI downloads, about 11 times more than Polars with about 50 million (pypistats.org, 7 October 2026). Many teams use both.

    What is DuckDB?

    DuckDB is an open-source SQL database for analytics that runs inside your application, with no server. In Python, it queries CSV and Parquet files directly and can query pandas and Polars DataFrames by name. It uses the MIT license.

    When should I use DuckDB instead of pandas?

    Use DuckDB when your team prefers SQL, when you query Parquet or CSV files directly, or when your data is close to or bigger than your RAM.

    What changed in pandas 3.0?

    Copy-on-Write is now the default, so chained assignment no longer works. A dedicated string type is on by default. pandas 3.0.0 was released on 21 January 2026.

    What changed in Polars 2.0?

    Lazy queries use the streaming engine by default, and spill to disk is on by default. Polars 2.0.0 was released on 6 October 2026.

    How much memory does pandas need?

    In our test, pandas 3.0 needed 3.5 to 4.9 times the CSV file size in RAM. Loading a 569 MB CSV file used up to 2.8 GB of memory.

    How useful was this post?

    No votes so far! Be the first to rate this post.

    Share:
    Polars vs pandas vs DuckDB: 2026 Benchmark on 10 and 50 Million Rows - 16

    Need to augment your IT team with top talents?

    Uvik can help!
    Contact
    Uvik Software
    Privacy Overview

    This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

    Get a free project quote!
    Fill out the inquiry form and we'll get back as soon as possible.

      Subscribe to TechTides – Your Biweekly Tech Pulse!
      Join 750+ subscribers who receive 'TechTides' directly on LinkedIn. Curated by Paul Francis, our founder, this newsletter delivers a regular and reliable flow of tech trends, insights, and Uvik updates. Don’t miss out on the next wave of industry knowledge!
      Subscribe on LinkedIn