CSV vs. Parquet: which file format should you choose for modern data platforms?

In almost every migration and MDM engagement I work on, the same question surfaces sooner or later:

Should we store this data as CSV or Parquet?

CSV has been the universal data-exchange format for decades – nearly every business application can export and import it. Parquet has become the default for modern analytics platforms: Spark, Databricks, Snowflake, DuckDB and the data lakes underneath them.

The honest answer is that neither format is simply "better". They solve different problems. So rather than argue from first principles, I ran both formats through the same tasks on my own laptop and let the numbers settle it. Below are the measured results, plus a practical framework for when to reach for each.

What is CSV?

CSV (Comma-Separated Values) is a plain-text format where each row is a record and values are separated by a delimiter:

CustomerId,Name,Country,Revenue
1001,Acme Ltd,Sweden,15000
1002,Globex,Denmark,24000

Strengths: human-readable, trivial to generate, universally supported, ideal for moving data between systems.

Weaknesses: large files, no type information, no compression by default, and every read has to scan whole rows even when you only need one column.

What is Parquet?

Parquet is a columnar storage format built for analytics. Instead of storing data row by row, it stores values column by column. The same table, laid out internally:

CustomerId:  1001, 1002
Name:        Acme Ltd, Globex
Country:     Sweden, Denmark
Revenue:     15000, 24000

Grouping each column's values together unlocks three things CSV can't do: aggressive compression (similar values sit next to each other), dictionary encoding for repeated values, and reading a single column without touching the rest of the file. It also embeds the schema – column names and types – directly in the file.

The test

I wanted measured numbers, not folklore, so I ran both formats through the same tasks on the same machine:

• Test rig: Apple MacBook Air (M4), 16 GB RAM, Python 3.14, pandas 3.0.3, pyarrow 24.0.0. Timing via timeit, median of up to 7 runs.
• Small / interoperability case: the classic Iris dataset – 5 columns, 150 rows. The kind of tiny file that pours out of ERP and Excel exports.
• Large / analytics case: a real financial-fraud detection dataset – 18 columns, 5 million rows, 754 MB as raw CSV. A realistic mix: low-cardinality categoricals (transaction type, merchant category, location) alongside high-cardinality keys (account numbers, IP addresses, device hashes).

For each dataset I compared file size on disk (CSV, gzipped CSV, Parquet) and the operations that matter: a full read, a write, reading a single column, a read-then-filter scan, and a grouped aggregation. On the large file I also swept every Parquet compression codec.

Results: the small file (Iris)

	CSV	Parquet
Size on disk	3.8 KB	4.6 KB
Full read	0.22 ms	0.47 ms
Write	0.27 ms	0.38 ms
Read 1 column	0.16 ms	0.32 ms
Read + filter	0.33 ms	0.56 ms

Read that table twice, because it's the opposite of what the marketing suggests: for a tiny file, CSV wins on every line. Parquet's file is actually larger (its footer, schema and per-column metadata cost more than the data itself at this size), and every operation is ~2x slower because there's column structure to set up before any work happens.

This is the interoperability sweet spot. A business user can open the CSV in Excel and review it in seconds; the Parquet file needs tooling. At 150 rows, the columnar machinery is pure overhead.

Results: the large file (5M fraud transactions)

Here CSV is up against Parquet's default codec (snappy):

	CSV	Parquet (snappy)	Parquet advantage
Size on disk	754.1 MB	318.7 MB	2.4x smaller
Full read	8.32 s	133 ms	~63x faster
Write	14.13 s	1.94 s	~7x faster
Read 1 column	1.56 s	17 ms	~92x faster
Read + filter	7.96 s	269 ms	~30x faster
Group by	1.86 s	129 ms	~14x faster

Now the picture inverts completely. The headline result is reading a single column – ~92x faster, because Parquet only touches that one column on disk while CSV has to parse all 18. Full reads land at ~63x, filtered scans ~30x. Even gzipped CSV (273 MB) is larger than snappy Parquet, and it still has to be fully decompressed and parsed before you can query it.

A note on that 2.4x storage ratio: this dataset is honest about its limits. Its biggest columns are high-cardinality keys – account numbers, IP addresses, device hashes, transaction IDs – which barely compress, because almost every value is unique. Parquet's dictionary encoding feeds on repetition, so on tables dominated by low-cardinality categoricals (statuses, country codes, product types) you'll see far more – 10x and beyond. The codec you choose moves this number too, which is the next thing I measured.

Parquet isn't one format: the compression sweep

"Parquet" is really a container plus a choice of compression codec, and that choice is not cosmetic. I wrote the same 5-million-row frame out with every codec pandas/pyarrow supports and measured the result:

Storage on disk (raw CSV is 754.1 MB):

Format	Size	vs CSV
CSV (gzip)	273.0 MB	2.8x
Parquet (uncompressed)	550.0 MB	1.4x
Parquet (snappy)	318.7 MB	2.4x
Parquet (lz4)	324.0 MB	2.3x
Parquet (zstd)	207.2 MB	3.6x
Parquet (gzip)	213.8 MB	3.5x
Parquet (brotli)	202.8 MB	3.7x

Write and read times:

Format	Write	Full read	1 column	Filter
Parquet (uncompressed)	1.36 s	135 ms	11 ms	293 ms
Parquet (snappy)	1.94 s	133 ms	17 ms	269 ms
Parquet (lz4)	1.96 s	123 ms	13 ms	257 ms
Parquet (zstd)	2.13 s	162 ms	21 ms	292 ms
Parquet (gzip)	65.74 s	260 ms	43 ms	389 ms
Parquet (brotli)	13.86 s	285 ms	50 ms	411 ms

A few things jump out:

• Every codec demolishes CSV – even uncompressed Parquet reads ~60x faster than CSV and a single column ~140x faster. That win comes from the columnar, binary, schema-aware layout, not from compression. Compression is a bonus on top.
• zstd is the sweet spot. It hits 3.6x compression – all but tied with the best – yet writes in ~2 s and reads almost as fast as snappy. Best storage-per-second by a distance.
• snappy and lz4 are the speed kings. Fastest reads and writes, but they trade away compression (~2.4x). The right default when write throughput or low CPU matters more than disk.
• Parquet + gzip is a trap. It took 65 seconds to write – 30x slower than zstd – for worse compression than zstd. Familiar name, wrong tool for this job.
• brotli wins on size (3.7x, the smallest file) but you pay for it in write and read time. Worth it for cold archival data you write once and rarely touch.

The practical default: zstd for storage you keep, snappy/lz4 for hot intermediate data you rewrite often. The read-speed spread across codecs stays inside ~2x, so storage and write cost are what actually decide the choice.

Why the crossover happens

The two result tables tell one story: Parquet's advantage scales with data size, and below some threshold it's a net cost.

• Columnar layout has fixed overhead. Schema, footer and per-column metadata are roughly constant whether the file has 150 rows or 5 million. On a tiny file that overhead dominates; on a large file it's a rounding error.
• Column projection is Parquet's superpower. The ~92x single-column result is the whole reason analytics engines love it. SELECT revenue FROM sales WHERE country = 'Sweden' against a 100-column Parquet table reads roughly two columns off disk. The same query against CSV reads all 100.
• Apple Silicon flatters both. The M4's fast SSD and high memory bandwidth keep absolute times low, but they don't change the shape of the result – the crossover is structural, not hardware-specific.

Schema: the quiet advantage

Beyond speed and size, Parquet solves a problem CSV creates. This is a valid CSV:

1001,15000
1002,24000

Is the first column a string or an integer? Is revenue a decimal? What does each column mean? CSV can't say – the importing system has to know. Parquet embeds the answer (CustomerId: INT64, Revenue: DECIMAL(18,2), Country: STRING), which means fewer import errors, stronger validation and easier governance. In migration work, where a mis-typed key column can quietly corrupt a load, that self-describing schema earns its keep.

What this means for migrations

During ERP migrations, CSV is still everywhere – and for good reason. Almost every platform speaks it: SAP, Microsoft Dynamics, Oracle, Infor, Salesforce. CSV is the lingua franca between systems:

Legacy system → Extract CSV → Transform → Validate → Target ERP

A business user can open the extract in Excel and eyeball it immediately. That's much harder with Parquet, and the small-file results above show there's nothing to gain by forcing it. For the exchange layer, CSV's simplicity and universality usually outweigh efficiency.

What this means for analytics

Analytics workloads have the opposite profile. Picture a data lake with billions of transactions and daily reporting queries. On CSV that means long loads, expensive scans and a steep cloud-storage bill. On Parquet it means faster queries, lower storage cost and queries that read only the columns they need – which is exactly why modern analytics platforms store data as Parquet internally:

ERP → Data lake → Parquet → Analytics & AI

The DuckDB asterisk

It's worth tempering the analytics case with a caveat the DuckDB team has made well: modern engines read CSV far faster than its reputation suggests. They parallelise parsing, auto-detect schemas and scan text efficiently. For a one-time read or a smaller dataset, CSV can be perfectly adequate – my Iris results are a miniature version of the same point.

The principle underneath both findings is the same:

The best format depends on how the data will be used. Read once, CSV is fine. Queried repeatedly, Parquet wins – and the bigger the data, the more it wins.

A practical recommendation

In most organisations the right answer isn't CSV or Parquet. It's both, each where it pays:

Source systems → CSV exchange layer → Transformation → Parquet storage → Analytics & AI

• Use CSV for moving data between systems, ERP imports/exports, supporting business users, and anywhere maximum compatibility matters.
• Use Parquet for data lakes, analytics workloads, large or repeatedly-queried datasets, and anything feeding Spark, Databricks, DuckDB or an AI pipeline.

CSV is the king of interoperability. Parquet is the king of analytics. The useful question is never "which format is better?" – it's "what are we actually trying to do with this data?"

Reproduce it yourself

Want to run the same comparison on your own machine? Grab the script – it's the exact code behind this article:

csv-vs-parquet-benchmark.py

It runs the small Iris head-to-head and the full Parquet compression sweep. Point --large at any CSV you like (it falls back to a synthetic stand-in if you don't), and --sample-rows for a quicker pass:

pip install numpy pandas pyarrow

# Out of the box (synthetic large dataset if you have none):
python csv-vs-parquet-benchmark.py

# Your own large CSV, sampled to 1M rows for speed:
python csv-vs-parquet-benchmark.py --large transactions.csv --sample-rows 1_000_000

The absolute numbers will differ on your hardware and data – run-to-run variance and, above all, the cardinality of your columns both matter – but the shape holds: CSV for small files and interoperability, Parquet for large, repeatedly-queried data, and zstd as a sane default codec.

Curious what the crossover looks like on your own datasets? Get in touch – or just run the script and see your actual times.