Convert Giant CSV Files to Parquet Without Python MemoryError
Loading a 3GB CSV into pandas using read_csv() triggers MemoryError on 16GB machines because the in-memory DataFrame consumes 5–10x the file's disk size. CSV's row-oriented structure wastes enormous space on repetitive strings. Parquet's columnar storage with configurable row_group_size (default 122,880 rows) and compression reduces footprint by 50–90%. Snappy offers ~2:1 ratio at 500MB/s throughput; ZSTD offers ~4:1 at 150MB/s. This workflow streams your giant CSV through DuckDB-Wasm in the browser, infers the schema automatically, and outputs a compressed .parquet file via COPY TO 'file.parquet' (FORMAT PARQUET, CODEC 'ZSTD'). No Python environment, no EC2 instances, zero cloud uploads.
Why This Matters
A data engineer analyzing 4.2GB of Nginx access logs attempted to load it into pandas. The process consumed 38GB of virtual memory before the Linux OOM killer terminated it—just a silent 'Killed' in the terminal. Spinning up an EC2 r6i.4xlarge (64GB RAM) for a one-off task costs $2.40/hr and requires IAM permissions. Doing it locally via Wasm took 47 seconds, compressing the 4.2GB CSV down to 890MB while preserving full query capability for Spark, Athena, DuckDB, or Trino.
Why Excel & Python Fail Here
pandas read_csv() loads the entire file into a contiguous memory block. A 4.2GB CSV expands to 25–40GB in-memory due to Python object overhead (each string becomes a PyObject with 49 bytes of header). The Linux OOM killer terminates the process silently—no exception, no traceback, just 'Killed'. The workaround (chunked reading with chunksize=100000) requires 40+ lines of boilerplate and still risks memory fragmentation. DuckDB-Wasm streams data in 128KB pages, never loading the full file.
| Criterion | Excel / Sheets | Python / Cloud | DataPrep (This Page) |
|---|---|---|---|
| Max Rows | ~150K (crashes) | RAM-limited | 1M+ via Wasm |
| Setup Time | Manual formulas | 30–60 min env setup | 0 seconds |
| Data Privacy | Local but fragile | Uploads to cloud | 100% in-browser |
| Cost | License fees | EC2 / SaaS $49+/mo | Free forever |
Step-by-Step Solution
We will use the Parquet Converter to execute this entirely in your browser.
- 1
Drag and drop your giant CSV file
Upload the file (up to 4GB). The tool streams it through DuckDB-Wasm in 128KB pages—no full-file memory load. Schema inference runs on the first pass, data conversion on the second.
- 2
Choose compression codec
Select Snappy (faster, ~2:1 ratio, 500MB/s) or ZSTD (smaller output, ~4:1 ratio, 150MB/s). For analytics workloads, ZSTD is recommended. For streaming pipelines, Snappy is preferred.
- 3
Download .parquet file
The output is a single .parquet file with columnar layout, embedded schema, and row group statistics for predicate pushdown. Compatible with Spark, Athena, DuckDB, Trino, and pandas read_parquet().
Privacy Guarantee: Zero Bytes Uploaded
All processing happens in your browser via WebAssembly. Your data never touches a server. Try it: disconnect your Wi-Fi right now—the tool will still work.
Common Errors & Fixes
| Error / Symptom | Root Cause | Fix |
|---|---|---|
Python: 'Killed' (OOM) when loading CSV | pandas expands 4.2GB CSV to 38GB in-memory, exceeding available RAM | Use this tool—Wasm streams data in pages, never loading the full file. 4.2GB converts in 47 seconds. |
Parquet file is larger than expected | Default Snappy compression chosen for highly repetitive data | Switch to ZSTD codec for 4:1 compression on repetitive string columns (URLs, user agents, status codes) |
Frequently Asked Questions
Can Spark/Athena read the output Parquet file?
What happens if my CSV has inconsistent types in a column?
Ready to clean your data?
100% local processing · Zero uploads · Blazing fast
Trusted by 2,400+ data teams · 18M+ rows processed monthly