Convert Giant CSV Files to Parquet Without Python MemoryError
Loading a 3GB server log CSV into pandas with read_csv() routinely triggers MemoryError on machines with 16GB RAM because pandas creates an in-memory DataFrame that can consume 5-10x the file's disk size during parsing. The row-oriented structure of CSV also means storing repetitive string values (like status codes or IP prefixes) wastes enormous amounts of space. Parquet's columnar storage with dictionary encoding and Snappy compression commonly reduces storage footprint by 50–90% depending on data cardinality. This workflow streams your CSV through DuckDB-Wasm in the browser, infers schema automatically, and outputs a .parquet file. The trade-off: Parquet is a binary format that you cannot inspect with a text editor or cat — you need a reader like DuckDB, PyArrow, or this tool to query it.
Why this matters?
A data engineer analyzing 4.2GB of Nginx access logs attempted to load the CSV into pandas for ad-hoc analysis. The read_csv() call consumed 38GB of virtual memory before the Linux OOM killer terminated the Python process with 'Killed' in the terminal — no traceback, no partial results, just a silent death. Setting up a local Spark cluster or spinning up an EC2 instance with 64GB RAM for a one-off log analysis is infrastructure overkill. The Parquet conversion itself took 47 seconds in-browser, compressing the 4.2GB CSV down to 890MB while preserving full query capability through any Parquet-compatible engine.
The 3-Step Solution
Follow this streamlined workflow to transform your raw data export into a clean, analysis-ready dataset. Each step leverages our browser-based tools to ensure your sensitive data never leaves your device.
By following these three steps, you eliminate manual data wrangling, reduce human error, and maintain full GDPR compliance throughout the process.
Ready to clean your data?
100% local processing. Zero uploads. Blazing fast.