How to Process Large CSV Files from the Command Line

Excel crashes on large files. Python pandas requires setup. Google Sheets has row limits. The simplest solution? Use the command line.

Why CLI for CSV Processing?

No size limits — Process files of any size
Faster — No GUI rendering overhead
Automation — Script your entire pipeline
Reproducible — Same results every time

Installing csv-toolkit

npm install -g csv-toolkit

Common Operations

1. Merging Multiple CSV Files

# Combine all sales reports csv-toolkit merge sales_jan.csv sales_feb.csv sales_mar.csv -o q1_sales.csv # Merge all CSV files in a directory csv-toolkit merge ./reports/*.csv -o consolidated.csv

2. Cleaning Messy Data

# Remove empty rows and duplicates csv-toolkit clean raw_data.csv --remove-empty --deduplicate -o clean_data.csv # Trim whitespace and normalize csv-toolkit clean dirty.csv --trim --normalize

3. Converting Formats

# Excel to CSV csv-toolkit convert input.xlsx -o output.csv # CSV to JSON csv-toolkit convert data.csv -o data.json

4. Analyzing Data

# Get column statistics csv-toolkit analyze sales.csv --stats # Count unique values in a column csv-toolkit analyze sales.csv --unique --column "region"

Real Pipeline Example

# Step 1: Convert raw Excel data to CSV csv-toolkit convert raw_data.xlsx -o stage1.csv # Step 2: Clean the data csv-toolkit clean stage1.csv --remove-empty --deduplicate -o stage2.csv # Step 3: Merge with historical data csv-toolkit merge stage2.csv historical.csv -o final_dataset.csv # Step 4: Analyze results csv-toolkit analyze final_dataset.csv --stats --column "revenue"

Automate with Cron

# Run daily at 2 AM 0 2 * * * cd /path/to/project && csv-toolkit merge ./daily/*.csv -o weekly_report.csv

Performance

csv-toolkit handles millions of rows efficiently because it's built on Node.js streams — files are processed line by line, never loaded entirely into memory.