Database
Measuring Tabular Data Formats Cold: A Replicated Cold-Cache Benchmark of Serialization, Scan, and Index Performance on Real Datasets
arXiv cs.SE0 views
arXiv:2611.10398v2 Announce Type: replace-cross Abstract: Choosing a file format for tabular data is a routine engineering decision, yet published comparisons often time reads served from the operating-system page cache, use one session, or rely on a single synthetic dataset. This article benchmarks fifteen format and access-path configurations (CSV, JSON Lines, Pickle, Feather, Parquet, ORC, DuckDB, and SQLite, with several access paths) on two real public datasets, New York City taxi trips and United States airline on-time records, at 50,000 to 3 million rows. Every measurement uses an explicitly cold cache, randomized execution order, and three independent sessions (with restarts on the laptop), and is analyzed with bootstrap intervals and Holm-adjusted permutation tests. Parquet answered a filtered aggregation 2.6 to 11 times faster than CSV, while JSON Lines was 10 to 27 times slower. A secondary index made SQLite queries 2.6 to 4.8 times slower on randomly ordered data and helped only on data sorted by the filtered column, at low selectivity. A row filter inside the Parquet reader was significantly faster than a projected scan on flight data from one million rows and on both datasets at three million, and row-group skipping needs several row groups, which the default writer produced only at 3 million rows. A 4 GB Linux host ran out of memory on configurations that a 24 GB laptop completed and reversed some orderings, such as Pickle versus Parquet. Code, derived data, and raw timings are released.
Source:https://arxiv.org/abs/2611.10398PreviousWriting Bug Reports for Software Repair Agents: What Information Matters Most?NextCross-Spectral Analysis of Bivariate Graph Signals
Related posts

Same MCP version. Different tool description. Who signed it?
0dev.to

Backups That Never Restored: Automated Restore Drills
0dev.to DevOps
Architectural Blueprints: Building High-Throughput Agentic & RAG Pipelines using C# and Vector Database Engine Tuning Executive Summary As enterprise AI systems evolve from simple single-turn Retrieva
0dev.to
Architectural blueprint
0dev.to

Should Developers Apply for Jobs in November and December? (Yes, and Here's Why)
0dev.to
One Rater to Score Them All: Frozen Large Language Model Anchoring for Video Quality of Experience
0arXiv cs.NI