Open Source
Benchmarking Content-Defined Chunking Algorithms for Data Deduplication
arXiv cs.DC0 views
arXiv:2610.13704v1 Announce Type: cross Abstract: Data deduplication relies on content-defined chunking (CDC) algorithms to decide where data is split. We observe that most CDC algorithms are modular: a base chunking algorithm fixes the byte-level cut-point condition, while high-level techniques (size normalization, min/max thresholds, caching) layer on top. Existing benchmarks conflate the two, obscuring which design choices actually drive performance. We isolate and evaluate eight state-of-the-art base chunking algorithms across four realistic workloads and four metrics: throughput, deduplication ratio, mean chunk size, and chunk-size variance. Further, we derive new formulas relating algorithm parameters to expected chunk size for AE, RAM, and MII. In doing so, we uncover a longstanding error in the widely-used AE parameterization reproduced across multiple papers and evaluation tools that produces chunks 13-39% smaller than intended and inflates AE's reported deduplication. Across our evaluation, AE is the most balanced base chunking algorithm across all four metrics, RAM's similarity to AE does not carry over to low-entropy data, and MII and PCI show limited advantages over simpler alternatives in storage-oriented workloads. We release all implementations, datasets, and analysis scripts as open source.
Source:https://arxiv.org/abs/2610.13704PreviousEmber: Making Unified Virtual Memory Practical for Discrete-Memory GPUsNextAdaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy
Related posts
A fresh wave of cyberattacks has hit Europe’s supermarket refrigerators, destroying food across countries
0TechCrunch

Onboarding Instrumentation: What to Actually Log Before You Redesign the Flow
0dev.to UX
Iceberg vs Parquet: do you need a table format?
0dev.to Data

Open Source Didn't Ban AI Code. 83% of Its AI Policies Allow It, With One Catch
0dev.to Git
Orbit: A High-Performance Terminal Workspace & System Monitor in Rust with WASM Plugins
0dev.to Rust

Forecasting the U.S. Senate with Prediction Markets and Monte Carlo Simulations
0dev.to AI