Backend
Asymptotic Analysis of Data Deduplication Under a Constant Edit Model
arXiv math.IT0 views
arXiv:2611.09842v1 Announce Type: new Abstract: Data deduplication has become increasingly important in large-scale storage systems, driven by the rapid growth of digital data. By identifying and removing repeats in the data stream, deduplication saves storage space more efficiently than traditional compression methods. In this paper, we study the performance of variable-length deduplication and multi-chunk deduplication algorithms from an information-theoretic perspective, focusing on data streams where repeats are approximate rather than exact. We consider two probabilistic source models in which source strings consist of repeated blocks that are modified by either a constant number of substitution edits or burst-insertion edits per block, the latter capturing the localized edit patterns commonly observed in real-world data. For both models, we show that variable-length deduplication achieves asymptotically arbitrarily large compression ratios, and that multi-chunk deduplication achieves compression within a constant factor of the source entropy under mild conditions.
Source:https://arxiv.org/abs/2611.09842PreviousAn Efficient Synchronization Scheme for Distributed BanditsNextHow far does a hash-only sender go? A pre-registered map of sender memory versus bits for short text messages
Related posts
Building a Private AI Control Plane for PS5 Porting: What We Learned After SQLite Locked Us Out at 2 AM
0dev.to
Monolith vs Microservices: A Founder's Guide
0dev.to
Sentinel Dev Diary: Read First, Run Second
0dev.to
A website for the café down the street, prepared before they ask
0dev.to

Where Temporary Files Come From and Why They Can Leak Office Data
0dev.to

How to Accept USDC Payments on EVM: A Reorg-Safe Stablecoin Payments Detector in Node.js
0dev.to