Backend
CheckCo: Checkpoint-Coordinated Load Balancingfor Skewed Event Streams at Scale
arXiv cs.DC0 views
arXiv:2611.22476v1 Announce Type: new Abstract: Event streams are rarely uniform: a few dominant keys can generate orders of magnitude more events than the rest, overloading the workers that receive them and capping throughput. We present CheckCo, a checkpoint-coordinated load-balancing algorithm built on a dynamic oracle that estimates per-key frequencies from the live stream, with no prior statistics. Workers count events per key locally; at each checkpoint barrier, the counts are merged into a partitioning function that divides each key into a number of splits roughly proportional to its estimated frequency and packs the splits onto workers, and this function routes the next checkpoint window. We derive approximation bounds on the resulting load imbalance from classical scheduling results and bound its total key fan-out. On a production pipeline processing over 100 billion events per day, the most loaded worker receives only 9.8% more events than the average over a day (imbalance ratio 1.10), against 8.03 for key-based routing in simulation; in simulation, CheckCo also needs less than half the fan-out of earlier partitioners that reach comparable balance. Shifts in the key distribution are reflected in routing from the next checkpoint on, and the load stays balanced through job restarts.
Source:https://arxiv.org/abs/2611.22476PreviousXM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory PoolsNextByzantine Resilience of Robot Gathering under Round-Robin Scheduling
Related posts
5 California App Development Companies and the Technical Questions to As
0dev.to Discuss
Content keeps getting longer, and the point is harder to find
0dev.to Discuss
I shipped one app to Shopify and one to Cafe24. Almost everything review flagged was copy and screens, not code.
0dev.to Discuss

Top WooCommerce Gift Card Plugins for Customers & Store Owners
0dev.to Productivity
Best Travel Apps for Vietnam
0dev.to Productivity
Is Opus Clip Legit? What the Billing Complaints Really Mean
0dev.to Productivity