Backend
CheckCo: Checkpoint-Coordinated Load Balancingfor Skewed Event Streams at Scale
arXiv cs.DC0 views
arXiv:2611.22476v1 Announce Type: new Abstract: Event streams are rarely uniform: a few dominant keys can generate orders of magnitude more events than the rest, overloading the workers that receive them and capping throughput. We present CheckCo, a checkpoint-coordinated load-balancing algorithm built on a dynamic oracle that estimates per-key frequencies from the live stream, with no prior statistics. Workers count events per key locally; at each checkpoint barrier, the counts are merged into a partitioning function that divides each key into a number of splits roughly proportional to its estimated frequency and packs the splits onto workers, and this function routes the next checkpoint window. We derive approximation bounds on the resulting load imbalance from classical scheduling results and bound its total key fan-out. On a production pipeline processing over 100 billion events per day, the most loaded worker receives only 9.8% more events than the average over a day (imbalance ratio 1.10), against 8.03 for key-based routing in simulation; in simulation, CheckCo also needs less than half the fan-out of earlier partitioners that reach comparable balance. Shifts in the key distribution are reflected in routing from the next checkpoint on, and the load stays balanced through job restarts.
Source:https://arxiv.org/abs/2611.22476PreviousXM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory PoolsNextByzantine Resilience of Robot Gathering under Round-Robin Scheduling
Related posts
Driving the Style: Velocity-Conditioned Whole-Body Control from Motion Demonstrations
0arXiv cs.RO
sqrtVINS-Based 360 Visual-Inertial SLAM Systems: Hilti-Trimble SLAM Challenge 2026 Entry
0arXiv cs.RO
Ontology-Compiled Control Barrier Functions for Heterogeneous Robot Teams
0arXiv cs.RO
I Used to Get Attached to My Code. I Don't Anymore.
0dev.to AI
Securing MCP Servers: Preventing Shadow Tool Declarations and JSON-RPC Parameter Poisoning
0dev.to AI
How do I find out why users abandon conversations with my AI agent?
0dev.to AI