AI & ML
Fault Tolerance for Stream Processing Engines
arXiv cs.DC0 views
arXiv:1605.00928v4 Announce Type: replace Abstract: Distributed Stream Processing Engines (DSPEs) target applications related to continuous computation, online machine learning and real-time query processing. DSPEs operate on high volume of data by applying lightweight operations on real-time and continuous streams. Such systems require clusters of hundreds of machines for their deployment. Streaming applications come with various requirements, i.e., low-latency, high throughput, scalability and high availability. In this survey, we study the fault tolerance problem for DSPEs. We discuss fault tolerance techniques that are used in stream processing engines that were state of the art at the time of writing (2016), namely S4, Storm, Flink, Spark Streaming, Samza and MillWheel. Further, we give insight on fault tolerance approaches that we categorize as active replication, passive replication and upstream backup. Finally, we discuss implications of the fault tolerance techniques for different streaming application requirements. This revised version corrects errors in the original 2016 manuscript, summarizes later developments in Flink and Spark, discusses durable logs and archival storage for replay, and points to more recent surveys.
Source:https://arxiv.org/abs/1605.00928PreviousThe Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per TokenNextMagneton: Optimizing Energy Efficiency of ML Systems via Differential Energy Debugging
Related posts
The Autonomous Edge — Issue #13: Cheaper Agents, Harder Governance (Week of November 2, 2026)
0dev.to AI

Plano Físico vs Plano Funcional en AWS Networking: cómo los arquitectos senior resuelven lo que otros no ven
0dev.to AWS
We built Origami Tutor: free paper-craft lessons you can turn and replay in 3D
0dev.to WebDev
From two authorized photos to a short AI rap video (consent-first workflow)
0dev.to WebDev
Check a File Hash in the Browser, Without Uploading the File
0dev.to WebDev
Your AI keys were leaked at a platform. A local guard you can paste catches the abuse before the invoice.
0dev.to Security