Hey all.

Would like to ask for some feedback on a protocol spec I've been working on.

I've got some dev experience in other languages, but am fairly new to Rust. So I've set up a project to learn more about it as well as building for edge devices, lower-level system stuff, and local ML/AI usage: GitHub - hyarthi/flwrs · GitHub

The gist of it is that I'm trying to build a dynamic pluggable workflow server, lightweight enough to run on edge devices like Raspberry Pi, but at the same time scalable to (potentially) run in clusters in cloud environments. The basic architecture is that there would be a central hub process with smaller plugin sub-processes running individual steps (either on same device or sitting in the same subnet) and communicating over TCP sockets.

As part of the project I've been looking at what protocol I could use for communication between the hub and the plugins. Looked at a few options out there (more and less mainstream), but in the end decided to hand roll my own. For one, I can strip it down to the bare minimum that I need to limit resource overhead on weaker edge devices. For another, it's a good learning in protocol design (which I've not really done before).

I've put together the spec here: flwrs/docs/protocol.md at main · hyarthi/flwrs · GitHub
I'd welcome any feedback or suggestions you may have.
Some specific open questions that I have at the moment are these:

  1. Credit units. There is an opt-in credit system for backpressure management. Credits currently count events only. Should they also count bytes (Credit.bytes), for producers whose event sizes vary a lot?
  2. Acks from the hub to producers mean "routed". Is that the right point, or should there be an option for "delivered to (and acked by) every consumer", end to end?
  3. Batching and compression are reserved in flags but not specified. Is per-frame compression worth it for the expected event sizes, or is batching (several events per frame) the better first step?
  4. Heartbeat defaults: a 10 s interval with three missed intervals, about 30 s to notice a dead link. Too slow for some deployments, too chatty for others?
  5. How far back a transform's output can refer: the hub remembers every unacked input and the last 1024 delivered. Should that window be negotiable, for accumulators with long windows? Or should an output be able to name several inputs (span links in tracing terms)?

Thank you!

1 post - 1 participant

Read full topic