Flow control in busy groups

Apr 5, 2012 at 2:09 PM

I'm giving a lot of thought to flow control lately, especially in large groups with a lot going on in them, and wanted to offer a few rules of thumb for those who might be interested in this topic:

1) If you can find a way to code your solution in terms of Send (or even the unreliable RawSend) those will certainly do best. My built in flow control scheme seems to work well for both cases, and you can also use the rate controller if you wish.

2) CausalSend isn't a great choice in these situations.  If you have lots of concurrent senders using CausalSend in a busy group, causal send can easily degenerate into a sequential ordering in which lots of messages get sent but only one can be delivered at a time.  This will tend to be slow.  I would limit the use of CausalSend to fairly small groups and avoid using it with huge message bodies.

3) Performance of OrderedSend and SafeSend/Paxos will be quite similar for in-memory configurations and both tend to slow down linearly in the size of the group and also to slow down more as a function of group size than Send or RawSend.

4) Among all possible options, SafeSend in the DiskLogger durability mode is by far the worst for raw speed at large scale

Summary:  For a small group, with perhaps 3-5 members, and of these options is actually fine depending on your goals.

For a group with hundreds or thousands of members, try to find a way to implement your algorithm using the FIFO Send primitive.,  This works better than any other protocol: they may be simpler to think about, but the fancier ordering and durability options come with costs and those can really bite in large, busy groups.