Apr 1, 2015 at 1:37 PM
Edited Apr 1, 2015 at 2:17 PM
The new system name (for the C++ version) will be Derecho, which refers to a super-fast wind that can occur in extreme weather situations, DMC (Derecho multicast) for short.
My decision is to really strip the system down to a minimum at least at the outset. So DMC will retain exactly the same process group and multicast structure as Isis2, but I'm getting rid of the OOB file transfers, DHT key-value store, Aggregation layer, as
well as some internal features such as UNICAST-only, USEINFINIBAND, Dr. Multicast, the specialized handling of very large groups, and support for Google proto-netbufs. None of this is permanently gone, since we can pull it back in from Isis2 later, but for
now the system will be a specialist in high speed multicast and locking. I'm keeping the on-wire cryptographic options, but will be making cryptographic signatures optional, since they require touching the data and at RDMA speeds, the processor is way slower
than the speed at which data can be moved, so you might not want that feature (and RDMA is reliable in hardware, so you don't have to worry about data corruption in any case). But I'll still have DMC provide cryptographic protection for those who want it,
as Isis2 does now.
Another shift is that DMC will be an RDMA specialist, designed to run "only" on top of Jonathan's new RDMC module (provides reliable multicast over RDMA unicast or hardware RDMA multicast).
Words don't easy convey quite how different the Derecho experience and behavior will be, because RDMA multicast can hit 18Gb/s today on 20Gb/s hardware and is projected to move to 100, 200, 400 and then 1Tb/s in the coming years. With this new approach, DMC
will run directly, zero-copy, so your code would see that sort of speed. This is something like 17,000x faster than Isis2 on an IPMC network with an MTU of 8k, where communication is through the kernel and occurs over IP multicast and UDP with packets fragmented,
flow control, acks/nacks, retransmission delays, etc. So we are talking major, massive, disruptive speedup. Moreover, since RDMC is reliable (unlike IPMC), and allows arbitrary packet sizes, whereas IPMC has some sort of MTU lurking in the background, I can
get rid of Isis2 flow control, retransmissions, and packet fragmentation/reassembly. On the other hand, we will need a different form of receiver "pushback" to avoid having a sender dump petabytes into someone else's memory. So... big changes ahead!
Once DMC is solidly working, I'm likely to look at the features I've removed one by one. I'm guessing that the DHT would be the first thing to be restored (or maybe reimplemented, since the performance properties of DMC will be so totally different). I'll probably
set up a DMC distribution site, but I may do it on GitHub, since the C++ crowd seems to hang around there more than here on Codeplex, which is mostly a .NET C# crowd...
Given that we still have Isis2 and it works perfectly well, I'm unlikely to shoehorn every single thing back into DMC. Instead I see it as a new system, born from the old one but heading into a different world...
Let me know if you are curious about the hardware RDMA multicast. I can put you in touch with a vendor working in that space.