Derecho progress report

Jun 19, 2015 at 1:29 PM
For all the C++ buffs out there, I thought I might share my insights gleaned from working on this port of Isis2 to C++. This has been both easier and harder than I expected.

[False start 1]: Back late last fall, I had hoped I could crowd-source a solution. But the proposals I ended up with wanted lots of money with little evidence of insight into how to do it.

[False start 2]: The Tangible C# to C++ conversion tool. In fact this is super useful, but I was hoping it could just do the translation. Nope.

(delay while distracted by Cornell teaching, research, etc)

[Apparently successful plan]: So here's what will apparently be the successful plan. I have a vacation coming up for 2 weeks but will carry this out when I get back.

1) Create a DMC.h file with my full external user API. This is key and needs to be done by hand.
a) Easy: expose the public methods and fields of my main classes.
b) Trickier: For methods like OrderedSend that have variable numbers and types of arguments, use recursive variadic templates in C++ to transform the argument list into a simple array of objects (call them DMCArg objects) that capture at compile time the typeid and some type traits that DMC needs to know about, such as whether this argument is a base type (int, string, etc), a pointer to an object in some registered class, a vector or array, and for the arrays, the dimensions.  You capture this with macros from type_traits.h.  These are C++11 features and actually some of them, like dimensions of 2-D arrays, are very tricky to figure out.  So there may be some small limitations here compared to Isis2 in C#.  But because I'll do this at compile time, the big benefit is faster runtime behavior.
c) Reimplement the overloaded += operator used to attach upcall handlers to the g.ViewHandlers and g.Handlers[] lists that DMC will use to dispatch upcalls. Again, the main trick is to capture the type information of the provided methods/lamdas. And again, there is a tricky way to do this with recursive variadic templates and type_traits.h
... as an aside, I've spent the last two weeks learning to do this step and experimenting to convince myself that this is all going to work, and I needed help from experts once or twice. Matthew Milano, a Cornell PhD student who works in this area, was invaluable. But folks on StackOverflow were helpful too.
2) OK, so now we need a DMC.cpp that matches this DMC.h. For this, you actually do start with the Tangible C# to C++ translation tool, and tell it to create a single .h file as its output. But then you rename that file to have a .cpp extension because we won't use it as a .h -- it will become the actual library.
a) Since all the variable argument list, variable types stuff will be handled with variadic templates, it isn't all that useful to keep the generics that Isis2 currently uses. So before doing this translation, it helps to remove all those generics. Internally, Isis2 was actually working mostly with a form of typeids (the C# type names, actually). So I edit Isis2 into DMC.cs and in doing that, I also replace the generics with code that wants things to be of object type, and wants the typeid passed in, or perhaps a 1-d vector of DMCArgs and the size of that vector. This means that Tangible won't try to create variadic templates on its own.
b) Most of the .NET features I use have direct equivalences in C++11. For example, I use List<x> in C#/.NET; one uses std::vector<x> for the same purposes in C++11, and foo.add(something) just becomes foo.push_back(something), etc. Equivalences include:
   b-1) List<x> becomes std::vector<x> or std::list<x> (double linked) or std::forward_list<x> (single linked), as appropriate
   b-2) Dictionary<x> becomes std::unordered_map<x>
   b-3) Sorted_List<x> becomes std::map<x>
   b-4) typeof(x) is replaced with typeid(x) but the trick is to have the typeid(x) evaluated at compile time, not runtime, and to pass the typeid values around internally as nonces -- just numbers that can be compared for exact match.  So I end up completely eliminating any other use of typeof(x) at runtime except for these type match operations.  Obviously this imposes some limitations where people might wish to use fancier forms of subtyping and inheritance in objects passed within DMC messages or delivered to callbacks, but I can actually be fairly sophisticated about it...  In the end it will look a lot like Isis2 (which is also kind of limited compared to what full C# would allow, for the same reasons)
  b-5) The "register type" API has to evolve slightly: now any type you register will need to support a particular interface (namely, your class will have to have the proper constructor running with a byte* argument, and a ToBArray method, and also a ToString method).  In C++11 I can't support [AutoMarshalled] but it looks as if C++17 might actually be able to do so.  This is because while C++11 has annotations, like [[something]], they aren't currently user-extensible.
c) Then there is a bit of work to do internal to DMC.cpp.  In particular, I have to use their safe pointer scheme (this is mostly automatic but in a few places, when doing callbacks, I would need to call the proper std::safe_pointer constructor), and then change to C++11 threads with guarded mutex objects where Isis2 currently uses C# locks, etc.  Similarly, I need to use getifaddrs/freeifaddrs to scan the network interfaces, and gettimeofday to get the time.
d) There are a bunch of situations where I need to know the length of an object passed in from the outside.  For the objects that are of type byte* and were allocated with malloc, I'll need a way to know their lengths, and of course the malloc/realloc/free library lacks a getlength operation.  So for this particular case I may need to put something in DMC.h that you would use as a wrapper: DMCArray(ptr, len), for example.  A shame, but C and C++ have never really believed that malloc should maintain and reveal the size of dynamically allocated objects.
So that's probably a month of editing when I get back from vacation but it appears to cover the entire job, and the cool thing is that DMC will then have nearly the identical API to the one Isis2 has. Far less runtime copying will occur so this version will probably be a bit faster than the C# version. But the main merit to using C++11 for DMC is that I can then work with network interface cards that allow you to drop code onto the card and we can play with DMC right on the NIC (should let me do things like blazingly fast distributed hash tables via the built-in DHT feature, and an amazingly fast lock manager). I can also pursue the very tight binding to RDMA I've mentioned in prior postings, so that DMC might be able to move data at optical line speeds. For people using multicast to replicate large objects this would be a dramatically fast capability, maybe 10,000x or more faster than what Isis2 in C# is able to do though its normal C# APIs, and maybe 100x faster than what you can pull off using Isis2 with the OOB features.

If I'm right about this, I would then probably deprecate the OOB API. Initially I'm thinking I will keep it for DMC just in case when I've done all of this, the core OrderedSend overheads remain high compared to the pure speed of the RDMA data path.

Then going forward I would maintain both systems; we won't abandon Isis2 in C#. DMC would just become a second option for people working in unmanaged C or C++.

Feel free to post questions, comments or suggestions!