Isis2 on Infiniband: 4Gbit/s even on relatively slow nodes...

Mar 22, 2014 at 1:18 PM
With help from Jonathan Behrens (I should almost say that Jonathan, with help from me...) we've created a new plug-in called ib.dll that will help Isis2 take full advantage of Infiniband, which is a kind of ultra-fast network interconnect technology used widely on HPC clusters. We've tested only on Linux but in fact the approach "should" work on Microsoft Windows platforms too.

The biggest win is seen for the OOB file replication code -- the "out of band" memory-to-memory copying subsystem in Isis2. When using this, you create a large file (or create a memory-mapped file) and put your data into the memory-mapped area. Then you ask Isis2 to "rereplicate" the file to other locations where your service is running. The system sets up to do the closest thing it can to a multicast DMA from source to destination(s). I should mention, though, that while we genuinely use IP multicast on Ethernet, Jonathan and I haven't added that piece of logic for Infiniband yet, so right now everything is done with unicast. Coming soon.

Anyhow, with this new capability on Infiniband, we configure the network to do a DMA transfer directly between the memory-mapped area on the sender into a memory-mapped area we create for the incoming data on the receiver. We've clocked this at 4Gbits/s on some pretty slow nodes (IB itself can hit 20Gbit/s) and I plan to retest on beefier systems at LLNL. I bet we can get close to the full 20Gb/s rate on some hardware configurations. When you compare that to sending data through normal Isis2 multicast (currently around 10-20 Mb/s for similar uses) you can see that we're getting a huge win with Infiniband -- potentially 1000x!

Isis2 on IB will use IB communication for its own data messages too, but because of all the costs of marshaling in and out of C#'s object representation format, we pay a CPU cost that gets skipped when just moving data via DMA without any form of "data touching" except by the IB device. So this 1000x difference in speed may drop a little, but I bet that it will remain 100x or more even with a lot of speedup to Isis2 itself.

To use it, just compile Isis2 on a platform that has Infiniband NICs and, if needed, tell the system to use those network interfaces via ISIS_NETWORK_INTERFACES. Then make sure that ib.dll is in the load-library search path so that when we try and access the dll, it can be found.