Replaced by new general release V2.2.2003 on July 16, 2015
I'm releasing V2.2.1962 as our newest general release, since it seems to work quite well. The main work has been on the "out of band" transfer layer (and the work isn't finished yet). But this stuff is really fast by now.
Note: I updated this release on March 16, 2015 to correct a minor bug in the DHT that arose in certain start sequences. Updated again on April 24&27, 2015 to fix additional minor issues seen when the system became overloaded during startup (my Cornell class was kind enough to generate some extreme overload cases and these revealed a bug). Then yet another small issue arose, leading to yet another update, this time on April 28. Regression tests and stress tests now seem to be solid again.
Use of the OOB code in V2.2.1962 involves understanding some parameter settings.
- By default, it runs over UDP multicast
- If you specify ISIS_UNICASTONLY=true you'll get pure point to point, but in two possible modes
- Mode 1: ISIS_OOBVIATCP=true (this is the default). A TCP connection will be used for each OOB file transfer. This is a bit faster than the Isis UDP option (not hugely so)
- Mode 2: ISIS_OOBVIATCP=false (needs to be explicitly set, e.g. using export in bash). Uses Isis UDP unicast connections to transfer your files.
- ISIS_USERDMA=true, ISIS_INFINIBAND=true (you need to set these; defaults are false, false): If you are on an Infiniband interconnect, and have the ib.dll file built and in a place where it can be found, does OOB transfers as big RDMA copies, DMA from memory-to-memory. You may need to email Ken (firstname.lastname@example.org) or Jonathan for help setting this up -- it isn't hard but it isn't properly documented yet.
Note: You'll ONLY need ib.dll (or ib.h and ib.c, from which one builds ib.dll) if you plan to use Infiniband or the new Fast Ethernet via ROE or ROCE (RDMA on Ethernet or RDMA on Converged Ethernet) feature, and obviously that only works if Isis is configured to use the Infiniband or Fast Ethernet ROE/ROCE network interface. So there is an easy but not automatic setup needed. Ken will be happy to help you get past that step. For this version of Isis2, there should be no real difference between ROE and ROCE, by the way.
For people who don't plan to play with these OOB file transfer features ib.dll won't even be loaded by Isis2, so you don't need to download the file or put it anywhere. Everything will work just fine. On the other hand, you won't get the blazingly fast OOB file transfers that we talk about so much lately.
All of this really pays off in terms of performance. Isis2 is WAY faster when using OOB in smart ways. Of course you'll need to learn how that works. But Weijia Song will soon make OOB nearly invisible: he's building a file system that does it for you, and a "command line" way of initiating OOB transfers too.
Note: In March 2015 I also modified the "research prototype" comment to tone down the warnings. By now the system seems very robust and if people are prepared to do the needed quality assurance for the applications they build, I think it can safely be used in any setting where it is safe to use computer technologies of other kinds.