Isis2 Cloud Computing Library
Isis2 is a new option for cloud computing that can enable reliable, secure replication of data even in the highly elastic first-tier of the cloud. Isis2
is a project of Ken Birman at Cornell University. Although the system started out as a data replication technology (groups of programs that can share updates), in 2013 Isis2 became much more big-data oriented.
The need for high assurance has never been greater: with the trends towards data centers of all sizes and shapes (ranging from small racks of just a dozen or two machines to massive cloud computing data centers with hundreds of thousands of them), developers of modern computing systems need to target the Web, employ Web Services APIs, and yet somehow ensure that the solutions they build can scale out without loss of assurance properties such as data security (who knows what the other users of the cloud might be doing… or what might be watching?), consistency and fault-tolerance.
library was built to help you solve this problem in an easy way, closely matched to the style of development used for standard object-oriented applications that use GUI builders.
Isis2 for people who don't want to use C#
I'm in the midst of a push now to make Isis2 more and more useful without ever programming against the Isis2
API. You'll find that I recently posted a tool written by one of our students, Kaveri Chaudry, that offers a small server in which she runs Isis2
(e.g. the server is a kind of stand-alone version of the library). You run one copy on each node in your cluster or HPC system. Then you can access the Isis2
OOB interface from programs on the same machines where you have copies of the server. Thus if you had 100 machines using Hadoop, you could put this server in the background and now it becomes possible to use the Isis2 in-memory replication API to share mapped files between your applications.
There are two ways to talk to Kaveri's server: via a command-line program she wrote, or from a very thin library for C++ or C users, who would work purely in native C++ or C, without the .NET CLI extensions.
In the coming weeks, she and are extending this to also talk to the Isis2
locking layer and the Isis2 DHT. At that point we can provide the key Isis2 functionality to people who are working in totally unrelated languages and using tools like Hadoop. We think the mix could be very powerful and it will be entirely transparent to the Hadoop user with existing code that needs to keep running -- these new features will add to Hadoop but won't require mods to the way you do things normally.
Drop an email to Ken (firstname.lastname@example.org) if you have a good application for this and would be willing to be a beta user. Later in the spring we'll do a user manual aimed just at this community, and it won't have a line of C# in it, or even mention that Isis2 is a library written in a strong programming language known only to an elite few...
A few quick comments aimed at those revisiting the site who might be wondering what's new.DHT
We added a distributed sharding scheme (a DHT) that integrates with LINQ and has the remarkable feature that it can do consistent queries even on rapidly changing data, and yet scales as well as any sharding scheme of which we're aware. This takes Isis2 squarely into the world of big data, and by using LINQ in conjunction with this new technology, you can even do database-like programming in a new and rather elegant way, especially if you combine your queries with our scalable "aggregation" layer. Out of Band files copying
A second new mechanism, added in summer of 2013 and just being released as of August is a file copying and "shuffling" tool. The basic idea is that you have a cluster of machines and are running, say, Hadoop. As part of your computation you generate intermediate results and store them as memory-mapped files (this is popular lately). With the new tool, you can declare these to Isis2 and then ask the system to move them around: maybe you had a copy at A and want copies added on B..F. You want to move some other file outright: it was at D and you want it now at H and J. The number of copies can be large and the files can be huge. Isis2 will concurrently carry out this file movement actions at very high speed using a form of multicast that is reliable and (if desired) secure. The speed is extremely good; we seem to be able to peg even a large cluster network at near 100% of capacity, with very low overheads.
Right now these features can only be used by linking to the Isis2 library, but within a few weeks we plan to release a "daemon" that will run Isis2 and be accessible via simple RPC commands and even via command line. At that point you can launch the daemon on your cluster nodes and use these features of Isis2 without needing to touch the library itself.Java
A number of people have asked how would use Isis2 from Java. This article can help:
As you will see, one option is to use a system called J#. With it your Java code runs without changes on .NET. On the other hand, you end up with a solution that won't port back to Linux because Mono lacks a J# compiler.
For this reason, the "best" option is to use a package called the JLCA. Microsoft no longer actively supports this, but you can find old Windows images with JLCA on them. You only need to use it once. You run it on the Java code, and it magically generates C# code that calls the native .NET library methods. This gives better performance and better interoperability. But the resulting code behaves in the identical manner to the original Java. In fact it even looks nearly identical: JLCA works because C# and Java are really almost identical languages. Thus not that many things need to be changed, and JLCA can do the necessary edits with surprisingly little impact on the look and feel of your code.Live Distributed Objects + Isis2
Last, I want to point people to the LDOForGridCloud https://ldo4gridcloud.codeplex.com/
project page. This revives a very comprehensive tool for building collaboration applications, that we call Live Distributed Objects. It offers a drag-and-drop way of creating visual GUIs that support point and click interactions and can capture data from various places. LDO used to have a dedicated but limited TCP multicast: an event at any object was applied to the others by relaying it through N TCP links from the place where the event happened to the other replicas. Now LDO also supports Isis2 multicast. It works on Windows or on Linux via Mono, just like Isis2, and we have some nice demos and online documentation. So check it out! Note: The basic Live Objects distribution no longer builds properly. But the one on this other project page does work. On the other hand, the documentation on the Live Objects distribution page remains valid.
So... check it out and let us know what you think!
Please see the Documentation
page for video summaries, release description, as well as complete documentation. There are also some self-test modules to evaluate your understanding of the videos, focused on the core Isis2
For compilation under Linux environment, please see the Compile
page. With proper setup, Isis2 can run on Windows, Linux, Amazon EC2 or Eucalyptus-style virtualization platforms, Azure... you name it!