Getting started - multiple unicast hosts on the same machine

Mar 11, 2015 at 9:18 AM
It's probably a silly question, but from reading the docs, I'm not sure if it's possible to have different unicast hosts running on the same machine as the port numbers are assumed to be global, and you apparently can't specify a port for each host in the ISIS hosts list?

Functionality looks fantastic by the way. I've been looking for something like this for ages for .NET
Mar 11, 2015 at 1:14 PM
Edited Mar 11, 2015 at 1:15 PM
You are correct, but the handling of this is automatic. If you run in the normal (default) mode with a shared multicast socket Isis2 will pick its own unicast socket numbers automatically and all you need to do is to unblock the built-in firewall by saying "yes" when the pop-up first appears (when you first launch your program). Then you can run, say, 20 instances which would each have its own command window or GUI window.

On Linux you have to configure the firewall through the iptables command, but the idea is the same. See the documentation on how this is done.

Or if you insist on using UNICAST-ONLY mode for some reason, then the first instance of your server that gets launched will end up exclusively grabbing the unicast sockets with the default port numbers. Other instances will automatically pick other port numbers, then connect to that first instance. This works fine but you end up with a non-replicated version of the built-in ORACLE service, which is a group running internal to Isis2 on your nodes. In this case killing the first instance you launched causes the whole application to shut down. The reasoning is that if an ORACLE instance isn't using the proper port numbers it can't be contacted, so in this mode, where only one instance can be using the right port numbers, only that one process can play the ORACLE role. That issue wouldn't arise in the normal multicast mode, where Isis2 can use multicast port numbers -- Windows and Linux allow multiple processes to share the same port numbers in that case.

Have fun!
Mar 11, 2015 at 3:13 PM
Hi birman,

Thanks, I think I understand. So if I want to run on a cluster that doesn't allow UDP across machines then I set up a service on each machine running on a known port that can be the oracle and then for other processes on that machine just let it pick the port (i.e don't override it for each process?). I assume I can have more than 3 oracles in ISIS_HOSTS and it will just pick three from those?

I may have more questions as I'm trying out some simple examples and they're not quite working as I'd expect so I may be missing something. ISIS2 seems an ideal opportunity for us to avoid some more "overblown" solutions like Coherence or NCache, so I'd really like to get a good understanding of it!

Thanks,
Ben Young


Mar 11, 2015 at 3:41 PM
Edited Mar 11, 2015 at 3:42 PM
You can call me Ken!

Anyhow, we used to support a TCP only mode in addition to everything else, but I ripped that code out two years ago -- I felt that the implementation was ugly and unstable.

I was planning to reimplement it but never did so because it turned out that the UDP tunneling support on systems like Amazon AWS is actually fairly fast. So although they "don't support UDP" actually they really do support it. It turns out that you don't often see environments that flat out don't support UDP. These days I do have an RDMAVIAUDP option you can set, but only for the transfer of large files via a reliable TCP-style session (intended for RDMA over converged Ethernet, like Intel iWarp). This is very fast and we have actually been working on it here at Cornell and think we can speed it up even more, maybe by 3x or 4x (for example, I currently used SHARED mapped memory, but we've discovered that Linux always does a disk I/O in such cases, unlike Windows where this mode is fast. So I should actually change my code to use the UNSHARED mapping mode whenever possible, eliminating this background cost).

So right now, Isis2 will always use UDP multicast and/or UDP unicast and there is no TCP-only mode available.

Getting back to your question, this actually adds up to: you can't use Isis2 in such a case. You are forced to open the firewall for UDP traffic specifically between Isis2 server instances and if you really can't do that, Isis2 isn't an option for you. You definitely can disable the use of UDP multicast, if the real policy is that UDP unicast is ok but not UDP multicast, although Isis2 performs best with UDP multicast.

The same UDP tunneling over TCP that Amazon AWS is really using could be an option for you. It was a freeware product of some kind -- a server that basically treats UDP as a kind of pub/sub feature. So one way to work around this is to track down that product, get your people to let you deploy it, and then the kernel thinks it is supporting UDP with a firewall that allows some tunneling, but in reality these guys have an encapsulating device driver that really maps UDP down to TCP to a server, which sits on a fast machine and just relays packets in the obvious way.
Mar 11, 2015 at 3:57 PM
Thanks Ken,

I'm not sure how much of an issue UDP is in practice (I don't work in the deployment side of things), but I do work in financial services, so I know they do tend to lock them down pretty tightly! I imagine if we said we needed it, it would be possible for a limited subnet or something.

Anyway, I'm just in the trying things out stage at the moment, so I'll have to see how things go. We're looking at things like Zookeeper and Ignite too, but I like the fact that this is native .NET very much (The .NET world seems short of some of the things in the enterprise area that Java seems to have grown, I wonder if people are nervous to develop things as they think Microsoft will swoop in and develop their own, but that's a side issue).

Thanks,
Ben


Mar 11, 2015 at 4:54 PM
Edited Mar 11, 2015 at 4:57 PM
Oh, this isn't special to Isis2. The Oracle commercial cluster product has the exact same need, and probably the exact same choices: open the firewall for those particular machines to talk to each other (you can install very specific iptables rules: "Allow Isis2 on a.bankofjapan.com to talk to Isis2 on b.bankofjapan.com via UDP on port 9876") or tunnel via that TCP thing I mentioned (I really should hunt it down, but popular open source and stable). The version called "We can't use Oracle at the Bank of Japan" isn't a serious option.

So it won't be such an issue. And keep in mind: Zookeeper needs to talk between its servers too. So would spread, or OpenSplice, or anything of this kind. The IT folks will be able to do a customized firewall policy for this. They probably moved to OpenFlow SDN precisely for this reason in the last year and are becoming experts.

One issue you will run into is that if you run more than one Isis2 application on the same machine, you can't predict the port numbers (obviously) so you end up needing to say "Allow Isis2 on a.bankofjapan.com to talk to Isis2 on b.bankofjapan.com via UDP on port *"). This is perhaps harder to convince the IT people to agree to do, since it leaves a much larger hole in the firewall. But if you only run one server using a given instance of the system on each machine, you can wire down the port numbers and you won't have any issue at all. Isis2 actually uses 2 ports per machine, by the way: one for UDP messages and a second one for acknowledgements. The numbers are assigned consecutively: if you tell us to use 9872 for the first one, we use 9873 for the second one.

Then you can also control the IPMC port range we will use, and how many IPMC ports in total we'll use. So that again allows pretty precise firewall rules to be designed.

Zookeeper is a cool and popular technology. Isis2 is way more flexible, but if you are aiming for something pretty specific that matches Zookeeper (e.g. you want files with built-in mutual exclusion), I would use Zookeeper for that. You can do the same thing in Isis2, but you have to write some code, and why write code to end up at the same place as Zookeeper if this is all you plan to do?

Internally, they (the Zookeeper guys) actually use protocols that originally were based on Paxos but over time, evolved to be like the ones in Isis2, except less fancy. Isis2 is very sophisticated at that protocol level and has a huge range of options, if you get into it. So Isis2 is the better choice for communication-oriented systems that do a lot of messaging and either don't maintain state in files, or do so just for checkpoint recovery but where the file system is a backup resource, not the main deal. But of course you need to understand precisely what your needs are to pick the fastest option for your purposes.

Also, the developers of Zookeeper are no longer so active; the system is stable and kind of mature and not evolving a ton at the basic level. So whereas I'm trying to figure out how to drop Isis2 onto a network card, for example, I doubt that you would see the core Zookeeper guys doing that -- the team seems to have sort of split up and gone different ways. But who knows? Anyhow, I love what they did. Zookeeper rocks. (But they should admit that basically, they ended up going with a virtual synchrony model...)
Mar 12, 2015 at 9:26 AM
Thanks for all the info Ken, it's very cool stuff. I'll let you know how our testing goes.

I've opened one issue with the DHT stuff. I'm sure it's something I'm doing wrong but I thought I was following the examples.