[scenario] Anticipating port conflicts with a local active server queue on a shared computer

[wlandau]

My colleagues and I work with a traditional SGE cluster which has a small handful of login nodes and several compute nodes. Everyone shares the nodes, particularly the login nodes, but even the compute nodes (even when specific cores are reserved).

Let's say Alice and Bob each log into the login node with IP address 192.168.0.2 on the cluster's local network. Each uses a local active server queue to run a large pipeline.

Alice listens for servers on client ports 50000 through 50999 and launches 1000 servers to dial into them:

daemons("tcp://192.168.0.2:50000", nodes = 1000) 

Similarly, Bob listens for servers on client ports 50050 through 51049 and launches 1000 servers to dial into them:

daemons("tcp://192.168.0.2:50050", nodes = 1000) 

Alice and Bob both launch tasks which go to servers that dial into port 50100.

mirai_alice <- mirai("Alice's task")

mirai_bob <- mirai("Bob's task")

What will mirai_alice$data return on Alice's client? Will it return "Alice's task" or "Bob's task"? How do the servers know which client should get the result? (This is what I meant by "crossed wires" in #24.)

[wlandau]

I anticipate similar situations when a mirai user likes to use port 8888 but it conflicts with someone else's Jupyter notebook running from the same port on the same login node of the cluster. That is the main reason why I would prefer to check parallelly::freePort() and then only use that port. But maybe I am missing something about how TCP sockets work.

[shikokuchuo]

I had a brief look at parallelly::freePort() and it seems to allocate a random port, so fundamentally should not be any better than asking NNG to assign a free port (by specifying the port as 0).

As the client is using listeners, they actually bind to the requested port. So if it is already in use you are sure to get an error 10 - address in use or 16 - permission denied (say you if you try listening to SSH port 22).

[shikokuchuo]

There is no way for both Bob and Alice to listen on the same port.

If they run nodes directly they will get:

Bob:

daemons("tcp://192.168.0.2:50000") 
#> [1] "remote"

Alice:

daemons("tcp://192.168.0.2:50000") 
#> Error in socket(protocol = "req", listen = value) : 10 | Address in use

If they add the nodes argument, they don't see the above error as it is the background process that listens.

However for Alice, her background queue process errors and exits. The easiest way of detecting this is simply querying daemons(), which will show 0 connections instead of 1.

[wlandau]

So maybe it's not as big of an issue as you are currently anticipating?

It is starting to look that way. (And I just posted a question asking about which ports NNG considers ephemeral.) I haven't decided whether I will add or keep schedules other than mirai, but in crew I now plan to build mirai-based worker launchers for both traditional clusters and cloud systems.

[wlandau]

As for #33: in crew's autoscaling, I need some way to check which servers are currently running. Because of #33, I plan to go forward and do it on a case-by-case basis, e.g. handle$is_alive() for callr::r_bg() objects and qstat for SGE jobs. However, calling qstat too many times is a demand on the cluster's job scheduler, and the equivalent operation in the cloud is a call to the AWS API (which is cheap, but it's not free, and those API calls could could add up in cost). If there were an NNG/mirai-centric way to do this, then those costs could be avoided, and the check on the actual connection would be extra assurance that each server is healthy.

[shikokuchuo]

So maybe it's not as big of an issue as you are currently anticipating?

It is starting to look that way. (And I just posted a question asking about which ports NNG considers ephemeral.) I haven't decided whether I will add or keep schedules other than mirai, but in crew I now plan to build mirai-based worker launchers for both traditional clusters and cloud systems.

The answer is that it is platform-dependent. NNG does not define this itself. Some operating systems you can even configure this port range system-wide. So a la wikipedia I guess.

[mschubert]

To add my 2 cents: the clustermq master listens on one free port while the worker pool is active, so there is no "unsafe" behavior (maybe except for using a specific port that another program wants to use later on). So the clustermq solution is very similar to the solution in this discussion (use multiple incoming connections to one port, don't listen to a whole range; in fact it uses REQ/ROUTER sockets, similar to the REQ/REP that @shikokuchuo mentioned).

The main conceptual difference between clustermq and mirai as I see it, is that the former uses what @wlandau calls persistent workers, while the latter uses transient workers (that and the actual job submission).

[shikokuchuo]

To add my 2 cents: the clustermq master listens on one free port while the worker pool is active, so there is no "unsafe" behavior (maybe except for using a specific port that another program wants to use later on). So the clustermq solution is very similar to the solution in this discussion (use multiple incoming connections to one port, don't listen to a whole range; in fact it uses REQ/ROUTER sockets, similar to the REQ/REP that @shikokuchuo mentioned).

The main conceptual difference between clustermq and mirai as I see it, is that the former uses what @wlandau calls persistent workers, while the latter uses transient workers (that and the actual job submission).

@mschubert thanks for making it so clear for us. Apologies, I only had a very quick glance through your code and came across where you sampled 'n' ports from a range and mistakenly came to the above conclusion. By 'unsafe', I only meant that there appears to be no guarantee that the selected port is free, in which case it would error - now thinking about it I am sure you handle this possibility in other parts of your code, which is probably why you sample a range of ports to fall back on.