Nested mirais and daemon pools

[noamross]

Hi, I'm looking for some guidance on what I should expect from nested mirais and daemon pools. To explore, I've constructed this example:

library(mirai)
x <- mirai(
  {
    Sys.sleep(5)
    x <- mirai::mirai({
      Sys.sleep(5)
      x <- mirai::mirai({
      Sys.sleep(3)
        v <- 1
        cat(v, "\n")
        v
      })
      v <- mirai::call_mirai(x)$data + 1
      cat(v, "\n")
      v
    })
    Sys.sleep(5)
  v <- mirai::call_mirai(x)$data + 1
  cat(v, "\n")
  v
  }
)
call_mirai_(x)$data

This works fine - three background process are spawned and then shut down, and the result is 3. What if I want to see the output of all three? I first add

daemons(3, output = TRUE)

In this case, three daemons are launched and one dispatcher. However, they are not used for the inner two mirais, so eventually five daemons and a dispatcher are running and only "3" is printed from the outermost call.

Instead I added mirai::daemons(1, output = TRUE, dispatcher = FALSE) to the top of each mirai expression. This worked fine, I had three daemons and the output printed to my main console.

I wanted to see if the nexted mirais could share a daemon pool, so I set

daemons(3, output = TRUE)
urls <- nextget("urls")

and added mirai::daemons(url = urls, output = TRUE, dispatcher = FALSE) to the top of each call, with urls=urls as arguments. This gave me the error:

Error in socket(protocol = "req", listen = url, tls = tls): 10 | Address in use

OK, so it seems one can't re-connect to the same daemons directly. This makes sense as processes would need a shared dispatcher to avoid running into each other. Is there a way for nested calls to dial in to the same pool? Per #81, it seems that having a dispatcher tied closely to the host would make this challenging without there being some other, joint dispatcher.

I briefly experimented with extracting the compute environment from mirai::..$default, but I see that it has the same URLs and the socket and cvs which are external pointers, so it couldn't be passed to the child processes.

I do note that if one were to have a shared pool for this mirai with only 2 daemons, it would never resolve, because both would be occupied when the innermost call was executed. That's an inevitable edge case. (Will Landau pointed out that it could occur with crew if the worker crew were constrained.) Any nested approach will have some such limitations, but I still think a shared pool would be helpful.

[shikokuchuo]

Nested mirai are fully supported, so if anything doesn’t work as expected do let me know.

The ‘output’ argument just diverts the R stdout connection stream, and was a request from @dfalbel when implementing https://github.com/mlverse/callq. I have not really spent time on this feature, and consider it only of limited value as it is just for local daemons. The below sort of works and results in the same number of daemons being created as your original example, however, it is perhaps easier to just log all output to file via a call within the mirai.

library(mirai)
daemons(1, dispatcher = FALSE, output = TRUE)
x <- mirai(
  {
    Sys.sleep(5)
    mirai::daemons(1, dispatcher = FALSE, output = TRUE)
    x <- mirai::mirai({
      Sys.sleep(5)
      mirai::daemons(1, dispatcher = FALSE, output = TRUE)
      x <- mirai::mirai({
        Sys.sleep(3)
        v <- 1
        cat(v, "\n")
        v
      })
      v <- mirai::call_mirai(x)$data + 1
      cat(v, "\n")
      v
    })
    Sys.sleep(5)
    v <- mirai::call_mirai(x)$data + 1
    cat(v, "\n")
    v
  }
)
call_mirai_(x)$data

On the other hand, the daemon pool that can be used by nested mirai is an interesting concept.

The functions in mirai are designed to be modular, so you can structure whatever network topology you need, but they do assume only one host process. Allowing multiple hosts to connect to a dispatcher – was an intention for a while before higher priorities overtook it – you would effectively make it the permanent entity that listens to (binds to) the ports, and clients would then dial in just as daemons do. However it is not quite as simple as this, and there are other things such as the handshake, TLS certificates etc. to accommodate.

I think the direction of travel is that we will ultimately have FIFO scheduling built in (either at the library or nanonext level), so that could then leave dispatcher to perform a role managing a pool of daemons.

Note that a by-product of #81 (#81 (comment)) is that you could have different processes connect to a pool of persistent daemons in sequence (not all at the same time). This may be good enough in the meantime, depending on your requirements.

[noamross]

Thanks! This is about where I got, and output = TRUE works fine for me as my current use case (#90). #81 is helpful, too, for a totally separate application (managing a background server across sessions that I previously used system2 for). I'm in no rush needing a shared dispatcher, but I'll keep an eye out for it :)