Evaluate AsyncSSH as a replacement for Paramiko + asyncio thread executor

[nchammas]

Right now Flintrock doesn't really use the full power of asyncio. We cheat by making regular SSH calls using Paramiko in asyncio's thread executor. So all we get out of asyncio right now is perhaps a nicer API to the threading library.

The real reason Flintrock uses asyncio is that, as an orchestration tool, it manages remote resources. That means, fundamentally, Flintrock's main job is to do I/O against multiple servers across the network as quickly as possible. Most of Flintrock's interactions with the cluster nodes it manages are carried out over SSH. (The major exception is perhaps the work it does talking to the cloud provider.)

This use case sounds exactly like what asyncio is for.

AsyncSSH is a library that offers SSH over asyncio. It would replace both Paramiko as well as our use of the asyncio thread executor. More importantly, it promises to offer asynchronous SSH without the burden of threading.

I've had my eye on it for some time and have experimented with it on several occasions. The API is clean and the maintainer is active and responsive.

I want to seriously evaluate AsyncSSH as a replacement for Paramiko + thread executor, focusing on the following factors:

1. Does it make Flintrock easier to maintain, or at least not more complicated to maintain?
2. Does it make cluster operations, especially cluster launches, faster? How much faster?
3. What is the risk to depending on this relatively new library that does not yet have as much adoption or popularity as Paramiko?

[nchammas]

FYI: My work on this will have to wait until I can get through the issues described in this comment.

[nchammas]

I gave this a shot some time ago in #77 and didn't succeed. It's probably worth another shot now since AsyncSSH has matured quite nicely since then, however I don't feel the payoff justifies the cost of doing this work. Flintrock works well as it is, and there are plenty of other things to work on with a higher payoff, whether from a maintenance perspective, like #70 or #157, or a user perspective, like #27.