Future: Allow for Client based API interfaces with remote clusters

[chrisjsewell]

Motivation

The current API, for interfacing with compute resources (local computer, remote cluster, HPC, etc) is principally split into two components:

1. The Transport class handles interfacing with the computer's filesystem
2. The Scheduler class handles interfacing with the computer's (calculation) job scheduler

This works well, for "de-duplicating" the many-to-many mapping of:

• Interfacing with (a) the local computer or (b) interfacing to a remote via SSH
• Interfacing with any of the possible schedulers on these computers (SLURM, PBS, SGE, ...)

However, there is currently a coupling between these two interfaces, since the Scheduler requires that the Transport can send BASH commands directly to the computer.

This split of components and requirement on that the Transport can send BASH commands is blocking for Client APIs, such as FirecREST, that tightly couple interaction with both the file system and the scheduler, and which have a different API for interacting with the Scheduler.

Desired Outcome

AiiDA should allow for compute resource plugins, which provide collectively filesystem and scheduler APIs, and which do not rely on the sending of BASH commands.

The roadmap item can be deemed closed, once an AiiDA FirecREST plugin has been fully realised.

Impact

FirecREST and, more broadly, REST based APIs for interacting with HPCs are a growing component of scientific compute clusters.
This is particularly critical in emergence of two-factor authentication (2FA) protocols.

Having an interface for such clients, would help future-proof AiiDA going forward.

Complexity

Due to the current code architecture in AiiDA, it is certainly not trivial to refactor it and achieve this.
Certainly, this will take a number of months to achieve:

1. Put the initial architecture refactoring in place, to allow for such a plugin, and
2. Implement and stress test such a plugin

Progress

Work is ongoing across:

• 🔃 REFACTOR: Move Scheduler/Transport behind Client API aiida-core#5901, to implement the changes in AiiDA
• https://github.com/chrisjsewell/fireflow, to prototype and test the FirecREST client API, outside of AiiDA
• https://github.com/aiidateam/aiida-firecrest, to eventually develop a fully formed AiiDA plugin

[khsrali]

Thanks @giovannipizzi
I believe this is addressed by @sphuber on aiidateam/aiida-core#6043
On plugin level, main functionalities of aiida-firecrest already works with that PR. However, since that is a vulnerable change I'm manually testing with SSH client, to insure its functioning as expected.

Will report back here, once the PR is merged.