SHPC Integration into OHPC

[georgiastuart]

This is something I'm working on so SHPC integrates better with our cluster tech stack. I wanted to post about it here for feedback on the overarching plan and see if it would be something that could be an "accessory" to SHPC once it's polished (nothing impacts core behavior and it would likely go in its own repository).

OpenHPC is a collection of building blocks and a design philosophy for deploying HPC clusters. OHPC provides RPM-based software arranged into a particular structure that is easy to deploy across nodes in a cluster (e.g., apps are in /opt/ohpc/pub/apps). OHPC is very opinionated with how things are deployed to make sure software is consistent and compatible.

I think SHPC can fit into the OHPC ecosystem quite easily and provide another avenue for integrating container-based software into OHPC clusters. In addition, there are a LOT of OHPC based clusters out there, and easy deployment for an OHPC system would mean a wider audience for SHPC. Here are the aspects that I've thought of so far.

• RPM-based deployment of SHPC. I have a working RPM for deploying SHPC on our clusters already, though it's a bit "janky" (technical term ;) ). This needs to be cleaned up significantly.
• Integration of SHPC views into OHPC moduledeps. OHPC provides several combinations of compilers and mpi stacks (e.g., gnu9 with openmpi4). On our current clusters, we are manually loading views for various compiler / mpi stacks, but loading the views can easily be stuck in the appropriate lmod files for the compiler / mpi stack (like here). Unfortunately those lmod files are created from the official OHPC RPMs so it's not really possible to make the modification automatic, so it may be easier to put the view logic in the SHPC rpm and have builds for different compiler / mpi stacks (though it's not actually compiler / mpi dependent). But then we have redundancy. Clearly I need to think about this point more :)
• Setup of user-space views. SHPC is already really friendly for doing stuff in user space, but I'd like to figure out how to have it automatically set up user-space views when pulling containers (that have no dependencies or an mpi or gpu dependency, for example). May require patching in the RPM.
• OHPC friendly registry. I'd like to be able to pull an appropriate mpi / gpu based container based on the loaded view without the user having to think about it. Ultimately, I'd like users to be able to easily control their own containers so our admins don't have to do it cluster-wide and so users don't have to set up shpc themselves and think about mpi / gpu compatibility.

Any thoughts? Glaring pitfalls?

I'll update this issue as I get things working!

[vsoch]

Oh this looks super cool @georgiastuart ! If you make an rpm recipe for shpc we can maybe add it alongside a repository here for others to use (or even make it's own repository?) I'd like to learn more about these module deps too - I think the shpc view.yaml (that you can store somewhere and then install from) might be a good starting point for a logical group of applications?

Can you keep this thread updated / show what you are doing at each step and we can help if needed - either talking something through or helping with some of the work?

The one blocker for shpc (that already existed before this) is really pulling containers for specific architectures. As far as I know biocontainers doesn't provide that yet, and even if we did, the registry pull would dictate what you get. I do think we eventually need to figure out a strategy (but it may not be directly related here).

[georgiastuart]

1. RPM: Yes! Once I have a cleaner spec file I'll upload the OHPC one to its own repo and make a more general one with standard system install locations for SHPC in general.
2. I'll check out view.yaml as a starting spot!
3. I'll definitely keep this thread updated and link to my working repo once I have it set up.

I think pulling containers for specific architectures is related here since my problem is pulling containers related to certain dependencies. I'll mull it over and see if I can find a starting solution to iterate on.

[georgiastuart]

Working on the OpenHPC compatible spec file here: https://github.com/georgiastuart/ohpc-application-spec-files/tree/add-python38/shpc

It works! But it's not particularly elegant. Feedback welcome in my internal PR here: georgiastuart/ohpc-application-spec-files#4

[georgiastuart]

Note: the shpc-ohpc spec brings a LOT of python requirements with it since the python38 install I made is bare bones. A system RPM can lean more heavily on system python38 packages. We choose not to do that on the clusters to not bloat the nodes.

[vsoch]

Awesome! Sometimes "it works" is really the most important part, and elegance can come later. Do you want to have a repository space here to maintain it?

[georgiastuart]

Starting to get the bandwidth to work on this again. I've been out on maternity since late December. I've also got a new situation that would benefit from pulling by arch! Is there somewhere that conversation is happening?

[vsoch]

Oh congrats @georgiastuart!! (and welcome back!) Right now the container pull doesn't have any understanding of arch beyond what the registry delivers based on the OS doing the pull. If you want to open a new issue to discuss that would be perfect.

[marcodelapierre]

This is such a beautiful news @georgiastuart , I am so happy for you and your growing family! :-)