Update appliance for stackhpc.openhpc nodegroup/partition changes

[sjpb]

TODO: Update description below once below PR merges

Changes to cope with stackhpc/ansible-role-openhpc#183 which changed openhpc_slurm_partitions to openhpc_nodegroups and openhpc_partitions:

• common openhpc config
• skeleton templating
• caas infra and ansible
• stackhpc environment openhpc overides
• rebuild config - now automatic
• stackhpc.openhpc:validate.yml should get called from ansible/validate.yml

Note that:

• This removes the templating of partitions from opentofu; these are now all configured via ansible, using a new
  cluster_compute_groups variable templated into the inventory/hosts.yml file.
• A default openhpc_nodegroups is calculated in environments/common/inventory/group_vars/all/openhpc.yml. This creates a node group per key in the OpenTofu compute variable. This is usually what is required, but this variable will overriding if custom Node-level Slurm parameters/configuration is required (e.g. for GRES).
• A default openhpc_partitions is calculated. This includes one partition per node group, plus a rebuild partition if the rebuild group is active. This should not generally be overriden. Instead normal (non-rebuild) partitions should be modified using openhpc_user_partitions, which is the same format.

#665 is replaced by this.

[sjpb]

TODO: test this with adding extra nodes in stackhpc env
[edit:] done, see below.

[sjpb]

Testing in .stackhpc env with adding "extra" nodes:

1. Default configuration, i.e. just adding nodes into 2nd compute group:

$ git diff environments/.stackhpc/tofu/main.tf
diff --git i/environments/.stackhpc/tofu/main.tf w/environments/.stackhpc/tofu/main.tf
index 8d78401b..ea58d4b0 100644
--- i/environments/.stackhpc/tofu/main.tf
+++ w/environments/.stackhpc/tofu/main.tf
@@ -84,7 +84,7 @@ module "cluster" {
         # Normally-empty partition for testing:
         extra: {
             nodes: []
-            #nodes: ["extra-0", "extra-1"]
+            nodes: ["extra-0", "extra-1"]
             flavor: var.other_node_flavor
         }
     }
$ ansible login -a sinfo
RL9-login-0 | CHANGED | rc=0 >>
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
extra        up 60-00:00:0      2   idle RL9-extra-[0-1]
standard*    up 60-00:00:0      2   idle RL9-compute-[0-1]

Ok.

2. With the above, changing to an explicitly-configured partition covering both compute groups:

$ git diff environments/.stackhpc/inventory/group_vars/all/openhpc.yml
diff --git i/environments/.stackhpc/inventory/group_vars/all/openhpc.yml w/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
index 5aac5f8a..6dd18e9e 100644
--- i/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
+++ w/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
@@ -1,3 +1,8 @@
 openhpc_config_extra:
   SlurmctldDebug: debug
   SlurmdDebug: debug
+openhpc_user_partitions:
+  - name: hpc
+    nodegroups:
+      - standard
+      - extra
$ ansible-playbook ansible/slurm.yml --tags openhpc
$ ansible login -a sinfo
RL9-login-0 | CHANGED | rc=0 >>
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
hpc*         up 60-00:00:0      4   idle RL9-compute-[0-1],RL9-extra-[0-1]

OK.

3. With overlapping partitions:

$ git diff environments/.stackhpc/inventory/group_vars/all/openhpc.yml
diff --git i/environments/.stackhpc/inventory/group_vars/all/openhpc.yml w/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
index 5aac5f8a..9235f8e0 100644
--- i/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
+++ w/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
@@ -1,3 +1,11 @@
 openhpc_config_extra:
   SlurmctldDebug: debug
   SlurmdDebug: debug
+openhpc_user_partitions:
+  - name: normal
+    nodegroups:
+      - standard
+  - name: all
+    nodegroups:
+      - standard
+      - extra
$ ansible-playbook ansible/slurm.yml --tags openhpc
$ ansible login -a sinfo
RL9-login-0 | CHANGED | rc=0 >>
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
normal       up 60-00:00:0      2   idle RL9-compute-[0-1]
all*         up 60-00:00:0      4   idle RL9-compute-[0-1],RL9-extra-[0-1]

OK

[sjpb]

WIP: testing on azimuth as slurm-v21 - running, but need to redeploy slurm after last force-push

[jovial]

Looks sensible to me. Just need to wait for the openhpc role changes to merge so that we can update requirements.yml.

[sjpb]

Now fully tested on CaaS.

[sjpb]

Converted to draft because #668 must merge first.