Harden wrapper scripts with explicit `UD_*` variables + environment profiles (wrappers remain source of truth)

[mvreeuwijk]

We need to make wrapper scripts robust across multiple clusters without changing the principle that wrappers are the source of truth. The biggest gap is that real cluster setups require multi-line environment instructions (modules + venv + overrides), and that context is currently scattered and hard-coded.

Examples of the problem

ICL cluster (current reality):
module load intel/2025a netCDF/4.9.2-iimpi-2023a netCDF-Fortran/4.6.1-iimpi-2023a FFTW/3.3.9-intel-2021a CMake/3.29.3-GCCcore-13.3.0 git/2.45.1-GCCcore-13.3.0
module load Python/3.9.6-GCCcore-11.2.0
source /rds/general/user/$USER/home/udales/.venv/bin/activate

ARCHER2 (different stack):
module load epcc-job-env
module load nco gsl
# different MPI launcher, different compiler stack

Local workstation (no modules):
# system Python + local venv
python3 -m venv .venv
source .venv/bin/activate

Right now, each script encodes part of this, and users must remember multi-line setup steps that are not centralized.

---

Proposal

1) Environment profiles

Add profiles with explicit multi-line instructions:

tools/env/profiles/icl.sh
tools/env/profiles/archer.sh
tools/env/profiles/common.sh

Each profile defines:

• module stack
• Python module + venv activation
• MPI launcher quirks
• scratch location

2) Environment detection

Add:

• tools/env_detect.sh — prints hostname, module tool, mpiexec flavor, Python header path
• tools/env_profile.sh — selects a profile (or respects UD_SITE) and exports defaults

3) Explicit UD_* overrides

Scripts should honor:

• UD_PYTHON
• UD_VENV
• UD_BUILD
• UD_MPIEXEC
• UD_SCRATCH
• UD_MODULES
• UD_SITE

So a full invocation can be:
UD_SITE=icl ./tools/build_preprocessing.sh
UD_VENV=/rds/general/user/$USER/home/udales/.venv UD_PYTHON=python3
./tools/build_preprocessing.sh

4) Wrapper scripts remain the source of truth

We are not replacing wrappers. We are making their environment setup explicit and portable.

---

Outcome

• users get copy-pasteable environment setup for each cluster
• wrappers continue to be the canonical entry point
• fewer silent failures due to wrong module or Python