diff --git a/docs/developers_guide/spack.md b/docs/developers_guide/spack.md index de1ce8fe..1314b12e 100644 --- a/docs/developers_guide/spack.md +++ b/docs/developers_guide/spack.md @@ -31,7 +31,16 @@ Each YAML file describes a Spack environment for a particular combination of com For example: `chicoma-cpu_gnu_mpich.yaml` These files are Jinja2 templates, allowing conditional inclusion of packages -(e.g., HDF5/NetCDF, LAPACK) based on user options. +(e.g., LAPACK) based on user options. + +Machine-provided HDF5/NetCDF packages should now be listed unconditionally in +the YAML templates. Downstream packages can opt out of them, or any other +machine-provided external such as `cmake`, with `exclude_packages`. Mache +filters the rendered YAML after Jinja expansion and removes matching: + +- `spack.specs` entries +- `spack.packages.` external-package sections +- matching provider specs under `spack.packages.all.providers` ### Typical External Packages @@ -104,21 +113,48 @@ This pipeline greatly reduces maintenance and prevents drift between Mache and E3SM’s authoritative machine configuration. In most cases, you do not need to author or maintain shell script templates in Mache. +### Package opt-outs and deprecated HDF5/NetCDF flag + +The preferred downstream-facing interface is `exclude_packages`, available in +the public `mache.spack` APIs and in `mache.deploy` runtime config. + +Examples: + +- `exclude_packages=["cmake"]`: build CMake with Spack instead of using the + machine-provided external package and module setup. +- `exclude_packages=["hdf5_netcdf"]`: opt out of the machine-provided + HDF5/NetCDF bundle. +- `exclude_packages=["hdf5", "netcdf-c", "netcdf-fortran", + "parallel-netcdf"]`: the same bundle, expressed explicitly. + +`e3sm_hdf5_netcdf` and `include_e3sm_hdf5_netcdf` remain supported as +deprecated compatibility flags in public `mache.spack` APIs. New code should +prefer `exclude_packages`. + ### When to add a template override Only provide a small override in `mache/spack/templates/` if you need to: - Apply an adjustment that’s not appropriate for the shared E3SM CIME config (machine-local quirk, temporary workaround, etc.). -- Add conditional behavior toggled by - `include_e3sm_lapack` or `e3sm_hdf5_netcdf` (both exposed as Jinja booleans - in templates) that cannot be expressed in the CIME config. +- Add conditional behavior toggled by `include_e3sm_lapack` or by package + helper functions such as `use_system_package('cmake')` and + `use_system_packages('netcdf-c', 'netcdf-fortran')` that cannot be + expressed in the CIME config. Note: `include_e3sm_hdf5_netcdf` remains supported as a deprecated alias for - `e3sm_hdf5_netcdf` in public `mache.spack` APIs. + `e3sm_hdf5_netcdf` in public `mache.spack` APIs, but new shell overrides + should prefer the package helpers over `e3sm_hdf5_netcdf`. Templates are Jinja2 files and can use the same conditional logic as YAML -templates. +templates. For shell overrides, the most useful helpers are: + +- `use_system_package('')` +- `use_system_packages('', '', ...)` +- `render_env_var(name, value, shell_type)` + +These helpers let shell overrides stay package-oriented even when the machine +module names do not match Spack package names exactly. ## Testing @@ -134,4 +170,3 @@ After adding or modifying YAML templates (or an exceptional shell override): - For the non-spack aspects of adding a new machine, see [Adding a New Machine to Mache](adding_new_machine.md). - For more details on Spack external packages, see the [Spack documentation on external packages](https://spack.readthedocs.io/en/latest/build_settings.html#external-packages). - diff --git a/docs/users_guide/deploy.md b/docs/users_guide/deploy.md index e13a22ec..45aecb5e 100644 --- a/docs/users_guide/deploy.md +++ b/docs/users_guide/deploy.md @@ -176,9 +176,96 @@ Important settings: - `spack.spack_path`: required when Spack support is enabled and no hook or CLI override provides it, unless the user disables Spack for that run with `--no-spack`. +- `spack.exclude_packages`: optional list of machine-provided Spack packages + that the target software wants Spack to build instead. - `jigsaw.enabled`: optional. - `hooks`: optional and disabled unless explicitly configured. +Example: + +If a downstream package such as Compass wants to use the machine-provided +libraries in general but build its own newer `cmake`, it can set +`spack.exclude_packages` in `deploy/config.yaml.j2`: + +```yaml +spack: + supported: true + deploy: true + spack_path: /path/to/shared/spack + exclude_packages: + - cmake +``` + +With this setting, `mache deploy run` will: + +- remove the machine-provided `cmake` external from the rendered Spack + environment YAML, +- remove matching machine-provided `cmake` module loads and related + environment-variable setup from generated shell snippets, and +- let Spack build the `cmake` version requested by `deploy/spack.yaml.j2`. + +The package list in `deploy/spack.yaml.j2` does not need any special syntax +for this. You simply request the Spack package you want, for example: + +```yaml +software: + - "cmake@{{ spack.cmake }}" +library: + - "trilinos@{{ spack.trilinos }}" +``` + +The same mechanism can be used for the machine-provided HDF5/NetCDF bundle: + +```yaml +spack: + exclude_packages: + - hdf5_netcdf +``` + +This is the preferred replacement for the older `e3sm_hdf5_netcdf` flag. + +Another common pattern is to keep a machine-specific boolean such as +`[deploy] use_e3sm_hdf5_netcdf` in the target repository's machine config and +translate that into `exclude_packages` in a `pre_spack` hook. + +For example, if a downstream project like Compass has a machine config with: + +```ini +[deploy] +use_e3sm_hdf5_netcdf = false +``` + +then `deploy/hooks.py` can map that to the new mechanism: + +```python +from __future__ import annotations + +from typing import Any + +from mache.deploy.hooks import DeployContext + + +def pre_spack(ctx: DeployContext) -> dict[str, Any] | None: + updates: dict[str, Any] = {} + + exclude_packages = list(ctx.config.get("spack", {}).get("exclude_packages", [])) + + use_bundle = False + if ctx.machine_config.has_section("deploy") and ctx.machine_config.has_option( + "deploy", "use_e3sm_hdf5_netcdf" + ): + use_bundle = ctx.machine_config.getboolean("deploy", "use_e3sm_hdf5_netcdf") + + if not use_bundle: + exclude_packages.append("hdf5_netcdf") + + updates.setdefault("spack", {})["exclude_packages"] = exclude_packages + return updates +``` + +This lets existing machine-config policy continue to drive behavior while +moving the actual Spack selection onto `exclude_packages`. + ### `deploy/pixi.toml.j2` Required: yes @@ -214,12 +301,20 @@ Purpose: specs, `software` specs, or both. - Supplies the target-specific package list used when `mache` constructs Spack environments. +- Receives `exclude_packages` as a deploy-time Jinja variable, reflecting any + configured opt-outs of machine-provided Spack packages. Edit policy: - Safe and expected to edit. - Leave it empty if you do not support Spack yet. +In most target repositories, package opt-outs belong in `deploy/config.yaml.j2` +under `spack.exclude_packages`, not in `deploy/spack.yaml.j2`. The Spack spec +template is where you request the package versions and variants you want to +build; `exclude_packages` controls which machine-provided externals are +suppressed before concretization. + ### `deploy/hooks.py` Required: no diff --git a/docs/users_guide/spack/build.md b/docs/users_guide/spack/build.md index a8b09344..3f86ef51 100644 --- a/docs/users_guide/spack/build.md +++ b/docs/users_guide/spack/build.md @@ -55,6 +55,7 @@ make_spack_env( config_file=machine_config, include_e3sm_lapack=include_e3sm_lapack, e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, yaml_template=yaml_template, tmpdir=tmpdir, spack_mirror=spack_mirror, @@ -71,8 +72,16 @@ make_spack_env( - `compiler`, `mpi`: Compiler and MPI library names. - `machine`: Machine name (optional, auto-detected if not provided). - `config_file`: Path to a machine config file (optional). -- `include_e3sm_lapack`, `e3sm_hdf5_netcdf`: Whether to include E3SM-specific - LAPACK or HDF5/NetCDF packages. +- `include_e3sm_lapack`: Whether to include E3SM-specific LAPACK packages. +- `exclude_packages`: A package name or list of package names whose + machine-provided externals, modules, and related environment variables + should be removed so Spack can build them instead. For example, setting + `exclude_packages=["cmake"]` lets a downstream package build a newer CMake + than the system provides. +- `e3sm_hdf5_netcdf`: Deprecated compatibility flag for opting into the + machine-provided HDF5/NetCDF bundle. New code should prefer + `exclude_packages=["hdf5_netcdf"]` (or the individual package names + `hdf5`, `netcdf-c`, `netcdf-fortran`, and `parallel-netcdf`) instead. - `yaml_template`: Path to a custom Jinja2 YAML template (optional). - `tmpdir`: Temporary directory for builds (optional). - `spack_mirror`: Path to a local Spack mirror (optional). @@ -85,6 +94,36 @@ make_spack_env( - Generates and runs a shell script to create the Spack environment. - Loads any required modules and sets up environment variables as needed. +**Recommended pattern for downstream packages:** + +```python +exclude_packages = [] +if needs_newer_cmake: + exclude_packages.append("cmake") + +make_spack_env( + ..., + exclude_packages=exclude_packages, +) +``` + +To opt out of the machine-provided HDF5/NetCDF bundle, use either: + +```python +exclude_packages=["hdf5_netcdf"] +``` + +or the individual package names: + +```python +exclude_packages=[ + "hdf5", + "netcdf-c", + "netcdf-fortran", + "parallel-netcdf", +] +``` + --- ## `get_spack_script` @@ -116,6 +155,7 @@ spack_script = get_spack_script( machine=machine, include_e3sm_lapack=include_e3sm_lapack, e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, ) ``` @@ -133,6 +173,13 @@ The returned snippet is assembled in three steps: This design keeps Mache aligned with E3SM’s authoritative machine configuration and minimizes maintenance. +`exclude_packages` applies here too, so `get_spack_script()` removes matching +machine-provided module loads and environment variables from both: + +- shell snippets derived from `config_machines.xml`, and +- any package-local shell overrides in `mache/spack/templates/*.sh` or + `*.csh`. + **Usage in activation scripts:** ```bash @@ -167,6 +214,7 @@ mpicc, mpicxx, mpifc, mod_env_commands = get_modules_env_vars_and_mpi_compilers( shell='sh', # or 'csh' include_e3sm_lapack=include_e3sm_lapack, e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, ) ``` @@ -177,6 +225,9 @@ mpicc, mpicxx, mpifc, mod_env_commands = get_modules_env_vars_and_mpi_compilers( - `mpifc`: Name of the MPI Fortran compiler wrapper (e.g., `mpif90` or `ftn`). - `mod_env_commands`: Shell commands to load modules and set environment variables. +As with `get_spack_script()`, `exclude_packages` can be used to remove +machine-provided package setup from the generated shell snippet. + **Notes and usage in build scripts:** ```bash @@ -184,10 +235,10 @@ mpicc, mpicxx, mpifc, mod_env_commands = get_modules_env_vars_and_mpi_compilers( # Now safe to use $mpicc, $mpicxx, $mpifc for building MPI-dependent software ``` -- This helper produces a minimal snippet based on Mache’s template overrides - (when present). It does not auto-generate content from - `config_machines.xml`. If you need the full environment setup used by - downstream activation scripts, prefer `get_spack_script`. +- This helper uses the same shell-generation logic as `get_spack_script()` + but does not activate a Spack environment. It therefore includes the + machine-derived setup from `config_machines.xml` plus any matching Mache + shell overrides. --- @@ -211,9 +262,11 @@ mpicc, mpicxx, mpifc, mod_env_commands = get_modules_env_vars_and_mpi_compilers( - These functions are intended for use in deployment scripts, not for interactive use. +- `e3sm_hdf5_netcdf` and `include_e3sm_hdf5_netcdf` remain supported for + backward compatibility, but new downstream code should use + `exclude_packages` instead. - The downstream package is responsible for determining the correct arguments (machine, compiler, MPI, etc.) and for integrating the generated scripts into their activation workflow. - For more details, see the source code and examples in the downstream packages listed above. - diff --git a/mache/deploy/spack.py b/mache/deploy/spack.py index a9b9fc8f..57b5006d 100644 --- a/mache/deploy/spack.py +++ b/mache/deploy/spack.py @@ -14,7 +14,11 @@ from mache.deploy.bootstrap import check_call from mache.deploy.hooks import DeployContext from mache.spack.script import get_spack_script -from mache.spack.shared import _get_yaml_data +from mache.spack.shared import ( + _get_yaml_data, + normalize_excluded_packages, + resolve_e3sm_hdf5_netcdf, +) from mache.version import __version__ @@ -146,6 +150,11 @@ def deploy_spack_software_env( option='use_e3sm_hdf5_netcdf', default=False, ) + exclude_packages = _get_excluded_spack_packages(spack_cfg) + e3sm_hdf5_netcdf, exclude_packages = resolve_e3sm_hdf5_netcdf( + e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, + ) spack_path = _resolve_spack_path( ctx=ctx, @@ -176,6 +185,7 @@ def deploy_spack_software_env( mpi=mpi, section='software', e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, ) yaml_path = _write_mache_spack_env_yaml( @@ -186,6 +196,7 @@ def deploy_spack_software_env( env_name=env_name, spack_specs=specs, e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, ) _install_spack_env( @@ -295,6 +306,11 @@ def deploy_spack_envs( option='use_e3sm_hdf5_netcdf', default=False, ) + exclude_packages = _get_excluded_spack_packages(spack_cfg) + e3sm_hdf5_netcdf, exclude_packages = resolve_e3sm_hdf5_netcdf( + e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, + ) env_name_prefix = str( spack_cfg.get('env_name_prefix') or 'spack_env' @@ -341,6 +357,7 @@ def deploy_spack_envs( mpi=mpi, section='library', e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, ) yaml_path = _write_mache_spack_env_yaml( @@ -351,6 +368,7 @@ def deploy_spack_envs( env_name=env_name, spack_specs=specs, e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, ) _install_spack_env( @@ -440,6 +458,11 @@ def load_existing_spack_envs( option='use_e3sm_hdf5_netcdf', default=False, ) + exclude_packages = _get_excluded_spack_packages(spack_cfg) + e3sm_hdf5_netcdf, _ = resolve_e3sm_hdf5_netcdf( + e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, + ) env_name_prefix = str( spack_cfg.get('env_name_prefix') or 'spack_env' @@ -680,6 +703,12 @@ def _resolve_software_toolchain( return compiler, mpi +def _get_excluded_spack_packages(spack_cfg: dict) -> set[str]: + """Read the opt-out list for machine-provided Spack packages.""" + + return normalize_excluded_packages(spack_cfg.get('exclude_packages')) + + def _render_spack_specs( *, template_path: str, @@ -688,6 +717,7 @@ def _render_spack_specs( mpi: str, section: str, e3sm_hdf5_netcdf: bool, + exclude_packages: set[str], ) -> list[str]: if not os.path.exists(template_path): raise FileNotFoundError( @@ -709,6 +739,7 @@ def _render_spack_specs( machine=ctx.machine or '', compiler=compiler, mpi=mpi, + exclude_packages=sorted(exclude_packages), # Naming: prefer e3sm_hdf5_netcdf in templates; keep use_* to align # with the machine-config option name. e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, @@ -777,6 +808,7 @@ def _write_mache_spack_env_yaml( env_name: str, spack_specs: list[str], e3sm_hdf5_netcdf: bool, + exclude_packages: set[str], ) -> Path: """Write the full spack environment YAML using mache's templates.""" @@ -803,6 +835,7 @@ def _write_mache_spack_env_yaml( e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, specs=spack_specs, yaml_template=yaml_template, + exclude_packages=exclude_packages, ) yaml_path = work / f'{env_name}.yaml' diff --git a/mache/deploy/templates/config.yaml.j2.j2 b/mache/deploy/templates/config.yaml.j2.j2 index 2ac166bf..3ceac014 100644 --- a/mache/deploy/templates/config.yaml.j2.j2 +++ b/mache/deploy/templates/config.yaml.j2.j2 @@ -170,6 +170,22 @@ spack: # each containing a list of spec strings. specs_template: "deploy/spack.yaml.j2" + # Optional list of machine-provided Spack packages to suppress so Spack can + # build them instead. + # + # Typical examples: + # exclude_packages: + # - cmake + # + # To opt out of the machine-provided HDF5/NetCDF bundle, prefer: + # exclude_packages: + # - hdf5_netcdf + # + # In some target repositories, this list is set dynamically in `pre_spack()` + # based on merged machine config, for example when [deploy] + # use_e3sm_hdf5_netcdf = false should map to excluding the bundle. + exclude_packages: [] + # Optional spack build settings tmpdir: null mirror: null diff --git a/mache/deploy/templates/spack.yaml.j2 b/mache/deploy/templates/spack.yaml.j2 index d73a9a52..3c3e7b92 100644 --- a/mache/deploy/templates/spack.yaml.j2 +++ b/mache/deploy/templates/spack.yaml.j2 @@ -5,6 +5,8 @@ # - software, machine, compiler, mpi # - pins (dict of dicts from deploy/pins.cfg) # - spack / pixi / all (shortcuts for pins sections) +# - exclude_packages (normalized list of machine-provided packages to build +# with Spack instead) # # Expected format: a YAML mapping with one or both keys: # - library: specs for the per-toolchain "library" environment(s) diff --git a/mache/spack/config_machines.py b/mache/spack/config_machines.py index 5df8dfa0..2ad6c4cb 100644 --- a/mache/spack/config_machines.py +++ b/mache/spack/config_machines.py @@ -4,6 +4,11 @@ from lxml import etree +from mache.spack.shared import ( + classify_module_command_package_group, + shell_group_condition, +) + def extract_machine_config(xml_file, machine, compiler, mpilib): """ @@ -80,22 +85,22 @@ def config_to_shell_script(config, shell_type): script_lines.append('') module_commands = defaultdict(list) - e3sm_hdf5_netcdf_modules = defaultdict(list) + package_modules: dict[str, dict[str, list[str]]] = defaultdict( + lambda: defaultdict(list) + ) for module in config.findall('.//module_system/modules'): for command in module.findall('command'): name = command.get('name') value = command.text if value: - if 'command' != 'unload' and 'python' in value: + if name != 'unload' and 'python' in value: # we don't want to load E3SM's python module continue - elif 'command' != 'unload' and re.search( - r'hdf5|netcdf', value, re.IGNORECASE - ): - # we want to remove hdf5 and netcdf in all cases - e3sm_hdf5_netcdf_modules[name].append(value) - else: + group_name = classify_module_command_package_group(value) + if group_name is None: module_commands[name].append(value) + else: + package_modules[group_name][name].append(value) elif name not in module_commands: module_commands[name] = [] @@ -104,11 +109,11 @@ def config_to_shell_script(config, shell_type): ) script_lines.append('') - if e3sm_hdf5_netcdf_modules: - script_lines.append('{%- if e3sm_hdf5_netcdf %}') + for group_name, grouped_commands in package_modules.items(): + script_lines.append(f'{{%- if {shell_group_condition(group_name)} %}}') script_lines.extend( _convert_module_commands_to_script_lines( - e3sm_hdf5_netcdf_modules, shell_type + grouped_commands, shell_type ) ) script_lines.append('{%- endif %}') @@ -222,7 +227,6 @@ def _convert_env_vars_to_script_lines(config, shell_type): List of script lines. """ script_lines = [] - e3sm_hdf5_netcdf_env_vars = [] for env_var in config.findall('.//environment_variables/env'): name = env_var.get('name') value = env_var.text @@ -240,30 +244,20 @@ def _convert_env_vars_to_script_lines(config, shell_type): continue value = re.sub(r'\$ENV{([^}]+)}', r'${\1}', value) - if 'NETCDF' in name and 'PATH' in name: - e3sm_hdf5_netcdf_env_vars.append((name, value)) - if name == 'NETCDF_PATH': - # also set the NETCDF_C_PATH and NETCDF_FORTRAN_PATH, needed - # by MPAS standalone components - e3sm_hdf5_netcdf_env_vars.append(('NETCDF_C_PATH', value)) - e3sm_hdf5_netcdf_env_vars.append( - ('NETCDF_FORTRAN_PATH', value) - ) - else: - if shell_type == 'sh': - script_lines.append(f'export {name}="{value}"') - elif shell_type == 'csh': - script_lines.append(f'setenv {name} "{value}"') + env_names = [(name, value)] + if name == 'NETCDF_PATH': + # also set the NETCDF_C_PATH and NETCDF_FORTRAN_PATH, needed + # by MPAS standalone components + env_names.append(('NETCDF_C_PATH', value)) + env_names.append(('NETCDF_FORTRAN_PATH', value)) + + for env_name, env_value in env_names: + script_lines.append( + '{{ ' + f'render_env_var({env_name!r}, {env_value!r}, {shell_type!r})' + ' }}' + ) script_lines.append('') - if e3sm_hdf5_netcdf_env_vars: - script_lines.append('{%- if e3sm_hdf5_netcdf %}') - for name, value in e3sm_hdf5_netcdf_env_vars: - if shell_type == 'sh': - script_lines.append(f'export {name}="{value}"') - elif shell_type == 'csh': - script_lines.append(f'setenv {name} "{value}"') - script_lines.append('{%- endif %}') - return script_lines diff --git a/mache/spack/env.py b/mache/spack/env.py index 41f71811..d0ee426a 100644 --- a/mache/spack/env.py +++ b/mache/spack/env.py @@ -8,7 +8,7 @@ from mache.machine_info import MachineInfo, discover_machine from mache.spack.script import get_spack_script -from mache.spack.shared import _get_yaml_data +from mache.spack.shared import _get_yaml_data, resolve_e3sm_hdf5_netcdf from mache.version import __version__ MPI_COMPILERS = { @@ -33,6 +33,7 @@ def make_spack_env( include_e3sm_hdf5_netcdf=None, e3sm_hdf5_netcdf=None, yaml_template=None, + exclude_packages=None, tmpdir=None, spack_mirror=None, custom_spack='', @@ -82,6 +83,13 @@ def make_spack_env( of the mache template. This allows you to use compilers and other modules that differ from E3SM. + exclude_packages : sequence of str or str, optional + System-provided Spack packages to opt out of when rendering the + machine template. These package entries are removed from the rendered + YAML so Spack can build them instead. The aliases + ``e3sm_hdf5_netcdf`` and ``hdf5_netcdf`` exclude the machine-provided + HDF5/NetCDF bundle. + tmpdir : str, optional A temporary directory for building spack packages @@ -112,6 +120,10 @@ def make_spack_env( e3sm_hdf5_netcdf = bool(e3sm_hdf5_netcdf) else: e3sm_hdf5_netcdf = bool(include_e3sm_hdf5_netcdf) + e3sm_hdf5_netcdf, exclude_packages = resolve_e3sm_hdf5_netcdf( + e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, + ) if machine is None: machine = discover_machine() @@ -132,6 +144,7 @@ def make_spack_env( e3sm_hdf5_netcdf, spack_specs, yaml_template, + exclude_packages=exclude_packages, ) yaml_filename = os.path.abspath(f'{env_name}.yaml') @@ -150,6 +163,7 @@ def make_spack_env( include_e3sm_lapack, load_spack_env=False, e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, ) modules = f'{modules}\n{bash_script}' @@ -200,6 +214,7 @@ def get_modules_env_vars_and_mpi_compilers( include_e3sm_hdf5_netcdf=None, *, e3sm_hdf5_netcdf=None, + exclude_packages=None, ): """ Get the non-spack modules, environment variables and compiler names for a @@ -231,6 +246,12 @@ def get_modules_env_vars_and_mpi_compilers( include_e3sm_hdf5_netcdf : bool, optional Deprecated alias for ``e3sm_hdf5_netcdf``. + exclude_packages : sequence of str or str, optional + System-provided Spack packages to opt out of. For this function, the + package bundle that affects shell setup is ``e3sm_hdf5_netcdf`` (or + ``hdf5_netcdf``), which disables the machine-provided HDF5/NetCDF + module and environment-variable setup. + Returns ------- mpicc : str @@ -265,6 +286,10 @@ def get_modules_env_vars_and_mpi_compilers( e3sm_hdf5_netcdf = bool(e3sm_hdf5_netcdf) else: e3sm_hdf5_netcdf = bool(include_e3sm_hdf5_netcdf) + e3sm_hdf5_netcdf, exclude_packages = resolve_e3sm_hdf5_netcdf( + e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, + ) if machine is None: machine = discover_machine() @@ -291,6 +316,7 @@ def get_modules_env_vars_and_mpi_compilers( load_spack_env=False, include_e3sm_lapack=include_e3sm_lapack, e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, ) mpicc, mpicxx, mpifc = _get_mpi_compilers( diff --git a/mache/spack/script.py b/mache/spack/script.py index 0fd6fda4..f2839267 100644 --- a/mache/spack/script.py +++ b/mache/spack/script.py @@ -5,6 +5,12 @@ from mache.machine_info import discover_machine from mache.spack.config_machines import extract_spack_from_config_machines +from mache.spack.shared import ( + render_env_var, + resolve_e3sm_hdf5_netcdf, + use_system_package, + use_system_packages, +) def get_spack_script( @@ -19,6 +25,7 @@ def get_spack_script( load_spack_env=True, *, e3sm_hdf5_netcdf=None, + exclude_packages=None, ): """ Build a snippet of a load script for the given spack environment @@ -55,6 +62,12 @@ def get_spack_script( include_e3sm_hdf5_netcdf : bool, optional Deprecated alias for ``e3sm_hdf5_netcdf``. + exclude_packages : sequence of str or str, optional + System-provided Spack packages to opt out of. For this function, the + package bundle that affects shell setup is ``e3sm_hdf5_netcdf`` (or + ``hdf5_netcdf``), which disables the machine-provided HDF5/NetCDF + module and environment-variable setup. + load_spack_env : bool, optional Whether to load the spack environment at the start of script. Must be set to False when initially building the environment @@ -87,6 +100,10 @@ def get_spack_script( e3sm_hdf5_netcdf = bool(e3sm_hdf5_netcdf) else: e3sm_hdf5_netcdf = bool(include_e3sm_hdf5_netcdf) + e3sm_hdf5_netcdf, exclude_packages = resolve_e3sm_hdf5_netcdf( + e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=exclude_packages, + ) if machine is None: machine = discover_machine() @@ -136,6 +153,16 @@ def get_spack_script( load_script = Template(load_script_template).render( e3sm_lapack=include_e3sm_lapack, e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, + exclude_packages=sorted(exclude_packages), + use_system_package=lambda package: use_system_package( + package, exclude_packages + ), + use_system_packages=lambda *packages: use_system_packages( + *packages, exclude_packages=exclude_packages + ), + render_env_var=lambda name, value, shell_type: render_env_var( + name, value, shell_type, exclude_packages + ), ) return load_script diff --git a/mache/spack/shared.py b/mache/spack/shared.py index f95581f2..5e8247e1 100644 --- a/mache/spack/shared.py +++ b/mache/spack/shared.py @@ -1,6 +1,313 @@ +import re +from collections.abc import Iterable from importlib import resources as importlib_resources from jinja2 import Template +from yaml import safe_dump, safe_load + +E3SM_HDF5_NETCDF_PACKAGES = frozenset( + {'hdf5', 'netcdf-c', 'netcdf-fortran', 'parallel-netcdf'} +) +E3SM_HDF5_NETCDF_ALIASES = frozenset({'e3sm_hdf5_netcdf', 'hdf5_netcdf'}) +SHELL_GROUP_RULES = ( + ( + 'cmake', + { + 'condition_packages': ('cmake',), + 'module_patterns': (r'(^|[ /_-])cmake($|[ /@._-])',), + 'env_name_patterns': (r'^CMAKE_', r'^ACLOCAL_PATH$'), + 'env_value_patterns': (r'(^|/)cmake($|/|-)',), + }, + ), + ( + 'hdf5', + { + 'condition_packages': ('hdf5',), + 'module_patterns': (r'(^|[ /_-])hdf5($|[ /@._-])',), + 'env_name_patterns': (r'^HDF5_', r'^PHDF5_'), + 'env_value_patterns': (r'(^|/)hdf5($|/|-)',), + }, + ), + ( + 'parallel-netcdf', + { + 'condition_packages': ('parallel-netcdf',), + 'module_patterns': ( + r'(^|[ /_-])pnetcdf($|[ /@._-])', + r'parallel-netcdf', + ), + 'env_name_patterns': (r'^PNETCDF_',), + 'env_value_patterns': ( + r'(^|/)pnetcdf($|/|-)', + r'parallel-netcdf', + ), + }, + ), + ( + 'netcdf', + { + 'condition_packages': ('netcdf-c', 'netcdf-fortran'), + 'module_patterns': ( + r'(^|[ /_-])netcdf($|[ /@._-])', + r'netcdf-hdf5parallel', + ), + 'env_name_patterns': ( + r'^NETCDF_', + r'^NETCDF_C_', + r'^NETCDF_FORTRAN_', + ), + 'env_value_patterns': ( + r'(^|/)netcdf($|/|-)', + r'netcdf-hdf5parallel', + ), + }, + ), +) +PATH_LIKE_ENV_VARS = frozenset( + { + 'PATH', + 'LD_LIBRARY_PATH', + 'LIBRARY_PATH', + 'CPATH', + 'C_INCLUDE_PATH', + 'CPLUS_INCLUDE_PATH', + 'F_INCLUDE_PATH', + 'PKG_CONFIG_PATH', + 'MANPATH', + 'ACLOCAL_PATH', + 'CMAKE_PREFIX_PATH', + } +) + + +def normalize_excluded_packages(exclude_packages): + """Normalize a package opt-out list into a set of package names.""" + + if exclude_packages is None: + return set() + + if isinstance(exclude_packages, str): + entries = exclude_packages.replace(',', ' ').split() + elif isinstance(exclude_packages, Iterable): + entries = [] + for item in exclude_packages: + if item is None: + continue + if not isinstance(item, str): + raise TypeError( + 'exclude_packages entries must be strings, got ' + f'{type(item).__name__}.' + ) + entries.extend(item.replace(',', ' ').split()) + else: + raise TypeError( + 'exclude_packages must be a string or an iterable of strings.' + ) + + excluded: set[str] = set() + for entry in entries: + package = entry.strip().lower() + if not package: + continue + if package in E3SM_HDF5_NETCDF_ALIASES: + excluded.update(E3SM_HDF5_NETCDF_PACKAGES) + else: + excluded.add(package) + + return excluded + + +def use_system_package(package, exclude_packages): + """Return whether the system-provided package should remain enabled.""" + + requested = normalize_excluded_packages([package]) + excluded = normalize_excluded_packages(exclude_packages) + return not bool(requested & excluded) + + +def use_system_packages(*packages, exclude_packages): + """Return whether all requested system packages remain enabled.""" + + return all( + use_system_package(package, exclude_packages) for package in packages + ) + + +def _matches_patterns(text, patterns): + return any(re.search(pattern, text, re.IGNORECASE) for pattern in patterns) + + +def classify_module_command_package_group(value): + """Classify a module command by the system-package group it belongs to.""" + + for group_name, rule in SHELL_GROUP_RULES: + if _matches_patterns(value, rule['module_patterns']): + return group_name + return None + + +def classify_env_var_package_group(name, value=''): + """Classify an environment variable by the system-package group it uses.""" + + for group_name, rule in SHELL_GROUP_RULES: + if _matches_patterns(name, rule['env_name_patterns']): + return group_name + if value and _matches_patterns(value, rule['env_value_patterns']): + return group_name + return None + + +def use_system_package_group(group_name, exclude_packages): + """Return whether a shell-side system-package group remains enabled.""" + + for candidate, rule in SHELL_GROUP_RULES: + if candidate == group_name: + return use_system_packages( + *rule['condition_packages'], + exclude_packages=exclude_packages, + ) + raise ValueError(f'Unknown system-package group: {group_name}') + + +def shell_group_condition(group_name): + """Return a Jinja expression for testing a shell package group.""" + + for candidate, rule in SHELL_GROUP_RULES: + if candidate != group_name: + continue + packages = rule['condition_packages'] + if len(packages) == 1: + return f"use_system_package('{packages[0]}')" + args = ', '.join(f"'{package}'" for package in packages) + return f'use_system_packages({args})' + raise ValueError(f'Unknown system-package group: {group_name}') + + +def _filter_system_package_path_segments(value, exclude_packages): + segments = value.split(':') + filtered = [] + for segment in segments: + group_name = classify_env_var_package_group('', segment) + if group_name is not None and not use_system_package_group( + group_name, exclude_packages + ): + continue + filtered.append(segment) + + while filtered and filtered[0] == '': + filtered = filtered[1:] + while filtered and filtered[-1] == '': + filtered = filtered[:-1] + return ':'.join(filtered) + + +def render_env_var(name, value, shell_type, exclude_packages): + """Render one shell export/setenv line after applying package opt-outs.""" + + if name in PATH_LIKE_ENV_VARS: + value = _filter_system_package_path_segments(value, exclude_packages) + if not value: + return '' + else: + group_name = classify_env_var_package_group(name, value) + if group_name is not None and not use_system_package_group( + group_name, exclude_packages + ): + return '' + + if shell_type == 'sh': + return f'export {name}="{value}"' + if shell_type == 'csh': + return f'setenv {name} "{value}"' + raise ValueError(f'Unexpected shell_type: {shell_type}') + + +def resolve_e3sm_hdf5_netcdf( + *, + e3sm_hdf5_netcdf, + exclude_packages, +): + """Resolve the legacy HDF5/NetCDF toggle against package opt-outs.""" + + excluded = normalize_excluded_packages(exclude_packages) + bundle_requested = bool(excluded & E3SM_HDF5_NETCDF_PACKAGES) + + if not e3sm_hdf5_netcdf: + excluded.update(E3SM_HDF5_NETCDF_PACKAGES) + return False, excluded + + if bundle_requested: + excluded.update(E3SM_HDF5_NETCDF_PACKAGES) + return False, excluded + + return True, excluded + + +def _extract_spack_package_name(spec): + if not isinstance(spec, str): + return None + + match = re.match(r'^\s*"?([A-Za-z0-9][A-Za-z0-9._-]*)', spec) + if match is None: + return None + return match.group(1).lower() + + +def _filter_yaml_data(yaml_data, exclude_packages): + excluded = normalize_excluded_packages(exclude_packages) + if not excluded: + return yaml_data + + data = safe_load(yaml_data) + if not isinstance(data, dict): + return yaml_data + + spack_data = data.get('spack') + if not isinstance(spack_data, dict): + return yaml_data + + specs = spack_data.get('specs') + if isinstance(specs, list): + spack_data['specs'] = [ + spec + for spec in specs + if _extract_spack_package_name(spec) not in excluded + ] + + packages = spack_data.get('packages') + if isinstance(packages, dict): + for package_name in list(packages): + if package_name.lower() in excluded: + del packages[package_name] + continue + + package_data = packages[package_name] + if package_name != 'all' or not isinstance(package_data, dict): + continue + + providers = package_data.get('providers') + if not isinstance(providers, dict): + continue + + for provider_name in list(providers): + provider_specs = providers[provider_name] + if not isinstance(provider_specs, list): + continue + + kept_specs = [ + spec + for spec in provider_specs + if _extract_spack_package_name(spec) not in excluded + ] + if kept_specs: + providers[provider_name] = kept_specs + else: + del providers[provider_name] + + filtered = safe_dump(data, sort_keys=False) + if filtered and not filtered.endswith('\n'): + filtered = f'{filtered}\n' + return filtered def _get_yaml_data( @@ -11,6 +318,7 @@ def _get_yaml_data( e3sm_hdf5_netcdf, specs, yaml_template, + exclude_packages=None, ): """Get the data from the jinja-templated yaml file based on settings""" if yaml_template is None: @@ -36,4 +344,4 @@ def _get_yaml_data( e3sm_lapack=include_e3sm_lapack, e3sm_hdf5_netcdf=e3sm_hdf5_netcdf, ) - return yaml_data + return _filter_yaml_data(yaml_data, exclude_packages) diff --git a/mache/spack/templates/aurora_oneapi-ifx_mpich.yaml b/mache/spack/templates/aurora_oneapi-ifx_mpich.yaml index dcc40f2d..60b45cf6 100644 --- a/mache/spack/templates/aurora_oneapi-ifx_mpich.yaml +++ b/mache/spack/templates/aurora_oneapi-ifx_mpich.yaml @@ -5,12 +5,10 @@ spack: specs: # don't include {{ compiler }} since there isn't a "oneapi" package - {{ mpi }}%{{ compiler }} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -119,7 +117,6 @@ spack: - spec: {{ mpi }}%{{ compiler }} prefix: /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/mpich-5.0.0.aurora_test.3c70a61-hlkigtk buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.14.6+cxx+fortran+hl+mpi @@ -140,7 +137,6 @@ spack: - spec: parallel-netcdf@1.14.1+cxx+fortran prefix: /opt/aurora/26.26.0/spack/unified/1.1.1/install/linux-x86_64/parallel-netcdf-1.14.1-44pgfig buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/chicoma-cpu_gnu_mpich.yaml b/mache/spack/templates/chicoma-cpu_gnu_mpich.yaml index 04e97bf1..57bf6bdc 100644 --- a/mache/spack/templates/chicoma-cpu_gnu_mpich.yaml +++ b/mache/spack/templates/chicoma-cpu_gnu_mpich.yaml @@ -5,12 +5,10 @@ spack: - {{ compiler }} - {{ mpi }}%{{ compiler }} - cray-libsci -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -123,7 +121,6 @@ spack: modules: - cray-libsci/23.12.5 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.12.2.9~cxx+fortran+hl~java+mpi+shared @@ -144,7 +141,6 @@ spack: - spec: netcdf-fortran@4.5.3 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.0.9/gnu/12.3 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/chicoma-cpu_nvidia_mpich.yaml b/mache/spack/templates/chicoma-cpu_nvidia_mpich.yaml index 9fd97daf..23fea210 100644 --- a/mache/spack/templates/chicoma-cpu_nvidia_mpich.yaml +++ b/mache/spack/templates/chicoma-cpu_nvidia_mpich.yaml @@ -4,12 +4,10 @@ spack: specs: - {{ mpi }}%{{ compiler }} - cray-libsci -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -105,7 +103,6 @@ spack: modules: - cray-libsci/23.12.5 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.12.2.9~cxx+fortran+hl~java+mpi+shared @@ -126,7 +123,6 @@ spack: - spec: netcdf-fortran@4.5.3 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.0.9/nvidia/23.3 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/chrysalis_gnu_openmpi.yaml b/mache/spack/templates/chrysalis_gnu_openmpi.yaml index cf01d6a5..afe4ee17 100644 --- a/mache/spack/templates/chrysalis_gnu_openmpi.yaml +++ b/mache/spack/templates/chrysalis_gnu_openmpi.yaml @@ -7,12 +7,10 @@ spack: {%- if e3sm_lapack %} - intel-oneapi-mkl {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -119,7 +117,6 @@ spack: modules: - intel-oneapi-mkl/2022.1.0-w4kgsn4 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.10.7+cxx+fortran+hl+mpi @@ -144,7 +141,6 @@ spack: modules: - parallel-netcdf/1.11.0-d7h4ysd buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/chrysalis_intel_impi.yaml b/mache/spack/templates/chrysalis_intel_impi.yaml index 33184302..4e21e6e4 100644 --- a/mache/spack/templates/chrysalis_intel_impi.yaml +++ b/mache/spack/templates/chrysalis_intel_impi.yaml @@ -7,12 +7,10 @@ spack: {%- if e3sm_lapack %} - intel-mkl {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -114,7 +112,6 @@ spack: modules: - intel-mkl/2020.4.304-g2qaxzf buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.10.7+cxx+fortran+hl+mpi~shared @@ -139,7 +136,6 @@ spack: modules: - parallel-netcdf/1.11.0-b74wv4m buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/chrysalis_intel_openmpi.yaml b/mache/spack/templates/chrysalis_intel_openmpi.yaml index eee43fbb..2cf3bf5b 100644 --- a/mache/spack/templates/chrysalis_intel_openmpi.yaml +++ b/mache/spack/templates/chrysalis_intel_openmpi.yaml @@ -7,12 +7,10 @@ spack: {%- if e3sm_lapack %} - intel-mkl {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -114,7 +112,6 @@ spack: modules: - intel-mkl/2020.4.304-g2qaxzf buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.10.7+cxx+fortran+hl+mpi @@ -139,7 +136,6 @@ spack: modules: - parallel-netcdf/1.11.0-icrpxty buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/chrysalis_oneapi-ifx_openmpi.yaml b/mache/spack/templates/chrysalis_oneapi-ifx_openmpi.yaml index bebdb532..103fc76d 100644 --- a/mache/spack/templates/chrysalis_oneapi-ifx_openmpi.yaml +++ b/mache/spack/templates/chrysalis_oneapi-ifx_openmpi.yaml @@ -7,12 +7,10 @@ spack: {%- if e3sm_lapack %} - intel-oneapi-mkl {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -126,7 +124,6 @@ spack: modules: - intel-oneapi-mkl/2025.2.0-bcimxay buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.14.6+cxx+fortran+hl+mpi @@ -151,7 +148,6 @@ spack: modules: - parallel-netcdf/1.14.1-f2fwvr2 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/compy_gnu_openmpi.csh b/mache/spack/templates/compy_gnu_openmpi.csh index c37c1063..1830333a 100644 --- a/mache/spack/templates/compy_gnu_openmpi.csh +++ b/mache/spack/templates/compy_gnu_openmpi.csh @@ -2,9 +2,12 @@ module load \ gcc/10.2.0 \ openmpi/4.0.1 -{%- if e3sm_hdf5_netcdf %} -module load \ - hdf5/1.10.5 \ - netcdf/4.6.3 \ - pnetcdf/1.9.0 +{%- if use_system_package('hdf5') %} +module load hdf5/1.10.5 +{%- endif %} +{%- if use_system_packages('netcdf-c', 'netcdf-fortran') %} +module load netcdf/4.6.3 +{%- endif %} +{%- if use_system_package('parallel-netcdf') %} +module load pnetcdf/1.9.0 {%- endif %} diff --git a/mache/spack/templates/compy_gnu_openmpi.sh b/mache/spack/templates/compy_gnu_openmpi.sh index c37c1063..1830333a 100644 --- a/mache/spack/templates/compy_gnu_openmpi.sh +++ b/mache/spack/templates/compy_gnu_openmpi.sh @@ -2,9 +2,12 @@ module load \ gcc/10.2.0 \ openmpi/4.0.1 -{%- if e3sm_hdf5_netcdf %} -module load \ - hdf5/1.10.5 \ - netcdf/4.6.3 \ - pnetcdf/1.9.0 +{%- if use_system_package('hdf5') %} +module load hdf5/1.10.5 +{%- endif %} +{%- if use_system_packages('netcdf-c', 'netcdf-fortran') %} +module load netcdf/4.6.3 +{%- endif %} +{%- if use_system_package('parallel-netcdf') %} +module load pnetcdf/1.9.0 {%- endif %} diff --git a/mache/spack/templates/compy_gnu_openmpi.yaml b/mache/spack/templates/compy_gnu_openmpi.yaml index 97c0349b..a2b3bd78 100644 --- a/mache/spack/templates/compy_gnu_openmpi.yaml +++ b/mache/spack/templates/compy_gnu_openmpi.yaml @@ -7,12 +7,10 @@ spack: {%- if e3sm_lapack %} - intel-mkl {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -132,7 +130,6 @@ spack: modules: - mkl/2020 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5~mpi+hl@1.10.5 @@ -161,7 +158,6 @@ spack: modules: - pnetcdf/1.9.0 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/compy_intel_impi.yaml b/mache/spack/templates/compy_intel_impi.yaml index 679eb378..8ce31c94 100644 --- a/mache/spack/templates/compy_intel_impi.yaml +++ b/mache/spack/templates/compy_intel_impi.yaml @@ -7,12 +7,10 @@ spack: {%- if e3sm_lapack %} - intel-mkl {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -133,7 +131,6 @@ spack: modules: - mkl/2020 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5~mpi+hl@1.10.5 @@ -162,7 +159,6 @@ spack: modules: - pnetcdf/1.9.0 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/dane_intel_mvapich2.yaml b/mache/spack/templates/dane_intel_mvapich2.yaml index 58344987..1fc4ca65 100644 --- a/mache/spack/templates/dane_intel_mvapich2.yaml +++ b/mache/spack/templates/dane_intel_mvapich2.yaml @@ -7,12 +7,10 @@ spack: {%- if e3sm_lapack %} - intel-oneapi-mkl {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -26,7 +24,6 @@ spack: {%- if e3sm_lapack %} lapack: [intel-oneapi-mkl@2022.1.0] {%- endif %} -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.14.0+cxx+fortran+hl~java+mpi~szip+threadsafe+tools api=v18 @@ -55,7 +52,6 @@ spack: modules: - parallel-netcdf/1.12.3 buildable: false -{%- endif %} bison: externals: - spec: bison@3.0.4 diff --git a/mache/spack/templates/frontier_crayamd-mphipcc_mpich.yaml b/mache/spack/templates/frontier_crayamd-mphipcc_mpich.yaml index e53211c8..08a30e03 100644 --- a/mache/spack/templates/frontier_crayamd-mphipcc_mpich.yaml +++ b/mache/spack/templates/frontier_crayamd-mphipcc_mpich.yaml @@ -6,12 +6,10 @@ spack: {%- if e3sm_lapack %} - cray-libsci {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -132,7 +130,6 @@ spack: modules: - cray-libsci/22.12.1.1 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.12.2.1~cxx+fortran+hl~java+mpi+shared @@ -153,7 +150,6 @@ spack: - spec: netcdf-fortran@4.5.3 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.0.1/amd/4.3 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/frontier_crayamd_mpich.yaml b/mache/spack/templates/frontier_crayamd_mpich.yaml index 6004bec7..8158e522 100644 --- a/mache/spack/templates/frontier_crayamd_mpich.yaml +++ b/mache/spack/templates/frontier_crayamd_mpich.yaml @@ -6,12 +6,10 @@ spack: {%- if e3sm_lapack %} - cray-libsci {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -132,7 +130,6 @@ spack: modules: - cray-libsci/22.12.1.1 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.12.2.1~cxx+fortran+hl~java+mpi+shared @@ -153,7 +150,6 @@ spack: - spec: netcdf-fortran@4.5.3 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.0.1/amd/4.3 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/frontier_craycray-mphipcc_mpich.yaml b/mache/spack/templates/frontier_craycray-mphipcc_mpich.yaml index 602e759f..94ec5a3a 100644 --- a/mache/spack/templates/frontier_craycray-mphipcc_mpich.yaml +++ b/mache/spack/templates/frontier_craycray-mphipcc_mpich.yaml @@ -6,12 +6,10 @@ spack: {%- if e3sm_lapack %} - cray-libsci {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -136,7 +134,6 @@ spack: - cray-libsci/22.12.1.1 buildable: false {%- endif %} -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.12.2.1~cxx+fortran+hl~java+mpi+shared @@ -157,7 +154,6 @@ spack: - spec: netcdf-fortran@4.5.3 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.0.1/crayclang/14.0 buildable: false -{%- endif %} compilers: - compiler: diff --git a/mache/spack/templates/frontier_craycray_mpich.yaml b/mache/spack/templates/frontier_craycray_mpich.yaml index 6e49b817..caad4f76 100644 --- a/mache/spack/templates/frontier_craycray_mpich.yaml +++ b/mache/spack/templates/frontier_craycray_mpich.yaml @@ -6,12 +6,10 @@ spack: {%- if e3sm_lapack %} - cray-libsci {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -136,7 +134,6 @@ spack: - cray-libsci/22.12.1.1 buildable: false {%- endif %} -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.12.2.1~cxx+fortran+hl~java+mpi+shared @@ -157,7 +154,6 @@ spack: - spec: netcdf-fortran@4.5.3 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.0.1/crayclang/14.0 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/frontier_craygnu-mphipcc_mpich.yaml b/mache/spack/templates/frontier_craygnu-mphipcc_mpich.yaml index 85a532a7..98dda209 100644 --- a/mache/spack/templates/frontier_craygnu-mphipcc_mpich.yaml +++ b/mache/spack/templates/frontier_craygnu-mphipcc_mpich.yaml @@ -7,12 +7,10 @@ spack: {%- if e3sm_lapack %} - cray-libsci {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -160,7 +158,6 @@ spack: - cray-libsci/24.11.0 buildable: false {%- endif %} -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.12.2.1~cxx+fortran+hl~java+mpi+shared @@ -181,7 +178,6 @@ spack: - spec: netcdf-fortran@4.5.3 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.0.1/GNU/9.1 buildable: false -{%- endif %} compilers: - compiler: diff --git a/mache/spack/templates/frontier_craygnu_mpich.yaml b/mache/spack/templates/frontier_craygnu_mpich.yaml index 7d2f512b..577ebf84 100644 --- a/mache/spack/templates/frontier_craygnu_mpich.yaml +++ b/mache/spack/templates/frontier_craygnu_mpich.yaml @@ -7,12 +7,10 @@ spack: {%- if e3sm_lapack %} - cray-libsci {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -158,7 +156,6 @@ spack: - cray-libsci/24.11.0 buildable: false {%- endif %} -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.12.2.1~cxx+fortran+hl~java+mpi+shared @@ -179,7 +176,6 @@ spack: - spec: netcdf-fortran@4.5.3 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.0.1/GNU/9.1 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/pm-cpu_gnu_mpich.yaml b/mache/spack/templates/pm-cpu_gnu_mpich.yaml index ce24419b..798b310a 100644 --- a/mache/spack/templates/pm-cpu_gnu_mpich.yaml +++ b/mache/spack/templates/pm-cpu_gnu_mpich.yaml @@ -5,12 +5,10 @@ spack: - {{ compiler }} - {{ mpi }}%{{ compiler }} - cray-libsci -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -139,7 +137,6 @@ spack: modules: - cray-libsci/25.09.0 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.14.3.7~cxx+fortran+hl~java+mpi+shared @@ -160,7 +157,6 @@ spack: - spec: netcdf-fortran@4.6.1 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.2.1/gnu/12.3 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/pm-cpu_intel_mpich.yaml b/mache/spack/templates/pm-cpu_intel_mpich.yaml index 916dd5ef..0e58e59b 100644 --- a/mache/spack/templates/pm-cpu_intel_mpich.yaml +++ b/mache/spack/templates/pm-cpu_intel_mpich.yaml @@ -5,12 +5,10 @@ spack: - {{ compiler }} - {{ mpi }}%{{ compiler }} - cray-libsci -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -129,7 +127,6 @@ spack: modules: - cray-libsci/25.09.0 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.14.3.7~cxx+fortran+hl~java+mpi+shared @@ -150,7 +147,6 @@ spack: - spec: netcdf-fortran@4.6.1 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.2.1/intel/2023.2 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/pm-cpu_nvidia_mpich.yaml b/mache/spack/templates/pm-cpu_nvidia_mpich.yaml index fd10c572..544f6055 100644 --- a/mache/spack/templates/pm-cpu_nvidia_mpich.yaml +++ b/mache/spack/templates/pm-cpu_nvidia_mpich.yaml @@ -4,12 +4,10 @@ spack: specs: - {{ mpi }}%{{ compiler }} - cray-libsci -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -117,7 +115,6 @@ spack: modules: - cray-libsci/25.09.0 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.14.3.7~cxx+fortran+hl~java+mpi+shared @@ -138,7 +135,6 @@ spack: - spec: netcdf-fortran@4.6.1 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.2.1/nvidia/23.3 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/pm-gpu_gnugpu_mpich.yaml b/mache/spack/templates/pm-gpu_gnugpu_mpich.yaml index 5b97db9b..cb6f8d4f 100644 --- a/mache/spack/templates/pm-gpu_gnugpu_mpich.yaml +++ b/mache/spack/templates/pm-gpu_gnugpu_mpich.yaml @@ -5,12 +5,10 @@ spack: - {{ compiler }} - {{ mpi }}%{{ compiler }} - cray-libsci -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -144,7 +142,6 @@ spack: modules: - cray-libsci/25.09.0 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.14.3.7~cxx+fortran+hl~java+mpi+shared @@ -165,7 +162,6 @@ spack: - spec: netcdf-fortran@4.6.1 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.2.1/gnu/12.3 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/pm-gpu_nvidiagpu_mpich.yaml b/mache/spack/templates/pm-gpu_nvidiagpu_mpich.yaml index 389d1723..2b44a2af 100644 --- a/mache/spack/templates/pm-gpu_nvidiagpu_mpich.yaml +++ b/mache/spack/templates/pm-gpu_nvidiagpu_mpich.yaml @@ -4,12 +4,10 @@ spack: specs: - {{ mpi }}%{{ compiler }} - cray-libsci -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -129,7 +127,6 @@ spack: modules: - cray-libsci/25.09.0 buildable: false -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.14.3.7~cxx+fortran+hl~java+mpi+shared @@ -150,7 +147,6 @@ spack: - spec: netcdf-fortran@4.6.1 prefix: /opt/cray/pe/netcdf-hdf5parallel/4.9.2.1/nvidia/23.3 buildable: false -{%- endif %} compilers: - compiler: spec: {{ compiler }} diff --git a/mache/spack/templates/ruby_intel_mvapich2.yaml b/mache/spack/templates/ruby_intel_mvapich2.yaml index d103e582..48b8abb2 100644 --- a/mache/spack/templates/ruby_intel_mvapich2.yaml +++ b/mache/spack/templates/ruby_intel_mvapich2.yaml @@ -7,12 +7,10 @@ spack: {%- if e3sm_lapack %} - intel-oneapi-mkl {%- endif %} -{%- if e3sm_hdf5_netcdf %} - "hdf5%{{ compiler }}" - "netcdf-c%{{ compiler }}" - "netcdf-fortran%{{ compiler }}" - "parallel-netcdf%{{ compiler }}" -{%- endif %} {%- for spec in specs %} - "{{ spec }}%{{ compiler }}" {%- endfor %} @@ -289,7 +287,6 @@ spack: - mkl-interfaces/2022.1.0 buildable: false {%- endif %} -{%- if e3sm_hdf5_netcdf %} hdf5: externals: - spec: hdf5@1.14.0+cxx+fortran+hl~java+mpi~szip+threadsafe+tools api=v18 @@ -314,6 +311,5 @@ spack: modules: - parallel-netcdf/1.12.3 buildable: false -{%- endif %} config: install_missing_compilers: false diff --git a/tests/test_deploy_spack.py b/tests/test_deploy_spack.py index 0fa0d26e..ac6a8e18 100644 --- a/tests/test_deploy_spack.py +++ b/tests/test_deploy_spack.py @@ -7,6 +7,11 @@ from mache.deploy import spack as deploy_spack from mache.deploy.hooks import DeployContext +from mache.spack.script import get_spack_script +from mache.spack.shared import ( + E3SM_HDF5_NETCDF_PACKAGES, + _get_yaml_data, +) def _logger(name: str) -> logging.Logger: @@ -151,3 +156,87 @@ def test_load_existing_spack_envs_respects_runtime_disable(tmp_path: Path): == [] ) assert deploy_spack.load_existing_spack_software_env(ctx=ctx) is None + + +def test_get_excluded_spack_packages_supports_bundle_aliases(): + excluded = deploy_spack._get_excluded_spack_packages( + {'exclude_packages': 'cmake, e3sm_hdf5_netcdf'} + ) + + assert 'cmake' in excluded + assert E3SM_HDF5_NETCDF_PACKAGES.issubset(excluded) + + +def test_get_yaml_data_can_exclude_cmake_external(): + yaml_text = _get_yaml_data( + machine='chicoma-cpu', + compiler='gnu', + mpi='mpich', + include_e3sm_lapack=False, + e3sm_hdf5_netcdf=False, + specs=['trilinos'], + yaml_template=None, + exclude_packages=['cmake'], + ) + + assert 'cmake:' not in yaml_text + assert 'curl:' in yaml_text + assert 'trilinos%gcc@12.3' in yaml_text + + +def test_get_yaml_data_can_exclude_e3sm_hdf5_netcdf_bundle(): + yaml_text = _get_yaml_data( + machine='chicoma-cpu', + compiler='gnu', + mpi='mpich', + include_e3sm_lapack=False, + e3sm_hdf5_netcdf=True, + specs=['trilinos'], + yaml_template=None, + exclude_packages=['hdf5_netcdf'], + ) + + assert 'hdf5:' not in yaml_text + assert 'netcdf-c:' not in yaml_text + assert 'netcdf-fortran:' not in yaml_text + assert 'parallel-netcdf:' not in yaml_text + assert 'hdf5%gcc@12.3' not in yaml_text + assert 'netcdf-c%gcc@12.3' not in yaml_text + + +def test_get_spack_script_filters_compy_template_modules(): + script = get_spack_script( + spack_path='/unused', + env_name='unused', + compiler='gnu', + mpi='openmpi', + shell='sh', + machine='compy', + load_spack_env=False, + e3sm_hdf5_netcdf=True, + exclude_packages=['hdf5_netcdf'], + ) + + assert 'gcc/10.2.0' in script + assert 'openmpi/4.0.1' in script + assert 'hdf5/1.10.5' not in script + assert 'netcdf/4.6.3' not in script + assert 'pnetcdf/1.9.0' not in script + + +def test_get_spack_script_filters_config_machine_cmake_module(): + script = get_spack_script( + spack_path='/unused', + env_name='unused', + compiler='gnu', + mpi='mpich', + shell='sh', + machine='chicoma-cpu', + load_spack_env=False, + e3sm_hdf5_netcdf=True, + exclude_packages=['cmake'], + ) + + assert 'PrgEnv-gnu/8.5.0' in script + assert 'cray-mpich/8.1.28' in script + assert 'cmake/3.29.6' not in script