Integrate papiex C library compilation into pip install (setup.py + compile_papiex.sh)

[Copilot]

Folds papiex native library compilation into pip install epmt using the same pyproject.toml + setup.py hybrid pattern as NOAA-GFDL/pyFMS. Compilation is best-effort: EPMT installs cleanly even when build tools or network are unavailable.

New files

• src/compile_papiex.sh — Downloads papiex source (or uses src/vendor/papiex/ if pre-populated) and runs make PREFIX=<epmt_lib_dir> install. Skips gracefully (exit 0) when .so files are already present, gcc/make/curl are missing, or the source download fails. Exits non-zero only on actual make failure (caught and warned by setup.py). Respects PAPIEX_SRC_BRANCH, CONFIG_PAPIEX_PAPI, CONFIG_PAPIEX_DEBUG.
• src/setup.py — Minimal hook following pyFMS pattern. All metadata stays in pyproject.toml.

class BuildPapiex(build):
    def run(self):
        script = Path(__file__).parent / "compile_papiex.sh"
        if script.exists():
            try:
                with open("install_papiex.log", "w", encoding="utf-8") as log_fh:
                    subprocess.run([str(script)], stdout=log_fh,
                                   stderr=subprocess.STDOUT, check=True)
            except subprocess.CalledProcessError:
                logger.warning("papiex compilation failed … see install_papiex.log")
        super().run()

Modified files

• src/MANIFEST.in — Include compile_papiex.sh in sdist so it ships with the package.
• Makefile — vendor-papiex target downloads papiex source into src/vendor/papiex/ for air-gapped installs; pip-install target runs vendor-papiex then pip3 install ./src; papiexclean now also removes src/vendor/papiex and src/epmt/lib.
• .gitignore — Exclude src/vendor/, src/epmt/lib/, install_papiex.log.
• build_and_test_epmt.yml — New CI step exercises pip install ./src (with papiex compiled from source) before the existing sdist-based path.
• epmtdocs/docs/INSTALL.md — Documents pip install epmt, the make vendor-papiex / make pip-install targets, and the controlling environment variables.

Backward compatibility

The existing Makefile workflow (make python-dist) pre-copies .so files into src/epmt/lib/ before building the sdist. compile_papiex.sh detects those files and skips compilation, so the current Docker/CI release path is unaffected.

Checklist

• the code runs
• the code is readable
• the code is commented
• there are no additional failures in GUARDED pipeline tasks
• a new test was written (if applicable)
• new instructions/doc was written (if applicable)
• I ran pylint and attempted to implement some of it's feedback

Original prompt

Goal

Integrate the papiex C library compilation step into pip install epmt, following the same pattern that NOAA-GFDL/pyFMS uses to compile cFMS during pip install. The approach uses declarative metadata in pyproject.toml and imperative build logic in setup.py — the recommended setuptools pattern for custom build steps.

Context

Currently, EPMT and papiex are built and distributed separately:

• EPMT is a Python package with src/pyproject.toml using setuptools.build_meta as the build backend.
• papiex is a pure C project (https://github.com/NOAA-GFDL/papiex) compiled via make PREFIX=... install, producing shared libraries (libpapiex.so, libmonitor.so, etc.).
• The EPMT Makefile downloads the papiex source tarball via curl, compiles it (natively or in Docker), and bundles the resulting papiex-epmt-*.tgz alongside the EPMT pip package in the release.
• Pre-compiled .so files are included in the EPMT package via [tool.setuptools.package-data] with "lib/*.so*".

pyFMS demonstrates the pattern we want to follow. Its pyproject.toml declares build-backend = "setuptools.build_meta" and its setup.py defines a CustomBuild class that runs compile_c_libs.sh before the normal build:

# pyFMS setup.py
class CustomBuild(build):
    def run(self):
        with open("install.log", "w") as f:
            subprocess.run(["./compile_c_libs.sh"], stdout=f, check=True)
        build.run(self)

setup(cmdclass={"build": CustomBuild})

Implementation Plan

1. Create compile_papiex.sh in the src/ directory (next to pyproject.toml)

This script should:

• Check if the papiex source is available (either vendored as a submodule under src/vendor/papiex or already present).
• If the source isn't present, attempt to download it from https://github.com/NOAA-GFDL/papiex (using the branch/tag configured in the top-level Makefile variables PAPIEX_SRC_BRANCH).
• Check that gcc and make are available on the system.
• If dependencies are missing or compilation fails, warn but do not fail — EPMT should still be installable without papiex (it already handles missing papiex gracefully at runtime via epmt check).
• Compile papiex using make PREFIX=<target> install into src/epmt/lib/ so the resulting .so files are picked up by the existing [tool.setuptools.package-data] glob "lib/*.so*".
• Respect environment variables like CONFIG_PAPIEX_PAPI and CONFIG_PAPIEX_DEBUG that control papiex build options.
• Log compilation output to an install_papiex.log file.

2. Create src/setup.py

Following pyFMS's pattern, create a minimal setup.py that hooks into setuptools' build step:

"""Custom build step: compile papiex native libraries during pip install."""
import logging
import subprocess
import sys
from pathlib import Path

from setuptools import setup
from setuptools.command.build import build

logger = logging.getLogger(__name__)


class BuildPapiex(build):
    """Compile papiex native libraries before the standard build."""

    def run(self):
        script = Path(__file__).parent / "compile_papiex.sh"
        if script.exists():
            try:
                with open("install_papiex.log", "w") as log_fh:
                    subprocess.run(
                        [str(script)],
                        stdout=log_fh,
                        stderr=subprocess.STDOUT,
                        check=True,
                    )
            except subprocess.CalledProcessError:
                logger.warning(
                    "papiex compilation failed. EPMT will still install "
                    "but hardware counter collection will be unavailable. "
                    "See install_papiex.log for details."
                )
            except FileNotFoundError:
                logger.warning(
                    "compile_papiex.sh not found ��� skipping papiex build."
                )
        super().run()


setup(cmdclass={"build": BuildPapiex})

3. Update src/pyproject.toml

• Ensure build-backend = "setuptools.build_meta" is set (it already is).
• The [tool.setuptools.package-data] section already includes "lib/*.so*" which will pick up the compiled papiex libraries.
• No other changes should be needed to pyproject.toml since all declarative metadata remains there.

4. Update the top-level Makefile

• Review the papiex-related targets (papiex-dist, $(PAPIEX_RELEASE), $(PAPIEX_SRC), $(PAPIEX_SRC_TARBALL)) to see if they can be simplified or if they need to reference the new compile_papiex.sh.
• Add a new target (e.g., pip-install or install-with-papiex) that demonstrates pip install ./src with papiex compilation.
• Keep existing Docker-based build targets working for backward compatibility.
• Consider adding a vendor-papiex target that clones/downloads the papiex source into src/vendor/papiex/.

5. Update .github/workflows

Review...

This pull request was created from Copilot chat.

---

💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.67%. Comparing base (1543528) to head (e2b7fa4).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #171      +/-   ##
==========================================
+ Coverage   77.63%   77.67%   +0.04%     
==========================================
  Files          34       34              
  Lines        6765     6765              
==========================================
+ Hits         5252     5255       +3     
+ Misses       1513     1510       -3     

Flag	Coverage Δ
unittests	77.67% <100.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ilaflott]

appears to work at this point- freezing but not merging yet

[ilaflott]

this will not be ready for releasing 5.0.0, that is OK.

[ilaflott]

this is doing somethings right...

• the baremetal compilation case of papiex is handled, allowing the OUTSIDE_DOCKER condtion in the Makefile to be removed
• links up the build products with metadata in pyproject.toml
• tests are passing

todo:
look closer at how epmt check performing now across workflows where the new pip install approach is actually used

forseeable problem, with possible alternative:
compile_papiex.sh is not going to be fun to maintain, CMake with (or maybe even without) scikit-build-core could be more helpful.

[ilaflott]

review your changes against the recent test-PyPI packaging efforts, see .github/workflows/pip_publish.yml

review the feedback to your code generally

we're going to off-ramp this PR into a set of plans/notes for a fresh attempt

[ilaflott]

probably not safe