diff --git a/docs/spec.md b/docs/spec.md index f49812f..a6d3a22 100644 --- a/docs/spec.md +++ b/docs/spec.md @@ -2048,6 +2048,12 @@ sampler.run_with_samples( the file may be much larger than RAM. Same key handling as the in-memory branch: missing keys raise, extra keys are ignored. +Both branches return parameters in declaration order: the prior's nonlinear +parameters, then the extensions', then any explicitly sampled linear ones. The +order never depends on set iteration, so it is the same in every process +whatever `PYTHONHASHSEED` is (results written by separate processes, as in +harv-hq's shards, must agree on it). + `randomize_prior_order` (HDF5 path only): when `True` (default), batch *order* is permuted via `jax.random.permutation(key, n_batches)`. Each batch is still a single contiguous h5py slice — no random seeks, no read diff --git a/packages/hq/docs/implementation-plan.md b/packages/hq/docs/implementation-plan.md index a5a89ad..6b64d3f 100644 --- a/packages/hq/docs/implementation-plan.md +++ b/packages/hq/docs/implementation-plan.md @@ -26,8 +26,8 @@ Updated as each phase lands. A phase is done when its PR is merged. | — | hq scaffolding, spec, and this plan | Done (merged, PR #58) | | 1 | config, model file, IDs, Parquet I/O, provenance, CLI | Done (merged, PR #59) | | 2 | prepare, `read_source`, `read_sources` | Done (PR #60) | -| 3 | prior cache, serial rejection, result parts, resume | Implemented on branch `hq-phase3`, awaiting merge | -| 4 | execution modes: shards, pool, MPI | Not started | +| 3 | prior cache, serial rejection, result parts, resume | Done (PR #61) | +| 4 | execution modes: shards, pool, MPI | Implemented on branch `hq-phase4`, awaiting merge | | 5 | compaction, summarize, status | Not started | | 6 | MCMC follow-up | Not started | | 7 | web viewer | Not started | @@ -59,7 +59,6 @@ ______________________________________________________________________ packages/hq/src/harv_hq/ ├── __init__.py # curated __all__ (tuple) ├── _version.py # generated by hatch-vcs -├── _threads.py # pin_threads() -- called by pool workers before importing JAX ├── config.py # Config (frozen dataclasses, Config.from_file), ConfigError ├── _model_file.py # ModelFile: import prior.py, expose make_setup & hooks ├── ids.py # stable_hash, source_key, prior_cache_key @@ -70,7 +69,7 @@ packages/hq/src/harv_hq/ ├── summary_stats.py # per-source summary statistics + weighted resample ├── rejection.py # process one source: rejection ├── mcmc.py # process one source: MCMC -├── execution.py # serial / pool / MPI drivers around a per-source fn +├── execution.py # serial / pool drivers, mpi_comm, thread pinning, Progress ├── compact.py # compact(): merge a stage's parts, drop superseded rows ├── summarize.py # summarize(), status() ├── run.py # Run, SourceResult @@ -273,7 +272,7 @@ linear time (loose bound). ______________________________________________________________________ -## Phase 3: prior cache, serial rejection, result parts, resume — awaiting merge +## Phase 3: prior cache, serial rejection, result parts, resume — done Implemented on branch `hq-phase3` (stacked on `hq-phase2`). Deviations from the text below: `Run.run_rejection` takes `shard` and `overwrite` @@ -362,7 +361,30 @@ rows. ______________________________________________________________________ -## Phase 4: execution modes (shards, pool, MPI) +## Phase 4: execution modes (shards, pool, MPI) — awaiting merge + +Implemented on branch `hq-phase4`. Deviations from the text below: there is +no `_threads.py`. A worker cannot pin its own threads, because unpickling its +initializer imports `harv_hq`, which loads NumPy and JAX first, so the parent +sets the thread variables in the environment the workers are spawned with +and restores its own afterwards (spec updated). The drivers take +`(ids, data, setup, sink)`: `setup` is a picklable +`functools.partial(rejection_setup, config, cache_path)` that builds the +per-source function (each pool worker calls it once), and `sink` adds to the +`PartWriter` and counts progress. There is no `run_mpi` driver: `--mpi` takes +the slice from the rank and then runs the serial driver, or the pool driver +with `--workers` (`--mpi --workers` is allowed, for one rank per node; spec +updated). With `--overwrite`, +only rank 0 supersedes, between two barriers (spec updated). Every mode logs +progress about every 5% of its slice. The real-`mpirun` test is skipped +unless `mpi4py` and `mpirun` are both present, and could not be run locally +(Homebrew's Open MPI crashes even a one-rank job on this machine), so the MPI +path is also tested with a stand-in communicator. Found along the way, and +fixed in harv: `run_with_samples` ordered parameters by set iteration, so two +processes could write differently ordered columns, and the structure check +then refused the second one (harv spec updated). From the Copilot review of +PR #63: a rank that raises calls `comm.Abort(1)`, so its peers cannot hang at +a barrier (spec updated). **`execution.py`** wraps a per-source function in three drivers sharing one contract: given `(ids, data, process_fn, writer)`, process every ID and pass diff --git a/packages/hq/docs/spec.md b/packages/hq/docs/spec.md index 7d7b243..2e77ec9 100644 --- a/packages/hq/docs/spec.md +++ b/packages/hq/docs/spec.md @@ -601,22 +601,33 @@ falls as `N` grows. | Explicit shard | `hq run --shard 3/16` | `3/16` | `0003-of-0016-` | | Local pool | `hq run --workers 8 [--shard 3/16]` | the process's slice | as above, by the parent | | MPI | `mpirun -n 64 hq run --mpi` | `rank/size` | `-of--` | +| MPI with a pool per rank | `hq run --mpi --workers 32` | `rank/size` | as above, by each rank | - **Explicit shards** are for job arrays and task launchers (SLURM `--array=0-15` with `--shard $SLURM_ARRAY_TASK_ID/16`, disBatch). Shards are fully independent processes. - **Local pool** runs a `ProcessPoolExecutor` with the `spawn` start method. - Each worker is pinned to one CPU thread (BLAS and XLA environment variables - set before JAX is imported), so `W` workers use `W` cores. The worker + Each worker is pinned to one CPU thread, so `W` workers use `W` cores: the + parent sets the BLAS thread variables (`OMP_NUM_THREADS` and friends) and + appends single-thread flags to `XLA_FLAGS` in the environment the workers + are spawned with, and restores its own environment afterwards. The worker initializer imports the model file and builds the sampler once. The parent - loads the slice's data and sends each task its source's arrays; workers - return results as plain NumPy payloads, and only the parent buffers and - writes parts. + loads the slice's data and sends each task its source's data; at most + `2 * W` sources are in flight. Workers return plain payloads, and only the + parent buffers and writes parts, in completion order. - **MPI** uses `mpi4py` (the `harv-hq[mpi]` extra), imported lazily so a missing install raises `ImportError` naming `hq run --mpi`. Rank `r` of `n` processes slice `r/n` and writes its own parts, so ranks need no - communication beyond a final barrier. Rank 0 additionally logs aggregate - progress. `--mpi` is mutually exclusive with `--shard` and `--workers`. + communication beyond barriers: one after the start (with `--overwrite`, + only rank 0 moves the old results aside, and the others wait for it) and + one at the end. Rank 0 additionally logs progress extrapolated to all + ranks. A rank that raises (anything outside a single source's + processing, which only marks that source `failed`) logs the error and calls + `MPI.COMM_WORLD.Abort(1)`, so no rank is left waiting at a barrier. + `--mpi` is mutually exclusive with `--shard`. It combines with + `--workers`: each rank runs a local pool over its slice, so a cluster job + can use one rank per node (`srun --ntasks-per-node=1 hq run --mpi --workers 32`), and only the pool workers are pinned to one thread. +- Every mode logs a progress line about every 5% of its slice. - GPU runs use the single-process or explicit-shard modes, one process per device. diff --git a/packages/hq/pyproject.toml b/packages/hq/pyproject.toml index cb4212c..798c8d0 100644 --- a/packages/hq/pyproject.toml +++ b/packages/hq/pyproject.toml @@ -39,6 +39,9 @@ dependencies = [ "pyarrow>=17", ] +[project.optional-dependencies] +mpi = ["mpi4py>=4"] + [project.scripts] hq = "harv_hq.cli:main" diff --git a/packages/hq/src/harv_hq/cli.py b/packages/hq/src/harv_hq/cli.py index 05ad6d9..580c414 100644 --- a/packages/hq/src/harv_hq/cli.py +++ b/packages/hq/src/harv_hq/cli.py @@ -65,9 +65,11 @@ def _prior_cache(args: argparse.Namespace) -> int: def _run(args: argparse.Namespace) -> int: - if args.workers != 1 or args.mpi: - sys.exit("hq run: --workers and --mpi are not implemented yet; use --shard") - Run(args.run_dir).run_rejection(shard=args.shard, overwrite=args.overwrite) + if args.workers < 1: + sys.exit("hq run: --workers must be at least 1") + Run(args.run_dir).run_rejection( + shard=args.shard, workers=args.workers, mpi=args.mpi, overwrite=args.overwrite + ) return 0 diff --git a/packages/hq/src/harv_hq/execution.py b/packages/hq/src/harv_hq/execution.py new file mode 100644 index 0000000..735d974 --- /dev/null +++ b/packages/hq/src/harv_hq/execution.py @@ -0,0 +1,182 @@ +"""Drivers that run a per-source function over a slice of sources. + +See ``packages/hq/docs/spec.md``, "Execution modes". A driver gets the +slice's IDs and data, a picklable ``setup`` that builds the per-source +function (``setup() -> process(source_id, data) -> Payload``), and a ``sink`` +that receives every payload. Only the calling process writes results, so +drivers know nothing about parts. The MPI mode is the serial driver run on +slice ``rank/size`` (:func:`mpi_comm`). +""" + +__all__ = ("Progress", "mpi_comm", "run_pool", "run_serial") + +import logging +import multiprocessing +import os +from collections.abc import Callable, Mapping, Sequence +from concurrent.futures import FIRST_COMPLETED, Future, ProcessPoolExecutor, wait +from typing import Any, cast, final +from unittest.mock import patch + +from harv_hq.results import Payload + +logger = logging.getLogger("harv_hq") + +ProcessFn = Callable[[Any, Any], Payload] +Setup = Callable[[], ProcessFn] +Sink = Callable[[Payload], None] + +# One CPU thread per pool worker, so W workers use W cores. Set in the +# environment the workers are spawned with: by the time a worker's own code +# runs, importing harv_hq has already loaded NumPy (BLAS) and JAX. +_THREAD_ENV = { + "OMP_NUM_THREADS": "1", + "OPENBLAS_NUM_THREADS": "1", + "MKL_NUM_THREADS": "1", + "VECLIB_MAXIMUM_THREADS": "1", +} +_XLA_SINGLE_THREAD = "--xla_cpu_multi_thread_eigen=false intra_op_parallelism_threads=1" + + +@final +class Progress: + """Log a progress line about every 5% of a slice. + + Parameters + ---------- + label + The stage, for the log line. + total + Sources in this process's slice. + ranks + MPI ranks; above 1, the line also extrapolates this rank's count to + the whole run (rank 0 only logs this). + """ + + def __init__(self, label: str, total: int, *, ranks: int = 1) -> None: + self.label = label + self.total = total + self.ranks = ranks + self.done = 0 + self._every = max(1, total // 20) + + def tick(self) -> None: + """Count one finished source.""" + self.done += 1 + if self.done % self._every and self.done != self.total: + return + overall = ( + f" (about {self.done * self.ranks}/{self.total * self.ranks} " + f"over {self.ranks} ranks)" + if self.ranks > 1 + else "" + ) + logger.info("%s: %d/%d sources%s", self.label, self.done, self.total, overall) + + +def run_serial( + ids: Sequence[Any], data: Mapping[Any, Any], setup: Setup, sink: Sink +) -> None: + """Process every source in this process, in order. + + Parameters + ---------- + ids + The sources to process, in order. + data + Source ID to its data. + setup + Builds the per-source function. + sink + Receives each payload. + """ + process = setup() + for source_id in ids: + sink(process(source_id, data[source_id])) + + +def run_pool( + ids: Sequence[Any], + data: Mapping[Any, Any], + setup: Setup, + sink: Sink, + *, + workers: int, +) -> None: + """Process sources on ``workers`` spawned processes; payloads return here. + + Each worker runs ``setup`` once (its initializer) and is pinned to one + CPU thread. At most ``2 * workers`` sources are in flight, so memory stays + flat however long ``ids`` is. Payloads reach ``sink`` in completion order. + + Parameters + ---------- + ids + The sources to process; submitted in this order. + data + Source ID to its data; each task carries one source's data. + setup + Builds the per-source function; must be picklable. + sink + Receives each payload, in this process. + workers + Number of worker processes. + """ + changes = {**_THREAD_ENV, "XLA_FLAGS": _XLA_SINGLE_THREAD} + if flags := os.environ.get("XLA_FLAGS"): + changes["XLA_FLAGS"] = f"{flags} {_XLA_SINGLE_THREAD}" + # Set only while the workers are spawned; this process's environment is + # restored afterwards. + with patch.dict(os.environ, changes): + pool = ProcessPoolExecutor( + workers, + mp_context=multiprocessing.get_context("spawn"), + initializer=_init_worker, + initargs=(setup,), + ) + try: + pending: set[Future[Payload]] = set() + for source_id in ids: + if len(pending) >= 2 * workers: + finished, pending = wait(pending, return_when=FIRST_COMPLETED) + for future in finished: + sink(future.result()) + pending.add(pool.submit(_process_in_worker, source_id, data[source_id])) + for future in wait(pending).done: + sink(future.result()) + finally: + # On an interruption, drop queued work rather than wait for it. + pool.shutdown(wait=True, cancel_futures=True) + + +def mpi_comm() -> Any: + """``MPI.COMM_WORLD``, importing ``mpi4py`` only when ``--mpi`` is used. + + Returns + ------- + The world communicator. + + Raises + ------ + ImportError + If ``mpi4py`` is not installed. + """ + try: + from mpi4py import MPI # noqa: PLC0415 # ty: ignore[unresolved-import] + except ImportError as err: + msg = "hq run --mpi needs mpi4py: pip install 'harv-hq[mpi]'" + raise ImportError(msg) from err + return MPI.COMM_WORLD + + +# The per-source function, built once per worker by _init_worker. +_worker_process: ProcessFn | None = None + + +def _init_worker(setup: Setup) -> None: + global _worker_process # noqa: PLW0603 -- one per worker process + _worker_process = setup() + + +def _process_in_worker(source_id: Any, data: Any) -> Payload: + return cast("ProcessFn", _worker_process)(source_id, data) diff --git a/packages/hq/src/harv_hq/rejection.py b/packages/hq/src/harv_hq/rejection.py index e938405..1a30c84 100644 --- a/packages/hq/src/harv_hq/rejection.py +++ b/packages/hq/src/harv_hq/rejection.py @@ -5,11 +5,13 @@ hand it across processes. """ -__all__ = ("process_rejection",) +__all__ = ("process_rejection", "rejection_setup") +import functools import time import traceback import warnings +from collections.abc import Callable from datetime import UTC, datetime from pathlib import Path from typing import Any @@ -18,12 +20,50 @@ from harv.data import GaiaAstrometryData, RVData from harv.samplers import RejectionSampler -from harv_hq.config import RejectionConfig +from harv_hq._model_file import ModelFile +from harv_hq.config import Config, RejectionConfig from harv_hq.ids import source_key, stable_hash from harv_hq.results import Payload, samples_table from harv_hq.summary_stats import summary_stats +def rejection_setup( + config: Config, prior_cache: Path +) -> Callable[[Any, RVData | GaiaAstrometryData], Payload]: + """Build a run's per-source rejection function, once per process. + + Imports the model file, calls ``make_setup()``, and builds one + ``RejectionSampler``. Pass it with :func:`functools.partial` as an + execution driver's ``setup``, so that pool workers build their own. + + Parameters + ---------- + config + The run's configuration. + prior_cache + Path to the shared prior cache. + + Returns + ------- + ``process(source_id, data) -> Payload``. + """ + prior, model = ModelFile.load(config.run.model_file).setup(config.run.kind) + rejection = config.rejection + sampler = RejectionSampler( + prior, + model, + batch_size=rejection.batch_size, + min_evidence_ess=rejection.min_evidence_ess, + ) + return functools.partial( + process_rejection, + sampler=sampler, + prior_cache=prior_cache, + config=rejection, + seed=config.run.seed, + ) + + def process_rejection( source_id: Any, data: RVData | GaiaAstrometryData, diff --git a/packages/hq/src/harv_hq/run.py b/packages/hq/src/harv_hq/run.py index f8924fb..2673ee2 100644 --- a/packages/hq/src/harv_hq/run.py +++ b/packages/hq/src/harv_hq/run.py @@ -6,6 +6,7 @@ __all__ = ("Run", "SourceResult") +import functools import json import logging import os @@ -17,14 +18,21 @@ import h5py from harv.data import GaiaAstrometryData, RVData -from harv.samplers import RejectionSampler, Samples, make_prior_cache +from harv.samplers import Samples, make_prior_cache from harv_hq._model_file import ModelFile from harv_hq.config import Config +from harv_hq.execution import Progress, mpi_comm, run_pool, run_serial from harv_hq.ids import prior_cache_key from harv_hq.prepare import PreparedData, prepare from harv_hq.provenance import check_provenance, make_provenance -from harv_hq.rejection import process_rejection -from harv_hq.results import PartWriter, ResultsIndex, read_source_samples, supersede +from harv_hq.rejection import rejection_setup +from harv_hq.results import ( + PartWriter, + Payload, + ResultsIndex, + read_source_samples, + supersede, +) logger = logging.getLogger("harv_hq") @@ -146,6 +154,8 @@ def run_rejection( self, *, shard: tuple[int, int] = (0, 1), + workers: int = 1, + mpi: bool = False, overwrite: bool = False, ) -> None: """Run the rejection sampler on every source in a slice (``hq run``). @@ -154,31 +164,68 @@ def run_rejection( ``hq.toml`` and model file, skips sources already done (see the spec's "Resume"), and writes result parts as it goes. Anything still buffered is written when the loop ends, including when it is - interrupted by an exception. + interrupted by an exception. See the spec's "Execution modes". Parameters ---------- shard This process's slice ``(i, N)``. + workers + Local worker processes; above 1, sources run on a spawned pool + and this process writes the parts. Combines with ``mpi`` (a pool + per rank, e.g. one rank per node). + mpi + Process slice ``rank/size`` of ``MPI.COMM_WORLD`` (needs + ``mpi4py``); excludes ``shard``. overwrite Move existing rejection results to ``results/superseded--rejection/`` and start over. Raises ------ + ValueError + If ``mpi`` is combined with ``shard``, or ``workers < 1``. FileNotFoundError If the data are not prepared or the prior cache is not built. ProvenanceError If an input or an existing part was built from different inputs. """ + if workers < 1: + msg = f"workers must be at least 1, got {workers}" + raise ValueError(msg) + comm = None + if mpi: + if shard != (0, 1): + msg = "mpi=True takes its slice from the MPI rank; omit shard" + raise ValueError(msg) + comm = mpi_comm() + shard = (comm.Get_rank(), comm.Get_size()) + try: + self._run_rejection(shard, workers, overwrite, comm) + except BaseException: + # A rank that stops early would leave the others waiting at a + # barrier forever; nothing above it can recover, so end them all. + if comm is not None: + logger.exception("rejection: rank %d failed; aborting", shard[0]) + comm.Abort(1) + raise + + def _run_rejection( + self, shard: tuple[int, int], workers: int, overwrite: bool, comm: Any + ) -> None: + i, n = shard prepared = PreparedData(self.run_dir) expected = make_provenance(self.config, data_id=prepared.data_id) check_provenance(prepared.provenance, expected, path=prepared.path) cache_path, cache_provenance = self._checked_prior_cache(expected) stage_dir = self.run_dir / "results" / "rejection" - if overwrite: + # Under MPI only rank 0 moves results aside, and every rank waits for + # it before reading what is done. + if overwrite and (comm is None or i == 0): supersede(stage_dir) + if comm is not None: + comm.Barrier() part_provenance = { **expected, "prior_cache_id": cache_provenance["prior_cache_id"], @@ -186,7 +233,6 @@ def run_rejection( existing = ResultsIndex.build(stage_dir, expected_provenance=part_provenance) done = existing.done() todo = [sid for sid in prepared.slice_ids(shard) if sid not in done] - i, n = shard log_handler, log_level = self._log_to_file(f"rejection-{i:04d}-of-{n:04d}.log") logger.info( "rejection: slice %d/%d, %d sources to run (%d already done)", @@ -197,14 +243,7 @@ def run_rejection( ) try: data = prepared.read(todo) - prior, model = self.model_file.setup(self.config.run.kind) - rejection = self.config.rejection - sampler = RejectionSampler( - prior, - model, - batch_size=rejection.batch_size, - min_evidence_ess=rejection.min_evidence_ess, - ) + setup = functools.partial(rejection_setup, self.config, cache_path) writer = PartWriter( stage_dir, stage="rejection", @@ -214,18 +253,19 @@ def run_rejection( flush_seconds=self.config.results.flush_seconds, structure=existing.structure, ) + progress = Progress( + "rejection", len(todo), ranks=n if comm is not None and i == 0 else 1 + ) + + def sink(payload: Payload) -> None: + writer.add(payload) + progress.tick() + try: - for source_id in todo: - writer.add( - process_rejection( - source_id, - data[source_id], - sampler=sampler, - prior_cache=cache_path, - config=rejection, - seed=self.config.run.seed, - ) - ) + if workers > 1: + run_pool(todo, data, setup, sink, workers=workers) + else: + run_serial(todo, data, setup, sink) finally: writer.close() finally: @@ -234,6 +274,8 @@ def run_rejection( logger.removeHandler(log_handler) logger.setLevel(log_level) log_handler.close() + if comm is not None: + comm.Barrier() def load_source(self, source_id: Any) -> SourceResult: """Everything hq holds for one source. diff --git a/packages/hq/tests/unit/test_hq_rejection.py b/packages/hq/tests/unit/test_hq_rejection.py index 0765fa7..14c9309 100644 --- a/packages/hq/tests/unit/test_hq_rejection.py +++ b/packages/hq/tests/unit/test_hq_rejection.py @@ -1,6 +1,11 @@ """Tests for ``hq prior-cache`` and ``hq run``: result parts, resume, provenance.""" +import importlib.util +import logging +import os import shutil +import subprocess +import sys from datetime import UTC, datetime import jax @@ -12,8 +17,10 @@ from harv.samplers import RejectionSampler, Samples from harv_hq import ProvenanceError, Run +from harv_hq import rejection as rejection_module from harv_hq import run as run_module from harv_hq.cli import main +from harv_hq.execution import Progress from harv_hq.ids import source_key from harv_hq.prepare import PreparedData from harv_hq.results import ( @@ -166,9 +173,9 @@ def flaky(self, data, *args: object, **kwargs: object): # Failed sources are not done: the next resume reruns them, and only them. monkeypatch.setattr(RejectionSampler, "run_with_samples", original) calls = [] - process = run_module.process_rejection + process = rejection_module.process_rejection monkeypatch.setattr( - run_module, + rejection_module, "process_rejection", lambda sid, *a, **k: calls.append(sid) or process(sid, *a, **k), ) @@ -179,7 +186,7 @@ def flaky(self, data, *args: object, **kwargs: object): def test_resume_after_an_interruption(self, rv_run, monkeypatch): run = prepared_run(rv_run) calls = [] - original = run_module.process_rejection + original = rejection_module.process_rejection def interrupted(source_id, *args: object, **kwargs: object): if len(calls) == 6: @@ -187,7 +194,7 @@ def interrupted(source_id, *args: object, **kwargs: object): calls.append(source_id) return original(source_id, *args, **kwargs) - monkeypatch.setattr(run_module, "process_rejection", interrupted) + monkeypatch.setattr(rejection_module, "process_rejection", interrupted) # Caches the (empty) index on this Run. assert run.load_source(next(iter(rv_run.kept))).rejection_status == "pending" with pytest.raises(KeyboardInterrupt): @@ -197,10 +204,10 @@ def interrupted(source_id, *args: object, **kwargs: object): # ...and the cached index was dropped, so this Run sees it too. assert run.load_source(calls[0]).rejection_status == "ok" - monkeypatch.setattr(run_module, "process_rejection", original) + monkeypatch.setattr(rejection_module, "process_rejection", original) rerun = [] monkeypatch.setattr( - run_module, + rejection_module, "process_rejection", lambda sid, *a, **k: rerun.append(sid) or original(sid, *a, **k), ) @@ -320,17 +327,178 @@ def test_a_parameter_named_like_a_bookkeeping_column_is_refused(self, name): samples_table(samples, "a", stage="rejection") -class TestShards: - def test_two_shards_cover_every_source(self, rv_run): +def all_samples(run): + """Every result's samples, in a fixed order, for comparing runs.""" + paths = sorted(str(p) for p in stage_dir(run).glob("*.samples.parquet")) + return ( + ds.dataset(paths) + .to_table() + .sort_by([("source_id", "ascending"), ("sample_index", "ascending")]) + ) + + +def twin_runs(fixture, tmp_path): + """Two prepared copies of one run directory (same seed, data, and cache).""" + run = prepared_run(fixture) + shutil.copytree(run.run_dir, tmp_path / "twin") + return run, Run(tmp_path / "twin") + + +class TestExecutionModes: + def test_shards_match_a_serial_run(self, rv_run, tmp_path): + serial, sharded = twin_runs(rv_run, tmp_path) + serial.run_rejection() + for i in range(3): + sharded.run_rejection(shard=(i, 3)) + assert all_samples(sharded).equals(all_samples(serial)) + names = {p.name[:12] for p in stage_dir(sharded).glob("*.sources.parquet")} + assert names == {f"000{i}-of-0003" for i in range(3)} + + def test_a_pool_matches_a_serial_run(self, rv_run, tmp_path): + serial, pooled = twin_runs(rv_run, tmp_path) + serial.run_rejection() + environ = dict(os.environ) + pooled.run_rejection(workers=2) + # The workers' thread pinning does not leak into this process. + assert dict(os.environ) == environ + assert all_samples(pooled).equals(all_samples(serial)) + assert index(pooled).status_counts() == {"ok": len(rv_run.kept)} + assert all( + p.name.startswith("0000-of-0001-") + for p in stage_dir(pooled).glob("*.parquet") + ) + + def test_changing_the_shard_count_recomputes_nothing(self, rv_run, monkeypatch): run = prepared_run(rv_run) - run.run_rejection(shard=(0, 2)) + run.run_rejection(shard=(0, 4)) + run.run_rejection(shard=(1, 4)) first = set(index(run).records) - run.run_rejection(shard=(1, 2)) assert first assert first != set(rv_run.kept) + calls = [] + process = rejection_module.process_rejection + monkeypatch.setattr( + rejection_module, + "process_rejection", + lambda sid, *a, **k: calls.append(sid) or process(sid, *a, **k), + ) + run.run_rejection() + assert set(calls) == set(rv_run.kept) - first assert set(index(run).records) == set(rv_run.kept) + + def test_two_processes_on_one_slice_resolve_to_the_newest( + self, rv_run, monkeypatch + ): + run = prepared_run(rv_run) + run.run_rejection() + first_parts = set(stage_dir(run).glob("*.sources.parquet")) + # A second process that started before the first wrote anything. + monkeypatch.setattr(ResultsIndex, "done", lambda _self: set()) + run.run_rejection() + second_parts = set(stage_dir(run).glob("*.sources.parquet")) - first_parts + assert len(second_parts) == len(first_parts) # distinct names, none lost + records = index(run).records + assert set(records) == set(rv_run.kept) + assert {r.sources_path for r in records.values()} <= second_parts + + def test_mpi_excludes_shard(self, rv_run): + run = Run(rv_run.run_dir) + with pytest.raises(ValueError, match="omit shard"): + run.run_rejection(mpi=True, shard=(0, 2)) + with pytest.raises(ValueError, match="at least 1"): + run.run_rejection(workers=0) + + def test_mpi_ranks_take_their_slices(self, rv_run, monkeypatch): + """The MPI path, with a stand-in communicator: ranks run one at a time.""" + run = prepared_run(rv_run) + run.run_rejection() + barriers = [] + + class FakeComm: + def __init__(self, rank): + self.rank = rank + + def Get_rank(self): + return self.rank + + def Get_size(self): + return 2 + + def Barrier(self): + barriers.append(self.rank) + + # --overwrite: only rank 0 moves results aside, so rank 1 does not move + # rank 0's new parts. Rank 1 also runs a pool (--mpi --workers 2). + for rank, workers in ((0, 1), (1, 2)): + monkeypatch.setattr(run_module, "mpi_comm", lambda r=rank: FakeComm(r)) + run.run_rejection(mpi=True, workers=workers, overwrite=True) + assert barriers == [0, 0, 1, 1] # after the overwrite, and at the end + assert len(list((run.run_dir / "results").glob("superseded-*"))) == 1 + assert index(run).status_counts() == {"ok": len(rv_run.kept)} names = {p.name[:12] for p in stage_dir(run).glob("*.sources.parquet")} assert names == {"0000-of-0002", "0001-of-0002"} + logs = {p.name for p in (run.run_dir / "logs").iterdir()} + assert {"rejection-0000-of-0002.log", "rejection-0001-of-0002.log"} <= logs + + def test_a_failing_rank_aborts_every_rank(self, rv_run, monkeypatch): + run = prepared_run(rv_run) + aborted = [] + + class FakeComm: + def Get_rank(self): + return 0 + + def Get_size(self): + return 2 + + def Barrier(self): + pass + + def Abort(self, code): + aborted.append(code) + + def broken(stage_dir): + raise OSError("disk full") + + monkeypatch.setattr(run_module, "mpi_comm", FakeComm) + monkeypatch.setattr(run_module, "supersede", broken) + with pytest.raises(OSError, match="disk full"): + run.run_rejection(mpi=True, overwrite=True) + assert aborted == [1] + + def test_mpi_without_mpi4py(self, rv_run, monkeypatch): + monkeypatch.setitem(sys.modules, "mpi4py", None) + with pytest.raises(ImportError, match="harv-hq\\[mpi\\]"): + Run(rv_run.run_dir).run_rejection(mpi=True) + + @pytest.mark.skipif( + importlib.util.find_spec("mpi4py") is None or shutil.which("mpirun") is None, + reason="needs mpi4py and mpirun", + ) + def test_mpi(self, rv_run): + run = prepared_run(rv_run) + subprocess.run( # noqa: S603 -- fixed argv + [ + *("mpirun", "-n", "2", sys.executable), + *("-m", "harv_hq.cli", "run", "--mpi", "--run-dir", str(run.run_dir)), + ], + check=True, + timeout=600, + ) + assert index(run).status_counts() == {"ok": len(rv_run.kept)} + names = {p.name[:12] for p in stage_dir(run).glob("*.sources.parquet")} + assert names == {"0000-of-0002", "0001-of-0002"} + + +def test_progress_logs_about_every_five_percent(caplog): + progress = Progress("rejection", 40, ranks=4) + with caplog.at_level(logging.INFO, logger="harv_hq"): + for _ in range(40): + progress.tick() + assert len(caplog.records) == 20 + assert caplog.messages[-1] == ( + "rejection: 40/40 sources (about 160/160 over 4 ranks)" + ) class TestProvenance: @@ -369,8 +537,8 @@ def test_cli(rv_run): ResultsIndex.build(rv_run.run_dir / "results" / "rejection").records ) == set(rv_run.kept) with pytest.raises(SystemExit) as excinfo: - main(["run", "--run-dir", run_dir, "--workers", "2"]) - assert "not implemented yet" in str(excinfo.value.code) + main(["run", "--run-dir", run_dir, "--workers", "0"]) + assert "at least 1" in str(excinfo.value.code) def test_copy_of_a_run_directory_still_resumes(rv_run, tmp_path): diff --git a/src/harv/samplers/rejection.py b/src/harv/samplers/rejection.py index 58a24a0..91f41f3 100644 --- a/src/harv/samplers/rejection.py +++ b/src/harv/samplers/rejection.py @@ -940,9 +940,13 @@ def run_with_samples( def _expected_prior_keys( self, prepared: _PreparedSamplerModel - ) -> tuple[set[str], set[str]]: + ) -> tuple[tuple[str, ...], tuple[str, ...]]: """Return ``(expected_nonlinear_keys, expected_explicit_linear_keys)``. + Both are in declaration order (prior, then extensions), never set + order: callers build the sample dicts by iterating them, and set order + varies between processes with ``PYTHONHASHSEED``. + The flat sample dict the sampler consumes contains base nonlinear + extension nonlinear under the first set, plus the linear params sampled explicitly under the second set. The explicit-linear set depends on the @@ -951,15 +955,15 @@ def _expected_prior_keys( otherwise it is only the non-Gaussian linear params (those that cannot be analytically marginalized). """ - nonlinear_keys = set(self.prior.nonlinear_priors) | set( - prepared.nonlinear_extension_priors - ) - explicit_linear_keys = set( - _explicit_linear_names( - prepared.effective_linear_prior or {}, - prepared.effective_marginalized_names, + nonlinear_keys = tuple( + dict.fromkeys( + [*self.prior.nonlinear_priors, *prepared.nonlinear_extension_priors] ) ) + explicit_linear_keys = _explicit_linear_names( + prepared.effective_linear_prior or {}, + prepared.effective_marginalized_names, + ) return nonlinear_keys, explicit_linear_keys def _flatten_prior_samples( @@ -980,7 +984,7 @@ def _flatten_prior_samples( got_nl = set(prior_samples.nonlinear) got_lin = set(prior_samples.linear) - missing = (expected_nl - got_nl) | (expected_lin - got_lin) + missing = (set(expected_nl) - got_nl) | (set(expected_lin) - got_lin) if missing: msg = ( "Prior samples key mismatch for this (prior, model) setup. " @@ -1057,7 +1061,7 @@ def _evaluate_from_hdf5( accumulated samples are not biased toward the start of the file. """ expected_nl, expected_lin = self._expected_prior_keys(prepared) - expected_keys = expected_nl | expected_lin + expected_keys = tuple(dict.fromkeys([*expected_nl, *expected_lin])) with h5py.File(path, "r") as f: nonlinear_group = f["nonlinear"] @@ -1068,7 +1072,7 @@ def _evaluate_from_hdf5( available_nl = set(nonlinear_group) available_lin = set(linear_group) available = available_nl | available_lin - missing = expected_keys - available + missing = set(expected_keys) - available if missing: msg = ( f"Prior cache at {path} is missing required keys: " diff --git a/tests/unit/samplers/test_prior_cache.py b/tests/unit/samplers/test_prior_cache.py index 1f578f3..1017c6f 100644 --- a/tests/unit/samplers/test_prior_cache.py +++ b/tests/unit/samplers/test_prior_cache.py @@ -377,6 +377,25 @@ def test_disk_matches_in_memory_sequential(self, tmp_path: Path): np.sort(np.asarray(mem["period"].value)), ) + def test_parameter_order_is_the_declaration_order(self, tmp_path: Path): + """Both paths order parameters as declared, never in set order. + + Set order changes between processes with ``PYTHONHASHSEED``, so results + written by different processes (harv-hq's shards and MPI ranks) would + disagree on column order. + """ + prior = _rv_prior_with_jitter() + model = RVModel(extensions=(Jitter(obs_unit="km/s"),)) + sampler = RejectionSampler(prior, model, batch_size=200) + path = tmp_path / "cache.h5" + make_prior_cache(prior, model, 400, path, key=jr.key(0), batch_size=200) + expected = [*prior.nonlinear_priors, "jitter"] + for cache in (path, Samples.from_hdf5(path)): + samples = sampler.run_with_samples( + _rv_data(), cache, key=jr.key(1), top_k=8 + ) + assert list(samples.nonlinear) == expected + def test_randomize_does_not_change_n_samples_dramatically(self, tmp_path: Path): """randomize_prior_order=True still produces a valid posterior.""" prior = _rv_prior() diff --git a/uv.lock b/uv.lock index 0b6543a..fa094da 100644 --- a/uv.lock +++ b/uv.lock @@ -1321,15 +1321,22 @@ dependencies = [ { name = "pyarrow" }, ] +[package.optional-dependencies] +mpi = [ + { name = "mpi4py" }, +] + [package.metadata] requires-dist = [ { name = "astropy", specifier = ">=6.1" }, { name = "h5py", specifier = ">=3.11" }, { name = "harv", editable = "." }, { name = "jax", specifier = ">=0.8.1" }, + { name = "mpi4py", marker = "extra == 'mpi'", specifier = ">=4" }, { name = "numpy", specifier = ">=2.0" }, { name = "pyarrow", specifier = ">=17" }, ] +provides-extras = ["mpi"] [[package]] name = "hatch" @@ -2288,6 +2295,44 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/98/6af411189d9413534c3eb691182bff1f5c6d44ed2f93f2edfe52a1bbceb8/more_itertools-11.0.2-py3-none-any.whl", hash = "sha256:6e35b35f818b01f691643c6c611bc0902f2e92b46c18fffa77ae1e7c46e912e4", size = 71939, upload-time = "2026-04-09T15:01:32.21Z" }, ] +[[package]] +name = "mpi4py" +version = "4.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/75/83/231445bbcf7ef10864746c244ff2d82000011449b79275642c5d4ed8c8f4/mpi4py-4.1.2.tar.gz", hash = "sha256:56860286dc45f20e8821e93cb06669e30462348bf866f685553fa4b712d58d02", size = 501709, upload-time = "2026-05-16T10:35:23.618Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bf/7e/1524baa98c3b4e3ddc030ec2ad5f6e5a362930317ed3e561d6da1c2e97b1/mpi4py-4.1.2-cp310-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ceb3b6e6f27ba7a39f7721583c04514013b860b829e721769e77398dee97bfa3", size = 1411317, upload-time = "2026-05-16T10:33:56.706Z" }, + { url = "https://files.pythonhosted.org/packages/2f/73/5918e062a65f40e3bd82a2dccf0c8ba22a284d327ccda46d84d2717985cc/mpi4py-4.1.2-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:251e8880f4cb98e9c8f63c6f6b2c7e819e22b5e4949d47767a0092eed6f814c2", size = 1304635, upload-time = "2026-05-16T10:33:58.685Z" }, + { url = "https://files.pythonhosted.org/packages/a3/0a/1da7f403e0d8ce0e26d541f7538302cec00cf5b0a98a7a52b929f938a25c/mpi4py-4.1.2-cp310-abi3-manylinux1_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:2ef63b2e3083e6062fd90e4de8c4e3acbf81e0772406e0226eb8dde6a48cab8e", size = 1327130, upload-time = "2026-05-16T10:34:00.269Z" }, + { url = "https://files.pythonhosted.org/packages/e6/f9/65999152ae82bad914c6a083821ee774afefd6d0544e633b940c9a9ebf3f/mpi4py-4.1.2-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6508e654b9c8ff9f611b19548b2a17d1e323b520a15168189f92221e6757b8ff", size = 1182268, upload-time = "2026-05-16T10:34:02.206Z" }, + { url = "https://files.pythonhosted.org/packages/66/8f/90de8d5e0676f86f7866fd4665193252ea17764a54800d9545c7b53ac00a/mpi4py-4.1.2-cp310-abi3-win_amd64.whl", hash = "sha256:eb69f6273ad155f191850a593deebdf52aed6722979ba0693e02db5663e59699", size = 1466751, upload-time = "2026-05-16T10:34:04.278Z" }, + { url = "https://files.pythonhosted.org/packages/3e/73/d3d747f74c5aa7be6faf84aef6b234458baeeb0acb35143c288bc04133c5/mpi4py-4.1.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:78983a9606e105af1af907878ec41627360344ea576c75afdaca4124db9d0b13", size = 1621733, upload-time = "2026-05-16T10:34:24.804Z" }, + { url = "https://files.pythonhosted.org/packages/cb/e0/2d029ea1907a477126039a48aadb30113cf3df5bd3c2f27331334b281415/mpi4py-4.1.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4e5fdba71cf27d0d3b45e706736d6690060976345f45a6408b3594884bb0cc6e", size = 1393929, upload-time = "2026-05-16T10:34:26.585Z" }, + { url = "https://files.pythonhosted.org/packages/9f/2b/1e48c4c5f9acbdca8dd28beeba9123dde140cd2ca520f8e3a3cf22faeeaa/mpi4py-4.1.2-cp312-cp312-manylinux1_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:00f4cce8999d19f35243c3442ea22debbe3336f69c309cd5d3176df4e51c717a", size = 1358844, upload-time = "2026-05-16T10:34:28.247Z" }, + { url = "https://files.pythonhosted.org/packages/94/46/a37225d47997fcf30adca25d3849d035bbb61d972118b024db900306e528/mpi4py-4.1.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:0a58e164776acb7b52414548b1bf0e5caafce0ee90345deff147873a64b6b2cc", size = 1227206, upload-time = "2026-05-16T10:34:29.866Z" }, + { url = "https://files.pythonhosted.org/packages/b5/48/14173d8baaf622b76e0e8983f53fffff8538edcf24a020028432e63f20d3/mpi4py-4.1.2-cp312-cp312-win_amd64.whl", hash = "sha256:0a7227a53ba24102f2c4c2c1b226d028fabc473b20ceef5435f1a7ac8cbd508c", size = 1706260, upload-time = "2026-05-16T10:34:31.806Z" }, + { url = "https://files.pythonhosted.org/packages/35/15/324cfa61674b8f73aa3347590df937232d898a993f3348121f9003f43b16/mpi4py-4.1.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:632a94e355cf1960eaae2b668d88399f53680207f5a4777e15fdaf7d96c72d09", size = 1660517, upload-time = "2026-05-16T10:34:33.836Z" }, + { url = "https://files.pythonhosted.org/packages/00/ec/b670c9250b59ed255571e679f1e0ae7bd4e08c67f2f7f7f11a928b4becd7/mpi4py-4.1.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0aedf3e6c937b48d244e7c6bb72d0b068a1b1adf99ec6001bc9c7e381c581806", size = 1426478, upload-time = "2026-05-16T10:34:35.342Z" }, + { url = "https://files.pythonhosted.org/packages/0d/5e/493415cdb0da0c1c0b6ec9a1fb65ab57a174e8111b25c87f7507f663d0b4/mpi4py-4.1.2-cp313-cp313-manylinux1_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:085e0cb05427398fe2281856f27a87984b9f234cd6d98a2a384a6fbfe679a56f", size = 1358571, upload-time = "2026-05-16T10:34:37.478Z" }, + { url = "https://files.pythonhosted.org/packages/6f/51/3822e834fc9cafd96811501b85232e02bd9b31596d42965536a269a6c112/mpi4py-4.1.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:429437883b4511583d56a57bc58ad0d02965fb6c19f84f39e30c0646e6c0cab9", size = 1226562, upload-time = "2026-05-16T10:34:39.652Z" }, + { url = "https://files.pythonhosted.org/packages/72/27/919831e5c843890b1a5b475ca1a8d83d8eb3453c1fe66f6bbdaca35ee548/mpi4py-4.1.2-cp313-cp313-win_amd64.whl", hash = "sha256:f984aa35d61fd95c7aa49829f94c3d61bb9ec2272bd8e404ae8d1ea84b620b58", size = 1703612, upload-time = "2026-05-16T10:34:41.133Z" }, + { url = "https://files.pythonhosted.org/packages/ef/ee/3c1128b5c393a8a2abeb9db6cc2aba2d3d604fee20d0fa99c02ba5aab5c6/mpi4py-4.1.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:16ae4d0fac11e60056ec7c46ce101f70f4941634ad8165c54a24304820a816b0", size = 1740466, upload-time = "2026-05-16T10:34:42.566Z" }, + { url = "https://files.pythonhosted.org/packages/17/ed/7a163be7da48ace29ab4c54407e7f2e510953df14dea90e122f9ca7310c7/mpi4py-4.1.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:364da611dda94e8a26e418ccfc4c8ff2a1492e74f4c28b24df09521fe888de3b", size = 1518426, upload-time = "2026-05-16T10:34:44.081Z" }, + { url = "https://files.pythonhosted.org/packages/ec/5e/d358fadb8672d58abd6dce16c95eda56f497b378dec5642a31cd6ededfc4/mpi4py-4.1.2-cp313-cp313t-manylinux1_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:85e332feaac3323d8ed1c71f17478fe3f529468f28f6f9bb4c9133ad2d3a3a6d", size = 1419007, upload-time = "2026-05-16T10:34:45.47Z" }, + { url = "https://files.pythonhosted.org/packages/27/eb/5cd53880337009cab9a9d17a007ad5aec731f3a55e211bdcc99dbb98a0e3/mpi4py-4.1.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3e05dba75f6ad17ae761bdc01ee5c5271d15cd296e8a508e22c99c91b540dc99", size = 1298482, upload-time = "2026-05-16T10:34:46.858Z" }, + { url = "https://files.pythonhosted.org/packages/13/d9/eb35b0e8328a02c4d6288bf2bcd156b387f08ff6146a5c8f3f2abd75a078/mpi4py-4.1.2-cp313-cp313t-win_amd64.whl", hash = "sha256:2d0a131a255f61a2d313623f91b496c345abd5976147f866b39383264f275449", size = 1835283, upload-time = "2026-05-16T10:34:48.899Z" }, + { url = "https://files.pythonhosted.org/packages/2e/0d/6ad35d0d50cb519f1c1658d754bfcf553a48611043d91d6b5e7c9d3cdca1/mpi4py-4.1.2-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:0bfd4739cf1189d3d1f44b360e8fab203cbab91d762e581b32764e7dbe7e8c44", size = 1672866, upload-time = "2026-05-16T10:34:50.293Z" }, + { url = "https://files.pythonhosted.org/packages/34/0b/f9bf119248f3269076a44ae4c83a06ae609e6aa11c04db7306aed011e0d9/mpi4py-4.1.2-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:a8e9d25dd3b304ac2a57ab8ac4eca87f81411e535dc24ac0cd044f6f8304d9d7", size = 1440551, upload-time = "2026-05-16T10:34:51.777Z" }, + { url = "https://files.pythonhosted.org/packages/26/e4/32575e9b0d5380b08ffa04958c7a13b1bab86e3674f33b6de3827fbbe14c/mpi4py-4.1.2-cp314-cp314-manylinux1_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:d53e8f7182bf125b37155f679b301a21cd64e5ae6861ca0ab989c9d2b2073bbd", size = 1368388, upload-time = "2026-05-16T10:34:53.665Z" }, + { url = "https://files.pythonhosted.org/packages/87/a8/ad5e925da9de402704ecfe8715348b068fb0b65059109c65dafb111f95e4/mpi4py-4.1.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7cbf982d3425fd06890fbcff79ba19a7435e93bc74e5711ca8ac0b64d0c3ece9", size = 1238211, upload-time = "2026-05-16T10:34:55.094Z" }, + { url = "https://files.pythonhosted.org/packages/c3/5e/3fbcbe18712fecd509eedb91dcd4947dba0618a4d2add6231b772604590b/mpi4py-4.1.2-cp314-cp314-win_amd64.whl", hash = "sha256:50a1f5c5b210b8175860174fe670e058883df6f50e8998356dca5c73e6177d89", size = 1712936, upload-time = "2026-05-16T10:34:56.745Z" }, + { url = "https://files.pythonhosted.org/packages/14/ea/6f1836e46355f46eca2bd8d2b989fb29d62acff0a6a5dc61d356e0e36e44/mpi4py-4.1.2-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:5f255ed29f4fc66bd1387956e5fe207cae19e418c7db5559f5d6a77370171389", size = 1753847, upload-time = "2026-05-16T10:34:58.184Z" }, + { url = "https://files.pythonhosted.org/packages/cc/b3/999f743330dec8543a2b0f804c12e28a45afc6e5546aa16ef849a67d91f1/mpi4py-4.1.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:d2704dfd19663200677a7c98b971a385d2ed58bb9deee544dbd38fbec1a91ef1", size = 1527795, upload-time = "2026-05-16T10:34:59.965Z" }, + { url = "https://files.pythonhosted.org/packages/39/29/45a41d081342896af52d38a848017728bf6834c4c92814c50aa791dc750b/mpi4py-4.1.2-cp314-cp314t-manylinux1_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:db189238c37be98933eeb078a3e2b827eef969837e8fe0f3a2e6cdb7f4b1b05f", size = 1423092, upload-time = "2026-05-16T10:35:01.672Z" }, + { url = "https://files.pythonhosted.org/packages/0f/6f/c2127d426d87f2f7cdb4beacbb4febe7b0c93047a3c615d0e56cecb42563/mpi4py-4.1.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d0fbf0bc100ed966c3d7f44231a19aef0025d523d1b813de468a1b114d0d9436", size = 1303134, upload-time = "2026-05-16T10:35:03.122Z" }, + { url = "https://files.pythonhosted.org/packages/24/9c/f80cd1f9591be4c5c55a60676306b19847194cf168e0cc9db07f2e181152/mpi4py-4.1.2-cp314-cp314t-win_amd64.whl", hash = "sha256:34d94bd254947d17033aa5a9b6b1872175bad7e73ccef6b0373f2b8054e30f2e", size = 1838114, upload-time = "2026-05-16T10:35:05.306Z" }, +] + [[package]] name = "multipledispatch" version = "1.0.0"