Repository navigation
Conversation
…set order _expected_prior_keys returned sets, and both cache paths built the sample dicts by iterating them, so the parameter order followed string hashing and changed between processes with PYTHONHASHSEED. Results written by separate processes (harv-hq shards, pool workers, MPI ranks) disagreed on column order. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- execution.py: run_serial and run_pool drivers over a picklable per-source setup and a sink; the pool spawns single-threaded workers with at most 2*W sources in flight. mpi_comm imports mpi4py lazily. - Run.run_rejection takes workers and mpi; under MPI the slice is rank/size, only rank 0 supersedes on --overwrite, and ranks meet at barriers after the start and at the end. Progress logged every ~5%. - hq run --workers / --mpi; the harv-hq[mpi] extra. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Rank-local failures can leave other MPI ranks blocked indefinitely at a barrier.
1 open finding
What changed in this PR
Adds parallel rejection sampling through local process pools, explicit shards, and MPI.
Changes:
- Adds serial/pool/MPI execution drivers and progress reporting.
- Adds MPI packaging, CLI support, and tests.
- Stabilizes sampler parameter ordering across processes.
| File | Description |
|---|---|
uv.lock |
Locks optional MPI dependency. |
tests/unit/samplers/test_prior_cache.py |
Tests deterministic parameter ordering. |
src/harv/samplers/rejection.py |
Preserves declaration order for sample keys. |
packages/hq/tests/unit/test_hq_rejection.py |
Tests pool, shard, MPI, and progress behavior. |
packages/hq/src/harv_hq/run.py |
Selects and coordinates execution modes. |
packages/hq/src/harv_hq/rejection.py |
Adds per-process sampler setup. |
packages/hq/src/harv_hq/execution.py |
Implements execution drivers. |
packages/hq/src/harv_hq/cli.py |
Enables pool and MPI options. |
packages/hq/pyproject.toml |
Adds the MPI optional extra. |
packages/hq/docs/spec.md |
Documents parallel execution. |
packages/hq/docs/implementation-plan.md |
Records phase-four implementation status. |
docs/spec.md |
Documents deterministic parameter ordering. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
A rank that raised outside per-source processing (supersede, provenance, reading, writing) left its peers blocked at a barrier. It now logs the error and calls COMM_WORLD.Abort(1). From the Copilot review of PR #63. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
patch.dict for the workers' thread environment, one progress log call, one try for the MPI abort, and no unreachable worker guard. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each rank runs a local pool over its slice, so a cluster job can use one rank per node. --mpi still excludes --shard. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Support parallelization