Partitioning of analysis specifications into smaller units

[anthonysena]

Idea

We'd like to be able to partition an analysis specification into smaller units of work to distribute the workload across a number of worker nodes for execution.

Current state

Today we can partition a large analysis specification by HADES module only. We can do this by using the modulesToExecute parameter of the execution settings (example). As an example, if we have an analysis specification that utilizes all of the available HADES modules, we can scale out an analysis across machines in the following way: we can set modulesToExecute = c("CohortGeneratorModule") to generate the cohorts required for the study. Once the cohorts are generated, if we had a machine available for each module in the analysis specification, we could execute each module in parallel to execute the full study. These machine could be connected to a shared drive to hold the results or later those results are aggregated to a single location for results upload and review.

Proposed solution

In addition to partitioning by module, we'd like to be able to extend this to partition tasks within a given module so that we can continue to scale out analyses. Given an analysis specification, each HADES module should be capable of partitioning the work to distribute and determining if the work completed successfully. This is related to #246 as well.

[anthonysena]

Design Summary: Partitioning Analysis Specifications in Strategus

Motivation

Large analysis specifications in Strategus may contain many independent analyses (e.g., multiple CohortMethod analyses), making execution difficult to distribute, resume, or reason about at finer granularity. While Strategus currently supports selecting modules to execute, it does not provide a way to decompose a single analysis specification into smaller, independently executable units.

This proposal introduces a deterministic, module-defined partitioning step that operates purely at the specification level and enables finer-grained execution without introducing workflow-engine behavior.

---

Core Idea: Partitioning as a Specification Transformation

Partitioning is a one-time transformation of an input analysis specification into a set of smaller, self-contained analysis specifications (“partitions”), each representing exactly one unit of analytical work.

Partitioning:

• does not execute analyses,
• does not schedule work, and
• does not manage execution state.

It only transforms spec → specs.

---

Definition of a Partition

A partitioned specification:

• contains exactly one analysis,
• includes all metadata and cohort definitions required for independent execution,
• is derived deterministically from the original specification, and
• can be executed by Strategus with no additional context.

Multi-analysis partitions and nested partitioning are explicitly unsupported.

---

Module-Defined Partitioning

Partitioning behavior is defined by analytical modules via a required interface on StrategusModule.

Each module:

• receives the full analysis specification as input,
• determines how its work can be decomposed into atomic analyses, and
• produces one partitioned specification per analysis.

Modules that cannot meaningfully partition their work must explicitly return a single partition representing the entire workload.

---

Files Produced by Partitioning

Partitioning produces a dedicated directory adjacent to the original analysis specification:

analysisSpecification.json
analysisSpecification.partitions/
├── manifest.json
├── part-001.json
├── part-002.json
└── ...

Partitioned specification files (part-*.json)

Each part-*.json file:

• is a valid analysis specification,
• contains exactly one analysis,
• duplicates all required cohort definitions and metadata, and
• can be executed independently by Strategus.

File naming is deterministic and stable to support inspection and reproducibility.

---

Partition manifest (manifest.json)

The manifest serves as the system of record for the partitioning operation. It captures how and when partitioning occurred and how partitions relate to the original specification.

The manifest includes:

• a schema version,
• a hash of the source analysis specification,
• the partitioning strategy used,
• the list of generated partitions and their identifiers.

Example (illustrative):

{
  "schemaVersion": "1.0",
  "partitionedAt": "2026-02-09T18:42:00Z",
  "partitionStrategy": "moduleDefined",
  "sourceSpecification": {
    "path": "../analysisSpecification.json",
    "hash": "sha256:..."
  },
  "units": [
    {
      "unitId": "cohortMethod_101",
      "module": "CohortMethod",
      "analysisId": 101,
      "file": "part-001.json"
    }
  ]
}

The original analysis specification is never modified.

---

Detecting Partition State

An analysis specification is considered partitioned if and only if:

• a partition directory exists,
• a valid manifest.json is present, and
• the source specification hash matches the current specification.

If the hash does not match, the partition is considered stale.

Partitioning fails if a partition directory already exists; automatic overwrite or forced re-partitioning is not supported.

---

Scope and Non-Goals

This proposal intentionally does not address:

• execution scheduling or orchestration,
• task execution status,
• result merging,
• shared or common specification artifacts.

Partitioning exists solely to make analysis specifications decomposable and inspectable.

---

Relationship to Execution

Partitioning enables—but does not require—distributed or resumable execution. Each partitioned specification naturally maps to a single executable task, allowing execution frameworks (including direct R execution or external tools) to operate at finer granularity while relying on Strategus for consistent specification semantics.