Skip to content

feat: accumulator engine — Build/Apply for maximal consistent rule combinations#42

Merged
discreteds merged 38 commits into
developfrom
feature/accumulator-engine
Apr 29, 2026
Merged

feat: accumulator engine — Build/Apply for maximal consistent rule combinations#42
discreteds merged 38 commits into
developfrom
feature/accumulator-engine

Conversation

@discreteds

Copy link
Copy Markdown
Member

Summary

  • New engine: AccumulatorEngine with explicit build() → Lattice and apply(lattice, context) → AccumulatorResult two-phase API. Builds the lattice of maximal consistent rule combinations with accumulated numerics (the "accumulator builds; the filter ranks" pattern from the architecture analysis).
  • Shared metadata extension: DimensionRole enum (CONSTRAINT / CONTEXT_KEY) on Dimension — backward-compatible, filter engine ignores it.
  • AccumulatorCompiler: Compiles coalesce, compatible, and NA flag expressions for EXACT, RANGE, GREATER_THAN, LESS_THAN strategies. Sentinel-safe conditional coalesce prevents corruption from greatest/least on sentinel values.
  • Prime-product combination DNA: Each rule assigned a unique prime per partition. Combination identity is the product of member primes. Frontier filter uses prime divisibility for Pareto-dominant subset detection. Checked multiplication raises OverflowError at int64 boundary.
  • 128 new tests: unit tests for all components, cross-backend parametrized apply tests (7 backends), edge cases (empty rules, multiple aggregates, frontier filter isolation, range/GT/LT integration), and performance benchmarks (build scaling by rules/dims/density, apply scaling).
  • Backlog captured: Multi-lattice composition discussion doc for cross-authority lattice joins.

Architecture

The accumulator engine answers "what is the maximal consistent combination of rules, and what numeric does it accumulate?" — complementing the existing filter engine which answers "which rules apply to this context?"

Build phase: Python-controlled breadth-first iteration where each step's work (cross-join, compatibility filter, dimension coalesce, numeric accumulation) runs as a relation() pipeline via mountainash.expressions. Frontier filter removes dominated combinations via prime-product divisibility anti-join.

Apply phase: reuses ExpressionRulesEngine by remapping co_ prefixed lattice columns to filter-engine-compatible DimensionsMetadata.

New Files

File Purpose
accumulator_engine.py AccumulatorEngine — build, build_all, apply, apply_auto
accumulator_compiler.py Coalesce/compatible/NA flag expression compilation
accumulator_result.py AccumulatorResult(RuleResult) with accumulator accessors
aggregate.py Aggregate pydantic model
lattice.py Lattice data class
primes.py Prime sieve, allocation, checked multiply

Test plan

  • 618 tests passing (128 new), 0 failures
  • Cross-backend apply phase verified across all 7 backends (3 xfailed on ibis-polars #78)
  • Worked example from architecture analysis reproduced end-to-end (3 rules → 6 outermost rulesets)
  • Edge cases: empty rules, multiple aggregates, unsupported operations, frontier filter isolation, sentinel coalesce
  • Performance benchmarks: build scales with unknown density (30ms at 10%, 215ms at 70% for 10 rules/4 dims); apply is ~4ms constant
  • Full existing test suite green — no regressions

🤖 Generated with Claude Code

discreteds and others added 30 commits April 14, 2026 21:10
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ibis-polars fails the engine pipeline due to missing with_row_index
translation (mountainash-io/mountainash#78). Use BENCH_BACKENDS (6
backends) instead of ALL_BACKENDS (7) for benchmark parametrisation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds tests/benchmark_data.py with four public functions (assign_strategies,
generate_rules, generate_context, build_engine) that produce deterministic
seeded rule sets and contexts across all 11 match strategies.
Adds tests/test_benchmark_data.py with 26 tests covering shape, sentinel
density, determinism, and engine round-trip evaluation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds tests/test_benchmarks.py with TestScalingMatrix (3x3x6 rule/dim/backend
grid) and TestStrategyIsolation (all 11 MatchStrategies × 6 backends) using
pytest-benchmark pedantic mode; excludes ibis-polars per upstream bug #78.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ls backends

PREFIX, SUFFIX, CONTAINS fail on pandas/narwhals backends where the
mountainash string operations pass Expr objects to str accessor methods
that expect plain strings. Added skip logic alongside existing SET
strategy skips. Also adds .benchmarks/ to .gitignore.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace auto-generated boilerplate with problem-first README covering:
- Quick start with working code example
- 11 match strategies table
- 7 backend support matrix
- Development commands reference
- Architecture overview (ternary logic, single-pass evaluation)

Also updated GitHub repo description and topics.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lead with the core value proposition: rules as data, not code.
Immediate updates without deploys, auditable per-dimension results,
scales to thousands of rules without branching logic.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- pyrightconfig: use local .venv instead of hatch ephemeral path, bump to 3.12
- coverage: remove tests from source_pkgs (not a coverage target)
- gitignore: add sp_productpricingmatrix_discretion_combos.sql

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These standalone scripts targeted the now-deleted HybridRulesEngine,
VectorizedRulesEngine, and NumpyRuleProcessor. Replaced by the
expression-based engine and tests/test_benchmarks.py benchmark suite.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Old JSON and markdown benchmark outputs from August 2025, targeting
engines that no longer exist. Current benchmarks use pytest-benchmark
with JSON saved to .benchmarks/ (gitignored).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tests for HybridRulesEngine, VectorizedRulesEngine, NumpyRuleProcessor,
and real-data infrastructure — all superseded by the expression-based
engine and its cross-backend test suite.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- docs/future opportunities/ — phase 1-3 hype documents
- docs/opencode/ — OpenCode session docs for deleted VectorizedRulesEngine
- docs/recommendations/ — outdated code consistency review

Current architecture docs live in docs/superpowers/specs/ and
docs/superpowers/plans/.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Test commands are documented in README's Development section and
CLAUDE.md's Build/Test/Lint Commands section. TESTING.md referenced
deleted test files (test_rule_engine.py) and was out of date.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All phase 1-4 planning, retrospectives, and performance analysis docs
are superseded by the expression-based engine rewrite. Current specs
and plans live in docs/superpowers/. The additive engine analysis is
preserved in docs/superpowers/specs/2026-04-07-additive-rules-architecture-analysis.md.

Removed:
- docs/planning/ (8 files — phases 1-6 roadmaps)
- docs/retrospectives/ (7 files — phase 1-4 retros, architecture analysis)
- docs/ root (benchmarking plan, outstanding tasks, optimisation strategies,
  performance analysis)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The PMX_DB SQL reference is documented in the additive-rules
architecture analysis spec; the raw file doesn't need to live in
this repo.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds DimensionRole (CONSTRAINT | CONTEXT_KEY) to constants.py and a
backward-compatible role field (default: CONSTRAINT) to the Dimension
model, enabling the accumulator engine to distinguish lattice-partitioning
dimensions from coalesce-participating ones.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces the Aggregate pydantic model for declaring numeric columns to
accumulate, and the Lattice data class wrapping the accumulator build output
with combination access, row count, and partition key support.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements AccumulatorResult with best_combination, accumulated(),
provenance, and depths accessors; adds aggregate.py and lattice.py stubs
required by the accumulator engine.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements Tasks 6 and 7: sentinel-safe coalesce expressions for RANGE
(intersection of bounds), GREATER_THAN (greatest wins), and LESS_THAN
(least wins), plus always-compatible threshold logic. All 13 new tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Core lattice-building engine that constructs maximal consistent rule
combinations via breadth-first expansion with prime-product tracking,
dimension coalescing, and frontier dominance filtering.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements apply() and apply_auto() methods that reuse ExpressionRulesEngine
to match contexts against lattice fingerprints via remapped co_ column metadata.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
discreteds and others added 6 commits April 29, 2026 15:22
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds TestApplyCrossBackend with 4 test methods × 7 backends (28 total).
ibis-polars xfails (3) are registered under the existing #78 entry in
conftest._UPSTREAM_XFAILS; apply_backend added to the param-name lookup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers build-phase scaling by rule count, dimension count, and unknown
density, plus apply-phase and end-to-end build+apply benchmarks using
pytest-benchmark pedantic mode.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
21 tests covering empty rules, multiple aggregates, unsupported aggregate ops,
frontier filter dominance semantics, no-constraint-dim builds, RANGE/GREATER_THAN/
LESS_THAN coalesce integration, and build-step level isolation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Captures the cross-authority lattice join pattern — risk × discretion ×
promotional lattices composed independently. Documents why multisets
become real at join time (validating the prime-over-bitset decision),
incremental rebuild benefits, and open questions on commutativity,
associativity, and prime allocation across authorities.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Apr 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (develop@db960b8). Learn more about missing BASE report.
⚠️ Report is 50 commits behind head on develop.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             develop      #42   +/-   ##
==========================================
  Coverage           ?   93.90%           
==========================================
  Files              ?       13           
  Lines              ?      623           
  Branches           ?       89           
==========================================
  Hits               ?      585           
  Misses             ?       28           
  Partials           ?       10           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

discreteds and others added 2 commits April 29, 2026 17:34
Add `benchmark` pytest marker to both benchmark test files. CI
test_github scripts now use `-m 'not benchmark'` to skip them since
pytest-benchmark is not in the CI dependency set. Benchmarks still
run locally via `hatch run test:test-perf`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

❌ The last analysis has failed.

See analysis details on SonarQube Cloud

@discreteds discreteds merged commit 3f8b145 into develop Apr 29, 2026
5 of 6 checks passed
@discreteds discreteds deleted the feature/accumulator-engine branch April 29, 2026 07:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant