feat: accumulator engine — Build/Apply for maximal consistent rule combinations#42
Merged
Conversation
release/26.4.0
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ibis-polars fails the engine pipeline due to missing with_row_index translation (mountainash-io/mountainash#78). Use BENCH_BACKENDS (6 backends) instead of ALL_BACKENDS (7) for benchmark parametrisation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds tests/benchmark_data.py with four public functions (assign_strategies, generate_rules, generate_context, build_engine) that produce deterministic seeded rule sets and contexts across all 11 match strategies. Adds tests/test_benchmark_data.py with 26 tests covering shape, sentinel density, determinism, and engine round-trip evaluation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds tests/test_benchmarks.py with TestScalingMatrix (3x3x6 rule/dim/backend grid) and TestStrategyIsolation (all 11 MatchStrategies × 6 backends) using pytest-benchmark pedantic mode; excludes ibis-polars per upstream bug #78. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ls backends PREFIX, SUFFIX, CONTAINS fail on pandas/narwhals backends where the mountainash string operations pass Expr objects to str accessor methods that expect plain strings. Added skip logic alongside existing SET strategy skips. Also adds .benchmarks/ to .gitignore. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace auto-generated boilerplate with problem-first README covering: - Quick start with working code example - 11 match strategies table - 7 backend support matrix - Development commands reference - Architecture overview (ternary logic, single-pass evaluation) Also updated GitHub repo description and topics. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lead with the core value proposition: rules as data, not code. Immediate updates without deploys, auditable per-dimension results, scales to thousands of rules without branching logic. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- pyrightconfig: use local .venv instead of hatch ephemeral path, bump to 3.12 - coverage: remove tests from source_pkgs (not a coverage target) - gitignore: add sp_productpricingmatrix_discretion_combos.sql Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These standalone scripts targeted the now-deleted HybridRulesEngine, VectorizedRulesEngine, and NumpyRuleProcessor. Replaced by the expression-based engine and tests/test_benchmarks.py benchmark suite. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Old JSON and markdown benchmark outputs from August 2025, targeting engines that no longer exist. Current benchmarks use pytest-benchmark with JSON saved to .benchmarks/ (gitignored). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tests for HybridRulesEngine, VectorizedRulesEngine, NumpyRuleProcessor, and real-data infrastructure — all superseded by the expression-based engine and its cross-backend test suite. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- docs/future opportunities/ — phase 1-3 hype documents - docs/opencode/ — OpenCode session docs for deleted VectorizedRulesEngine - docs/recommendations/ — outdated code consistency review Current architecture docs live in docs/superpowers/specs/ and docs/superpowers/plans/. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Test commands are documented in README's Development section and CLAUDE.md's Build/Test/Lint Commands section. TESTING.md referenced deleted test files (test_rule_engine.py) and was out of date. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All phase 1-4 planning, retrospectives, and performance analysis docs are superseded by the expression-based engine rewrite. Current specs and plans live in docs/superpowers/. The additive engine analysis is preserved in docs/superpowers/specs/2026-04-07-additive-rules-architecture-analysis.md. Removed: - docs/planning/ (8 files — phases 1-6 roadmaps) - docs/retrospectives/ (7 files — phase 1-4 retros, architecture analysis) - docs/ root (benchmarking plan, outstanding tasks, optimisation strategies, performance analysis) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The PMX_DB SQL reference is documented in the additive-rules architecture analysis spec; the raw file doesn't need to live in this repo. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Release 26.4.0
Adds DimensionRole (CONSTRAINT | CONTEXT_KEY) to constants.py and a backward-compatible role field (default: CONSTRAINT) to the Dimension model, enabling the accumulator engine to distinguish lattice-partitioning dimensions from coalesce-participating ones. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces the Aggregate pydantic model for declaring numeric columns to accumulate, and the Lattice data class wrapping the accumulator build output with combination access, row count, and partition key support. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements AccumulatorResult with best_combination, accumulated(), provenance, and depths accessors; adds aggregate.py and lattice.py stubs required by the accumulator engine. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements Tasks 6 and 7: sentinel-safe coalesce expressions for RANGE (intersection of bounds), GREATER_THAN (greatest wins), and LESS_THAN (least wins), plus always-compatible threshold logic. All 13 new tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Core lattice-building engine that constructs maximal consistent rule combinations via breadth-first expansion with prime-product tracking, dimension coalescing, and frontier dominance filtering. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements apply() and apply_auto() methods that reuse ExpressionRulesEngine to match contexts against lattice fingerprints via remapped co_ column metadata. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds TestApplyCrossBackend with 4 test methods × 7 backends (28 total). ibis-polars xfails (3) are registered under the existing #78 entry in conftest._UPSTREAM_XFAILS; apply_backend added to the param-name lookup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers build-phase scaling by rule count, dimension count, and unknown density, plus apply-phase and end-to-end build+apply benchmarks using pytest-benchmark pedantic mode. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
21 tests covering empty rules, multiple aggregates, unsupported aggregate ops, frontier filter dominance semantics, no-constraint-dim builds, RANGE/GREATER_THAN/ LESS_THAN coalesce integration, and build-step level isolation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Captures the cross-authority lattice join pattern — risk × discretion × promotional lattices composed independently. Documents why multisets become real at join time (validating the prime-over-bitset decision), incremental rebuild benefits, and open questions on commutativity, associativity, and prime allocation across authorities. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #42 +/- ##
==========================================
Coverage ? 93.90%
==========================================
Files ? 13
Lines ? 623
Branches ? 89
==========================================
Hits ? 585
Misses ? 28
Partials ? 10 ☔ View full report in Codecov by Sentry. |
Add `benchmark` pytest marker to both benchmark test files. CI test_github scripts now use `-m 'not benchmark'` to skip them since pytest-benchmark is not in the CI dependency set. Benchmarks still run locally via `hatch run test:test-perf`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
❌ The last analysis has failed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AccumulatorEnginewith explicitbuild() → Latticeandapply(lattice, context) → AccumulatorResulttwo-phase API. Builds the lattice of maximal consistent rule combinations with accumulated numerics (the "accumulator builds; the filter ranks" pattern from the architecture analysis).DimensionRoleenum (CONSTRAINT/CONTEXT_KEY) onDimension— backward-compatible, filter engine ignores it.coalesce,compatible, and NA flag expressions for EXACT, RANGE, GREATER_THAN, LESS_THAN strategies. Sentinel-safe conditional coalesce prevents corruption fromgreatest/leaston sentinel values.OverflowErrorat int64 boundary.Architecture
The accumulator engine answers "what is the maximal consistent combination of rules, and what numeric does it accumulate?" — complementing the existing filter engine which answers "which rules apply to this context?"
Build phase: Python-controlled breadth-first iteration where each step's work (cross-join, compatibility filter, dimension coalesce, numeric accumulation) runs as a
relation()pipeline viamountainash.expressions. Frontier filter removes dominated combinations via prime-product divisibility anti-join.Apply phase: reuses
ExpressionRulesEngineby remappingco_prefixed lattice columns to filter-engine-compatibleDimensionsMetadata.New Files
accumulator_engine.pyAccumulatorEngine— build, build_all, apply, apply_autoaccumulator_compiler.pyaccumulator_result.pyAccumulatorResult(RuleResult)with accumulator accessorsaggregate.pyAggregatepydantic modellattice.pyLatticedata classprimes.pyTest plan
🤖 Generated with Claude Code