WIREFRAME.md

SolverForge Bench Wireframe

This is the as-built map for the current benchmark repository. It is a documentation contract: when source layout, benchmark ownership, run flow, persistence, or CI changes, update this file with README.md and AGENTS.md.

Repository Surfaces

• pyproject.toml declares the Python 3.14 package, shared dependencies, and package discovery across src/, list-variable/cvrp/src/, and scalar-variable/employee-scheduling/src/, and scalar-variable/job-shop-scheduling/src/.
• Makefile is the root build and execution surface. It owns virtualenv setup, native adapter builds, benchmark smoke/full runs, database helpers, normalization, the SolverForge banner, and CPU-pinned harness execution.
• src/solverforge_bench/ is the shared framework. It owns CLI parsing, TOML loading, benchmark registry, exact run matrix construction and hashing, runtime artifact provenance, timed execution, watchdog containment, row construction, CSV writing, logging, solver output capture, solver-version collection, secret-safe command metadata, ETL, and optional PostgreSQL writes.
• scripts/run_benchmark.py, scripts/verify_solverforge_py_guardrails.py, and benchmark-local validation scripts bootstrap through scripts/_venv_bootstrap.py when they require the repository virtualenv. scripts/test_verify_solverforge_py_guardrails.py and scripts/test_benchmark_contracts.py and scripts/test_solver_output_contracts.py run through their root Make targets.
• list-variable/cvrp/ is the list-variable benchmark package for CVRP.
• scalar-variable/employee-scheduling/ is the scalar-variable benchmark package for INRC-II nurse scheduling.
• scalar-variable/job-shop-scheduling/ is the scalar-variable benchmark package for classic JSPLIB job-shop scheduling.
• migrations/ holds SQLx-compatible PostgreSQL warehouse migrations.
• .github/workflows/ci.yml holds the GitHub-hosted CI workflow.
• .forgejo/workflows/ci.yml holds the local Forgejo CI workflow.
• archive/ holds historical reports and older standalone scripts only.

Shared Harness Flow

1. scripts/run_benchmark.py bootstraps into the root .venv, adds the four source roots to sys.path, and delegates to solverforge_bench.cli.main. scripts/verify_model_parity.py uses the same virtualenv bootstrap before importing the employee-scheduling package.
2. cli.py loads optional TOML configuration, selects the benchmark spec, applies CLI overrides, finalizes run catalog fields, validates solver names, and passes the selected spec to runner.py.
3. registry.py exposes the canonical benchmark specs: cvrp, employee-scheduling, and job-shop-scheduling.
4. runner.py materializes a nonempty, duplicate-free case list, positive unique time limits, unique solvers, and the complete expected Cartesian matrix. It resolves content-hashed runtime provenance before opening output, then iterates case -> time_limit -> solver and refuses completion unless every expected key was emitted exactly once.
5. execution.py runs each solver in a child process. Only the instance payload and nominal time limit are passed to the solver callable; the nominal time limit is not the hard kill deadline. The watchdog only terminates runaway processes after max(time_limit * multiplier, time_limit + grace_seconds).
6. Each benchmark spec validates and evaluates returned solutions externally, then the shared runner writes an incremental CSV row and, when enabled, a PostgreSQL row. Reference solutions may be used here for scoring, not as solver starts.
7. Solver exceptions, including NoSolutionFoundError, become result rows with run_error. CSV or PostgreSQL write failures remain fatal output-integrity failures. A SolverForge mandatory-construction stop at the requested time limit is a no_solution row with no objective and preserved native failure details; unrecognized SolverForge exceptions remain adapter_error.
8. scripts/verify_solverforge_py_guardrails.py is a release/local wrapper over the same root harness. It runs fixed solverforge-py smoke slices and paired native/Python comparison slices. Its one-second employee feasibility probe covers n030w4, n050w8, and n080w8 and requires hard-feasible returned schedules. Before execution it resolves every requested dataset selector; afterward it requires the exact instance/time-limit/solver matrix, then writes build/solverforge-py-guardrails/summary.json. Database URL values are redacted from recorded commands and persisted run metadata.

Benchmark Specs

Spec	Category	Default solvers	Default time limits	Native columns
cvrp	list_variable	pyvrp, ortools, vroom, timefold, rustvrp, pyhygese, solverforge, solverforge-py	1, 10, 60	none
employee-scheduling	scalar_variable	solverforge, solverforge-py, timefold, ortools	1, 10, 60	nurses, weeks, validator_model_delta, score_drift
job-shop-scheduling	scalar_variable	solverforge, solverforge-py, timefold, ortools	1, 10, 60	num_jobs, num_machines, num_operations, source_family, known_best_makespan, lower_bound_makespan, upper_bound_makespan, makespan_gap_to_best

CVRP Adapter Shape

• Data lives under list-variable/cvrp/data/X/ as CVRPLIB-X .vrp and .sol pairs.
• spec.py exposes dataset CVRPLIB-X, dataset set canonical, and --num-instances for smoke selection.
• domain/models.py defines Pydantic instance and solution contracts.
• domain/utils.py validates route feasibility and cost.
• solver/solver.py registers pyvrp, ortools, vroom, timefold, rustvrp, pyhygese, solverforge, and solverforge-py.
• Native solver builds are rooted in solver/ortools/, solver/rustvrp/, solver/vroom/, solver/timefold/, and solver/solverforge/.
• The SolverForge CVRP manifest and committed registry lockfile target the published 0.19.4 crates.
• The CVRP model uses public SolverForge CVRP list-variable hook bundles: VrpSolution, matrix distance meters, stock route hooks, stock savings depot/distance/metric-class hooks, and strict route feasibility for construction pruning. The benchmark budget is applied through the model config provider.
• The SolverForge and Timefold CVRP list variables start from empty route lists; adapter-owned incumbents, route hints, and reference-solution reads are not part of solver input.
• The solverforge-py CVRP adapter builds a public Python-binding list-variable model from the same CVRPLIB instance. Its canonical 0.6.6 declaration uses independent ListRouteHooks and ListSavingsHooks, explicit row-scoped capacity/demand/distance metadata, and explicit cross/intra-position distance sources. It starts all route lists empty and reports the installed solverforge Python distribution version.
• solverforge/solver.toml uses reproducible mode, seed 42, a 60 second internal termination cap, list construction phases, and a local-search union of nearby list moves, reverse moves, k-opt, ruin, and limited-neighborhood sublist change moves.
• solverforge_py.toml is the separate Python adapter copy of that complete policy. The adapter loads it directly and overlays only the requested termination seconds.
• Native OR-Tools no-solution exits are normalized to NoSolutionFoundError rather than benchmark-aborting runtime errors.
• Every CVRP wrapper emits a fair-start witness before solving. The native OR-Tools and SolverForge adapters also include native witness checks in their JSON output.

Employee Scheduling Adapter Shape

• Data lives under scalar-variable/employee-scheduling/data/inrc2/ as bundled INRC-II scenario, history, week-data, and reference solution TXT files.
• manifest.json defines dataset groups: quick has 1 group and 3 cases, test_with_solutions has 3 groups and 9 cases, canonical has 14 groups and 42 cases, and late has 6 groups and 18 cases.
• loader.py parses INRC-II TXT files and enumerates concrete cases.
• validation.py is the shared Python referee. It checks hard constraints first and then computes the soft-cost breakdown.
• scripts/verify_model_parity.py verifies that the Python validator, OR-Tools model, Timefold model, and SolverForge model encode the same model contract.
• spec.py exposes --dataset-set and --datasets, writes solution JSON artifacts for hard-feasible runs, and reports validator/model deltas through native columns.
• solver/solver.py registers solverforge, solverforge-py, timefold, and ortools.
• The SolverForge NRP manifest and committed registry lockfile target the published 0.19.4 crates with serde enabled.
• The SolverForge NRP model uses public scalar APIs: per-shift candidate values, unassigned scalar variables for optional slots, nearby value/entity candidates, and one ScalarGroup::assignment for required minimum slots, one nurse per day capacity, adjacent forbidden-succession assignment rules, ordered shift positions, and nurse sequence keys.
• SolverForge initializes each shift with nurse_idx = None, Timefold leaves each nurse planning variable unset, and OR-Tools performs one CP-SAT solve without adapter hints, hard seeds, warm starts, or fallback schedules. OR-Tools uses a deterministic coverage-order, round-robin assignment strategy within that single solve and the shared time limit.
• The solverforge-py employee adapter builds a public Python-binding scalar model with unassigned nurse_idx variables. Immutable required, capacity, position, sequence, and same-nurse forbidden-succession conflict metadata stays in native row fields. The static conflict graph gives the assignment engine the same adjacency rule as the native Rust adapter without a Python callback on each candidate edge. Per-shift nurse candidates remain the native legality boundary for construction, swaps, and rematches, so an assignment cannot migrate onto a shift whose skill or initial-history domain excludes that nurse. Hard feasibility, indexed presence penalties, and shift-off request penalties remain in the constraint model; the shared validator remains the source of result feasibility and cost. This adapter is a first-class default performance row.
• The Python wrappers emit a witness before solving. SolverForge Rust counts preassigned scalar variables, Timefold Java counts preassigned nurse planning variables, and OR-Tools C++ inspects CP-SAT solution-hint fields in the native model proto.
• solverforge_nrp/solver.toml sets environment_mode = "non_reproducible" and random_seed = 1. It intentionally does not impose an independent termination cap; the shared harness passes the requested benchmark budget.
• solverforge_py.toml is the separate Python adapter copy. Both files omit explicit phases so both bindings select the same model-aware default construction and local-search profile.

Job-Shop Scheduling Adapter Shape

• Data lives under scalar-variable/job-shop-scheduling/data/jsplib/ as bundled classic JSPLIB instance files sourced from tamy0612/JSPLIB.
• manifest.json defines dataset groups: quick has ft06 and la01, and canonical has all 162 bundled JSPLIB instances across abz, ft, la, orb, swv, ta, and yn.
• loader.py parses standard JSPLIB text files with optional comments.
• validation.py is the shared Python referee. It checks operation coverage, job precedence, machine non-overlap, and returned makespan.
• spec.py exposes --dataset-set and --datasets, reports JSPLIB family, size, known optimum, lower/upper bounds, and makespan gap through native columns.
• solver/solver.py registers solverforge, solverforge-py, timefold, and ortools.
• The SolverForge JSSP manifest and committed registry lockfile target the published SolverForge facade, SolverForge Core, and SolverForge Scoring 0.19.4 crates.
• Its list model declares each operation's fixed machine owner with element_owner_fn; SolverForge construction and list neighborhoods must not move an operation to a non-required machine.
• The SolverForge JSSP score path uses the stock upstream ListPrecedenceMakespanConstraint: job precedence is fixed precedence, each machine sequence contributes list precedence, missing/duplicate/wrong-owner assignments are hard penalties, and makespan is the soft objective. The adapter maps JSPLIB data into that generic constraint; it does not own a benchmark-local full-score search path.
• solverforge_jssp/solver.toml remains a stock SolverForge selector configuration. It may choose upstream list neighborhoods, but it does not add benchmark-local solver/search helpers, config probes, warm starts, or reference-solution hints.
• SolverForge and Timefold JSSP machine operation lists start empty. Known best bounds and validation data stay in specs and validators, not in solver-start incumbents.
• The solverforge-py JSSP adapter builds a public Python-binding list model with one empty machine sequence per machine and owner-constrained operation elements. The public first-class list precedence/makespan constraint scores operation ownership, assignment uniqueness, job precedence, machine order, and makespan; the shared validator remains the source of returned schedule feasibility and cost. This adapter is a first-class default performance row.
• The JSSP wrappers emit witnesses before solving. SolverForge Rust and Timefold Java count prefilled machine lists, and OR-Tools C++ records CP-SAT solution-hint counts from the model proto.
• solverforge_jssp/solver.toml and the separate solverforge_py.toml both set random_seed = 1 and omit explicit phases so native and Python use the same model-aware default construction and local-search profile.
• SolverForge Rust, SolverForge Python, and Timefold reject incomplete, duplicated, wrong-owner, or cyclic JSSP structure. They never replace an unknown operation start with zero.
• Both SolverForge CVRP adapters require every customer exactly once. Both employee-scheduling adapters require every callback-required shift while preserving optional unassigned shifts. Native adapters accept a solution only from Completed, never from a prior best after cancellation.

Makefile Contract

• make install-python-deps creates or refreshes the root .venv and is configured to install the exact published solverforge==0.6.6 wheel.
• make build-cvrp builds Python dependencies plus CVRP Timefold, SolverForge, OR-Tools, rustvrp, and VROOM integrations.
• make build-employee-scheduling builds Python dependencies plus employee Timefold, SolverForge, and OR-Tools integrations.
• make build-job-shop-scheduling builds Python dependencies plus job-shop Timefold, SolverForge, and OR-Tools integrations.
• make verify-fair-start enforces that active solver adapters start from unassigned scalar variables or empty list variables, emit runtime witnesses, and do not read reference solutions or inject adapter-owned incumbents.
• make verify-benchmark-contracts exercises exact matrix completion, runtime provenance, and all SolverForge output-completeness boundaries without running a full benchmark suite.
• make verify-fair-start-rows RUN_ID=<uuid> checks persisted PostgreSQL rows for valid fair-start witnesses after a DB smoke run.
• make verify-solverforge-config-parity parses each separate native/Python TOML pair, requires semantic equality, and pins the result to the qualified strongest-policy hash so jointly weakening both copies also fails.
• make verify-stock-solverforge-guardrails builds the active native adapters, runs stock SolverForge guardrail benchmarks, and parses the resulting CSVs. JSSP quick plus the fixed canonical subset must produce hard-feasible SolverForge rows with valid fair-start witnesses; CVRP and employee SolverForge smoke rows must remain hard-feasible with valid fair-start witnesses. Pass --require-jssp-win through GUARDRAIL_ARGS when the run should enforce that SolverForge ties or beats the best feasible JSSP solver row.
• make verify-solverforge-py-smoke builds the native SolverForge adapters, verifies fair-start source checks, runs solverforge-py smoke rows through the shared harness, and parses the generated CSVs. Employee rows check fair-start, runtime, version, and score-reporting integrity while accepting an honest infeasible or no-incumbent result. Its production-scale one-second feasibility probe covers n030w4, n050w8, and n080w8 and requires every row to return a hard-feasible schedule without a validation error.
• make verify-solverforge-py-guardrail-contract runs the focused exact-matrix and command-redaction regression suite without invoking solvers.
• make verify-solverforge-py-comparison runs paired native solverforge and solverforge-py rows through the shared harness and records parsable quality and wall-time summaries without PostgreSQL by default. Employee infeasible and no-incumbent rows remain honest outcomes, not gate failures.
• make verify-solverforge-py-release combines compileall, benchmark validators, fair-start checks, native adapter builds, smoke, and paired comparison into one release-mode guardrail invocation and summary.
• make bench-cvrp-quick runs three CVRP instances at 1 and 10 seconds with all registered CVRP solvers.
• make bench-cvrp-quick-db runs the same CVRP smoke path after applying migrations and persists it to PostgreSQL.
• make bench-cvrp-solverforge-quick runs the same quick CVRP slice with only SolverForge.
• make bench-cvrp-solverforge-quick-db persists that same SolverForge-only CVRP smoke path after applying migrations.
• make bench-employee-scheduling-quick runs n005w4 at 1 and 10 seconds with all registered employee-scheduling solvers.
• make bench-employee-scheduling-quick-db runs the same employee smoke path after applying migrations and persists it to PostgreSQL.
• make bench-employee-scheduling-solverforge-quick runs the same quick employee slice with only SolverForge.
• make bench-employee-scheduling-solverforge-quick-db persists that same SolverForge-only employee smoke path after applying migrations.
• make bench-job-shop-scheduling-quick runs ft06 and la01 at 1 and 10 seconds with all registered job-shop solvers.
• make bench-job-shop-scheduling-quick-db runs the same job-shop smoke path after applying migrations and persists it to PostgreSQL.
• make bench-job-shop-scheduling-solverforge-quick runs the same quick job-shop slice with only SolverForge.
• make bench-job-shop-scheduling-solverforge-quick-db persists that same SolverForge-only job-shop smoke path after applying migrations.
• make bench-cvrp, make bench-employee-scheduling, and make bench-job-shop-scheduling run canonical benchmark paths at 1, 10, and 60 seconds.
• make bench-nightly-db builds all benchmark stacks, applies migrations once, and invokes the root harness for CVRP, employee scheduling, and job-shop scheduling in parallel on their per-suite pinned cores.
• make db-check, make db-create, make db-migrate, and make db-reset operate on DATABASE_URL, then BENCH_DATABASE_URL, then postgresql://postgres@localhost/solverforge_bench.
• make normalize-results converts generated global CSV artifacts to normalized CSV or NDJSON through scripts/normalize_results.py.
• Benchmark run targets use per-suite pinned cores: CVRP_BENCH_CPU ?= 0, EMPLOYEE_BENCH_CPU ?= 1, and JOBSHOP_BENCH_CPU ?= 2. They set OMP_NUM_THREADS=1, MKL_NUM_THREADS=1, and per-core BENCH_LOCK values so different suites can run in parallel while accidental same-core runs serialize. BENCH_CPU=<n> intentionally forces all suites onto one core.
• Override child harness arguments with BENCH_ARGS, nightly child arguments with NIGHTLY_ARGS, config path with BENCH_CONFIG, and SQLx reset flags with DB_RESET_FLAGS.

Configuration Contract

• Root TOML keys are benchmark, solver, time_limits, wall_time_tolerance, watchdog_multiplier, watchdog_grace_seconds, output, run_kind, nightly, and release_tag.
• [postgres] accepts save and url.
• [logging] accepts level, dir, file, show_solver_output, and capture_solver_output.
• [benchmarks.cvrp] accepts num_instances.
• [benchmarks.employee-scheduling] accepts dataset_set and datasets.
• [benchmarks.job-shop-scheduling] accepts dataset_set and datasets.
• run_kind is one of quick, candidate, or tag; tag requires release_tag.
• A CLI --postgres-url enables PostgreSQL persistence unless --no-save-postgres is also supplied. A TOML PostgreSQL URL alone does not.

Output And Persistence

• CSV output uses one global snake_case schema with stable optional native columns.
• BenchmarkRow.as_dict() emits core fields and merges native fields.
• CVRP output defaults to list-variable/cvrp/data/benchmark_cvrp_<stamp>.csv.
• Employee scheduling output defaults to scalar-variable/employee-scheduling/data/benchmark_employee_scheduling_<stamp>.csv.
• Job-shop scheduling output defaults to scalar-variable/job-shop-scheduling/data/benchmark_job_shop_scheduling_<stamp>.csv.
• SolverForge-Py guardrail CSVs, logs, and summary.json are written under build/solverforge-py-guardrails/ by the wrapper. Benchmark-local solution artifact writing remains shared harness behavior and uses the spec artifact directories.
• Employee scheduling solution artifacts are written under scalar-variable/employee-scheduling/data/artifacts/employee_scheduling_<stamp>/.
• Job-shop scheduling solution artifacts are written under scalar-variable/job-shop-scheduling/data/artifacts/job_shop_scheduling_<stamp>/.
• PostgreSQL stores run catalog rows in benchmark_runs, expected keys in benchmark_run_matrix_entries, solver-version rows in benchmark_solver_versions, and result rows in benchmark_results.
• benchmark_result_facts, latest_benchmark_runs, and latest_benchmark_result_facts are diagnostic warehouse views.
• benchmark_run_publication_audit explains every rejected run. publishable_benchmark_runs, publishable_benchmark_result_facts, and the latest_publishable_* views are the only public publication surface.
• Completed result rows are written immediately. Interrupted runs keep partial rows but are excluded from latest-run views unless their run status is completed.

CI Contract

• Python CI runs on ubuntu-latest in .github/workflows/ci.yml and on python runner labels in .forgejo/workflows/ci.yml.
• Rust CI runs on ubuntu-latest in .github/workflows/ci.yml and on rust runner labels in .forgejo/workflows/ci.yml.
• Python CI uses Python 3.14, creates the root .venv through make install-python-deps HOST_PYTHON=..., compiles Python source, parses benchmark TOML examples, verifies separate SolverForge config parity, the shared matrix/provenance contract, and the Python guardrail regression contract, validates bundled CVRP instances, and validates employee model parity. GitHub uses actions/setup-python@v6; Forgejo uses a shell Python 3.14 bootstrap because the local Forgejo action mirror does not provide that interpreter version.
• Rust CI preserves strict --locked resolution. Adapter manifests and committed locks target SolverForge 0.19.4. CI checks formatting, runs cargo clippy --locked --all-targets -- -D warnings, and runs cargo build --locked for the CVRP SolverForge adapter, CVRP rustvrp adapter, employee SolverForge adapter, and job-shop SolverForge adapter.

Generated Artifacts

• The ignored build and output surfaces are .venv/, __pycache__/, Python package/build outputs, Rust/Java target/ outputs, logs/*, generated benchmark CSVs, build/solverforge-py-guardrails/, and generated solution artifact directories.
• Generated CSVs are evidence artifacts. Commit them only when the run output is intentionally part of the change.