Repository navigation
Merge pull request #54 from aurascoper/fix/persist-target-and-pin-bas… #113
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: CI | |
| on: | |
| push: | |
| branches: [main, "feature/**"] | |
| pull_request: | |
| concurrency: | |
| group: ci-${{ github.ref }} | |
| cancel-in-progress: true | |
| jobs: | |
| build-and-test: | |
| # macos-26 → Xcode 26 → Swift 6.3.x, matching the developer toolchain. | |
| # macos-15's Swift 6.1.2 had over-eager region-based-isolation diagnostics | |
| # (false-positive "sending 'self' risks causing data races" on @MainActor | |
| # Sendable types, e.g. AppViewModel's detached pipeline tasks) that Swift 6.2 | |
| # relaxed. It fails to build correct code that Xcode 26 accepts. The Swift | |
| # version guard below fails fast if the runner ever regresses. | |
| runs-on: macos-26 | |
| steps: | |
| # fetch-depth: 0 because the ledger gate resolves the commit SHAs that | |
| # ledger nodes pin. The default depth of 1 leaves every historical SHA | |
| # unresolvable, which would make that check inert (it degrades to | |
| # warnings rather than lying, but it stops catching anything). Measured: | |
| # a depth-1 checkout of this repository cannot resolve node 1's two real | |
| # pins. Full history is cheap here — ~300 commits, single-digit MB. | |
| - uses: actions/checkout@v4 | |
| with: | |
| fetch-depth: 0 | |
| - name: Swift version | |
| run: | | |
| swift --version | |
| # This package builds under Swift 6.2+ (Xcode 26+). Fail fast and | |
| # legibly if the runner regresses to an older toolchain, whose | |
| # region-isolation false-positives would otherwise surface as a wall | |
| # of confusing "sending 'self'" build errors. | |
| swift --version 2>&1 | grep -qE "Swift version (6\.[2-9]|[7-9]\.)" \ | |
| || { echo "::error::Need Swift 6.2+ (Xcode 26+); this runner has an older toolchain"; exit 1; } | |
| # GoldenRecordingRegressionTests skips silently if its fixture is absent, | |
| # which would let the suite "pass" vacuously (README advertises this | |
| # regression). Fail CI if the committed fixture ever goes missing. | |
| - name: Assert golden fixture is present | |
| run: | | |
| test -f Tests/BCIEEGTests/Fixtures/reference_pipeline.json \ | |
| || { echo "::error::Golden fixture missing — GoldenRecordingRegressionTests would skip vacuously"; exit 1; } | |
| # CONTRIBUTING.md makes docs/architecture/decision-log/ the canonical | |
| # location for architectural decisions, so a reference that resolves to no | |
| # file — or to two — is a provenance defect, not a typo. ADR-004 was | |
| # double-assigned for a year: the roadmap recorded it, a code review | |
| # re-found it, and neither caused the fix. | |
| # | |
| # A direct python3 step rather than a job: the checker is stdlib-only and | |
| # macos-26 ships python3, so this needs no setup-python and no new runner. | |
| # Tests run first — a gate whose only evidence is that it never complained | |
| # is not a gate. | |
| - name: ADR reference integrity | |
| run: | | |
| python3 Tests/eval/test_adr_references.py | |
| python3 Scripts/check_adr_references.py --mode all | |
| # Same reasoning one lane over. EXPERIMENT_LEDGER nodes are cited by | |
| # number, and a bare number silently borrows this ledger's namespace — | |
| # "node 33" is a dialectic-session node, referenced 14 times and | |
| # qualified exactly once. It also resolves the commit SHAs nodes pin, so | |
| # an entry can no longer point at a commit that was never pushed. That is | |
| # the half of the citation problem a machine can check; whether a node | |
| # contains the numbers attributed to it stays a review question. | |
| # | |
| # Tests first, same rule as above. See | |
| # docs/architecture/experiment-ledger-citation-integrity.md. | |
| - name: Ledger citation integrity | |
| run: | | |
| python3 Tests/eval/test_ledger_references.py | |
| python3 Scripts/check_ledger_references.py --mode hard | |
| # Compiles every target (incl. BCILLM's MLX Metal kernels via the runner's | |
| # full Xcode) — the primary gate that the whole graph builds. | |
| - name: Build (all targets) | |
| run: swift build | |
| # Run the deterministic stub/mock/pure suites only: headless runners have | |
| # no GPU (so MLX .eval() would fault), no `claude` CLI (network egress), | |
| # no Core ML models, and no live EEG hardware. BCIEEGTests includes the | |
| # golden-recording + BGE-replay regressions against the committed fixtures. | |
| # The MLX-eval, egress-smoke, and hardware paths stay local-only. | |
| - name: Test (deterministic suites) | |
| run: | | |
| swift test \ | |
| --filter BCICoreTests \ | |
| --filter BCIClassifierTests \ | |
| --filter BCIEEGTests \ | |
| --filter BCICloudBridgeTests \ | |
| --filter BCIVoiceTests \ | |
| --filter AsyncTimeout \ | |
| --skip MindMonitorOSCStreamTests | |
| # MindMonitorOSCStreamTests binds a real UDP socket on a FIXED high port | |
| # (57182–57186, 0.0.0.0) and waits on a 1–3s stale watchdog — a genuine | |
| # collision/flake risk on a headless runner. The OSC *decode* logic is | |
| # covered socket-free by MuseOSCDecoderTests / MindMonitorDecoderTests, | |
| # which still run. Run the socket integration test locally. | |
| # Tests/eval had no CI at all before this job. Seven modules, and only | |
| # test_adr_references.py ran (via the direct python3 step above) — the other | |
| # six executed nowhere, which is how three failing assertions and an | |
| # import-time crash survived in a green repository. | |
| # | |
| # pytest, not unittest, and the choice is forced rather than stylistic: | |
| # six of the seven modules are bare `def test_*` with no `unittest.TestCase`, | |
| # so `python -m unittest` collects ZERO from them, and `unittest discover` | |
| # fails outright here (`Tests/eval` has no __init__.py). Either way the suite | |
| # would report success while running nothing. | |
| python-eval: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: actions/setup-python@v5 | |
| with: | |
| python-version: "3.11" | |
| - name: Install Python dependencies | |
| run: | | |
| # scipy is required, not optional: three modules import it | |
| # transitively and fail collection without it. pandas is a harness | |
| # dependency of the session/channel-quality frames. | |
| # | |
| # Majors are bounded because this gate's semantics depend on them: | |
| # the trapz/trapezoid fix is numpy-2 specific, and the Mann-Whitney | |
| # contract depends on SciPy's exact/asymptotic behaviour. An | |
| # unconstrained install would let a major bump change a p-value | |
| # without any commit here. | |
| pip install \ | |
| 'pytest>=8,<9' \ | |
| 'numpy>=2,<3' \ | |
| 'scipy>=1.14,<2' \ | |
| 'pandas>=2,<3' | |
| # A --filter or --ignore that matches nothing exits 0, and a collection | |
| # error can silently shrink the suite. Assert the floor before running so | |
| # a module that stops being collected fails loudly instead of vanishing. | |
| - name: Assert Tests/eval is discoverable | |
| run: | | |
| python -m pytest Tests/eval --collect-only -q \ | |
| --ignore=Tests/eval/test_joint_embedding.py > /tmp/collect.txt 2>&1 || true | |
| tail -1 /tmp/collect.txt | |
| collected=$(grep -oE '^[0-9]+ tests? collected' /tmp/collect.txt | grep -oE '^[0-9]+' | head -1) | |
| errors=$(grep -cE '^ERROR ' /tmp/collect.txt || true) | |
| echo "discovered: Tests/eval=${collected:-0} collection_errors=${errors:-0}" | |
| [ "${errors:-0}" -eq 0 ] \ | |
| || { echo "::error::${errors} module(s) failed to collect"; exit 1; } | |
| # Floor sits below the current 63 so ordinary additions never trip it. | |
| [ "${collected:-0}" -ge 55 ] \ | |
| || { echo "::error::Tests/eval collected ${collected:-0}, expected >= 55"; exit 1; } | |
| - name: Test (eval suites) | |
| run: | | |
| # test_joint_embedding.py imports mlx (Apple Silicon only) and needs a | |
| # converted model; it is the one module excluded, by name and reason. | |
| python -m pytest Tests/eval -v \ | |
| --ignore=Tests/eval/test_joint_embedding.py | |
| # NeuralComposeEEG is not on main yet — it arrives with lane B. Absent is | |
| # expected and must be said out loud; present-but-empty is a defect, since | |
| # a package whose tests silently collect nothing is the failure this whole | |
| # job exists to prevent. | |
| - name: Assert NeuralComposeEEG is absent or non-vacuous | |
| run: | | |
| if [ ! -d NeuralComposeEEG ]; then | |
| echo "NeuralComposeEEG/ absent on this branch — expected until lane B integrates it" | |
| exit 0 | |
| fi | |
| echo "NeuralComposeEEG/ present — its suite must not be vacuous" | |
| n=$(PYTHONPATH=NeuralComposeEEG/src python -m pytest NeuralComposeEEG/tests \ | |
| --collect-only -q 2>/dev/null \ | |
| | grep -oE '^[0-9]+ tests? collected' | grep -oE '^[0-9]+' | head -1) | |
| echo "discovered: NeuralComposeEEG=${n:-0}" | |
| [ "${n:-0}" -gt 0 ] \ | |
| || { echo "::error::NeuralComposeEEG/ is present but collected 0 tests"; exit 1; } | |
| PYTHONPATH=NeuralComposeEEG/src python -m pytest NeuralComposeEEG/tests -v |