Refresh Card Data Catalogs #5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Refresh Card Data Catalogs | |
| # The engine embeds three git-tracked data files at compile time — | |
| # crates/engine/data/known-tokens.toml (named-token catalog; build.rs -> include_bytes!) | |
| # crates/engine/data/oracle-subtypes.json (creature-subtype vocabulary; include_str!) | |
| # crates/engine/data/mtgjson-vintage (vintage write-gate date stamp) | |
| # — that are generated from MTGJSON but committed by hand. When MTGJSON | |
| # publishes new cards/tokens/subtypes (weekly, ~Monday) these drift until | |
| # someone regenerates and commits them, which periodically turned CI red and | |
| # required a manual catalog PR. This workflow does that job automatically. | |
| # | |
| # It force-refreshes MTGJSON, runs the single-source-of-truth generator | |
| # (scripts/gen-card-data.sh — no YAML reimplementation of the pipeline), and | |
| # opens an auto-merge PR only when the tracked catalogs actually change. | |
| # | |
| # The vintage write-gate inside gen-card-data.sh makes every trigger | |
| # self-correcting: if the freshly-downloaded MTGJSON is not newer than the | |
| # committed vintage stamp, the catalogs are NOT rewritten, `git diff` is empty, | |
| # and no PR opens. So it is safe to fire on any of the triggers below. | |
| # | |
| # Triggers: | |
| # - schedule: daily, after MTGJSON's publish window. | |
| # - workflow_dispatch: manual run from the Actions tab. | |
| # - workflow_run on "Clear Caches": clearing the mtgjson/scryfall caches only | |
| # DELETES them; this workflow is the thing that re-fetches, re-processes, | |
| # and commits the result afterward. | |
| on: | |
| schedule: | |
| # Daily at 15:00 UTC, after MTGJSON's publish window. | |
| # | |
| # Was weekly (Mondays), which under-served the actual publish cadence: | |
| # MTGJSON ships roughly daily (5.3.0+20260723 -> +20260724 on consecutive | |
| # days), so for six days out of seven the committed known-tokens.toml was | |
| # stale. That is not cosmetic — build.rs embeds that file, so when | |
| # gen-card-data.sh regenerates it mid-run the deploy has to recompile the | |
| # engine a SECOND time to embed the new catalog. Measured at 5m35s on top | |
| # of the first 5m39s compile (run 30114030288), on a job that was timing | |
| # out at 30m. Keeping the committed catalog fresh is what removes it. | |
| # | |
| # Firing daily is safe by construction: the vintage write-gate above means | |
| # a run against unchanged MTGJSON rewrites nothing and opens no PR. | |
| - cron: "0 15 * * *" | |
| workflow_dispatch: | |
| workflow_run: | |
| workflows: ["Clear Caches"] | |
| types: [completed] | |
| permissions: | |
| contents: write | |
| pull-requests: write | |
| jobs: | |
| refresh-catalogs: | |
| name: Regenerate & Commit Catalogs | |
| # Forks inherit the schedule but not CI_PAT, so a fork run would only fail | |
| # and spam the owner — keep it on the canonical repo. On a workflow_run | |
| # trigger, only proceed if the cache clear actually succeeded (a failed or | |
| # cancelled clear left the caches as-is, so there is nothing new to fetch). | |
| if: >- | |
| github.repository == 'phase-rs/phase' && | |
| (github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success') | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 30 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: actions-rust-lang/setup-rust-toolchain@v1 | |
| with: | |
| # Share the tool-binary build cache with deploy.yml's card-data job. | |
| cache-shared-key: rust-tool | |
| - name: Regenerate card-data catalogs from fresh MTGJSON | |
| # PHASE_REFRESH_MTGJSON=1 forces a full re-download of every MTGJSON | |
| # input (AtomicCards, Meta, CardTypes, SetList, token sets, decks) so the | |
| # vintage gate sees the latest published date. gen-card-data.sh is the | |
| # single source of truth: it runs tokens-gen + oracle-gen behind the | |
| # vintage write-gate and bumps crates/engine/data/mtgjson-vintage when | |
| # the input date advances. It needs only cargo + jq (preinstalled) — no | |
| # node/scryfall — so nothing but MTGJSON is on the critical path. | |
| env: | |
| PHASE_REFRESH_MTGJSON: "1" | |
| run: | | |
| set -euo pipefail | |
| mkdir -p data | |
| ./scripts/gen-card-data.sh 2>&1 | tee /tmp/gen-card-data.log | |
| - name: Verify token-set fetch was complete | |
| # A catalog is only trustworthy if EVERY token-bearing set downloaded. | |
| # fetch-token-sets.sh tolerates a per-set download failure — it drops a | |
| # `.missing` marker and continues, exiting 0 — so a flaky MTGJSON pull | |
| # would otherwise ship a silently partial known-tokens.toml (fewer | |
| # source_card_refs from the sets that failed). That is the exact failure | |
| # the engine's token_coverage floor was left to catch after the fact. | |
| # Enforce completeness at the source instead: refuse to open a PR unless | |
| # the fetch reported zero failures. PHASE_REFRESH_MTGJSON=1 forces a clean | |
| # pull (FORCE=1), so `skipped` must be 0 too — any skip means a leftover | |
| # `.missing` marker from a set that never came down. | |
| run: | | |
| set -euo pipefail | |
| tally="$(grep -E '^Token sets: downloaded [0-9]+, skipped [0-9]+, failed [0-9]+' /tmp/gen-card-data.log | tail -1 || true)" | |
| if [ -z "$tally" ]; then | |
| echo "::error::No 'Token sets:' fetch tally in the generator log; cannot confirm a complete fetch." >&2 | |
| exit 1 | |
| fi | |
| echo "Fetch tally: $tally" | |
| [[ "$tally" =~ skipped\ ([0-9]+),\ failed\ ([0-9]+) ]] | |
| skipped="${BASH_REMATCH[1]}" | |
| failed="${BASH_REMATCH[2]}" | |
| if [ "$failed" -ne 0 ] || [ "$skipped" -ne 0 ]; then | |
| echo "::error::Token-set fetch incomplete (skipped=$skipped, failed=$failed). Refusing to open a catalog PR from a partial regen; re-run once MTGJSON is reachable." >&2 | |
| exit 1 | |
| fi | |
| echo "Token-set fetch complete: every token-bearing set downloaded." | |
| - name: Detect tracked-catalog changes | |
| id: diff | |
| # card-data.json and the downloaded MTGJSON inputs are gitignored; only | |
| # these three engine-embedded files are tracked. Scope the check (and the | |
| # commit below) to exactly them. | |
| run: | | |
| if git diff --quiet -- \ | |
| crates/engine/data/known-tokens.toml \ | |
| crates/engine/data/oracle-subtypes.json \ | |
| crates/engine/data/mtgjson-vintage; then | |
| echo "changed=false" >> "$GITHUB_OUTPUT" | |
| echo "Catalogs unchanged — MTGJSON input is not newer than the committed vintage; no PR needed." | |
| else | |
| echo "changed=true" >> "$GITHUB_OUTPUT" | |
| fi | |
| - name: Create or update catalog refresh PR | |
| if: steps.diff.outputs.changed == 'true' | |
| uses: ./.github/actions/create-auto-merge-pr | |
| with: | |
| token: ${{ secrets.CI_PAT }} | |
| add-paths: | | |
| crates/engine/data/known-tokens.toml | |
| crates/engine/data/oracle-subtypes.json | |
| crates/engine/data/mtgjson-vintage | |
| commit-message: "chore(card-data): refresh MTGJSON token & subtype catalogs" | |
| branch: chore/refresh-card-data-catalogs | |
| title: "chore(card-data): refresh MTGJSON token & subtype catalogs" | |
| body: | | |
| Automated refresh of the engine-embedded card-data catalogs from the latest MTGJSON publish. | |
| - `crates/engine/data/known-tokens.toml` — named-token catalog (`tokens-gen`) | |
| - `crates/engine/data/oracle-subtypes.json` — creature-subtype vocabulary (`oracle-gen --write-subtypes`) | |
| - `crates/engine/data/mtgjson-vintage` — vintage write-gate stamp | |
| Generated by `PHASE_REFRESH_MTGJSON=1 scripts/gen-card-data.sh`; the vintage gate only promotes catalogs when the MTGJSON input date advances, so this diff reflects a genuine upstream data change. Review, then let it auto-merge once CI is green. |