Skip to content

fix: compile-time checks for the tables, switches, masks and layouts this tree's defects came from; the wrong answers they surfaced #1003

fix: compile-time checks for the tables, switches, masks and layouts this tree's defects came from; the wrong answers they surfaced

fix: compile-time checks for the tables, switches, masks and layouts this tree's defects came from; the wrong answers they surfaced #1003

Workflow file for this run

name: CI
# Guardrails as automated gates (CLAUDE.md "The guardrails as automated gates"):
# - release job: configure (portable default, no RIPWIRE_NATIVE — CI hardware is not "this dev machine")
# + build + the full gate suite (170+ gates incl. determinism/G4/quality) on SIX legs:
# build-flavour (Release / plain) x front end (macOS AppleClang, Linux gcc, Linux
# clang) — see the job comment for why each axis exists — run in parallel via
# test/pargates.py -j 3 (same authoritative list as regression.sh), each leg's suite
# SHARDED across 4 runner jobs on every leg (--shard K/N, cost-balanced by
# .github/pargates-shard-weights.json) so the workflow's wall clock is one shard, not one suite.
# ubuntu-24.04 here is also the FIRST real proof of L1's portability fix: real x86-64
# Linux hardware, not the local RIPWIRE_PRETEND_LINUX proxy test/portablebuildcheck.sh
# uses on the (Apple Silicon) dev machines this repo is normally built on.
# - rhel job: the platform this project ships a Linux prebuilt for, checked at PR time instead of only
# at release. RHEL 9 userland (ubi9) via `container:` on a hosted Ubuntu runner, built with
# Red Hat's own gcc-toolset (RHEL 9's default gcc 11 cannot do C++23). Build + fixture
# self-run + the two headline guarantees, NOT the full suite — see the job comment for why.
# - asan job: configure -DRIPWIRE_ASAN=ON (G1: address,undefined,integer,float-divide-by-zero,
# float-cast-overflow, fail-fast) + build + the fixture self-run + the 5 heaviest verbs
# (--pack-task/--from-trace/--quality-delta/--merge-scout/--edit-check)
# via their own gate scripts pointed at the ASan binary. NOT the full regression.sh under
# ASan — that would run 170+ mostly-non-heavy gates at sanitizer speed for little extra
# signal; the release job already proves those gates pass on a normal binary per OS.
# - both jobs end with an explicit 2-run byte-diff det-gate + xmllint --noout, even though
# test/regression.sh already covers both, so a CI log reader sees the two headline product
# guarantees (determinism, well-formed XML / G4) called out as their own named steps.
#
# There is no dependency-download step and no dependency cache: tree-sitter core, all 15 grammars and
# doctest are VENDORED under third_party/deps (see CMakeLists.txt), so `actions/checkout` is the whole
# provisioning story. That also means CI exercises the same hermetic build a stranger's clone gets —
# a cache step would have hidden a re-introduced network fetch behind a warm runner.
#
# LIGHT SET vs. FULL MATRIX (owner decision, 2026-09-17: CI was the bottleneck — a merge to main re-ran all
# 31 jobs on a tree its PR had already tested, and per-lane full CI on the landing queue found almost
# nothing). The `plan` job below computes ONE `full` output from the event name, the pull request's labels
# and the ref; every heavy job reads that output through its own `if:` instead of repeating the same
# event/label logic.
# LIGHT (style + the one ubuntu-24.04/Release/clang release leg, all 4 shards — that leg already runs the
# det-gate and G4 xmllint checks): a push to main, or a pull_request carrying the `train-member` label.
# Labelling is maintainer-only by construction — a fork PR cannot add a label to itself.
# FULL (all 31 jobs): every other pull_request (integration/* train PRs, direct-land PRs, contributor
# PRs), workflow_dispatch — the owner requires a full build and test on the EXACT commit before any
# release tag, so dispatch this manually against the commit you are about to tag rather than trusting an
# earlier green nightly — and a nightly `schedule` against main. See the report-failure/report-green jobs
# at the bottom for where a red scheduled run is reported: its OWN tracking issue (label
# nightly-full-matrix, alongside the shared nightly-failure), separate from nightly.yml's TSan issue.
on:
push:
branches: [main]
pull_request:
# The GitHub default (opened/synchronize/reopened) plus labeled/unlabeled, so adding or removing
# `train-member` re-evaluates the plan job's decision on the SAME pull request without a new push.
types: [opened, synchronize, reopened, labeled, unlabeled]
workflow_dispatch:
schedule:
# Full-matrix proof against main, independent of nightly.yml's own 07:17 UTC ThreadSanitizer run — off
# :00 and a different minute from that cron so the two scheduled workflows don't start together.
- cron: "41 5 * * *" # 05:41 UTC daily
permissions:
contents: read # least privilege at the top; only report-failure/report-green below widen, and only for themselves
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
# Cancel a superseded run on a PULL REQUEST only. On main every push is a distinct commit that deserves its
# own verdict: on 2026-09-07 three consecutive main runs were cancelled by unrelated pushes inside the
# ~26-min window, with zero failures among the 45 jobs that had completed — the signal was thrown away, not
# superseded. A queued main run costs runner minutes; a cancelled one costs the answer.
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
# Never RIPWIRE_NATIVE in CI: -march=native would bake in whatever ISA the CI runner's host happens to
# expose that week, defeating the entire point of testing the portable default.
jobs:
# ─── plan: light set or full matrix — computed ONCE, so every heavy job reads one output via `if:` ───────
#
# No extra permission needed: a pull_request event's own webhook payload already lists its labels
# (github.event.pull_request.labels), so this reads the trigger event instead of calling the API.
plan:
name: plan (light set or full matrix?)
runs-on: ubuntu-24.04
outputs:
full: ${{ steps.decide.outputs.full }}
release_matrix: ${{ steps.decide.outputs.release_matrix }}
steps:
# release_matrix is computed here, not job-level `if:` on `release`, because the matrix context is
# not available to a job-level `if:` (only to steps already inside an expanded matrix job — actionlint
# catches this if it regresses: "context matrix is not allowed here"). Generating it also means there
# is exactly ONE source of truth for the full 24-row matrix instead of a second hand-typed copy that
# can drift from it: six legs {macos-26 AppleClang, ubuntu gcc, ubuntu clang} x {Release, plain}, each
# sharded 4 ways — the same axes `release`'s own job comment documents. LIGHT keeps only the one leg
# the light set always runs, ubuntu-24.04/Release/clang, all 4 shards (it already carries the
# det-gate and G4 xmllint checks).
- name: Decide, from event name, PR labels and ref — see the header comment for the rule
id: decide
env:
EVENT: ${{ github.event_name }}
REF: ${{ github.ref }}
PR_LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }}
run: |
full=true
case "$EVENT" in
push)
# on.push.branches is already scoped to [main]; the ref check is belt-and-suspenders so this
# job's own decision does not silently depend on that trigger filter never changing.
if [ "$REF" = refs/heads/main ]; then
full=false
fi
;;
pull_request)
if jq -e 'index("train-member") != null' <<<"$PR_LABELS" >/dev/null; then
full=false
fi
;;
workflow_dispatch|schedule)
full=true
;;
esac
echo "event=$EVENT ref=$REF labels=$PR_LABELS -> full=$full"
echo "full=$full" >> "$GITHUB_OUTPUT"
if [ "$full" = true ]; then
legs='[
{"os":"macos-26","flavor":"Release","cc":"appleclang"},
{"os":"macos-26","flavor":"plain","cc":"appleclang","skiplist":".github/pargates-macos-plain-skip.txt"},
{"os":"ubuntu-24.04","flavor":"Release","cc":"gcc"},
{"os":"ubuntu-24.04","flavor":"plain","cc":"gcc"},
{"os":"ubuntu-24.04","flavor":"Release","cc":"clang"},
{"os":"ubuntu-24.04","flavor":"plain","cc":"clang"}
]'
else
legs='[{"os":"ubuntu-24.04","flavor":"Release","cc":"clang"}]'
fi
release_matrix="$( jq -c '{ include: [ .[] as $leg | range(1;5) as $shard | ($leg + {shard:$shard, shards:4}) ] }' <<<"$legs" )"
echo "release_matrix=$release_matrix" >> "$GITHUB_OUTPUT"
# ─── style: one gating clang-format check, one advisory clang-format report, one advisory clang-tidy ───
#
# CLANG_VERSION is a PIN, not a floor. clang-format's output changes across major releases, so an
# unpinned checker reports "unformatted" on a tree that was formatted correctly by a different release.
# .clang-format was authored and verified against clang-format 22 on the dev machine; scripts/formatcheck.sh
# refuses to run under any other major, so bumping this number without re-running `clang-format -i` over the
# gated file set fails loudly instead of silently re-styling the repo.
#
# The clang-format GATE covers only the files listed in scripts/formatcheck.sh, which are the ones that
# already match .clang-format byte for byte. It cannot cover the tree: the house style in CONTRIBUTING.md §3
# is hand-formatted in ways clang-format has no option to preserve (multi-statement one-liners,
# `for( … ) if( … ) return i;`, packed initialiser rows and `case` labels, hand-chosen 160-200 column wrap
# seams), and reformatting all 98 first-party C++ files changes 11837 lines that survive `git diff -w` —
# real joins and splits — across 89 of them. A whole-tree --Werror check would therefore be red on a
# correctly-styled tree, which is a gate that punishes the documented style. The --advisory step below
# prints that gap on every run so it stays visible rather than forgotten.
#
# clang-tidy is ADVISORY and must stay that way: continue-on-error, and .clang-tidy's WarningsAsErrors is
# empty. Its default catalogue argues for a different C++ than the data-oriented one G2 mandates (POD/SoA,
# C arrays, 32-bit handles, VERIFY instead of exceptions), so the config is curated down to bugprone-* /
# clang-analyzer-* / performance-* / misc-dangling-* and even those only report. Baseline at the commit
# that added it: 312 unique sites, of which 126 are bugprone-easily-swappable-parameters and 85 are
# bugprone-exception-escape. Re-measured 2026-09-10 at d90acb2c, unique file:line:check: 358 while main.cpp's
# TU failed to parse (see the configure step below), 396 once it parsed, 359 after triage — 204 of them
# bugprone-easily-swappable-parameters and 30 bugprone-exception-escape. Every remaining class was read and
# refused with evidence; the record is in the commit that wrote these numbers. Read it as a to-triage list,
# not a queue of defects.
style:
name: style (clang-format gate + advisory clang-tidy)
runs-on: ubuntu-24.04
env:
CLANG_VERSION: "22"
steps:
# L4 (Linux probe): checkout@v4 defaults to a --depth 1 clone, which leaves ONE commit of history
# in the tree. The churn / co-change / ownership gates (churnjoincheck, hotspots, --owners, the
# quality short-horizon-churn kind) mine `git log` for real, so on a shallow checkout they do not
# error — they measure zero and fail, or worse, pass while measuring nothing. Full history it is.
- uses: actions/checkout@v4
with:
fetch-depth: 0 # churn/co-change gates read real git history — a shallow clone reddens them
persist-credentials: false # no step in this job pushes or fetches with the token
- name: Install clang-format / clang-tidy (PINNED major — see the job comment)
run: |
# Third-party apt sources ship in the runner image and are outside our control. TWICE now
# one has been mid-publish when a job ran and taken every Linux job on every branch down
# with it: `apt-get update` exits 100 and the leg dies in setup, before a single gate runs,
# with a failure that looks nothing like a package problem. google-chrome on 2026-09-09
# ("Hash Sum mismatch"; deb822 .sources on 24.04, not .list) and packages.microsoft.com on
# 2026-09-10 ("403 Forbidden ... is no longer signed"). Naming vendors one at a time only
# ever fixes the last outage, so: we install ubuntu-archive packages ONLY, and every
# non-ubuntu source goes. -print names what went, so a future reader can tell whether a
# missing package traces back to here.
ls /etc/apt/sources.list.d/ || true
sudo find /etc/apt/sources.list.d -type f \( -name '*.list' -o -name '*.sources' \) \
! -name 'ubuntu*' -print -delete || true
wget -qO /tmp/llvm.sh https://apt.llvm.org/llvm.sh
chmod +x /tmp/llvm.sh
sudo /tmp/llvm.sh "$CLANG_VERSION" all
clang-format-"$CLANG_VERSION" --version
clang-tidy-"$CLANG_VERSION" --version
# GATING. Runs clang-format --dry-run --Werror over scripts/formatcheck.sh's file list.
# Local equivalent, proven on the dev machine before this step was written:
# CLANG_FORMAT=/opt/homebrew/opt/llvm/bin/clang-format scripts/formatcheck.sh
- name: clang-format — gate (scripts/formatcheck.sh)
run: CLANG_FORMAT=clang-format-"$CLANG_VERSION" bash scripts/formatcheck.sh
# NON-GATING by construction: --advisory always exits 0. It names every first-party C++ file
# clang-format would rewrite, so the size of the ungated remainder is on the record every run.
- name: clang-format — advisory report over the whole first-party set (non-gating)
run: CLANG_FORMAT=clang-format-"$CLANG_VERSION" bash scripts/formatcheck.sh --advisory
# embedded_queries.h is written at CONFIGURE time (file(WRITE) in CMakeLists.txt). version.h is NOT: it is
# the BYPRODUCT of the ripwire_version_stamp custom target, which only runs at build time. src/cli.h
# includes it, so a configure-only database handed clang-tidy a fatal "'version.h' file not found" in
# every TU and the report was taken over a broken parse. Building that one target runs a cmake -P script,
# not the compiler, so there is still no compile needed to lint.
- name: Configure a compile database for clang-tidy
run: |
cmake -S . -B build-tidy -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
cmake --build build-tidy --target ripwire_version_stamp
- name: clang-tidy — ADVISORY, never gates
continue-on-error: true
run: |
clang-tidy-"$CLANG_VERSION" -p build-tidy --quiet \
src/main.cpp src/ingest.cpp src/pagerank.cpp src/tsprobe.cpp src/infra/diagnostics.cpp
release:
needs: plan
# LIGHT set still runs this job — just ONE leg of it, ubuntu-24.04/Release/clang (all 4 shards): see
# `plan`'s own step, which computes the matrix below to exactly that one leg when light and to all six
# legs when full, instead of this job carrying an `if:` (job-level `if:` cannot read the matrix context
# — only a step already inside an expanded matrix job can, so pruning legs needs the matrix ITSELF to
# be conditional, not a condition alongside it).
# Two independent flavours per OS, each on its OWN runner — they used to run stacked in one job,
# doubling wall time for no coverage gain:
# Release — defines NDEBUG, which compiles DEGRADED_PATH_ALERT out; the optimizer-visible build.
# plain — NDEBUG off, so DEGRADED_PATH_ALERT compiles in and the degrade-path gates can observe
# the alert they assert. The e7405e7 qsnap fix was invisible to CI before this flavour
# existed. estchargecheck #14 / qualitystalecheck arms 7/8c probe the flavour (an
# already-gated degrade path + --version's build-type token) and SKIP honestly on the
# Release leg, pointing at the plain leg as where the alert is proven; a missing alert
# on the plain flavour still FAILS. Neither flavour alone is enough.
#
# And BOTH FRONT ENDS on Linux, not just the platform default. gcc is the one this job used to have,
# and it is a feature — a second front end over the same tree, which found three real portability bugs
# on its first run. But it is not the whole story either: RIPWIRE_OPT_REMARKS and RIPWIRE_PGO are
# Clang-only by construction (-Rpass=/-fsave-optimization-record, .profdata/llvm-profdata) and
# CMakeLists FATAL_ERRORs on anything else, so on a gcc-only Linux leg optremarkscheck's configure arms
# cannot be expressed and can only skip. macOS covers Clang but not Linux's libc, headers or linker.
# Six legs = {macos-26 AppleClang, ubuntu gcc, ubuntu clang} x {Release, plain}: every guardrail is
# observed by at least one leg on the OS it matters for, and no leg has to assert the impossible.
name: release (${{ matrix.os }}, ${{ matrix.flavor }}, ${{ matrix.cc }}, shard ${{ matrix.shard }}/${{ matrix.shards }})
strategy:
fail-fast: false
# There is no gcc on the macOS runners (`g++` there is an AppleClang shim, so a `cc: gcc` macOS leg
# would silently be a second AppleClang leg claiming to be gcc), and a leg that lies about its
# toolchain is worse than a leg that is missing — `plan`'s generator uses an explicit include list
# for exactly that reason, rather than crossing three axes and excluding the impossible cell.
# Each leg's gate suite is SHARDED across runner jobs (2026-09-07): at 560+ gates the suite is ~150
# CPU-minutes, ~60 min wall at -j 3 on a 4-vCPU runner, and that one number was the wall clock of the
# whole workflow. test/pargates.py --shard K/N cuts the list into N cost-balanced, deterministic slices
# (weights in .github/pargates-shard-weights.json); every shard job builds its own binary (~3 min), stages the
# HEAD comparison binary in its own step (a second build, outside every gate budget -- see that step), and
# runs its slice. Every leg takes 4 shards. macOS took 2 until PR #110, on the grounds that GitHub caps
# concurrent macOS jobs far lower than Linux; 2 put ~300 gates in each macOS job, doubling the -j 3 contention
# every gate there ran under. The cost of 4 is stated rather than hidden: when the eight macOS shard jobs
# (plus asan's) exceed the account's concurrent macOS cap, the excess waits for a free runner.
#
# The matrix itself is `plan`'s computed output, not a literal list here (see that job's step): its
# SIZE is conditional on the light/full decision, which a static `include:` cannot express, and a
# second hand-typed copy of the full 24-row list would drift from the one `plan` generates.
matrix: ${{ fromJSON(needs.plan.outputs.release_matrix) }}
runs-on: ${{ matrix.os }}
# The macOS legs test the toolchain release.yml's macos-arm64 leg ships: the same runner, Xcode 26.6 and the pinned
# minimum macOS 14.0 (test/portablebuildcheck.sh #2i holds all three against release.yml). Xcode 26.6's Apple clang
# 21 is LLVM 18+, so its loop vectorizer reads VERIFY_NO_ALIAS's separate-storage promise (test/noaliascheck.sh arm
# 6); its libc++ still defines __cpp_lib_print at a 14.0 deployment target, and it decides what else it declares
# available by that target, so the legs build at the one the release ships. Through 0.6.1 these legs ran Xcode
# 16.2 (LLVM 17) on macos-14, which retires 2026-11-02. Both variables are empty, and inert, on the Linux legs.
env:
DEVELOPER_DIR: ${{ matrix.os == 'macos-26' && '/Applications/Xcode_26.6.app/Contents/Developer' || '' }}
MACOSX_DEPLOYMENT_TARGET: ${{ matrix.os == 'macos-26' && '14.0' || '' }}
steps:
# L4 (Linux probe): checkout@v4 defaults to a --depth 1 clone, which leaves ONE commit of history
# in the tree. The churn / co-change / ownership gates (churnjoincheck, hotspots, --owners, the
# quality short-horizon-churn kind) mine `git log` for real, so on a shallow checkout they do not
# error — they measure zero and fail, or worse, pass while measuring nothing. Full history it is.
- uses: actions/checkout@v4
with:
fetch-depth: 0 # churn/co-change gates read real git history — a shallow clone reddens them
persist-credentials: false # no step in this job pushes or fetches with the token
# clang is installed on BOTH Linux legs, not only the clang one. optremarkscheck's Clang-only
# configure arms pin `clang++` when the default front end is not Clang, so the gcc leg keeps that
# coverage instead of skipping it — the same remedy the asan job applies for G1's Clang-only
# sanitizer members. It is a package install, not a change of the leg's own compiler: the gcc leg
# still BUILDS ripwire with gcc, which is the point of having it.
- name: Install tooling (Linux)
if: runner.os == 'Linux'
run: |
# Third-party apt sources ship in the runner image and are outside our control. TWICE now
# one has been mid-publish when a job ran and taken every Linux job on every branch down
# with it: `apt-get update` exits 100 and the leg dies in setup, before a single gate runs,
# with a failure that looks nothing like a package problem. google-chrome on 2026-09-09
# ("Hash Sum mismatch"; deb822 .sources on 24.04, not .list) and packages.microsoft.com on
# 2026-09-10 ("403 Forbidden ... is no longer signed"). Naming vendors one at a time only
# ever fixes the last outage, so: we install ubuntu-archive packages ONLY, and every
# non-ubuntu source goes. -print names what went, so a future reader can tell whether a
# missing package traces back to here.
ls /etc/apt/sources.list.d/ || true
sudo find /etc/apt/sources.list.d -type f \( -name '*.list' -o -name '*.sources' \) \
! -name 'ubuntu*' -print -delete || true
sudo apt-get update
# poppler-utils is pdftotext: ripwirepubliccheck arm 1b FAILS a tracked PDF deck it cannot read.
sudo apt-get install -y --no-install-recommends libxml2-utils ripgrep bc clang poppler-utils
- name: Install tooling (macOS)
if: runner.os == 'macOS'
# poppler is pdftotext, which ripwirepubliccheck arm 1b needs: it FAILS a tracked deck it cannot read. Only the
# Release legs install it, because the plain legs skip that gate (.github/pargates-macos-plain-skip.txt).
run: brew install ripgrep ${{ matrix.flavor == 'Release' && 'poppler' || '' }}
# CC/CXX: the clang leg names clang; the gcc leg names gcc-14 (the std::print floor, 2026-09-08 — the
# ubuntu-24.04 image ships GNU C++ 12.4 / 13.3 / 14.2, and the SYSTEM default `g++` is 13.3, whose
# libstdc++ has <format> but not <print>); the macOS leg is left to its platform default under the
# DEVELOPER_DIR pinned at the job level. The stranger's-clone question this used to answer — does
# the stock ubuntu g++ 13 still build? — is answered by the `fallback-emitter` job below instead,
# which builds with g++-13 on purpose and proves the fallback path emits the same bytes.
- name: Configure (portable — no RIPWIRE_NATIVE; empty build type = the plain flavour)
env:
CC: ${{ matrix.cc == 'clang' && 'clang' || matrix.cc == 'gcc' && 'gcc-14' || '' }}
CXX: ${{ matrix.cc == 'clang' && 'clang++' || matrix.cc == 'gcc' && 'g++-14' || '' }}
run: cmake -S . -B build ${{ matrix.flavor == 'Release' && '-DCMAKE_BUILD_TYPE=Release' || '' }}
# The front end is recorded per leg so a log reader never has to infer it from the job name alone —
# a `cc: clang` leg whose CMake cache says GNU is a broken matrix, not a passing run.
# A macOS leg also proves the release toolchain pin reached it. The job env's matrix.os condition falls back to an
# empty string on any other runner label, so a runner bump without the condition would otherwise build silently
# with the image's default Xcode at the runner's own minimum macOS.
- name: Confirm the front end this leg actually configured with
run: |
grep -h 'CMAKE_CXX_COMPILER_ID\|CMAKE_CXX_COMPILER_VERSION' build/CMakeFiles/*/CMakeCXXCompiler.cmake
if [ "$RUNNER_OS" = macOS ]; then
if [ -z "$DEVELOPER_DIR" ] || [ -z "$MACOSX_DEPLOYMENT_TARGET" ] \
|| ! grep -qx "CMAKE_OSX_DEPLOYMENT_TARGET:STRING=$MACOSX_DEPLOYMENT_TARGET" build/CMakeCache.txt; then
echo "ci: macOS leg without the release toolchain pin: DEVELOPER_DIR='$DEVELOPER_DIR' MACOSX_DEPLOYMENT_TARGET='$MACOSX_DEPLOYMENT_TARGET' cache='$( grep '^CMAKE_OSX_DEPLOYMENT_TARGET:' build/CMakeCache.txt )'" >&2
exit 1
fi
echo "DEVELOPER_DIR=$DEVELOPER_DIR MACOSX_DEPLOYMENT_TARGET=$MACOSX_DEPLOYMENT_TARGET (in CMakeCache.txt)"
fi
- name: Build
run: cmake --build build -j
# The floor this matrix exists to hold, read off the BINARY rather than inferred from the compiler
# version: emit= names the emitter that compiled in (src/infra/emit.h). A leg whose standard library
# lacks <print> still builds — the emitter falls back to std::format+fputs by feature test — so
# without this line such a leg would build green and read as "the std::print floor holds".
- name: The emitter this leg compiled in is std::print
run: |
./build/ripwire --version
./build/ripwire --version | grep -q 'emit=std::print'
# The HEAD comparison binary, built ONCE per job in its own step: before the suite, outside every gate budget.
# Six gates (crossdirinclude, nestedimport, preproccond, py/rust/tsimportprecise; binoverridecheck re-runs them)
# compare this tree's binary against a second ripwire built from git HEAD. test/lib/headbinlib.sh used to build
# it INSIDE whichever of them ran first: a parallel cmake build inside a wall-clock budget, beside -j 3
# neighbours, whose cost is super-linear in that contention. On macos-14 Release shard 2/2 the xmlwellformed
# runner-speed probe moved 4.6x across five draws while crossdirincludecheck moved more than 15x, and it was
# killed at 900 s and then at 1200 s with the budget floor applied (runs 34479806177, 34495265793; again in
# 34514091873). No budget survives the next slow draw, so the build moved here, where it gets the idle runner
# the Build step above gets. RIPWIRE_HEADBIN puts headbinlib.sh in STAGED mode for every gate in this job: it
# never builds and never waits, and a missing or wrong binary FAILS the gate instead of rebuilding inside its
# budget. test/headbinstagecheck.sh gates that this step exists, precedes the suite, verifies and exports.
- name: Stage the HEAD comparison binary (once per job, before the suite, outside every gate budget)
run: |
export RIPWIRE_HEADBIN_BUILD_LOG="$RUNNER_TEMP/headbin-build.log"
. test/lib/headbinlib.sh
hb="$( ripwire_head_binary "$PWD" "$RUNNER_TEMP" )" || { echo "HEAD binary build failed; last 80 lines of its log:"; tail -n 80 "$RIPWIRE_HEADBIN_BUILD_LOG"; exit 1; }
ripwire_headbin_verify "$hb" "$( git rev-parse HEAD )"
"$hb" --version
echo "RIPWIRE_HEADBIN=$hb" >> "$GITHUB_ENV"
# test/pargates.py runs the SAME authoritative gate list regression.sh holds (manifestcheck.sh
# keeps that list honest) in parallel, with the timing-sensitive edit-check budget isolated via
# its `exclusive` group — the exact runner CLAUDE.md prescribes locally. Sequential regression.sh
# cost ~26 min per suite pass at 170 gates; at 560+ gates the whole suite measured ~62 min wall per
# leg at -j 3 (run 34109497781, 2026-09-07), so each job now runs ONE shard of it — see the matrix
# comment. The shard's own gates are still LPT-ordered by the scratch timings; -j 3 stays because
# the 4-vCPU runners saturate (many gates are internally multi-threaded).
- name: "gate suite, parallel, this job's shard (560+ gates: determinism, cache transparency, G4 XML, quality, …)"
# --budget-scale 4: the harness's flat 300 s default is a hang tripwire calibrated on an idle dev machine;
# here -j 3 on a 4-vCPU runner is a 3-8x multiplier (two gates hit rc=124 at exactly 300 s in the first
# sharded runs). The scale applies to the default only; GATE_BUDGET_SEC's explicit entries stay as declared.
# `skiplist` (macOS plain only): a committed list of platform-independent gates this leg does not run —
# see .github/pargates-macos-plain-skip.txt for the measurement and the per-gate reason.
run: python3 test/pargates.py . build/ripwire -j 3 --shard ${{ matrix.shard }}/${{ matrix.shards }} --budget-scale 4 ${{ matrix.skiplist && format('--exclude-list {0}', matrix.skiplist) || '' }}
- name: det-gate — 2-run byte-identical diff
run: |
./build/ripwire test/fixture --no-cache >/tmp/run_a.xml
./build/ripwire test/fixture --no-cache >/tmp/run_b.xml
diff -q /tmp/run_a.xml /tmp/run_b.xml
- name: G4 — xmllint --noout
run: ./build/ripwire test/fixture --no-cache | xmllint --noout -
# ─── fallback-emitter: the stock ubuntu compiler still builds, and emits the SAME bytes ───────────────
#
# Every leg above pins a toolchain with <print>. A stranger's `apt install g++` on ubuntu-24.04 is still
# 13.3, whose libstdc++ has no <print>; src/infra/emit.h then compiles the std::format+fputs arm. This job
# builds that arm ON PURPOSE (CC/CXX = gcc-13/g++-13, both on the image), asserts the binary discloses it,
# and runs the printf-family parity fence against the manifest the std::print legs pinned — so the two
# arms are proven byte-identical on every CI run, not assumed. Without this job the fallback would rot
# unobserved until the next stranger hit it.
fallback-emitter:
needs: plan
if: needs.plan.outputs.full == 'true' # not part of the light set — see the header comment
name: fallback-emitter (ubuntu gcc-13, std::format+fputs)
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
persist-credentials: false # no step in this job pushes or fetches with the token
- name: Install tooling
run: |
# Third-party apt sources ship in the runner image and are outside our control. TWICE now
# one has been mid-publish when a job ran and taken every Linux job on every branch down
# with it: `apt-get update` exits 100 and the leg dies in setup, before a single gate runs,
# with a failure that looks nothing like a package problem. google-chrome on 2026-09-09
# ("Hash Sum mismatch"; deb822 .sources on 24.04, not .list) and packages.microsoft.com on
# 2026-09-10 ("403 Forbidden ... is no longer signed"). Naming vendors one at a time only
# ever fixes the last outage, so: we install ubuntu-archive packages ONLY, and every
# non-ubuntu source goes. -print names what went, so a future reader can tell whether a
# missing package traces back to here.
ls /etc/apt/sources.list.d/ || true
sudo find /etc/apt/sources.list.d -type f \( -name '*.list' -o -name '*.sources' \) \
! -name 'ubuntu*' -print -delete || true
sudo apt-get update
sudo apt-get install -y --no-install-recommends libxml2-utils ripgrep bc
- name: Configure with the image's stock gcc 13 (plain flavour)
env:
CC: gcc-13
CXX: g++-13
run: cmake -S . -B build
- name: Confirm the front end this leg actually configured with
run: grep -h 'CMAKE_CXX_COMPILER_ID\|CMAKE_CXX_COMPILER_VERSION' build/CMakeFiles/*/CMakeCXXCompiler.cmake
- name: Build
run: cmake --build build -j
- name: The emitter this leg compiled in is the fallback, and says so
run: |
./build/ripwire --version
./build/ripwire --version | grep -q 'emit=std::format+fputs'
- name: Byte parity — the fallback arm reproduces the manifest the std::print legs pinned
run: bash test/printffmtparitycheck.sh
- name: versioncheck — the disclosure arm on this arm
run: bash test/versioncheck.sh
- name: det-gate — 2-run byte-identical diff
run: |
./build/ripwire test/fixture --no-cache >/tmp/run_a.xml
./build/ripwire test/fixture --no-cache >/tmp/run_b.xml
diff -q /tmp/run_a.xml /tmp/run_b.xml
# ─── rhel: the platform this project actually ships a prebuilt for, checked at PR time ───────────────
#
# GitHub hosts no RHEL runner image, but a job can run inside RHEL userland ON a hosted Ubuntu runner via
# `container:`. UBI (Universal Base Image) pulls from registry.access.redhat.com with no subscription,
# which is what makes this work on public CI — release.yml's `smoke-rhel` job already relies on it.
#
# WHY IT IS WORTH A LEG. release.yml builds the Linux artifact in a manylinux_2_28 container and smoke-runs
# it on ubi9 precisely because a binary built on stock ubuntu-24.04 needs GLIBCXX_3.4.31 and DIES on RHEL 9
# (verified, and recorded in that workflow's own comment). But that check only runs when a release is cut:
# every ordinary PR was tested on glibc 2.39 Ubuntu and macOS only, so a RHEL-incompatible change stayed
# invisible until publish time. This leg moves that signal to the pull request.
#
# SCOPE, STATED HONESTLY. It builds and runs the two headline product guarantees (determinism, well-formed
# XML) plus the fixture self-run — NOT the 361-gate suite. That is deliberate, not laziness: the suite needs
# ripgrep, which is not in the RHEL repositories (it lives in EPEL), and the RHEL-specific signal here is
# "does this toolchain compile it and does this userland run it", which the ubuntu legs cannot give and
# which these steps do give. The full suite stays on the four ubuntu legs and the two macOS ones.
rhel:
needs: plan
if: needs.plan.outputs.full == 'true' # not part of the light set — see the header comment
name: rhel (ubi9, ${{ matrix.flavor }})
runs-on: ubuntu-24.04
container: registry.access.redhat.com/ubi9/ubi
strategy:
fail-fast: false
matrix:
flavor: [Release, plain]
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
persist-credentials: false # no step in this job pushes or fetches with the token
# RHEL 9's default gcc is 11, which does not implement C++23 — CMakeLists sets CXX_STANDARD 23 with
# STANDARD_REQUIRED ON, so the configure would fail outright. gcc-toolset-N is Red Hat's own supported
# answer (a parallel-installable newer GCC in /opt/rh). Probe newest-first rather than pinning one exact
# version: which toolsets a given UBI9 AppStream carries changes over the RHEL 9.x lifecycle, and a hard
# pin on one N turns a routine repo refresh into a red leg the day that exact package is dropped.
# Whichever is found is ECHOED, so the log always records which compiler actually built this leg rather
# than leaving it to be inferred.
#
# Floor is 14, not merely "whichever toolset exists": the house emitter (src/infra/emit.h) is
# std::print, which libstdc++ implements starting at GCC 14 (<format> arrived at 13, <print> at 14).
# A lower toolset still BUILDS — the emitter falls back by feature test — which is exactly why the
# probe must not accept one: the leg would report "toolset found, leg is fine" while the binary it
# built took the fallback. The assertion step after Build reads the truth off the binary.
- name: Install the RHEL build toolchain
run: |
set -eu
dnf install -y cmake git libxml2 python3 make diffutils findutils
found=""
for v in $( seq 20 -1 14 ); do
if dnf install -y "gcc-toolset-$v-gcc-c++"; then
found="$v"
break
fi
done
[ -n "$found" ] || { echo "no gcc-toolset-N (N>=14) in this UBI9 AppStream — the house emitter is std::print, which needs libstdc++ 14+ (<print>); gcc-toolset-13 and older would silently take the std::format+fputs fallback"; exit 1; }
echo "TOOLSET=$found" >> "$GITHUB_ENV"
- name: Record the toolchain this leg is using
run: |
echo "gcc-toolset-$TOOLSET"
"/opt/rh/gcc-toolset-$TOOLSET/root/usr/bin/g++" --version
cmake --version | head -1
cat /etc/redhat-release
# CC/CXX point straight at the toolset's binaries rather than `source .../enable`, because every `run:`
# step is its own shell — a sourced environment would not survive to the next step.
- name: Configure (portable — no RIPWIRE_NATIVE; empty build type = the plain flavour)
env:
CC: /opt/rh/gcc-toolset-${{ env.TOOLSET }}/root/usr/bin/gcc
CXX: /opt/rh/gcc-toolset-${{ env.TOOLSET }}/root/usr/bin/g++
run: cmake -S . -B build ${{ matrix.flavor == 'Release' && '-DCMAKE_BUILD_TYPE=Release' || '' }}
- name: Build
run: cmake --build build -j
- name: The emitter this leg compiled in is std::print (gcc-toolset-14's libstdc++ has <print>)
run: |
./build/ripwire --version
./build/ripwire --version | grep -q 'emit=std::print'
- name: Self-run on the fixture (it parses and ranks under RHEL userland)
run: ./build/ripwire test/fixture --no-cache | head -c 400; echo
- name: det-gate — 2-run byte-identical diff
run: |
./build/ripwire test/fixture --no-cache >/tmp/run_a.xml
./build/ripwire test/fixture --no-cache >/tmp/run_b.xml
diff -q /tmp/run_a.xml /tmp/run_b.xml
- name: G4 — xmllint --noout
run: ./build/ripwire test/fixture --no-cache | xmllint --noout -
asan:
needs: plan
if: needs.plan.outputs.full == 'true' # not part of the light set — see the header comment
name: asan (${{ matrix.os }})
strategy:
fail-fast: false
matrix:
os: [macos-26, ubuntu-24.04]
runs-on: ${{ matrix.os }}
env:
# Darwin's arm64 runtime rejects the standalone leak sanitizer at startup (CMakeLists.txt's own
# comment on this, `ripwire_asan_fixture`); Linux gets full leak detection.
# matrix.os, not runner.os: the runner context does not exist at job-level env — GitHub's
# parser rejects the whole workflow (found by the first real Actions run; local YAML lint
# cannot see it). Step-level `if: runner.os` below is legal and stays.
# log_path: a sanitizer abort inside a gate that pipes 2>/dev/null leaves NO report in the log
# (CI round 3: an asan/x86-64 abort read as "pin no longer resolves"). With log_path the report
# lands in a file the failure artifact below preserves.
ASAN_OPTIONS: ${{ matrix.os == 'macos-26' && 'detect_leaks=0:halt_on_error=1:abort_on_error=1:log_path=/tmp/asanlog' || 'detect_leaks=1:halt_on_error=1:abort_on_error=1:log_path=/tmp/asanlog' }}
UBSAN_OPTIONS: halt_on_error=1:print_stacktrace=1:log_path=/tmp/asanlog
LSAN_OPTIONS: suppressions=${{ github.workspace }}/lsan_suppressions.txt
# The release toolchain on the macOS leg, as in the build matrix: Xcode 26.6 at the pinned minimum macOS 14.0
# (test/portablebuildcheck.sh #2i). Inert (empty) on Linux.
DEVELOPER_DIR: ${{ matrix.os == 'macos-26' && '/Applications/Xcode_26.6.app/Contents/Developer' || '' }}
MACOSX_DEPLOYMENT_TARGET: ${{ matrix.os == 'macos-26' && '14.0' || '' }}
steps:
# L4 (Linux probe): checkout@v4 defaults to a --depth 1 clone, which leaves ONE commit of history
# in the tree. The churn / co-change / ownership gates (churnjoincheck, hotspots, --owners, the
# quality short-horizon-churn kind) mine `git log` for real, so on a shallow checkout they do not
# error — they measure zero and fail, or worse, pass while measuring nothing. Full history it is.
- uses: actions/checkout@v4
with:
fetch-depth: 0 # churn/co-change gates read real git history — a shallow clone reddens them
persist-credentials: false # no step in this job pushes or fetches with the token
- name: Install tooling (Linux)
if: runner.os == 'Linux'
run: |
# Third-party apt sources ship in the runner image and are outside our control. TWICE now
# one has been mid-publish when a job ran and taken every Linux job on every branch down
# with it: `apt-get update` exits 100 and the leg dies in setup, before a single gate runs,
# with a failure that looks nothing like a package problem. google-chrome on 2026-09-09
# ("Hash Sum mismatch"; deb822 .sources on 24.04, not .list) and packages.microsoft.com on
# 2026-09-10 ("403 Forbidden ... is no longer signed"). Naming vendors one at a time only
# ever fixes the last outage, so: we install ubuntu-archive packages ONLY, and every
# non-ubuntu source goes. -print names what went, so a future reader can tell whether a
# missing package traces back to here.
ls /etc/apt/sources.list.d/ || true
sudo find /etc/apt/sources.list.d -type f \( -name '*.list' -o -name '*.sources' \) \
! -name 'ubuntu*' -print -delete || true
sudo apt-get update
sudo apt-get install -y --no-install-recommends libxml2-utils ripgrep bc clang
- name: Install tooling (macOS)
if: runner.os == 'macOS'
run: brew install ripgrep
# CLANG, not the default cc. G1's `integer` group is Clang-only and gcc rejects the whole option
# ("unrecognized argument to '-fsanitize=' option: 'integer'" — the first public CI run). CMakeLists.txt
# now filters the Clang-only members out under GCC, but that is the HONEST-DEGRADE path for a
# contributor, not something public CI should be running: pinning clang here keeps the COMPLETE G1
# stack on Linux, so the gates below mean the same thing on both matrix legs.
#
# Deliberately NOT applied to the release job: gcc there is a FEATURE. It is a second front end over
# the same tree and it found three real portability bugs on its first run.
- name: Configure (G1 sanitizer stack — clang, see the step comment)
if: runner.os == 'Linux'
env:
CC: clang
CXX: clang++
run: cmake -S . -B asan -DRIPWIRE_ASAN=ON
# Fails, as the build matrix's front-end step does, when the release toolchain pin did not reach this leg.
- name: Configure (G1 sanitizer stack)
if: runner.os == 'macOS'
run: |
cmake -S . -B asan -DRIPWIRE_ASAN=ON
if [ -z "$DEVELOPER_DIR" ] || [ -z "$MACOSX_DEPLOYMENT_TARGET" ] \
|| ! grep -qx "CMAKE_OSX_DEPLOYMENT_TARGET:STRING=$MACOSX_DEPLOYMENT_TARGET" asan/CMakeCache.txt; then
echo "ci: macOS asan leg without the release toolchain pin: DEVELOPER_DIR='$DEVELOPER_DIR' MACOSX_DEPLOYMENT_TARGET='$MACOSX_DEPLOYMENT_TARGET' cache='$( grep '^CMAKE_OSX_DEPLOYMENT_TARGET:' asan/CMakeCache.txt )'" >&2
exit 1
fi
- name: Build
run: cmake --build asan -j
- name: The emitter this leg compiled in is std::print
run: |
./asan/ripwire --version
./asan/ripwire --version | grep -q 'emit=std::print'
- name: G1 fixture self-run (ripwire_asan_fixture)
run: cmake --build asan --target ripwire_asan_fixture
- name: Heavy verb — --pack-task (test/packtaskcheck.sh)
run: RIPWIRE_BIN=asan/ripwire bash test/packtaskcheck.sh
- name: Heavy verb — --from-trace (test/tracecheck.sh)
run: RIPWIRE_BIN=asan/ripwire bash test/tracecheck.sh
- name: Heavy verb — --quality-delta (test/qualitycheck.sh)
run: RIPWIRE_BIN=asan/ripwire bash test/qualitycheck.sh
- name: Heavy verb — --merge-scout (test/mergescoutcheck.sh)
run: RIPWIRE_BIN=asan/ripwire bash test/mergescoutcheck.sh
- name: Heavy verb — --edit-check (test/editcheckcheck.sh)
run: RIPWIRE_BIN=asan/ripwire bash test/editcheckcheck.sh
# cachefuzzcheck's two mutation tables ARE sanitizer sweeps: they drive corrupt/truncated/hostile cache
# blobs and corrupt qsnap blobs through the readers and assert that no sanitizer report fires. That needs
# an instrumented binary — RIPWIRE_ASAN_BIN is the whole point of the gate — and the `release` jobs, which
# configure only build/, cannot supply one. Until now the gate hard-FAILED there for the missing binary
# (PR #1, run 30732976779: "absorb gate (cachefuzzcheck.sh failed)" on BOTH release legs, that one line
# its only failure) while NO job anywhere ran the sweeps it exists for. The gate now skips those two arms
# with a named reason on a leg that has no ASan build, and this step is where they actually run: both
# variables point at the instrumented binary, so the ground-truth arms and the sanitizer arms agree by
# construction. Adding it here is what makes that skip reason true rather than an excuse.
- name: Sanitizer sweep — corrupt cache / qsnap blobs (test/cachefuzzcheck.sh)
run: RIPWIRE_BIN=asan/ripwire RIPWIRE_ASAN_BIN=asan/ripwire bash test/cachefuzzcheck.sh
# NOT a heavy verb — here because it is the ONLY gate whose subject is an integer-overflow class that
# a plain binary cannot observe. cppqualcheck §9 drives an unbalanced `operator>` frame through
# --from-trace; before the H4 W2b fixup that aborted the G1 build (rc=134) while the plain build ran
# it in silence. The suite job above runs the whole suite TWICE and both times PLAIN, so this gate was
# green-by-construction in CI until it appeared here (V3-H-2).
- name: Sanitizer-only class — cppqualcheck §9 operator-frame scan (test/cppqualcheck.sh)
run: RIPWIRE_BIN=asan/ripwire bash test/cppqualcheck.sh
# crashsweepcheck B3 is the other sanitizer-only class: a file dated past 2262 overflowed the stat readers'
# nanosecond product, silent UB in the plain build and an abort here. The job's Linux leg is the one whose
# filesystem stores such a date (APFS clamps at 2262, and the arm says so on macOS). B2's hostile
# fixed-name files ride along on the instrumented binary; B1's preload shim skips a sanitizer build by name.
- name: Sanitizer-only class — crashsweepcheck far-future timestamp and fixed-name shapes (test/crashsweepcheck.sh)
run: RIPWIRE_BIN=asan/ripwire RIPWIRE_ASAN_BIN=asan/ripwire bash test/crashsweepcheck.sh
- name: det-gate — 2-run byte-identical diff (ASan binary)
run: |
./asan/ripwire test/fixture --no-cache >/tmp/run_a.xml
./asan/ripwire test/fixture --no-cache >/tmp/run_b.xml
diff -q /tmp/run_a.xml /tmp/run_b.xml
- name: G4 — xmllint --noout (ASan binary)
run: ./asan/ripwire test/fixture --no-cache | xmllint --noout -
# Self-dogfood, STILL NON-GATING — but for a smaller and better-measured reason than before.
#
# The verb now separates the historical floor from the rot: a failed anchor the AUTHOR dated is
# kind="dated-record" and counts in dated=, leaving drift= for live claims. On this repo's own docs
# that splits 104 failed anchors into 63 rot + 41 records, and BOTH constants this comment used to
# name (`kMcpVerbCount = 22`, `kCacheVersion=7`) are now records — their authors had already written
# the "as of <date>" hedge the lane reads.
#
# `|| true` stays, and note first that it was never suppressing an exit code: --doc-drift always
# exits 0 by design (a report, not a gate), so this only guards a crash. Gating would mean asserting
# drift="0", and the residual 63 will not go to zero by any rule change — they are genuine stale
# file:line maps in undated DESIGN studies, plus two audits that are manifestly artifacts-of-a-date
# to a human but never write that date where a machine can read it. The lane refuses to guess genre
# (measured: a bare date in the opening prose mis-dates three LIVE documents here), and git history
# cannot help either — 90 of 98 stale anchors were CORRECT at their own doc's last commit, records
# and rot alike. So the honest options are a zero-gate that is red on day one, or a ratchet ceiling
# that goes red whenever a doc is edited in a shared tree. Neither earns its noise.
#
# What DOES gate is test/docdriftcheck.sh, which pins every lane and every rec= kind on a labelled
# fixture, including the negative controls. Re-open this decision when the DESIGN docs' anchors are
# fixed and the review notes carry a self-date: at that point drift= can be gated at 0.
- name: Self-report (non-gating) — --doc-drift on our own docs
run: ./asan/ripwire . --doc-drift --with-history || true
# Sanitizer reports written via log_path survive gates that swallow stderr; keep them when a
# step failed so the log tells us WHICH abort happened, not just that one did.
- name: Upload sanitizer reports on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: sanitizer-reports-${{ matrix.os }}
path: /tmp/asanlog*
if-no-files-found: ignore
# ─── report-failure / report-green: this schedule's OWN tracking issue, separate from nightly.yml's ─────
#
# ci.yml's 05:41 UTC schedule and nightly.yml's 07:17 UTC schedule are two independent full runs against
# main. They used to report to the SAME issue, and that was a bug: nightly.yml's report-green closes on a
# green TSan run alone, which is not evidence THIS workflow's full matrix is also clean, so a green TSan
# night could close an issue this workflow opened while its own matrix was still red — and the reverse,
# a green full-matrix night closing an issue TSan opened while TSan was still red. Neither green run has
# any way to know the other workflow's current state.
#
# THE FIX: two labels, not one. `nightly-failure` stays as the SHARED label (so "every nightly-scale
# failure on main" is still one query), but every open/comment/close here also requires
# `nightly-full-matrix` — the label ONLY this workflow's report-failure applies, and ONLY this workflow's
# report-green matches on. nightly.yml's own pair does the same with `nightly-tsan` instead, on ITS own
# issue titled "Nightly checks failing on main (TSan)". The two issues, two titles, two label pairs: a
# green run in one workflow can now only ever touch the issue carrying ITS OWN second label. Neither job
# here runs for push, pull_request or workflow_dispatch — only the scheduled full-matrix run reports,
# matching the "scheduled full-matrix failures" scope this pair exists for; a workflow_dispatch run is
# watched live by whoever triggered it. See CONTRIBUTING.md §5 for the paragraph-length version.
report-failure:
name: report-failure (open or update the tracking issue)
needs: [plan, style, release, fallback-emitter, rhel, asan]
if: failure() && github.event_name == 'schedule' && github.ref == 'refs/heads/main'
runs-on: ubuntu-24.04
permissions:
issues: write
steps:
- name: Open "Nightly checks failing on main (full matrix)", or comment on the open one
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
HEAD_SHA: ${{ github.sha }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
NEEDS_JSON: ${{ toJSON(needs) }}
run: |
label=nightly-failure
ownLabel=nightly-full-matrix
body="$RUNNER_TEMP/body.md"
failed_jobs="$( jq -r 'to_entries[] | select(.value.result == "failure") | .key' <<<"$NEEDS_JSON" | paste -sd, - )"
{
echo "The ci.yml full-matrix schedule failed on \`main\` at $HEAD_SHA."
echo
echo "- run: $RUN_URL (schedule)"
echo "- failing jobs: ${failed_jobs:-none recorded}"
echo
echo "This is ci.yml's full 31-job matrix (release/fallback-emitter/rhel/asan/style), not the"
echo "nightly.yml ThreadSanitizer run — read this run's own job logs for the failing gate."
} > "$body"
if ! gh api "repos/$REPO/labels/$label" --silent 2>/dev/null; then
gh label create "$label" --repo "$REPO" --color B60205 \
--description "A nightly-scale workflow run failed on main; see the workflow-specific label for which one"
fi
if ! gh api "repos/$REPO/labels/$ownLabel" --silent 2>/dev/null; then
gh label create "$ownLabel" --repo "$REPO" --color 1D76DB \
--description "The full-matrix nightly failed on main; opened and closed by .github/workflows/ci.yml only"
fi
open="$( gh issue list --repo "$REPO" --label "$label,$ownLabel" --state open --json number --jq '.[0].number // empty' )"
if [ -n "$open" ]; then
gh issue comment "$open" --repo "$REPO" --body-file "$body"
else
gh issue create --repo "$REPO" --title "Nightly checks failing on main (full matrix)" --label "$label,$ownLabel" --body-file "$body"
fi
report-green:
name: report-green (close the tracking issue on this workflow's own green schedule run)
needs: [plan, style, release, fallback-emitter, rhel, asan]
if: success() && github.event_name == 'schedule' && github.ref == 'refs/heads/main'
runs-on: ubuntu-24.04
permissions:
issues: write
steps:
- name: Comment "green again" on the open tracking issue and close it
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
HEAD_SHA: ${{ github.sha }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
numbers="$( gh issue list --repo "$REPO" --label nightly-failure,nightly-full-matrix --state open --json number --jq '.[].number' )"
if [ -z "$numbers" ]; then
echo "no open nightly-full-matrix issue; nothing to close"
exit 0
fi
for n in $numbers; do
gh issue comment "$n" --repo "$REPO" --body "green again at $HEAD_SHA ($RUN_URL, ci.yml full matrix)"
gh issue close "$n" --repo "$REPO"
done