Repository navigation
ci: PR behaviour gates -- diff coverage 80%, diff budget, PR template check - #1814
Closed
zoroyihan7 wants to merge 4 commits into
Closed
zoroyihan7 wants to merge 4 commits into
zoroyihan7 wants to merge 4 commits into
Conversation
- tests-coverage.yml: on pull requests, the Python 3.10 coverage job writes coverage.xml and runs diff-cover 9.7.1 against the merge commit's base parent with --fail-under=80; the markdown report goes to the job summary. - pr-hygiene.yml (new, always runs, read-only token): diff budget over production Python lines (src/, scripts/, tests excluded; warn > 400, fail > 1000 unless a writer added the size-exception label), the agent-doc CLI reference check moved from docs.yml, and an advisory PR template completeness check that reads the body only as data. - scripts/pr_hygiene.py + tests pinning each refusal message. Co-Authored-By: Claude <noreply@anthropic.com>
A pure rename has zero numstat lines, so moving a 1001-line test file to a production path passed the diff budget at 0 lines. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
… text diffs check_cli_references.py prints PR-controlled file paths; a crafted path could start a workflow command. A PR's .gitattributes marking *.py as -diff hid its lines from both the budget and diff-cover; .git/info/ attributes takes precedence over it. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Author
|
Superseded by #1812, which now carries this change. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tests-coverage.yml,coverage (Python 3.10)job, pull requests only). After the existing combine,coverage xml+diff-cover==9.7.1 --fail-under=80againstHEAD^1of the PR merge commit (checkoutfetch-depth: 2, so no full-history fetch). The markdown report is appended to the job summary. It runs underalways()after the totalfail_undergate, and is skipped when a shard failed or is missing (the shard-results gate already reds that case). On 17 measurable recent PRs, 1 was below 80% (feat(tools): kbmine, a standalone no-run uplift estimator over the Recipe KB and Pulse #1778, 77%).pr-hygiene.yml, jobpr-hygiene).git diff -M --numstat -z HEAD^1 HEADover production Python (src/,scripts/;tests//test/dirs,test_*.py,*_test.py,conftest.pyexcluded; renames count only their edits). Warn above 400, fail above 1000. Thesize-exceptionlabel exempts only if the most recent user who added it has write/maintain/admin on the repo, checked with the read-only token via the issue timeline andcollaborators/{user}/permission; any API failure means not exempt. Counts match the research data exactly on 11 recent merged PRs (fix(enablement): make enablement_setting.sh reproduce the setup on its own #1789 703, refactor(kernelforge): restructure knowledge into domain packages #1788 763, fix(sweep): stop a cancelled conc_sweep, and let SWEEP wait for the sweep it granted #1767 151, ...).PR_BODYenv (never interpolated into shell) and checked for Tests, Size/complexity, Observable effect, Breaking changes and "PR addresses single concern" (must start with yes/no). An untouched template prompt counts as unanswered. Findings are warning annotations + step summary; the job stays green. Making it blocking is one line:PR_TEMPLATE_ENFORCE: "true"inpr-hygiene.yml. Bot-authored PRs are skipped. On the last 50 merged PRs, 20 of 48 non-bot PRs would get a warning.scripts/check_cli_references.pymoved fromdocs.yml(which has a paths filter, so a required check there would sit pending) into the always-runpr-hygienejob.No PR comments are posted (the sticky-comment mechanism is in ci: code-metrics gate on touched files with baseline-free checks; fold in import-linter, PR hygiene and ruff principle rules #1812).
scripts/tests/test_pr_hygiene.py(41 tests) pins each refusal by message: budget exceeded (1001 lines), a 1001-line test file renamed into production (counted in full), at-limit warning (1000 prod lines with 5000 test lines and a rename), label present but unverifiable, labeler with triage only, latest labeler wins across timeline pages, permission API error; template findings per field, yes/no, untouched template, nested/colon-less bullets, workflow-command injection in the body, advisory vs--enforce. Workflow contracts: no paths filter, read-only permissions, body passed via env, the CLI check runs inside::stop-commands::(it prints PR-controlled paths),*.py diffis forced via.git/info/attributesbefore both diff-based checks (a PR.gitattributeswith*.py -diffotherwise hides every line), diff-cover pinned and PR-only. diff-cover invocation reverse-proved locally on a synthetic merge commit: an added function with 5 of 8 lines uncovered -> exit 1 "Failure. Coverage is below 80%"; fully covered -> exit 0; no diff -> exit 0.pr-hygienecheck (diff budget errors, template warnings, CLI reference errors) and a diff-coverage section in thecoverage (Python 3.10)summary; a PR whose new lines are under 80% covered now fails that job.pr-hygieneandcoverage (Python 3.10)as required status checks, and create thesize-exceptionlabel (it does not exist yet). Note thattests-coverage.ymlhas apaths-ignorefor docs-only changes, so a requiredcoverage (Python 3.10)would stay pending on docs-only PRs until that workflow gets an always-run wrapper;pr-hygienehas no paths filter.Known limits: a PR can edit
pr-hygiene.ymlorscripts/pr_hygiene.pyitself to bypass the gate (true of everypull_requestworkflow);/.github/workflows/and*are owned by @AMD-AGI/SaFE, so this is closed oncerequire_code_owner_reviewis enabled. A writer who addssize-exceptionbefore a later large push keeps the exemption. Pure code moves count as new lines for diff-cover.🤖 Generated with Claude Code