Check license headers across the repository

[laughingman7743]

WHAT

Add scripts/check_license_headers.py, which reports files without the The PyAthena authors MIT header described in docs/contributing.md.

• Scope: every tracked and untracked, non-ignored file in the working tree (git ls-files --cached --others --exclude-standard).

• Accepted header: the documented six lines with any four-digit year, at the start of the file, in the comment syntax of the file's format:

  • HTML comments for .md
  • .. comments for .rst
  • Jinja {# -#} comments for .jinja2, and Jinja or HTML comments for .html
  • /* */ for .css
  • // for .jsonc
  • # for everything else

  The header may follow a shebang, an encoding declaration, or a leading YAML document marker. In a Markdown file with YAML front matter, it may be YAML comments anywhere inside the front matter, or an HTML comment directly after it.

• Configuration: scripts/config/license_headers.toml holds the exemptions:

  • exempt-suffixes: .csv, .tsv, .json, .gz, .png, and .lock.
  • unheaded-files: the existing files without the header.

  Unknown keys, non-string lists, and duplicate entries are rejected. Python 3.10 reads the file with tomli, now an explicit dev dependency for Python < 3.11 (already locked through pytest).

• Automatic exemptions: symbolic links, empty or whitespace-only files, binary files (NUL bytes or invalid UTF-8), and the configured suffixes.

• Explicit exemptions: unheaded-files lists the 77 existing files without the header, as classified in Standardize license headers and document PyHive acknowledgements #790 and described in NOTICE. The checker also reports a listed file that gains the header, becomes exempt, or no longer exists. A new file without the header, such as third-party material, is added only as agreed in its issue; docs/contributing.md says so.

• Behavior: it prints problems and exits 1. It never modifies files.

The check runs in these places:

• just license-headers, a new recipe (uv run python scripts/check_license_headers.py). just lint depends on it, and so do just scripts, just test pyathena, and the pyathena tox job of the Test workflow. Locally, an untracked, non-ignored file without the header also fails just lint.
• A new License Headers workflow on every pull request, which runs just license-headers after setting up uv and just. The Test workflow skips fork PRs and docs/**.md-only PRs, and new Markdown files are the most likely to miss the header. The job needs no dependencies and no AWS credentials.

docs/contributing.md now says the header goes at the start of the file (the checker enforces this) and points to the checker. AGENTS.md and scripts/README.md describe the extended checks. Another comment syntax needs an entry in SUFFIX_BLOCKS.

WHY

Closes #806.

#806 left open whether to check only added files or the whole tree with an exception list. The maintainer chose the whole tree with an exception list, because the number of files without the header should not grow.

Compared with a diff-based check, it:

• also catches a header removed from an existing file, and a new file that git reports as a rename;
• works in shallow checkouts without resolving a merge base.

TEST

Tested at 026587b, rebased onto master after #811 and #813:

• just scripts: passes (ruff, format check, mypy, cfn-lint, the new check, shellcheck, actionlint, and 88 script tests including 59 new ones).
• just license-headers on the current tree: exit 0.
• uv lock --check: passes with the uv workspace lock from Manage the benchmark project as a uv workspace member #813.
• just benchmark test: 80 passed, 1 skipped.
• Python 3.10 (uv run --python 3.10 in a separate environment): the checker exits 0, and the 59 checker tests pass, which covers the tomli path.
• Manually injected failures: a new unheaded file reports missing license header; moving a listed file away reports listed as unheaded in scripts/config/license_headers.toml but not found; just license-headers exits 1.
• The unit tests cover:
  • each accepted placement and per-format comment syntax, including headers inside and after front matter;
  • rejected variants: wrong attribution, a two-digit year, a different SPDX identifier, a missing line, an unclosed block, a header not at the start, another format's syntax (for example # lines in Markdown), and a header after front matter ends in the wrong syntax or after content;
  • each automatic exemption and each stale-entry case;
  • loading the repository config, and rejecting a missing or unknown key, a non-list value, and duplicate entries;
  • file listing in a temporary git repository, including a non-ASCII name with a trailing space.

No AWS resources are needed.

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one (implementation behavior): CLEAN

Base 82e88f4ebbd89a4dad8b3b57ac412012601f0c45, head f6db31e0f953a7ae178113e4348b5c3d65d6f95b.

Covered:

• header matching for all six comment syntaxes;
• shebang, encoding, and front-matter offsets;
• escaping of regex metacharacters in prefixes (*, ..);
• the automatic exemptions (symlink, empty, NUL/invalid UTF-8, suffixes);
• all three stale-entry states, including a listed file deleted only from the working tree;
• git ls-files --cached --others --exclude-standard with the current .gitignore (.venv, .tox, caches, htmlcov, .env are ignored);
• gitlinks and directories are skipped by is_file();
• CRLF working trees: Path.read_text uses universal newlines.

Tests exercise both acceptance and rejection, and each problem message.
Before the review, manually injected failures (an unheaded new file, a missing listed file) failed as expected, and the fixes for escaping and missing listed files were validated then.

[laughingman7743]

Self-review round two (claims and operational behavior): FINDINGS, repaired in eaaab65

Base 82e88f4ebbd89a4dad8b3b57ac412012601f0c45, head f6db31e0f953a7ae178113e4348b5c3d65d6f95b.

Claims checked:

• The Test workflow skips fork PRs and docs/**.md-only PRs: confirmed from test-suite.yaml (if: on the head repo) and test.yaml (paths-ignore).
• 77 listed files: confirmed with len(UNHEADED_FILES).
• 39 new tests and 68 script tests: confirmed from the pytest output.
• The contributing guide's placement wording matches the checker.

Findings and repairs:

1. This README said all script tests use botocore stubs and did not mention the new check. Both are reworded here.
2. The PR body said the tox runs execute just lint. Only the pyathena env does (just test pyathena depends on lint; sqla does not). The PR body is corrected.
3. Operational effect: locally, untracked, non-ignored files are now checked by just lint. This is stated in the PR body.
4. A new format with a different comment syntax needs a HEADER_BLOCKS entry. A comment was added above HEADER_BLOCKS.

Revalidated at eaaab65 with just scripts (68 passed) and markdownlint.

[laughingman7743]

Independent review (relayed): FINDINGS, repaired in 4bbc563

• Reviewer: OpenAI Codex CLI 0.156.0 (codex exec --sandbox read-only, reasoning effort high), session 01a0d0de-9035-71d3-a719-00186dc07d5e.
• Scope: static review of 82e88f4ebbd89a4dad8b3b57ac412012601f0c45..eaaab658ee265a4f92084ca0c9310b8421c141e8 in a detached snapshot, without the PR number, description, or prior findings. The reviewer confirmed that the 77-entry UNHEADED_FILES baseline matches the current non-exempt unheaded files.

Findings and dispositions:

1. There was no permitted path for a new third-party/generated file that cannot carry the header. Fixed: the UNHEADED_FILES comment and docs/contributing.md now say an entry is added only as agreed in the issue.
2. Header syntax was accepted regardless of format, so six # lines in Markdown passed while rendering as headings. Fixed: SUFFIX_BLOCKS selects each format's syntax, and tests reject # in .md, # in .rst, and HTML in .py.
3. A YAML-comment header after keys inside front matter was rejected, contrary to docs/contributing.md. Fixed: _front_matter_header accepts it anywhere inside the front matter, or in the file's syntax directly after the closing ---.
4. ASCII-only binary formats (for example PDF) weren't detected as binary. Rejected: no such file exists; a new binary format gets its suffix added when the file is proposed.
5. stdout.strip() broke repository paths ending in whitespace, and strict UTF-8 decoding broke non-UTF-8 file names. Fixed: rstrip("\n") and os.fsdecode, with a test for a non-ASCII name with a trailing space.

Both self-review perspectives were applied to the repair:

• just scripts: 78 passed.
• The checker exits 0 on the tree.
• The PR body is updated.

An independent follow-up review of eaaab65..4bbc563 is pending.

[laughingman7743]

Independent follow-up reviews (relayed)

Follow-up 1 — Codex CLI 0.156.0, read-only, effort high, session 01a0d0eb-0ee8-7101-b9a8-7a2d1c58dd73. Static review of eaaab65..4bbc563 at the snapshot 4bbc563ed4526a249e435c41d910d61233e7d59e. Result: FINDINGS, all fixed in fa99986594d7f00fbc82a7738880f1e171c5fc8d.

1. .html accepted only Jinja comments. .html now also accepts HTML comments.
2. In multi-document YAML, the second --- was treated as the end of front matter, so a header only in a later document passed. Front-matter handling is now limited to .md; other files accept the header only directly after a leading document marker. Tests were added.
3. The root path was decoded as text with rstrip("\n"). It is now decoded from bytes with removesuffix(b"\n") and os.fsdecode.
4. docs/contributing.md conflicted with the automatic exemptions. The guide now names the exemptions and says only non-exempt unheaded files are listed.

Repair validation: just scripts (83 passed), the checker exits 0 on the tree, and markdownlint passes. The PR body is updated.

Follow-up 2 — same reviewer configuration, session 01a0d0f4-6a44-7d81-aeb4-2440328fc7ae, static review of 4bbc563..fa99986 at fa99986594d7f00fbc82a7738880f1e171c5fc8d. Result: CLEAN. All four findings are resolved, with no regression for current-tree formats or tests, and the docs match the behavior.

The two non-blocking suggestions are deferred:

• A test for repository roots containing a newline or non-UTF-8 bytes (the fix is one expression, and such roots are impractical).
• Stating that Markdown front matter needs a closing --- (front matter without one isn't front matter).

[laughingman7743]

Maintainer-requested change and independent follow-up 3 (relayed)

At the maintainer's request, 9315fb63f05c74431c5fa793176fe8cb80d6bd47:

• moves the exempt suffixes and the unheaded file list to scripts/config/license_headers.toml, read with tomllib (tomli on Python 3.10, now an explicit dev dependency);
• runs the checker through a just license-headers recipe that lint depends on. The License Headers workflow now sets up uv and just and runs the recipe.

Validation:

• just scripts: 88 passed.
• Python 3.10: the checker exits 0 and the 59 checker tests pass.
• Injected failures report the new messages.

Both self-review perspectives were applied, and the PR body and docs/contributing.md are updated.

Follow-up 3 — Codex CLI 0.156.0, read-only, effort high, session 01a0d374-3fd9-7531-8f64-dfd46ccebff3. Static review of fa99986..9315fb6 at 9315fb6. Result: CLEAN.

• The migrated entries and suffixes match the prior constants.
• Invalid, malformed, or missing config fails explicitly.
• The Python 3.10 path is covered by the marked tomli dependency.
• just lint reaches the checker first.
• The workflow actions are SHA-pinned with read-only permissions.

Non-blocking suggestion, deferred: pin an interpreter in the License Headers workflow. uv currently selects the runner's Python, which satisfies requires-python, and uv run installs the default dev dependencies.

[laughingman7743]

CI repair and independent follow-up 4 (relayed)

• CI failure at 9315fb6: the Benchmark tooling offline job failed in uv sync --project benchmarks --locked. benchmarks/uv.lock also records the root project's dev group, so it needed the new tomli entry.
• Repair: 1e9787a7210c455891013e3e182789c2b5f6e0e1 regenerates it with just benchmark lock. just benchmark test passes (80 passed, 1 skipped), and the Benchmark tooling workflow now passes.

Follow-up 4 — Codex CLI 0.156.0, read-only, effort high, session 01a0d377-d150-79f3-9475-f7e15b1259ee. Static review of 9315fb6..1e9787a. Result: CLEAN.

• The change adds only the root dev dependency's metadata entry, matching pyproject.toml and the root lock.
• The benchmark project targets Python 3.12, so no resolved tomli package entry is needed.
• No other lock file, workflow, or recipe needs a matching change.

[laughingman7743]

Rebase onto master after #811 and #813

026587bc98289fdd66ade30d40104102c780066f, rebased from 1e9787a.

git range-diff 82e88f4..1e9787a origin/master..026587b:

• The five checker commits are identical (=).
• The benchmark-lock-only commit is dropped. Manage the benchmark project as a uv workspace member #813 removed benchmarks/uv.lock and made benchmarks a member of the uv workspace, so the root uv.lock now carries the tomli entry.

Upstream effects checked:

• just license-headers runs uv run python, which syncs only the root project.
• Rerun tests once on Athena service-side query failures #811 changes only the test recipes.

Validation:

• uv lock --check passes.
• just license-headers exits 0 on the rebased tree.
• just scripts: 88 passed.
• just benchmark test: 80 passed, 1 skipped.

No new independent review: the patches are unchanged.