feat: say how many documents were not read, and where they are

README.md

@@ -123,6 +123,11 @@ documented path.
 Every skip carries a reason and `--show-skips` prints all of them, because a checker that
 silently skips everything looks exactly like a clean one.
 
+The same rule applies one level up, to whole documents. Every run says how many
+documentation files it did not read and which directories they are in, so a clean report
+over two files in a project with three hundred cannot be mistaken for a clean report over
+three hundred. See [what gets read](#what-gets-read).
+
 ## What it checks
 
 Two things, each done properly:
@@ -383,6 +388,34 @@ Suppose your entry point is `src/pkg/main.py` and you move it.
 This README uses it twice, for exactly that reason. It was the first thing running
 `docproof` on `docproof` found.
 
+### What gets read
+
+Top-level documentation files, plus everything under `doc/` or `docs/`. Not every Markdown
+file in the tree: a fixture, a vendored README or a changelog fragment deep in a package is
+not a promise the project is making.
+
+**Every run says what that left out**, because the scope being deliberate does not make its
+silence harmless:
+
+```
+docproof 0.1.4 - myproject, 6 document(s)
+   41 documentation file(s) elsewhere in the tree were NOT read; the default scope is top-level files plus doc/ and docs/
+     guides/ 22, website/ 14, handbook/ 4, .github/ 1
+     read them too with --docs 'guides/**/*.md' or [tool.docproof] docs = ["guides/**/*.md"]
+```
+
+and when there is nothing outside the scope it says that instead, so the line is never
+missing:
+
+```
+   every documentation file in the tree was in scope
+```
+
+Only files the project **tracks in git** are counted. A gitignored tree is not your
+documentation, and the alternative measures the wrong thing badly: on one repository that
+keeps 217 cloned repositories for testing, walking the filesystem reported 38,402 unread
+documents against a true figure of 294.
+
 ### What is already skipped, and what you still have to say
 
 Some of the class above is recognised without being told. The list matters because **the

src/docproof/cli.py

@@ -18,10 +18,11 @@
     superseded_lines,
     suppressed_lines,
 )
-from .docs import find_docs, read
+from .docs import by_directory, find_docs, read, unread_documents
 from .history import classify, vanished_documents
 from .project import Project, find_root
 from .report import Report
+from .vcs import Git
 from .verifiers.base import Document, Verifier
 from .verifiers.cli_flags import DocumentedFlags
 from .verifiers.paths import DocumentedPaths
@@ -115,6 +116,45 @@ def survive_a_narrow_console() -> None:
             reconfigure(errors="backslashreplace")
 
 
+TOP_DIRECTORIES = 5
+SET_ASIDE_NAMES = 10
+
+
+def report_coverage(project: Project, unread: list[Path]) -> None:
+    """Say how much of the tree was in scope at all, whether or not any of it was missed.
+
+    **This prints on every run, including the clean one, and that is the whole point.** The
+    other skip reports in this file stay quiet when they have nothing to say, which is right
+    for them: the count of documents they set aside is visible in the header line beside the
+    count they judged. Documents that were never DISCOVERED have no such counterpart. Left
+    silent, "4 document(s)" over a tree of 168 is indistinguishable from "4 document(s)"
+    over a tree of 4, and the reader has no way to tell which one they are looking at.
+
+    The directory list is capped at five, and the line says so with the remainder counted -
+    a truncated list under an untruncated total is how a reader concludes from an absence
+    somebody else manufactured.
+    """
+    if not unread:
+        print("   every documentation file in the tree was in scope")
+        return
+    groups = by_directory(project.root, unread)
+    shown = groups[:TOP_DIRECTORIES]
+    rest = groups[TOP_DIRECTORIES:]
+    where = ", ".join(f"{name}/ {count}" for name, count in shown)
+    if rest:
+        where += f", and {len(rest)} more director{'y' if len(rest) == 1 else 'ies'} holding "
+        where += str(sum(count for _, count in rest))
+    print(
+        f"   {len(unread)} documentation file(s) elsewhere in the tree were NOT read; the "
+        f"default scope is top-level files plus doc/ and docs/"
+    )
+    print(f"     {where}")
+    print(
+        f"     read them too with --docs '{groups[0][0]}/**/*.md' or "
+        f'[tool.docproof] docs = ["{groups[0][0]}/**/*.md"]'
+    )
+
+
 def report_set_aside(historical: list[str], disclaimed: dict[tuple[str, str], list[str]]) -> None:
     """Name every document that was found and deliberately not judged, and why.
 
@@ -127,7 +167,18 @@ def report_set_aside(historical: list[str], disclaimed: dict[tuple[str, str], li
     told "No documentation found", which is the same lie in a friendlier voice.
     """
     if historical:
-        print(f"   describing the past, not judged: {', '.join(sorted(historical))}")
+        # **Capped, with the remainder counted, and that was learned from trino.** This list
+        # was unbounded, and trino keeps 330 release notes under `docs/src/main/sphinx/release/`.
+        # The run printed every one of their filenames on a single line thousands of characters
+        # long, which buried the report underneath it and told the reader nothing that the
+        # count does not. Ten names is enough to see WHAT KIND of document was set aside,
+        # which is the only question a reader has here.
+        #
+        # The remainder is stated rather than trimmed away. A truncated list under no total
+        # is how somebody concludes from an absence the tool manufactured.
+        shown = sorted(historical)[:SET_ASIDE_NAMES]
+        tail = "" if len(historical) <= SET_ASIDE_NAMES else f", and {len(historical) - SET_ASIDE_NAMES} more"
+        print(f"   describing the past, not judged ({len(historical)}): {', '.join(shown)}{tail}")
     # The directory reason is QUOTED, and it has to be. Every other skip can be audited by
     # opening the document that was skipped; this one's reason lives in a neighbouring file
     # the reader is not looking at, so a bare count would be a skip nobody could check.
@@ -151,14 +202,21 @@ def main(argv: Sequence[str] | None = None) -> int:
 
     project = Project(root=find_root(Path(args.path)))
     config = Config.from_pyproject(project.pyproject)
+    # An empty set means git could not answer, NOT that the project tracks nothing;
+    # passing it through as-is would report every document as read.
+    tracked = Git(root=project.root).tracked_files or None
 
     documents = []
     historical: list[str] = []
     # Keyed by (directory, the sentence it said), so the report can quote the reason once and
     # list the files under it rather than repeating a disclaimer thirty times.
     disclaimed: dict[tuple[str, str], list[str]] = {}
     read_directory: dict[Path, str | None] = {}
-    for path in find_docs(project.root, tuple(args.docs) + config.docs):
+    # Held rather than iterated directly, because `unread_documents` needs to know what WAS
+    # in scope. Comparing against `documents` instead would double-count: a document set
+    # aside as historical or disclaimed was found, and `report_set_aside` already names it.
+    in_scope = find_docs(project.root, tuple(args.docs) + config.docs)
+    for path in in_scope:
         relative = project.relative(path)
         if config.excludes(relative):
             continue
@@ -208,6 +266,7 @@ def main(argv: Sequence[str] | None = None) -> int:
         # found" here would be the same lie in a friendlier voice, so the reasons are
         # printed on this path exactly as they are on the ordinary one.
         print(f"docproof {__version__} — {project.root.name}, every document set aside")
+        report_coverage(project, unread_documents(project.root, in_scope, tracked))
         report_set_aside(historical, disclaimed)
         print()
         print("Nothing was judged, and each reason is above. Nothing to prove.")
@@ -233,6 +292,7 @@ def main(argv: Sequence[str] | None = None) -> int:
         outcomes.append(outcome)
     report = Report(project=project, outcomes=outcomes)
     print(f"docproof {__version__} — {project.root.name}, {len(documents)} document(s)")
+    report_coverage(project, unread_documents(project.root, in_scope, tracked))
     report_set_aside(historical, disclaimed)
     # Same principle, one level down, and it applies harder: nobody asked for this rule.
     # A `Before:` label is the tool deciding by itself that a block is not a claim, so the

src/docproof/docs.py

@@ -14,11 +14,12 @@
 
 from __future__ import annotations
 
+import os
 import re
 from bisect import bisect_right
-from collections.abc import Iterator
+from collections.abc import Iterable, Iterator
 from dataclasses import dataclass
-from pathlib import Path
+from pathlib import Path, PurePosixPath
 
 DOC_SUFFIXES = (".md", ".markdown", ".rst", ".txt")
 
@@ -120,6 +121,79 @@ def find_docs(root: Path, extra: tuple[str, ...] = ()) -> list[Path]:
     return sorted(set(found))
 
 
+def unread_documents(root: Path, scoped: Iterable[Path], tracked: frozenset[str] | None = None) -> list[Path]:
+    """Documentation files in the project that `find_docs` never offered.
+
+    **The scope above is deliberate and its silence was not.** `find_docs` reads top-level
+    files plus `doc/` and `docs/`, and the docstring there argues for why. What nothing
+    said, until this function existed, was how much of the project that leaves out. A
+    project keeping its documentation in `guides/`, `website/content/` or `handbook/` got
+    a confident clean run over the handful of files that happened to sit at the root, and
+    the report read exactly like a run that had read everything. That is the failure this
+    tool's own README already names for claims - *"a checker that silently skips
+    everything looks exactly like a clean one"* - one level up, at the document rather
+    than the claim.
+
+    **`tracked` is what makes the number mean anything, and the first version did not have
+    it.** Walking the filesystem instead, the repository this was written in service of
+    reported **38,402 unread documents**, because it keeps 217 cloned repositories under
+    `work/` for testing. Every one of them is gitignored. Filtered to what git tracks the
+    same tree reports **294**, which is the true answer: 299 documentation files, 5 in
+    scope. A directory heuristic would have needed a new exception for every project;
+    asking the index costs one `git ls-files` and is the same truth source the rest of
+    this package already trusts.
+
+    So: a documentation file the project does not TRACK is not the project's documentation.
+    That disposes of vendored trees, build output, caches and virtualenvs at once, without
+    this module holding an opinion about any of their names.
+
+    When git cannot answer - no repository, a tarball, git not installed - it falls back to
+    walking. Pruning then happens DURING the walk rather than after it, because `rglob` has
+    no way to skip a subtree, so a filter applied to its output still descends into
+    `node_modules` to produce an answer that throws the result away.
+    """
+    scoped_relative = {p.relative_to(root).as_posix() for p in scoped}
+
+    def wanted(relative: str) -> bool:
+        parts = PurePosixPath(relative).parts
+        if Path(relative).suffix.lower() not in DOC_SUFFIXES:
+            return False
+        if relative in scoped_relative:
+            return False
+        return not (SKIP_DIRS & set(parts[:-1]))
+
+    if tracked is not None:
+        return [root / rel for rel in sorted(tracked) if wanted(rel)]
+
+    found: list[Path] = []
+    for dirpath, dirnames, filenames in os.walk(root):
+        # In place, and it has to be: os.walk reads this list back to decide where to go.
+        dirnames[:] = sorted(d for d in dirnames if d not in SKIP_DIRS)
+        here = Path(dirpath)
+        for name in sorted(filenames):
+            path = here / name
+            if wanted(path.relative_to(root).as_posix()):
+                found.append(path)
+    return found
+
+
+def by_directory(root: Path, paths: Iterable[Path]) -> list[tuple[str, int]]:
+    """(top-level directory, count) for a set of paths, biggest first.
+
+    Grouped rather than listed because the useful question is *which part of my tree did
+    it not read*, and a project with three hundred unread files wants one line naming
+    `website/`, not three hundred lines. The grouping is also what makes the number
+    self-auditing: `.changeset/ 40` is instantly recognisable as fragments nobody wants
+    checked, where a bare total of 340 would look like a problem.
+    """
+    counts: dict[str, int] = {}
+    for path in paths:
+        parts = path.relative_to(root).parts
+        top = parts[0] if len(parts) > 1 else "."
+        counts[top] = counts.get(top, 0) + 1
+    return sorted(counts.items(), key=lambda item: (-item[1], item[0]))
+
+
 def read(path: Path) -> str:
     """UTF-8 with universal newlines, so a CRLF checkout reads the same as a LF one."""
     return path.read_text(encoding="utf-8", errors="replace").replace("\r\n", "\n")