Discovery silently drops first-party source: extensionless shebang'd files are never scanned

[gadievron]

Summary

Two rules in parsers/python/repository_scanner.py drop first-party source before any analysis runs.
Because discovery precedes seeding and reachability, anything lost here is lost from every downstream
stage — and in neither case does the artifact record what was dropped.

Both are reproduced below against the real parse_repository() at b5019628, each in an isolated
fixture with a control differing in exactly one variable.

---

1. Extension-only discovery skips shebang'd executables

repository_scanner.py:139-142:

    def is_source_file(self, file_name: str) -> bool:
        """Check if a file is a Python source file."""
        ext = os.path.splitext(file_name)[1].lower()
        return ext in self.source_extensions

For an extensionless file ext is "", so the file is never considered. There is no shebang,
magic-byte, mimetype or exec-bit fallback anywhere in the scanner — the only st_mode use (:285)
is S_ISDIR/S_ISREG.

Fixture — two byte-identical Python files (cmp reports IDENTICAL), one extensionless with a shebang
and mode 0755, one with a .py extension:

$ file tool_shebang
Python script text executable, ASCII text

  Found 1 Python files
  Total units: 1

Only tool_twin.py produces units. The skip is uncounted: :309-310 is a bare
if not self.is_source_file(entry.name): continue, so nothing records that a file was passed over.

2. A first-party package named build/ is excluded as build output

RETRACTED 2026-08-21 — this half re-litigates a documented deliberate decision. Please disregard
it; only defect (1) above stands.

My provenance argument was git log -S "'build'," -- parsers/python/repository_scanner.py
returning only the initial commit. That is a single-file text match, and it missed where the policy
actually lives:

• config/languages.json carries build in skip_dirs — the exclusion is registry-level, not a
  stray literal in one scanner.
• core/parser_adapter.py:49-52 documents the at-any-depth behaviour as intentional and as a
  deliberate reconciliation with the Go detector: "Directories named in skip_dirs are PRUNED
  rather than filtered per-file. This matches the Go detector's filepath.SkipDir semantics exactly
  (the two implementations previously disagreed on what 'skip' meant)".
• utilities/path_filters.py — the very file I nominated as the fix's home — codifies the contract
  with a worked example asserting should_exclude_directory("src/build/x.rb", {"build"}) -> True.

So pruning by directory name at any depth is the specified behaviour, agreed across two
implementations and pinned by a test. Asking for it to be narrowed is a design change, not a defect
report, and I did not present evidence that meets that bar. The fixture below reproduces, but it
demonstrates intended behaviour.

Leaving the original text below unedited so the thread reads honestly.

repository_scanner.py:80 lists 'build' among the default directory exclusions, applied at :129
by bare directory name:

        if dir_name in self.exclude_patterns:

Applied at :291 against entry.name, so it matches the segment anywhere in the tree, not only
at the repository root. The same list bare-name excludes env (:74), dist (:79) and
migrations (:90).

Fixture — core/build/mod.py and core/built/mod.py, byte-identical (cmp IDENTICAL), differing
only in the parent directory name:

  Found 3 Python files
  Total units: 1

The three found files are the __init__.py files plus built/mod.py; the entire core/build/
subtree is absent — nested at depth, with no build-output evidence consulted.

Intent versus effect: every neighbour in that list (node_modules, site-packages, dist,
egg-info, .eggs, .tox, .nox, virtualenv) is a generated-artifact directory, so the intent is
plainly to skip build output. The matcher does not check whether the directory is build output —
only whether it is named like one.

A second copy of the list exists at :426-433, with 'build' at :429, used as the CLI
--exclude merge default. A fix at :80 alone would leave that path unchanged.

Provenance: git log -S "'build'," -- libs/openant-core/parsers/python/repository_scanner.py returns
only 0d729f6 (initial commit), so this has not been revisited.

---

What the artifacts record

Defect (1) is entirely uncounted. Defect (2) is counted — directories_excluded (:118,
incremented :292, in the returned schema :350) and printed to stderr at :451:

            print(f"Directories excluded: {result['statistics']['directories_excluded']}", file=sys.stderr)

But the count names no directory, so an excluded first-party package is indistinguishable from an
excluded node_modules. The figure does reach disk — parse_repository:153-155 writes the stats
block to scan_result.json, and the real run carries 'directories_excluded': 6 there — but it has
no reader: nothing in core/reporter.py or the deliverables consumes it.

This is the same reasoning the project already applied to symlinks. _note_symlink's docstring
(the quoted sentences are :167-170, within the docstring at :165-172):

Symlinks are refused by policy, which means code reachable only that way is not scanned. Folding
the count into directories_excluded (as this did) hid it among ordinary prunes like node_modules
— an unscanned path that leaves no distinguishable trace is a silent false negative.

Symlinks were given their own key (symlinks_skipped) with examples, mirrored across all five
scanners, for exactly this reason. Extension-skipped files and name-excluded first-party directories
are in the same position and have not had the same treatment.

Suggested fix

For (1):

1. Fall back to a shebang check when the extension is empty — read the first line and match
   #!.*python. Cheap and precise.
2. Count what is skipped, in the shape _note_symlink established: its own stats key plus a few
   example paths.

For (2):
3. Require corroboration before excluding a directory named like build output — e.g. skip build/
only when it contains no __init__.py, or only at the repository root, or when a sibling
setup.py/pyproject.toml implies it is a build target.
4. Record excluded directory names (or examples), not only a count, so a first-party exclusion is
distinguishable from node_modules.
5. Apply any change to both copies of the list — :80 and :426-433.

For both:
6. Forward the discovery-stage counters through parse_repository so they reach the parse step
report; today they stop at stderr.

Relationship to existing work

Issue #295 references defect (1) in passing — its "what I am not claiming" section cites a
base-rate control showing missing shebang-launcher discovery "accounts for only ~4.1% recovery" on
one run. That is a magnitude for the discovery gap, not a filing of it; the mechanism itself is
unfiled. I have not re-measured that figure.

Checked and clear: #94 is context/application_context.py::detect_entry_points() — a different
file and function, closed-unmerged, and its body preserves the dist/build exclusion rather than
questioning it. #90 touches function_extractor.py. #192/#184 harden the zig and ruby
scanners. #164 is ruby+zig but adds utilities/path_filters.py, which may be the natural home
for a shared fix.

What I am not claiming

• Not claiming any specific vulnerability was missed. The evidence is scanner behaviour on
  fixtures executed at b5019628.
• Not claiming excluding real build output is wrong — it is deliberate and sensible. The claim is
  that the test is the directory's name, at any depth, with no check that it is generated.
• Not claiming a magnitude. The ~4.1% above is Python call graph records no edge for a function referenced as a dict value or list element, so dispatch-table targets are pruned with their dispatcher kept #295's figure for one run, quoted, not re-derived.
• A missed CLI entry point would also be a missed BFS root, which compounds; ENTRY_POINT_TYPES
  does carry cli_handler and main. But the Python extractor emits no cli_handler, and these
  fixtures demonstrate discovery loss only, not seeding loss.
• Fixing either will increase analysed units and cost. For a scanner that is the conservative
  direction, but it is a real trade-off.