Docs build (build-book / pydoc2json) broken on main since lazy-loading PR #2456

[romanlutz]

Summary

The build-book workflow's Build latest job has been failing on every run against main since PR #2456 ("PERF: Lazily load all package exports") merged.

Evidence

• docs.yml's latest matrix entry checks out ref: main (see .github/docs-versions.yml), so this job always builds the tip of main, independent of whatever PR triggered the workflow.
• Recent build-book runs on main (branch push trigger):
  • FIX bound OpenAI Realtime completion grace (#2420) — success (17:45:31)
  • PERF: Lazily load all package exports (#2456) — first failure (17:51:29, right after PERF: Lazily load all package exports #2456 merged)
  • All subsequent runs on main — failing the same way
• Failure is in the "Generate API reference (pydoc2json + gen_api_md)" step:

  Written to doc/_api/pyrit_all.json (27 members)
  ...
  Written doc/api/pyrit.md (2 members)
  VALIDATION ERRORS FOUND:
    • [myst.yml] File referenced in myst.yml TOC not found: 'api/pyrit_converter.md'
    • [myst.yml] File referenced in myst.yml TOC not found: 'api/pyrit_memory.md'
    ... (one per top-level subpackage)
    • [orphaned] File exists but not in myst.yml: api/pyrit.md
  

  i.e. pydoc2json pyrit --submodules now only discovers pyrit's own 2 lazy __all__ exports instead of walking into all subpackages (pyrit.converter, pyrit.memory, pyrit.executor, etc.), so almost none of the expected api/pyrit_*.md pages get generated.

Likely root cause

PR #2456 replaced eager submodule imports / a static pyrit/__init__.py with a lazy-loading scheme (_LazyPyRITModule, __getattr__, __dir__ driven by a small _LAZY_EXPORTS dict). build_scripts/pydoc2json.py --submodules apparently relies on dir(pyrit) / pyrit.__all__ (or similar static introspection) to enumerate submodules to recurse into, and that mechanism no longer surfaces the real subpackages since they're no longer eagerly imported/registered as attributes.

Suggested fix

Update build_scripts/pydoc2json.py's submodule discovery to use pkgutil.walk_packages/iter_modules against pyrit.__path__ (filesystem-based discovery) rather than relying on dir()/__all__ of the lazily-loaded pyrit package, so it's compatible with the new lazy-loading __init__.py.

Impact

• build-book fails on every push to main and on every PR that touches .github/workflows/docs.yml, .github/docs-versions.yml, build_scripts/**, doc/**, or pyrit/** (i.e. most PRs), because the latest build always builds main, which is currently broken.
• Not caused by, and not fixable within, PRs that don't touch pyrit/__init__.py/build_scripts/pydoc2json.py — it's a standing regression on main.