Deferred from #329 to keep that PR reviewable.
What
The Hugo site under site/ has no llms.txt or llms-full.txt (verified: no llms*.txt anywhere in the tree). An llms.txt gives LLM-based tools a curated, plain-text index of the docs corpus instead of leaving them to scrape rendered HTML.
Suggested approach
A Hugo custom output format is the clean way to do this:
- media type
text/plain, isPlainText: true, baseName: llms
- walk the docs corpus and emit a curated index; optionally a fuller
llms-full.txt
Why it should be generated, not hand-written
site/content/source/ is already generated by site/scripts/sync_source_docs.py, and hugo-site CI runs sync_source_docs.py --check to fail on drift. A hand-maintained llms.txt would drift with nothing to catch it. Either generate it through the Hugo output format from the same corpus, or extend sync_source_docs.py so the existing --check gate covers it.
Constraints worth respecting
site/README.md notes the hosted Pages site deploys from main, not dev.- The
secret-scan llm-model-names gate scans *.md/*.yml/*.json; whatever generates this must not emit specific model identifiers.
Deferred from #329 to keep that PR reviewable.
What
The Hugo site under
site/has nollms.txtorllms-full.txt(verified: nollms*.txtanywhere in the tree). Anllms.txtgives LLM-based tools a curated, plain-text index of the docs corpus instead of leaving them to scrape rendered HTML.Suggested approach
A Hugo custom output format is the clean way to do this:
text/plain,isPlainText: true,baseName: llmsllms-full.txtWhy it should be generated, not hand-written
site/content/source/is already generated bysite/scripts/sync_source_docs.py, andhugo-siteCI runssync_source_docs.py --checkto fail on drift. A hand-maintainedllms.txtwould drift with nothing to catch it. Either generate it through the Hugo output format from the same corpus, or extendsync_source_docs.pyso the existing--checkgate covers it.Constraints worth respecting
site/README.mdnotes the hosted Pages site deploys frommain, notdev.secret-scanllm-model-namesgate scans*.md/*.yml/*.json; whatever generates this must not emit specific model identifiers.