fix: prevent indexing wiki app routes

[NagariaHussain]

Summary

• add a noindex, nofollow robots meta directive to the /wiki frontend entry
• return X-Robots-Tag: noindex, nofollow for the wiki.html controller used by /wiki and /wiki/**
• remove the app entry page from generated sitemaps with sitemap = 0

This intentionally applies to the Wiki application routes only; published Wiki Document routes remain indexable.

Verification

• python -m compileall -q wiki/www/wiki.py
• git diff --check
• yarn build in frontend/
• verified GET /wiki/spaces and GET /wiki.html return X-Robots-Tag: noindex, nofollow and the matching HTML meta tag
• verified /sitemap.xml does not list /wiki

Summary by CodeRabbit

• Chores
  • Updated pages to prevent appearance in search engine results and prevent following of links by search engines.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 090d809f-f2d8-4d0a-9cbe-6c2d1177ce9b

📥 Commits

Reviewing files that changed from the base of the PR and between 30b4ac7 and 27874a7.

📒 Files selected for processing (2)

• frontend/index.html
• wiki/www/wiki.py

---

Walkthrough

This pull request adds search engine exclusion directives to prevent indexing of the wiki. The backend introduces a ROBOTS_DIRECTIVE constant set to "noindex, nofollow" in wiki/www/wiki.py and sets the X-Robots-Tag HTTP response header in the page context. The frontend reinforces this by adding a <meta name="robots" content="noindex, nofollow" /> tag to frontend/index.html. Together, these changes instruct search engines not to index the page or follow its links using both client-side meta tags and server-side HTTP headers.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix: prevent indexing wiki app routes' directly and specifically describes the main change: adding noindex/nofollow directives to prevent search engine indexing of wiki app routes.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}