[Docs]: Missing Documentation for `INTERNAL_RAG_TOKEN` Protected Endpoint Model

[Srejoye]

Summary

Description

rag-service/main.py defines two protection mechanisms — an exact-match set (PROTECTED_RAG_PATHS) and a prefix set (PROTECTED_RAG_PREFIXES) — that gate service-to-service access via the X-Internal-Token header. However, neither README.md, .env.example, nor any inline documentation explains:

• Which endpoints are protected and why the split between exact-match and prefix exists.
• What happens when INTERNAL_RAG_TOKEN is absent (503 is returned, but this is only discoverable by reading the middleware source).
• That the RAG service must not be exposed on a public port — the middleware comment says "when port 5000 is accidentally exposed" but no deployment guide reinforces this.
• That adding a new route under /ask/ is automatically protected, while a new top-level route (e.g. /ask-v2) is not — operators extending the API have no way to discover this without reading internal_auth_middleware.

docker-compose.yml does not bind the RAG service port to localhost only, so a developer standing up the stack locally exposes port 5000 publicly on their machine with no documentation warning.

Affected Files

• README.md — no section on service-to-service auth or port exposure
• .env.example — INTERNAL_RAG_TOKEN listed but not explained
• rag-service/main.py lines 556–603 (PROTECTED_RAG_PATHS, PROTECTED_RAG_PREFIXES, internal_auth_middleware)
• docker-compose.yml — RAG service port binding

Steps to reproduce

N/A

Expected behavior

N/A

Actual behavior

N/A

Additional context

No response

[Srejoye]

Hi — requesting GSSoC'26 assignment.

Technical understanding: The two-tier protection model (exact set + prefix set) was introduced to auto-protect future sub-routes under /ask/ without requiring code changes. This is a sound design but completely implicit. Operators adding routes, writing deployment configs, or setting up firewalls have no documented contract to rely on. The 503 behaviour on missing token is especially surprising — it looks like a service outage rather than a misconfiguration.

Affected modules/files:

• README.md — needs a "Security / Deployment" section
• .env.example — INTERNAL_RAG_TOKEN entry needs an inline comment
• docker-compose.yml — RAG service port binding needs a comment or localhost restriction

Proposed documentation additions:

1. README.md: Add a "Service-to-Service Auth" section listing all protected paths, explaining the exact-match vs prefix split, documenting the 503 behaviour, and explicitly warning that the RAG service port must not be publicly reachable.
2. .env.example: Expand the INTERNAL_RAG_TOKEN entry with a comment: # Required. Shared secret between Express gateway and RAG service. RAG service returns 503 on all protected endpoints until this is set.
3. docker-compose.yml: Add a comment on the RAG service port binding warning against exposing it externally, or change the binding to 127.0.0.1:5000:5000.

Regression/unit test considerations: No code changes — documentation only. A CI lint step (e.g. a shell script that greps README.md for INTERNAL_RAG_TOKEN) could act as a guardrail against future documentation drift.

[garima-agarwall]

Hii @FireFistisDead
I would like to work on this issue.
Please assign this issue to me if possible.