feat(api): fetch full JDs at discovery through SSRF-hardened fetcher + extend local prerank (#263)

[Taleef7]

Closes #263 (Epic #245 — Feed Engine).

Summary of Changes

1. SSRF-Hardened JD Upgrade (apps/api/src/lib/jd-upgrade.ts):

   • upgradeToFullJd(job, deps) fetches job pages through the existing SSRF-hardened fetchJobPage (DNS pinning, 8s timeout, 2MB limit, max 3 redirects).
   • Upgrades snippet descriptions when extracted full text is >= 600 characters and > 1.5x the snippet length.
   • Updates only descriptionText; preserves title, company, URL, and all other metadata.
   • Fails closed on any fetch failure or block without throwing.

2. Discovery Pipeline Integration (apps/api/src/lib/discovery.ts & apps/api/src/routes/discovery.ts):

   • Injects upgradeJd?: typeof upgradeToFullJd into DiscoveryDeps.
   • Bounded outbound JD fetches capped per run via JD_FETCH_CAP = Number(process.env.DISCOVERY_JD_FETCH_CAP ?? 25).
   • Upgrades descriptions before createJob so that salary parsing, seniority inference, content hashing, and local prerank operate on the full text.
   • Injected upgradeJd: upgradeToFullJd into routes/discovery.ts default dependencies.

3. Keyword Catalog Widening (apps/api/src/lib/analysis-core.ts):

   • Widened keywordCatalog from 20 to 75 unique technologies (Docker, Kubernetes, Terraform, AWS, Azure, GCP, GraphQL, Kafka, etc.).
   • Preserves original 20 catalog entries in order.
   • Preserves MIN_EVIDENCE_SKILLS = 3 and the null-score contract in local-fit.ts.

4. Testing & Verification:

   • Unit tests in jd-upgrade.test.ts (short JD upgrade, skipping >=600 chars, skipping missing URL, failing closed on blocked fetch, refusing < 1.5x expansion).
   • Integration tests in discovery.test.ts (fetch cap bounding at 25, prerank scoring on upgraded text).
   • Fit tests in local-fit.test.ts (full JD clearing evidence floor) and catalog uniqueness test in analysis-core.test.ts.
   • 345 API unit tests pass. Full npm run check (monorepo lint + typecheck + Next.js build + API build) clean.

[Taleef7]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 42c6c85b7c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".