[Feature]: Add hybrid semantic search to project knowledge

[roomote-community]

Problem

Full-text search cannot reliably retrieve conceptually related code, plans, decisions, and history when terminology differs.

Proposed behavior

Add embedding provider/model/dimension configuration, portable model-versioned canonical vectors, hybrid lexical/vector ranking, bounded exact-vector fallback, and an optional capability-detected pgvector projection. Expose semantic coverage, cost, freshness, and degraded status.

Reuse the inference gateway embedding endpoints and accounting plus the canonical knowledge chunks and authorization from lexical retrieval.

Acceptance criteria

• Missing embedding configuration leaves lexical retrieval functional.
• Model migration serves the previous complete index until replacement is ready.
• pgvector absence never breaks Roomote.
• Semantic retrieval cannot bypass project boundaries or permissions.
• Evaluations improve conceptual retrieval without weakening citations.

Dependency

• [Feature]: Add deterministic project knowledge boundaries and cited full-text search #1048

[roomote-community]

I've analyzed this issue and here's my proposed implementation plan:

1. Implement this after [Feature]: Add deterministic project knowledge boundaries and cited full-text search #1048 and reuse its authorized, non-tombstoned canonical chunks, immutable citations, policy versions, and lexical query. Semantic search must not introduce a separate source boundary or authorization path.
2. Add shared embedding configuration and status types in packages/types/src/project-knowledge.ts. Persist an admin-managed deployment configuration containing provider, model, expected dimension, batching limits, and exact-fallback cap; validate the provider against the embedding routes already described by packages/types/src/inference-gateway.ts. No configuration must mean lexical-only operation, not an error.
3. Extend packages/db/src/schema.ts additively with model-versioned embedding indexes and chunk embeddings. Store a portable canonical real[] vector plus provider/model/dimension, content hash, policy version, build state, timestamps, and failure metadata. Add a nullable project relation to llm_usage_events so existing usageType: 'embedding' accounting can report project indexing and query costs through recordLlmUsage.
4. Treat each provider/model/dimension/content-normalization combination as an immutable index version. Configuration changes create a replacement in building state while searches continue using the previous complete version; activate the replacement atomically only after every currently eligible chunk is embedded or explicitly accounted for, then retire old versions according to retention policy.
5. Extract reusable control-plane provider resolution from apps/api/src/handlers/inference/registry.ts into a server-only inference helper that both the API gateway and BullMQ jobs can call. Batch embedding requests through the existing allowlisted provider endpoints, resolve encrypted deployment credentials with resolveModelProviderEnvValue, validate response dimensions, use deterministic usage event keys, and record provider/model, tokens, cost when available, latency, and failures as embedding usage.
6. Add a dedicated BullMQ embedding/reconciliation queue alongside apps/bullmq/src/snapshot-queue.ts. Enqueue idempotently from [Feature]: Add deterministic project knowledge boundaries and cited full-text search #1048 ingestion, re-embed only changed content hashes, re-check the active policy and authorization before every persisted batch, discard stale jobs, remove or tombstone vectors with their chunks, retry transient provider failures with bounded concurrency, and persist progress so interrupted migrations resume safely.
7. Implement hybrid retrieval in the package-owned [Feature]: Add deterministic project knowledge boundaries and cited full-text search #1048 search helper. Build the eligible-chunks relation first, compute lexical and semantic candidate lists only inside that relation, fuse them with deterministic reciprocal-rank fusion and stable tie-breakers, and return the original immutable citation unchanged. If query embedding or semantic indexing is unavailable, transparently return lexical results with degraded metadata.
8. Keep canonical vectors independent of pgvector. Probe to_regtype('vector') at runtime and, only when available, maintain a rebuildable pgvector projection and dimension-specific similarity index for the active model version. Otherwise fetch at most the configured authorized candidate cap and calculate exact cosine similarity from canonical vectors in the server helper; never install the extension automatically or make startup, migrations, ingestion, or search depend on it.
9. Extend the focused project-knowledge tRPC/MCP surfaces planned in [Feature]: Add deterministic project knowledge boundaries and cited full-text search #1048 to return retrieval mode, active/building model versions, semantic coverage, last successful indexing time, backlog/failure counts, degraded reason, and aggregated embedding cost. Add admin controls to validate/save configuration and trigger or retry a rebuild without exposing provider credentials or accepting caller-supplied project grants.
10. Add real-Postgres coverage for lexical-only operation, project and permission isolation before ranking, tombstones, deterministic hybrid ordering, fallback caps, malformed/dimension-mismatched responses, partial and resumed builds, atomic model cutover, concurrent configuration changes, and equivalent results with and without pgvector. Add gateway/accounting and BullMQ retry/idempotency tests plus an evaluation fixture set containing terminology-mismatched queries; require improved conceptual recall while preserving citation validity and lexical baseline behavior.
11. Document embedding configuration, model migration, cost and freshness semantics, degraded behavior, exact-fallback limits, and optional pgvector acceleration in apps/docs/, including that standard PostgreSQL remains a complete supported deployment.

This approach keeps #1048's authorization and citation guarantees authoritative, makes vectors portable and safely replaceable, and treats pgvector as an acceleration layer rather than a runtime dependency.

Please tag @roomote if you'd like me to implement this, or reply with feedback on the plan.