Add HNSW indexes for scalable vector similarity search

[dhruvi-16-me]

Overview

This PR improves the performance of semantic retrieval using HNSW (Hierarchical Navigable Small World) indexes provided by pgvector.

Previous PR established the embedding infrastructure by generating and storing embeddings for meetings, tasks, and tickets. While semantic search was already functional, vector similarity queries would become increasingly expensive as the dataset grows because PostgreSQL would need to compare the query embedding against every stored embedding.

This PR introduces HNSW indexes to enable efficient Approximate Nearest Neighbor (ANN) search without changing the existing retrieval logic or application behavior.

Motivation

Before this change, vector similarity queries relied on brute-force comparisons over all available embeddings.

Although this approach is acceptable for small datasets, it does not scale well as more meetings, tasks, and tickets are added.

By introducing HNSW indexes, PostgreSQL can efficiently locate embeddings that are close to the query vector instead of comparing against every row.

This optimization is purely infrastructural and prepares the semantic search layer for larger datasets while preserving the current API and retrieval behavior.

Changes

1. Added HNSW vector indexes

Created HNSW indexes for embedding columns using cosine similarity (vector_cosine_ops).

Indexed columns:

• meetings.summary_embedding
• tasks.description_embedding
• tickets.description_embedding

These indexes optimize existing vector similarity queries without modifying query semantics.

2. Benchmark script

Added:

scripts/benchmark_hnsw_vector_search.sql

This script contains reusable benchmark queries using:

EXPLAIN (ANALYZE, BUFFERS)

to inspect execution plans and measure retrieval performance for:

• Meetings
• Tasks
• Tickets

The benchmark allows developers to verify query planning and evaluate vector search performance as datasets grow.

---

3. Verification script

Added:

scripts/verify_hnsw_indexes.sql

This script verifies:

• pgvector extension is installed
• HNSW indexes exist
• Index definitions are correct
• PostgreSQL execution plans can utilize the indexes

This provides a repeatable post-deployment validation process.

Screenshots

Verified creation of HNSW indexes. The query confirms that HNSW indexes were successfully created on the embedding columns used for semantic retrieval (summary_embedding for meetings and description_embedding for tasks and tickets). These indexes enable PostgreSQL to perform efficient Approximate Nearest Neighbor (ANN) searches as the dataset grows.

Verified pgvector installation. This confirms that the pgvector extension is available in the database, which is required for storing embedding vectors and creating HNSW indexes.

Benchmark executed successfully. The benchmark queries completed successfully, confirming that vector similarity searches execute correctly after the migration. On the current development dataset, PostgreSQL selected a Sequential Scan because the tables contain very few embedded rows. This is expected planner behavior; HNSW indexes become beneficial as the dataset grows.

What did NOT change

This PR intentionally does not modify retrieval behavior
The only change is how PostgreSQL internally executes existing vector similarity queries.

Validation

The following validation steps were performed:

• Applied the migration successfully.
• Verified that the pgvector extension is available.
• Verified creation of HNSW indexes.
• Executed benchmark queries using EXPLAIN (ANALYZE, BUFFERS).
• Confirmed existing embedding generation remains unaffected.
• Confirmed no application or API behavior changed.

During benchmarking, PostgreSQL may choose a Sequential Scan on very small datasets. This is expected planner behavior and does not indicate a problem with the indexes. As the number of embedded rows increases, PostgreSQL will begin favoring HNSW index scans where they provide a lower estimated execution cost.

Impact

This PR improves the scalability of semantic retrieval while keeping existing functionality unchanged.

Before

• Semantic search compared query embeddings against every stored embedding.

After

• PostgreSQL can leverage HNSW indexes to perform efficient Approximate Nearest Neighbor search as the dataset grows.

From the application's perspective, the returned results remain the same; only the database execution strategy becomes more efficient.

✅ Checklist

• I have read the contributing guidelines.
• I have added tests that prove my fix is effective or that my feature works.
• I have added necessary documentation (if applicable).
• Any dependent changes have been merged and published in downstream modules.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

• main
• develop

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 87741375-706f-483c-825a-7e2ce05603dc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}