Add Amazon DynamoDB vector search example - #3003
Conversation
Getting-started notebook covering table creation with vector indexes, loading the shared Wikipedia embeddings dataset, SearchVectors queries with OpenAI embeddings, mandatory tenant scoping via HASH search schema elements, inline filters, and grounded completion from retrieved context. All cells executed against Amazon DynamoDB and the OpenAI API. Includes an optional docker-compose local backend.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 793c3f6cad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "\n", | ||
| "Real applications rarely search a whole table. A `SearchSchema` on the vector index declares attributes you can constrain at query time:\n", | ||
| "\n", | ||
| "- `HASH` elements partition the index. Scoping on them becomes **mandatory**: the service rejects any search that does not pin them, so cross-tenant reads are impossible to express rather than merely discouraged.\n", |
There was a problem hiding this comment.
When a multi-tenant app lets callers influence the tenant value, this is not an isolation boundary: the AWS DynamoDB vector-search docs say a vector index partition key is for data locality/performance and that any principal with dynamodb:SearchVectors can search any partition key value. This text teaches readers that cross-tenant reads are impossible to express, so they could omit real authorization checks; describe the HASH element as mandatory query scoping and still require tenant authorization separately. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/VectorSearchWorkingWith.html
Useful? React with 👍 / 👎.
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "! pip install \"boto3>=1.43.64\" openai pandas numpy wget" |
There was a problem hiding this comment.
This install line can leave an existing openai 1.x package in place, and requirements.txt also allows openai>=1.0.0, but the notebook later calls openai_client.responses.create. The v1.0.0 OpenAI Python client exposes resources like chat and embeddings but not responses, so users running under a resolver/constraint that selects the documented minimum will hit AttributeError in the final RAG cell; require a Responses-capable SDK version, such as openai>=1.66.0, everywhere the notebook documents installation. https://raw.githubusercontent.com/openai/openai-python/v1.0.0/src/openai/_client.py
Useful? React with 👍 / 👎.
| | Index declaration | `VectorIndexes` on `CreateTable` or `UpdateTable` | | ||
| | Distance functions | `COSINE`, `DOT_PRODUCT`, `EUCLIDEAN` | | ||
| | Query API | `SearchVectors` with `SearchVector`, `TopK`, optional `SearchConditionExpression` | | ||
| | Scoring | `Score` is the distance: lower is more similar | |
There was a problem hiding this comment.
Because the table lists DOT_PRODUCT as a supported distance function two rows above, this scoring summary is wrong for that option: DynamoDB reports lower scores as more similar for COSINE/EUCLIDEAN, but higher scores as more similar for DOT_PRODUCT. Readers who switch the sample index to dot product would invert result interpretation or evaluation if they follow this README; qualify the row by distance function. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/VectorSearchWorkingWith.html
Useful? React with 👍 / 👎.
…PRODUCT scoring Search schemas are mandatory query scoping, not an authorization boundary; callers with SearchVectors permission can search any partition value, so the notebook and README now say to enforce tenant access control with IAM or the application layer. The openai floor is raised to 1.66.0 (first release with the Responses API used in the final cell). The README scoring row is qualified per distance function: DOT_PRODUCT reports higher as more similar.
|
Addressed all three review findings in 4ba282f:
|
1 participant
Summary
This PR adds a new vector database example for Amazon DynamoDB under
examples/vector_databases/dynamodb/. The notebook shows how to use DynamoDB vector indexes with OpenAI embeddings end to end: creating a table with a vector index, loading the shared Wikipedia embeddings dataset, running similarity searches withtext-embedding-ada-002query embeddings, using search schemas for tenant isolation and inline filtering, and feeding retrieved context to a model for a grounded answer.The notebook runs against Amazon DynamoDB by default and includes an optional local backend via a
docker-compose.yml, so readers can complete the whole walkthrough without an AWS account and move to DynamoDB by flipping one flag. All cells are executed and the committed outputs come from a real DynamoDB and OpenAI run.Motivation
DynamoDB recently added native vector search (
VectorIndexeson the table, queried withSearchVectors), and there is currently no DynamoDB example in the cookbook's vector database collection. DynamoDB is one of the most widely used operational databases, and its vector search model differs from most entries in the collection in a way that's useful to show: embeddings live in the same table as the operational data they describe, written in the samePutItemcall, with no separate cluster to manage.Two things in the notebook go beyond the standard getting-started pattern:
HASHsearch schema elements make query scoping mandatory - the service rejects unscoped searches outright rather than relying on query discipline. The notebook demonstrates the rejection and the scoped results, which matters for multi-tenant and agent-memory workloads.SearchVectors, so readers can run the full notebook without an AWS account.Disclosure: I work at AWS on DynamoDB and I maintain ExtendDB. The notebook defaults to Amazon DynamoDB; the emulator is an optional convenience for readers without an AWS account.
For new content
When contributing new content, read through our contribution guidelines, and mark the following action items as completed: