Add SearchIndex and VectorSearchIndex #264

WaVEV · 2025-03-03T02:45:35Z

Search Indexes and Vector search indexes

This PR introduces new index classes to encapsulate the definitions and details of Atlas indexes.

Notes.

I have to create my own action in order to deploy MongoDB Atlas local image. Maybe we have to include in Mongo forks.

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

timgraham · 2025-03-04T02:10:34Z

django_mongodb_backend/base.py

@@ -127,6 +127,38 @@ def _isnull_operator(a, b):
        "iregex": lambda a, b: regex_match(a, b, insensitive=True),
    }

+    mongo_data_types = {


I guess this is needs to be different than DatabaseWrapper.data_types but we need a comment explaining the distinction.

Would it be better organized in the index class or do you see some future use of it in other areas? I don't want to make things too tricky, but perhaps a mapping of data_types values to these search data types would simply this a bit. (e.g. string -> string, int -> number, with some exceptions....)

I think this is only used in atlas. The types that are described in base. So I agree to move this mapping in the index class.

To the other question, it will work with something like double mapping with exceptions. 🤔 but it should be a function/method instead of a dictionaries. Let me know if you agree with a method for those mappings

If we do a double mapping, connection should be passed to the function. So, it is not bad to have a double mapping but it will cost to add a new parameter. I would like to keep this as it is, maybe the dictionary is long but the flow is more simple. (given we don't need to pass connection)

django_mongodb_backend/base.py

requirements.txt

tests/indexes_/test_atlas_indexes.py

timgraham · 2025-03-04T02:27:54Z

tests/indexes_/test_atlas_indexes.py

+class AtlasSearchIndexTests(TestCase):
+    # Schema editor is used to  create the index to test that it works.
+    # available_apps = ["indexes"]
+    available_apps = None  # could be removed?


It's not currently that helpful, but I'd include it for future possibilities. Without it, if we try to run Django's test suite without arguments, Django has to flush all test apps between each test (see 86d5885). If we added available_apps to all of Django's TestCase classes, then it would be feasible to run the Django test suite (without any test app names).

django_mongodb_backend/introspection.py

django_mongodb_backend/schema.py

timgraham · 2025-03-11T00:16:27Z

django_mongodb_backend/base.py

@@ -127,6 +127,38 @@ def _isnull_operator(a, b):
        "iregex": lambda a, b: regex_match(a, b, insensitive=True),
    }

+    mongo_data_types = {


Would it be better organized in the index class or do you see some future use of it in other areas? I don't want to make things too tricky, but perhaps a mapping of data_types values to these search data types would simply this a bit. (e.g. string -> string, int -> number, with some exceptions....)

django_mongodb_backend/indexes.py

django_mongodb_backend/schema.py

django_mongodb_backend/indexes.py

timgraham · 2025-03-20T23:24:07Z

django_mongodb_backend/indexes.py

+            elif isinstance(
+                field_,
+                BooleanField
+                | IntegerField
+                | DateField
+                | DateTimeField
+                | CharField
+                | TextField
+                | UUIDField
+                | ObjectIdField
+                | ObjectIdAutoField,
+            ):


Is there any documentation you can link to that explains this list?

Yes, it's here. It is in fields.type table entry. I think copying the exact text:

filter - for fields that contain boolean, date, objectId, numeric, string, or UUID values.

Maybe we could make use the type mapping here? Instead of enumerating all the supporting fields?

It seems type checking may be more extensible, e.g. for custom fields.

🤔 ok, so, checking by the internal type is better and more flexible. Maybe with the double mapping idea or similar.

django_mongodb_backend/indexes.py

timgraham · 2025-03-20T23:35:10Z

django_mongodb_backend/fields/array.py

@@ -27,14 +28,23 @@ class ArrayField(CheckFieldDefaultMixin, Field):
    }
    _default_hint = ("list", "[]")

-    def __init__(self, base_field, size=None, **kwargs):
+    def __init__(self, base_field, size=None, fixed_size=None, **kwargs):


The parameter names don't look clear. I'm inclined to rename size to max_size and fixed_size to size. What do you think? Let's add this feature in a separate PR to precede this feature. I'm thinking first commit: rename size to max_size, second commit: add size.

I did part 1 in #273. Want to send a PR to my branch with part 2? There is a bit more to do than you already did (like handle the parameter in deconstruct(), add it to test migrations, support in forms, and update documentation). Hopefully my PR gives hints about the places that need updating.

Ok, shall I add those change also to my PR?

django_mongodb_backend/schema.py

timgraham · 2025-03-21T23:18:22Z

django_mongodb_backend/indexes.py

+            elif isinstance(
+                field_,
+                BooleanField
+                | IntegerField
+                | DateField
+                | DateTimeField
+                | CharField
+                | TextField
+                | UUIDField
+                | ObjectIdField
+                | ObjectIdAutoField,
+            ):


It seems type checking may be more extensible, e.g. for custom fields.

django_mongodb_backend/schema.py

django_mongodb_backend/indexes.py

timgraham · 2025-03-23T01:30:31Z

django_mongodb_backend/indexes.py

+                try:
+                    vector_size = int(field_.size)
+                except (ValueError, TypeError) as err:
+                    raise ValueError("Atlas vector search requires size.") from err
+                if not isinstance(field_.base_field, FloatField | DecimalField):
+                    raise ValueError("Base type must be Float or Decimal.")


This sort of validation really belongs as a system check, since at this point the user has already generated an invalid migration and has to do quite a lot to fix the problem. Unfortunately, the Index.check() hook is missing. Until then, however, I think we can write a system check that iterates over all the models in app_configs and does this validation. Let me know if you run into trouble with that idea.

I am creating the system check, the problem here is: can I import connection. Can I get that?

A first approach to this is implemented.

timgraham · 2025-03-23T01:33:06Z

tests/indexes_/test_atlas_indexes.py

+    def test_field_size_required(self):
+        index = VectorSearchIndex(
+            name="recent_article_idx",
+            fields=["number_list"],
+        )
+        msg = "Atlas vector search requires size."
+        with self.assertRaisesMessage(ValueError, msg), connection.schema_editor() as editor:
+            editor.add_index(index=index, model=Article)


For testing invalid model logic (which can hopefully be done as system checks), see
https://github.com/django/django/blob/922c1c732a47c02aa5ef28b0b1a2bd9bc9b92d87/tests/check_framework/test_model_checks.py

timgraham · 2025-03-23T01:37:32Z

django_mongodb_backend/indexes.py

+        if isinstance(similarities, str):
+            self._check_similarity_functions([similarities])
+        else:
+            self._check_similarity_functions(similarities)


I'm not sure I like accepting both a string and a list. Did you find a precedent for that? Even if so, self.similarities could at least be stored as a list to simplify future usage. Since you have a custom parameter, you need to add a deconstruct() method.

I tried the pattern:
single value applies to all elements, and a list assigns values one-to-one.
Like:
border parameter of expand in pillow
But I can make it one to one.

I didn't realize that, so I'll look again.

timgraham · 2025-03-23T02:07:35Z

tests/indexes_/test_atlas_indexes.py

+    def test_field_not_exists(self):
+        index = SearchIndex(
+            name="recent_article_idx",
+            fields=["headline", "non_existing_name"],
+        )
+        msg = "Article has no field named 'non_existing_name'"
+        with self.assertRaisesMessage(FieldDoesNotExist, msg), connection.schema_editor() as editor:
+            editor.add_index(index=index, model=Article)


This test may be unnecessary since it's already validated by a system check:

search_indexes.Article: (models.E012) 'indexes' refers to the nonexistent field 'headline'.

timgraham · 2025-03-25T14:35:53Z

django_mongodb_backend/indexes.py

+from django.db.models import (
+    DecimalField,
+    FloatField,
+    Index,
+)


Items were removed, so this can be a single line.

timgraham · 2025-03-25T14:36:11Z

django_mongodb_backend/indexes.py

@@ -101,6 +109,156 @@ def where_node_idx(self, compiler, connection):
    return mql


+def get_pymongo_index_model(


This is added redundantly after the rebase.

django_mongodb_backend/base.py

Jibola · 2025-03-23T23:06:18Z

.github/workflows/mongodb_settings.py

@@ -17,10 +17,12 @@
        "default": {
            "ENGINE": "django_mongodb_backend",
            "NAME": "djangotests",
+            "OPTIONS": {"directConnection": True},


Let's add a comment explaining this parameter is necessary against local atlas instances running in docker.

Docker site don't provide much info about it. Just give the connection uri format.
I think is to connect to a single MongoDB instance like standalone mode. But need to confirm that.

Jibola · 2025-03-23T23:09:23Z

django_mongodb_backend/fields/array.py

@@ -27,13 +28,21 @@ class ArrayField(CheckFieldDefaultMixin, Field):
    }
    _default_hint = ("list", "[]")

-    def __init__(self, base_field, size=None, **kwargs):
+    def __init__(self, base_field, max_size=None, size=None, **kwargs):


If this is just meant for vector_searching then can we have this be "fixed_size=True" rather than max_size? Then the validator would just use a ternary between ArrayMaxLengthValidator and LengthValidator. I think that may make understanding the difference easier to intuit.

Array is not only for vector search, is for general purpose. Like array of array. Or array of char field or array of embedded fields

Jibola · 2025-03-25T16:35:38Z

django_mongodb_backend/indexes.py

+                    {
+                        "type": "vector",
+                        "numDimensions": vector_size,
+                        "similarity": next(similarities),


I'm still not sure why similarities needs to be a list?
I saw earlier that it was one value per field, but I'm still not sure why we can't enforce just one index type as the similarity?

Another NIT: this mapping doesn't include optional quantization. I don't think will be necessary until we support BSONVector, but just wanted to note it here.

Well, the code is a bit weird. Maybe it's not easy to read.
The idea here is that we can have a vector index with more than one vector, right? Like creating an index with two vectors—one using cosine similarity and the other using dot product. So the idea is that if you define multiple vector fields, you can assign them different similarity functions. If a single string is passed as the similarity function, it will be applied to all vectors.

django_mongodb_backend/introspection.py

Jib's suggestion Co-authored-by: Jib <[email protected]>

WaVEV requested review from timgraham and Jibola March 3, 2025 02:45

timgraham reviewed Mar 4, 2025

View reviewed changes

WaVEV force-pushed the create-atlas-indexes branch from 03629ae to de3d245 Compare March 8, 2025 03:50

timgraham reviewed Mar 11, 2025

View reviewed changes

WaVEV force-pushed the create-atlas-indexes branch 2 times, most recently from 1bf4717 to 7dc04ab Compare March 20, 2025 21:49

WaVEV commented Mar 20, 2025

View reviewed changes

django_mongodb_backend/indexes.py Outdated Show resolved Hide resolved

WaVEV marked this pull request as ready for review March 20, 2025 23:05

timgraham reviewed Mar 20, 2025

View reviewed changes

timgraham reviewed Mar 21, 2025

View reviewed changes

timgraham mentioned this pull request Mar 23, 2025

Rename ArrayField.size to max_size #273

Open

timgraham reviewed Mar 23, 2025

View reviewed changes

timgraham changed the title ~~Create atlas indexes~~ Add SearchIndex and VectorSearchIndex Mar 23, 2025

timgraham reviewed Mar 23, 2025

View reviewed changes

WaVEV added 15 commits March 25, 2025 00:00

Refactor index creation

f21bb59

stubed up method

5ff00c0

Atlas index creation

d1e74f1

Test

dee6231

Fix mapping typing issue

776da14

Refactor unit test

ca8cf45

Add test with data

ccf05a7

Add atlas vector search index

a47877b

Change minimun pymongo version

ee80da2

ObjectId Fields does not support integers

e1a1842

Refactor: remove singledispatchmethod

fc8d08d

Add directConnection in testing settings

efcfc0f

Add lenght validator

583c084

Add unit test

c069618

Using fixed_size instead of size

967a77a

WaVEV added 15 commits March 25, 2025 00:04

Add validators test

df2baf5

Refactor fixed_size and size in array field

9ed469b

Add create_search_index to OperationDebugWrapper

8dd70b0

change tuple for frozenset

6b793d0

Refactor

7fc5924

Move import to the top

1e289fc

Add drop_search_index and list_search_indexes to OperationDebugWrapper

c0a8323

rename size and fixed_size

f44ef59

Docstring and search type mapping

ab4c1bb

Refactor if conditions

67c7cf1

Add unit tests

5d5f225

Add array field unit test

cbbe2fb

Simplify with

ac07863

Remove blank line

6460625

Add deconstruct method

2865e13

WaVEV force-pushed the create-atlas-indexes branch from 60d49de to 2865e13 Compare March 25, 2025 03:05

timgraham reviewed Mar 25, 2025

View reviewed changes

Jibola reviewed Mar 25, 2025

View reviewed changes

WaVEV and others added 4 commits March 26, 2025 13:16

Remove duplicate function

a3efe35

Update django_mongodb_backend/introspection.py

6223ea1

Jib's suggestion Co-authored-by: Jib <[email protected]>

Remove redundant tests

3154ac6

Index prefix cannot be longer than 3 chars

977da85

WaVEV force-pushed the create-atlas-indexes branch from b992aa7 to 977da85 Compare March 27, 2025 04:21

WaVEV added 7 commits March 27, 2025 19:44

Check function in VectorSearchIndex.

4263fe6

Add parameter connection to the vector search index check

26aa350

add system check and unit test

bd33b3f

Refactor imports

9f08f3a

Fix invalid similarity checker

bc82a58

Add similarities unit test

5016356

Remove check test, were moved to system check

9fdc143

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SearchIndex and VectorSearchIndex #264

Add SearchIndex and VectorSearchIndex #264

WaVEV commented Mar 3, 2025 •

edited by timgraham

Loading

timgraham Mar 4, 2025

timgraham Mar 11, 2025

WaVEV Mar 11, 2025

WaVEV Mar 12, 2025 •

edited

Loading

WaVEV Mar 20, 2025

timgraham Mar 4, 2025

timgraham Mar 11, 2025

timgraham Mar 20, 2025

WaVEV Mar 21, 2025

timgraham Mar 21, 2025

WaVEV Mar 22, 2025 •

edited

Loading

timgraham Mar 20, 2025

timgraham Mar 23, 2025

WaVEV Mar 23, 2025

timgraham Mar 21, 2025

timgraham Mar 23, 2025

WaVEV Mar 27, 2025

WaVEV Mar 28, 2025

timgraham Mar 23, 2025

timgraham Mar 23, 2025

WaVEV Mar 23, 2025

timgraham Mar 23, 2025

timgraham Mar 23, 2025

timgraham Mar 25, 2025

timgraham Mar 25, 2025

Jibola Mar 23, 2025

WaVEV Mar 27, 2025

Jibola Mar 23, 2025

WaVEV Mar 27, 2025

Jibola Mar 25, 2025

WaVEV Mar 27, 2025

		@@ -101,6 +109,156 @@ def where_node_idx(self, compiler, connection):
		return mql


		def get_pymongo_index_model(

Add SearchIndex and VectorSearchIndex #264

Are you sure you want to change the base?

Add SearchIndex and VectorSearchIndex #264

Conversation

WaVEV commented Mar 3, 2025 • edited by timgraham Loading

Notes.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WaVEV Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WaVEV Mar 22, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WaVEV commented Mar 3, 2025 •

edited by timgraham

Loading

WaVEV Mar 12, 2025 •

edited

Loading

WaVEV Mar 22, 2025 •

edited

Loading