Add SearchIndex and VectorSearchIndex

[WaVEV]

No description provided.

[timgraham]

As currently implemented, I believe this should be string:

django-mongodb-backend/django_mongodb_backend/base.py

Line 62 in 43f477c

"UUIDField": "string",

Separately, maybe it's worth confirming that UUIDField can't store it values as BSON uuid. Setting DatabaseFeatures.has_native_uuid_field = True raises the error: ValueError: cannot encode native uuid.UUID with UuidRepresentation.UNSPECIFIED. UUIDs can be manually converted to bson.Binary instances using bson.Binary.from_uuid() or a different UuidRepresentation can be configured. See the documentation for UuidRepresentation for more information.

I don't think I tried any further to make that work.

[WaVEV]

yes, good catch. I think we could support UUIDFields if we adapt UUIDField.get_db_prep_value and maybe to_python (have to check); like DurationField was adapted. It seems to be out of scope, we could add another task for that.

[timgraham]

This is the type we also will want for EmbeddedModelArrayField, correct? We might check if EmbeddedModelField.db_type() should be defined and could return embeddedDocuments.

How about support for ArrayField? https://www.mongodb.com/docs/atlas/atlas-search/field-types/array-type/

And JSONField? (search type="document"?)

[WaVEV]

JSONField is supported if I am not mistaken. It types as object so is mapped as document. We could map ArrayField also. I don't see any cons to define EmbeddedModelField.db_type() as embeddedDocuments.

EDIT: the only cons is the last S. All the other types are in singular.

[WaVEV]

Supporting ArrayField would be as I explained in here. I will add and if it is not very convincing, we could remove it. Also I noticed that we could have an array of integers or float that not necessarily be an embedded thing (a vector to do a search), also it could be with size. So the current classification won't work.

[WaVEV]

For Vector search I cannot add the support as simple as Search Indexes. It might need something that separates filter fields from vector fields. It will change the class sign. It is not very difficult but I suggest to work on that in a next iteration.

[timgraham]

Is isinstance() definitely what we want here as opposed to say checking db_type()? (That would be double and decimal for those fields.) On the one hand, an error message that references FloatField & DecimalField probably covers most common use cases and is going to be easier to understand that referencing double/decimal. Just wondering if you had any thoughts on this.

Is this the relevant documentation for this restriction? https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-type/#about-the-vector-type I see double there but it's not clear to me how DecimalField qualifies. I'm not sure what the doucmentation means when it mentions:

• BSON BinData vector subtype float32
• BSON BinData vector subtype int1
• BSON BinData vector subtype int8

[WaVEV]

DecimalField is not used for that, is Integer field. But Given that not all the similarities are supported for int1. Shall we support only float32?

[timgraham]

I'm not sure, maybe we ask the team.

[timgraham]

The linked doc says, "You can filter on boolean, date, objectId, numeric, string, and UUID values, including arrays of these types."
Does this account for arrays?

[WaVEV]

Well, will try to add some light over this behaviours. The array could be defined in the left hand side if I try to make a query like:

db.t.t.aggregate([
  {
    "$vectorSearch": {
      "index": "vector_index",
      "filter": {
        "name": ["example", "example3"]
      },
      "path": "values",
      "queryVector": [0,0,0,0,0],
      "numCandidates": 150,
      "limit": 10,
      "quantization": "scalar"
    }
  }])

I got the following error:
MongoServerError[UnknownError]: PlanExecutor error during aggregation :: caused by :: "filter" must be a boolean, objectId, number, string, date, uuid, or null
So I cannot pass an array as a filter.

But I can define a documents like:

db.t.t.insertOne({
  name: "example",
  values: [1.23, 4.56, 7.89, 0.12,3.45]
})

db.t.t.insertOne({
  name: ["example", "example2"],
  values: [1.23, 4.56, 7.89, 0.12,3.45]
})

Then I filter by:

db.t.t.aggregate([
  {
    "$vectorSearch": {
      "index": "vector_index",
      "filter": {
        "name": "example2"
      },
      "path": "values",
      "queryVector": [0,0,0,0,0],
      "numCandidates": 150,
      "limit": 10,
      "quantization": "scalar"
    }
  }])

I will get the second and if I filter by example I will get both.

[WaVEV]

But indeed we could support array of those types.