-
Notifications
You must be signed in to change notification settings - Fork 25
INTPYTHON-527 Add Queryable Encryption support #329
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Wrong commit message for 65bd15a and I don't want to force push yet. It should have said:
I'm aware that
|
It's not working as you think it is. As I said elsewhere, Does this fix the "command not supported for auto encryption: buildinfo" error? If so, it's perhaps because I'd suggest to use my patch is as a starting point for maintaining two connections. |
I don't disagree, but it feels a lot like
Yes it works by design, not a side effect. I'm
I'd make a few passes at it but did not get anywhere, I'll try again though. |
Your "stumble" theory of how it's working isn't correct. |
Copy that, thanks! I've removed
Still working on an unencrypted connection, but perhaps the only time we need it is for the version check. |
@ShaneHarvey @Jibola @timgraham FYI here is the
And here is the error again with some additional debug:
And the full traceback:
Test settings:
This is happening in the |
Maybe folks can use the mixin with any Django fields we don't provide ?
Subclassing `dict` to support `queries=EqualityQuery()` API
- Move aws creds to on-demand credentials provided by libmongocrypt (requires `pip install pymongo[aws]`. - Mock boto3 response - Not sure if KMS_CREDENTIALS are being used since the tests succeed after they pass the boto3 mock. - Test var cleanup
- Local provider has no configurable env setting - Kmip provider has configurable provider env only
- Schema map in the client is for development. - Schema map in collection creation is for production. - Create data keys for schema map in the client. - If a schema map is found in the client, use it.
Still leaving the assert failed in because the diff now looks like { "bsonType": "bool", "path": "is_active", "queries": { "queryType": "equality" }, "keyId": { "$binary": { - "base64": "srXESzUzQdq5Vqapl5TqOw==", + "base64": "AaTpZO7vSCiDQ/zH7+dfzw==", "subType": "04" } } } which is expected since the command is generating new data keys and we're comparing the map to the map from the client.
@@ -588,9 +588,17 @@ def django_test_expected_failures(self): | |||
}, | |||
} | |||
|
|||
@cached_property | |||
def mongodb_version(self): | |||
return self.connection.get_database_version() # e.g., (6, 3, 0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally speaking, version checks by drivers should be made through the wire protocol version, not the output of commands like buildInfo
(which may be unavailable in situations like e.g. using the stable/versioned API)
Via Anna Henningsen - Server-side schemas prevent a misconfigured client from accidentally writing unencrypted data - Client-side schemas prevent a malicious or compromised server from advertising an incorrect schema
- ip address field supported - slug field unsupported
- TimeField - URLField
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds comprehensive support for MongoDB's Queryable Encryption feature to Django MongoDB Backend. It introduces new encrypted model fields, router support, management commands, and documentation for using Queryable Encryption in Django applications.
- Adds
EncryptedModel
base class and encrypted field types for sensitive data storage - Implements router support for directing encrypted models to encrypted databases
- Provides management commands and helper utilities for encryption configuration
Reviewed Changes
Copilot reviewed 30 out of 31 changed files in this pull request and generated 6 comments.
Show a summary per file
File | Description |
---|---|
tests/encryption_/tests.py |
Comprehensive test suite covering encrypted fields, KMS credentials, and database operations |
tests/encryption_/routers.py |
Test router for encrypted models directing them to encrypted database |
tests/encryption_/models.py |
Test models demonstrating various encrypted field types and query configurations |
tests/backend_/test_features.py |
Tests for queryable encryption feature detection |
docs/source/topics/encrypted-models.rst |
Documentation explaining encrypted models usage and querying |
docs/source/topics/known-issues.rst |
Known limitations and restrictions for Queryable Encryption |
docs/source/topics/index.rst |
Added encrypted-models to documentation index |
docs/source/releases/5.2.x.rst |
Release notes mentioning Queryable Encryption support |
docs/source/ref/models/models.rst |
Documentation for EncryptedModel class |
docs/source/ref/models/fields.rst |
Documentation for encrypted field types and unsupported fields |
docs/source/ref/django-admin.rst |
Documentation for get_encrypted_fields_map command |
docs/source/intro/configure.rst |
Configuration guidance for encrypted models |
docs/source/index.rst |
Updated main documentation index |
docs/source/howto/index.rst |
Added encryption howto guide |
docs/source/howto/encryption.rst |
Detailed encryption configuration guide |
docs/source/contents.rst |
Updated table of contents |
docs/source/conf.py |
Removed root_doc configuration |
django_mongodb_backend/schema.py |
Schema editor support for creating encrypted collections |
django_mongodb_backend/routers.py |
Router extensions for KMS provider support |
django_mongodb_backend/models.py |
EncryptedModel base class implementation |
django_mongodb_backend/management/commands/get_encrypted_fields_map.py |
Management command for generating encryption schema maps |
django_mongodb_backend/fields/encrypted_model.py |
Encrypted field implementations |
django_mongodb_backend/fields/__init__.py |
Exports for encrypted fields |
django_mongodb_backend/features.py |
Feature detection for queryable encryption support |
django_mongodb_backend/encryption.py |
Helper classes and settings for encryption configuration |
django_mongodb_backend/base.py |
Database version detection fix for encrypted connections |
django_mongodb_backend/__init__.py |
Registration of router extensions |
.evergreen/setup.sh |
CI setup improvements |
.evergreen/run-encryption-tests.sh |
Encryption test runner script |
.evergreen/config.yml |
CI configuration for encryption tests |
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
# On Evergreen jobs, "CI" will be set, and if "CI" is set, add | ||
# "/opt/python/Current/bin" to PATH to pick up `just` and `uv`. | ||
if [ "${CI:-}" == "true" ]; then | ||
PATH_EXT="opt/python/Current/bin:\$PATH" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PATH_EXT variable is missing a leading slash. It should be PATH_EXT="/opt/python/Current/bin:\$PATH"
to properly reference the absolute path.
PATH_EXT="opt/python/Current/bin:\$PATH" | |
PATH_EXT="/opt/python/Current/bin:\$PATH" |
Copilot uses AI. Check for mistakes.
(see previous attempts in #318, #319 and #323 for additional context)