Customising `django_db_modify_db_settings` is superseeded by Django's own caching.

[cameronmaske]

Hey all,

I ran into an issue trying to customize django_db_modify_db_settings.

In my root level conftest.py I tried to make the tests run a sqlite in-memory database using (instead of my project settings.py which point to a postgres database).

@pytest.fixture(scope='session')
def django_db_modify_db_settings():
    from django.conf import settings
    settings.DATABASES = {
        'default': {
            'ENGINE':'django.db.backends.sqlite3',
            'NAME': ':memory:'
        }
    }

However, when the tests were run, any database interaction (Model's get, create, etc) would attempt to reach out to the original postgres database setting (aka

Doing a bit of digging, I think I can see what is happening.

When setup_databases (from here) is called, it in turn calls:

• get_unique_databases_and_mirrors, which in turn calls:
• ConnectionHandler which caches the databases from settings.

After ConnectionHandler is initalized, which I believe is at django.setup() it is fixed and stays pointed to whatever database settings it initalized with, even if they change down the line.

My current work around to this, is to override the DATABASE settings in my own pytest_configure in my project's conftest.py AND to set DJANGO_SETTINGS_MODULE there (e.g. no pytest.ini).
e.g.

os.environ['DJANGO_SETTINGS_MODULE'] = 'app.config.settings'

def pytest_configure():
   settings.DATABASES = {
        'default': {
            'ENGINE':'django.db.backends.sqlite3',
            'NAME': ':memory:'
        }
    }

I believe setting the DJANGO_SETTINGS_MODULE there causes the _setup_django in pytest_load_initial_conftests to be skipped over, thus the it is called later, in the plugin's pytest_configure.

Resource:

Here is a sample project with it in action. It's a bit big, so here are the important bits.

• conftest.py
• settings.py
• Sample test
• pytest outpyt showing test failures for db tests, unit tests pass

I'm using Python: 3.6.6 and the following package versions...

Django==2.1
pytest==3.7.4
pytest-django==3.0.0

[blueyed]

Thanks for the detailed report.

Have you tried using the TEST settings for this?
(see also #559)

[mattaw]

Can I add a +1 to this? Exactly the same issue and the same fix in conftest.py worked.

[blueyed]

@mattaw
Have you seen my previous (unanswered) comment?
It is also not clear if you're using django_db_modify_db_settings yourself (initially).

From the initial comment it appears like a lot of details are known already, so a PR based on that then would have made sense probably.

[Dhananjay1992]

The same issue for me as well.

I am using below in rot conftest.py and by using test_context I want to override the DATABASE which is not happening. It still points to the project.settings settings file

@pytest.fixture(scope='session') def django_db_setup(test_context): settings.DATABASES['default'] = test_context.env_config.get('TEST_DATABASE')

[timthelion]

@blueyed the initial reporter included a complete sample project.

[kdam0]

any update on this would be appreciated? Without this we cannot point settings to use an existing db as mentioned in the docs.

The fix mentioned above is not a working workaround for me, I get connection refused when trying to connect to a postgres instance created by the testing.postgresql library. But, at least Django is attempting to connect to the correct instance.

[seaw688]

Same problem and the solution suggested here doesn't work.

[vazkir]

I honestly don't think this will get fixed anytime soon, perhaps only if you create a PR yourself. There have been multiple issues opened over the years, 2017 was the first one. Basically they all experience the same issue, here are a few examples:

• DATABASES 'TEST' setting is ignored by pytest-django #559
• Tests accessing wrong database #892
• django_db_modify_db_settings fixture does nothing by default #814

Maybe this is the only workaround that can work?

• Tests accessing wrong database #892 (comment)

[tempoxylophone]

I have encountered this same issue when attempting to override the database settings the root conftest.py. This is my workaround:

@pytest.fixture(scope="session", autouse=True)
def configure_test_db(
    database_info: Dict[str, str],
) -> None:
    """
    Add this to your conftest.py
    """
    from django.conf import settings
    from django.db import connections

    # remove cached_property of connections.settings from the cache
    del connections.__dict__["settings"]

    # define settings to override during this fixture
    settings.DATABASES = {
        "default": {
            "ENGINE": "django.db.backends.postgresql",
            **database_info,
        }
    }

    # re-configure the settings given the changed database config
    connections._settings = connections.configure_settings(settings.DATABASES)

    # open a connection to the database with the new database config
    # here the database is called 'default', but one can modify it to whatever fits their needs
    connections["default"] = connections.create_connection("default")

The idea is to

1. Evict the @cached_property of django.db.connections. In Django 5, connections is an instance of django.db.utils.ConnectionHandler: src
2. Make the desired modifications to Django project settings in django.conf.settings and reflect those changes in django.db.connections
3. Refresh the connection to the database that was modified

I have not tested this on a fixture that is function-scoped, i.e. does not set scope="session". It's possible something like this could work for using a database setup for a single test:

@pytest.fixture
def configure_test_db(
    request: pytest.FixtureRequest, database_info: Dict[str, str],
) -> None:
    from django.conf import settings
    from django.db import connections

    # remove cached_property of connections.settings from the cache
    del connections.__dict__["settings"]

    prev_db_setting = settings.DATABASES

    def on_teardown() -> None:
        settings.DATABASES = prev_db_setting

    # define settings to override during this fixture
    settings.DATABASES = {
        "default": {
            "ENGINE": "django.db.backends.postgresql",
            **database_info,
        }
    }

    # re-configure the settings given the changed database config
    connections._settings = connections.configure_settings(settings.DATABASES)

    # open a connection to the database with the new database config
    # here the database is called 'default', but one can modify it to whatever fits their needs
    connections["default"] = connections.create_connection("default")

    request.addfinalizer(on_teardown)

I suppose another way to do this is to change the patch the behavior of functools.cached_property or of django.db.utils.ConnectionHandler somewhere in https://github.com/pytest-dev/pytest-django/blob/master/pytest_django/fixtures.py.

[stefanomunarini]

To add to the previous explanation, since it was leaving the testing process dangling.

In my case, I wanted to use the same database without pytest creating a new one with the ‘test_’ prefix, so I set the test database name to match the default database.

Instead of creating a new fixture, I cleared the cache and recreated the connection within the django_db_setup fixture, as recommended in the pytest-django documentation. This approach effectively resolved the issue for me.

@pytest.fixture(scope='session')
def django_db_setup():
    # remove cached_property of connections.settings from the cache
    del connections.__dict__["settings"]

    settings.DATABASES['default'] = {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'name',
        'USER': 'user',
        'PASSWORD': 'password',
        'HOST': 'host',
        'PORT': '5432',
        "TEST": 
            "NAME": 'name',
        }
    }

    # re-configure the settings given the changed database config
    connections._settings = connections.configure_settings(settings.DATABASES)
    # open a connection to the database with the new database config
    connections["default"] = connections.create_connection("default")

[mfdebby]

To add to the previous explanation, since it was leaving the testing process dangling.

In my case, I wanted to use the same database without pytest creating a new one with the ‘test_’ prefix, so I set the test database name to match the default database.

Instead of creating a new fixture, I cleared the cache and recreated the connection within the django_db_setup fixture, as recommended in the pytest-django documentation. This approach effectively resolved the issue for me.

@pytest.fixture(scope='session')
def django_db_setup():
# remove cached_property of connections.settings from the cache
del connections.dict["settings"]
here's
settings.DATABASES['default'] = {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'name',
'USER': 'user',
'PASSWORD': 'password',
'HOST': 'host',
'PORT': '5432',
"TEST":
"NAME": 'name',
}
}

# re-configure the settings given the changed database config
connections._settings = connections.configure_settings(settings.DATABASES)
# open a connection to the database with the new database config
connections["default"] = connections.create_connection("default")

The above setup also works for me, I'm using Django 5.1.4.

Here's my setup:

import pytest
from decouple import config
from dj_database_url import parse as db_url
from django.conf import settings
from django.db import connections

@pytest.fixture(scope="session")
def django_db_setup() -> None:
    # remove cached_property of connections.settings from the cache
    del connections.__dict__["settings"]

    settings.DATABASES["default"] = config("DATABASE_URL_TEST", cast=db_url)

    # re-configure the settings given the changed database config
    connections.settings = connections.configure_settings(settings.DATABASES)

    # open a connection to the database with the new database config
    connections["default"] = connections.create_connection("default")