In dev environment occasional exception 'Too many clients already'

[noisecapella]

This happens sometimes when there's a code change and the Django server attempts to restart. I have only seen this on my own dev environment, not sure it's an issue on production

db_1       | FATAL:  sorry, too many clients already
db_1       | FATAL:  sorry, too many clients already
db_1       | FATAL:  sorry, too many clients already
db_1       | FATAL:  sorry, too many clients already
db_1       | FATAL:  sorry, too many clients already
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
web_1      |     self.connect()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 171, in connect
web_1      |     self.connection = self.get_new_connection(conn_params)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
web_1      |     connection = Database.connect(**conn_params)
web_1      |   File "/usr/local/lib/python3.5/site-packages/psycopg2/__init__.py", line 164, in connect
web_1      |     conn = _connect(dsn, connection_factory=connection_factory, async=async)
web_1      | psycopg2.OperationalError: FATAL:  sorry, too many clients already
web_1      | 
web_1      | 
web_1      | The above exception was the direct cause of the following exception:
web_1      | 
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 39, in inner
web_1      |     response = get_response(request)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 244, in _legacy_get_response
web_1      |     response = middleware_method(request)
web_1      |   File "/usr/local/lib/python3.5/site-packages/wagtail/wagtailcore/middleware.py", line 22, in process_request
web_1      |     request.site = Site.find_for_request(request)
web_1      |   File "/usr/local/lib/python3.5/site-packages/wagtail/wagtailcore/models.py", line 154, in find_for_request
web_1      |     'root_page'
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 256, in __iter__
web_1      |     self._fetch_all()
web_1      |   File "/usr/local/lib/python3.5/site-packages/nplusone/ext/django/patch.py", line 275, in fetch_all
web_1      |     original_fetch_all(self)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 1087, in _fetch_all
web_1      |     self._result_cache = list(self.iterator())
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 54, in __iter__
web_1      |     results = compiler.execute_sql()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 833, in execute_sql
web_1      |     cursor = self.connection.cursor()
web_1      |   File "/usr/local/lib/python3.5/site-packages/debug_toolbar/panels/sql/tracking.py", line 42, in cursor
web_1      |     return state.Wrapper(connection._djdt_cursor(), connection, panel)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 231, in cursor
web_1      |     cursor = self.make_debug_cursor(self._cursor())
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 204, in _cursor
web_1      |     self.ensure_connection()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
web_1      |     self.connect()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
web_1      |     six.reraise(dj_exc_type, dj_exc_value, traceback)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
web_1      |     raise value.with_traceback(tb)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
web_1      |     self.connect()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 171, in connect
web_1      |     self.connection = self.get_new_connection(conn_params)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
web_1      |     connection = Database.connect(**conn_params)
web_1      |   File "/usr/local/lib/python3.5/site-packages/psycopg2/__init__.py", line 164, in connect
web_1      |     conn = _connect(dsn, connection_factory=connection_factory, async=async)
web_1      | django.db.utils.OperationalError: FATAL:  sorry, too many clients already
web_1      | 
web_1      | 
web_1      | During handling of the above exception, another exception occurred:
web_1      | 
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
web_1      |     self.connect()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 171, in connect
web_1      |     self.connection = self.get_new_connection(conn_params)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
web_1      |     connection = Database.connect(**conn_params)
web_1      |   File "/usr/local/lib/python3.5/site-packages/psycopg2/__init__.py", line 164, in connect
web_1      |     conn = _connect(dsn, connection_factory=connection_factory, async=async)
web_1      | psycopg2.OperationalError: FATAL:  sorry, too many clients already
web_1      | 
web_1      | 
web_1      | The above exception was the direct cause of the following exception:
web_1      | 
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/local/lib/python3.5/site-packages/dj_static.py", line 83, in __call__
web_1      |     return self.application(environ, start_response)
web_1      |   File "/usr/local/lib/python3.5/site-packages/raven/middleware.py", line 98, in __call__
web_1      |     iterable = self.application(environ, start_response)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 170, in __call__
web_1      |     response = self.get_response(request)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 124, in get_response
web_1      |     response = self._middleware_chain(request)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
web_1      |     response = response_for_exception(request, exc)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 86, in response_for_exception
web_1      |     response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception
web_1      |     return debug.technical_500_response(request, *exc_info)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/views/debug.py", line 84, in technical_500_response
web_1      |     html = reporter.get_traceback_html()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/views/debug.py", line 317, in get_traceback_html
web_1      |     return t.render(c)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 208, in render
web_1      |     return self._render(context)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/test/utils.py", line 94, in instrumented_test_render
web_1      |     return self.nodelist.render(context)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 994, in render
web_1      |     bit = node.render_annotated(context)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 961, in render_annotated
web_1      |     return self.render(context)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/template/defaulttags.py", line 315, in render
web_1      |     return nodelist.render(context)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 994, in render
web_1      |     bit = node.render_annotated(context)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 961, in render_annotated
web_1      |     return self.render(context)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/template/defaulttags.py", line 314, in render
web_1      |     if match:
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 234, in inner
web_1      |     self._setup()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 380, in _setup
web_1      |     self._wrapped = self._setupfunc()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/contrib/auth/middleware.py", line 24, in <lambda>
web_1      |     request.user = SimpleLazyObject(lambda: get_user(request))
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/contrib/auth/middleware.py", line 12, in get_user
web_1      |     request._cached_user = auth.get_user(request)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/contrib/auth/__init__.py", line 187, in get_user
web_1      |     user = backend.get_user(user_id)
web_1      |   File "/usr/local/lib/python3.5/site-packages/social/backends/base.py", line 188, in get_user
web_1      |     return strategy.get_user(user_id)
web_1      |   File "/usr/local/lib/python3.5/site-packages/social/strategies/base.py", line 56, in get_user
web_1      |     return self.storage.user.get_user(*args, **kwargs)
web_1      |   File "/usr/local/lib/python3.5/site-packages/social/storage/django_orm.py", line 83, in get_user
web_1      |     return cls.user_model().objects.get(**kwargs)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
web_1      |     return getattr(self.get_queryset(), name)(*args, **kwargs)
web_1      |   File "/usr/local/lib/python3.5/site-packages/nplusone/core/signals.py", line 23, in wrapped
web_1      |     ret = func(*args, **kwargs)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 379, in get
web_1      |     num = len(clone)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 238, in __len__
web_1      |     self._fetch_all()
web_1      |   File "/usr/local/lib/python3.5/site-packages/nplusone/ext/django/patch.py", line 275, in fetch_all
web_1      |     original_fetch_all(self)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 1087, in _fetch_all
web_1      |     self._result_cache = list(self.iterator())
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 54, in __iter__
web_1      |     results = compiler.execute_sql()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 833, in execute_sql
web_1      |     cursor = self.connection.cursor()
web_1      |   File "/usr/local/lib/python3.5/site-packages/debug_toolbar/panels/sql/tracking.py", line 42, in cursor
web_1      |     return state.Wrapper(connection._djdt_cursor(), connection, panel)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 231, in cursor
web_1      |     cursor = self.make_debug_cursor(self._cursor())
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 204, in _cursor
web_1      |     self.ensure_connection()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
web_1      |     self.connect()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
web_1      |     six.reraise(dj_exc_type, dj_exc_value, traceback)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
web_1      |     raise value.with_traceback(tb)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
web_1      |     self.connect()
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 171, in connect
web_1      |     self.connection = self.get_new_connection(conn_params)
web_1      |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
web_1      |     connection = Database.connect(**conn_params)
web_1      |   File "/usr/local/lib/python3.5/site-packages/psycopg2/__init__.py", line 164, in connect
nginx_1    | 2016/12/28 18:59:00 [error] 7#7: *437 upstream prematurely closed connection while reading response header from upstream, client: 172.18.0.1, server: , request: "GET /learners/ HTTP/1.1", upstream: "uwsgi://172.18.0.6:8077", host: "micromasters.local:8079", referrer: "http://micromasters.local:8079/learners/"
web_1      |     conn = _connect(dsn, connection_factory=connection_factory, async=async)
web_1      | django.db.utils.OperationalError: FATAL:  sorry, too many clients already
web_1      |

[pdpinch]

@noisecapella is the sentry event you saw? https://sentry.io/mit-office-of-digital-learning/micromasters/issues/199716942/

[noisecapella]

No, it's this one: https://sentry.io/mit-office-of-digital-learning/micromasters/issues/220000377/

[blarghmatey]

Both of those Sentry events are from Elasticsearch and they are because I was doing a rolling restart of the ES services to update the logging configuration to not be DEBUG, as it was using a huge amount of disk space for the log files.

[rhysyngsun]

Figured out the culprit, max_connections is 100, but we end up with UWSGI_PROCESS_COUNT * UWSGI_THREAD_COUNT + 1 connections (+1 for the celery process), the defaults give that an upper bound of 201. Setting those two env variables lower locally will make the issue go away.

Not sure what/if there is an appropriate fix to the defaults in uwsgi.ini, I defer to @blarghmatey on that.

[rhysyngsun]

After discussing with @blarghmatey I'm going to set the connection max_age in settings.py to 0 as that won't make a huge impact locally and in production and other envs we set it to that anyway and use pgbouncer to pool.