Websockets via channels are failing when using self-signed Redis stores (as required by Heroku paid-tier Redis instances)

[sunweiyang]

My configuration

I have a Django 3.1.3 app with channels 3.0.3 and channels_redis 3.2.0, deployed on Heroku. In production, I'm using a paid Heroku Redis instance for the channel layer, which is on Redis 6 (which requires the use of self-signed certificates):

For Premium, Private, and Shield plans on version 6 your REDIS_URL config var will use the rediss: scheme, and you must enable TLS in your Redis client’s configuration in order to connect to a Redis 6 database.

My CHANNEL_LAYERS is defined as follows:

CHANNEL_LAYERS = {
    "default": {
        'BACKEND': 'channels_redis.core.RedisChannelLayer',
        "CONFIG": {
            "hosts": [REDIS_URL],
        },
    }
}

I'm using Uvicorn, and below is my Procfile:

release: python manage.py migrate
web: gunicorn config.asgi:application -k config.workers.UvicornWorker
worker: celery -A skale.taskapp worker -P gevent -l info
beat: celery -A skale.taskapp beat -l info -S django

The problem

Websocket connections work fine for my staging server, where I'm using a free-tier Redis instance (redis://). However, on my production server (rediss://), websockets are failing and producing the following error:

 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain

I'll also add that I'm using the same Redis instance for our cache layer, and it's working with a self-signed certificate in production without any issues. It's only channels that's having this problem.

It seems like both channels and channels_redis are missing SSL/TLS instructions, so I'm running into a dead end without more information. What can I do to get this setup working in production?

Appendix

Here is the full traceback from our logs:

Jan 26 22:03:48 mydjangoapp-prod app/web.3 [2021-01-26 22:03:48 -0800] [9] [ERROR] Exception in ASGI application
Jan 26 22:03:48 mydjangoapp-prod app/web.3 Traceback (most recent call last):
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 154, in run_asgi
Jan 26 22:03:48 mydjangoapp-prod app/web.3     result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
Jan 26 22:03:48 mydjangoapp-prod app/web.3     return await self.app(scope, receive, send)
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels/routing.py", line 71, in __call__
Jan 26 22:03:48 mydjangoapp-prod app/web.3     return await application(scope, receive, send)
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/config/asgi.py", line 56, in __call__
Jan 26 22:03:48 mydjangoapp-prod app/web.3     return await self.app(scope, receive, send)
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels/routing.py", line 150, in __call__
Jan 26 22:03:48 mydjangoapp-prod app/web.3     return await application(
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels/consumer.py", line 94, in app
Jan 26 22:03:48 mydjangoapp-prod app/web.3     return await consumer(scope, receive, send)
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels/consumer.py", line 58, in __call__
Jan 26 22:03:48 mydjangoapp-prod app/web.3     await await_many_dispatch(
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels/utils.py", line 58, in await_many_dispatch
Jan 26 22:03:48 mydjangoapp-prod app/web.3     await task
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels/utils.py", line 50, in await_many_dispatch
Jan 26 22:03:48 mydjangoapp-prod app/web.3     result = task.result()
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels_redis/core.py", line 468, in receive
Jan 26 22:03:48 mydjangoapp-prod app/web.3     message_channel, message = await self.receive_single(
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels_redis/core.py", line 523, in receive_single
Jan 26 22:03:48 mydjangoapp-prod app/web.3     content = await self._brpop_with_clean(
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels_redis/core.py", line 356, in _brpop_with_clean
Jan 26 22:03:48 mydjangoapp-prod app/web.3     async with self.connection(index) as connection:
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels_redis/core.py", line 884, in __aenter__
Jan 26 22:03:48 mydjangoapp-prod app/web.3     self.conn = await self.pool.pop()
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/channels_redis/core.py", line 78, in pop
Jan 26 22:03:48 mydjangoapp-prod app/web.3     conn = await aioredis.create_redis(**self.host)
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/aioredis/commands/__init__.py", line 168, in create_redis
Jan 26 22:03:48 mydjangoapp-prod app/web.3     conn = await create_connection(address, db=db,
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/aioredis/connection.py", line 111, in create_connection
Jan 26 22:03:48 mydjangoapp-prod app/web.3     reader, writer = await asyncio.wait_for(open_connection(
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
Jan 26 22:03:48 mydjangoapp-prod app/web.3     return await fut
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/site-packages/aioredis/stream.py", line 23, in open_connection
Jan 26 22:03:48 mydjangoapp-prod app/web.3     transport, _ = await get_event_loop().create_connection(
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "uvloop/loop.pyx", line 2019, in create_connection
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "uvloop/loop.pyx", line 2014, in uvloop.loop.Loop.create_connection
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "uvloop/sslproto.pyx", line 515, in uvloop.loop.SSLProtocol._on_handshake_complete
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "uvloop/sslproto.pyx", line 497, in uvloop.loop.SSLProtocol._do_handshake
Jan 26 22:03:48 mydjangoapp-prod app/web.3   File "/app/.heroku/python/lib/python3.8/ssl.py", line 944, in do_handshake
Jan 26 22:03:48 mydjangoapp-prod app/web.3     self._sslobj.do_handshake()
Jan 26 22:03:48 mydjangoapp-prod app/web.3 ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1125)

[carltongibson]

Hi @sunweiyang. This looks like #235 -- we need to be able to pass an SSLContext to the connection constructor.

Does that seem right to you too?

I'd be happy to take a PR, and do a release adding that.

[nmacianx]

@carltongibson @sunweiyang any updates here? I'm facing the same issue when deploying to Heroku. Help is very much appreciated!!

[carltongibson]

@nmacianx Need someone (you?) to make a pull request adding support here.

[dablak]

See my comment in the other issue #235 (comment).

[carltongibson]

Duplicate of #235