Cannot connect to redis, using self-signed TLS and sentinels

[ajgon]

I'm not sure if it's a bug or me being stupid (as I'm not a python developer), so sorry in advance :)

I'm trying to setup channels-redis (4.0.0b2, the one from main branch) with redis sentinel via SSL (self-signed) - SSL is both on sentinels and underlying redises. I have successfully managed to set up sentinels connection, and fetch master node - however I'm unable to join underlying redis. I'm getting CERTIFICATE_VERIFY_FAILED no matter what I try. This is what I'm trying to do:

_ssl_context = ssl.create_default_context()
_ssl_context.check_hostname = False
_ssl_context.verify_mode = ssl.CERT_NONE

CHANNEL_LAYERS = {
  "default": {
    "BACKEND": "channels_redis.core.RedisChannelLayer",
    "CONFIG": {
      "hosts": [
        {
          "sentinels": [('sentinel.host', 26379)],
          "master_name": "mymaster",
          "sentinel_kwargs": {
            "password": "sentinel password",
            "ssl": True,
            "ssl_cert_reqs": "none",
          },
          "ssl": _ssl_context,
          "db": 0,
          "password": "redis password"
        }
      ],
    }
  }
}

and here is the exception:

  File "/usr/local/lib/python3.9/site-packages/redis/asyncio/connection.py", line 709, in connect
    await self._connect()
  File "/usr/local/lib/python3.9/site-packages/redis/asyncio/connection.py", line 744, in _connect
    reader, writer = await asyncio.open_connection(
  File "/usr/local/lib/python3.9/asyncio/streams.py", line 52, in open_connection
    transport, _ = await loop.create_connection(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1090, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1120, in _create_connection_transport
    await waiter
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 534, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 188, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 945, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/paperless/src/src/django-q/django_q/cluster.py", line 454, in worker
    res = f(*task["args"], **task["kwargs"])
  File "/usr/src/paperless/src/documents/tasks.py", line 172, in consume_file
    document = Consumer().try_consume_file(
  File "/usr/src/paperless/src/documents/consumer.py", line 255, in try_consume_file
    self._send_progress(0, 100, "STARTING", MESSAGE_NEW_FILE)
  File "/usr/src/paperless/src/documents/consumer.py", line 76, in _send_progress
    async_to_sync(self.channel_layer.group_send)(
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 218, in __call__
    return call_result.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/asgiref/sync.py", line 284, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/channels_redis/core.py", line 548, in group_send
    await connection.zremrangebyscore(
  File "/usr/local/lib/python3.9/site-packages/redis/asyncio/client.py", line 484, in execute_command
    conn = self.connection or await pool.get_connection(command_name, **options)
  File "/usr/local/lib/python3.9/site-packages/redis/asyncio/connection.py", line 1516, in get_connection
    await connection.connect()
  File "/usr/local/lib/python3.9/site-packages/redis/asyncio/sentinel.py", line 51, in connect
    await self.connect_to(await self.connection_pool.get_master_address())
  File "/usr/local/lib/python3.9/site-packages/redis/asyncio/sentinel.py", line 41, in connect_to
    await super().connect()
  File "/usr/local/lib/python3.9/site-packages/redis/asyncio/connection.py", line 715, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 1 connecting to redis-master:6379.

I also tried to pass ssl_cert_reqs=none as redis connection kwargs, but this parameter is not supported there. I have a feeling that this may be a bug, as by default redis.py sets ssl_certs_reqs to required regardless of context 🤔

[carltongibson]

This isn’t really a channels-redis issue. Maybe redis-py either have a solution or would take it as an issue. (We just pass through the params)

[ajgon]

I've dig it some more, and it turned out - it is a bug, here:

channels_redis/channels_redis/core.py

Lines 138 to 142 in a993f3f

	return aioredis.sentinel.SentinelConnectionPool(
	master_name,
	aioredis.sentinel.Sentinel(sentinels, sentinel_kwargs=sentinel_kwargs),
	**host
	)

The problem is, that host is passed to SentinelConnectionPool, not to the aioredis.sentinel.Sentinel itself. So the whole **connection_kwargs which are passed later to the redis are skipped. The fix will be either passing **host to sentinel as well, i.e.

            return aioredis.sentinel.SentinelConnectionPool(
                master_name,
                aioredis.sentinel.Sentinel(sentinels, sentinel_kwargs=sentinel_kwargs, **host),
                **host
            )

or somehow differentiate and them separately, something like:

            return aioredis.sentinel.SentinelConnectionPool(
                master_name,
                aioredis.sentinel.Sentinel(sentinels, sentinel_kwargs=sentinel_kwargs, **host['redis_connection_kwargs']),
                **host
            )

As I mentioned, I'm not a python dev, so I'm not sure what is the correct way, so I'll leave it up to you :)

[carltongibson]

OK, let's reopen to look. If you want to make a PR with a regression test quickly we can get it in the for release. Thanks

[ajgon]

Looks like, something is also broken on redis-py side. Here are the results of my tests:

Having connection built like these:

return aioredis.sentinel.SentinelConnectionPool(
    master_name,
    aioredis.sentinel.Sentinel(sentinels, sentinel_kwargs=sentinel_kwargs, **connection_kwargs),
    **host_kwargs
)

I'm getting following results basing on given arguments (I'm skipping sentinel_kwargs and master_name, as sentinel iteself works correctly)

Without host kwargs, and connection kwargs configured - ssl to redis master doesn't work.

connection_kwargs = {'password': 'mypass', 'ssl': True, 'ssl_cert_reqs': 'none'}
host_kwargs = {}
# redis.exceptions.ConnectionError: Error while reading from master-node-resolved-from-sentinel:6379 : (104, 'Connection reset by peer')
# 104 - means no SSL connection at all

With ssl configured in host kwargs - password is not used (checked redis-side, no AUTH is sent at all:

connection_kwargs = {'password': 'mypass', 'ssl': True, 'ssl_cert_reqs': 'none'}
host_kwargs = {'ssl': True, 'ssl_cert_reqs': 'none'}
# redis.exceptions.AuthenticationError: Authentication required.

When I try add password to host kwargs, it gets more bizzare, as now despite sentinels were asked for masters, redis py connects to localhost 🤔

connection_kwargs = {'password': 'mypass', 'ssl': True, 'ssl_cert_reqs': 'none'}
host_kwargs = {'ssl': True, 'ssl_cert_reqs': 'none', 'password': 'mypass'}
# OSError: Multiple exceptions: [Errno 111] Connect call failed ('::1', 6379, 0, 0), [Errno 111] Connect call failed ('127.0.0.1', 6379)

And last but not least - I can skip connection_kwargs completely, and all three behaviors repeat:

connection_kwargs = {}
host_kwargs = {}

# redis.exceptions.ConnectionError: Error while reading from master-node-resolved-from-sentinel:6379 : (104, 'Connection reset by peer')
# 104 - means no SSL connection at all

host_kwargs = {'ssl': True, 'ssl_cert_reqs': 'none'}
# redis.exceptions.AuthenticationError: Authentication required.

host_kwargs = {'ssl': True, 'ssl_cert_reqs': 'none', 'password': 'mypass'}
# OSError: Multiple exceptions: [Errno 111] Connect call failed ('::1', 6379, 0, 0), [Errno 111] Connect call failed ('127.0.0.1', 6379)

I officially throw in the towel. It seems channels_redis is either doing everything correctly - however, I'm not sure if it should be done this way on redis-py side... Anyway, there is no point for PR at the moment, I'll copy paste most of this comment to redis-py and ask support there...

Edit: redis-py ticket

[marcinowski]

I've had a similar issue (with straightforward hosts, not sentinels) and I needed to pass extra connection parameters. I've opened a PR to allow for it #337