Serverless AWS Redis results in `redis.exceptions.ResponseError: Lua scripts without any input keys are not supported.`

[Jonathan-Landeed]

Works on "design your own cache" redis clusters but not serverless (engine version 7.1 for both).

Running daphne in docker on aws ec2.

Traceback:

[1] [ERROR] Exception inside application: Lua scripts without any input keys are not supported.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/channels/routing.py", line 62, in __call__
    return await application(scope, receive, send)
  File "/app/./landeed_api/asgi.py", line 54, in __call__
    return await self.application(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/channels/security/websocket.py", line 37, in __call__
    return await self.application(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/channels/sessions.py", line 47, in __call__
    return await self.inner(dict(scope, cookies=cookies), receive, send)
  File "/usr/local/lib/python3.10/site-packages/channels/sessions.py", line 263, in __call__
    return await self.inner(wrapper.scope, receive, wrapper.send)
  File "/usr/local/lib/python3.10/site-packages/channels/auth.py", line 185, in __call__
    return await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/channels/middleware.py", line 24, in __call__
    return await self.inner(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/channels/routing.py", line 116, in __call__
    return await application(
  File "/usr/local/lib/python3.10/site-packages/channels/consumer.py", line 94, in app
    return await consumer(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/channelsmultiplexer/demultiplexer.py", line 61, in __call__
    await future
  File "/usr/local/lib/python3.10/site-packages/channels/consumer.py", line 94, in app
    return await consumer(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/channels/consumer.py", line 58, in __call__
    await await_many_dispatch(
  File "/usr/local/lib/python3.10/site-packages/channels/utils.py", line 57, in await_many_dispatch
    await task
redis.exceptions.ResponseError: Lua scripts without any input keys are not supported.

Pip freeze:

asgiref==3.7.2
attrs==23.2.0
autobahn==23.6.2
Automat==22.10.0
cffi==1.16.0
channels==4.0.0
channels-redis==4.2.0
channelsmultiplexer==0.0.3
constantly==23.10.4
cryptography==41.0.7
daphne==4.0.0
Django==4.2.9
djangochannelsrestframework==1.2.0
djangorestframework==3.14.0
hyperlink==21.0.0
idna==3.6
incremental==22.10.0
msgpack==1.0.7
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
pyOpenSSL==23.3.0
pytz==2023.3.post1
redis==5.0.1
service-identity==24.1.0
six==1.16.0
sqlparse==0.4.4
Twisted==23.10.0
txaio==23.1.1
typing_extensions==4.9.0
zope.interface==6.1

[carltongibson]

Grrr. There's not going to be anything we can do about this.

The PubSub channel layer doesn't use Lua so you could try that.

[root4417]

I'm using the ServerLess Redis Cache from ElastiCache AWS and getting the error.

Exception inside application: Lua scripts without any input keys are not supported.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/channels/routing.py", line 62, in call
return await application(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/channels/routing.py", line 62, in call
return await application(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/channels/sessions.py", line 47, in call
return await self.inner(dict(scope, cookies=cookies), receive, send)
File "/usr/local/lib/python3.10/site-packages/channels/sessions.py", line 263, in call
return await self.inner(wrapper.scope, receive, wrapper.send)
File "/usr/local/lib/python3.10/site-packages/channels/auth.py", line 185, in call
return await super().call(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/channels/middleware.py", line 24, in call
return await self.inner(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/channels/routing.py", line 116, in call
return await application(
File "/usr/local/lib/python3.10/site-packages/channels/consumer.py", line 94, in app
return await consumer(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/channels/consumer.py", line 58, in call
await await_many_dispatch(
File "/usr/local/lib/python3.10/site-packages/channels/utils.py", line 57, in await_many_dispatch
await task
redis.exceptions.ResponseError: Lua scripts without any input keys are not supported.

Here is the config:

CHANNEL_LAYERS = {
    'default': {
        'BACKEND': 'channels_redis.core.RedisChannelLayer',
        'CONFIG': {
            'hosts': [{
                "address": REDIS_HOST,  # "REDIS_TLS_URL"
                "ssl_cert_reqs": None,
            }],
            'expiry': 360,

        },
    },
}

.env redis host:

REDIS_HOST=rediss://<>.serverless.euw2.cache.amazonaws.com:6379

[connor-makowski]

For posterity: Confirming I can bypass the Lua issues for Serverless caches (and replication groups) using the pubsub channel layer. (Confirmed on valkey and redis).

Note: on single instances in aws, the endpoints support redis:// but on serverless you must use rediss://

Also worth noting: Not all redis features are supported on serverless caches. Like: keys(), set_many(), delete_many(). Other features are fairly fragile get_many() for example requires all items to be in the same hash location in memory (which is not always the case) so it fails sometimes.

Some of this is noted in the AWS docs here: https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/SupportedCommands.html#RestrictedCommandsRedis

I was not able to find other AWS docs on non supported items like Lua (which are supported in single instances but not multi) and directly relevant to this thread.

Some other functions (similar to get_many) that appear to be supported are not quite there yet. The basics seem to work well (get, set, delete ...), so you may just have to write loops for get_many and set_many. Methods like keys are just outright not supported and probably wont be.

Anyway, here is my connection code. Hope it can help.

## Channels layers
CHANNEL_LAYERS = {
    'default': {
        'BACKEND': 'channels_redis.pubsub.RedisPubSubChannelLayer',
        'CONFIG': {
            "hosts": [{
                'address': "rediss://" + ebconfig("CACHE_ELASTICACHE_HOSTNAME") + ":6379",
            }],
        },
    },
}
## General Cache support
CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.redis.RedisCache",
        "LOCATION": "rediss://" + ebconfig("CACHE_ELASTICACHE_HOSTNAME") + ":6379",
        # Cache Timeout comes from the development settings
        "TIMEOUT": CACHE_TIMEOUT,
    }
}

Special shout out / thanks to: @carltongibson for the great pointer.

[connor-makowski]

Since this is marked as not planned, I did end up having to make my own socket system for general serverless cache support (vakey and redis). My team at MIT's Computational Analytics, Visualization and Education lab was able to use it to cut our cache costs by 10x.

You can reference it here if you ever change your mind and want to work on this issue: https://github.com/connor-makowski/django_sockets