[12.x] Handle AWS ElasticCache failover by reconnecting when READONLY

[wsamoht]

I am currently running AWS ElasticCache Valkey with multi-AZ enabled. When the primary node fails over to the replica (promoting it to primary), our Horizon daemons continually throw the following error until manually restarted.

READONLY You can't write against a read only replica.

The full stacktrace (from Laravel 11)

READONLY You can't write against a read only replica. script: 230148430cb441a76f6486ae084405a674e00385, on @user_script:2.

#0 /home/forge/***/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(116): Redis->eval()
#1 /home/forge/***/vendor/laravel/framework/src/Illuminate/Redis/Connections/PhpRedisConnection.php(530): Illuminate\Redis\Connections\Connection->command()
#2 /home/forge/***/vendor/laravel/framework/src/Illuminate/Redis/Connections/PhpRedisConnection.php(447): Illuminate\Redis\Connections\PhpRedisConnection->command()
#3 /home/forge/***/vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(295): Illuminate\Redis\Connections\PhpRedisConnection->eval()
#4 /home/forge/***/vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(239): Illuminate\Queue\RedisQueue->retrieveNextJob()
#5 /home/forge/***/vendor/laravel/horizon/src/RedisQueue.php(139): Illuminate\Queue\RedisQueue->pop()
#6 /home/forge/***/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(350): Laravel\Horizon\RedisQueue->pop()
#7 /home/forge/***/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(364): Illuminate\Queue\Worker->Illuminate\Queue\{closure}()
#8 /home/forge/***/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(163): Illuminate\Queue\Worker->getNextJob()
#9 /home/forge/***/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(148): Illuminate\Queue\Worker->daemon()
#10 /home/forge/***/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(131): Illuminate\Queue\Console\WorkCommand->runWorker()
#11 /home/forge/***/vendor/laravel/horizon/src/Console/WorkCommand.php(51): Illuminate\Queue\Console\WorkCommand->handle()
#12 /home/forge/***/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Laravel\Horizon\Console\WorkCommand->handle()
#13 /home/forge/***/vendor/laravel/framework/src/Illuminate/Container/Util.php(43): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#14 /home/forge/***/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\Container\Util::unwrapIfClosure()
#15 /home/forge/***/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod()
#16 /home/forge/***/vendor/laravel/framework/src/Illuminate/Container/Container.php(690): Illuminate\Container\BoundMethod::call()
#17 /home/forge/***/vendor/laravel/framework/src/Illuminate/Console/Command.php(213): Illuminate\Container\Container->call()
#18 /home/forge/***/vendor/symfony/console/Command/Command.php(279): Illuminate\Console\Command->execute()
#19 /home/forge/***/vendor/laravel/framework/src/Illuminate/Console/Command.php(182): Symfony\Component\Console\Command\Command->run()
#20 /home/forge/***/vendor/symfony/console/Application.php(1047): Illuminate\Console\Command->run()
#21 /home/forge/***/vendor/symfony/console/Application.php(316): Symfony\Component\Console\Application->doRunCommand()
#22 /home/forge/***/vendor/symfony/console/Application.php(167): Symfony\Component\Console\Application->doRun()
#23 /home/forge/***/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(197): Symfony\Component\Console\Application->run()
#24 /home/forge/***/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1205): Illuminate\Foundation\Console\Kernel->handle()
#25 /home/forge/***/artisan(13): Illuminate\Foundation\Application->handleCommand()
#26 {main}

It would seem this should be handled by phpredis itself, but Laravel has this continually growing array of errors to force a reconnect 🤷 phpredis on our servers is the latest 6.2.0.

For the moment we switched back to predis as the reconnect works automatically. See predis/predis#720.

We tried setting retries with no difference in our database.php config:

'options' => [
    'cluster' => env('REDIS_CLUSTER', 'redis'),
    'prefix' => '',
    'max_retries' => 10,
    'retry_interval' => 200,
    'backoff_algorithm' => Redis::BACKOFF_ALGORITHM_DECORRELATED_JITTER,
    'backoff_base' => 250,
    'backoff_cap' => 750,
],

That's probably because it's not trying to reconnect but using the existing connection that is now read-only.

I did try this by overriding the Redis classes, but it's a lot of code to replicate where Laravel should handle it itself.

// AppServiceProvider.php
Redis::extend('phpredis', fn () => new CustomPhpRedisConnector());

// CustomPhpRedisConnection.php
<?php

namespace App\Redis;

use Illuminate\Redis\Connections\PhpRedisConnection;
use Illuminate\Support\Str;
use RedisException;

class CustomPhpRedisConnection extends PhpRedisConnection
{
    public function command($method, array $parameters = [])
    {
        try {
            return parent::command($method, $parameters);
        } catch (RedisException $e) {
            if (Str::contains($e->getMessage(), ['READONLY'])) {
                $this->client = $this->connector ? call_user_func($this->connector) : $this->client;
            }

            throw $e;
        }
    }
}

// CustomPhpRedisConnector.php
<?php

namespace App\Redis;

use Illuminate\Redis\Connectors\PhpRedisConnector;
use Illuminate\Support\Arr;

class CustomPhpRedisConnector extends PhpRedisConnector
{
    public function connect(array $config, array $options)
    {
        $formattedOptions = Arr::pull($config, 'options', []);

        if (isset($config['prefix'])) {
            $formattedOptions['prefix'] = $config['prefix'];
        }

        $connector = function () use ($config, $options, $formattedOptions) {
            return $this->createClient(array_merge(
                $config,
                $options,
                $formattedOptions
            ));
        };

        return new CustomPhpRedisConnection($connector(), $connector, $config);
    }
}