Horizon auto scaling not moving processes

[markieo1]

Hello,

We are having an issue with our horizon queues.
We are running PHP 8.4.2 with Horizon 5.30.2
We've recently switched from simple balancing to auto.
Our horizon.php config looks like this (only the part of the supervisors):

'defaults' => [
        'syncs' => [
            'connection' => 'horizon_jobs',
            'queue' => ['dispatchers', 'default'],
            'autoScalingStrategy' => 'size',
            'balance' => 'auto',
            // The number of seconds to wait in between auto-scaling attempts.
            'balanceCooldown' => 3,
            // The maximum number of processes that can be scaled up or down in a single auto-scaling attempt.
            'balanceMaxShift' => 1,
            'memory' => 256,
            'tries' => 1,
            'timeout' => 3600,
            'minProcesses' => 1,
            'maxProcesses' => env('HORIZON_SYNC_POOL_PROCESSES', 9),
            'maxJobs' => 1000
        ],
    ],

    'environments' => [
        '*' => []
    ],
],

In our .env we've adjusted HORIZON_SYNC_POOL_PROCESSES to 21.

Some of the times we see that all processes keep "hanging" on a queue with 0 jobs pending. Even after the balanceCooldown of 3 seconds. See the screenshot below:

We'd expect that Horizon would auto scale the processes from the default queue to the dispatchers queue. Since there are a lots of jobs pending there. After some time it would automatically begin to move processes, this seems to be happening when there are lots of jobs scheduled in the default queue.

The same issue is happening when there are no pending jobs. The processes are still there on some queues while nothing is pending. It is also not happening all the time. When we restart Horizon the issue is gone for some time and then returns.

How could we overcome this issue?

[macropay-solutions]

Is this happening after a deploy?

[markieo1]

No this also happens after just restarting Horizon. A deploy is unrelated to this issue.

[markieo1]

I've created laravel/horizon#1539 for this.