Redis liveness and readiness is failing due to AOF fsync is taking too long

[vrevanth007]

Name and Version

bitnami/redis:7.2.4-debian-11-r0

What architecture are you using?

None

What steps will reproduce the bug?

Redis liveness and readiness started failing automatically. To debug that issue tried removing the statefulset. Later got the below log.

1:C 07 Jan 2025 11:50:37.931 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 07 Jan 2025 11:50:37.931 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 07 Jan 2025 11:50:37.931 * Configuration loaded
1:M 07 Jan 2025 11:50:37.933 * monotonic clock: POSIX clock_gettime
1:M 07 Jan 2025 11:50:37.934 * Running mode=standalone, port=6379.
1:M 07 Jan 2025 11:50:37.935 * Server initialized
1:M 07 Jan 2025 11:50:37.936 * Reading RDB base file on AOF loading...
1:M 07 Jan 2025 11:50:37.936 * Loading RDB produced by version 7.2.4
1:M 07 Jan 2025 11:50:37.936 * RDB age 606741 seconds
1:M 07 Jan 2025 11:50:37.936 * RDB memory usage when created 1590.68 Mb
1:M 07 Jan 2025 11:50:37.936 * RDB is base AOF
1:M 07 Jan 2025 11:50:44.638 * Done loading RDB, keys loaded: 496444, keys expired: 0.
1:M 07 Jan 2025 11:50:44.638 * DB loaded from base file appendonly.aof.523177.base.rdb: 6.702 seconds
1:M 07 Jan 2025 11:52:24.365 * DB loaded from incr file appendonly.aof.523180.incr.aof: 99.727 seconds
1:M 07 Jan 2025 11:52:46.256 * DB loaded from incr file appendonly.aof.523181.incr.aof: 21.891 seconds
1:M 07 Jan 2025 11:52:46.256 * DB loaded from append only file: 128.321 seconds
1:M 07 Jan 2025 11:52:46.294 * Opening AOF incr file appendonly.aof.523181.incr.aof on server start
1:M 07 Jan 2025 11:52:46.294 * Ready to accept connections tcp
1:M 07 Jan 2025 11:52:46.301 * Starting automatic rewriting of AOF on 1173% growth
1:M 07 Jan 2025 11:52:46.491 * Creating AOF incr file appendonly.aof.523182.incr.aof on background rewrite
1:M 07 Jan 2025 11:52:46.539 * Background append only file rewriting started by pid 13
1:M 07 Jan 2025 11:52:57.093 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:04.059 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:06.074 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:08.006 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:12.044 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:17.023 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:19.006 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:21.070 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:24.050 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:26.012 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:28.041 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:30.061 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:32.038 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:35.005 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
13:C 07 Jan 2025 11:53:35.678 * Successfully created the temporary AOF base file temp-rewriteaof-bg-13.aof
13:C 07 Jan 2025 11:53:35.688 * Fork CoW for AOF rewrite: current 4 MB, peak 13 MB, average 10 MB
1:M 07 Jan 2025 11:53:35.811 * Background AOF rewrite terminated with success
1:M 07 Jan 2025 11:53:36.079 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-13.aof into appendonly.aof.523178.base.rdb
1:M 07 Jan 2025 11:53:36.738 * Removing the history file appendonly.aof.523180.incr.aof in the background
1:M 07 Jan 2025 11:53:36.738 * Removing the history file appendonly.aof.523181.incr.aof in the background
1:M 07 Jan 2025 11:53:36.738 * Removing the history file appendonly.aof.523177.base.rdb in the background
1:M 07 Jan 2025 11:53:37.660 * Background AOF rewrite finished successfully
1:M 07 Jan 2025 11:53:43.030 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:53.029 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:53:59.007 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:54:05.051 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:54:10.047 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:54:13.079 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:54:15.049 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:54:17.028 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:54:19.019 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:54:21.074 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 07 Jan 2025 11:54:25.065 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.

Are you using any custom parameters or values?

No

What is the expected behavior?

The pod should run without restarting

What do you see instead?

Pod is restarting.

[carrodher]

Hi, the issue may not be directly related to the Bitnami container image/Helm chart, but rather to how the application is being utilized, configured in your specific environment, or tied to a particular scenario that is not easy to reproduce on our side.

If you think that's not the case and want to contribute a solution, we'd like to invite you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Please feel free to contact us if you have any questions or need assistance.

If you have any questions about the application, customizing its content, or using technology and infrastructure, we highly recommend that you refer to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

[okgolove]

@vrevanth007 hi there! I am experiencing the same issue with Redis in Kubernetes. What cloud are you running Kubernetes on?

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.