Compactor stuck. not finishing compactor runs, but seems to making some progress.

[alanwguo]

Hello, I need some help understanding why my compactor is stuck or taking a very long time.

We have a sharded compactor setup. Currently, we have 30 replicas.
For each replica, the compactor runs metrics looks like this:
cortex_compactor_runs_started_total: 1
cortex_compactor_runs_completed_total: 0
cortex_compactor_runs_failed_total: 0
cortex_compactor_runs_interrupted_total: 0

We have kept the compactor running for days and weeks and did not seem to finish.

In the logs, we do see progress, for some tenants they finish quickly. We see both starting compaction of user blocks and successfully compacted user blocks.

But for other tenants, we only see starting compaction of user blocks and never finishes. We do see other progress for those tenants such as: finished compacting blocks, block has been marked for deletion, uploaded block, compacted blocks, downloaded and verified blocks; compacting blocks, etc.

One last metric is that we see: rate(cortex_compactor_blocks_cleaned_total[5m]) is non-zero in spikes. So it seems like some progress is being made.

should i be concerned that cortex_compactor_runs_completed_total never increases?