What happens if the ShardAwareDeduplicateFilter is not used

[ASatanicPickle]

Hi,

We have a very large tenant with about 800K blocks and the when the compactor runs the ShardAwareDeduplicateFilter, that filter takes about 24h - 30h to complete.

What would happen if that filter wasn't run? Would it be okay to compact with duplicate blocks? Or, if there's something else we could do that would be awesome too.

Thanks

[deniszh]

Hi @ASatanicPickle ,
Could you please share your compactor config? How many compactors do you use?

PS: we're using following for one of big tenants:

...
      compactor_split_and_merge_shards: 16 # recommended - 1 shard per every 8 million
      compactor_split_groups: 16 # recommended - 1 shard per every 8 million
      compactor_tenant_shard_size: 16 # Give 90% of total compactor replicas to this tenant
...
compactor:
...
  replicas: 20
  podDisruptionBudget:
    maxUnavailable: 20%
  resources:
    limits:
      cpu: 4
      memory: 12Gi
    requests:
      cpu: 2
      memory: 12Gi
  extraArgs:
    compactor.compaction-concurrency: 1
    compactor.compaction-interval: 1h
    compactor.ring.wait-stability-max-duration: 30m
    compactor.meta-sync-concurrency: 200
    compactor.block-sync-concurrency: 50
    compactor.max-compaction-time: 0
...

[ASatanicPickle]

Hi,

It's a bit of a story. We suffered an infrastructure issue that prevented the compactors from running for a while. And we have a very large tenant with about ~950 million active series. So we built up a lot of blocks, about 800,000

The Sync Meta portion of the compactor, runs some filters and the ShardAwareDeduplicateFilter we found takes a long time to run, 24h - 30h with 800K blocks.

The addSuccessorIfPossible func iterates over the list of blocks with a sort of polynomial runtime. And it just takes a very long time.

Here is an issue link to somebody else noticing it as well. #5742

We got the compactors running again but with a delay of 24h hours per sync metadata run, its real slow going. I know our compactor shard settings are not right for the size of our tenant, but the the runtime of this filter is the our primary concern at the moment.

Many thanks,
Mark.