Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[out-of-space-prevention] stop writes to the replica at configurable disk usage threshold #22921

Open
denesb opened this issue Feb 19, 2025 · 0 comments
Labels
area/elastic cloud area/storage Storage space reduction and other storage enhancements enhancement

Comments

@denesb
Copy link
Contributor

denesb commented Feb 19, 2025

Simple hit-the-wall behaviour. When disk usage goes above the configured threshold, stop all user writes to the replica. Allow writes to system tables so tablet state machine can be advanced and tablets can be migrated away.
We have to consider all possible source of disk writes:

  • regular mutation writes
  • streaming
  • repair
  • sstable upload
  • hints
  • snapshots
  • compactions

We will probably make some exceptions for system tables too, e.g. don't accept writes to system_tracing keyspace.

This feature will only work in tablet-only cluster. With vnodes, there are too many things to consider, especially around bootstrapping new nodes with RBNO.
With tablet-only cluster, only a few system tables use vnodes and these have small amount of data, so allowing RBNO (which may bring in data to the local node too) is safe.

@denesb denesb added area/elastic cloud area/storage Storage space reduction and other storage enhancements enhancement labels Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/elastic cloud area/storage Storage space reduction and other storage enhancements enhancement
Projects
None yet
Development

No branches or pull requests

1 participant