Draft: Change OOM protection logic #1317
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The current logic relies on hardcoded values which are not suitable for large hosts. The new logic takes into account the size of hosts and also tries to be more aggressive with misbehaving frames. Prevent host from entering an OOM state where oom-killer might start killing important OS processes. The kill logic will kick in one of the following conditions is met: - Host has less than OOM_MEMORY_LEFT_THRESHOLD_PERCENT memory available - A frame is taking more than OOM_FRAME_OVERBOARD_PERCENT of what it had reserved For frames that are using more than they had reserved but not above the threshold, negotiate expanding the reservations with other frames on the same host (cherry picked from commit e88a5295f23bd927614de6d5af6a09d496d3e6ac) Signed-off-by: Diego Tavares <[email protected]>
DiegoTavares
requested review from
bcipriano,
gregdenton,
jrray,
smith1511,
larsbijl,
IdrisMiles and
splhack
as code owners
Signed-off-by: Diego Tavares <[email protected]>
DiegoTavares
changed the title
|
Standby for review. Still working on a loose end found while running this on production. |
(cherry picked from commit b88f7bcb1ad43f83fb8357576c33483dc2bf4952)
(cherry picked from commit 647e75e2254c7a7ff68c544e438080f412bf04c1)
(cherry picked from commit aea4864ef66aca494fb455a7c103e4a832b63d41)
|
Will submit another PR to avoid having to deal with conflicts |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current logic relies on hardcoded values which are not suitable for large hosts. The new logic takes into account the size of hosts and also tries to be more aggressive with misbehaving frames.
Prevent host from entering an OOM state where oom-killer might start killing important OS processes. The kill logic will kick in one of the following conditions is met:
(cherry picked from commit e88a5295f23bd927614de6d5af6a09d496d3e6ac)
Signed-off-by: Diego Tavares [email protected]