Skip to content

Stability fixes, and related logging for slowdowns in dispatcher task processing #12676

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Aug 17, 2022
Merged

Stability fixes, and related logging for slowdowns in dispatcher task processing #12676

merged 9 commits into from
Aug 17, 2022

Conversation

AlanCoding
Copy link
Member

SUMMARY

Things this does

  • Add logging of things which might contribute to an instance's failure (task processing delays, for example)
  • Provision extra dispatch workers to prevent delays in running the heartbeat
  • Be extra sure to not reap running tasks, passing excluded_uuids into the waiting job reaper
  • Add lots of custom error messages - like distinguishing a control process SIGTERM from execution process
ISSUE TYPE
  • Bug, Docs Fix or other nominal change
COMPONENT NAME
  • API

AlanCoding and others added 9 commits August 17, 2022 11:33
@AlanCoding
Add logs to debug waiting bottlenecking
a3fef27
@AlanCoding @shanemcd
Add logs about heartbeat skew
43a53f4 
Co-authored-by: Shane McDonald <[email protected]>
@shanemcd @AlanCoding
Remove debug method that calls cleanup
c649809 
- It's unclear why this was here.
- Removing it doesnt appear to cause any problems.
- It still gets called during heartbeats.
@shanemcd @AlanCoding
Add grace period settings for task manager timeout, and pod / job wai…
3c51cb1 
…ting reapers

Co-authored-by: Alan Rominger <[email protected]>
@shanemcd @AlanCoding
Add setting for missed heartbeats before marking node offline
c5976e2
@shanemcd @AlanCoding
Allow for passing custom job_explanation to reaper methods
16be38b 
Co-authored-by: Alan Rominger <[email protected]>
@AlanCoding @kdelee
Add extra workers if computing based on memory
621833e 
Co-authored-by: Elijah DeLee <[email protected]>
@AlanCoding
Apply a failed status if cancel_flag is not set
1ea3c56
@AlanCoding
Use delay_update to set error message, according to merge note
56739ac
@AlanCoding AlanCoding requested a review from shanemcd August 17, 2022 16:05
@AlanCoding AlanCoding marked this pull request as ready for review August 17, 2022 16:58
@AlanCoding AlanCoding requested a review from kdelee August 17, 2022 17:02
kdelee
kdelee approved these changes Aug 17, 2022
View reviewed changes
Copy link
Member

@kdelee kdelee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

one day maybe we can fix DISPATCHER_DB_DOWNTOWN_TOLLERANCE 😮‍💨

@AlanCoding AlanCoding merged commit 85beb9e into ansible:devel Aug 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shanemcd shanemcd shanemcd approved these changes

@kdelee kdelee kdelee approved these changes

Assignees
No one assigned
Labels
component:api
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@AlanCoding @shanemcd @kdelee