Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: while jobset-wait retry on hydra failures #70

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

amerocu
Copy link

@amerocu amerocu commented Oct 1, 2023

While waiting for a jobset to complete, sometimes network requests can fail for many reasons. In these cases, the CLI will retry for 5 times in one minute, instead of just failing.

While waiting for a jobset to complete, sometimes network requests can fail for many reasons. In these cases, the CLI will retry for 5 times in one minute, instead of just failing.
@414owen
Copy link

414owen commented Oct 2, 2023

Oops, I just saw the value of max_retries is hard-coded. I guess you can disregard most of my review :)

@amerocu
Copy link
Author

amerocu commented Oct 2, 2023

I see your points, it would be better to have those parameters configured from outside, but given that we already have the sleep value hard-coded I tried to set reasonable defaults, and keep the MR small.

I think we can tackle the point of making this command more configurable and still keep these defaults. Would also be interesting to have dynamic seeps depending on the state of the jobset. Given how much jobset-wait does, it could make sense to just manage the sleeps automatically instead of having fixed parameters from the user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants