Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitor service crashes when GitHub returns HTTP 503 #9

Open
1 of 3 tasks
delan opened this issue Nov 18, 2024 · 0 comments
Open
1 of 3 tasks

Monitor service crashes when GitHub returns HTTP 503 #9

delan opened this issue Nov 18, 2024 · 0 comments

Comments

@delan
Copy link
Member

delan commented Nov 18, 2024

When this happens, the service needs to be restarted manually.

Nov 15 14:05:09 ci0 monitor-start[2405852]: [2024-11-15T14:05:09Z INFO  monitor::profile] Destroying runner 3493 with base vm name servo-windows10
Nov 15 14:05:09 ci0 monitor-start[2540097]: Domain 'ci-servo-windows10.3493' destroyed
Nov 15 14:05:09 ci0 monitor-start[2540136]: Domain 'ci-servo-windows10.3493' has been undefined
Nov 15 14:05:10 ci0 monitor-start[2540145]: will destroy tank/ci/servo-windows10.3493
Nov 15 14:05:15 ci0 monitor-start[2405852]: [2024-11-15T14:05:15Z INFO  monitor] Did not receive an API request
Nov 15 14:05:16 ci0 monitor-start[2540153]: gh: Server Error (HTTP 503)
Nov 15 14:05:16 ci0 monitor-start[2405852]: [2024-11-15T14:05:16Z ERROR monitor] Monitor thread error: Command exited with status exit status: 1
Nov 15 14:05:16 ci0 systemd[1]: monitor.service: Main process exited, code=exited, status=1/FAILURE
Nov 15 14:05:16 ci0 systemd[1]: monitor.service: Failed with result 'exit-code'.
Nov 15 14:05:16 ci0 systemd[1]: monitor.service: Consumed 4min 55.506s CPU time, 47.8M memory peak, 4K read from disk, 7.6M incoming IP traffic, 2.4M outgoing IP traffic.
  • Restart systemd services on failure (55b3f42)
  • Restart systemd services after extended failures (5 times in 10 seconds) with an hourly timer
  • Don’t crash when GitHub API requests fail
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant