Skip to content

Remove redundant restart #279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Oct 16, 2023
Merged

Remove redundant restart #279

merged 27 commits into from
Oct 16, 2023

Conversation

widhalmt
Copy link
Member

@widhalmt widhalmt commented Sep 28, 2023

Restarting Elasticsearch takes quite a while and may lead to connection issues as well as sync issues. So keeping restarts to a minimum is important. These changes will make sure that, even when the Restart Elasticsearch handler is notified, it will only restart if Elasticsearch was running before. If there's a fresh start (after reconfiguration) we don't need to restart again.

Same goes for Logstash and Kibana. Some restarts of these tools happen fairly fast. But others (like after fresh installs or updates) will trigger internal jobs that should not be intercepted by another restart.

Beats restart very fast and as far as I know there's not a big downside to restarting them right after the first start so I didn't include them in the change.

Additionally, this PR will make sure some tasks in verify.yml of the full stack are only run when the service to be checked is actually running on this node. This helps with spreading services over nodes to save ressources.

Since GitHub hosted runners are quite low on ressources we can't run every service on every node in a cluster setup anymore. So this PR will make sure that only Elasticsearch runs everywhere and the others are spread out.

Caches get cleared after every role in during a Molecule test. This helps with saving ressources, too.

Elasticsearch still won't sync all shards due to full volumes, the watermarks for Elasticseach are set to extremely high volumes so that the cluster can at least get into sync.

fixes #278
fixes #141
fixes #194

We don't need to restart Elasticseach after this task. Everything is set
in a similar task earlier. This one is only to change the start
bevaviour to a safer one (not reinitializing the cluster). The change is
only needed during restarts, so whenever Elasticsearch is restarted, the
new version will be used.

fixes #278
@widhalmt widhalmt added bug Something isn't working feature New feature or request labels Sep 28, 2023
@widhalmt widhalmt self-assigned this Sep 28, 2023
@widhalmt widhalmt marked this pull request as draft September 28, 2023 12:48
@widhalmt widhalmt marked this pull request as ready for review September 30, 2023 19:08
@widhalmt widhalmt enabled auto-merge September 30, 2023 19:10
@widhalmt
Copy link
Member Author

Seems like the Elasticsearch cluster still has problems syncing. I'll withdraw the request for review and get back to Draft.

@widhalmt widhalmt marked this pull request as draft September 30, 2023 19:21
auto-merge was automatically disabled September 30, 2023 19:21

Pull request was converted to draft

@widhalmt widhalmt marked this pull request as ready for review October 11, 2023 16:12
@widhalmt widhalmt requested a review from a team October 11, 2023 16:13
@widhalmt widhalmt enabled auto-merge October 11, 2023 16:13
@widhalmt widhalmt added this pull request to the merge queue Oct 16, 2023
Merged via the queue into main with commit d5f7f54 Oct 16, 2023
@widhalmt widhalmt deleted the fix/redundant_restarts-278 branch October 16, 2023 17:28
github-merge-queue bot pushed a commit that referenced this pull request Oct 17, 2023
This will need #279 to be merged before tests can pass.

fixes #283
ivareri pushed a commit to ivareri/ansible-collection-elasticstack that referenced this pull request Jun 17, 2025
Restarting Elasticsearch takes quite a while and may lead to connection
issues as well as sync issues. So keeping restarts to a minimum is
important. These changes will make sure that, even when the `Restart
Elasticsearch` handler is notified, it will only restart if
Elasticsearch was running before. If there's a fresh start (after
reconfiguration) we don't need to restart again.

Same goes for Logstash and Kibana. Some restarts of these tools happen
fairly fast. But others (like after fresh installs or updates) will
trigger internal jobs that should not be intercepted by another restart.

Beats restart very fast and as far as I know there's not a big downside
to restarting them right after the first start so I didn't include them
in the change.

Additionally, this PR will make sure some tasks in `verify.yml` of the
full stack are only run when the service to be checked is actually
running on this node. This helps with spreading services over nodes to
save ressources.

Since GitHub hosted runners are quite low on ressources we can't run
every service on every node in a cluster setup anymore. So this PR will
make sure that only Elasticsearch runs everywhere and the others are
spread out.

Caches get cleared after every role in during a Molecule test. This
helps with saving ressources, too.

Elasticsearch still won't sync all shards due to full volumes, the
watermarks for Elasticseach are set to extremely high volumes so that
the cluster can at least get into sync.

fixes NETWAYS#278
fixes NETWAYS#141 
fixes NETWAYS#194
ivareri pushed a commit to ivareri/ansible-collection-elasticstack that referenced this pull request Jun 17, 2025
This will need NETWAYS#279 to be merged before tests can pass.

fixes NETWAYS#283
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Redundant restarts Fix full stack workflow on all containers Restarting Elasticsearch after the first start is not ideal
2 participants