Add code for automatic updates (#250)

Adding playbook contributed by @xtruthx

I have to admit, I kept this playbook for waaaaay too long. @xtruthx
sent it to me a long while back and I always planned to integrate it
into the codebase of this collection.

Now I had to face that I took too long and so I put it up publicly. Now
I worked it over so that it will fit into the `elasticsearch` role just
well.

This PR also introduces a new role where all "global" variables can be
found. This will help determining the correct version of a package to
install. Some tasks that were repeated in every role get moved into this
role called `elasticstack` to streamline execution.

This PR now reads the current version of Elasticsearch installed on the
CA host and uses it as new installation target for all further
installations. This not only helps with upgrading, it will also make
sure that you won't have a version mismatch if you install components
after the initial install.

Special thanks to @hryamzik for the workaround to run a task file like
with `serial: 1`. Found in
ansible/ansible#12170 (comment)

Very special thanks to CID GmbH for sponsoring parts of the the
development of this PR.


fixes #216
fixes #286 
fixes #236
fixes #205

---------

Co-authored-by: Daniel Neuberger <[email protected]>

Author: widhalmt

NOTICE.md

@@ -4,4 +4,4 @@
 
 Here's a list of sponsors who contributed by having the collection improved via outsourcing to NETWAYS.
 
-* CID GmbH : Thank you so much for sponsoring. Especially the feature to have different types of Elasticsearch nodes in the cluster.
+* CID GmbH : Thank you so much for sponsoring. Especially the feature to have different types of Elasticsearch nodes in the cluster and the ingetration of rolling upgrades.

README.md

@@ -79,13 +79,13 @@ You will want to have reliable DNS resolution or enter all hosts of the stack in
 
 The variable `elasticstack_no_log` can be set to `false` if you want to see the output of all tasks. It defaults to `true` because some tasks could reveal passwords in production.
 
-### Versioning
+### Versions and upgrades
 
-*elasticstack_version*: Version number of tools to install. Only set if you don't want the latest. (default: none).
+*elasticstack_version*: Version number of tools to install. Only set if you don't want the latest on new setups. (default: none). If you already have an installation of Elastic Stack, this collection will query the version of Elasticsearch on the CA host and use it for all further installations in the same setup. (Only if you run the `elasticsearch` role before all others) Example: `7.17.2`
 
-*elasticstack_release*: Major release version of Elastic stack to configure. (default: `7`)
+*elasticstack_release*: Major release version of Elastic stack to configure. (default: `7`) Make sure it corresponds to `elasticstack_version` if you set both.
 
-For OSS version see `elasticstack_variant` below. **IMPORTANT** Do not change the version once you have set up the stack. There are unpredictable effects to be expected when using this for upgrades. And upgrade mechanism is already on it's way. (default: none. Example: `7.17.2`)
+For OSS version see `elasticstack_variant` below.
 
 *elasticstack_variant*: Variant of the stack to install. Valid values: `elastic` or `oss`. (default: `elastic`)
 
@@ -99,6 +99,14 @@ roles:
         elasticstack_version: 8.8.1
 ```
 
+#### Upgrades ####
+
+Set `elasticstack_version` to the version you want to upgrade to. Positively do read and understand Elastics changelog and "breaking changes" of your target version and all between your current and the target version. Do not use unless you have a valid backup.
+
+If an upgrade fails, you can try re-running the collection with the same settings. There are several tasks that can provide "self-healing". Please do not rely on these mechanisms, they are more of a "convenience recovery" for easier steps.
+
+The collection will make sure to upgrade Elasticsearch nodes one by one.
+
 ### Default Passwords
 
 Default passwords can be seen during generation, or found later in `/usr/share/elasticsearch/initial_passwords`

docs/role-elasticsearch.md

@@ -13,6 +13,7 @@ Role Variables
 --------------
 
 * *elasticsearch_node_types*: List of types of this very node. Please refer to [official docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html) for details. (default: not set. allowed value: array of types)
++ *elasticsearch_nodename*': Node name of the Elasticsearch node. (default: value of `ansible_hostname`)
 * *elasticsearch_clustername*: Name the Elasticsearch Cluster (default: `elasticsearch`)
 * *elasticsearch_heap*: Heapsize for Elasticsearch. (Half of free memory on host. Maximum 30GB. (default: Half of hosts memory. Min 1GB, Max 30GB)
 * *elasticsearch_tls_key_passphrase*: Passphrase for elasticsearch certificates (default: `PleaseChangeMeIndividually`)
@@ -53,6 +54,10 @@ This variable activates a workaround to start on systems that have certain harde
 * *elasticsearch_seed_hosts*: Set elasticsearch seed hosts
 * *elasticsearch_security_enrollment*: Controls enrollment (of nodes and Kibana) to a local node that’s been autoconfigured for security.
 
+The following variable was only integrated to speed up upgrades of non-production clusters. Use with caution and at your own risk:
+
+* *elasticsearch_unsafe_upgrade_restart*: This will still perform rolling upgrades, but will first update the package and then restart the service. In contrast the default behaviour is to stop the service, do the upgrade and then start again. (default: `false`)
+
 These variables are identical over all our elastic related roles, hence the different naming schemes.
 
 * *elasticstack_ca*: Set to the inventory hostname of the host that should house the CA for certificates for inter-node communication. (default: First node in the `elasticsearch` host group)

molecule/elasticstack_default/verify.yml

@@ -136,4 +136,3 @@
         success_msg: "'{{ item }}' was found in nodes.content"
       with_inventory_hostnames: all
       when: groups[elasticstack_elasticsearch_group_name] | length > 1
-

roles/beats/defaults/main.yml

@@ -6,7 +6,6 @@ beats_auditbeat: false
 beats_metricbeat: false
 beats_target_hosts:
   - localhost
-elasticstack_beats_port: 5044
 beats_logging: file
 beats_logpath: /var/log/beats
 beats_loglevel: info
@@ -58,23 +57,6 @@ beats_metricbeat_modules:
   - system
 beats_metricbeat_loadbalance: true
 
-elasticstack_release: 8
-elasticstack_full_stack: true
-elasticstack_variant: elastic
-elasticstack_security: true
-
-elasticstack_elasticsearch_group_name: elasticsearch
-elasticstack_logstash_group_name: logstash
-
-elasticstack_ca_dir: /opt/es-ca
-elasticstack_ca_pass: PleaseChangeMe
-elasticstack_initial_passwords: /usr/share/elasticsearch/initial_passwords
-elasticstack_elasticsearch_http_port: 9200
-elasticstack_no_log: true
 beats_cert_validity_period: 1095
 beats_cert_expiration_buffer: "+30d"
 beats_cert_will_expire_soon: false
-
-# Variables for debugging and development
-
-elasticstack_override_beats_tls: false

roles/beats/tasks/auditbeat.yml

@@ -16,20 +16,26 @@
     name: "{{ beats_auditbeat_package }}"
     enablerepo:
       - 'elastic-{{ elasticstack_release }}.x'
+  notify:
+    - Restart Auditbeat
   when:
     - ansible_os_family == "RedHat"
     - elasticstack_full_stack | bool
 
 - name: Install Auditbeat - rpm - standalone
   ansible.builtin.package:
     name: "{{ beats_auditbeat_package }}"
+  notify:
+    - Restart Auditbeat
   when:
     - ansible_os_family == "RedHat"
     - not elasticstack_full_stack | bool
 
 - name: Install Auditbeat - deb
   ansible.builtin.package:
     name: "{{ beats_auditbeat_package }}"
+  notify:
+    - Restart Auditbeat
   when:
     - ansible_os_family == "Debian"
 

roles/beats/tasks/beats-security.yml

@@ -1,17 +1,5 @@
 ---
 
-- name: Install packages for security tasks
-  ansible.builtin.package:
-    name:
-      - unzip
-      - python3-cryptography
-      - openssl
-  tags:
-    - certificates
-    - renew_ca
-    - renew_kibana_cert
-    - renew_beats_cert
-
 - name: Ensure beats certificate exists
   ansible.builtin.stat:
     path: "/etc/beats/certs/{{ inventory_hostname }}-beats.crt"

roles/beats/tasks/filebeat.yml

@@ -15,20 +15,26 @@
     name: "{{ beats_filebeat_package }}"
     enablerepo:
       - 'elastic-{{ elasticstack_release }}.x'
+  notify:
+    - Restart Filebeat
   when:
     - ansible_os_family == "RedHat"
     - elasticstack_full_stack | bool
 
 - name: Install Filebeat - rpm - standalone
   ansible.builtin.package:
     name: "{{ beats_filebeat_package }}"
+  notify:
+    - Restart Filebeat
   when:
     - ansible_os_family == "RedHat"
     - not elasticstack_full_stack | bool
 
 - name: Install Filebeat - deb
   ansible.builtin.package:
     name: "{{ beats_filebeat_package }}"
+  notify:
+    - Restart Filebeat
   when:
     - ansible_os_family == "Debian"
 

roles/beats/tasks/main.yml

@@ -1,10 +1,8 @@
 ---
 
-- name: Include OS specific vars
-  ansible.builtin.include_vars: '{{ item }}'
-  with_first_found:
-    - '{{ ansible_os_family }}_{{ ansible_distribution_major_version }}.yml'
-    - '{{ ansible_os_family }}.yml'
+- name: Include global role
+  ansible.builtin.import_role:
+    name: netways.elasticstack.elasticstack
 
 - name: Update apt cache.
   ansible.builtin.apt:
@@ -25,18 +23,6 @@
         - elasticstack_variant != "oss"
         - not elasticstack_override_beats_tls | bool
 
-    - name: Set elasticstack_ca variable if not already done by user
-      ansible.builtin.set_fact:
-        elasticstack_ca: "{{ groups[elasticstack_elasticsearch_group_name][0] }}"
-      when:
-        - beats_security | bool
-        - elasticstack_ca is undefined
-        - groups[elasticstack_elasticsearch_group_name] is defined
-      tags:
-        - certificates
-        - renew_ca
-        - renew_beats_cert
-
     - name: Set beats_ca_dir if whole stack is used
       ansible.builtin.set_fact:
         beats_ca_dir: "/etc/beats/certs"

roles/beats/tasks/metricbeat.yml

@@ -16,20 +16,26 @@
     name: "{{ beats_metricbeat_package }}"
     enablerepo:
       - 'elastic-{{ elasticstack_release }}.x'
+  notify:
+    - Restart Metricbeat
   when:
     - ansible_os_family == "RedHat"
     - elasticstack_full_stack | bool
 
 - name: Install Metricbeat - rpm - standalone
   ansible.builtin.package:
     name: "{{ beats_metricbeat_package }}"
+  notify:
+    - Restart Metricbeat
   when:
     - ansible_os_family == "RedHat"
     - not elasticstack_full_stack | bool
 
 - name: Install Metricbeat - deb
   ansible.builtin.package:
     name: "{{ beats_metricbeat_package }}"
+  notify:
+    - Restart Metricbeat
   when:
     - ansible_os_family == "Debian"
 

roles/beats/vars/Debian.yml

@@ -31,39 +31,18 @@ elasticsearch_heap_dump_path: "/var/lib/elasticsearch"
 
 elasticsearch_jna_workaround: false
 
-# The following variables are to be used when activating security
-# They follow a different naming scheme to show that they are global
-# to our set of Elastic Stack related Ansible roles
-
-# elasticstack_ca: First host in the `elasticsearch` group
-elasticstack_ca_dir: /opt/es-ca
-elasticstack_initial_passwords: /usr/share/elasticsearch/initial_passwords
 elasticsearch_initialized_file: "{{ elasticstack_initial_passwords | dirname }}/cluster_initialized"
-elasticstack_ca_name: "CN=Elastic Certificate Tool Autogenerated CA"
-elasticstack_ca_pass: PleaseChangeMe
-elasticstack_ca_validity_period: 1095
 elasticsearch_tls_key_passphrase: PleaseChangeMeIndividually
 elasticsearch_cert_validity_period: 1095
-elasticstack_ca_expiration_buffer: 30
 elasticsearch_cert_expiration_buffer: 30
-elasticstack_ca_will_expire_soon: false
 elasticsearch_cert_will_expire_soon: false
 elasticsearch_ssl_verification_mode: full
 
+# use this only for non-prod environments and at your own risk!
+elasticsearch_unsafe_upgrade_restart: false
+
 # only used internally
 elasticsearch_freshstart:
   changed: false
 elasticsearch_freshstart_security:
   changed: false
-
-# "global" variables for all roles
-
-elasticstack_release: 8
-elasticstack_full_stack: true
-elasticstack_variant: elastic
-elasticstack_elasticsearch_http_port: 9200
-elasticstack_no_log: true
-
-elasticstack_elasticsearch_group_name: elasticsearch
-elasticstack_logstash_group_name: logstash
-elasticstack_kibana_group_name: kibana