Allow disabling repos per default (NETWAYS#108)

This introduces a new variable that will disable yum repositories per
default and re-enables them temporarily when modifying packages. It's
not possible to disable apt repositories temporarily so we need to have
two different approaches.

fixes NETWAYS#106
fixes NETWAYS#76
fixes NETWAYS#98
supersedes NETWAYS#101

Author: widhalmt

README.md

@@ -2,8 +2,6 @@
 
 This collection installs and manages the Elastic Stack. It provides roles every component which is part of the Stack. Furthermore it is possible to differentiate between Enterprise or OSS releases. Every role is documented with all variables, please refer to the documentation found in **[Getting-Started](./docs/getting-started.md)**
 
-
-
 ## Roles Documentation
 
 * [Beats](docs/role-beats.md)
@@ -56,6 +54,10 @@ We have known issues with the following Distributions.
 
 ## Usage
 
+* *elastic_version*: Version number of tools to install Only set if you don't want the latest. (default: none). For OSS version see `elastic_variant` below. **IMPORTANT** Do not change the version once you have set up the stack. There are unpredictable effects to be expected when using this for upgrades. And upgrade mechanism is already on it's way. (default: none. Example: `7.17.2`
+*elastic_release*: Major release version of Elastic stack to configure. (default: `7`)
+*elastic_variant*: Variant of the stack to install. Valid values: `elastic` or `oss`. (default: `elastic`)
+
 Make sure all hosts that should be configured are part of your playbook. (See below for details on groups etc.). The collection is built to first collect all facts from all hosts (including those only running beats) and then use facts like hostnames or ip addresses to connect the tools to each other.
 
 You will want to have reliable DNS resolution or enter all hosts of the stack into your systems hosts files.

docs/role-beats.md

@@ -14,7 +14,6 @@ Role Variables
 --------------
 
 * *beats_filebeat*: Install and manage filebeat (Default: `true`)
-* *beats_filebeat_version*: Install specific version (Default: none. Possible values: e.g. `-7.10.1` for RedHat compatible systems or `=1:7.10.1-1` for Debian compatible systems or `latest`)
 * *filebeat_enable*: Automatically start Filebeat (Default: `true`)
 * *filebeat_output*: Set to `logstash` or `elasticsearch`. (default: `logstash`)
 * *filebeat_syslog_udp*: Use UDP Syslog input (Default: `false`)
@@ -66,14 +65,12 @@ filebeat_journald_inputs:
 * *filebeat_modules*: **EXPERIMENTAL**: Give a list of modules to enable. (default: none)
 
 * *beats_auditbeat*: Install and manage filebeat (Default: `false`)
-* *beats_auditbeat_version*: Install specific version (Default: none. Possible values: e.g. `-7.10.1` for RedHat compatible systems or `=1:7.10.1-1` for Debian compatible systems or `latest`)
 * *auditbeat_output*: Output for Auditbeat Set to `logstash` or `elasticsearch`. (default: `elasticsearch`)
 * *auditbeat_enable*: Automatically start Auditbeat (Default: `true`)
 * *auditbeat_setup*: Run Auditbeat Setup (Default: `true`) (Only works with Elasticsearch output)
 * *auditbeat_loadbalance*: Enable loadbalancing for Auditbeats Logstash output (default: `true`)
 
 * *beats_metricbeat*: Enable installation and management of Metricbeat (Default: `false`)
-* *beats_metricbeat_version*: Install specific version (Default: none. Possible values: e.g. `-7.10.1` for RedHat compatible systems or `=1:7.10.1-1` for Debian compatible systems or `latest`)
 * *metricbeat_enable*: Start Metricbeat automatically (Default: `true`)
 * *metricbeat_output*: Set to `logstash` or `elasticsearch`. (default: `elasticsearch`)
 * *metricbeat_modules*: List of modules to enable. (Default: `- system`)
@@ -96,6 +93,7 @@ The following variables only apply if you use this role together with our other
 * *elastic_ca_dir*: Directory where on the Elasticsearch CA host certificates are stored. This is only useful in connection with out other Elastic Stack related roles. (default: `/opt/es-ca`)
 * *elastic_ca_pass*: Password for Elasticsearch CA (default: `PleaseChangeMe`)
 * *elastic_initial_passwords*: Path to file with initical elasticsearch passwords (default: `/usr/share/elasticsearch/initial_passwords`)
+* *elastic_version*: Install specific version (Default: none. Possible values: e.g. `7.10.1` or `latest`)
 
 If you want to use this role with your own TLS certificates, use these variables.
 

docs/role-logstash.md

@@ -29,7 +29,7 @@ If you want to use the default pipeline (or other pipelines communicating via Re
 Role Variables
 --------------
 
-* *logstash_version*: Version number of Logstash to install (use os specific version string. e.g. `-7.10.1` for RedHat compatible systems or `=1:7.10.1-1` for Debian compatible systems). Only set if you don't want the latest. (default: none). For OSS version see `elastic_variant` below.
+* *elastic_version*: Version number of Logstash to install (e.g. `7.10.1`). Only set if you don't want the latest. (default: none). For OSS version see `elastic_variant` below.
 * *logstash_enable*: Start and enable Logstash service (default: `true`)
 * *logstash_config_backup*: Keep backups of all changed configuration (default: `no`)
 * *logstash_manage_yaml*: Manage and overwrite `logstash.yml` (default: `true`)

docs/role-repos.md

@@ -18,12 +18,21 @@ Role Variables
 
 * *elastic_release*: Major release version of Elastic stack to configure. (default: `7`). `7` and `8` are supported.
 * *elastic_variant*: Variant of the stack to install. Valid values: `elastic` or `oss`. (default: `elastic`).
+* *elastic_enable_repos*: Enable repositories after creating them. (default: `true`) Only works on RPM based distributions!
 
 Please note that no `oss` versions are available for Elastic Stack later than `7`. This role will fail if you try to install them.
 
 Usage
 --------
 
+Upgrades
+========
+
+If you want to be able to update your operating system without worrying about accidentally upgrading Elastic Stack, set `elastic_enable_repos` to `false`. The roles in this collection will enable the repository in case they need it. Keep in mind that this will only work on rpm based distributions.
+
+Example playbook
+================
+
 ```
   - hosts: all
     become: yes

molecule/beats_peculiar/converge.yml

@@ -23,7 +23,6 @@
     elastic_stack_full_stack: false
     filebeat_mysql_slowlog_input: true
     beats_auditbeat: true
-    beats_auditbeat_version: latest
     auditbeat_output: logstash
     auditbeat_enable: false # can't run on GitHub because of permissions
     filebeat_journald_inputs:
@@ -36,18 +35,18 @@
     #filebeat_docker: true
     elastic_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
   tasks:
-    # Looks like Elastic isn't providing all old releases
-    # anymore
-    #
-    #- name: Set Filebeat version on RedHat
-    #  set_fact:
-    #    beats_filebeat_version: "-7.16.1"
-    #  when: ansible_os_family == "RedHat"
 
-    #- name: Set Filebeat version on Debian
-    #  set_fact:
-    #    beats_filebeat_version: "=7.16.1"
-    #  when: ansible_os_family == "Debian"
+    - name: Set Filebeat version for 7.x
+      set_fact:
+        elastic_version: "7.17.1"
+      when:
+        - elastic_release == 7
+
+    - name: Set Filebeat version for 8.x
+      set_fact:
+        elastic_version: "8.4.1"
+      when:
+        - elastic_release == 8
 
     - name: "Include Elastics repos role"
       include_role:

molecule/beats_peculiar/verify.yml

@@ -2,6 +2,8 @@
 
 - name: Check if Logstash configuration does what it should
   hosts: all
+  vars:
+    elastic_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
   tasks:
   - name: fetch Filebeat version
     command: "filebeat version | grep ^filebeat"
@@ -11,8 +13,16 @@
     debug:
       var: filebeat_version.stdout
 
-  #- name: Fail if Filebeat has the wrong version
-  #  fail:
-  #    msg: "Filebeat has the wrong version"
+  - name: Fail if Filebeat has the wrong version
+    fail:
+      msg: "Filebeat has the wrong version"
+    when:
+      - filebeat_version.stdout.find('7.17.1') == -1
+      - elastic_release == 7
 
-  #  when: filebeat_version.stdout.find('7.16.1') == -1
+  - name: Fail if Filebeat has the wrong version
+    fail:
+      msg: "Filebeat has the wrong version"
+    when:
+      - filebeat_version.stdout.find('8.4.1') == -1
+      - elastic_release == 8

molecule/logstash_full_stack-oss/verify.yml

@@ -3,6 +3,8 @@
 
 - name: Check if Logstash configuration does what it should
   hosts: all
+  vars:
+    elastic_elasticsearch_http_port: 9200
   tasks:
   - name: Give some time for tools to connect
     wait_for:

molecule/logstash_full_stack/verify.yml

@@ -3,6 +3,8 @@
 
 - name: Check if Logstash configuration does what it should
   hosts: all
+  vars:
+    elastic_elasticsearch_http_port: 9200
   tasks:
   - name: Give some time for tools to connect
     wait_for:

molecule/logstash_specific_version/converge.yml

@@ -12,24 +12,26 @@
     logstash_logging_console: false
     logstash_logging_slow_file: false
     logstash_pipeline_identifier: false
-    elastic_release: 7
+    elastic_release: "{{ lookup('env', 'ELASTIC_RELEASE') | int}}"
     elastic_stack_full_stack: false
   tasks:
 
-  - name: Set Logstash version on RedHat
-    set_fact:
-      logstash_version: "-7.10.1"
-    when: ansible_os_family == "RedHat"
+    - name: Set Filebeat version for 7.x
+      set_fact:
+        elastic_version: "7.17.1"
+      when:
+        - elastic_release == 7
 
-  - name: Set Logstash version on Debian
-    set_fact:
-      logstash_version: "=1:7.10.1-1"
-    when: ansible_os_family == "Debian"
+    - name: Set Filebeat version for 8.x
+      set_fact:
+        elastic_version: "8.4.1"
+      when:
+        - elastic_release == 8
 
-  - name: "Include Elastics repos role"
-    include_role:
-      name: repos
+    - name: "Include Elastics repos role"
+      include_role:
+        name: repos
 
-  - name: "Include Logstash"
-    include_role:
-      name: logstash
+    - name: "Include Logstash"
+      include_role:
+        name: logstash

molecule/logstash_specific_version/molecule.yml

@@ -4,7 +4,7 @@ dependency:
 driver:
   name: docker
 platforms:
-  - name: logstash_version
+  - name: elastic_version
     image: "geerlingguy/docker-${MOLECULE_DISTRO:-centos7}-ansible:latest"
     command: ${MOLECULE_DOCKER_COMMAND:-""}
     volumes: