Upgrade el8/changes

[widhalmt]

Minor fixes for problems that were found during tests of the collection (most were previously undetected in the separate roles). And fixes for a stable handling of Elastic Stack version 8.

ToDos

Here are some items we discovered during testing. They need to be addressed in one way or another. Either fix them here or create an issue to fix it. (Tick them off, when a fix is in this branch, in main or when an issue is open)

• @pdolinic / @afeefghannam89 Critical: Make sure the users /tmp directory is cleared of the old .p12 & cert file, before another rollout - Idea: Delete files in local /tmp after copying to target host. Will increase traffic, but make overall system more secure and stable for rebuilds. Alternatively create a unique temp directory (e.g. with mktemp) and use that. Partially focused in Make sure certificate handling is idempotent and reacts to changes #35
• @afeefghannam89 Critical: Certificates for TLS between Beats and Logstash seem to be broken. We need a fix for that. See comments for error message. -> Fix TLS certificates for Beats connection #40
• Investigate behavior of IP vs Hostnames in inventory. - Idea: ip address, arbitrary inventory name, hostname in inventory etc.
• Optional: Kibana Web default Certificate - Idea: Use netways.ca role or have a small autogenerated certificate as default. Don't break different scenarios like company CA, local CA, letsencrypt etc. -> Provide examples for Kibana web TLS certificates #39
• @pdolinic Optinal: Default Password for PleaseChangeMe - Idea: Create random passwords for everything where a default password is set and find a way so securly show it to the user. E.g. put them in a file with restricted access. -> first release testing of pwgenerator #38
• Handle hosts with more than one IP address. Currently there are hardcoded addresses chosen from Ansible facts and you can't choose which to use. This can (and did) lead to broken certificates. -> Handle hosts with multiple IP addresses #22
• Allow for disabling TLS connection between Beats and Logstash. It regularly gave problems in the past (due to bugs in upstream code) and it would help at least with developing and testing.
• Disable cluster discovery for single node clusters >= 8. There's a warning in the logs we could suppress with that -> Disable cluster discovery for single node clusters in version 8 #8 / Disable discovery on single node clusters #9
• Fix system user for Kibana to access Elasticseach -> "kibana" user doesn't work anymore #20
• The logstash_writer user seems to be created but is not allowed to access Elasticsearch
• Make sure to use automatic security of Elastic Stack 8 if that makes sense. We shouldn't pervert ES8 to run like ES7. ES 8 still allows to set everything manuall. Automatic security is only provided for convenience.
• @widhalmt Introduce checks for version 8 (and keep them for version 7) - see 8e44781 for an example
• Set default of elastic_version to 8
• Enable "full stack mode" by default. But allow to deactivate (e.g. for OpenSearch based installations)
• Test if security is really a must with version 8 -> Upgrade el8/changes #5 (comment) -> https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-stack-security.html#configuring-stack-security you can explicitly deactivate security even in 8. That's only interesting for testing, though.
• List, which passwords can be changed by Ansible after initial setup and which can't.

Tests

Every test is run with Elastic Stack version 8. oss variant is not supported in versions > 7 so no tests are neccessary.

• Single node with default variables (as much as possible) on Ubuntu 22.04 LTS
• Single node with default variables (as much as possible) on RedHat 8 (or similar)
• Multinode cluster with default variables (as much as possible) on Ubuntu 22.04 LTS
• Multinode cluster with default variables (as much as possible) on RedHat 8 (or similar)

[widhalmt]

The PKCS8 key we create seems to look invalid to Logstash:

[2023-02-01T07:39:18,401][ERROR][logstash.inputs.beats    ][ansible-input] SSL configuration invalid {:exception=>Java::JavaLang::IllegalArgumentException, :message=>"File does not contain valid private key: /etc/logstash/certs/twidhalm-elasticcollection.novalocal-pkcs8.key", :cause=>{:exception=>Java::JavaSecurity::NoSuchAlgorithmException, :message=>"1.2.840.113549.1.5.13 SecretKeyFactory not available"}}
[2023-02-01T07:39:18,500][ERROR][logstash.javapipeline    ][ansible-input] Pipeline error {:pipeline_id=>"ansible-input", :exception=>#<LogStash::ConfigurationError: File does not contain valid private key: /etc/logstash/certs/twidhalm-elasticcollection.novalocal-pkcs8.key>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:239:in `new_ssl_handshake_provider'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:193:in `create_server'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.6-java/lib/logstash/inputs/beats.rb:178:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:391:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:316:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/ansible-input/input.conf", "/etc/logstash/conf.d/ansible-input/output.conf"], :thread=>"#<Thread:0x1a3df954 run>"}
[2023-02-01T07:39:18,501][INFO ][logstash.javapipeline    ][ansible-input] Pipeline terminated {"pipeline.id"=>"ansible-input"}
[2023-02-01T07:39:18,511][ERROR][logstash.agent           ] Failed to execute action {:id=>:"ansible-input", :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<ansible-input>, action_result: false", :backtrace=>nil}

[widhalmt]

I disabled a check for enabled security in https://github.com/NETWAYS/ansible-collection-elasticstack/pull/5/files#diff-4001bacb15c0da113993bf42c647ba9b0445d698d6a2b40733e0fbf2ed1c2397L2-R8 . Honestly, I don't know if this just works by accident, maybe only with single node clusters. It did work in my tests but we need to test more thoroughly before removing the code.

[widhalmt]

I added a new milestone: https://github.com/NETWAYS/ansible-collection-elasticstack/milestone/4 . We can use it to keep track of things we stumble upon that only work with version 7. Thanks @danopt for the contribution.