Add output of Mermaid syntax overview to Logstash

[widhalmt]

This feature was born out of an Elastic Stack workshop I did with customers. We used Mermaid to visualize how pipelines are interconnected. So the idea was born that it might be very helpful if the collection could create Mermaid code automatically.

I tried the following to test this code.

Added the following variable to group_vars/logstash.yml.

logstash_pipelines:
  - name: sorter
    exclusive: true
    input:
      - name: input
        key: input
    output:
      - name: syslog
        key: syslog-input
        condition: '[log][file][path] == "/var/log/syslog"'
        congestion: 5000
      - name: apache
        key: apache
        condition: '[log][file][path] =~ /\/var\/log\/apache2\/.*access.*log$/'
        congestion: 5000
      - name: apache-error
        key: apache-error
        condition: '[log][file][path] =~ /\/var\/log\/apache2\/.*error.*log$/'
        congestion: 5000
      - name: haproxy
        key: syslog-input
        condition: '[log][file][path] == "/var/log/haproxy.log"'
        congestion: 5000
      - name: mysql-error
        key: mysql-error
        condition: '[log][file][path] == "/var/log/mysql/error.log" or [log][file][path] == "/var/log/mysql/mysql.err"'
        congestion: 5000
      - name: mysql-slow
        key: mysql-slow
        condition: '[log][file][path] =~ /\/var\/log\/mysql\/.*-slow.log/'
        congestion: 5000
      - name: haproxy
        key: syslog-input
        condition: '[log][file][path] =~ /^\/var\/log\/mail/'
        congestion: 5000
      - name: forwarder
        key: forwarder
        congestion: 5000
  - name: syslog
    source: https://github.com/NETWAYS/syslog-logstash-pipeline.git
    version: master
    exclusive: true
    input:
      - name: syslog-input
        key: syslog-input
    output:
      - name: postfix
        condition: '[program] =~ /^postfix/ or [log][file][path] =~ /^\/var\/log\/mail/'
        key: postfix
        congestion: 5000
      - name: haproxy
        key: haproxy
        condition: '[program] == "haproxy"'
        congestion: 5000
      - name: forwarder
        key: forwarder
        congestion: 5000
  - name: postfix
    source: https://github.com/NETWAYS/postfix-logstash-pipeline.git
    version: master
    exclusive: false
    input:
      - name: postfix
        key: postfix
    output:
      - name: forwarder
        key: forwarder
        congestion: 5000
  - name: apache
    source: https://github.com/widhalmt/apache-access-logstash-pipeline.git
    version: master
    exclusive: false
    input:
      - name: apache
        key: apache
    output:
      - name: forwarder
        key: forwarder
        congestion: 5000
  - name: apache-error
    #source: https://github.com/widhalmt/apache-error-logstash-pipeline.git
    #version: main
    exclusive: false
    input:
      - name: apache-error
        key: apache-error
    output:
      - name: forwarder
        key: forwarder
        congestion: 5000
  - name: haproxy
    source: https://github.com/widhalmt/haproxy-logstash-pipeline.git
    version: master
    exclusive: false
    input:
      - name: haproxy
        key: haproxy
    output:
      - name: forwarder
        key: forwarder
        congestion: 5000
  - name: mysql-error
    source: https://github.com/widhalmt/mysql-error-logstash-pipeline.git
    version: main
    exclusive: false
    input:
      - name: mysql-error
        key: mysql-error
    output:
      - name: forwarder
        key: forwarder
        congestion: 5000
  - name: mysql-slow
    source: https://github.com/widhalmt/mysql-slowlog-logstash-pipeline.git
    version: main
    exclusive: false
    input:
      - name: mysql-slow
        key: mysql-slow
    output:
      - name: forwarder
        key: forwarder
        congestion: 1000

This resulted in the following contents in /etc/logstash/pipelines.mermaid.

# Managed via Ansible role
# https://github.com/netways/ansible-role-logstash

# Use the following code with your favorite Mermaid editor
# Or paste into: https://mermaid.live/
# To get a graphical overview of your Logstash pipelines

flowchart TD
p_ansible-input[ansible-input] --> k_input{input}
k_forwarder{forwarder} --> p_ansible-forwarder[ansible-forwarder]
k_input{input} --> p_sorter[sorter]
p_sorter[sorter] --> |if log.file.path == /var/log/syslog|k_syslog-input{syslog-input}
p_sorter[sorter] --> |if log.file.path =~ /\/var\/log\/apache2\/.*access.*log$/|k_apache{apache}
p_sorter[sorter] --> |if log.file.path =~ /\/var\/log\/apache2\/.*error.*log$/|k_apache-error{apache-error}
p_sorter[sorter] --> |if log.file.path == /var/log/haproxy.log|k_syslog-input{syslog-input}
p_sorter[sorter] --> |if log.file.path == /var/log/mysql/error.log or log.file.path == /var/log/mysql/mysql.err|k_mysql-error{mysql-error}
p_sorter[sorter] --> |if log.file.path =~ /\/var\/log\/mysql\/.*-slow.log/|k_mysql-slow{mysql-slow}
p_sorter[sorter] --> |if log.file.path =~ /^\/var\/log\/mail/|k_syslog-input{syslog-input}
p_sorter[sorter] --> k_forwarder{forwarder}
k_syslog-input{syslog-input} --> p_syslog[syslog]
p_syslog[syslog] --> |if program =~ /^postfix/ or log.file.path =~ /^\/var\/log\/mail/|k_postfix{postfix}
p_syslog[syslog] --> |if program == haproxy|k_haproxy{haproxy}
p_syslog[syslog] --> k_forwarder{forwarder}
k_postfix{postfix} --> p_postfix[postfix]
p_postfix[postfix] --> k_forwarder{forwarder}
k_apache{apache} --> p_apache[apache]
p_apache[apache] --> k_forwarder{forwarder}
k_apache-error{apache-error} --> p_apache-error[apache-error]
p_apache-error[apache-error] --> k_forwarder{forwarder}
k_haproxy{haproxy} --> p_haproxy[haproxy]
p_haproxy[haproxy] --> k_forwarder{forwarder}
k_mysql-error{mysql-error} --> p_mysql-error[mysql-error]
p_mysql-error[mysql-error] --> k_forwarder{forwarder}
k_mysql-slow{mysql-slow} --> p_mysql-slow[mysql-slow]
p_mysql-slow[mysql-slow] --> k_forwarder{forwarder}

When I post it into an online Merm

aid Editor, then I get the following: