Skip to content

Latest commit

 

History

History
100 lines (82 loc) · 4.8 KB

cluster-logging-log-forwarding-about.adoc

File metadata and controls

100 lines (82 loc) · 4.8 KB

Understanding cluster log forwarding

The {product-title} cluster log forwarding feature uses a combination of outputs and pipelines defined in the Log Forwarding Custom Resource to send logs to specific endpoints inside and outside of your {product-title} cluster.

Note

If you want to use only the default internal {product-title} Elasticsearch instance, do not configure any outputs and pipelines.

An output is the destination for log data and a pipeline defines simple routing for one source to one or more outputs.

An output can be either:

  • elasticsearch to forward logs to an external Elasticsearch v5.x cluster, specified by server name or FQDN, and/or the internal {product-title} Elasticsearch instance.

  • forward to forward logs to an external log aggregation solution. This option uses the Fluentd forward plug-ins.

A pipeline associates the type of data to an output. A type of data you can forward is one of the following:

  • logs.app - Container logs generated by user applications running in the cluster, except infrastructure container applications.

  • logs.infra - Logs generated by both infrastructure components running in the cluster and {product-title} nodes, such as journal logs. Infrastructure components are pods that run in the openshift*, kube*, or default projects.

  • logs.audit - Logs generated by the node audit system (auditd), which are stored in the /var/log/audit/audit.log file, and the audit logs from the Kubernetes apiserver and the OpenShift apiserver.

Note the following:

  • The internal {product-title} Elasticsearch instance does not provide secure storage for audit logs. We recommend you ensure that the system to which you forward audit logs is compliant with your organizational and governmental regulations and is properly secured. {product-title} cluster logging does not comply with those regulations.

  • An output supports TLS communication using a secret. Secrets must have keys of: tls.crt, tls.key, and ca-bundler.crt which point to the respective certificates for which they represent. Secrets must have the key shared_key for use when using forward in a secure manner.

  • You are responsible to create and maintain any additional configurations that external destinations might require, such as keys and secrets, service accounts, port opening, or global proxy configuration.

The following example creates three outputs:

  • the internal {product-title} Elasticsearch instance,

  • an unsecured externally-managed Elasticsearch instance,

  • a secured external device using the forward plug-in.

Three pipelines send:

  • the application logs to the internal {product-title} Elasticsearch,

  • the infrastructure logs to an external Elasticsearch instance,

  • the audit logs to the secured device over the forward plug-in.

Sample log forwarding outputs and pipelines
apiVersion: "logging.openshift.io/v1alpha1"
kind: "LogForwarding"
metadata:
  name: instance (1)
  namespace: openshift-logging
spec:
  disableDefaultForwarding: true (2)
  outputs: (3)
   - type: "elasticsearch" (4)
     name: elasticsearch (5)
     endpoint: elasticsearch.openshift-logging.svc:9200 (6)
     secret: (7)
        name: fluentd
   - type: "elasticsearch"
     name: elasticsearch-insecure
     endpoint: elasticsearch-insecure.svc.messaging.cluster.local
     insecure: true (8)
   - type: "forward"
     name: secureforward-offcluster
     endpoint: https://secureforward.offcluster.com:9200
     secret:
        name: secureforward
  pipelines: (9)
   - name: container-logs (10)
     inputSource: logs.app (11)
     outputRefs: (12)
     - elasticsearch
     - secureforward-offcluster
   - name: infra-logs
     inputSource: logs.infra
     outputRefs:
     - elasticsearch-insecure
   - name: audit-logs
     inputSource: logs.audit
     outputRefs:
     - secureforward-offcluster

  1. The name of the log forwarding CR must be instance.

  2. Parameter to enable log forwarding. Set to true to enable log forwarding.

  3. Sets of outputs.

  4. The type of output, either elasticsearch or forward.

  5. A name to describe the output.

  6. The log forwarding endpoint, either the server name or FQDN. For the internal {product-title} Elasticsearch instance, specify elasticsearch.openshift-logging.svc:9200.

  7. Optional name of the secret required by the endpoint for TLS communication. The secret must exist in the openshift-logging project.

  8. Optional setting if the endpoint does not use a secret, resulting in insecure communication.

  9. Sets of pipelines.

  10. A name to describe the pipeline.

  11. The source type, logs.app, logs.infra, or logs.audit.

  12. The name of one or more outputs configured in the CR.