diff --git a/README.md b/README.md index 22c9ad9..aa9db5b 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,5 @@ # Kubernetes Logging with Fluent Bit - - [Fluent Bit](http://fluentbit.io) is a lightweight and extensible __Log Processor__ that comes with full support for Kubernetes: - Read Kubernetes/Docker log files from the file system or through systemd Journal @@ -16,72 +14,36 @@ This repository contains a set of Yaml files to deploy Fluent Bit which consider ``` $ kubectl create namespace logging -$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml -$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml -$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml +$ kubectl apply -k ./base ``` +(`./base` can be replaced with a URL if [#90](https://github.com/fluent/fluent-bit-kubernetes-logging/pull/90) gets merged) + If you are deploying fluent-bit on openshift, you additionally need to run: ``` $ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-openshift-security-context-constraints.yaml ``` -#### Fluent Bit to Elasticsearch - -The next step is to create a ConfigMap that will be used by our Fluent Bit DaemonSet: - -``` -$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml -``` - -If the cluster uses a CRI runtime, like containerd or CRI-O, change the `Parser` described in `input-kubernetes.conf` from docker to cri. - -Fluent Bit DaemonSet ready to be used with Elasticsearch on a normal Kubernetes Cluster: - -``` -$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds.yaml -``` - -#### Fluent Bit to Elasticsearch on Minikube - -If you are using Minikube for testing purposes, use the following alternative DaemonSet manifest: - -``` -$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds-minikube.yaml -``` - -#### Fluent Bit to Kafka - -Create a ConfigMap that will be used by our Fluent Bit DaemonSet: - -``` -$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/kafka/fluent-bit-configmap.yaml -``` - -Fluent Bit DaemonSet ready to be used with Kafka on a normal Kubernetes Cluster: +The base only configures [Stdout](https://docs.fluentbit.io/manual/pipeline/outputs/standard-output) output. +Log records are found using `kubectl logs` on fluentbit pods. +This is a good way to experiment with configuration. +Switch to [Format](https://docs.fluentbit.io/manual/pipeline/outputs/standard-output#configuration-parameters) `msgpack` to also see the "tag" (helpful when developing your pipeline). -``` -$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/kafka/fluent-bit-ds.yaml -``` +## Logs ingested to Loki -#### Fluent Bit to Elasticsearch on Minikube +This repository serves as example of Kubernetes yaml for a log processing stack, +not as example of [how](https://docs.fluentbit.io/manual/concepts/data-pipeline) Fluent-bit can process, refine and forward log entries. -If you are using Minikube for testing purposes, use the following alternative DaemonSet manifest: - -``` -$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-ds-minikube.yaml -``` +Use the [loki](./loki) base to replace Stdout with forwarding to a standalone Loki instance. Loki can do some processing at [query time](https://grafana.com/docs/loki/latest/logql/) which helps keep this repo light on pipeline config. -## Details +The [loki-plugin](./loki-plugin) base uses an out-of-tree plugin with `DropSingleKey` +so that the log records forwarded are the actual logged lines from pods, +but still queryable on labels like `pod` and `container`. -The default configuration of Fluent Bit makes sure of the following: +See the [lokitest](./loki/test/lokitest-job.yaml) Job for some examples of how to consume logs through Loki. -- Consume all containers logs from the running Node. -- The [Tail input plugin](http://fluentbit.io/documentation/0.12/input/tail.html) will not append more than __5MB__ into the engine until they are flushed to the Elasticsearch backend. This limit aims to provide a workaround for [backpressure](http://fluentbit.io/documentation/0.13/configuration/backpressure.html) scenarios. -- The Kubernetes filter will enrich the logs with Kubernetes metadata, specifically _labels_ and _annotations_. The filter only goes to the API Server when it cannot find the cached info, otherwise it uses the cache. -- The default backend in the configuration is Elasticsearch set by the [Elasticsearch Output Plugin](http://fluentbit.io/documentation/0.13/output/elasticsearch.html). It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments. -- There is an option called __Retry_Limit__ set to False that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeeds. +The loki folders also serve as [example](./loki/kustomization.yaml) of how to use Kustomize to override selected parts of the `base`'s *.conf. ## Get in touch with us! diff --git a/base/filter-kubernetes.conf b/base/filter-kubernetes.conf new file mode 100644 index 0000000..642586d --- /dev/null +++ b/base/filter-kubernetes.conf @@ -0,0 +1,11 @@ + +[FILTER] + Name kubernetes + Match kube.* + Kube_Tag_Prefix kube. + Regex_Parser kube-tag + # Merge_Log On + # Merge_Log_Key log_processed + K8S-Logging.Parser On + K8S-Logging.Exclude On + Annotations Off diff --git a/base/fluent-bit.conf b/base/fluent-bit.conf new file mode 100644 index 0000000..2c536d6 --- /dev/null +++ b/base/fluent-bit.conf @@ -0,0 +1,13 @@ +[SERVICE] + Flush 1 + Log_Level ${LOG_LEVEL} + Daemon off + Parsers_File parser-cri.conf + Parsers_File parsers.conf + HTTP_Server On + HTTP_Listen 0.0.0.0 + HTTP_Port 2020 + +@INCLUDE input-kubernetes.conf +@INCLUDE filter-kubernetes.conf +@INCLUDE outputs.conf diff --git a/base/fluentbit-daemonset.yaml b/base/fluentbit-daemonset.yaml new file mode 100644 index 0000000..dabea98 --- /dev/null +++ b/base/fluentbit-daemonset.yaml @@ -0,0 +1,73 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: fluentbit + labels: + app.kubernetes.io/name: fluentbit +spec: + selector: + matchLabels: + app.kubernetes.io/name: fluentbit + template: + metadata: + labels: + app.kubernetes.io/name: fluentbit + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "2020" + prometheus.io/path: /api/v1/metrics/prometheus + spec: + serviceAccountName: fluentbit + terminationGracePeriodSeconds: 25 + tolerations: + - key: node-role.kubernetes.io/master + operator: Exists + effect: NoSchedule + containers: + - name: fluentbit + image: docker.io/fluent/fluent-bit + command: + - /fluent-bit/bin/fluent-bit + - -c + - /fluent-bit/etc/fluent-bit.conf + env: + - name: LOG_LEVEL + value: debug + - name: MEM_BUF_LIMIT + value: 5MB + ports: + - name: http + containerPort: 2020 + livenessProbe: + httpGet: + path: / + port: http + readinessProbe: + httpGet: + path: /api/v1/health + port: http + volumeMounts: + - name: varlogpods + mountPath: /var/log/pods + readOnly: true + - name: db + mountPath: /var/run/fluentbit + - name: config + mountPath: /fluent-bit/etc + # - mountPath: /etc/machine-id + # name: etcmachineid + # readOnly: true + volumes: + - name: varlogpods + hostPath: + path: /var/log/pods + - name: db + hostPath: + path: /var/run/fluentbit + # - name: etcmachineid + # hostPath: + # path: /etc/machine-id + # type: File + - name: config + configMap: + name: fluentbit diff --git a/base/input-kubernetes.conf b/base/input-kubernetes.conf new file mode 100644 index 0000000..d457ae3 --- /dev/null +++ b/base/input-kubernetes.conf @@ -0,0 +1,11 @@ +[INPUT] + Name tail + DB /var/run/fluentbit/tail-positions.db + Mem_Buf_Limit ${MEM_BUF_LIMIT} + Skip_Long_Lines On + DB.locking true + Path /var/log/pods/*/*/*.log + Exclude_Path /var/log/pods/*/fluentbit/*.log + Parser cri + Tag kube.... + Tag_Regex /var/log/pods/(?[^_/]+)_(?[^_/]+)_(?[^_/]+)/(?[^/]+)/.* diff --git a/base/kustomization.yaml b/base/kustomization.yaml new file mode 100644 index 0000000..89d2813 --- /dev/null +++ b/base/kustomization.yaml @@ -0,0 +1,25 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: logging + +images: +- name: docker.io/fluent/fluent-bit + newTag: 1.8.3@sha256:10ea2709cef6e7059d980b4969d5f9d753ef97278a817c214cbe9120b1152082 + +commonLabels: + app.kubernetes.io/name: fluentbit + +resources: +- ../rbac +- fluentbit-daemonset.yaml + +configMapGenerator: +- name: fluentbit + files: + - fluent-bit.conf + - parser-cri.conf + - input-kubernetes.conf + - filter-kubernetes.conf + - parsers.conf + - outputs.conf diff --git a/base/outputs.conf b/base/outputs.conf new file mode 100644 index 0000000..fa4ac9e --- /dev/null +++ b/base/outputs.conf @@ -0,0 +1,6 @@ +[OUTPUT] + Name stdout + # Switch to format msgpack to see the tags + Format json_lines + json_date_key t + json_date_format iso8601 diff --git a/base/parser-cri.conf b/base/parser-cri.conf new file mode 100644 index 0000000..635c71a --- /dev/null +++ b/base/parser-cri.conf @@ -0,0 +1,12 @@ + +[PARSER] + Name cri + Format regex + Regex ^(?