Does it work with ES datastreams?

[applike-ss]

For the opstree kubernetes-event-exporter i had to hack together a solution to support elasticsearch datastreams, see: opsgenie#165

I know that this is far from ideal, but i needed something to worked quickly at that point and couldn't spend more time on making it proper.

Has a fix like this (or proper 😉 ) been applied to this fork?

[azlev]

datastreams need the @timestamp field. I also couldn't point to one.

@applike-ss I read your PR opsgenie#165 and I think it's a good solution. I suggest to send the PR 🙂 .

[azlev]

a workaround to people willing to use datastreams right now:

PUT _ingest/pipeline/kubernetes-events
{
  "version": 1,
  "description": "Put @timestamp in document.",
  "processors": [
    {
      "set": {
        "field": "@timestamp",
        "override": false,
        "copy_from": "firstTimestamp"
      }
    }
  ]
}
PUT _index_template/ds-kubernetes-events
{
  "priority": 600,
  "template": {
    "settings": {
      "index": {
        "default_pipeline": "kubernetes-events",
        "refresh_interval": "30s"
      }
    }
  },
  "index_patterns": [
    "ds-kubernetes-events*"
  ],
  "data_stream": {
    "hidden": false
  },
  "composed_of": []
}

[fernferret]

I realize the question here was for ElasticSearch, but I stumbled upon this issue when trying to make DataStreams work with OpenSearch.

Unfortunately it doesn't look like OpenSearch 2.4 has the "copy_from": "firstTimestamp" directive.

However, in OpenSearch you can just natively set the DataStream timestamp_field when you create your _index_template, here's my OpenSearch 2.4.0 datastream config:

PUT _index_template/kube-events-template
{
  "index_patterns": [
    "kube-events"
  ],
  "data_stream": {
    "timestamp_field": {
      "name": "firstTimestamp"
    }
  },
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1
    }
  },
  "priority": 100
}

And here's my kubernetes-event-exporter config:

config:
  logLevel: debug
  logFormat: pretty
  receivers:
    - name: "opensearch"
      opensearch:
        hosts:
          - https://opensearch-cluster.metrics:9200
        index: kube-events
        username: eventlog
        password: XXXXXXXXXXXXXXX
        useEventID: false
        deDot: true
        tls:
          ...
  route:
    routes:
      - match:
          - receiver: "opensearch"

I didn't see any references to the timestamp_field on Elastic's documentation but I could have missed it.

I prefer this solution a lot more than adding an extra field to the Kubernetes Event Exporter source, so I'm curious if ElasticSearch has something similar. It seems a bit silly to totally lock DataStreams to the @timestamp field.

[azlev]

There is no timestamp_field field in Elasticsearch Data Streams currently implementation, https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html :

Every document indexed to a data stream must contain a @timestamp field, mapped as a date or date_nanos field type. If the index template doesn’t specify a mapping for the @timestamp field, Elasticsearch maps @timestamp as a date field with default options.

Maybe I missed something inside the mapping documentation, but as far I know it's not possible to rename a field without a ingestion pipeline.

The copy_from is inside a ingestion pipeline, I think this is the equivalent, but I'm not sure: https://opensearch.org/docs/latest/data-prepper/data-prepper-reference/#copy_values

It seems a bit silly to totally lock DataStreams to the @timestamp field.

Well, it is hard-coded inside elastic, so it makes sense to hard code here in the patch too.