Support system tests otel input

[mrodm]

Relates #2835
Depends on elastic/kibana#233090
Depends on elastic/fleet-server#5469

Add support of system tests in input packages using otelcol input.

How to test this PR locally

Use the kibana code from elastic/kibana#233090. Enable the feature flag enableOtelIntegrations.

Using elastic-package to install the scenario (with these instructions: https://github.com/elastic/elastic-package/blob/main/docs/howto/use_existing_stack.md).

mage build:cover docker:customagentimage
export ELASTIC_AGENT_IMAGE_REF_OVERRIDE=<image tag generated in previous step>
export ELASTIC_PACKAGE_ELASTICSEARCH_PASSWORD=changeme
export ELASTIC_PACKAGE_ELASTICSEARCH_USERNAME=elastic
export ELASTIC_PACKAGE_KIBANA_HOST=http://localhost:5601
export ELASTIC_PACKAGE_ELASTICSEARCH_HOST=http://localhost:9200
elastic-package stack up -v -d --provider environment --version 9.2.0-SNAPSHOT

elastic-package -C test/packages/parallel/httpcheck test system -v

[mrodm]

@jsoriano it looks like there is no "Index Template" created when these input packages using otelcol input are installed:

As a reference, for instance sql_input input package creates this Index Template:

This makes the system test fail with this error:

[0] failed to load mappings from index template preview (metrics-httpcheck.check.otel): error getting mapping: [400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"unable to simulate template [metrics-httpcheck.check.otel] that does not exist"}],"type":"illegal_argument_exception","reason":"unable to simulate template [metrics-httpcheck.check.otel] that does not exist"},"status":400}

This is used to get the mappings that were installed to compare them with those that are available after being ingested the documents.

[mrodm]

Ok, it seems that it is pending as mentioned in elastic/kibana#232628

In addition they should contain @package and @custom templates included in current packages, so package developers and users can still customize mappings or pipelines.

Do you think there is going to be an Index Template installed for that as in the other packages?

[mrodm]

Deleted. Outdated message.

[mrodm]

It looks like that it does not need to update the datasets the Integration Policies created by elastic-package via:

ds := createPackageDatastream(*policyToTest, *r.pkgManifest, policyTemplate, *r.dataStreamManifest, *config, policyToTest.Namespace)

[mrodm]

Documents ingested contain the dataset with suffix otel:

    "data_stream": {
      "dataset": "httpcheck.check.otel",
      "namespace": "83653",
      "type": "metrics"
    },

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 68d15da

Failed CI Steps

• :go: Integration test: httpcheck

History

• 💔 Build #6083 failed 81df3a5
• 💔 Build #6070 failed 0a5db1b
• 💔 Build #6069 failed 82f082f
• 💔 Build #6066 failed 2bfebb8

cc @mrodm