Update coverage information with tags and transforms

[jsoriano]

Related to #2704.

Add some information for coverage reports:

• validation.yml is ignored.
• Kibana tags are added as covered on asset reports.
• Transform files are added as covered on system tests.

[jsoriano]

test integrations

[jsoriano]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15143

[jsoriano]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15143

[jsoriano]

Tag detection for tags in tags.yml is flaky in CI 🤔 Not sure if removing this by now...

[jsoriano]

/test

[mrodm]

Tag detection for tags in tags.yml is flaky in CI 🤔 Not sure if removing this by now...

Looking at the packages that have failed those asset tests, they have defined the kibana/tags.yml but they do not have any Kibana asset in the package.
For instance azure_blob_storage, cel or gcp_pubsub packages:

 $ ls gcp_pubsub/kibana/
tags.yml
 $ ls cel/kibana/
tags.yml
 $ ls azure_blob_storage/kibana/
tags.yml

Could it be checked if there are Kibana assets in the directory first ?

However, there are two packages that they have kibana assets and a tags.yml and there is no tag found in that CI build: kubernetes_otel and ti_util.
I've tried to install locally those two packages in a stack version 9.0.0 and the tag is shown:

In case of ti_util package is defined twice Security Solution. For that package there is a tag defined for Security Solution:
https://github.com/elastic/integrations/blob/f12a71ef0cc898557be4f1ec7cf10a7fb6f6b748/packages/ti_util/kibana/tag/ti_util-3e4bef62-18b5-4f4a-96fa-45b0ec39bd73.json

It looks like that tags created by fleet have a tooltip Tag defined in package-spec:

https://github.com/elastic/kibana/blob/f7add87c5a13d37ebcd60d100bfcee06b6960459/x-pack/platform/plugins/shared/fleet/server/services/epm/kibana/assets/tag_assets.ts#L225

Would it be ok to filter the packages with the description somehow when validating the kibana/tags.yml?

[jsoriano]

Looking at the packages that have failed those asset tests, they have defined the kibana/tags.yml but they do not have any Kibana asset in the package.

This is expected, isn't it? Tags defined in kibana/tags.yml don't need to be defined as assets, right?

I've tried to install locally those two packages in a stack version 9.0.0 and the tag is shown:

Yes, I haven't managed to make it fail locally, and in CI it sometimes works, even here we have now a green check.

I am starting to think that these tags are installed asynchronously, and then we may have some kind of race condition.
For the rest of assets we are not actually checking if they are installed, we check that they are listed in the package response from the API, we might have the same problem if we would check for them.

It looks like that tags created by fleet have a tooltip Tag defined in package-spec:

https://github.com/elastic/kibana/blob/f7add87c5a13d37ebcd60d100bfcee06b6960459/x-pack/platform/plugins/shared/fleet/server/services/epm/kibana/assets/tag_assets.ts#L225

Yeah, a bit confusing I think, because the tag is not really defined in the spec. Also looking to how ids are generated we cannot trust on them being just lowercased versions of the names, this only works for the security tag, others contain random uuids.

Would it be ok to filter the packages with the description somehow when validating the kibana/tags.yml?

I think the problem is that they are not there yet, so we would need to introduce some kind of wait until mechanism in any case.

So I think that we would need to do at least:

• Introduce the wait mechanism.
• Change detection to look for tags by name, or expect ids like the ones generated by getPackageSpecTagId.

Apart from that, once we have the wait mechanism, we could check for the rest of assets to be actually present.

Maybe I could split this PR in two, here I only mark the files as covered, and in a follow up we add the other enhancements, wdyt?

[jsoriano]

So I think that we would need to do at least:

• Introduce the wait mechanism.
• Change detection to look for tags by name, or expect ids like the ones generated by getPackageSpecTagId.

Doing this in dadf441, I will leave further improvements for future changes.

[mrodm]

Looking at the packages that have failed those asset tests, they have defined the kibana/tags.yml but they do not have any Kibana asset in the package.

This is expected, isn't it? Tags defined in kibana/tags.yml don't need to be defined as assets, right?

You're right, they do not have to be defined as assets. But in those packages that I mentioned (e.g. azure_blob_storage) there are no Kibana assets. And the tags from kibana/tags.yaml are loaded as Kibana assets (via LoadPackageAssets), and they are checked to be present. But in those packages there will no tag present since there are no kibana asset defined:

https://github.com/elastic/elastic-package/pull/2831/files#diff-a7276c75e5369ba9f671fcfe2fc1b53ea49dc2b2f560b6266958d37c0c52e154R147-R162

[mrodm]

I am starting to think that these tags are installed asynchronously, and then we may have some kind of race condition.
For the rest of assets we are not actually checking if they are installed, we check that they are listed in the package response from the API, we might have the same problem if we would check for them.

That scenario looks likely to happen 👍 .

I think the problem is that they are not there yet, so we would need to introduce some kind of wait until mechanism in any case.

So I think that we would need to do at least:

• Introduce the wait mechanism.
• Change detection to look for tags by name, or expect ids like the ones generated by getPackageSpecTagId.

It sounds a good plan to follow.

But I still think that tags defined in kibana/tags.yml should not be checked for their existence if there are no Kibana assets defined in the package (e.g. azure_blob_storage or cel packages). In those packages, those tags are not going to be created.

[jsoriano]

But I still think that tags defined in kibana/tags.yml should not be checked for their existence if there are no Kibana assets defined in the package (e.g. azure_blob_storage or cel packages). In those packages, those tags are not going to be created.

You are right, fixing this and adding test case in 039273f.

[jsoriano]

This would be actually wrong in the package, and can require quite some changes to be supported as we would need to bring here the whole tags.yml. So if we find a package with this I would prefer to fix this in the package.

[jsoriano]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15143

[jsoriano]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15143

[jsoriano]

test integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15143

[jsoriano]

Moving back to draft, there are still cases causing problems with tags.

[jsoriano]

Putting this on hold as this will require more changes than expected.

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 3a9d607

Failed CI Steps

• :go: Integration test: other

History

• 💚 Build #6044 succeeded cb95d65
• 💚 Build #6043 succeeded 9dfb308
• 💚 Build #6042 succeeded 0e89dca
• 💚 Build #6039 succeeded 3322db3
• 💚 Build #6028 succeeded 8dc311d
• 💚 Build #6027 succeeded 8dc311d

cc @jsoriano