[DON'T MERGE] [aws_cloudtrail_otel] Content pack of EDOT Cloud Forwarder for AWS - CloudTrail Logs

[mykola-elastic]

Content pack for EDOT Cloud Forwarder for AWS - CloudTrail Logs - Dashboard

Redo of AWS CloudTrail dashboard from AWS integration but using data from https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding/awslogsencodingextension#cloudtrail-log-record-fields

Proposed commit message

See title.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices
• Add auto-install via discovery.datasets

Screenshots

Screenshot of AWS CloudTrail dashboard from AWS integration (for comparison)

Screenshot of the dashboard for content pack

[mykola-elastic]

The EDOT Cloud Forwarder page doesn't yet contain anything about CloudTrail Logs

[mykola-elastic]

ping @ishleenk17 @daniela-elastic marked this one ready for review. In the PR description there are screenshots of the existing dashboard for Cloudtrail logs (from AWS integration), and the one I made with ESQL using data from aws.cloudtrail.otel data_stream for this content pack

[mykola-elastic]

@daniela-elastic @ishleenk17 I have a few questions:

• Is it OK for pies/donuts to display all the data without putting the rest to "Others" category?
• Most of the events come with User ID set to null (see the panel on the bottom right). Should I display them or remove from the table?
• Who else can I invite for review?

[ishleenk17]

• Is it OK for pies/donuts to display all the data without putting the rest to "Others" category?

Is this because there is lesser data to categorize ? Is there is enough data tp categorize, we shoul dmaybe do the top 10 and put rest to others.

• Most of the events come with User ID set to null (see the panel on the bottom right). Should I display them or remove from the table?

If user ID's are NULL mostly, I would assume its not of much relevance to the customer/user ?

• Who else can I invite for review?

Added @ShourieG from Security team.

[ishleenk17]

Use the word " OpenTelemetry" in the Title of the dashboard.
What are the titlees we have used till now for other dashbaords ? I think we should be consistent in how we name them

[mykola-elastic]

In the dashboard titles:

• system_otel uses OTel
• nginx_otel, mysql_otel, iis_otel, and aws elb content pack use OTEL

[mykola-elastic]

Is this because there is lesser data to categorize ? Is there is enough data tp categorize, we shoul dmaybe do the top 10 and put rest to others.

I haven't figured out a way yet how to do LIMIT while having the Others category to display the sum of those which didn't get into the "limit".
There are no configuration options on pie chart to do this, this needs to be done in ES|QL with some complex logic it seems

UPDATE: Confirmed, there is no way to do that in ES|QL at the moment - to show top 10 results and group the rest as "Others" in Kibana pie chart

[daniela-elastic]

How do we plan to update this if in the future ECF supports more than just serverless? CC @ishleenk17

[ishleenk17]

When ECF supports more than serverless they will do that with a particular new stack version.
We will also update our kibana version then and along with that the README as well

[daniela-elastic]

Do we have a process to ensure that we don't miss this and then proactively update the README?

[ishleenk17]

UPDATE: Confirmed, there is no way to do that in ES|QL at the moment - to show top 10 results and group the rest as "Others" in Kibana pie chart

So, its an ESQL issue. In KQL it would be possible right ?

[mykola-elastic]

So, its an ESQL issue. In KQL it would be possible right ?

Yes, it is possible to do using Lens (screenshot below).
Though I suggest sticking with ES|QL and fix it when such feature appears. The package version is still 0.1.0 and maybe being consistent is better in this case. What do you think?

[ishleenk17]

So, its an ESQL issue. In KQL it would be possible right ?

Yes, it is possible to do using Lens (screenshot below). Though I suggest sticking with ES|QL and fix it when such feature appears. The package version is still 0.1.0 and maybe being consistent is better in this case. What do you think?

If we don't limit it, its looking cluttered and TMI for the customer.
I would let @daniela-elastic take the final call on this.

[mykola-elastic]

@daniela-elastic ESQL (top) vs LENS (bottom):

Note: in the "User agents" the labels for ESQL panel are short and nice as I used REPLACE with regex to remove clutter. In Lens I have to use labels (user agents) as they are in the data

[daniela-elastic]

@daniela-elastic ESQL (top) vs LENS (bottom):

Note: in the "User agents" the labels for ESQL panel are short and nice as I used REPLACE with regex to remove clutter. In Lens I have to use labels (user agents) as they are in the data

If we already have a workaround for ESQL suggest keeping this and submitting a request to platform / Kibana team to show top 10 results and group the rest as "Others" in Kibana pie chart (submit a ticket and also add to kibana wishlist spreadsheet)

[mykola-elastic]

Opened issue in Kibana elastic/kibana#239279 (and added to the wishlist)

[mykola-elastic]

Can we move forward with this one? @ishleenk17 @MichaelKatsoulis

[MichaelKatsoulis]

LGTM. Just a nit

[ishleenk17]

1. We are using Logs by Service and Action. In elastic agent Integration we say only by Action. Whats the difference ?
2. For the User Agents panel, is it meant to be Logs by User agents ?
3. In the table of some fields that we have created is it possible to rename the heading of the columns. Eg: rpc.service might not be intuitive to customer. But looks like that is EventType. Not a blocker!

[mykola-elastic]

@ishleenk17

1. We are using Logs by Service and Action. In elastic agent Integration we say only by Action. Whats the difference ?

Yeah, they are the same, it is just that I changed the title - seemed better.

2. For the User Agents panel, is it meant to be Logs by User agents ?

Yeah, it is. I thought that the short "User Agents" is understandable. What do you think? Shall I change it?

3. In the table of some fields that we have created is it possible to rename the heading of the columns. Eg: rpc.service might not be intuitive to customer. But looks like that is EventType. Not a blocker!

Yes, it is possible to rename the heading, doing that now

[ishleenk17]

Yeah, they are the same, it is just that I changed the title - seemed better.

FMU, whats service and action here ?

Yeah, it is. I thought that the short "User Agents" is understandable. What do you think? Shall I change it?

I think Logs by User Agents would be better considering the other panels whrre we are mentioning what we are breaking it down by.

Yes, it is possible to rename the heading, doing that now

Thanks

[mykola-elastic]

FMU, whats service and action here ?

Service corresponds to:

• rpc.service (awslogsencodingextension)
• eventSource (CloudTrail doc) - "The service that the request was made to"

Action corresponds to:

• rpc.method (awslogsencodingextension)
• eventName (CloudTrail doc) - "The requested action, which is one of the actions in the API for that service."

[mykola-elastic]

@ishleenk17 updated the dashboard and screenshots

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b731c73

History

• 💚 Build #33091 succeeded 0a4cdef
• 💚 Build #32803 succeeded ad22e61
• 💚 Build #32748 succeeded 467c6eb
• 💔 Build #32747 failed 2b9c97d
• 💚 Build #32705 succeeded aca5d11
• 💚 Build #32699 succeeded d7931e0

cc @mykola-elastic

[ishleenk17]

Looks good!

[ShourieG]

LGTM