[Netskope] Add alerts_events_v2 data stream to fetch the data for alerts_v2 and events_v2 from a single queue #15697

moxarth-rathod · 2025-10-20T17:05:53Z

Proposed commit message

netskope: add alerts_events_v2 data stream to fetch the data for alerts_v2 and events_v2 from a single queue

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/netskope directory.
Run the following command to run tests.

elastic-package test -v

Additionally, the following cloud credentials are required to setup:

AWS:

AWS_DEFAULT_PROFILE
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN.

Related issues

Closes [Netskope]: Using Log Streaming mode does not support to get both Events and Alerts #15459

…ents_v2 data from a single queue

elasticmachine · 2025-10-20T17:05:58Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

leandrojmp · 2025-10-21T15:49:39Z

packages/netskope/data_stream/alerts_events_v2/agent/stream/aws-s3.yml.hbs

+bucket_arn: {{bucket_arn}}
+{{/if}}
+{{#if number_of_workers}}
+number_of_workers: {{number_of_workers}}


Hello @moxarth-rathod

This needs to be outside #if collect_s3_logs and #if queue_url as the setting number_of_workers is applied to both ways of getting data from S3, polling and SQS, the setting max_number_of_messages is ignored on agents higher than 8.16+

For more context this was reported on #13179 and fixed on multiple integrations on #13350

Thanks! I've made the changes accordingly.

leandrojmp · 2025-10-21T16:12:06Z

packages/netskope/_dev/build/docs/README.md


+**Note**: It is recommended to use the combined alerts_events_v2 data stream rather than configuring the individual events_v2 or alerts_v2 data stream. The alerts_events_v2 stream automatically directs logs to the appropriate individual data streams.
+
+If the individual v2 data streams, events_v2 or alerts_v2, are used via SQS, it is necessary to implement event-based bucket segregation.


This may change in the future, but according to current Netskope documentation, if you choose any event type and alerts, they will be streamed together.

The user may choose to stream only events or to stream only alerts, but when choosing alerts and any other event type, they will be streamed together.

packages/netskope/changelog.yml

efd6 · 2025-10-29T05:16:46Z

...ges/netskope/data_stream/alerts_events_v2/_dev/test/pipeline/test-combined.log-expected.json

+                    "city_name": "Ikebukuro",
+                    "country_iso_code": "AU",
+                    "location": {
+                        "lat": 37.7749,
+                        "lon": 151.2093
+                    },
+                    "postal_code": "2099",
+                    "region_name": "Western Australia",
+                    "timezone": "America/Los_Angeles"


🌏 🌍 🌎

packages/netskope/data_stream/alerts_events_v2/routing_rules.yml

packages/netskope/data_stream/alerts_v2/manifest.yml

packages/netskope/data_stream/events_v2/manifest.yml

packages/netskope/data_stream/alerts_events_v2/manifest.yml

packages/netskope/data_stream/alerts_events_v2/elasticsearch/ingest_pipeline/default.yml

elastic-vault-github-plugin-prod · 2025-10-29T05:41:21Z

🚀 Benchmarks report

Package `netskope` 👍(3) 💚(1) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`alerts_v2`	1212.12	861.33	-350.79 (-28.94%)	💔
`events`	2070.39	1636.66	-433.73 (-20.95%)	💔

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-10-30T10:34:44Z

💚 Build Succeeded

Buildkite Build
Commit: abce8b4

History

💚 Build #33328 succeeded 2fd48a6
💔 Build #33185 failed 5c3f5a1
💔 Build #33057 failed 5afc25a
💔 Build #32912 failed 1483abd

cc @moxarth-rathod

kcreddy · 2025-10-30T11:54:46Z

...s/netskope/data_stream/alerts_events_v2/elasticsearch/ingest_pipeline/pipeline_events_v2.yml

      field: ecs.version
      tag: set_ecs_version
-      value: 8.17.0
+      value: 8.17.


Suggested change

value: 8.17.

value: 8.17.0

kcreddy · 2025-10-30T11:55:30Z

packages/netskope/data_stream/alerts_events_v2/elasticsearch/ingest_pipeline/default.yml

+  - json:
+      field: event.original
+      tag: json_event_original
+      target_field: netskope.alerts_or_events


Any reason why we are not using the data stream name i.e., netskope.alerts_events_v2 but instead opting for netskope.alerts_or_events ?

kcreddy · 2025-10-30T12:01:10Z

packages/netskope/_dev/build/docs/README.md


 Considering you already have an AWS S3 bucket setup, to configure it with Netskope, follow [these steps](https://docs.netskope.com/en/stream-logs-to-amazon-s3) to enable the log streaming.

+**Note**: It is recommended to use the combined alerts_events_v2 data stream rather than configuring the individual events_v2 or alerts_v2 data stream. The alerts_events_v2 stream automatically directs logs to the appropriate individual data streams.


Unless users look at the integration code, they are not aware of alerts_events_v2, events_v2, or alerts_v2. It might be better to use titles instead.
Alerts V2 and Events V2, Events V2, and Alerts V2.

kcreddy · 2025-10-30T12:06:50Z

packages/netskope/_dev/build/docs/README.md

-### Alerts V2
-
-{{fields "alerts_v2"}}
-
-{{event "alerts_v2"}}
-
 ### Events

 {{fields "events"}}

 {{event "events"}}

-### Events V2
-
-{{fields "events_v2"}}
-
-{{event "events_v2"}}
-


Can we also add system tests from this routing data stream alerts_events_v2, and add sample event and field mappings to README? Looks like it is possible, example

efd6

LGTM, but please wait for @kcreddy and @ShourieG.

ShourieG · 2025-10-31T09:27:11Z

@moxarth-rathod, the field definitions are not present in each of the routed data streams, It will likely cause errors when the data is routed and published in the respective dataset. Have we tested this with live data ? I believe it's necessary to test this with the fields absent & present. Can we do this ?

ShourieG · 2025-10-31T10:12:11Z

@moxarth-rathod, the field definitions are not present in each of the routed data streams, It will likely cause errors when the data is routed and published in the respective dataset. Have we tested this with live data ? I believe it's necessary to test this with the fields absent & present. Can we do this ?

After discussing with @moxarth-rathod, the respective field definitions are present and also a live test has been performed.

ShourieG

LGTM

Add alerts_events_v2 data stream to support fetching alerts_v2 and ev…

1483abd

…ents_v2 data from a single queue

moxarth-rathod self-assigned this Oct 20, 2025

moxarth-rathod requested a review from a team as a code owner October 20, 2025 17:05

moxarth-rathod added breaking change Integration:netskope Netskope Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Oct 20, 2025

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Oct 20, 2025

leandrojmp reviewed Oct 21, 2025

View reviewed changes

Address PR comments

5afc25a

moxarth-rathod requested a review from leandrojmp October 23, 2025 05:56

moxarth-rathod and others added 2 commits October 27, 2025 11:55

Update README

5c3f5a1

Merge branch 'main' into netskope-fix-alerts_events_v2

2fd48a6

efd6 reviewed Oct 29, 2025

View reviewed changes

Address PR comments

abce8b4

moxarth-rathod requested review from ShourieG and efd6 October 30, 2025 10:04

kcreddy reviewed Oct 30, 2025

View reviewed changes

efd6 approved these changes Oct 30, 2025

View reviewed changes

ShourieG approved these changes Oct 31, 2025

View reviewed changes


		Note: It is recommended to use the combined alerts_events_v2 data stream rather than configuring the individual events_v2 or alerts_v2 data stream. The alerts_events_v2 stream automatically directs logs to the appropriate individual data streams.

		If the individual v2 data streams, events_v2 or alerts_v2, are used via SQS, it is necessary to implement event-based bucket segregation.


		Considering you already have an AWS S3 bucket setup, to configure it with Netskope, follow [these steps](https://docs.netskope.com/en/stream-logs-to-amazon-s3) to enable the log streaming.

		Note: It is recommended to use the combined alerts_events_v2 data stream rather than configuring the individual events_v2 or alerts_v2 data stream. The alerts_events_v2 stream automatically directs logs to the appropriate individual data streams.

[Netskope] Add alerts_events_v2 data stream to fetch the data for alerts_v2 and events_v2 from a single queue #15697

Are you sure you want to change the base?

[Netskope] Add alerts_events_v2 data stream to fetch the data for alerts_v2 and events_v2 from a single queue #15697

Conversation

moxarth-rathod commented Oct 20, 2025

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Oct 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leandrojmp Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 29, 2025

🚀 Benchmarks report

Package netskope 👍(3) 💚(1) 💔(2)

Uh oh!

elasticmachine commented Oct 30, 2025

💚 Build Succeeded

History

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

ShourieG commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShourieG commented Oct 31, 2025

Uh oh!

ShourieG left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

leandrojmp Oct 21, 2025 •

edited

Loading

Package `netskope` 👍(3) 💚(1) 💔(2)

ShourieG commented Oct 31, 2025 •

edited

Loading