NGINX OpenTelemetry Input Package #15651

ishleenk17 · 2025-10-14T18:02:15Z

This is a draft version of NGINX OpenTelemetry Input Package.
Since this is the very first OTEL Input package, this PR is being used to figure out the structure, template, what works what doesn't while creating packages.

FYI, using HTTPChecker example as reference

Relates: https://github.com/elastic/obs-integration-team/issues/622

ishleenk17 · 2025-10-27T10:51:01Z

@jsoriano @cmacknz : Could you please review this first OTEL input package

Point to discuss:

We need to have a check in data_stream.dataset. It should always be same as the receiver name (for metrics) for the content package to autoinstall. For logs we need to see how will we ensure about the datastream field.
Under compatibility section of documentation, what all should we mention or shall we just point to the upstream receiver ? I think pointing to upstream receiver might not be right as our EDOT collector might not be at that the upstream collector label ?
Ideally we don't need base-fileds.yml. But elastic package doesn't allow to build the package without that. We should remove this check for these fields in OTEL Input package.
If we are updating this, we should also update the corresponding content package documentation. We need to discuss how. Currently it contains how a collecotr config looks like. But since now that info is there in the input package. Should we have that in content package ? We might need it if someone tries to install that content package via upstream collector and not EDOT. Thoughts ?

cc: @tommyers-elastic

tommyers-elastic · 2025-10-27T11:37:05Z

packages/nginx_input_otel/_dev/build/docs/README.md

@@ -0,0 +1,63 @@
+# NGINX OpenTelemetry Input Package 


i wonder how much of this we should just delegate to the upstream receiver. i.e. i could see an argument for literally just something like

An Elastic Agent wrapper around the nginxreceiver. Compatible with the Nginx Content Pack.

I am open to discussions on this one.

@daniela-elastic WDYT ?

I like that, if we could delegate to the upstream receiver this means less work on our side. Do you foresee any cons to delegating to the upstream receiver? Wouldn't want to go into one way doors.

@tommyers-elastic : Maybe we can write this in the Metric reference section. Other parts of the documentation should be there IMO. WDYT ?

tommyers-elastic · 2025-10-27T11:39:08Z

packages/nginx_input_otel/agent/input/input.yml.hbs

+  nginx:
+    endpoint: {{nginx_endpoint}}
+    collection_interval: {{period}}
+processors:


it would be great if agent could provide more flexibility for resource detection processing. i.e it could be a setting at the agent policy level. system seems a sensible default, but ec2 looks out of place to me here.

additionally, do you know if agent injects any batching or memory limiter processor configs?

Yes, I would expect the RDP processor to be a default one by Fleet at policy level.

additionally, do you know if agent injects any batching or memory limiter processor configs?

Nothing that I see in the agent policy that is created. @jsoriano is there any ?

it would be great if agent could provide more flexibility for resource detection processing. i.e it could be a setting at the agent policy level. system seems a sensible default, but ec2 looks out of place to me here.

Do you think most packages are going to have this resourcedetection processor? If that's the case this should be probably managed by Fleet, and not included in all packages.

additionally, do you know if agent injects any batching or memory limiter processor configs?

Nothing that I see in the agent policy that is created. @jsoriano is there any ?

Not that I know.

resourcedetection should be a default global processor, and the list of global processors needs to be editable (unlike the hard coded list in agent today). I always envisioned that global processors would be associated with exporters.

resourcedetection should be a default global processor, and the list of global processors needs to be editable (unlike the hard coded list in agent today).

So in the case of Fleet-managed integrations they should be handled by Fleet, right?
Do they need to be included in packages then as is being done here? We risk having duplicates if this starts being managed by Fleet.

I always envisioned that global processors would be associated with exporters.

There probably needs to be both, processors associated to exporters and to policies.

For now are we going to go ahead with processor as part of the agent config ?
@jsoriano @cmacknz

So in the case of Fleet-managed integrations they should be handled by Fleet, right?

Yes, the default processors would have to be defined in Fleet which we don't do yet and has some problems e.g. needing a Kibana release to change the config. Creating an output package type was one idea for how to solve this.

Per policy global processors would also make sense but this could just be syntactic sugar that inserts them into each exporter pipeline.

For now are we going to go ahead with processor as part of the agent config ?

Yes there is no other way for us to do this right now. I think you would want the detectors or even the whole processor configuration to be configurable though.

packages/nginx_input_otel/fields/base-fields.yml

tommyers-elastic · 2025-10-27T11:43:18Z

packages/nginx_input_otel/img/sample-logo.svg

shouldn't ths be the nginx logo?

I was thinking about generating NGINX+OTEL logo. The way we have been doing for content packs.
WDYT ?

packages/nginx_input_otel/manifest.yml

jsoriano · 2025-10-27T19:01:07Z

packages/nginx_input_otel/agent/input/input.yml.hbs

+  nginx:
+    endpoint: {{nginx_endpoint}}
+    collection_interval: {{period}}
+processors:


it would be great if agent could provide more flexibility for resource detection processing. i.e it could be a setting at the agent policy level. system seems a sensible default, but ec2 looks out of place to me here.

Do you think most packages are going to have this resourcedetection processor? If that's the case this should be probably managed by Fleet, and not included in all packages.

additionally, do you know if agent injects any batching or memory limiter processor configs?

Nothing that I see in the agent policy that is created. @jsoriano is there any ?

Not that I know.

jsoriano · 2025-10-27T19:02:51Z

packages/nginx_input_otel/fields/base-fields.yml

This file can be removed. OTel input packages rely on pre-built component templates.

If this is the only fields file, build will fail, till we release elastic/package-spec#994 (probably this week).

Yes, this is mostly going to be the only file under fields. I don't expect any mappings to go in here.
Do we actually expect predefined mappings for OTEL Input packages.
Or we can rely on the otel template and the dynamic mappings ?

Do we actually expect predefined mappings for OTEL Input packages.
Or we can rely on the otel template and the dynamic mappings ?

We can rely on the pre-installed template and the dynamic mappings it includes.

We still have the option of defining fields if needed in some case, but defining fields will be completely optional for OTel input packages. It will be actually expected for OTel input packages to don't contain any field definition.

packages/nginx_input_otel/manifest.yml

Co-authored-by: Jaime Soriano Pastor <[email protected]>

elasticmachine · 2025-10-28T04:41:26Z

💚 Build Succeeded

Buildkite Build
Commit: 87a3579

History

💚 Build #33219 succeeded 31c428a
💚 Build #33200 succeeded b560639
💚 Build #33198 succeeded 0c7ad17
💔 Build #33197 failed 43d3c73
💔 Build #33192 failed 8ce1be2
💔 Build #33189 failed bffc9a2

cc @ishleenk17

ishleenk17 added 2 commits October 14, 2025 23:29

Draft version of OTEL Input Pacakge

71c03fd

Update version

45958b4

andrewkroh added New Integration Issue or pull request for creating a new integration package. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Oct 14, 2025

ishleenk17 added 4 commits October 27, 2025 12:38

Update README

bffc9a2

Update codeowners

8ce1be2

codeeowner

43d3c73

Manifest update

0c7ad17

ishleenk17 self-assigned this Oct 27, 2025

ishleenk17 requested review from jsoriano and tommyers-elastic October 27, 2025 10:45

ishleenk17 marked this pull request as ready for review October 27, 2025 10:51

ishleenk17 requested a review from a team as a code owner October 27, 2025 10:51

ishleenk17 changed the title ~~[Draft] NGINX OpenTelemetry Input Package~~ NGINX OpenTelemetry Input Package Oct 27, 2025

tommyers-elastic reviewed Oct 27, 2025

View reviewed changes

packages/nginx_input_otel/fields/base-fields.yml Show resolved Hide resolved

tommyers-elastic reviewed Oct 27, 2025

View reviewed changes

packages/nginx_input_otel/manifest.yml Outdated Show resolved Hide resolved

Update defaults

b560639

jsoriano reviewed Oct 27, 2025

View reviewed changes

ishleenk17 and others added 2 commits October 28, 2025 00:51

Update packages/nginx_input_otel/manifest.yml

31c428a

Co-authored-by: Jaime Soriano Pastor <[email protected]>

Update packages/nginx_input_otel/manifest.yml

87a3579

Co-authored-by: Jaime Soriano Pastor <[email protected]>

NGINX OpenTelemetry Input Package #15651

Are you sure you want to change the base?

NGINX OpenTelemetry Input Package #15651

Uh oh!

Conversation

ishleenk17 commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ishleenk17 commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Point to discuss:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tommyers-elastic Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ishleenk17 Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Oct 28, 2025

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ishleenk17 commented Oct 14, 2025 •

edited

Loading

ishleenk17 commented Oct 27, 2025 •

edited

Loading

tommyers-elastic Oct 27, 2025 •

edited

Loading

ishleenk17 Oct 27, 2025 •

edited

Loading

jsoriano Oct 27, 2025 •

edited

Loading