Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -339,6 +339,7 @@
/packages/nginx_ingress_controller @elastic/obs-ds-hosted-services
/packages/nginx_ingress_controller_otel @elastic/obs-infraobs-integrations
/packages/nginx_otel @elastic/obs-infraobs-integrations
/packages/nginx_input_otel @elastic/obs-infraobs-integrations
/packages/nozomi_networks @elastic/security-service-integrations
/packages/nvidia_gpu @elastic/obs-infraobs-integrations
/packages/o365 @elastic/security-service-integrations
Expand Down
93 changes: 93 additions & 0 deletions packages/nginx_input_otel/LICENSE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
Elastic License 2.0

URL: https://www.elastic.co/licensing/elastic-license

## Acceptance

By using the software, you agree to all of the terms and conditions below.

## Copyright License

The licensor grants you a non-exclusive, royalty-free, worldwide,
non-sublicensable, non-transferable license to use, copy, distribute, make
available, and prepare derivative works of the software, in each case subject to
the limitations and conditions below.

## Limitations

You may not provide the software to third parties as a hosted or managed
service, where the service provides users with access to any substantial set of
the features or functionality of the software.

You may not move, change, disable, or circumvent the license key functionality
in the software, and you may not remove or obscure any functionality in the
software that is protected by the license key.

You may not alter, remove, or obscure any licensing, copyright, or other notices
of the licensor in the software. Any use of the licensor’s trademarks is subject
to applicable law.

## Patents

The licensor grants you a license, under any patent claims the licensor can
license, or becomes able to license, to make, have made, use, sell, offer for
sale, import and have imported the software, in each case subject to the
limitations and conditions in this license. This license does not cover any
patent claims that you cause to be infringed by modifications or additions to
the software. If you or your company make any written claim that the software
infringes or contributes to infringement of any patent, your patent license for
the software granted under these terms ends immediately. If your company makes
such a claim, your patent license ends immediately for work on behalf of your
company.

## Notices

You must ensure that anyone who gets a copy of any part of the software from you
also gets a copy of these terms.

If you modify the software, you must include in any modified copies of the
software prominent notices stating that you have modified the software.

## No Other Rights

These terms do not imply any licenses other than those expressly granted in
these terms.

## Termination

If you use the software in violation of these terms, such use is not licensed,
and your licenses will automatically terminate. If the licensor provides you
with a notice of your violation, and you cease all violation of this license no
later than 30 days after you receive that notice, your licenses will be
reinstated retroactively. However, if you violate these terms after such
reinstatement, any additional violation of these terms will cause your licenses
to terminate automatically and permanently.

## No Liability

*As far as the law allows, the software comes as is, without any warranty or
condition, and the licensor will not be liable to you for any damages arising
out of these terms or the use or nature of the software, under any kind of
legal claim.*

## Definitions

The **licensor** is the entity offering these terms, and the **software** is the
software the licensor makes available under these terms, including any portion
of it.

**you** refers to the individual or entity agreeing to these terms.

**your company** is any legal entity, sole proprietorship, or other kind of
organization that you work for, plus all organizations that have control over,
are under the control of, or are under common control with that
organization. **control** means ownership of substantially all the assets of an
entity, or the power to direct its management and policies by vote, contract, or
otherwise. Control can be direct or indirect.

**your licenses** are all the licenses granted to you for the software under
these terms.

**use** means anything you do with the software requiring one of your licenses.

**trademark** means trademarks, service marks, and similar rights.
63 changes: 63 additions & 0 deletions packages/nginx_input_otel/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# NGINX OpenTelemetry Input Package
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wonder how much of this we should just delegate to the upstream receiver. i.e. i could see an argument for literally just something like

An Elastic Agent wrapper around the nginxreceiver. Compatible with the Nginx Content Pack.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am open to discussions on this one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that, if we could delegate to the upstream receiver this means less work on our side. Do you foresee any cons to delegating to the upstream receiver? Wouldn't want to go into one way doors.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tommyers-elastic : Maybe we can write this in the Metric reference section. Other parts of the documentation should be there IMO. WDYT ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tommyers-elastic : we would go ahead with the documentation as is for now, as we discussed.
I can maybe remove the metrics reference section here as for that they can refer to the upstream receiver.


## Overview
The NGINX OpenTelemetry Input Package integration for Elastic enables collection of telemetry data from NGINX web servers through OpenTelemetry protocols using the [nginxreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/nginxreceiver#nginx-receiver).

This integration facilitates comprehensive monitoring of NGINX web server performance, request processing, error tracking, and operational metrics to provide insights into web application infrastructure health and performance.

### Compatibility
This Integration uses the upstream [nginxreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/nginxreceiver#nginx-receiver) to collect the metrics.

### How it works
This integration receives telemetry data from NGINX servers by configuring the NGINX endpoint in the integration, which then gets applied to the nginxreceiver present in the EDOT collector, which then forwards the data to Elastic Agent. The Elastic Agent processes and enriches the data before sending it to Elasticsearch for indexing and analysis. Once the data arrives into Elasticsearch, its corresponding [NGINX OpenTelemetry Assets Package](https://www.elastic.co/docs/reference/integrations/nginx_otel) gets auto installed and the dashboards light up.

## What data does this integration collect?
The NGINX OpenTelemetry Input Package integration collects telemetry data of the following types:

* **Metrics** - Performance metrics including request rates, response times, connection counts, and server status


## What do I need to use this integration?
1. Permissions required: The collector requires access to the NGINX `stub_status` endpoint (for example, http://localhost:80/nginx_status). When running the collector, make sure you have the appropriate permissions to access this endpoint.

2. NGINX configuration: The NGINX `stub_status` module must be enabled, and the status endpoint must be accessible. For example:
```
server {
listen 80;
server_name localhost;
location /nginx_status {
stub_status on;
allow 127.0.0.1;
deny all;
}
}
```
3. The NGINX endpoint configured in the Integration configuration from the UI.


## Metrics reference

### NGINX metrics
The [NGINX receiver]((https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/nginxreceiver/documentation.md)) collects performance metrics from the NGINX `stub_status` module. Key metrics include:


| Metric Name | Description | Type | Attributes |
|-------------|-------------|------|------------|
| `nginx.requests` | Total number of client requests | Counter | - |
| `nginx.connections_accepted` | Total number of accepted client connections | Counter | - |
| `nginx.connections_handled` | Total number of handled connections | Counter | - |
| `nginx.connections_current` | Current number of client connections by state | Gauge | `state`: `active`, `reading`, `writing`, `waiting` |

#### Connection States

- `active`: Currently active client connections
- `reading`: Connections currently reading request headers
- `writing`: Connections currently writing response to client
- `waiting`: Idle client connections waiting for a request

These metrics provide insights into:
- **Request volume and patterns** through request counts
- **Connection health** via accepted, and handled connection statistics
- **Server performance** using active, reading, writing, and waiting connection states

For a complete list of all available metrics and their detailed descriptions, refer to the [NGINX Receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/nginxreceiver/documentation.md) in the upstream OpenTelemetry Collector repository.
12 changes: 12 additions & 0 deletions packages/nginx_input_otel/agent/input/input.yml.hbs
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
receivers:
nginx:
endpoint: {{nginx_endpoint}}
collection_interval: {{period}}
processors:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be great if agent could provide more flexibility for resource detection processing. i.e it could be a setting at the agent policy level. system seems a sensible default, but ec2 looks out of place to me here.

Copy link
Contributor

@tommyers-elastic tommyers-elastic Oct 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

additionally, do you know if agent injects any batching or memory limiter processor configs?

Copy link
Member Author

@ishleenk17 ishleenk17 Oct 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I would expect the RDP processor to be a default one by Fleet at policy level.

additionally, do you know if agent injects any batching or memory limiter processor configs?

Nothing that I see in the agent policy that is created. @jsoriano is there any ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be great if agent could provide more flexibility for resource detection processing. i.e it could be a setting at the agent policy level. system seems a sensible default, but ec2 looks out of place to me here.

Do you think most packages are going to have this resourcedetection processor? If that's the case this should be probably managed by Fleet, and not included in all packages.

additionally, do you know if agent injects any batching or memory limiter processor configs?

Nothing that I see in the agent policy that is created. @jsoriano is there any ?

Not that I know.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resourcedetection should be a default global processor, and the list of global processors needs to be editable (unlike the hard coded list in agent today). I always envisioned that global processors would be associated with exporters.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now are we going to go ahead with processor as part of the agent config ?
@jsoriano @cmacknz

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So in the case of Fleet-managed integrations they should be handled by Fleet, right?

Yes, the default processors would have to be defined in Fleet which we don't do yet and has some problems e.g. needing a Kibana release to change the config. Creating an output package type was one idea for how to solve this.

Per policy global processors would also make sense but this could just be syntactic sugar that inserts them into each exporter pipeline.

For now are we going to go ahead with processor as part of the agent config ?

Yes there is no other way for us to do this right now. I think you would want the detectors or even the whole processor configuration to be configurable though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ishleenk17 did you consider making this configurable as suggested by Craig?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano As we discussed in our meeting. Shall we just give a yaml box for procesors, keeping the whole yaml configuration configurable.
I don't think parts of the rdp processor should be made configurable.
WDYT @tommyers-elastic

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed in our meeting. Shall we just give a yaml box for procesors, keeping the whole yaml configuration configurable.

We discussed this solution in the context of the hostmetrics receiver, which has a pretty extensive set of options. I wouldn't generalize the use of yaml boxes for all cases.

Thinking on this case of the resourcedetection processor, in the mid-long term I would like to see this managed by Fleet, so in the meantime as you prefer.

resourcedetection:
detectors: ["system", "ec2"]
service:
pipelines:
metrics:
receivers: [nginx]
processors: [resourcedetection]
6 changes: 6 additions & 0 deletions packages/nginx_input_otel/changelog.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# newer versions go on top
- version: "0.1.0"
changes:
- description: Initial draft of the package
type: enhancement
link: https://github.com/elastic/integrations/pull/15651
63 changes: 63 additions & 0 deletions packages/nginx_input_otel/docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# NGINX OpenTelemetry Input Package

## Overview
The NGINX OpenTelemetry Input Package integration for Elastic enables collection of telemetry data from NGINX web servers through OpenTelemetry protocols using the [nginxreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/nginxreceiver#nginx-receiver).

This integration facilitates comprehensive monitoring of NGINX web server performance, request processing, error tracking, and operational metrics to provide insights into web application infrastructure health and performance.

### Compatibility
This Integration uses the upstream [nginxreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/nginxreceiver#nginx-receiver) to collect the metrics.

### How it works
This integration receives telemetry data from NGINX servers by configuring the NGINX endpoint in the integration, which then gets applied to the nginxreceiver present in the EDOT collector, which then forwards the data to Elastic Agent. The Elastic Agent processes and enriches the data before sending it to Elasticsearch for indexing and analysis. Once the data arrives into Elasticsearch, its corresponding [NGINX OpenTelemetry Assets Package](https://www.elastic.co/docs/reference/integrations/nginx_otel) gets auto installed and the dashboards light up.

## What data does this integration collect?
The NGINX OpenTelemetry Input Package integration collects telemetry data of the following types:

* **Metrics** - Performance metrics including request rates, response times, connection counts, and server status


## What do I need to use this integration?
1. Permissions required: The collector requires access to the NGINX `stub_status` endpoint (for example, http://localhost:80/nginx_status). When running the collector, make sure you have the appropriate permissions to access this endpoint.

2. NGINX configuration: The NGINX `stub_status` module must be enabled, and the status endpoint must be accessible. For example:
```
server {
listen 80;
server_name localhost;
location /nginx_status {
stub_status on;
allow 127.0.0.1;
deny all;
}
}
```
3. The NGINX endpoint configured in the Integration configuration from the UI.


## Metrics reference

### NGINX metrics
The [NGINX receiver]((https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/nginxreceiver/documentation.md)) collects performance metrics from the NGINX `stub_status` module. Key metrics include:


| Metric Name | Description | Type | Attributes |
|-------------|-------------|------|------------|
| `nginx.requests` | Total number of client requests | Counter | - |
| `nginx.connections_accepted` | Total number of accepted client connections | Counter | - |
| `nginx.connections_handled` | Total number of handled connections | Counter | - |
| `nginx.connections_current` | Current number of client connections by state | Gauge | `state`: `active`, `reading`, `writing`, `waiting` |

#### Connection States

- `active`: Currently active client connections
- `reading`: Connections currently reading request headers
- `writing`: Connections currently writing response to client
- `waiting`: Idle client connections waiting for a request

These metrics provide insights into:
- **Request volume and patterns** through request counts
- **Connection health** via accepted, and handled connection statistics
- **Server performance** using active, reading, writing, and waiting connection states

For a complete list of all available metrics and their detailed descriptions, refer to the [NGINX Receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/nginxreceiver/documentation.md) in the upstream OpenTelemetry Collector repository.
12 changes: 12 additions & 0 deletions packages/nginx_input_otel/fields/base-fields.yml
Copy link
Member

@jsoriano jsoriano Oct 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file can be removed. OTel input packages rely on pre-built component templates.

If this is the only fields file, build will fail, till we release elastic/package-spec#994 (probably this week).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is mostly going to be the only file under fields. I don't expect any mappings to go in here.
Do we actually expect predefined mappings for OTEL Input packages.
Or we can rely on the otel template and the dynamic mappings ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually expect predefined mappings for OTEL Input packages.
Or we can rely on the otel template and the dynamic mappings ?

We can rely on the pre-installed template and the dynamic mappings it includes.

We still have the option of defining fields if needed in some case, but defining fields will be completely optional for OTel input packages. It will be actually expected for OTel input packages to don't contain any field definition.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider removing this file? You can leave it empty or wait for next elastic-package release (coming soon) if you have problems with validation.

Copy link
Member Author

@ishleenk17 ishleenk17 Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano On removing the base-fields.yml

  1. file "/Users/ishleenkaur/go/ish_12Oct_integ/ish_integrations/build/packages/nginx_otel_input-0.0.1.zip/fields/base-fields.yml" is invalid: file is empty, but media type is defined

  2. On having this file empty, we get below message:

Error: building package failed: invalid content found in built zip package: found 4 validation errors:

  1. expected field "data_stream.dataset" with type "constant_keyword" not found
  2. expected field "data_stream.namespace" with type "constant_keyword" not found
  3. expected field "@timestamp" with type "date" not found
  4. expected field "data_stream.type" with type "constant_keyword" not found

So for now we need to have this file with the fields, or we wait for the next release of elastic package.

Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
- name: data_stream.type
type: constant_keyword
description: Data stream type.
- name: data_stream.dataset
type: constant_keyword
description: Data stream dataset.
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
- name: '@timestamp'
type: date
description: Event timestamp.
Loading