-
Notifications
You must be signed in to change notification settings - Fork 513
NGINX OpenTelemetry Input Package #15651
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
71c03fd
45958b4
bffc9a2
8ce1be2
43d3c73
0c7ad17
b560639
31c428a
87a3579
45c7269
581b15a
40cbd02
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| Elastic License 2.0 | ||
|
|
||
| URL: https://www.elastic.co/licensing/elastic-license | ||
|
|
||
| ## Acceptance | ||
|
|
||
| By using the software, you agree to all of the terms and conditions below. | ||
|
|
||
| ## Copyright License | ||
|
|
||
| The licensor grants you a non-exclusive, royalty-free, worldwide, | ||
| non-sublicensable, non-transferable license to use, copy, distribute, make | ||
| available, and prepare derivative works of the software, in each case subject to | ||
| the limitations and conditions below. | ||
|
|
||
| ## Limitations | ||
|
|
||
| You may not provide the software to third parties as a hosted or managed | ||
| service, where the service provides users with access to any substantial set of | ||
| the features or functionality of the software. | ||
|
|
||
| You may not move, change, disable, or circumvent the license key functionality | ||
| in the software, and you may not remove or obscure any functionality in the | ||
| software that is protected by the license key. | ||
|
|
||
| You may not alter, remove, or obscure any licensing, copyright, or other notices | ||
| of the licensor in the software. Any use of the licensor’s trademarks is subject | ||
| to applicable law. | ||
|
|
||
| ## Patents | ||
|
|
||
| The licensor grants you a license, under any patent claims the licensor can | ||
| license, or becomes able to license, to make, have made, use, sell, offer for | ||
| sale, import and have imported the software, in each case subject to the | ||
| limitations and conditions in this license. This license does not cover any | ||
| patent claims that you cause to be infringed by modifications or additions to | ||
| the software. If you or your company make any written claim that the software | ||
| infringes or contributes to infringement of any patent, your patent license for | ||
| the software granted under these terms ends immediately. If your company makes | ||
| such a claim, your patent license ends immediately for work on behalf of your | ||
| company. | ||
|
|
||
| ## Notices | ||
|
|
||
| You must ensure that anyone who gets a copy of any part of the software from you | ||
| also gets a copy of these terms. | ||
|
|
||
| If you modify the software, you must include in any modified copies of the | ||
| software prominent notices stating that you have modified the software. | ||
|
|
||
| ## No Other Rights | ||
|
|
||
| These terms do not imply any licenses other than those expressly granted in | ||
| these terms. | ||
|
|
||
| ## Termination | ||
|
|
||
| If you use the software in violation of these terms, such use is not licensed, | ||
| and your licenses will automatically terminate. If the licensor provides you | ||
| with a notice of your violation, and you cease all violation of this license no | ||
| later than 30 days after you receive that notice, your licenses will be | ||
| reinstated retroactively. However, if you violate these terms after such | ||
| reinstatement, any additional violation of these terms will cause your licenses | ||
| to terminate automatically and permanently. | ||
|
|
||
| ## No Liability | ||
|
|
||
| *As far as the law allows, the software comes as is, without any warranty or | ||
| condition, and the licensor will not be liable to you for any damages arising | ||
| out of these terms or the use or nature of the software, under any kind of | ||
| legal claim.* | ||
|
|
||
| ## Definitions | ||
|
|
||
| The **licensor** is the entity offering these terms, and the **software** is the | ||
| software the licensor makes available under these terms, including any portion | ||
| of it. | ||
|
|
||
| **you** refers to the individual or entity agreeing to these terms. | ||
|
|
||
| **your company** is any legal entity, sole proprietorship, or other kind of | ||
| organization that you work for, plus all organizations that have control over, | ||
| are under the control of, or are under common control with that | ||
| organization. **control** means ownership of substantially all the assets of an | ||
| entity, or the power to direct its management and policies by vote, contract, or | ||
| otherwise. Control can be direct or indirect. | ||
|
|
||
| **your licenses** are all the licenses granted to you for the software under | ||
| these terms. | ||
|
|
||
| **use** means anything you do with the software requiring one of your licenses. | ||
|
|
||
| **trademark** means trademarks, service marks, and similar rights. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # NGINX OpenTelemetry Input Package | ||
|
|
||
| ## Overview | ||
| The NGINX OpenTelemetry Input Package integration for Elastic enables collection of telemetry data from NGINX web servers through OpenTelemetry protocols using the [nginxreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/nginxreceiver#nginx-receiver). | ||
|
|
||
| This integration facilitates comprehensive monitoring of NGINX web server performance, request processing, error tracking, and operational metrics to provide insights into web application infrastructure health and performance. | ||
|
|
||
| ### Compatibility | ||
| This Integration uses the upstream [nginxreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/nginxreceiver#nginx-receiver) to collect the metrics. | ||
|
|
||
| ### How it works | ||
| This integration receives telemetry data from NGINX servers by configuring the NGINX endpoint in the integration, which then gets applied to the nginxreceiver present in the EDOT collector, which then forwards the data to Elastic Agent. The Elastic Agent processes and enriches the data before sending it to Elasticsearch for indexing and analysis. Once the data arrives into Elasticsearch, its corresponding [NGINX OpenTelemetry Assets Package](https://www.elastic.co/docs/reference/integrations/nginx_otel) gets auto installed and the dashboards light up. | ||
|
|
||
| ## What data does this integration collect? | ||
| The NGINX OpenTelemetry Input Package integration collects telemetry data of the following types: | ||
|
|
||
| * **Metrics** - Performance metrics including request rates, response times, connection counts, and server status | ||
|
|
||
|
|
||
| ## What do I need to use this integration? | ||
| 1. Permissions required: The collector requires access to the NGINX `stub_status` endpoint (for example, http://localhost:80/nginx_status). When running the collector, make sure you have the appropriate permissions to access this endpoint. | ||
|
|
||
| 2. NGINX configuration: The NGINX `stub_status` module must be enabled, and the status endpoint must be accessible. For example: | ||
| ``` | ||
| server { | ||
| listen 80; | ||
| server_name localhost; | ||
| location /nginx_status { | ||
| stub_status on; | ||
| allow 127.0.0.1; | ||
| deny all; | ||
| } | ||
| } | ||
| ``` | ||
| 3. The NGINX endpoint configured in the Integration configuration from the UI. | ||
|
|
||
|
|
||
| ## Metrics reference | ||
|
|
||
| ### NGINX metrics | ||
| The [NGINX receiver]((https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/nginxreceiver/documentation.md)) collects performance metrics from the NGINX `stub_status` module. Key metrics include: | ||
|
|
||
|
|
||
| | Metric Name | Description | Type | Attributes | | ||
| |-------------|-------------|------|------------| | ||
| | `nginx.requests` | Total number of client requests | Counter | - | | ||
| | `nginx.connections_accepted` | Total number of accepted client connections | Counter | - | | ||
| | `nginx.connections_handled` | Total number of handled connections | Counter | - | | ||
| | `nginx.connections_current` | Current number of client connections by state | Gauge | `state`: `active`, `reading`, `writing`, `waiting` | | ||
|
|
||
| #### Connection States | ||
|
|
||
| - `active`: Currently active client connections | ||
| - `reading`: Connections currently reading request headers | ||
| - `writing`: Connections currently writing response to client | ||
| - `waiting`: Idle client connections waiting for a request | ||
|
|
||
| These metrics provide insights into: | ||
| - **Request volume and patterns** through request counts | ||
| - **Connection health** via accepted, and handled connection statistics | ||
| - **Server performance** using active, reading, writing, and waiting connection states | ||
|
|
||
| For a complete list of all available metrics and their detailed descriptions, refer to the [NGINX Receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/nginxreceiver/documentation.md) in the upstream OpenTelemetry Collector repository. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| receivers: | ||
| nginx: | ||
| endpoint: {{nginx_endpoint}} | ||
| collection_interval: {{period}} | ||
| processors: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it would be great if agent could provide more flexibility for resource detection processing. i.e it could be a setting at the agent policy level.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. additionally, do you know if agent injects any batching or memory limiter processor configs?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, I would expect the RDP processor to be a default one by Fleet at policy level.
Nothing that I see in the agent policy that is created. @jsoriano is there any ?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Do you think most packages are going to have this
Not that I know.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yes, the default processors would have to be defined in Fleet which we don't do yet and has some problems e.g. needing a Kibana release to change the config. Creating an output package type was one idea for how to solve this. Per policy global processors would also make sense but this could just be syntactic sugar that inserts them into each exporter pipeline.
Yes there is no other way for us to do this right now. I think you would want the detectors or even the whole processor configuration to be configurable though.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @ishleenk17 did you consider making this configurable as suggested by Craig?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jsoriano As we discussed in our meeting. Shall we just give a yaml box for procesors, keeping the whole yaml configuration configurable.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
We discussed this solution in the context of the Thinking on this case of the |
||
| resourcedetection: | ||
| detectors: ["system", "ec2"] | ||
| service: | ||
| pipelines: | ||
| metrics: | ||
| receivers: [nginx] | ||
| processors: [resourcedetection] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # newer versions go on top | ||
| - version: "0.1.0" | ||
| changes: | ||
| - description: Initial draft of the package | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/15651 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # NGINX OpenTelemetry Input Package | ||
|
|
||
| ## Overview | ||
| The NGINX OpenTelemetry Input Package integration for Elastic enables collection of telemetry data from NGINX web servers through OpenTelemetry protocols using the [nginxreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/nginxreceiver#nginx-receiver). | ||
|
|
||
| This integration facilitates comprehensive monitoring of NGINX web server performance, request processing, error tracking, and operational metrics to provide insights into web application infrastructure health and performance. | ||
|
|
||
| ### Compatibility | ||
| This Integration uses the upstream [nginxreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/nginxreceiver#nginx-receiver) to collect the metrics. | ||
|
|
||
| ### How it works | ||
| This integration receives telemetry data from NGINX servers by configuring the NGINX endpoint in the integration, which then gets applied to the nginxreceiver present in the EDOT collector, which then forwards the data to Elastic Agent. The Elastic Agent processes and enriches the data before sending it to Elasticsearch for indexing and analysis. Once the data arrives into Elasticsearch, its corresponding [NGINX OpenTelemetry Assets Package](https://www.elastic.co/docs/reference/integrations/nginx_otel) gets auto installed and the dashboards light up. | ||
|
|
||
| ## What data does this integration collect? | ||
| The NGINX OpenTelemetry Input Package integration collects telemetry data of the following types: | ||
|
|
||
| * **Metrics** - Performance metrics including request rates, response times, connection counts, and server status | ||
|
|
||
|
|
||
| ## What do I need to use this integration? | ||
| 1. Permissions required: The collector requires access to the NGINX `stub_status` endpoint (for example, http://localhost:80/nginx_status). When running the collector, make sure you have the appropriate permissions to access this endpoint. | ||
|
|
||
| 2. NGINX configuration: The NGINX `stub_status` module must be enabled, and the status endpoint must be accessible. For example: | ||
| ``` | ||
| server { | ||
| listen 80; | ||
| server_name localhost; | ||
| location /nginx_status { | ||
| stub_status on; | ||
| allow 127.0.0.1; | ||
| deny all; | ||
| } | ||
| } | ||
| ``` | ||
| 3. The NGINX endpoint configured in the Integration configuration from the UI. | ||
|
|
||
|
|
||
| ## Metrics reference | ||
|
|
||
| ### NGINX metrics | ||
| The [NGINX receiver]((https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/nginxreceiver/documentation.md)) collects performance metrics from the NGINX `stub_status` module. Key metrics include: | ||
|
|
||
|
|
||
| | Metric Name | Description | Type | Attributes | | ||
| |-------------|-------------|------|------------| | ||
| | `nginx.requests` | Total number of client requests | Counter | - | | ||
| | `nginx.connections_accepted` | Total number of accepted client connections | Counter | - | | ||
| | `nginx.connections_handled` | Total number of handled connections | Counter | - | | ||
| | `nginx.connections_current` | Current number of client connections by state | Gauge | `state`: `active`, `reading`, `writing`, `waiting` | | ||
|
|
||
| #### Connection States | ||
|
|
||
| - `active`: Currently active client connections | ||
| - `reading`: Connections currently reading request headers | ||
| - `writing`: Connections currently writing response to client | ||
| - `waiting`: Idle client connections waiting for a request | ||
|
|
||
| These metrics provide insights into: | ||
| - **Request volume and patterns** through request counts | ||
| - **Connection health** via accepted, and handled connection statistics | ||
| - **Server performance** using active, reading, writing, and waiting connection states | ||
|
|
||
| For a complete list of all available metrics and their detailed descriptions, refer to the [NGINX Receiver documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/nginxreceiver/documentation.md) in the upstream OpenTelemetry Collector repository. |
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This file can be removed. OTel input packages rely on pre-built component templates. If this is the only fields file, build will fail, till we release elastic/package-spec#994 (probably this week).
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, this is mostly going to be the only file under fields. I don't expect any mappings to go in here.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
We can rely on the pre-installed template and the dynamic mappings it includes. We still have the option of defining fields if needed in some case, but defining fields will be completely optional for OTel input packages. It will be actually expected for OTel input packages to don't contain any field definition.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Did you consider removing this file? You can leave it empty or wait for next elastic-package release (coming soon) if you have problems with validation.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jsoriano On removing the base-fields.yml
Error: building package failed: invalid content found in built zip package: found 4 validation errors:
So for now we need to have this file with the fields, or we wait for the next release of elastic package. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| - name: data_stream.type | ||
tommyers-elastic marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| type: constant_keyword | ||
| description: Data stream type. | ||
| - name: data_stream.dataset | ||
| type: constant_keyword | ||
| description: Data stream dataset. | ||
| - name: data_stream.namespace | ||
| type: constant_keyword | ||
| description: Data stream namespace. | ||
| - name: '@timestamp' | ||
| type: date | ||
| description: Event timestamp. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i wonder how much of this we should just delegate to the upstream receiver. i.e. i could see an argument for literally just something like
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am open to discussions on this one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@daniela-elastic WDYT ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like that, if we could delegate to the upstream receiver this means less work on our side. Do you foresee any cons to delegating to the upstream receiver? Wouldn't want to go into one way doors.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tommyers-elastic : Maybe we can write this in the Metric reference section. Other parts of the documentation should be there IMO. WDYT ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tommyers-elastic : we would go ahead with the documentation as is for now, as we discussed.
I can maybe remove the metrics reference section here as for that they can refer to the upstream receiver.