Improve Prometheus docs, include nginx ingress

Author: Joshua Lambert

config/prometheus/additional_metrics.yml

@@ -1,3 +1,33 @@
+- group: Response metrics (NGINX Ingress)
+  priority: 10
+  metrics:
+  - title: "Throughput"
+    y_label: "Requests / Sec"
+    required_metrics:
+      - nginx_upstream_requests_total
+    weight: 1
+    queries:
+    - query_range: 'sum(rate(nginx_upstream_requests_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m]))'
+      label: Total
+      unit: req / sec
+  - title: "Latency"
+    y_label: "Latency (ms)"
+    required_metrics:
+      - nginx_upstream_response_msecs_avg
+    weight: 1
+    queries:
+    - query_range: 'avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"})'
+      label: Average
+      unit: ms
+  - title: "HTTP Error Rate"
+    y_label: "HTTP 500 Errors / Sec"
+    required_metrics:
+      - nginx_upstream_responses_total
+    weight: 1
+    queries:
+    - query_range: 'sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m]))'
+      label: HTTP Errors
+      unit: "errors / sec"
 - group: Response metrics (HA Proxy)
   priority: 10
   metrics:
@@ -68,18 +98,18 @@
       - nginx_upstream_response_msecs_avg
     weight: 1
     queries:
-    - query_range: 'avg(nginx_upstream_response_msecs_avg{%{environment_filter}}) * 1000'
+    - query_range: 'avg(nginx_upstream_response_msecs_avg{%{environment_filter}})'
       label: Upstream
       unit: ms
   - title: "HTTP Error Rate"
-    y_label: "Error Rate (%)"
+    y_label: "HTTP 500 Errors / Sec"
     required_metrics:
       - nginx_responses_total
     weight: 1
     queries:
-    - query_range: 'sum(rate(nginx_responses_total{status_code="5xx", %{environment_filter}}[2m])) / sum(rate(nginx_requests_total{server_zone!="*", server_zone!="_", %{environment_filter}}[2m]))'
+    - query_range: 'sum(rate(nginx_responses_total{status_code="5xx", %{environment_filter}}[2m]))'
       label: HTTP Errors
-      unit: "%"
+      unit: "errors / sec"
 - group: System metrics (Kubernetes)
   priority: 5
   metrics:

doc/ci/environments.md

@@ -607,10 +607,9 @@ exist, you should see something like:
 - With GitLab 9.2, all deployments to an environment are shown directly on the
   monitoring dashboard
 
-If you have enabled Prometheus for collecting metrics, you can monitor the performance behavior of your app
-through the environments.
+If you have enabled [Prometheus for monitoring system and response metrics](https://docs.gitlab.com/ee/user/project/integrations/prometheus.html), you can monitor the performance behavior of your app running in each environment.
 
-Once configured, GitLab will attempt to retrieve performance metrics for any
+Once configured, GitLab will attempt to retrieve [supported performance metrics](https://docs.gitlab.com/ee/user/project/integrations/prometheus_library/metrics.html) for any
 environment which has had a successful deployment. If monitoring data was
 successfully retrieved, a Monitoring button will appear on the environment's
 detail page.

doc/user/project/integrations/prometheus.md

@@ -40,7 +40,7 @@ Installing and configuring Prometheus to monitor applications is fairly straight
 ### Configuring Omnibus GitLab Prometheus to monitor Kubernetes deployments
 
 With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled
-version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>). 
+version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>).
 
 1. Read how to configure the bundled Prometheus server in the
    [Administration guide][gitlab-prometheus-k8s-monitor].
@@ -133,6 +133,8 @@ to integrate with.
 Once configured, GitLab will attempt to retrieve performance metrics for any
 environment which has had a successful deployment.
 
+GitLab will automatically scan the Prometheus server for known metrics and attempt to identify the metrics for a particular environment. The supported metrics and scan process is detailed in our [Prometheus Metric Library documentation](prometheus_library/metrics.html).
+
 [Learn more about monitoring environments.](../../../ci/environments.md#monitoring-environments)
 
 ## Determining the performance impact of a merge
@@ -174,7 +176,7 @@ If the "Attempting to load performance data" screen continues to appear, it coul
 [prometheus-docker-image]: https://hub.docker.com/r/prom/prometheus/
 [prometheus-yml]:samples/prometheus.yml
 [gitlab.com-ip-range]: https://gitlab.com/gitlab-com/infrastructure/issues/434
-[ci-environment-slug]: https://docs.gitlab.com/ce/ci/variables/#predefined-variables-environment-variables
+[ci-environment-slug]: ../../../ci/variables/#predefined-variables-environment-variables
 [ce-8935]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935
 [ce-10408]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10408
 [promgldocs]: ../../../administration/monitoring/prometheus/index.md

doc/user/project/integrations/prometheus_library/metrics.md

@@ -4,6 +4,7 @@
 GitLab offers automatic detection of select [Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/). Currently supported exporters are:
 * [Kubernetes](kubernetes.md)
 * [NGINX](nginx.md)
+* [NGINX Ingress Controller](nginx_ingress.md)
 * [HAProxy](haproxy.md)
 * [Amazon Cloud Watch](cloudwatch.md)
 
@@ -14,10 +15,7 @@ We have tried to surface the most important metrics for each exporter, and will
 GitLab retrieves performance data from the configured Prometheus server, and attempts to identifying the presence of known metrics. Once identified, GitLab then needs to be able to map the data to a particular environment.
 
 In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do that,
-GitLab will look for the required metrics which have a label that
-matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug].
-
-For example if you are deploying to an environment named `production`, there must be a label for the metric with the value of `production`.
+GitLab uses the defined queries and fills in the environment specific variables. Typically this involves looking for the [$CI_ENVIRONMENT_SLUG](https://docs.gitlab.com/ee/ci/variables/#predefined-variables-environment-variables), but may also include other information such as the project's Kubernetes namespace. Each search query is defined in the [exporter specific documentation](#prometheus-metrics-library).
 
 ## Adding to the library
 

doc/user/project/integrations/prometheus_library/nginx.md

@@ -8,8 +8,8 @@ GitLab has support for automatically detecting and monitoring NGINX. This is pro
 | Name | Query |
 | ---- | ----- |
 | Throughput (req/sec) | sum(rate(nginx_requests_total{server_zone!="*", server_zone!="_", %{environment_filter}}[2m])) |
-| Latency (ms) | avg(nginx_upstream_response_msecs_avg{%{environment_filter}}) * 1000 |
-| HTTP Error Rate (%) | sum(rate(haproxy_frontend_http_responses_total{code="5xx",%{environment_filter}}[2m])) / sum(rate(haproxy_frontend_http_responses_total{%{environment_filter}}[2m])) |
+| Latency (ms) | avg(nginx_upstream_response_msecs_avg{%{environment_filter}}) |
+| HTTP Error Rate (HTTP Errors / sec) | rate(nginx_responses_total{status_code="5xx", %{environment_filter}}[2m])) |
 
 ## Configuring Prometheus to monitor for NGINX metrics
 

doc/user/project/integrations/prometheus_library/nginx_ingress.md

@@ -0,0 +1,25 @@
+# Monitoring NGINX Ingress Controller
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13438) in GitLab 9.5
+
+GitLab has support for automatically detecting and monitoring the Kubernetes NGINX ingress controller. This is provided by leveraging the built in Prometheus metrics included in [version 0.9.0](https://github.com/kubernetes/ingress/blob/master/controllers/nginx/Changelog.md#09-beta1) of the ingress.
+
+## Metrics supported
+
+| Name | Query |
+| ---- | ----- |
+| Throughput (req/sec) | sum(rate(nginx_upstream_requests_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) |
+| Latency (ms) | avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}) |
+| HTTP Error Rate (HTTP Errors / sec) | sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) |
+
+## Configuring Prometheus to monitor for NGINX ingress metrics
+
+The easiest way to get started is to use at least version 0.9.0 of [NGINX ingress](https://github.com/kubernetes/ingress/tree/master/controllers/nginx). If you are using NGINX as your Kubernetes ingress, there is [direct support](https://github.com/kubernetes/ingress/pull/423) for enabling Prometheus monitoring in the 0.9.0 release.
+
+If you have deployed with the [gitlab-omnibus](https://docs.gitlab.com/ee/install/kubernetes/gitlab_omnibus.md) Helm chart, these metrics will be automatically enabled and annotated for Prometheus monitoring.
+
+## Specifying the Environment label
+
+In order to isolate and only display relevant metrics for a given environment
+however, GitLab needs a method to detect which labels are associated. To do this, GitLab will search metrics with appropriate labels. In this case, the `upstream` label must be of the form `<Kubernetes Namespace>-<CI_ENVIRONMENT_SLUG>-*`.
+
+If you have used [Auto Deploy](https://docs.gitlab.com/ee/ci/autodeploy/index.html) to deploy your app, this format will be used automatically and metrics will be detected with no action on your part.