[prometheus] Cannot scrape only from required namespaces

[vaibhhavv]

Describe the bug a clear and concise description of what the bug is.

Use case: I want my Prometheus to scrape metrics only from a few namespaces inside my Kubernetes cluster
I am using Prometheus helm chart v26.0.0 and it has Prometheus v3.0.0.

I only want to scrape metrics from a few namespaces. So I used the below parameters in values.yaml, which tells that it can help in achieving that use case.

  # Enable only the release namespace for monitoring. By default all namespaces are monitored.
  # If releaseNamespace and namespaces are both set a merged list will be monitored.
  releaseNamespace: true

  ## namespaces to monitor (instead of monitoring all - clusterwide). Needed if you want to run without Cluster-admin privileges.
  namespaces:
    - namespace1
    - namespace2

Actual Behaviour: It stops scraping from all namespaces, includingnamespace1 & namespace2 from which we want to scrape.
Also, below are the Prometheus logs from that time.

k8s_client_runtime logger=UnhandledError err="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:prometheus-namespace:my-prometheus-server\" cannot list resource \"services\" in API group \"\" at the cluster scope"

Expected Behaviour: It should only scrape from namespace1 & namespace2, and should not scrape from other namespaces.

Note: If I comment out the above-mentioned parameters or do not set the, then my Prometheus scrape from all namespaces smoothly without any error in the logs.

I think these parameters are not doing their intended work and are misbehaving. Experts, could you please help with this?

What's your helm version?

v3.16.3

What's your kubectl version?

v1.31.3

Which chart?

prometheus

What's the chart version?

26.0.0

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

NA

Anything else we need to know?

No response

[zeritti]

If server.namespaces is set, a role binding to the cluster role given in server.useExistingClusterRoleName will be created for prometheus in each of the namespaces that will provide access to namespaced-only resources per namespace.

The likely reason for the error is that scrape configs do not indicate namespaced discovery - prometheus attempts cluster-wide discovery but finds no targets as it does not have permissions for such discovery.
If you limit prometheus to selected namespaces through a rolebinding as opposed to cluster-wide discovery, the scrape configs must be adjusted, i.e. telling prometheus to discover targets per given namespaces. Consider a sample scrape config like this which won't work in this scenario:

- job_name: 'myapps'
  kubernetes_sd_configs:
   - role: endpoints
  scheme: http

If setting e.g

server:
  namespaces:
    - ns1
    - ns2
    - ns3
  releaseNamespace: true
  useExistingClusterRoleName: myclusterrole

then each kubernetes_sd_config must be adjusted with the namespaces field:

- job_name: 'myapps'
  kubernetes_sd_configs:
   - role: endpoints
     namespaces:
       names:
       - ns1
       - ns2
       - ns3
       own_namespace: true

See also kubernetes_sd_config.

[vaibhhavv]

Hi @zeritti, thanks for the brief input.
I tried the way you suggested, but still no luck.
With the mentioned configuration, it stops scraping from all the namespaces, even those namespaces which we mention under server.namespaces or under every kubernetes_sd_configs.

[zeritti]

There is an example of the namespaced discovery in the chart's CI directory which works as intended (validated in different clusters) - prometheus executes SD against its own namespace and successfully discovers and scrapes targets in this namespace, in this case at least that of kube-state-metrics. NOTE This example assumes the permission to create a cluster role is available to the deployer.