Prometheus /metrics Exposed on 8080 and not Default 9000

[cmcguirk17]

System Info

Docker Image: huggingface/text-embeddings-inference
Version: 1.8.1

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Calling curl localhost:9000/metrics returns nothing while curl localhost:8080/metrics seems to return the expected metrics.
9000 is listed as being the default, and no PROMETHEUS_PORT env is set. It seems that 8080 is actually the default port?

Expected behavior

9000 should be the default port.

Adding to this, changing the PROMETHEUS_PORT environment variable did not seem to successfully apply to the container.

[alvarobartt]

Hey @cmcguirk17 thanks for opening the issue! Indeed the --prometheus-port argument only applies for gRPC where the /metrics endpoint is deployed via HTTP under that route (we could maybe improve the documentation around it to be more explicit), whereas for the HTTP API there's no need to open another port and instead the default port will be used (either 3000 locally or 80 via Docker), but correct me if I'm wrong @kozistr 🤗

[alvarobartt]

Also may I ask what was the intent on changing the default HTTP port for the /metrics endpoint to something different e.g. 9000? Or was that just because the argument felt misleading? That would help clarify things, and we'll improve on the documentation side of things to make sure that's clear (once @kozistr confirms as per #589), thanks again!

[cmcguirk17]

Thanks @alvarobartt, yes I think I was confused by the argument here README.md.
I'm using 8080 for some other networking, so hoping keep metrics confined to their own port.

[kozistr]

Hey @cmcguirk17 thanks for opening the issue! Indeed the --prometheus-port argument only applies for gRPC where the /metrics endpoint is deployed via HTTP under that route (we could maybe improve the documentation around it to be more explicit), whereas for the HTTP API there's no need to open another port and instead the default port will be used (either 3000 locally or 80 via Docker), but correct me if I'm wrong @kozistr 🤗

Hi @alvarobartt @cmcguirk17! As far as I know, --prometheus-port argument applies to both gRPC and http. The actual Prometheus will be available at host:{prometheus-port}, and its metrics can be accessed through the TEI /metrics API (instead of connecting directly to Prometheus). In the meantime, maybe we cannot directly access the Prometheus currently unless we spawn the exporter, and this was resolved at PR 678.

In short, if you have a problem with port 8080, then running the TEI with a different port should resolve the issue!

I hope it helps you :) And please feel free to correct me if I'm wrong!

[cmcguirk17]

Hey, I have tried setting the --prometheus-port argument but it doesn't seem to apply to HTTP. Is there any solution for this? Or will I only be able to access /metrics on 8080?

[kozistr]

Hey, I have tried setting the --prometheus-port argument but it doesn't seem to apply to HTTP. Is there any solution for this? Or will I only be able to access /metrics on 8080?

Hi! Maybe, you can start the TEI with a different port, likewise 8888 via --port argument, and then you can get the prometheus metrics by calling like localhost:8888/metrics!

[OlivierDehaene]

@kozistr, @alvarobartt,

The --prometheus-port only applies to gRPC because in HTTP we only use the prom_builder to create a recorder

text-embeddings-inference/router/src/http/server.rs

Line 1699 in 9ef569d

let (recorder, _) = prom_builder

and the prom_handle is then served with Axum directly on the same port as the other routes

text-embeddings-inference/router/src/http/server.rs

Line 1594 in 9ef569d

async fn metrics(prom_handle: Extension<PrometheusHandle>) -> String {

For gRPC, the prom_builder is actually installed with the port specified by the user as --prometheus-port

text-embeddings-inference/router/src/grpc/server.rs

Line 1376 in 9ef569d

prom_builder.install()?;

I guess a way to streamline this behavior could be to use Axum to serve the metrics if prometheus_port == port and install the prom_builder otherwise but migrating this change in default port might be hard (since the default is 8080 for HTTP and 9000 for gRPC).

[kozistr]

@kozistr, @alvarobartt,

The --prometheus-port only applies to gRPC because in HTTP we only use the prom_builder to create a recorder

text-embeddings-inference/router/src/http/server.rs

Line 1699 in 9ef569d

let (recorder, _) = prom_builder
and the prom_handle is then served with Axum directly on the same port as the other routes
text-embeddings-inference/router/src/http/server.rs

Line 1594 in 9ef569d

async fn metrics(prom_handle: Extension) -> String {
For gRPC, the prom_builder is actually installed with the port specified by the user as --prometheus-port

text-embeddings-inference/router/src/grpc/server.rs

Line 1376 in 9ef569d

prom_builder.install()?;
I guess a way to streamline this behavior could be to use Axum to serve the metrics if prometheus_port == port and install the prom_builder otherwise but migrating this change in default port might be hard (since the default is 8080 for HTTP and 9000 for gRPC).

oh i missed that part. thanks for the correction!

and sorry for the confusion by my comments above

[alvarobartt]

Fair enough and thanks for the clarification @OlivierDehaene, then guessing that it makes sense to say that --prometheus-port only applies for gRPC, as mentioned on the very first reply, and there's not a clear need of changing the behaviour for HTTP, but rather only update the default value based on the active feature (http or grpc) and update the --prometheus-port argument description, right? Thanks in advance 🤗

[augustas1]

Just for consideration - there are some edge cases where people would want metrics on a separate port, though (the reason I created PR #678).
For example we use mTLS on the main port to secure transferring user data in our cluster, but we don't need that for operational data (metrics) - and the only way to do that is serve metrics on a different port.

Or are there alternatives - e.g. are metrics exported via OTLP as well?