Healthy endpoint

[rotarur]

Problem statement
When some pod has an issue to connect to the database or any other dependent service, it's still healthy for the loadbalancer which send traffic to it increasing the errors for the customers.

In order to reduce or completely remove the errors during these connection issues I'd like to propose to add a /healtz (yes it's written healthz instead of health to avoid overlapping with any other similar endpoint for the application and it's a common usage to name this one like that for the kubernetes) which will tell the loadbalancer that the pod is not healthy and to stop sending traffic to it. This endpoint will also help kubelet in kubernetes (or any other automated process) to restart that unhealthy pod before it's becoming critical to the whole system.

Expectation proposal
To have an endpoint exposed by the indexer, like /healthz, which I can hit to know that the indexer and all its dependencies are healthy (these dependencies can be critical to the indexer and not critical which don't affect the healthy endpoint)