monitoring: expose storage limit and disk metrics for TBS monitoring

[carsonip]

From comment #14247 (comment)

8.x and 9.x: As apm-server exposes lsm_size and value_log_size as monitoring metrics, expose configured storage limit as well. Then it will be possible to plot db size vs storage limit, and removes the need to dig into the logs for the configured storage limit.

9.x: Add monitoring metrics to monitor the disk utilization check.

It is up to the implementer to decide what metric to emit, whether it is a combined metric or a few separate metrics. The actual work involves adding this metric in apm-server code, and update relevant mappings in ES, integrations and metricbeat repo. See #13475 for an example.

[raultorrecilla]

this is important as we would like to add some UI to tell the customers how much disk they are using elastic/kibana#226600

[carsonip]

This is less straightforward than before. In 8.x there is one configured storage limit, and we compare db size against that. However, from 9.0, by default the storage limit is 0, and instead of comparing db size against that, we compare the disk_used vs disk_total so that TBS doesn't write to the last 20% of the disk. It does not involve db size and storage limit. I'm a bit hesitant to plumb these implementation details to monitoring metrics and all the way to the UI. We should rethink what should be exposed.

[carsonip]

apm-server.sampling.tail.events.failed_writes is a counter metric that records the number of failed writes, no matter it is then discarded or directly indexed to ES. If this per-apm-server counter increases, it means apm-server is running into storage issues. I believe it is good enough for elastic/kibana#226600

[carsonip]

That said, for elastic/kibana#226600 , I wonder how kibana would have access to monitoring metrics (think about stack monitoring). If we cannot surface this by default on ECH, I wonder how useful this will be.

[carsonip]

In case we move ahead with implementing this, the actual work involves adding this metric in apm-server code, and update relevant mappings in ES, integrations and metricbeat repo. See #13475 for an example.

[carsonip]

As discussed during weekly, we would like to expand the scope of this task and ensure that it is relevant and valuable to both 8.x and 9.x. The description has been updated to reflect that. The actual metrics used and the design are up to the implementer.

[isaacaflores2]

As mentioned during the weekly, this story is similar to: #18084. Cross posting just to make sure everyone is aware. Depending on how the metric for 9.X is implement it might cover one of the cases in #18084 (I could be wrong, so feel free to disregard)

[raultorrecilla]

we need to remember to add docs as done here