cardinality explosion due to thread.name in runtime-telemetry-java17 metrics

[stevesea]

Describe the bug

in some of our apps, we're seeing a cardinality explosion when enabling runtime-telemetry-java17

the culprits seem to be the following metrics:

• jvm.network.io
• jvm.network.time
• jvm.memory.allocation

each of those metrics include a thread.name attribute that can have many variations.

Steps to reproduce

set otel.instrumentation.runtime-telemetry-java17.enable-all=true for an app that allocates memory or does network io on many different threads.

Expected behavior

thread.name should only be included if it's low cardinality

Actual behavior

thread.name has many, many values (depending on app framework/libraries used)

Javaagent or library instrumentation version

2.10.0, 2.11.0, and 2.13.1

Environment

eclipse-temurin:17-jre

Additional context

No response

[stevesea]

There's also a thread.name on the jvm.cpu.longlock metric. But, I didn't see that causing a cardinality problem in these applications.

[jeanbisutti]

Are you using the OpenTelemetry Java agent, the OpenTelemetry starter or another thing?

Perhaps you could try to drop some metrics at the metric export level.

[stevesea]

we're using the OpenTelemetry Java agent in two of the apps with problems (2.10.1, and 2.11.0), and started using a custom agent distribution with the other app with the issue (2.13.1).

yes, I plan to workaround the problem by:

• dropping thread.name for these metrics in our gateway otel-collectors
• updating our custom agent distribution with a metrics view like :

      for (String metricName : Arrays.asList("jvm.network.io", "jvm.network.time", "jvm.memory.allocation", "jvm.cpu.longlock")) {
        meterProviderBuilder.registerView(
            InstrumentSelector.builder().setName(metricName).build(),
            View.builder().setAttributeFilter(attrName -> !"thread.name".equals(attrName)).build()
        );
      }

• either disabling runtime-telemetry-java17 in the apps using the standard OTel agent, or updating them to use our custom agent distribution

[trask]

I think I'd support dropping thread.name from these experimental metrics (or making them opt-in), I'm not sure if these are very widely used.

@SylvainJuge @robsunday @PeterF778 are we capturing thread.name in any of the jmx insights / scraper metrics?

[SylvainJuge]

@SylvainJuge @robsunday @PeterF778 are we capturing thread.name in any of the jmx insights / scraper metrics?

No, the thread.name is not captured through JMX, even with JMX gatherer.

I am not aware of the usages in other platforms, but this sounds like an attribute where the cardinality will need to be assessed carefully for every usage, so maybe adding something in the semconv about this would be relevant. The same probably applies to thread.id.

For example:

• using it as a metric attribute is likely to create an high cardinality value.
• using it for an in-flight information like the sampler in contrib should be fine because it's not stored nor sent.