Missing metrics on services overview page

[CaueP]

Hello 👋🏻

I noticed that one of my services doesn't have the metrics on the overview dashboard. After some time loading, it shows an error API responded with 400 - error in builder querie status: error, errors: {}

Is there something that I can do to fill these charts?

Some screenshots:
Empty charts (loading) on the Overview tab of my service:

Error that pops up after a while:

The other tabs works fine:

Debugging

Logs that looks related to the error

Chrome's Network tab with errors

[srikanthccv]

This page shows the RED metrics based on the entry-point span operations for the service. When such entry-points operations are unusually high this error occurs. The span names (referred to as endpoints) should have low cardinal values however you seem to have some id values in it. Please refer to the OTEL docs and address the high cardinal span names issue https://opentelemetry.io/docs/specs/otel/trace/api/#span.

[CaueP]

Another screen that we can see that I have only spans from 6/26 and on:

[srikanthccv]

It will be fixed in #5404 soon. Meanwhile, you can exec into clickhouse and run DELETE FROM signoz_traces.distributed_top_level_operations ON CLUSTER cluster . This will remove all the old data and repopulate the table again as the data gets inserted.

[srikanthccv]

and attempt to fix the issue in the instrumentation library open-telemetry/opentelemetry-python-contrib#2650

[CaueP]

It will be fixed in #5404 soon. Meanwhile, you can exec into clickhouse and run DELETE FROM signoz_traces.distributed_top_level_operations ON CLUSTER cluster . This will remove all the old data and repopulate the table again as the data gets inserted.

Good to know, thanks a lot!

and attempt to fix the issue in the instrumentation library open-telemetry/opentelemetry-python-contrib#2650

Nice! 🚀

[CaueP]

It will be fixed in #5404 soon. Meanwhile, you can exec into clickhouse and run DELETE FROM signoz_traces.distributed_top_level_operations ON CLUSTER cluster . This will remove all the old data and repopulate the table again as the data gets inserted.

Just for the record, this delete operation on distributed_top_level_operations table didn't work, probably because it is distributed table as I found out on this migration. The table name should be top_level_operations instead.

I changed the query to perform the delete on the table top_level_operations only for my service with problems, and it worked. The statement used was this:

DELETE FROM signoz_traces.top_level_operations WHERE serviceName = 'MY_SERVICE_NAME'