Implement relabeling debugger/inspector in Alloy UI and or documentation.

[simonswine]

Background

I am a Prometheus user for many years, but when it comes to relabelling rules, that still doesn't help me from avoiding unexpected outcomes and debugging sessions.

They are pretty hard to debug and once there is enough complexity either in terms of rules or discovered targets, it gets quite hard to understand.

Proposal

I would like Alloy to implement an UI based relabel debugger, inspired by the existing examples like:

• https://play.victoriametrics.com/select/0/vmui/?#/relabeling
• https://relabeler.promlabs.com/

As a major difference I would like Alloy UI to introduce a link from relabel compoments to that debugger, that would feed the current rule set and/or selected discovery.Targets.

I think this could be achieved frontend only, and then we would also be able to host a public play ground (like https://go.dev/play/), that purely works on information submitted in the URL fragment, without that many privacy concerns:

http://grafana.com/docs/alloy/latest/relabel-debug#

While this is helpful for us as experts, it will also make it easier for newer users that can play with example relabelling rules, share them to get help or just tweak them to do what they intend to.

cc @clayton-cornell @korniltsev @jdbaldry

[korniltsev]

I love this idea and wanted to implement this myself looong time. May be a good goal for a hackathon project?

Our profiling configuration is 1k loc. Combined with discovery.kubernetes for a big cluster makes it imposible to debug anything.

What I currently do to debug any issues:

1. copy the configuration to run locally
2. modify config to mock/replace discovery.kubernetes and use only one target
3. obtaining a single target is also a challenge, alloy UI hangs with all the targets from a big cluster, so I curl alloy (megabytes of json) and process it.
4. Then I try to remove /add rules until I understand why something is broken

Unresponsive alloy UI probably worth a separate issue

Another idea for debugger inspiration could be this feature (not directly applicable, but still) https://www.jetbrains.com/guide/java/tips/debugging-streams/

[ptodev]

Related to #1921

[wildum]

Hey, thanks for the proposal :)

I see two use cases:

• the user wants to build his real pipeline correctly
• the user wants to experiment with Alloy without setting up an entire env

For the 1st one, my approach to processing steps like relabeling is to use live debugging. You can check in the UI the data that exits the different components in the pipeline and modify your relabelling rules till the data that reaches your last component matches your expectations. Here is a video where I do this: https://www.youtube.com/watch?v=_MbB8IVKMfw&ab_channel=Grafana.
It's fairly easy to do because you can edit the config and hit the reload endpoint to see the changes. No need to restart Alloy or reload the UI. I'm also currently implementing a live graph where you see the data passing through the graph.

For the 2nd use case where the user just want synthetic data to try things, there is a killer coda tutorial available to send logs to loki: https://grafana.com/docs/alloy/latest/tutorials/send-logs-to-loki/#before-you-begin. Maybe we could have a tutorial like this for the other main pipelines?