Idea for discussion: GUI for enrichment

[GeorgLink]

The enrichment of raw indexes is following a set of formulas. Right now the formulas are embedded in code.

The idea I want to discuss is to make the formulas dynamic and customizable by the user of the platform. A preset of formulas may be exported and imported. But it would be nice to have a GUI for adding more formulas.

This is similar to Painless, but in a way that the data is stored in the enriched indexes.

This will make the platform more configurable and customizable for different users.
The users can exchange formulas on the raw data, e.g., through CHAOSS discussions.

[valeriocos]

I guess pandas notebooks could work, ad-hoc scripts or a study could also do the job.

Feel free to elaborate more on this idea.

[valeriocos]

@GeorgLink there is no activity on this issue. Can you elaborate a bit more on this idea? Do you have some refs of tools doing something similar?

[GeorgLink]

I don't know a good comparison.

My idea is to have very few hard-coded enrichment formulas.
Let's take the utc_weekday and utc_time_of_day calculations that we added on our call as an example.
Rather than writing it into GELK, if we had a GUI for defining, importing, and exporting this kind of calculations, we could have created the new fields without having to update GrimoireLab.

The imaginary GUI provides a list of available indexes and fields. From those fields, formulas are created for enriched fields. If an enriched field is the input to another formula, the system has to detect the dependency and resolve the formulas in the right order.

The goal of this GUI:

• reduce the number of fields hardcoded in GELK enrichment
• make enrichment formulas sharable (similar to how Sigils are used for sharing visualizations)
• enable users to create custom enrichments that we haven't even thought of.

[GeorgLink]

@valeriocos and I had a chat about this idea today.

What was a revelation for me is that following above idea, the enrichment process would be expressed in a domain specific language (DSL) and GELK would become a parser or compilor for the DSL and execute it. The benefit is that the DSL can be shared with others while providing the flexibility to define custom metrics. The big challenge would be aligning the visualizations because they depend on the data fields created by DSL, so minor differences in the DSL of one instance may break visualizations.