Querying contexts column in Clickhouse is expensive and can cause incidents.

[wedamija]

From @dbanda

We would like some clarity on what fields are allowed to be queried in alerts. We had an incident where a subscription query was querying the contexts column in clickhouse on the errors dataset . This column is unbounded and we got OOMs. Can we ban querying contexts ? If not, are there any constraints in place to control what gets stuffed in the field? Ideally, we shouldn’t allow querying fields where customers can stuff in arbitrarily large data.

Basically, we had an incident caused by an alert query on the contexts columns. Because the column is unbounded it leads to OOMs

• Ban querying contexts. Our preferred solution. We shouldn’t be allowing queries on such fields where the payload can be arbitrarily large.
• If we must query it (or allow users to query it), cap its size so that its bounded.
• If there’s specific keys we need to query from there, those keys should probably be promoted.

It's not clear if we intentionally allow contexts to be queried. We typically don't really like it to be, since the data in the column is unbounded and can cause OOMs in Clickhouse due to the way arrayjoins work.

Metric alerts should support the same searches as issue search and discover, so that we have a consistent experience. So whichever path we take will need to be consistent.

Ideally we would be able to ban querying contexts, but it's not clear on if this is acceptable and how often people are querying that field. We should investigate this and figure out whether it's ok to ban querying contexts, and if not figure out what the best path forward is.