🚀 Proposal: Add Pipeline (|) Query Syntax to UQL for "Mirandor Core".

[dostiharise]

🚀 Proposal: Add Pipeline (|) Query Syntax to UQL for "Mirandor Core".

Summary

UQL provides unified querying across Metrics, Logs, and Traces, but it lacks a pipeline-style syntax, now the dominant standard across modern observability and analytics systems.
Introducing a pipe-driven DSL improves composability, readability, and aligns Mirador-core with leading systems such as Lightstep UQL, Elasticsearch ES|QL, Azure Log Analytics (KQL), OpenSearch PPL, and BigQuery’s SQL-style staged operators.

---

🌍 Industry Precedents (Expanded)

Lightstep UQL

• Fully pipeline-driven.
• “A UQL query is piped, made up of two or more stages.”

Elasticsearch ES|QL

• Described as a “new piped language for fast, multi-stage data analysis.”
• Uses | to chain analytics steps.

Azure Log Analytics (KQL)

• Entire language is built on pipe-based tabular expressions.
• Microsoft: “Operators are sequenced by a |. Data flows from one operator to the next.”

OpenSearch PPL (Piped Processing Language)

• Log analytics engine with first-class pipeline syntax.

• Syntax example:

  source=logs 
  | where status=500 
  | stats count(*) by service 
  

• Defines a clear sequence of pipeline operators similar to Splunk and KQL.

BigQuery SQL (Staged Transformations)

• While SQL-based, BigQuery strongly encourages stepwise transformations via CTE pipelines (WITH → SELECT → nested transforms).
• Equivalent to pipeline semantics—each stage refines the dataset.
• Common pattern in their docs: “Break analysis into sequential transformation blocks.”

Together, these establish a unified direction: pipeline-based transformations are the industry default for massive-scale log, metric, trace, and analytic systems.

---

💡 Motivation

Pipeline syntax matches the natural mental model for telemetry workflows:
fetch → transform → filter → aggregate → correlate.

It reduces cognitive load, improves step-by-step reasoning, and creates a foundation for future operators (AI, anomaly detection, join heuristics).

---

📈 Benefits at Large Scale

1. Streaming execution
Stages allow streaming instead of building huge intermediate tables → far lower RAM.

2. Pushdown and pruning
Structured pipelines enable the query planner to push filters earlier and drop unused fields.

3. Predictable CPU per operator
Each stage is a well-defined operator optimized in isolation.

4. Horizontal parallelism
Clear stage boundaries allow sharding log/metric/trace operations across cores or nodes.

5. Faster interactive dashboards
Upstream filtering and projection shrink data early → lower latency.

6. Efficient cross-domain correlation
Pipeline-model enables reorganizing joins between metrics/logs/traces to minimize data movement.

7. Extensibility
Add new operators (predict, trace_align, vector_enrich) without grammar breakage.

8. Proven at hyperscale
KQL, ES|QL, UQL, PPL, Splunk SPL, and BigQuery-style stage flows all rely on pipeline semantics for scale.

---

🧱 Proposed Pipeline Grammar

FETCH <source>
| FILTER <predicate>
| TRANSFORM <fn>
| GROUP_BY <keys>, <agg>
| JOIN <other> ON <condition>
| TIME_SHIFT <duration>
| OUTPUT <format>

Examples

Metrics

metrics http_requests_total
| rate 5m
| filter status >= 500
| group_by [service], sum

Logs

logs syslog
| filter message contains "timeout"
| parse message with /user=(?<user>\w+)/
| stats count() by user

Cross-domain correlation

traces service="checkout"
| join logs on trace_id
| filter log.level == "error"
| group_by [span_name], count

---

🧭 Backward Compatibility

• Current UQL syntax remains valid.
• Pipeline is an additive enhancement.
• Compiler translates pipeline stages into existing engine operations.

---

🛠 Implementation Notes

• Needs grammar extension + stage compiler.
• Must handle intermediate dataset limits safely.
• Stage-level error diagnostics required for UX.
• Allows later introduction of AI-powered operators.

---

✅ Request

Please consider implementing a pipeline-style DSL for UQL. This aligns Mirador-core with the best-in-class observability ecosystems (UQL, ES|QL, KQL, PPL, BigQuery) and unlocks high-scale, intuitive, modular query authoring.

Happy to assist with grammar design or reference operator mappings.

[dostiharise]

Reference Implementations:

• https://pql.dev (go)
• https://github.com/PRQL/prql (rust)