feat(logql): Add LogQL Benchmark Suite

[cyriltovena]

This PR introduces a comprehensive benchmark suite for LogQL, Loki's query language. The suite is designed to help measure and improve the performance of LogQL queries by providing tools to generate realistic log data and benchmark various query patterns against different storage implementations.

What this PR does

• Adds a complete benchmark suite for LogQL in pkg/logql/bench/
• Provides tools to generate synthetic log data with configurable cardinality and patterns
• Implements multiple storage backends for comparison (chunk-based and dataobj columnar)
• Includes a variety of benchmark queries covering log filtering, pattern extraction, and metrics
• Adds a comprehensive README with usage instructions and architecture details
• Includes Makefile targets for easy data generation and benchmark execution

Why this is valuable

This benchmark suite will help:

1. Measure performance impact of changes: Developers can use this to measure the performance impact of changes to the LogQL engine
2. Compare storage implementations: The suite allows comparing different storage implementations to identify optimization opportunities
3. Identify bottlenecks: The profiling capabilities help identify performance bottlenecks in the query execution
4. Regression testing: The suite can be used to detect performance regressions before they reach production

Key features

• Reproducible datasets: Uses a fixed random seed to ensure benchmark results are comparable across different runs and environments
• Realistic log patterns: Generates logs that mimic real-world applications like nginx, postgres, java, etc.
• Multiple storage formats: Supports different storage implementations for performance comparison
• Comprehensive query types: Tests various LogQL features including filtering, pattern extraction, and metrics
• Extensible framework: Easy to add new application types, storage implementations, and query patterns

How to use it

1. Generate test data: make -C pkg/logql/bench generate
2. Run benchmarks through the CLI: make -C pkg/logql/bench run
3. For more options, see the README in pkg/logql/bench/

Future improvements

• Integrate with CI for automated performance regression testing
• Add more complex query patterns as LogQL evolves with new data example.

CLI

This PR comes with a a CLI tools to select which benchmark to run, compare result, re-run and see performance profiles.

To use it run make run