Enhanced Label Management (drop/keep with regex matchers)

[frisbeeman]

Overview

Enhance the drop and keep pipeline stages to support all matcher types, not just simple name equality.

Parent Issue

Part of #6 (Query Service Implementation), delegated from #8 (LogQL Query Compiler).

Context

The drop and keep pipeline stages currently only support simple label names with = matchers. The full LogQL specification supports all matcher types.

Current Support

{job="app"} | drop method        ✅ Works
{job="app"} | keep level, service  ✅ Works

Missing Support

{job="app"} | drop level="debug"        ❌ Not implemented
{job="app"} | drop level=~"debug|info"  ❌ Not implemented
{job="app"} | keep level!="error"       ❌ Not implemented
{job="app"} | keep level!~"debug.*"     ❌ Not implemented

Requirements

Matcher Types to Support

Matcher	Syntax	Description
=	drop level="debug"	Drop if label equals value
!=	keep level!="debug"	Keep if label not equals value
=~	drop level=~"debug|info"	Drop if label matches regex
!~	keep level!~"error.*"	Keep if label doesn't match regex

Implementation Approach

1. Update planner in src/query/logql/datafusion/planner.rs:

   • Enhance apply_drop() and apply_keep() methods
   • Handle matcher expressions, not just label names

2. For value-based matchers:

   • Filter rows based on label value match
   • Then apply map key filtering

3. Example translation:

// {job="app"} | drop level="debug"
// 1. Filter: WHERE attributes['level'] = 'debug'
// 2. Then drop the 'level' key from matching rows
// Or: conditionally drop based on value match

Acceptance Criteria

• drop label="value" removes label when value matches
• drop label=~"regex" removes label when value matches regex
• keep label!="value" keeps label when value doesn't match
• keep label!~"regex" keeps label when value doesn't match regex
• Multiple matchers in single drop/keep work correctly
• Combination with simple names works: | drop method, level="debug"
• Unit tests for each matcher type
• Integration tests with full LogQL queries

Technical References

• LogQL Drop Labels
• LogQL Keep Labels
• Current implementation: src/query/logql/datafusion/planner.rs
• UDFs: src/query/logql/datafusion/udf.rs (map_keep_keys, map_drop_keys)