(feat) Add metrics persistence with dual storage strategy for Iceberg table operations (fixes #3337)

[obelix74]

Checklist

• 🛡️ Don't disclose security issues! (contact [email protected])
• 🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
• 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
• 💡 Added comments for complex logic
• 🧾 Updated CHANGELOG.md (if needed)
• 📚 Updated documentation in site/content/in-dev/unreleased (if needed)

Overview

Implements comprehensive metrics persistence for Iceberg table operations with support for both unified audit trail and analytics-optimized storage strategies. Enables tracking and analysis of scan and commit metrics from compute engines (Spark, Trino, Flink) with full OpenTelemetry trace context integration.

There is a document that explains the overall plan for supporting end to end auditing here: https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0

Fixes #3337

Key Features

Metrics Persistence

• Dual Storage Strategy: Supports persisting metrics to both events table (audit trail) and dedicated tables (analytics)
• Flexible Reporting: Four reporter implementations (default, events, persistence, composite)
• Schema v4: New dedicated tables for scan and commit metrics with proper indexing
• OpenTelemetry Integration: Full trace context propagation for correlation across operations
• Automated Cleanup: Configurable retention policies with scheduled cleanup service

Implementation Details

New Components

• PersistingMetricsReporter: Persists to dedicated scan/commit metrics tables
• EventsMetricsReporter: Persists to events table as JSON
• CompositeMetricsReporter: Multi-destination reporting
• MetricsReportCleanupService: Automated retention management
• MetricsReportingConfiguration: Flexible configuration interface

Database Schema

• scan_metrics_report: 30+ columns for scan metrics with indexes on timestamp, table, trace_id
• commit_metrics_report: 30+ columns for commit metrics with indexes on timestamp, table, operation
• Schema version upgraded from v3 to v4

JDBC Persistence Layer

• Write methods: writeScanMetricsReport(), writeCommitMetricsReport()
• Query methods: Filter by table, time range, trace ID
• Cleanup methods: deleteAllMetricsReportsOlderThan()

Event System Integration

• AfterReportMetricsEvent: Emitted after metrics reports are processed
• PolarisPersistenceEventListener: Extracts and persists metrics data
• InMemoryBufferEventListener: Batch writes for performance

Configuration

   polaris:
     iceberg-metrics:
       reporting:
         type: composite  # or: default, events, persistence
         targets:
                 • events
                 • persistence
         retention:
           enabled: true
           retention-period: P30D
           cleanup-interval: PT6H

Testing

• 933 new test lines across unit and integration tests
• All unit tests passing
• Comprehensive coverage for reporters, persistence, and model classes

Breaking Changes

None. All changes are additive with backward compatibility maintained.

Documentation

• Added telemetry documentation in site/content/in-dev/unreleased/telemetry.md
• Configuration examples for all reporter types
• Query examples for common analytics use cases

[obelix74]

Dual Storage Strategy: We use dedicated scan_metrics_report and commit_metrics_report tables rather than storing metrics as JSON blobs in the existing events table. This enables efficient SQL queries for analytics (e.g., "find slow table scans") and proper indexing on metrics fields, at the cost of a more rigid schema. The metadata JSONB column provides extensibility for future metric fields without requiring schema migrations.

[obelix74]

Index Strategy: Indexes are designed around common query patterns: time-range queries (timestamp_ms DESC), table-specific queries (composite on catalog/namespace/table), and trace correlation (otel_trace_id with partial index to skip NULLs). The realm_id index supports multi-tenant queries without requiring it in every composite index.

[obelix74]

CDI-based Reporter Selection: The @Identifier annotation pattern allows new reporter implementations to be added without modifying this factory. The composite reporter is assembled dynamically from the targets config, with safeguards against infinite recursion (ignoring "composite" as a target) and graceful fallback to default if no valid targets are resolved.

[obelix74]

Backend Compatibility Guard: This explicit check for JdbcBasePersistenceImpl ensures the persistence reporter degrades gracefully when running with a non-JDBC backend (e.g., in-memory for testing). The warning log makes misconfiguration obvious in production logs.

[obelix74]

Iceberg's CommitReport.operation() can be null in some edge cases (e.g., failed commits before operation is determined), but our DB schema defines operation TEXT NOT NULL. We default to "UNKNOWN" to satisfy the constraint while making these cases visible in analytics queries.

[obelix74]

Capturing the OpenTelemetry trace/span IDs enables correlating metrics reports with the HTTP request that triggered them. This is essential for debugging slow scans or failed commits by joining metrics data with distributed traces. Span.current() returns the active span from the thread-local context propagated through the Quarkus request handling.

[obelix74]

The cleanup interval is injected via config expression ${polaris.iceberg-metrics.reporting.retention.cleanup-interval:6h}. This runs in a Quarkus scheduler thread, not the request thread, so the realm context must be explicitly created (line 143) rather than injected.

[obelix74]

Retest this please.