[Feature Request]: [IcebergIO] Configure data writers to track metrics

[ahmedabu98]

What would you like to happen?

We're configuring our Iceberg data writers without a metrics config! So we're losing out on writing important statistics for each data file (statistics that help with query planning).

This can be easily added over here with:

   .metricsConfig(metricsConfig)

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Suvrat1629]

@ahmedabu98 Is this issue available to work on?

[ahmedabu98]

Hey @Suvrat1629! Yeah give it a shot and tag me in the PR to review it. Let's make sure we add some unit testing that columns marked for statistic collection are respected.

Take a look at the following table options for enabling metrics on table columns:

• write.metadata.metrics.default
• write.metadata.metrics.column.

[Suvrat1629]

@ahmedabu98
Thank you i will give it a shot but since I am new to the codebase so before i start working I would like some guidance and like to verify my approach:

• first we add the metricsConfig(metricsConfig) to the RecordWriter.java file
• then update all the corresponding Write*.java files.

does this approach fit right? Do I need to look into anymore files?

[ahmedabu98]

I think it'll end up being a lot simpler than that. RecordWriter is the essential piece that actually writes data. The other Write* files just take care of moving data around. The essential piece that actually writes data to files is RecordWriter. We just need to create a metrics config (something like MetricsConfig metricsConfig = MetricsConfig.forTable(table);) and pass it into the data writers

[Suvrat1629]

@ahmedabu98 So I need to add MetricsConfig metricsConfig = MetricsConfig.forTable(table); and then add test cases for the same?

[Suvrat1629]

@ahmedabu98 Since now that I have made the necessary change as you told me, do i add a new Test case file for the tests or make changes to and existing one?

[ahmedabu98]

You can add a new test case to an existing class: RecordWriterManagerTest.java