Materializing table transformations helps improving runtime performance if the transformations are to be execute multiple times.
Some transformations on a table also change table schema based on content in the table.
When to materialize
- When user observe the table: We already compute all the transformations that builds up the table. We should materialize it, unless the user only observe a part of the table. Ex:
table.print({limit: 2})
- When user say so: We should allow users to control when a table can materialized on demand.
- When transforming with a verb that requires the knowledge of the data countent: For example,
pivot needs to know all the possible values of the pivoting key column before calculating the schema of the output table.
Materializing table transformations helps improving runtime performance if the transformations are to be execute multiple times.
Some transformations on a table also change table schema based on content in the table.
When to materialize
table.print({limit: 2})pivotneeds to know all the possible values of the pivoting key column before calculating the schema of the output table.