[WIP] - DO NOT REVIEW - Deserializer hive 28197 #6067
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WIP for deserializing JSON plans to RELNODE Add table from passed object Clean up RelJson, make varchars with Charset Fix changes in plan related to GROUPING__ID clean up Fixed error in HiveSortExchange because of a check in org.apache.calcite.rel.core.Exchange assert distribution != RelDistributions.ANY Fixed NPE caused by null table for Values When timestamp literal was an Integer, we got a ClassCastException because in RexBuilder.clean(), there is an explicit cast TimestampString.fromMillisSinceEpoch((Long) o). With this change, for all Number types, we will first get the longvalue() and use that to create the timestamp literal. Although even with this change, I saw a diff in cast4.q.out: -1945-12-31 23:59:59.1234 +1945-12-31 23:59:59.123 Will look into this later Adding support for complex types - Structs, Arrays, and Maps If Calcite is upgraded to 1.29, we might not need these changes Added dummy table to tabNameToTabObject in SemanticAnalyzer so the parser can get this later Disable serialization for JDBC plans When table:alias is not present in the JSON plan, try setting tableAlias to table name Always return SqlStdOperatorTable.MAP_VALUE_CONSTRUCTOR for a MAP operator, as MAP_QUERY_CONSTRUCTOR doesn't override inferType method Add element types of MAPs and ARRAYs in the JSON, so that these types are available during deserialization. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapCliDriver -Dtest.output.overwrite=true -Dqfile="complex_datatype.q" Setting SqlExplainLevel.ALL_ATTRIBUTES for insideView in HiveTableScan, so that it shows up for views. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapCliDriver -Dtest.output.overwrite=true -Dqfile="create_view.q" Serialize with String format for TIMESTAMPS instead of LONG to prevent millis and nanos from getting trimmed. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestCliDriver -Dtest.output.overwrite=true -Dqfile="timestamptz_2.q" mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="cast4.q" Cleaned up a bit. Also, added a JSON property for materializedTable. When it is set to true, we can look for the table in the Context. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapCliDriver -Dtest.output.overwrite=true -Dqfile="cte_4.q" mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="cte_mat_11.q,cte_mat_8.q,cte_mat_9.q,cte_mat_10.q,cte_3.q" Always return SqlStdOperatorTable.ARRAY_VALUE_CONSTRUCTOR for a ARRAY operator, as ARRAY_QUERY_CONSTRUCTOR doesn't override inferType method Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="parquet_array_map_emptynullvals.q" Skip serialization and deserialization if plan is an instance of HiveJdbcConverter tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="jdbc_table_with_schema_postgres.q" Added support for different RelDistributions. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="cbo_rp_auto_join17.q" Support serialization and deserialization of HiveMultiJoin. TODO: Serialize joinInputs and joinTypes separately for JSON output tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="cbo_rp_join0.q" Fix bug where virtual columns were getting added with regular columns. Also cleaned up a little bit. tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="cbo_rp_join0.q" With this patch, AggregateCalls get serialized correctly when it has a complex type like Map. This helped resolve compilation errors from cbo_rp_udf_percentile.q, however, we get incorrect results. I also noticed that all Handlers in RelMetaDataQuery object has a NPE. Not sure if that is a new problem or not. Added type for RexCalls to the JSON plan. Without this the types were getting derived and sometimes they would be a little different. tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestTezTPCDS30TBPerfCliDriver -Dtest.output.overwrite=true -Dqfile="query1.q,query12.q,query1b.q,query20.q,query23.q,query24.q,query32.q,query4.q,query58.q,query6.q,query61.q,query65.q,query66.q,query67.q,query81.q,query92.q,query93.q,query98.q" Get ENUMs from a registry while making literal. HiveRelEnumTypes.java is a copy of Calcite's RelEnumTypes as we need to use the toEnum method. Ideally toEnum in RelEnumTypes should be public. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestCliDriver -Dtest.output.overwrite=true -Dqfile="localtimezone.q" Added a new method to wrap the output of serialized JSON object Add benign plan changes. Note that these changes are not actually going to be in the q.out file - these are going to be in the cache only. Currently they show up in the q.out files only because we run all tests through the serialization-deserialization step. However, in the final patch, we'll have one or several configs for this. Minor clean up Timestamps can be case to Integer or Longs, so we need to handle that while deserializing. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="vector_ptf_part_simple.q,vector_ptf_bounded_start.q" Support deserialization of BigIntegers to DECIMALs. This change is needed because right now Calcite doesn't check for BigIntegers while making literals, and trims a BigInteger to LONG: https://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/rex/RexBuilder.java#L1843 This should ideally be fixed in Calcite. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="bucket_decimal_col_select.q" Resolving JSON parsing failures due to not escaping characters correctly in Calcite 1.25. This patch is probably not needed with Calcite 1.31 Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="quote2.q,replication_metrics_ingest.q,resourceplan.q" quote2.q runs successfully, while the other two still fails - but with a different error. Resolving failures when first node is not HiveJdbcConverter. Now we check all nodes in the plan, and if convention of any node contains "jdbc", we skip serialization and deserialization step. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="external_jdbc_auth.q" More benign plan changes Introduce config for query plan cache Fix NPE in HiveRelMdSelectivity.computeInnerJoinSelectivity(HiveRelMdSelectivity.java:142) by using correct cluster Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="vector_outer_join6.q,vector_outer_join5.q,vector_outer_join4.q" NPE in FilterSelectivityEstimator.getMaxNulls(FilterSelectivityEstimator.java:418) Tested with: mvn test -pl itests/qtest-iceberg -Piceberg -Pitests -Dtest=TestIcebergCliDriver -Dtest.output.overwrite=true -Dqfile="query_iceberg_metadata_with_joins.q" NPE due to bug in RelMdUtil#estimateFilteredRows In Calcite 1.25, the above function calls mq.getSelectivity which is nullable, and tries to multiply with another double resulting in NPE. This behaviour is better in later versions of Calcite where mq.getSelectivity returns null instead of throwing an NPE. Currently, I am just not getting row counts because they are not needed to deserialize. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="vector_empty_where.q" AssertionError in Exchange because of an equality check using == for distribution in traits Created a static final variable for HiveRelDistribution, which is used everytime we see a RelDistribution.ANY Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="parquet_ppd_boolean.q,union6.q" Revert "NPE in FilterSelectivityEstimator.getMaxNulls(FilterSelectivityEstimator.java:418)" Not inserting null values in the list causes IndexOutOfBoundsException. It it better to handle this downstream. This reverts commit ac455a8. NPE in FilterSelectivityEstimator.getMaxNulls(FilterSelectivityEstimator.java:418) Resolved by ignoring null ColStatistics in getMaxNulls Tested with: mvn test -pl itests/qtest-iceberg -Piceberg -Pitests -Dtest=TestIcebergCliDriver -Dtest.output.overwrite=true -Dqfile="query_iceberg_metadata_with_joins.q" Remove unnecessary insideView and materializedTable info from JSON plan. Also revert these from benign plan changes. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="ctas.q,excluded_rule_explain.q,input4.q,materialized_view_create.q,plan_json.q" Another benign change For RANK-style funtions, ROWS/RANGE are skipped, as mentioned in https://github.com/apache/calcite/blob/calcite-1.25.0/core/src/main/java/org/apache/calcite/rex/RexWindow.java#L141 Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="join46.q" Fix UDFArgumentTypeException due to missing collation in AggregateCall Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="sketches_materialized_view_percentile_disc.q,sketches_rewrite_percentile_disc.q" Return HiveSqlSumEmptyIsZeroAggFunction instead of Calcite's SqlSumEmptyIsZeroAggFunction as it sets operandTypeInference to null. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestCliDriver -Dtest.output.overwrite=true -Dqfile="cbo_rp_auto_join1.q" Fix plan changes due to trimming of FLOAT values. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="float_failure.q" Fix error due to getting different table with the same alias. As an ad hoc, now we are checking if the table we get has the <db>.<tableName> that we expect, but the code can be optimized - I have added a TODO for now. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="schq_ingest.q,schq_analyze.q,schq_materialized.q" Fix error due to huge query In TestHiveIcebergVectorization, huge insert queries are generated and the JSON parser fails while reading them. Tested with: mvn test -Dtest=TestHiveIcebergVectorization -pl iceberg/iceberg-handler -Piceberg Support serialization and deserialization for dynamic paramters. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestNegativeLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="execute_mismatched_parameters.q" mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="prepare_plan_partition_pruning.q,prepare_plan.q" If operands list is not empty, pass HiveToUnixTimestampSqlOperator instead of HiveUnixTimestampSqlOperator Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="udf_to_unix_timestamp.q" Fix errors due to missing tableScanTrait while constructing HiveTableScans. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="fetch_deleted_rows_vector.q,fetch_deleted_rows.q" Fix errors due to missing value of ignoreNulls in RexOver Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="windowing_navfn.q" Refactor and clean up JSON parser Fix authorization errors due to missing values of insideView in digest Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestNegativeLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="authorization_view_6.q" Fix plan changes where virtual columns were missing in ALTER MATERIALIZED REBUILD. This was caused because of 'E' in scientific format representation of doubles. The digests for predicates didn't match up and that resulted in different plans in the HepPlanner. More plan changes should be expected after this commit, but they will be benign and will be addressed in a later commit. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="materialized_view_partition_cluster.q,materialized_view_distribute_sort.q,materialized_view_cluster.q" Reverting changes to a file, which was part of benign changes in an earlier commit. This was fixed by adding correct conditions to 'itemIf' of HiveTableScan's explainTerms All benign changes where 'E' is removed from doubles. This will revert some of the earlier changes and add a couple of new files Add snapshotRef to JSON when it's available. Tested with: mvn test -pl itests/qtest-iceberg -Piceberg -Pitests -Dtest=TestIcebergCliDriver -Dtest.output.overwrite=true -Dqfile="write_iceberg_branch.q" Add partitionListKey to fix incorrect joincost errors Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestTezTPCDS30TBPerfCliDriver -Dtest.output.overwrite=true -Dqfile="cbo_ext_query1.q" Revert "Add partitionListKey to fix incorrect joincost errors" Reverting because this doesn't work when a table has different aliases and there are different conditions on both, as in that case the key to the map, table name, is the same. This reverts commit 8406d3f. Serialize pruned partition list to JSON, so that it can be used in HiveTableScan. Tested with: mvn test -pl itests/qtest -Pitests -Dtest=TestTezTPCDS30TBPerfCliDriver -Dtest.output.overwrite=true -Dqfile="cbo_ext_query1.q" Add correct joinAlgorithm to HiveJoin Tested with: mvn test -pl itests/qtest-iceberg -Piceberg -Pitests -Dtest=TestIcebergCliDriver -Dtest.output.overwrite=true -Dqfile="mv_iceberg_orc4.q,mv_iceberg_orc7.q" mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="materialized_view_create_rewrite_7.q,materialized_view_create_rewrite_nulls.q" Do serialization and deserialization only for tests where it is enabled. Disabled by default For TestTezTPCDS30TBPerfCliDriver create new test files for plan changes For TestIcebergCliDriver create new test files for plan changes For TestMiniLlapLocalCliDriver create new test files for plan changes Revert "For TestTezTPCDS30TBPerfCliDriver create new test files for plan changes" This reverts commit 36eb530. Revert "For TestIcebergCliDriver create new test files for plan changes" This reverts commit ed39df7. Revert "For TestMiniLlapLocalCliDriver create new test files for plan changes" This reverts commit eeec453. Enable serialization-deserialization test for a certain qtests Make RelPlanParser testable Clean up TestRelPlanParser Add more unit tests to TestRelPlanParser Revert "Enable serialization-deserialization test for a certain qtests" This reverts commit cef3527. Update q.out results after removing set hive.test.cbo.plan.serialization.deserialization.enabled=true Remove test code from CalcitePlanner Address nits from PR review Move stats collection during serialization to a new class Wrap AssertionErrors with IOException during deserialization Update expected output for testJoining3TablesAndAllChanged Rowcount moved in the JSON output after stats collection was moved to a different class
When a table is not present in HMS, we can still create a Table object. This helps for materialized tables/ctes as they are not present in HMS. Also mocking is no longer required in TestRelPlanParser because of this.
…s correctly in Calcite 1.25." This reverts commit d9dd986.
…o the parser can get this later" This reverts commit 7088de2.
soumyakanti3578
force-pushed
the
deserializer-HIVE-28197
branch
from
a63b1f9 to
39b0ec0
Compare
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?