Skip to content

Conversation

@szehon-ho
Copy link
Member

@szehon-ho szehon-ho commented Jul 4, 2025

What changes were proposed in this pull request?

Fix a warning that nulls out the schema.

In #51007 we decide to use the proper Catalog API alterDataSchema for column changes. However, that one pass in the table.schema (including partition columns) and not the expected table.dataSchema (ie, without partition columns). For reference, the add column command uses table.dataSchema : https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala#L250

Change to use the table.dataSchema

Why are the changes needed?

If we use table.schema, it adds the partition column twice. This triggers a Hive Metastore exception, which causes us to null out the schema.

18:20:02.660 WARN org.apache.spark.sql.hive.test.TestHiveExternalCatalog: Could not alter schema of table `default`.`t1` in a Hive compatible way. Updating Hive metastore in Spark SQL specific format.java.lang.reflect.InvocationTargetException	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)	at java.base/java.lang.reflect.Method.invoke(Method.java:580)	at org.apache.spark.sql.hive.client.Shim_v2_1.alterTable(HiveShim.scala:1115)	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$alterTableDataSchema$1(HiveClientImpl.scala:639)	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)	at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:237) 

Does this PR introduce any user-facing change?

No

How was this patch tested?

Add unit test

@szehon-ho
Copy link
Member Author

Ill fix this as part of #51373

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant