You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-51384][SQL] Support java.time.LocalTime as the external type of TimeType
### What changes were proposed in this pull request?
In the PR, I propose to support `java.time.LocalTime` as the external type of new data type `TIME` introduced by apache#50103. After the changes, users can create Datasets with `TimeType` columns and collect them back as instances of `java.time.LocalTime`. For example:
```scala
scala> val df = Seq(LocalTime.of(12, 15)).toDF
val df: org.apache.spark.sql.DataFrame = [value: time(6)]
scala> df.printSchema
root
|-- value: time(6) (nullable = true)
scala> df.first.getAs[LocalTime](0)
val res8: java.time.LocalTime = 12:15
```
By default the external type is encoded to the `TIME` type column with precision = 6 (microseconds).
### Why are the changes needed?
1. To allow creation of TIME columns using public Scala/Java API otherwise new type is useless.
2. To be able to write tests when supporting new type in other parts of Spark SQL.
### Does this PR introduce _any_ user-facing change?
Yes, in some sense since the PR allow to create `TimeType` columns using Scala/Java APIs.
### How was this patch tested?
By running new tests:
```
$ build/sbt "test:testOnly *DateTimeUtilsSuite"
$ build/sbt "test:testOnly *CatalystTypeConvertersSuite"
$ build/sbt "test:testOnly *DatasetSuite"
```
and modified:
```
$ build/sbt "test:testOnly *DataTypeSuite"
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closesapache#50153 from MaxGekk/time-localtime.
Authored-by: Max Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
|**ArrayType**|java.util.List|DataTypes.createArrayType(*elementType*)<br/>**Note:** The value of *containsNull* will be true.<br/>DataTypes.createArrayType(*elementType*, *containsNull*).|
0 commit comments