Skip to content

Add support for VARBYTE type#171

Open
aajisaka wants to merge 1 commit into
spark-redshift-community:masterfrom
aajisaka:varbyte
Open

Add support for VARBYTE type#171
aajisaka wants to merge 1 commit into
spark-redshift-community:masterfrom
aajisaka:varbyte

Conversation

@aajisaka
Copy link
Copy Markdown

Problem

Fails to read from Redshift table if the table has VARBYTE type column

An error occurred while calling o156.load. : java.sql.SQLException: Unsupported type -4
    at io.github.spark_redshift_community.spark.redshift.data.JDBCWrapper.getCatalystType(JDBCWrapper.scala:445)
    at io.github.spark_redshift_community.spark.redshift.data.JDBCWrapper.resolveTable(JDBCWrapper.scala:353)
    at io.github.spark_redshift_community.spark.redshift.RedshiftRelation.$anonfun$schema$1(RedshiftRelation.scala:74)
    at scala.Option.getOrElse(Option.scala:189)
    at io.github.spark_redshift_community.spark.redshift.RedshiftRelation.schema$lzycompute(RedshiftRelation.scala:68)
    at io.github.spark_redshift_community.spark.redshift.RedshiftRelation.schema(RedshiftRelation.scala:67)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:460)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:345)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:278)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:210)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)

Testing

Built from source, add the jar in Glue 5.0, and ran a Spark job to write binary data to Redshift table and read them from the same table.

I didn't update the unit tests, but I'm willing to work on that. Please let me know how I can efficiently implement unit tests for the new type.

Notes

@aajisaka
Copy link
Copy Markdown
Author

@bsharifi Would you review?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant