Add spark sql integration test for Hudi #3194
Conversation
| // TODO: extract a polaris-rest module as a thin layer for | ||
| // client to depends on. | ||
| implementation(project(":polaris-core")) { isTransitive = false } | ||
| testImplementation("org.apache.hudi:hudi-spark3.5-bundle_${scalaVersion}:1.1.0") |
There was a problem hiding this comment.
Nit: we put versions in the file pluginlibs.versions.toml, refer it as line 35 does.
github-project-automation
bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
| // TODO: extract a polaris-rest module as a thin layer for | ||
| // client to depends on. | ||
| implementation(project(":polaris-core")) { isTransitive = false } | ||
| testImplementation("org.apache.hudi:hudi-spark3.5-bundle_${scalaVersion}:1.1.0") |
There was a problem hiding this comment.
for the actual spark project, we don't really intend to introduce any table format specific dependency, even for testing. i didn't see any change in the actual spark project, is there a reason that we need this?
| exclude("org.slf4j", "jul-to-slf4j") | ||
| } | ||
|
|
||
| // Add spark-hive for Hudi integration - provides HiveExternalCatalog that Hudi needs |
There was a problem hiding this comment.
@rahil-c could you also update the readme to include the support for hudi?
It would be great if we could also have a notebook in the get-started to help people to onboard for hudi, we could do that in follow up, we should also extend the regress test to include actual end to end test for hudi to avoid any potential break of the feature
4 participants
Description
SparkDeltaITChecklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)