Spark 4 support

[m-aciek]

Is your feature request related to a problem? Please describe.
We are considering using great expectations in our project. Though we are about to be switching to Spark 4 soon. What's the project stance on Spark 4 support? Is it planned? Thank you.

Describe the solution you'd like
Great expectations supporting Spark 4.

Describe alternatives you've considered
Using alternatives: cuallee or spark-expectations.

[adeola-ak]

hi there, thank you for bringing this to our attention.

out of curiosity, about when will you be switching to Spark 4? We will discuss this as a team and get back to you with an update.

[m-aciek]

out of curiosity, about when will you be switching to Spark 4? We will discuss this as a team and get back to you with an update.

at latest in June this year, thank you

[keenanwells-tatari]

We are currently using great expectations in many places and are looking to upgrade to spark 4. Wondering if this is possible for our case or if we are blocked due to GX.

[postatum]

Hello @adeola-ak . Sorry for tagging you directly, but do you have an ETA of Spark 4 support in GE?

[joshua-stauffer]

hey @m-aciek, @keenanwells-tatari, @postatum, thanks for your patience on this. I'm prioritizing this work next, and expect Spark 4 support to ship with the next release.