feat: convert anomaly demo to spark-connect #209
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* make token a stack parameter * use 8080 * start in /notebook
razvan
moved this from Development: In Progress
to Development: Waiting for Review
in Stackable Engineering
NickLarsenNZ
reviewed
May 2, 2025
| name: notebook | ||
| initContainers: | ||
| - name: download-notebook | ||
| image: oci.stackable.tech/sdp/spark-connect-client:3.5.5-stackable0.0.0-dev |
There was a problem hiding this comment.
This might change depending on the outcome mentioned in the dependent PR:
There was a problem hiding this comment.
yes, let's wait with merging this one
NickLarsenNZ
moved this from Development: Waiting for Review
to Development: In Review
in Stackable Engineering
NickLarsenNZ
reviewed
May 2, 2025
docs/modules/demos/pages/jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data.adoc
Outdated
Show resolved
Hide resolved
docs/modules/demos/pages/jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data.adoc
Outdated
Show resolved
Hide resolved
| NOTE: Using a custom image requires access to a repository where the image can be made available. | ||
| The Python notebook uses libraries such as `pandas` and `scikit-learn` to analyze the data. | ||
| In addition, since the model training is delegated to a Spark Connect server, some of these dependencies, most notably `scikit-learn`, must also be made available on the Spark Connect pods. | ||
| For convenience, a custom image is used in this demo that bundles all the required libraries for both the notebook and the Spark Connect server. |
There was a problem hiding this comment.
optional: We could link to the Dockerfile (so that others can take the next steps for their use case).
docs/modules/demos/pages/jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data.adoc
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Were the comments supposed to be in there?
#SCL = spark.sparkContext.broadcast(scaler)
#CLF = spark.sparkContext.broadcast(clf) # No broadcast variables when using Spark Connect
# x_test = SCL.value.transform(x_test)
# prediction = CLF.value.predict(x_test)[0]There was a problem hiding this comment.
Yeah, I left them as an explanation / reminder that connect only supports a subset of the spark api
| image: | ||
| # Using an image that includes scikit-learn (among other things) | ||
| # because this package needs to be available on the executors. | ||
| custom: oci.stackable.tech/sdp/spark-connect-client:3.5.5-stackable0.0.0-dev |
There was a problem hiding this comment.
This might change depending on the outcome mentioned in the dependent PR:
There was a problem hiding this comment.
updated according to stackabletech/docker-images#1072
…tion-taxi-data.adoc Co-authored-by: Nick <[email protected]>
…tion-taxi-data.adoc Co-authored-by: Nick <[email protected]>
…tion-taxi-data.adoc Co-authored-by: Nick <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of stackabletech/spark-k8s-operator#284
Depends on stackabletech/docker-images#1071