[WIP] Expose spatial partitioning from SpatialRDD #1751
+60
−5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
paleolimbot
commented
Jan 21, 2025
There was a problem hiding this comment.
This also would benefit from a SpatialPartitioner that removes duplicates (perhaps by wrapping a SpatialPartitioner, consuming the result of placeObject and deterministically choosing one of the results), since most of the time having duplicates when partitioning is not really desired.
Comment on lines
+330
to
+348
| public JavaPairRDD<Integer, T> spatialPartitioningWithIds(GridType gridType, int numPartitions) | ||
| throws Exception { | ||
| calc_partitioner(gridType, numPartitions); | ||
| return spatialPartitioningWithIds(partitioner); | ||
| } | ||
|
|
||
| public JavaPairRDD<Integer, T> spatialPartitioningWithIds(final SpatialPartitioner partitioner) { | ||
| this.partitioner = partitioner; | ||
| return this.rawSpatialRDD | ||
| .flatMapToPair( | ||
| new PairFlatMapFunction<T, Integer, T>() { | ||
| @Override | ||
| public Iterator<Tuple2<Integer, T>> call(T spatialObject) throws Exception { | ||
| return partitioner.placeObject(spatialObject); | ||
| } | ||
| }) | ||
| .partitionBy(partitioner); | ||
| } | ||
|
|
There was a problem hiding this comment.
This is probably not needed (the changes in the Adapter to preserve the partitioning of spatialPartitionedRDD into the output data frame should eliminate the need to keep any identifier alongside the partition).
| val stringRow = extractUserData(geom) | ||
| castRowToSchema(stringRow = stringRow, schema = schema) | ||
| }) | ||
| val rdd = spatialRDD.rawSpatialRDD.rdd.mapPartitions( |
There was a problem hiding this comment.
(But moved to a different overload since most of the time this will introduce duplicates)
Suggested change
| val rdd = spatialRDD.rawSpatialRDD.rdd.mapPartitions( | |
| val rdd = spatialRDD.spatialPartitionedRDD.rdd.mapPartitions( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a JIRA ticket?
[SEDONA-695] my subject.Closes #1268.
What changes were proposed in this PR?
This PR exposes spatial partitioning information from the SpatialRDD API. Sedona is exceptionally good at this and the spatial community would love to have access to this information!
There are two pieces of information that would be helpful:
There are a few ideas in this PR...the boundaries seem straightforward but I'm a little new to the RDD API to know what the options are for returning these things.
How was this patch tested?
Working on it!
Did this PR include necessary documentation updates?
vX.Y.Zformat.