-
Notifications
You must be signed in to change notification settings - Fork 329
Prep 0.11.0 release #507
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Prep 0.11.0 release #507
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
03b7939
Adding section for UDF serialization
Niharikadutta 4ef693d
removing guides from master
Niharikadutta 81145ca
Merge latest from master
Niharikadutta 49a437d
Prep changes for 0.11.0 version
Niharikadutta 014e359
Merge branch 'master' into nidutta/0.11.0-release
imback82 bde7231
Adding table for breaking changes
Niharikadutta ac0db5e
Merge branch 'nidutta/0.11.0-release' of github.com:Niharikadutta/spa…
Niharikadutta 598ea15
table formatting error
Niharikadutta 4ddfb2c
Added Compatibility section
Niharikadutta d748beb
Added description for compatibility table
Niharikadutta dfadf09
Added link to PRs of incompatible features
Niharikadutta e9eb4c8
testing remove the link name
Niharikadutta d27baae
test 2
Niharikadutta fb625b3
test3
Niharikadutta 0dfe622
using href tags
Niharikadutta 62f91c4
using href tags to remove link url from table
Niharikadutta 63f2cac
PR review changes
Niharikadutta e06399a
PR comment changes
Niharikadutta b843cd8
PR comment changes
Niharikadutta a8dab1f
add forward compatibility table
elvaliuliuliu da5585a
resolve comments
elvaliuliuliu 7b72e24
Merge pull request #1 from elvaliuliuliu/elva/addFC
Niharikadutta e7cd165
Merge branch 'master' into nidutta/0.11.0-release
imback82 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
# .NET for Apache Spark 0.11 Release Notes | ||
|
||
### New Features and Improvements | ||
|
||
* Ability to pass and return [corefxlab](https://github.com/dotnet/corefxlab) DataFrames to UDF APIs ([#277](https://github.com/dotnet/spark/pull/277)) | ||
* Support for ML TF-IDF (Term frequency-inverse document frequency) feature vectorization method ([#394](https://github.com/dotnet/spark/pull/394)) | ||
* Support for TimestampType in `DataFrame.Collect()`, `CreateDataFrame` and UDFs ([#428](https://github.com/dotnet/spark/pull/428)) | ||
* Support for Broadcast Variables ([#414](https://github.com/dotnet/spark/pull/414)) | ||
* Support for ML feature Word2Vec ([#491](https://github.com/dotnet/spark/pull/491)) | ||
* Streamline logging when there is a failure ([#439](https://github.com/dotnet/spark/pull/439)) | ||
|
||
|
||
### Breaking Changes | ||
|
||
* `SparkSession.Catalog` is changed from a method to a property ([#508](https://github.com/dotnet/spark/pull/508)) | ||
|
||
### Compatibility | ||
|
||
#### Backward compatibility | ||
|
||
The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker. | ||
|
||
<table> | ||
<thead> | ||
<tr> | ||
<th>Oldest compatible Microsoft.Spark.Worker version</th> | ||
<th>Incompatible features</th> | ||
</tr> | ||
</thead> | ||
<tbody align="center"> | ||
<tr> | ||
<td rowspan=4>v0.9.0</td> | ||
<td>DataFrame with Grouped Map UDF <a href="https://github.com/dotnet/spark/pull/277">(#277)</a></td> | ||
</tr> | ||
<tr> | ||
<td>DataFrame with Vector UDF <a href="https://github.com/dotnet/spark/pull/277">(#277)</a></td> | ||
</tr> | ||
<tr> | ||
<td>Support for Broadcast Variables <a href="https://github.com/dotnet/spark/pull/414">(#414)</a></td> | ||
</tr> | ||
<tr> | ||
<td>Support for TimestampType <a href="https://github.com/dotnet/spark/pull/428">(#428)</a></td> | ||
</tr> | ||
</tbody> | ||
</table> | ||
|
||
#### Forward compatibility | ||
|
||
The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with. | ||
|
||
<table> | ||
<thead> | ||
<tr> | ||
<th>Oldest compatible .NET for Apache Spark release version</th> | ||
</tr> | ||
</thead> | ||
<tbody align="center"> | ||
<tr> | ||
<td>v0.9.0</td> | ||
</tr> | ||
</tbody> | ||
</table> | ||
|
||
### Supported Spark Versions | ||
|
||
The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with: | ||
|
||
<table> | ||
<thead> | ||
<tr> | ||
<th>Spark Version</th> | ||
<th>microsoft-spark JAR</th> | ||
</tr> | ||
</thead> | ||
<tbody align="center"> | ||
<tr> | ||
<td>2.3.*</td> | ||
<td>microsoft-spark-2.3.x-0.11.0.jar</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.0</td> | ||
<td rowspan=5>microsoft-spark-2.4.x-0.11.0.jar</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.1</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.3</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.4</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.5</td> | ||
</tr> | ||
<tr> | ||
<td>2.4.2</td> | ||
<td><a href="https://github.com/dotnet/spark/issues/60">Not supported</a></td> | ||
</tr> | ||
</tbody> | ||
</table> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.