Skip to content
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
03b7939
Adding section for UDF serialization
Niharikadutta Apr 20, 2020
4ef693d
removing guides from master
Niharikadutta Apr 20, 2020
81145ca
Merge latest from master
Niharikadutta May 6, 2020
49a437d
Prep changes for 0.11.0 version
Niharikadutta May 6, 2020
014e359
Merge branch 'master' into nidutta/0.11.0-release
imback82 May 6, 2020
bde7231
Adding table for breaking changes
Niharikadutta May 6, 2020
ac0db5e
Merge branch 'nidutta/0.11.0-release' of github.com:Niharikadutta/spa…
Niharikadutta May 6, 2020
598ea15
table formatting error
Niharikadutta May 6, 2020
4ddfb2c
Added Compatibility section
Niharikadutta May 6, 2020
d748beb
Added description for compatibility table
Niharikadutta May 6, 2020
dfadf09
Added link to PRs of incompatible features
Niharikadutta May 6, 2020
e9eb4c8
testing remove the link name
Niharikadutta May 6, 2020
d27baae
test 2
Niharikadutta May 6, 2020
fb625b3
test3
Niharikadutta May 6, 2020
0dfe622
using href tags
Niharikadutta May 6, 2020
62f91c4
using href tags to remove link url from table
Niharikadutta May 6, 2020
63f2cac
PR review changes
Niharikadutta May 6, 2020
e06399a
PR comment changes
Niharikadutta May 6, 2020
b843cd8
PR comment changes
Niharikadutta May 6, 2020
a8dab1f
add forward compatibility table
elvaliuliuliu May 6, 2020
da5585a
resolve comments
elvaliuliuliu May 6, 2020
7b72e24
Merge pull request #1 from elvaliuliuliu/elva/addFC
Niharikadutta May 6, 2020
e7cd165
Merge branch 'master' into nidutta/0.11.0-release
imback82 May 7, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
<tbody align="center">
<tr>
<td >2.3.*</td>
<td rowspan=6><a href="https://github.com/dotnet/spark/releases/tag/v0.10.0">v0.10.0</a></td>
<td rowspan=6><a href="https://github.com/dotnet/spark/releases/tag/v0.11.0">v0.11.0</a></td>
</tr>
<tr>
<td>2.4.0</td>
Expand Down
2 changes: 1 addition & 1 deletion benchmark/scala/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>com.microsoft.spark</groupId>
<artifactId>microsoft-spark-benchmark</artifactId>
<version>0.10.0</version>
<version>0.11.0</version>
<inceptionYear>2019</inceptionYear>
<properties>
<encoding>UTF-8</encoding>
Expand Down
83 changes: 83 additions & 0 deletions docs/release-notes/0.11/release-0.11.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# .NET for Apache Spark 0.11 Release Notes

### New Features and Improvements

* Ability to pass and return corefxlab DataFrames to UDF APIs ([#277](https://github.com/dotnet/spark/pull/277))
* Supporting ML TF-IDF (Term frequency-inverse document frequency) feature vectorization method ([#394](https://github.com/dotnet/spark/pull/394))
* Support for TimestampType in `DataFrame.Collect()`, `CreateDataFrame` and UDFs ([#428](https://github.com/dotnet/spark/pull/428))
* Support for Broadcast Variables ([#414](https://github.com/dotnet/spark/pull/414))
* Implement ML feature Word2Vec ([#491](https://github.com/dotnet/spark/pull/491))
* Streamline logging when there is a failure ([#439](https://github.com/dotnet/spark/pull/439))


### Breaking Changes

* SparkSession.Catalog call changed from a method to a property ([#508](https://github.com/dotnet/spark/pull/508))

### Compatibility

The following table describes the oldest version of the worker that this current version is compatible with, excluding some incompatible features as shown below.

<table>
<thead>
<tr>
<th>Oldest compatible .NET for Apache Spark worker</th>
<th>Incompatible features</th>
</tr>
</thead>
<tbody align="center">
<tr>
<td rowspan=4>v0.9.0</td>
<td>DataFrame with Grouped Map UDF <a href="https://github.com/dotnet/spark/pull/277">(#277)</a></td>
</tr>
<tr>
<td>DataFrame with Vector UDF <a href="https://github.com/dotnet/spark/pull/277">(#277)</a></td>
</tr>
<tr>
<td>Support for Broadcast Variables <a href="https://github.com/dotnet/spark/pull/414">(#414)</a></td>
</tr>
<tr>
<td>Support for TimestampType <a href="https://github.com/dotnet/spark/pull/428">(#428)</a></td>
</tr>
</tbody>
</table>


### Supported Spark Versions

The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:

<table>
<thead>
<tr>
<th>Spark Version</th>
<th>microsoft-spark JAR</th>
</tr>
</thead>
<tbody align="center">
<tr>
<td>2.3.*</td>
<td>microsoft-spark-2.3.x-0.11.0.jar</td>
</tr>
<tr>
<td>2.4.0</td>
<td rowspan=5>microsoft-spark-2.4.x-0.11.0.jar</td>
</tr>
<tr>
<td>2.4.1</td>
</tr>
<tr>
<td>2.4.3</td>
</tr>
<tr>
<td>2.4.4</td>
</tr>
<tr>
<td>2.4.5</td>
</tr>
<tr>
<td>2.4.2</td>
<td><a href="https://github.com/dotnet/spark/issues/60">Not supported</a></td>
</tr>
</tbody>
</table>
2 changes: 1 addition & 1 deletion eng/Versions.props
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<VersionPrefix>0.10.0</VersionPrefix>
<VersionPrefix>0.11.0</VersionPrefix>
<PreReleaseVersionLabel>prerelease</PreReleaseVersionLabel>
<RestoreSources>
$(RestoreSources);
Expand Down
2 changes: 1 addition & 1 deletion src/scala/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<version>${microsoft-spark.version}</version>
<properties>
<encoding>UTF-8</encoding>
<microsoft-spark.version>0.10.0</microsoft-spark.version>
<microsoft-spark.version>0.11.0</microsoft-spark.version>
</properties>

<modules>
Expand Down