Merge pull request #4300 from NvTimLiu/release-tmp

Merge source to main branch from branch-21.12 [skip ci]

Author: NvTimLiu

.github/workflows/auto-merge.yml

@@ -18,7 +18,7 @@ name: auto-merge HEAD to BASE
 on:
   pull_request_target:
     branches:
-    - branch-21.10
+    - branch-21.12
     types: [closed]
 
 jobs:
@@ -29,13 +29,13 @@ jobs:
     steps:
       - uses: actions/checkout@v2
         with:
-          ref: branch-21.10 # force to fetch from latest upstream instead of PR ref
+          ref: branch-21.12 # force to fetch from latest upstream instead of PR ref
 
       - name: auto-merge job
         uses: ./.github/workflows/auto-merge
         env:
           OWNER: NVIDIA
           REPO_NAME: spark-rapids
-          HEAD: branch-21.10
-          BASE: branch-21.12
+          HEAD: branch-21.12
+          BASE: branch-22.02
           AUTOMERGE_TOKEN: ${{ secrets.AUTOMERGE_TOKEN }} # use to merge PR

.github/workflows/auto-merge/automerge

@@ -90,6 +90,21 @@ def auto_merge(number, sha):
 ```
 {r.json()}
 ```
+
+Please use the following steps to fix the merge conflicts manually:
+```
+# Assume upstream is NVIDIA/spark-rapids remote
+git fetch upstream {HEAD} {BASE}
+git checkout -b fix-auto-merge-conflict-{number} upstream/{BASE}
+git merge upstream/{HEAD}
+# Fix any merge conflicts caused by this merge
+git commit -am "Merge {HEAD} into {BASE}"
+git push <personal fork> fix-auto-merge-conflict-{number}
+# Open a PR targets NVIDIA/spark-rapids {BASE}
+```
+**IMPORTANT:** Before merging this PR, be sure to change the merging strategy to `Create a merge commit` (repo admin only).
+
+Once this PR is merged, the auto-merge PR should automatically be closed since it contains the same commit hashes
 """)
         print(f'status code: {r.status_code}')
         print(r.json())

.github/workflows/blossom-ci.yml

@@ -63,6 +63,7 @@ jobs:
       zhanga5,\
       nvliyuan,\
       res-life,\
+      HaoYang670,\
       ', format('{0},', github.actor)) && github.event.comment.body == 'build'
     steps:
       - name: Check if comment is issued by authorized person

CHANGELOG.md

@@ -50,39 +50,56 @@ You can find all available build versions in the top level pom.xml file. If you
 for Databricks then you should use the `jenkins/databricks/build.sh` script and modify it for
 the version you want.
 
-To get an uber jar with more than 1 version you have to `mvn install` each version
-and then use one of the defined profiles in the dist module. See the next section
-for more details.
+To get an uber jar with more than 1 version you have to `mvn package` each version
+and then use one of the defined profiles in the dist module, or a comma-separated list of
+build versions. See the next section for more details.
 
 ### Building a Distribution for Multiple Versions of Spark
 
 By default the distribution jar only includes code for a single version of Spark. If you want
-to create a jar with multiple versions we currently have 4 options.
+to create a jar with multiple versions we have the following options.
 
 1. Build for all Apache Spark versions and CDH with no SNAPSHOT versions of Spark, only released. Use `-PnoSnapshots`.
 2. Build for all Apache Spark versions and CDH including SNAPSHOT versions of Spark we have supported for. Use `-Psnapshots`.
 3. Build for all Apache Spark versions, CDH and Databricks with no SNAPSHOT versions of Spark, only released. Use `-PnoSnaphsotsWithDatabricks`.
 4. Build for all Apache Spark versions, CDH and Databricks including SNAPSHOT versions of Spark we have supported for. Use `-PsnapshotsWithDatabricks`
+5. Build for an arbitrary combination of comma-separated build versions using `-Dincluded_buildvers=<CSV list of build versions>`.
+   E.g., `-Dincluded_buildvers=312,330`
 
-You must first build and install each of the versions of Spark and then build one final time using the profile for the option you want.
-
-There is a build script `build/buildall` to build everything with snapshots and this will have more options to build later.
+You must first build each of the versions of Spark and then build one final time using the profile for the option you want.
 
 You can also install some manually and build a combined jar. For instance to build non-snapshot versions:
 
 ```shell script
-mvn -Dbuildver=301 clean install -DskipTests
-mvn -Dbuildver=302 clean install -Drat.skip=true -DskipTests
-mvn -Dbuildver=303 clean install -Drat.skip=true -DskipTests
-mvn -Dbuildver=311 clean install -Drat.skip=true -DskipTests
-mvn -Dbuildver=312 clean install -Drat.skip=true -DskipTests
-mvn -Dbuildver=311cdh clean install -Drat.skip=true -DskipTests
+mvn clean
+mvn -Dbuildver=301 install -DskipTests
+mvn -Dbuildver=302 install -Drat.skip=true -DskipTests
+mvn -Dbuildver=303 install -Drat.skip=true -DskipTests
+mvn -Dbuildver=311 install -Drat.skip=true -DskipTests
+mvn -Dbuildver=312 install -Drat.skip=true -DskipTests
+mvn -Dbuildver=320 install -Drat.skip=true -DskipTests
+mvn -Dbuildver=311cdh install -Drat.skip=true -DskipTests
 mvn -pl dist -PnoSnapshots package -DskipTests
 ```
+#### Building with buildall script
+
+There is a build script `build/buildall` that automates the local build process. Use
+`./buid/buildall --help` for up-to-date use information.
+
+By default, it builds everything that is needed to create a distribution jar for all released (noSnapshots) Spark versions except for Databricks. Other profiles that you can pass using `--profile=<distribution profile>` include
+- `snapshots`
+- `minimumFeatureVersionMix` that currently includes 302, 311cdh, 312, 320 is recommended for catching incompatibilities already in the local development cycle
+
+For initial quick iterations we can use `--profile=<buildver>` to build a single-shim version. e.g., `--profile=301` for Spark 3.0.1.
+
+The option `--module=<module>` allows to limit the number of build steps. When iterating, we often don't have the need for the entire build. We may be interested in building everything necessary just to run integration tests (`--module=integration_tests`), or we may want to just rebuild the distribution jar (`--module=dist`)
+
+By default, `buildall` builds up to 4 shims in parallel using `xargs -P <n>`. This can be adjusted by
+specifying the environment variable `BUILD_PARALLEL=<n>`.
 
 ### Building against different CUDA Toolkit versions
 
-You can build against different versions of the CUDA Toolkit by using one of the following profiles:
+You can build against different versions of the CUDA Toolkit by using qone of the following profiles:
 * `-Pcuda11` (CUDA 11.0/11.1/11.2, default)
 
 ## Code contributions
@@ -98,6 +115,13 @@ dedicated shim modules.
 Thus, the conventional source code root directories `src/main/<language>` contain the files that
 are source-compatible with all supported Spark releases, both upstream and vendor-specific.
 
+The following acronyms may appear in directory names:
+
+|Acronym|Definition  |Example|Example Explanation                           |
+|-------|------------|-------|----------------------------------------------|
+|cdh    |Cloudera CDH|311cdh |Cloudera CDH Spark based on Apache Spark 3.1.1|
+|db     |Databricks  |312db  |Databricks Spark based on Spark 3.1.2         |
+
 The version-specific directory names have one of the following forms / use cases:
 - `src/main/312/scala` contains Scala source code for a single Spark version, 3.1.2 in this case
 - `src/main/312+-apache/scala`contains Scala source code for *upstream* **Apache** Spark builds,
@@ -107,14 +131,18 @@ The version-specific directory names have one of the following forms / use cases
 3.1.2 *exclusive*
 - `src/main/302to312-cdh` contains code that applies to Cloudera CDH shims between 3.0.2 *inclusive*,
    3.1.2 *inclusive*
+- `src/main/pre320-treenode` contains shims for the Catalyst `TreeNode` class before the
+  [children trait specialization in Apache Spark 3.2.0](https://issues.apache.org/jira/browse/SPARK-34906).
+- `src/main/post320-treenode` contains shims for the Catalyst `TreeNode` class after the
+  [children trait specialization in Apache Spark 3.2.0](https://issues.apache.org/jira/browse/SPARK-34906).
 
 
 ### Setting up an Integrated Development Environment
 
-Our project currently uses `build-helper-maven-plugin` for shimming against conflicting definitions of superclasses 
-in upstream versions that cannot be resolved without significant code duplication otherwise. To this end different 
-source directories with differently implemented same-named classes are 
-[added](https://www.mojohaus.org/build-helper-maven-plugin/add-source-mojo.html) 
+Our project currently uses `build-helper-maven-plugin` for shimming against conflicting definitions of superclasses
+in upstream versions that cannot be resolved without significant code duplication otherwise. To this end different
+source directories with differently implemented same-named classes are
+[added](https://www.mojohaus.org/build-helper-maven-plugin/add-source-mojo.html)
 for compilation depending on the targeted Spark version.
 
 This may require some modifications to IDEs' standard Maven import functionality.
@@ -123,27 +151,98 @@ This may require some modifications to IDEs' standard Maven import functionality
 
 _Last tested with 2021.2.1 Community Edition_
 
-To start working with the project in IDEA is as easy as 
-[opening](https://blog.jetbrains.com/idea/2008/03/opening-maven-projects-is-easy-as-pie/) the top level (parent) 
-[pom.xml](pom.xml). 
+To start working with the project in IDEA is as easy as
+[opening](https://blog.jetbrains.com/idea/2008/03/opening-maven-projects-is-easy-as-pie/) the top level (parent)
+[pom.xml](pom.xml).
 
 In order to make sure that IDEA handles profile-specific source code roots within a single Maven module correctly,
 [unselect](https://www.jetbrains.com/help/idea/2021.2/maven-importing.html) "Keep source and test folders on reimport".
 
 If you develop a feature that has to interact with the Shim layer or simply need to test the Plugin with a different
 Spark version, open [Maven tool window](https://www.jetbrains.com/help/idea/2021.2/maven-projects-tool-window.html) and
-select one of the `release3xx` profiles (e.g, `release320`) for Apache Spark 3.2.0, and click "Reload" 
+select one of the `release3xx` profiles (e.g, `release320`) for Apache Spark 3.2.0, and click "Reload"
 if not triggered automatically.
 
-There is a known issue with the shims/spark3xx submodules. After being enabled once, a module such as shims/spark312 
+There is a known issue with the shims/spark3xx submodules. After being enabled once, a module such as shims/spark312
 may remain active in IDEA even though you explicitly disable the Maven profile `release312` in the Maven tool window.
-With an extra IDEA shim module loaded the IDEA internal build "Build->Build Project" is likely to fail 
+With an extra IDEA shim module loaded the IDEA internal build "Build->Build Project" is likely to fail
 (whereas it has no adverse effect on Maven build). As a workaround, locate the pom.xml under the extraneous IDEA module,
 right-click on it and select "Maven->Ignore Projects".
 
 If you see Scala symbols unresolved (highlighted red) in IDEA please try the following steps to resolve it:
-- Make sure there are no relevant poms in "File->Settings->Build Tools->Maven->Ignored Files" 
-- Restart IDEA and click "Reload All Maven Projects" again 
+- Make sure there are no relevant poms in "File->Settings->Build Tools->Maven->Ignored Files"
+- Restart IDEA and click "Reload All Maven Projects" again
+
+#### Bloop Build Server
+
+[Bloop](https://scalacenter.github.io/bloop/) is a build server and a set of tools around Build
+Server Protocol (BSP) for Scala providing an integration path with IDEs that support it. In fact,
+you can generate a Bloop project from Maven just for the Maven modules and profiles you are
+interested in. For example, to generate the Bloop projects for the Spark 3.2.0 dependency
+just for the production code run:
+
+```shell script
+mvn install ch.epfl.scala:maven-bloop_2.13:1.4.9:bloopInstall -pl aggregator -am \
+  -DdownloadSources=true \
+  -Dbuildver=320 \
+  -DskipTests \
+  -Dskip \
+  -Dmaven.javadoc.skip \
+  -Dmaven.scalastyle.skip=true \
+  -Dmaven.updateconfig.skip=true
+```
+
+With `--generate-bloop` we integrated Bloop project generation into `buildall`. It makes it easier
+to generate projects for multiple Spark dependencies using the same profiles as our regular build.
+It makes sure that the project files belonging to different Spark dependencies are
+not clobbered by repeated `bloopInstall` Maven plugin invocations, and it uses
+[jq](https://stedolan.github.io/jq/) to post-process JSON-formatted project files such that they
+compile project classes into non-overlapping set of output directories.
+
+You can now open the spark-rapids as a
+[BSP project in IDEA](https://www.jetbrains.com/help/idea/bsp-support.html)
+
+# Bloop, Scala Metals, and Visual Studio Code
+
+_Last tested with 1.63.0-insider (Universal) Commit: bedf867b5b02c1c800fbaf4d6ce09cefba_
+
+Another, and arguably more popular, use of Bloop arises in connection with
+[Scala Metals](https://scalameta.org/metals/) and [VS @Code](https://code.visualstudio.com/).
+Scala Metals implements the
+[Language Server Protocol (LSP)](https://microsoft.github.io/language-server-protocol/) for Scala,
+and enables features such as context-aware autocomplete, and code browsing between Scala symbol
+definitions, references and vice versa. LSP is supported by many editors including Vim and Emacs.
+
+Here we document the integration with VS code. It makes development on a remote node almost
+as easy as local development, which comes very handy when working in Cloud environments.
+
+Run `./build/buildall --generate-bloop --profile=<profile>` to generate Bloop projects
+for required Spark dependencies, e.g. `--profile=320` for Spark 3.2.0. When developing
+remotely this is done on the remote node.
+
+Install [Scala Metals extension](https://scalameta.org/metals/docs/editors/vscode) in VS Code,
+either locally or into a Remote-SSH extension destination depending on your target environment.
+When your project folder is open in VS Code, it may prompt you to import Maven project.
+IMPORTANT: always decline with "Don't ask again", otherwise it will overwrite the Bloop projects
+generated with the default `301` profile. If you need to use a different profile, always rerun the
+command above manually. When regenerating projects it's recommended to proceed to Metals
+"Build commands" View, and click:
+1. "Restart build server"
+1. "Clean compile workspace"
+to avoid stale class files.
+
+Now you should be able to see Scala class members in the Explorer's Outline view and in the
+Breadcrumbs view at the top of the Editor with a Scala file open.
+
+Check Metals logs, "Run Doctor", etc if something is not working as expected. You can also verify
+that the Bloop build server and the Metals language server are running by executing `jps` in the
+Terminal window:
+```shell script
+jps -l
+72960 sun.tools.jps.Jps
+72356 bloop.Server
+72349 scala.meta.metals.Main
+```
 
 #### Other IDEs
 We welcome pull requests with tips how to setup your favorite IDE!
@@ -254,7 +353,7 @@ Please visit the [testing doc](tests/README.md) for details about how to run tes
 
 ### Pre-commit hooks
 We provide a basic config `.pre-commit-config.yaml` for [pre-commit](https://pre-commit.com/) to
-automate some aspects of the development process. As a convenience you can enable automatic 
+automate some aspects of the development process. As a convenience you can enable automatic
 copyright year updates by following the installation instructions on the
 [pre-commit homepage](https://pre-commit.com/).
 
@@ -299,7 +398,7 @@ manually trigger it by commenting `build`. It includes following steps,
 1. Mergeable check
 2. Blackduck vulnerability scan
 3. Fetch merged code (merge the pull request HEAD into BASE branch, e.g. fea-001 into branch-x)
-4. Run `mvn verify` and unit tests for multiple Spark versions in parallel. 
+4. Run `mvn verify` and unit tests for multiple Spark versions in parallel.
 Ref: [spark-premerge-build.sh](jenkins/spark-premerge-build.sh)
 
 If it fails, you can click the `Details` link of this check, and go to `Upload log -> Jenkins log for pull request xxx (click here)` to

CONTRIBUTING.md

@@ -178,15 +178,15 @@
    APPENDIX: How to apply the Apache License to your work.
 
       To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "{}"
+      boilerplate notice, with the fields enclosed by brackets "[]"
       replaced with your own identifying information. (Don't include
       the brackets!)  The text should be enclosed in the appropriate
       comment syntax for the file format. We also recommend that a
       file or class name and description of purpose be included on the
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2019 nvspark
+   Copyright [yyyy] [name of copyright owner]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.