You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: README.md
+17-8
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
##### \* LEGAL NOTICE: Your use of this software and any required dependent software (the "Software Package") is subject to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party or open source software included in or with the Software Package, and your use indicates your acceptance of all such terms. Please refer to the "TPP.txt" or other similarly-named text file included with the Software Package for additional details.
2
+
3
+
##### \* Optimized Analytics Package for Spark* Platform is under Apache 2.0 (https://www.apache.org/licenses/LICENSE-2.0).
4
+
1
5
# OAP MLlib
2
6
3
7
## Overview
@@ -17,13 +21,13 @@ You can find the all the OAP MLlib documents on the [project web page](https://o
17
21
18
22
### Java/Scala Users Preferred
19
23
20
-
Use a pre-built OAP MLlib JAR to get started. You can firstly download OAP package from [OAP-JARs-Tarball](https://github.com/Intel-bigdata/OAP/releases/download/v1.1.0-spark-3.0.0/oap-1.1.0-bin-spark-3.0.0.tar.gz) and extract this Tarball to get `oap-mllib-x.x.x-with-spark-x.x.x.jar` under `oap-1.1.0-bin-spark-3.0.0/jars`.
24
+
Use a pre-built OAP MLlib JAR to get started. You can firstly download OAP package from [OAP-JARs-Tarball](https://github.com/oap-project/oap-tools/releases/download/v1.1.1-spark-3.1.1/oap-1.1.1-bin-spark-3.1.1.tar.gz) and extract this Tarball to get `oap-mllib-x.x.x.jar` under `oap-1.1.1-bin-spark-3.1.1/jars`.
21
25
22
26
Then you can refer to the following [Running](#running) section to try out.
23
27
24
28
### Python/PySpark Users Preferred
25
29
26
-
Use a pre-built JAR to get started. If you have finished [OAP-Installation-Guide](./docs/OAP-Installation-Guide.md), you can find compiled OAP MLlib JAR `oap-mllib-x.x.x-with-spark-x.x.x.jar` in `$HOME/miniconda2/envs/oapenv/oap_jars/`.
30
+
Use a pre-built JAR to get started. If you have finished [OAP-Installation-Guide](./docs/OAP-Installation-Guide.md), you can find compiled OAP MLlib JAR `oap-mllib-x.x.x.jar` in `$HOME/miniconda2/envs/oapenv/oap_jars/`.
27
31
28
32
Then you can refer to the following [Running](#running) section to try out.
29
33
@@ -49,13 +53,17 @@ Users usually run Spark application on __YARN__ with __client__ mode. In that ca
OAP MLlib adopted oneDAL as implementation backend. oneDAL requires enough native memory allocated for each executor. For large dataset, depending on algorithms, you may need to tune `spark.executor.memoryOverhead` to allocate enough native memory. Setting this value to larger than __dataset size / executor number__ is a good starting point.
66
+
59
67
### Sanity Check
60
68
61
69
#### Setup `env.sh`
@@ -103,10 +111,10 @@ Intel® oneAPI Toolkits and its components can be downloaded and install from [h
103
111
104
112
More details about oneAPI can be found [here](https://software.intel.com/content/www/us/en/develop/tools/oneapi.html).
105
113
106
-
You can refer to [this script](dev/install-build-deps-centos.sh) to install correct dependencies.
107
-
108
114
Scala and Java dependency descriptions are already included in Maven POM file.
109
115
116
+
***Note:*** You can refer to [this script](dev/install-build-deps-centos.sh) to install correct dependencies: DPC++/C++, oneDAL, oneTBB, oneCCL.
117
+
110
118
### Build
111
119
112
120
#### Building oneCCL
@@ -161,12 +169,13 @@ To build, run the following commands:
161
169
$ cd mllib-dal
162
170
$ ./build.sh
163
171
```
172
+
164
173
The target can be built against different Spark versions by specifying profile with <spark-x.x.x>. E.g.
165
174
```
166
175
$ ./build.sh spark-3.1.1
167
176
```
168
177
If no profile parameter is given, the Spark version 3.0.0 will be activated by default.
169
-
The built JAR package will be placed in `target` directory with the name `oap-mllib-x.x.x-with-spark-x.x.x.jar`.
178
+
The built JAR package will be placed in `target` directory with the name `oap-mllib-x.x.x.jar`.
0 commit comments