fix: trigger doc build all condition #2242

yuejiaointel · 2025-01-06T23:14:47Z

Description

in this PR I fixed several problems so now doc can build without autodoc warnings, also I added code so in future if autodoc gives a warning it will be marked as an error so pipeline will fail. It still has duplicate warnings because we are having multiple same classes def in daal4py.

_List associated issue number(s) if exist(s): #8786

PR should start as a draft, then move to ready for review state after CI is passed and all applicable checkboxes are closed.
This approach ensures that reviewers don't spend extra time asking for regular requirements.

You can remove a checkbox as not applicable only if it doesn't relate to this PR in any way.
For example, PR with docs update doesn't require checkboxes for performance while PR with any change in actual code should have checkboxes and justify how this code change is expected to affect performance (or justification should be self-evident).

Checklist to comply with before moving PR from draft:

PR completeness and readability

I have reviewed my changes thoroughly before submitting this pull request.
I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes or created a separate PR with update and provided its number in the description, if necessary.
Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
I have added a respective label(s) to PR if I have a permission for that.
I have resolved any merge conflicts that might occur with the base branch.

Testing

I have run it locally and tested the changes extensively.
All CI jobs are green or I have provided justification why they aren't.
I have extended testing suite if new functionality was introduced in this PR.

Performance

I have measured performance for affected algorithms using scikit-learn_bench and provided at least summary table with measured data, if performance change is expected.
I have provided justification why performance has changed or why changes are not expected.
I have provided justification why quality metrics have changed or why changes are not expected.
I have extended benchmarking suite and provided corresponding scikit-learn_bench PR if new measurable functionality was introduced in this PR.

codecov · 2025-01-06T23:54:54Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Flag	Coverage Δ
azure	`78.01% <ø> (+1.22%)`	⬆️
github	`71.04% <ø> (+0.85%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 14 files with indirect coverage changes

david-cortes-intel · 2025-01-07T13:01:44Z

Would the problem that this PR addresses get fixed as part of this other PR?
#2214

…nd sklearn doc build

…d circular import

yuejiaointel · 2025-01-10T05:24:58Z

Would the problem that this PR addresses get fixed as part of this other PR? #2214

Hi David,
I think it is different, the ref pr seems about making sure the doc are generated with the right version, but this one is mainly about fix the warning about doc, making sure doc can be generated completely, and bring back doc in ci workflow, now it is skipped.
Best,
Yue

.ci/pipeline/docs.yml

david-cortes-intel · 2025-01-16T07:55:53Z

.ci/pipeline/docs.yml

@@ -66,16 +66,23 @@ jobs:
  - script: |
      export PREFIX=$(dirname $(dirname $(which python)))
      export DALROOT=$PREFIX
+      export LD_LIBRARY_PATH=$(dirname $(dirname $(which python)))/lib:$LD_LIBRARY_PATH


Sklearnex has an option to add oneDAL's path to its RPATH which could be used here - see e.g. here:

scikit-learn-intelex/conda-recipe/build.sh

Line 31 in 6742689

export DALRPATH=--abs-rpath

scikit-learn-intelex/setup.py

Line 43 in 6742689

USE_ABS_RPATH = False

Hi David,
I think by setting DALROOT, and since this pipeline is not running in conda, in build.sh will set DALRPATH automatically by entering the elif statement since "${DALROOT}" != "${CONDA_PREFIX}"
Best,
Yue

It will set it in the call to build.sh, but not in the call to setup.py build_ext.

.ci/pipeline/docs.yml

doc/Makefile

david-cortes-intel · 2025-01-16T08:21:11Z

@yuejiaointel @icfaust Is there some way that the generated HTMLs could be linked as a clickable and rendable link from the "Docs" CI that we see here at the bottom of PRs without having to manually download them from AzureDevOps?

icfaust

Some initial thoughts, I've never looked at this pipeline before, and seeing a daal-devel install makes me worried how outdated these scripts are.

icfaust · 2025-01-16T07:38:34Z

.ci/pipeline/docs.yml

@@ -48,13 +47,14 @@ jobs:
    vmImage: 'ubuntu-22.04'
  steps:
  - script: |
+      source /opt/intel/oneapi/compiler/latest/env/vars.sh


Suggested change

source /opt/intel/oneapi/compiler/latest/env/vars.sh

The compiler is not available by default in the azure pipeline, it raises some interesting questions. (see follow-up comments)

Hi Ethan,
Thx reviewing this PR! I removed this this source. I was using it to ensure tbb.so file but it was already in the daal-devel dependency
Best,
Yue

icfaust · 2025-01-16T07:41:22Z

.ci/pipeline/docs.yml

      bash .ci/scripts/describe_system.sh
    displayName: 'System info'
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.11'
      addToPath: true
-  - script: sudo apt-get update && sudo apt-get install -y clang-format
+  - script: sudo apt-get update && sudo apt-get install -y clang-format && sudo apt-get install -y pandoc 


Suggested change

- script: sudo apt-get update && sudo apt-get install -y clang-format && sudo apt-get install -y pandoc

- script: sudo apt-get update && sudo apt-get install -y clang-format pandoc

apt-get install can be merged to one call.

Hi Ethan,
Good catch! I merged the apt get command
Best,
Yue

icfaust · 2025-01-16T08:25:55Z

.ci/pipeline/docs.yml

+      export LD_LIBRARY_PATH=$(dirname $(dirname $(which python)))/lib:$LD_LIBRARY_PATH
      ./conda-recipe/build.sh
+      cp ${DALROOT}/lib/python3.11/site-packages/daal4py/_daal4py.cpython*.so daal4py/


Suggested change

export LD_LIBRARY_PATH=$(dirname $(dirname $(which python)))/lib:$LD_LIBRARY_PATH

./conda-recipe/build.sh

cp ${DALROOT}/lib/python3.11/site-packages/daal4py/_daal4py.cpython*.so daal4py/

NO_DPC=1

python setup.py build_ext --inplace

If I understand correctly, I think you want the _daal4py.so just so you can import the modules of the repo in place. You can just build the extension, which will suffice. Do you also need to import onedal for the docs?

I think NO_DPC should not be necessary if the compiler doesn't support NO_DPC, but NO_DIST=1 might be required.

for build_ext that is clear that NO_DPC isn't necessary, if we want to build onedal pybind11, and we are using the intel compiler, it may save some time. Will wait on @yuejiaointel to clarify what all is necessary.

Note that build_ext will only build daal4py. For sklearnex you need build.

Hi Ethan Hi David!
Thx for this good information, setyp.py build_ext --inplace can build daal4py but still need build.sh for slearnex doc to be built

What is removed:

step of setting LD_LIBRARY_PATH in build daal4py/sklearnex stage

step of cp the build file back in the daal4py folder

What is needed:

NO_DIST=1 otherwise build.sh will complain MPIROOT not being set.

./conda-recipe/build.sh otherwise sklearnex doc will give module not found warnings

LD_LIBRARY_PATH in the 2 build doc stages, pipeline will gives the module not found warning if not set.

Given this I am not sure if it is better to run setup.py twice (once in place and once by running build.sh) or do the cp command. For now I'll have setup.py run twice here. I also noticed the last step of python setup.py build_ext --inplace is copy daal4py .so file from lib back to daal4py folder.

Best,
Yue

@yuejiaointel A call to python setup.py build also builds the module in-place. One is for daal4py and the other for onedal+sklearnex. They are built using different frameworks which follow different setup commands.

Also for paths involving python modules, please use PYTHONPATH if needed to make it more obvious that it's about python things.

david-cortes-intel · 2025-01-16T08:39:58Z

@yuejiaointel I'm seeing some missing things in the docs generated here. Particularly, autoclass is missing the constructors of daal4py algorithm classes. This is for example how Cholesky looks like in the current online docs:

And this is how it renders in the generated artifact:

I have the same issue when building the docs locally so not sure if it's related to this PR.

…en copy

david-cortes-intel · 2025-01-17T07:42:18Z

.ci/pipeline/docs.yml

      ./conda-recipe/build.sh
    displayName: 'Build daal4py/sklearnex'
  - script: |
      export LD_LIBRARY_PATH=$(dirname $(dirname $(which python)))/lib:$LD_LIBRARY_PATH
      cd doc/daal4py
-      make html
+      make html 2>&1 | tee build.log


Please add this file to gitignore.

david-cortes-intel · 2025-01-17T07:48:51Z

doc/build-doc.sh

@@ -25,8 +25,16 @@ mkdir $SAMPLES_DIR

 # copy jupyter notebooks
 cd ..
-rsync -a --exclude='doc/$SAMPLES_DIR/daal4py_data_science.ipynb' examples/notebooks/*.ipynb doc/$SAMPLES_DIR


Why would this change be needed?

Hi David,
The exclude command does not recognize a path it just read as a string literal, so I changed to just file name. If we don't exclude this file when cp from examples, sklearnex doc will give a warning that this file is not included in any of the rst files
Best,
Yue

That's curious - it does work for me when I try it locally. It also appears to show up like that in stackexchange posts:
https://askubuntu.com/questions/1420321/how-to-exclude-the-multiple-directory-in-rsync

.. but it looks like just pattern matching should also work. Would it is still work if you prefix it with .*?

david-cortes-intel · 2025-01-17T07:50:45Z

doc/daal4py/Makefile

@@ -33,4 +33,4 @@ help:
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)


Better to leave the empty line at the end here.

david-cortes-intel · 2025-01-17T07:54:20Z

.ci/pipeline/docs.yml

+      python setup.py build_ext --inplace
      ./conda-recipe/build.sh


There's a call to both build_ext and conda recipe's build. What's the idea for having both? That'd leave you with one importable module (daal4py) being in two places, with one of them having oneDAL in the rpath and the other without.

If you want to build both extensions in-place, you should replace the call to build.sh with this:

python setup.py build --abs-rpath

If that doesn't work, perhaps would require these two:

python setup.py build_ext --inplace --abs-rpath python setup.py build --abs-rpath

david-cortes-intel · 2025-01-17T07:58:10Z

doc/build-doc.sh

+cat build.log
+
+# check for autodoc warnings
+if grep "WARNING" build.log; then


On a second look, maybe it'd be easier to follow if having both of these checks (daal4py/sklearnex) in the .yaml files.

david-cortes-intel · 2025-01-17T07:58:27Z

doc/build-doc.sh

+# check for autodoc warnings
+if grep "WARNING" build.log; then
+    echo "Warnings detected, build failed!"
+    exit 1


This block should not be needed if you are passing option -W to sphinx. Also please leave the empty lines at the end of files.

fix: trigger doc build all condition

e2fdedc

yue.jiao added 3 commits January 6, 2025 21:36

test-ci-fix: try not reset daalroot in doc.yml

aba7949

test-ci-fix: revert dalroot set in doc.yml

4be9a0a

temp: print statements to pipline to see the sys path behavior

f979c5e

yue.jiao added 9 commits January 7, 2025 20:49

fix: add sklearn and daal4py folder to sys path

bc0b5c2

fix: revert change to conf.py

091f9d0

fix: confirm change needed in conf.py

598b2c2

fix: import daal4py and sklearn to path in conf.py for both daal4py a…

8a6bbdf

…nd sklearn doc build

ci-print: remove useless print in doc.yml and sys.path import to avoi…

2344410

…d circular import

Merge branch 'main' into fix_build_doc_ci

1c42d17

test-ci: additional print

edbcd5b

fix: import build .so file

0238639

fix: update intelex to include build dir as well

2bff139

yue.jiao added 3 commits January 9, 2025 21:27

fix: import build only test

fdf20aa

fix: test need the lib folder

b910ada

fix: test add one dal to dep

81f96d9

david-cortes-intel requested changes Jan 10, 2025

View reviewed changes

.ci/pipeline/docs.yml Show resolved Hide resolved

.ci/pipeline/docs.yml Outdated Show resolved Hide resolved

.ci/pipeline/docs.yml Outdated Show resolved Hide resolved

.ci/pipeline/docs.yml Outdated Show resolved Hide resolved

yue.jiao added 7 commits January 10, 2025 11:24

fix: remove unused env var

afd1876

test-ci: print the sys path library path

17dc7c9

ci-test: remove some prints test import

64b04a0

ci-test: print the import

6761c36

ci-test: print

ea567fd

ci-test: print

af4afc7

ci-test: print

b030f59

yuejiaointel force-pushed the fix_build_doc_ci branch from c40e68d to b030f59 Compare January 13, 2025 06:37

yue.jiao added 2 commits January 12, 2025 22:39

ci-test: cp so file

8879aae

fix: fix syntax

201f24e

yuejiaointel requested review from Alexsandruss, samir-nasibli, icfaust, homksei, ahuber21 and ethanglaser as code owners January 16, 2025 05:56

david-cortes-intel requested changes Jan 16, 2025

View reviewed changes

icfaust reviewed Jan 16, 2025

View reviewed changes

yuejiaointel added 17 commits January 16, 2025 11:58

fix: comnine apt get and remove useless source command

840eee1

fix: move warning detect from makefile to build-doc.sh and yml

c68de58

ci-test: print

42eb9b5

ci-test: print

d4276a6

ci-test: print

1229c77

ci-test: print

f78c8d2

fix: fix exclude command to not include daal4py_data_science.ipynb wh…

1478c48

…en copy

fix: use --inplace flag for setup.py isntead

9ea61f5

fix: remove cp command

463b270

fneed dalroot

12707fc

ci-test: print

2e1aac3

ci-test: print

36a2b0c

ci-test: print

6528f6c

ci-test: print

527a78f

ci-test: print

4e61a01

fix: remove build.sh

ca1488e

fix: add build.sh back

b9a2b0e

david-cortes-intel reviewed Jan 17, 2025

View reviewed changes

uxlfoundation deleted a comment from yuejiaointel Jan 17, 2025

fix: empty lines at end replace build.sh with another setup.py

6a19911

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: trigger doc build all condition #2242

fix: trigger doc build all condition #2242

yuejiaointel commented Jan 6, 2025 •

edited

Loading

codecov bot commented Jan 6, 2025 •

edited

Loading

david-cortes-intel commented Jan 7, 2025

yuejiaointel commented Jan 10, 2025

david-cortes-intel Jan 16, 2025

yuejiaointel Jan 16, 2025

david-cortes-intel Jan 17, 2025

david-cortes-intel commented Jan 16, 2025

icfaust left a comment

icfaust Jan 16, 2025

yuejiaointel Jan 16, 2025

icfaust Jan 16, 2025

yuejiaointel Jan 16, 2025

icfaust Jan 16, 2025

david-cortes-intel Jan 16, 2025

icfaust Jan 16, 2025

david-cortes-intel Jan 16, 2025

yuejiaointel Jan 17, 2025 •

edited

Loading

david-cortes-intel Jan 17, 2025

david-cortes-intel Jan 17, 2025

david-cortes-intel commented Jan 16, 2025

david-cortes-intel Jan 17, 2025

david-cortes-intel Jan 17, 2025

yuejiaointel Jan 17, 2025

david-cortes-intel Jan 17, 2025

david-cortes-intel Jan 17, 2025

david-cortes-intel Jan 17, 2025

david-cortes-intel Jan 17, 2025

david-cortes-intel Jan 17, 2025

	- script: sudo apt-get update && sudo apt-get install -y clang-format && sudo apt-get install -y pandoc
	- script: sudo apt-get update && sudo apt-get install -y clang-format pandoc

fix: trigger doc build all condition #2242

Are you sure you want to change the base?

fix: trigger doc build all condition #2242

Conversation

yuejiaointel commented Jan 6, 2025 • edited Loading

Description

codecov bot commented Jan 6, 2025 • edited Loading

Codecov Report

david-cortes-intel commented Jan 7, 2025

yuejiaointel commented Jan 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david-cortes-intel commented Jan 16, 2025

icfaust left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuejiaointel Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david-cortes-intel commented Jan 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuejiaointel commented Jan 6, 2025 •

edited

Loading

codecov bot commented Jan 6, 2025 •

edited

Loading

yuejiaointel Jan 17, 2025 •

edited

Loading