[Bug] license_select failing kfp_ray test

[daw3rd]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/Other

What happened + What you expected to happen

With the recent PR to properly enabled kfp workflow tests (PR #672) on transforms, we found that repo_level_ordering was not passing (see log here). To get the PR to pass, license_select/kfp_ray/Makefile was renamed to Makefile.disable-cicd.

Reproduction script

To reproduced

cd transforms/code/license_select/kfp_ray
git mv Makefile.disable-cicd Makefile
git commit -s -a -m "restore kfp_ray test for license_select"
git push

Make a PR and watch the cicd

Anything else

Image: "quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" with ID "sha256:37d2e98f3b1b4f406f20254b3ae5e96a1081bd731917aa62848a95bcadda080c" not yet present on node "dataprep-worker2", loading...
Image: "quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" with ID "sha256:37d2e98f3b1b4f406f20254b3ae5e96a1081bd731917aa62848a95bcadda080c" not yet present on node "dataprep-control-plane", loading...
Image: "quay.io/dataprep1/data-prep-kit/kfp-data-processing:latest" with ID "sha256:37d2e98f3b1b4f406f20254b3ae5e96a1081bd731917aa62848a95bcadda080c" not yet present on node "dataprep-worker", loading...
make[3]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/kfp/kfp_ray_components'
. /home/runner/work/data-prep-kit/data-prep-kit/transforms/code/license_select/kfp_ray/../../../..//transforms/venv/bin/activate && python -m workflow_support.pipeline_utils.pipelines_tests_utils -c "sanity-test" -p /home/runner/work/data-prep-kit/data-prep-kit/transforms/code/license_select/kfp_ray/license_select_wf.yaml -e "http://localhost:8080/"
12:25:34 WARNING - Failure to get pipeline. Number of pipelines with name license_select_wf is 0
12:25:34 INFO - Pipeline uploaded
12:25:34 INFO - Pipeline run license_select_wf 2024_10_05_12_25_34 submitted
12:26:34 INFO - Got pipeline execution status Running
12:27:34 INFO - Got pipeline execution status Running
12:28:34 INFO - Got pipeline execution status Running
12:29:34 INFO - Got pipeline execution status Running
12:30:34 INFO - Got pipeline execution status Running
12:31:34 INFO - Got pipeline execution status Running
12:32:35 INFO - Got pipeline execution status Running
12:33:35 INFO - Got pipeline execution status Failed
12:33:35 WARNING - Pipeline license_select_wf failed with error None and status Failed
make[2]: *** [/home/runner/work/data-prep-kit/data-prep-kit/transforms/code/license_select/kfp_ray/../../../..//transforms/.make.workflows:46: .workflows.test-pipeline] Error 1
make[1]: *** [Makefile:45: workflow-test] Error 2
make[2]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/code/license_select/kfp_ray'
make[1]: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/code/license_select/kfp_ray'
make: *** [Makefile:56: workflow-test] Error 2
make: Leaving directory '/home/runner/work/data-prep-kit/data-prep-kit/transforms/code/license_select'
Error: Process completed with exit code 2.

OS

Ubuntu

Python

3.10.x

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[revit13]

It is added to the kfp black list in #684. Additionally, the kfp_ray/Makefile has been restored to enable local testing and to verify the workflow build in the CI/CD pipeline.

[revit13]

With #781 the pipeline still fails with:

03:46:04 INFO - Getting supported licenses from file test/license_select/sample_approved_licenses.json
03:46:04 ERROR - Error reading file test/license_select/sample_approved_licenses.json: [Errno 2] No such file or directory: 'test/license_select/sample_approved_licenses.json'
03:46:04 ERROR - Failed to read file: test/license_select/sample_approved_licenses.json due to [Errno 2] No such file or directory: 'test/license_select/sample_approved_licenses.json'
03:46:04 INFO - pipeline id runtime_pipeline_id