Fix flaky WPS provider responses (i.e.: other WPS birds) causing failure during their registration in weaver

[fmigneault]

Overview

Fix flaky WPS provider responses (i.e.: other WPS birds) causing failure during their registration in weaver.

Changes

Non-breaking changes

• Fix flaky WPS provider responses (i.e.: other WPS birds) causing failure during their registration in weaver.

  In some cases, the WPS birds would not respond properly when starting the stack, either because they are still
  initiating or due to other temporary failures such as services being restarted until healthy. This fix introduces
  a retry mechanism to attempt WPS registration in weaver up to WEAVER_WPS_PROVIDERS_RETRY_COUNT=5 times
  (1 initial attempt + 5 retries), and with WEAVER_WPS_PROVIDERS_RETRY_AFTER=5 second intervals between each retry.
  If the maximum number of retries for any WPS provider or the WEAVER_WPS_PROVIDERS_MAX_TIME across all registrations
  are reached, the operation is aborted.

Breaking changes

• n/a

Related Issue / Discussion

• Close fix flaky WPS provider responses causing failure during their registration in weaver #318 (using fix-canarieapi as base branch) instead of master (this PR)

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1501/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : fix-flaky-wps-provider-standalone
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-92.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1090/

NOTEBOOK TEST RESULTS

    
[2023-04-20T00:12:47.177Z] ============================= test session starts ==============================
[2023-04-20T00:12:47.178Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-20T00:12:47.178Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@3
[2023-04-20T00:12:47.178Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-20T00:12:47.178Z] collected 268 items
[2023-04-20T00:12:47.178Z] 
[2023-04-20T00:12:56.916Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-20T00:13:09.574Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-20T00:13:26.922Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-20T00:13:37.041Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 11%]
[2023-04-20T00:13:40.297Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-20T00:26:44.289Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-20T00:26:46.848Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-20T00:26:52.021Z] ...............                                                          [ 26%]
[2023-04-20T00:27:02.867Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-20T00:27:10.754Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-20T00:27:32.553Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-20T00:27:34.474Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-20T00:27:38.930Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-20T00:27:45.332Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-04-20T00:30:51.377Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-20T00:32:09.709Z] .............                                                            [ 47%]
[2023-04-20T00:32:15.064Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-20T00:32:17.812Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-20T00:32:53.977Z] .................                                                        [ 58%]
[2023-04-20T00:32:59.700Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-20T00:33:01.077Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-20T00:33:17.701Z] .........                                                                [ 64%]
[2023-04-20T00:33:29.553Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-20T00:33:40.168Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-20T00:34:28.234Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-20T00:34:30.780Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-20T00:34:46.353Z] ......                                                                   [ 81%]
[2023-04-20T00:39:53.008Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-20T00:41:42.838Z] .............                                                            [ 86%]
[2023-04-20T00:42:00.946Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-20T00:48:40.027Z] ....s.                                                                   [ 89%]
[2023-04-20T00:48:50.030Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-20T00:48:54.147Z] ...                                                                      [ 91%]
[2023-04-20T00:49:09.086Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-20T00:49:35.300Z] ......                                                                   [ 93%]
[2023-04-20T00:49:37.229Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-20T00:53:11.391Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-04-20T00:53:11.391Z] 
[2023-04-20T00:53:11.391Z] ================= 265 passed, 3 skipped in 2419.84s (0:40:19) ==================
    
  

[tlvu]

+1

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1506/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-flaky-wps-provider-standalone
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-92.rdext.crim.ca

Infrastructure deployment failed. Instance has not been destroyed. @matprov