Skip to content

Add cancel endpoint, R progress monitor, and upgrade PyWPS#68

Merged
QSparks merged 47 commits intoci-climatex-demofrom
ci-climatex-demo-pywps46
Apr 3, 2025
Merged

Add cancel endpoint, R progress monitor, and upgrade PyWPS#68
QSparks merged 47 commits intoci-climatex-demofrom
ci-climatex-demo-pywps46

Conversation

@QSparks
Copy link
Contributor

@QSparks QSparks commented Mar 26, 2025

Adds support for cancelling running WPS processes. Upgrades to PyWPS v4.6.0 for improved job queue handling and adds finer-resolution progress updates via R console callbacks.

cancel_process.py

  • New cancel endpoint (POST /wps/cancel-process) that:
    • Sends SIGINT to the running process
    • Retrieves tracked response and calls response.clean()
    • Updates DB status (WPS_STATUS.FAILED)

response_tracker.py

  • In-memory registry to track active response objects by UUID

utils.py

  • Integrated R console progress monitor:
    • Parses R output to update WPS progress bar
    • Detects cancellation mid-process, interrupts R via stop(...)
    • Triggers cleanup via response.clean() if cancelled
  • Added update_status_with_check() to avoid updating progress on cancelled/failed jobs
  • Includes a fallback raise_if_failed() for missed cancellations

Other

  • Upgrades to latest pywps and nchelpers
  • Updated WSGI config for the new endpoint.
  • Adds docs/cancellation.md with usage instructions.

QSparks added 30 commits February 14, 2025 11:07
@QSparks
Use PyWPS 4.6.0, .references for OPeNDAP inputs
412ca17
@QSparks
Handle OPeNDAP vs. local data in select_args util
01fb841
@QSparks
Revert select_args_from_input_list
1fe48fb
@QSparks
extract .data if it exists in select_args_from_input_list
ebf3b0f
@QSparks
Revert .data extract, add logging
af6a5cb
@QSparks
Use as_reference for ComplexInput
fcb7a19
@QSparks
Black
2ac6641
@QSparks
Revet as_reference usage
0d2c00d
@QSparks
Test OPeNDAP input fix
6dd6276
@QSparks
Revert test OPeNDAP input fix
f2e5efa
@QSparks
Log wps_CI inputs
a7abef7
@QSparks
Test set gcm and obs files to href
022eb8d
@QSparks
Test override collect_args for gcm and obs files
75b86aa
@QSparks
Remove as_reference to allow container starts
5153ba4
@QSparks
Use updated wps_tools
9d98fd9
@QSparks
Install git in Dockerfile
76f9aa7
@QSparks
Use python3.9 required by wps_tools
9a82fc0
@QSparks
Use nchelpers v5.5.11
68e789f
@QSparks
Add fine-grained progress steps
526ab0c
@QSparks
Use wps-tools v2.1.2 on pypi
d1358bf
@QSparks
Add cancellation process
07d19ff
@QSparks
Add check for failed status
ae822a3
@QSparks
SIGTERM target process in wps_cancel
3ec8557
@QSparks
Revert sigterm calls in wps_cancel
58eeed0
@QSparks
Move WPS cancellation to a separate endpoint
961acb2
@QSparks
Add logging to wsgi.py
b2d796f
@QSparks
Prevent updates after cancellation
ba06924
@QSparks
Update in-memory status on cancel
13445d4
@QSparks
Add graceful termination on cancel
c48d21a
@QSparks
Check running before cancel
91c72c6
@QSparks QSparks requested a review from Nospamas March 26, 2025 23:08
@QSparks QSparks self-assigned this Mar 26, 2025
Nospamas
Nospamas approved these changes Mar 27, 2025
View reviewed changes
Copy link

@Nospamas Nospamas left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (Pending CI tests passing?) 👍

chickadee/cancel_process.py
return error(f"Failed to update status: {str(e)}", "500 Internal Server Error")
finally:
session.close()

Copy link

@Nospamas Nospamas Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, good error handling 👍

@QSparks
Copy link
Contributor Author

QSparks commented Apr 1, 2025

LGTM (Pending CI tests passing?) 👍

CI fixed with updates to ClimDown finishing some refactoring that was already underway with the ci-climatex branch & validating process errors via WPS-compliant response body instead of relying on stderr in wps-tools.

Nospamas
Nospamas approved these changes Apr 2, 2025
View reviewed changes
Copy link

@Nospamas Nospamas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

requirements.txt Outdated
rpy2==3.3.6
werkzeug==1.0.1
wps-tools[r]==2.1.2
wps-tools @ git+https://github.com/pacificclimate/wps-tools.git@i60-file-url-handling#egg=wps-tools
Copy link

@Nospamas Nospamas Apr 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably don't forget to release this one and use the version number once its ready :)

@QSparks QSparks merged commit 8391acb into ci-climatex-demo Apr 3, 2025
4 checks passed
@QSparks QSparks mentioned this pull request Nov 24, 2025
Merged
1 task
QSparks added a commit that referenced this pull request Dec 12, 2025
@QSparks
Ci climatex demo (#70)
90af236 
* Add and update CI/demo notebooks
* Update CI environment: Python versions, system deps, packages, Dockerfile, and Black workflow
* Add cancel endpoint, R progress monitor, and upgrade PyWPS (#68)
* Maintenance updates  (#69)
* Add USE_PYPROJECT option in black workflow
* Use ClimDown v2.0.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Nospamas Nospamas Nospamas approved these changes

Assignees

@QSparks QSparks

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@QSparks @Nospamas