Create pipeline and workflow with specified values for cpu_limit and memory_limit

[jennydaman]

I want to be able to create a pipeline where a piping has non-default values for cpu_limit and memory_limit, e.g.

❯ cat fetal_brain_reconstruction_pipeline.yml
name: "Automatic Fetal Brain Reconstruction Pipeline v1.2.0-20231206.1"
authors: "Jennings Zhang <[email protected]>"
category: "MRI"
description: "Automatic fetal brain reconstruction pipeline using machine learning developed by Kiho's group at the FNNDS
locked: true
plugin_tree:
- title: "Unstack folders"
  plugin: pl-unstack-folders v1.0.0
  previous: null

- title: "Convert DICOM to NIFTI"
  plugin: pl-dcm2niix v1.0.0
  previous: "Unstack folders"
  plugin_parameter_defaults:
    b: n  # no BIDS sidecar
- title: "Brain Mask"
  plugin: pl-emerald v0.2.1
  previous: "Convert DICOM to NIFTI"
  plugin_parameter_defaults:
    mask_suffix: _mask.nii
    outputs: '0.0:_brain.nii,0.2:_overlay02.nii'
- title: "Select brain extraction"
  previous: "Brain Mask"
  plugin: pl-bulk-rename v0.1.2
  plugin_parameter_defaults:
    filter: ".*_brain\\.nii$"
    expression: "(.*)_brain\\.nii"
    replacement: "$1.nii"
- title: "N4 Bias Field Correction"
  plugin: pl-ANTs_N4BiasFieldCorrection v0.2.7.4
  plugin_parameter_defaults:
    # !!!!! look here! increased resources !!!!!
    cpu_limit: 12000m
    memory_limit: 18Gi
  previous: "Select brain extraction"
- title: "Automatic Fetal Brain Assessment"
  plugin: pl-fetal-brain-assessment v1.3.1
  previous: "N4 Bias Field Correction"
- title: "Reconstruction"
  plugin: pl-NeSVoR-reconstruct v0.5.0
  previous: "Automatic Fetal Brain Assessment"

  # optional
- title: "Brain extraction preview figures"
  plugin: pl-mri-preview v1.2.2
  previous: "Brain Mask"
- title: "Brain reconstruction preview figures"
  plugin: pl-mri-preview v1.2.2
  previous: "Reconstruction"

CUBE accepts this YAML as a valid pipeline sourcefile, however the cpu_limit and memory_limit are not set on the plugin instance when the workflow is created.

I also tried to create the workflow instance with the parameters set:

curl 'http://rc-live.tch.harvard.edu:32000/api/v1/pipelines/75/workflows/' -X POST -H 'Accept: application/vnd.collection+json' -H 'Content-Type: application/vnd.collection+json' -H 'Authorization: Token REDACTED' --data '{"template":{"data":[{"name":"previous_plugin_inst_id","value":4985},{"name":"nodes_info","value":"[{\"piping_id\":501,\"previous_piping_id\":500,\"compute_resource_name\":\"galena\",\"title\":\"Convert DICOM to NIFTI\"},{\"piping_id\":502,\"previous_piping_id\":501,\"compute_resource_name\":\"galena-avx\",\"title\":\"Brain Mask\"},{\"piping_id\":503,\"previous_piping_id\":502,\"compute_resource_name\":\"galena\",\"title\":\"Select brain extraction\"},{\"piping_id\":504,\"previous_piping_id\":502,\"compute_resource_name\":\"galena\",\"title\":\"Brain extraction preview figures\"},{\"piping_id\":505,\"previous_piping_id\":503,\"compute_resource_name\":\"galena\",\"title\":\"N4 Bias Field Correction\",\"plugin_parameter_defaults\":[{\"name\":\"cpu_limit\",\"default\":\"12000m\"},{\"name\":\"memory_limit\",\"default\":\"12GiB\"}]},{\"piping_id\":506,\"previous_piping_id\":505,\"compute_resource_name\":\"galena\",\"title\":\"Automatic Fetal Brain Assessment\",\"plugin_parameter_defaults\":[{\"name\":\"inputPathFilter\",\"default\":\"*.nii\"},{\"name\":\"output_filename\",\"default\":\"quality_assessment.csv\"},{\"name\":\"crop_destination\",\"default\":\"\"},{\"name\":\"destination_folder\",\"default\":\"Best_Images_crop\"},{\"name\":\"threshold\",\"default\":0.4}]},{\"piping_id\":507,\"previous_piping_id\":506,\"compute_resource_name\":\"galena\",\"title\":\"Reconstruction\"},{\"piping_id\":508,\"previous_piping_id\":507,\"compute_resource_name\":\"galena\",\"title\":\"Brain reconstruction preview figures\"}]"}]}}'

Once again, CUBE was happy to create the workflow for me, but cpu_limit and memory_limit were ignored.