Mega-issue tracking all of the problems with running the "Fetal Brain MRI Reconstruction Pipeline"

[jennydaman]

The "Fetal Brain MRI Reconstruction Pipeline" is an important pipeline we run at BCH.

• https://chrisproject.org/blog/2023/11/14/fetal-reconstruction-pipeline
• https://github.com/FNNDSC/Fetal_Brain_MRI_Reconstruction_Pipeline

It works on our internal deployment of CUBE, but is suboptimal.

Feature Requests

This issue is an aggregation of all feature requests which would improve how the pipeline is used.

• It should be possible to set what compute environment is preferred for pipings of a pipeline.
• When creating a workflow of a pipeline, cpu_limit, gpu_limit, and memory_limit should be somehow configurable.
• A good solution for parallelism: every step before the last can be embarrassingly parallel, however there is no good solution for doing this in a pipeline.

UI requests

• Make it easier to select some (but not all) series of a study as the input for a new feed.

ML Issues

• Improve brain masking model. pl-emerald, the brain masking ChRIS plugin, is giving bad results which include a lot of non-brain tissue. Could this be related to different MRI acquisition parameters?

Deployment Considerations

Container "cold boot" problem: pulling container images which have large files (such as ML weights) takes a long time. Possible solutions:

• Use SLURM+E3, which uses Apptainer images from the NFS
• Set up an internal container registry cache for Kubernetes
• Confine the jobs to a single machine

The CPU speed varies wildly across nodes of our Kubernetes cluster. E.g. to run https://github.com/fnndsc/pl-emerald on CPU (not GPU) my computer takes 5-10s per stack. Our Kubernetes nodes take anywhere between 60-300s per stack.

[rudolphpienaar]

Another issue:

• End-to-End, i.e. DICOM-to-DICOM and transmission of final recon to clinical PACS