CWLProv: add option exclude raw copies of the data

[mr-c]

https://matrix.to/#/!RQMxrGNGkeDmWHOaEs:gitter.im/$AJGFCdt6jVAn3aR5lQ0PK3_0SGgvFrubf5SMClsOgGA (a.k.a https://gitter.im/common-workflow-language/common-workflow-language?at=61d6a7bfbfe2f54b2e04661d )

• --prov-exclude-inputs Skips copying the input files into the CWLProv ResearchObject
• --prov-exclude-intermediates Skips copying the intermediate files into the CWLProv ResearchObject
• --prov-exclude-outputs Skips copying the output files into the CWLProv ResearchObject

[jjkoehorst]

Thanks for creating the ticket for me personally only the metadata (rdf files, workflows files) are needed. As the input and output files are preserved on a cloud store.

[mr-c]

Areas to investigate, (add flag to skip the copying, but still calculate and store the checksums)

cwltool/cwltool/provenance.py

Line 790 in a1e3449

def add_data_file(

cwltool/cwltool/provenance.py

Line 929 in a1e3449

def _relativise_files(

cwltool/cwltool/provenance.py

Line 741 in a1e3449

def generate_snapshot(self, prov_dep: CWLObjectType) -> None:

called from

cwltool/cwltool/main.py

Line 1413 in a1e3449

research_obj.generate_snapshot(

cwltool/cwltool/provenance.py

Line 894 in a1e3449

def create_job(

[jjkoehorst]

To update this, when providing Directory or Files as input it will copy the entire content to /tmp. Solution for now is to use Strings instead of Directory when possible.

[mr-c]

Solution for now is to use Strings instead of Directory when possible.

FYI, while that may work for now, that will break mulit-node execution of the workflow