snakemake 6.8.0 → CWL

[mr-c]

From @sofiakarb

The issue I am currenlty facing has to do with Snakemake (v:6.8.0) - cwltool (v:3.1.20210521105815).
More specifically, my Snakefile is the following:

from snakemake.remote.HTTP import RemoteProvider as HTTPRemoteProvider
HTTP = HTTPRemoteProvider()

rule workflow:
    input:
        input_file=HTTP.remote("remote_link", keep_local=True)
    params:
        channels=[0, 1, 2, 3, 4],
        psd_output_file_name="psd.json",
        output_file_name="output.png"
    output:
        output="output.png"
    cwl:
        "file:workflow.cwl"

While executing snakemake -c 1, I get an error related to cwltool :

INFO /../bin/cwltool 3.1.20210521105815
ERROR Workflow error, try again with --debug for more information:
Invalid job input record:
../../../../tmp/tmpw06vyrgf:1:2: the `input_file` field is not valid because
                                   is not a dict

This is because .yml file that holds the information about Class: File is skipped due to the fact that Snakemake only needs the Snakefile (in which all input, parameters, outputs and cwl workflow file are provided). I can see two solutions, one from Snakemake side, where I must specificly provide the Class:File in the input_file field, and the other solution having a way cwltool to not provide errors (maybe just a warning?)

Thank you beforehand for your support.

Originally posted by @sofiakarb in #758 (comment)

[mr-c]

@sofiakarb Can you share the CWL document that Snakemake produces? I get errors when I run snakemake -c 1 with your snakefile, so I think I'm missing some details

[sofiakarb]

Sorry @mr-c , I think I forgot to change the remote_link in my Snakefile:

from snakemake.remote.HTTP import RemoteProvider as HTTPRemoteProvider
HTTP = HTTPRemoteProvider()

rule workflow:
    input:
        input_file=HTTP.remote("https://object.cscs.ch/v1/AUTH_25b4e28a742d4987a7b6f84c0c36512e/hbp-01852/data-originals/sub-1/hbp-01852_sub-1_chs-32_hem-RH_ana-ISO000_stim-SPN.smr", keep_local=True)
    params:
        channels=[0, 1, 2, 3, 4],
        psd_output_file_name="psd.json",
        output_file_name="output.png"
    output:
        output="output.png"
    cwl:
        "file:workflow.cwl"

Can you please try again?
I am not sure which is the "CWL document that Snakemake produces", are you referring to the
snakemake --export-cwl ?

[sofiakarb]

A solution (temporal?) that worked for us is the following:

rule workflow:
    input:
    params:
        input_file={"class": "File", "path": "https://object.cscs.ch/v1/AUTH_25b4e28a742d4987a7b6f84c0c36512e/hbp-01852/data-originals/sub-1/hbp-01852_sub-1_chs-32_hem-RH_ana-ISO000_stim-SPN.smr"},
        channels=[0, 1, 2, 3, 4],
        psd_output_file_name="psd.json",
        output_file_name="output.png"
    output:
        output="output.png"
    cwl:
        "file:workflow.cwl"

We removed the HTTPRemoteProvider, and placed everything inside the params: (including the input_file). This remote location is then taking care of by the cwltool which also sees the correct Class:File and stops the error.

[mr-c]

A solution (temporal?) that worked for us is the following:

Great! Is this not a valid snakefile? Perhaps it needs to be documented on their side. Or snakemake updated to do the class: File clarification itself.

I am not sure which is the "CWL document that Snakemake produces", are you referring to the
snakemake --export-cwl ?

I was trying to figure out the cwltool connection. I see now that you are calling another CWL workflow from this snakemake rule.

I can't follow along further as I lack your workflow.cwl

[Errno 2] No such file or directory: '/home/michael/src/snakemake/1555/workflow.cwl'

[sofiakarb]

Dear @mr-c, sorry for the late response. Our workflow definition is under an internal gitlab repo so it will not be possible for you to reproduce it -for now. I will try to find some time to work an a simpler example and let you know. For now the solution previously shared seems to work fine