GA4GH TRS: linked files fail to resolve

[keiranmraine]

Expected Behavior

Template generation or execution should be successful

Actual Behavior

$ cwltool --make-template quay.io/wtsicgp/dockstore-cgpmap:3.0.0
/home/kr2/.local/bin/cwltool 1.0.20181217162649
Resolved 'quay.io/wtsicgp/dockstore-cgpmap:3.0.0' to 'https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor'
Tool definition failed initialization:
https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor:21:5: ('https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml', HTTPError('404 Client Error: Not Found for url: https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml',))

Workflow Code

Commands instead:

Fails:

cwltool --make-template quay.io/wtsicgp/dockstore-cgpmap:3.0.0

Success with a full URL (no GA4GH TRS):

$ cwltool --make-template https://dockstore.org:443/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor/%2FDockstore.cwl
/home/kr2/.local/bin/cwltool 1.0.20181217162649
sample: a_string  # type "string"
reference:  # type "File"
    class: File
    path: a/file/path
mmqcfrac: 0.05  # default value of type "float". (optional)
mmqc: false  # type "boolean"
groupinfo:  # type "File" (optional)
    class: File
    path: a/file/path
bwa_idx:  # type "File"
    class: File
    path: a/file/path
bwa: ' -Y -K 100000000'  # default value of type "string". (optional)
bams_in:  # array of type "File" (optional)
  - class: File
    path: a/file/path

Full Traceback

Traceback (most recent call last):
  File "/home/kr2/.local/lib/python3.6/site-packages/schema_salad/ref_resolver.py", line 143, in fetch_text
    resp.raise_for_status()
  File "/home/kr2/.local/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kr2/.local/lib/python3.6/site-packages/schema_salad/ref_resolver.py", line 590, in resolve_ref
    doc = self.fetch(doc_url, inject_ids=(not mixin))
  File "/home/kr2/.local/lib/python3.6/site-packages/schema_salad/ref_resolver.py", line 955, in fetch
    text = self.fetch_text(url)
  File "/home/kr2/.local/lib/python3.6/site-packages/schema_salad/ref_resolver.py", line 145, in fetch_text
    raise RuntimeError(url, e)
RuntimeError: ('https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml', HTTPError('404 Client Error: Not Found for url: https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kr2/.local/lib/python3.6/site-packages/cwltool/main.py", line 661, in main
    do_validate=loadingContext.do_validate)
  File "/home/kr2/.local/lib/python3.6/site-packages/cwltool/load_tool.py", line 284, in validate_document
    workflowobj, fileuri, checklinks=do_validate)
  File "/home/kr2/.local/lib/python3.6/site-packages/schema_salad/ref_resolver.py", line 896, in resolve_all
    val, base_url+subscope, file_base=file_base, checklinks=False)
  File "/home/kr2/.local/lib/python3.6/site-packages/schema_salad/ref_resolver.py", line 909, in resolve_all
    val, base_url=file_base, checklinks=False)
  File "/home/kr2/.local/lib/python3.6/site-packages/schema_salad/ref_resolver.py", line 590, in resolve_ref
    doc = self.fetch(doc_url, inject_ids=(not mixin))
  File "/home/kr2/.local/lib/python3.6/site-packages/schema_salad/sourceline.py", line 168, in __exit__
    raise self.makeError(six.text_type(exc_value))
RuntimeError: https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/descriptor:21:5: ('https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml', HTTPError('404 Client Error: Not Found for url: https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fwtsicgp%2Fdockstore-cgpmap/versions/3.0.0/plain-CWL/cwls/mixins/requirements.yml',))

Your Environment

$ cwltool --version
/home/kr2/.local/bin/cwltool 1.0.20181217162649

[mr-c]

I made a few edits, the issue is with using a GA4GH TRS reference vs. the full URL; not cwltool vs. cwl-runner (which, in this case, is the same program).