OSError: [Errno 18] Invalid cross-device link error in move os.rename(src, real_dst)

[AriBad]

Hi, I use cwltool to run a pipeline and I sometimes have an error when I run it. The error is : OSError: [Errno 18] Invalid cross-device link and File "/path/to/.conda/envs/name_env/lib/python3.7/shutil.py", line 566, in move os.rename(src, real_dst)
I think the error is the same as here and the problem is the using of os.rename instead of shutil.move().
This error appears at the end of the pipeline when cwltool moves temp files into local.
Have you ever encountered this error ? If yes, is there a way to troubleshoot it ?

---

Expected Behavior

no error

Actual Behavior

OSError: [Errno 18] Invalid cross-device link and File "/path/to/.conda/envs/name_env/lib/python3.7/shutil.py", line 566, in move os.rename(src, real_dst)

Full Traceback

^[[1;30mERROR^[[0m ^[[31mUnhandled error:
  [Errno 2] No such file or directory: '/path/bins_directory/reads_full_pe_A.fq_1_bins_metabat.1.fa'^[[0m
Traceback (most recent call last):
  File "/path/to/.conda/envs/env_conda/lib/python3.7/shutil.py", line 566, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/npmifkcp/MetaBat/reads_full_pe_A.fq_1_bins_metabat.1.fa' -> '/path/bins_directory/reads_full_pe_A.fq_1_bins_metabat.1.fa'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/.local/lib/python3.7/site-packages/cwltool/main.py", line 1249, in main
    tool, initialized_job_order_object, runtimeContext, logger=_logger
  File "/home/.local/lib/python3.7/site-packages/cwltool/executors.py", line 59, in __call__
    return self.execute(process, job_order_object, runtime_context, logger)
  File "/home/.local/lib/python3.7/site-packages/cwltool/executors.py", line 162, in execute
    path_mapper=runtime_context.path_mapper,
  File "/home/.local/lib/python3.7/site-packages/cwltool/process.py", line 400, in relocateOutputs
    stage_files(pm, stage_func=_relocate, symlink=False, fix_conflicts=True)
  File "/home/.local/lib/python3.7/site-packages/cwltool/process.py", line 296, in stage_files
    stage_func(entry.resolved, entry.target)
  File "/home/.local/lib/python3.7/site-packages/cwltool/process.py", line 371, in _relocate
    _relocate(dir_entry.path, fs_access.join(dst, dir_entry.name))
  File "/home/.local/lib/python3.7/site-packages/cwltool/process.py", line 373, in _relocate
    shutil.move(src, dst)
  File "/home/.conda/envs/name_env/lib/python3.7/shutil.py", line 580, in move
    copy_function(src, real_dst)
  File "/home/.conda/envs/name_env/lib/python3.7/shutil.py", line 266, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/.conda/envs/name_env/lib/python3.7/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/path/bins_directory/reads_full_pe_A.fq_1_bins_metabat.1.fa'

Your Environment

• cwltool version: 3.1.20210816212154

[mr-c]

Hello @AriBad ; can you re-run with --debug so I can see which part of cwltool lead to that error? Alas the stack trace shown is from a follow-on error, not the original error.

Checking the latest version of cwltool, th only calls to os.rename are from the --provenance feature; are you using that?

Also, I would recommend upgrading your cwltool to a newer version as well. Conda has version 3.1.20220224085855 for example: https://anaconda.org/conda-forge/cwltool

[mr-c]

a workaround would be to set TMPDIR to a path on the same device as /path/bins_directory/reads_full_pe_A.fq_1_bins_metabat.1.fa

TMPDIR=/path/tmp cwltool ….

[AriBad]

Hi, sorry for the late reply. Thanks for your reply. Indeed, by precising the path of TMP dir allow to not have the error with os.rename. However. the second part of the error is already here :

[1;30mINFO�[0m [step individual_metagenopic] completed success
[1;30mINFO�[0m [workflow ] completed success
[1;30mDEBUG�[0m �[32m[workflow ] outputs {
[...]
}
 [Errno 2] No such file or directory: '/calrissian/output-data/output_Assembly/reads_full_pe_A.fq_1_output_megahit/reads_full_pe_A.fq_1_megahit_final.contigs.fa'�[0m
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/cwltool/main.py", line 1152, in main
    tool, initialized_job_order_object, runtimeContext, logger=_logger
  File "/usr/local/lib/python3.7/site-packages/cwltool/executors.py", line 60, in __call__
    return self.execute(process, job_order_object, runtime_context, logger)
  File "/usr/local/lib/python3.7/site-packages/cwltool/executors.py", line 163, in execute
    path_mapper=runtime_context.path_mapper,
  File "/usr/local/lib/python3.7/site-packages/cwltool/process.py", line 425, in relocateOutputs
    outputObj, ("File",), functools.partial(compute_checksums, fs_access)
  File "/usr/local/lib/python3.7/site-packages/cwltool/utils.py", line 295, in visit_class
    visit_class(rec[d], cls, op)
  File "/usr/local/lib/python3.7/site-packages/cwltool/utils.py", line 298, in visit_class
    visit_class(d, cls, op)
  File "/usr/local/lib/python3.7/site-packages/cwltool/utils.py", line 295, in visit_class
    visit_class(rec[d], cls, op)
  File "/usr/local/lib/python3.7/site-packages/cwltool/utils.py", line 298, in visit_class
    visit_class(d, cls, op)
  File "/usr/local/lib/python3.7/site-packages/cwltool/utils.py", line 295, in visit_class
    visit_class(rec[d], cls, op)
  File "/usr/local/lib/python3.7/site-packages/cwltool/utils.py", line 298, in visit_class
    visit_class(d, cls, op)
  File "/usr/local/lib/python3.7/site-packages/cwltool/utils.py", line 293, in visit_class
    op(rec)
  File "/usr/local/lib/python3.7/site-packages/cwltool/process.py", line 1281, in compute_checksums
    with fs_access.open(location, "rb") as f:
  File "/usr/local/lib/python3.7/site-packages/cwltool/stdfsaccess.py", line 42, in open
    return open(self._abs(fn), mode)
FileNotFoundError: [Errno 2] No such file or directory: '/calrissian/output-data/output_Assembly/reads_full_pe_A.fq_1_output_megahit/reads_full_pe_A.fq_1_megahit_final.contigs.fa'
[...]

I have different using of my pipeline with the using of scatter at a step or not. And depending on the scenario of the pipeline that I choose, the error is present or not. I think the error come when cwltool try to copy file in tmp dir in output dir. As The error appear in some cases and not in another I am not be able to give you a toy script to show you the problem because I don't understand the cause of it. Maybe it is because I have many output to copy.

(Here the log is little different because I have used cwltool with calrissian on k8s. But the error is the same. )