azcopy creates directory on download failure which triggers further issue

[adamrtalbot]

Bug report

Based on comment: #5799 (comment)

Due to code in link it will then try to generate a folder that also crashs. It took a bit of time to find this bug :)

Expected behavior and actual behavior

We can probably be more intelligent about the difference between downloading a directory or a path here.

The current situation:

1. azcopy tries to download the input files
2. It fails, then creates a directory
3. It then tries to download a directory
4. It also fails

This obfuscates the failure in step 2 making it hard to debug.

Instead, we could handle downloading paths or directories better. This will probably require some investigation and/or an update to azcopy to see the behaviour but I think it should be possible.

Steps to reproduce the problem

TBC! But it should be possible by:

1. Set process.memory to 1GB
2. Try to download a large (>50GB) file
3. Watch the task on Azure Batch
4. Check the working directory.

Program output

🤷

Environment

• Nextflow version: [?]
• Java version: [?]
• Operating system: [macOS, Linux, etc]
• Bash version: (use the command $SHELL --version)

Additional context

Originally described by @KH-NN, any additional context would be helpful.

[malthesr]

To reproduce, allocate a large-ish file and transfer to Azure storage,

setup.sh

fallocate -l 50G test.file
azcopy cp test.file https://$AZURE_STORAGE_ACCOUNT.blob.core.windows.net/container/path/to/

Then, using the following workflow,

main.nf

process do_sth {
    conda "conda-forge::gawk"

    machineType "Standard_D16d_v5"
    memory 1.GB

    input:
    path(input)

    output:
    path("out.txt")

    """
    ls > out.txt
    """
}

workflow {
    do_sth(Channel.fromPath(params.input, checkIfExists: true, glob: false))
}

with config,

nextflow.config

plugins {
    id 'nf-azure@1.16.0'
}

env {
    AZCOPY_AUTO_LOGIN_TYPE = 'MSI'
}

executor {
    name = 'azurebatch'
}

wave {
    enabled = true
    strategy = ['conda']
}

azure {
    managedIdentity {
        system = true
    }

    storage {
        accountName = "$AZURE_STORAGE_ACCOUNT"
    }

    azcopy {
        // To make sure we're not hitting
        // https://github.com/nextflow-io/nextflow/issues/5797
        blockSize = 32
    }

    batch {
        location = "$AZURE_LOCATION"
        accountName = "$AZURE_BATCH_ACCOUNT"
        autoPoolMode = true
    }
}

fails on an invocation like,

run.sh

NXF_VER=25.04.2 NXF_DEBUG=1 \
    nextflow -C nextflow.config \
    run main.nf \
    -work-dir az://container/tmp \
    --input az://container/path/to/test.file

The .command.log file contains only:

mkdir: cannot create directory ‘test.file‘: File exists

However, the same thing works when changing to nf-azure@1.13. Likewise, it works on nf-azure@1.16 when increasing process.memory to e.g. 16.GB.

[adamrtalbot]

Thanks @malthesr, that's really helpful. The change is because previously there was no cap to azcopy memory usage and now there is a hard cap.

Could you add the environment variable AZCOPY_BUFFER_GB as per #6161 and see if it fixes the problem?

env {
    AZCOPY_AUTO_LOGIN_TYPE = 'MSI'
    AZCOPY_BUFFER_GB = 1
}

This will slow down the process, but it should improve reliability when running a high number of tasks on a single node.

[malthesr]

That does not appear to make a difference.

[malthesr]

However, this does work again when setting AZCOPY_BUFFER_GB = 0.1.

I wonder if this is because, as I understand it, nxf_parallel runs azcopy jobs in parallel? I.e. the correct value of AZCOPY_BUFFER_GB actually needs to be process.memory.toGiga() / [nxf_parallel jobs]?

EDIT: Although there's only really a single large file in play here, so for that explanation to make sense it would have to be the concurrent copying of test.file and the various (small) nextflow auxillary files.

[KH-NN]

Based on @malthesr comment, another thing to consider is if it is possible to set the ENV AZCOPY_BUFFER_GB as a closure based on the memory requirements for a given process. Currently all azcopy processes for all nextflow processes will be slow if we set ENV AZCOPY_BUFFER_GB to e.g. 1gb just because you have a single process that requires 1GB memory.

[adamrtalbot]

It should be possible to do this in the beforeScript, which is executed before staging the data.

So two workarounds right now:

• process.memory = null
• process.beforeScript = "export AZCOPY_BUFFER_GB=${task.memory.toGiga() / $n_files}" (work out n_files beforehand)

Thank you for your hard work!

[malthesr]

Thank you.

Just to emphasize my edit above, I think export AZCOPY_BUFFER_GB=${task.memory.toGiga() / $n_files} is not quite right, seeing as the MWE has only a single file (i.e. export AZCOPY_BUFFER_GB=1 ought to work and does not). It's possible that azcopy of the nextflow files (.command.* etc.) takes up non-negligible memory also, although I doubt that?

More generally, I think it'd be great to somehow improve on the failure mode around azcopy. @KH-NN and I were really led astray by the errors about attempting to copy directories, and spent a significant amount of time debugging as a result.

[adamrtalbot]

Agreed, I will continue discussing the memory issue on #6161

[adamrtalbot]

Playing around with this, if we bypass the error we end up with another problem where it creates a file then downloads the file to that dir:

We need to handle the difference between files and directories better. We might be able to use --as-subdir false somehow.

[adamrtalbot]

@malthesr and @KH-NN try #6187 as a solution for raising the error, you should see an appropriate error message when azcopy fails.