Feature request: recognize requeued jobs in SLURM

[Hoeze]

On our SLURM cluster we have a preemptible lowprio partition.
It seems like Nextflow does not recognize job preemtions. What is the correct way to configure Nextflow here?

• clusterOptions = --requeue: Can this cause duplicated job submissions?
• clusterOptions = --no-requeue: The whole pipeline breaks all the time due to many job preemptions

[Hoeze]

I just checked the source code: Nextflow just assumes preempted jobs to have errored and also always adds --no-requeue to the sbatch command.

However, it would make more sense to add --requeue by default to all submissions and check whether the job is being cancelled or just restarted later.
Here is some python snippet on how to do that:

def run_scontrol_on_preempted_job(cluster, jobid):
    sctrl_res = sp.check_output(
        shlex.split(f"scontrol {cluster} -o show job {jobid}")
    )
    m = re.search(r"Requeue=(\w+)", sctrl_res.decode())
    requeueable = m.group(1)

    if requeueable == "1":
        job_state = "REQUEUED"
    else:
        job_state = "CANCELLED"

    return job_state

This would need to be put here:

nextflow/modules/nextflow/src/main/groovy/nextflow/executor/SlurmExecutor.groovy

Line 181 in 827ee97

result.put( cols[0], STATUS_MAP.get(cols[1]) )

Unfortunately I have no idea how to translate that snippet to Groovy.
Could someone from the Nextflow devs do that?

[bentsherman]

Hi @Hoeze , we probably aren't gonna get around to trying this anytime soon. Since you already have a code snippet in Python and you have a SLURM environment where you can test the behavior, your best bet is to try to implement this change in Nextflow and create a PR, demonstrating that it works in your environment.

I know you said you aren't familiar with Groovy, but you already found the correct spot in the Nextflow codebase to implement the change, and there is plenty of Groovy code in Nextflow that you can use as a reference.

To build and test Nextflow, all you need is Java, clone the Nextflow repo, make your changes, run make compile to build it, and use the launch.sh the same way you would use nextflow.

[pditommaso]

Nextlow adds --no-requeue by default, don't think it's possible to override it

nextflow/modules/nextflow/src/main/groovy/nextflow/executor/SlurmExecutor.groovy

Line 55 in 827ee97

result << '--no-requeue' << '' // note: directive need to be returned as pairs

[bentsherman]

I think he is saying that we should change the SLURM executor to use --requeue and either fail or restart a job based on its queue status from scontrol.

[bentsherman]

This issue has a long history: #226, #234, #3422

But to be honest, I'm having a hard time following these discussions. They talk about jobs that are suspended by SLURM due to preemption or node failure, so it seems like you would want to use --requeue instead.

[Hoeze]

@bentsherman is it possible to have some sidecar process similar to Snakemake's --cluster-sidecar?
https://github.com/gagneurlab/snakemake-slurm-template/blob/master/slurm-sidecar.py

• This script is extremely helpful on our cluster to reduce the load on the Slurmmaster while polling 1000s of jobs in parallel.
• Also, it correctly recognizes requeued jobs

[bentsherman]

Looks like it is doing something similar to what you proposed. But how does it help reduce the load on the SLURM scheduler? Is it just a higher polling interval? By default Nextflow checks the status of jobs once per minute.

[Hoeze]

Ah yes, indeed Nextflow also polls all jobs at once.
So the only difference is the recognition of requeued jobs.

[Hoeze]

Snakemake runs for every job a separate command to check the state, so I had to implement this cluster sidecar which one can query with a simple cURL command as often as Snakemake likes.

[sidevshiy]

@Hoeze Hi, Hoeze!
Did you manage to make your changes work ? I would be really glad if we can figure out somehow! Thank you!

[Hoeze]

Hi @sidevshiy, I did not investigate this further as I currently do not have a direct use case.
I might have a use case the next months which I can test this on, but if you need this now you should be fine implementing the changes I described here:
#3573 (comment)

[shellywanamaker]

Hi, recent updates to my cluster are preventing me from running jobs that specify --norequeue and I'm getting the following error message:

sbatch: error: When submitting a Checkpoint job, you cannot specify --no-requeue
sbatch: error: Batch job submission failed: Unspecified error

Is there a way to modify my .config file to remove this parameter from the sbatch scripts that nf-core pipelines generate?

#5276

[finchnSNPs]

I'm not a nextflow user so I'm not sure what it would take to implement such a solution, but adding

REQUEUE_COUNT=$(sacct -j "$SLURM_JOB_ID" --duplicates --noheader --format=State | grep -c REQUEUED)

# Exit the job if it has been requeued at least once
if [ "$REQUEUE_COUNT" -gt 0 ]; then
    echo "Job $SLURM_JOB_ID has been requeued $REQUEUE_COUNT time(s). Exiting."
    exit 0
fi

to the Slurm script will check if the job has been requeued and stop it before it starts. This rather than adding the --no-requeue parameter would be a more generalized solution. However, I'm gathering this is unreasonable to add to all the generated scripts manually, so I'm sure it is not a simple fix in the source.

[shellywanamaker]

From @CarsonJM:
A quick "hack" to workaround this for the time being would be to do include the following in your nextflow.config .
process {
clusterOptions = { "--requeue" }
errorStrategy = { task.exitStatus in ((130..145)) ? 'ignore' : 'retry' }
}
Basically this will include --requeue in the SLURM submission so that you can still use checkpoint queue. The caveat is that if a job is pre-empted, Nextflow will think that the job failed and will try to create a new job, but the SLURM scheduler will also resubmit the job (even though this won't be tracked by Nextflow), so what you'll end up with is duplication of jobs and a potentially very large overhead. So I added the errorStrategy line so that Nextflow will ignore pre-emptions, with the idea that I will just resume the pipeline to clean up tasks that were pre-empted.

I can report that this worked in my case.