Skip to content

Queue adapters and executable templates

Jump to bottom
vingman edited this page Apr 27, 2021 · 6 revisions

Table of Contents

Queue adapters

These are templates that FireWorks uses to request appropriate resources when submitting jobs to the queue. Adapters for various queuing systems are found in the AaronJr/templates directory, with names of the form QUEUETYPE_qadpter.template. However, these should be checked for accuracy and any missing job script directives. Since we do not have access to every queuing system for testing, we would greatly appreciate any errors found in the template files provided by AaronJr be reported through the Issues tab on GitHub. If you need to edit the queue adapter template, you may copy the file to your $AARONLIB/templates folder and make any necessary changes there.

Using the configuration files to define template variables

The appropriate queue adapter template file is chosen based on the value of the queue_type option in the [HPC] section of your configuration file. For example: queue_type=SLURM will tell AaronJr to look for a queue adapter template file named SLURM_qadapter.template in first your $AARONLIB/templates directory then, if not found, in the AaronJr/templates directory.

The queue adapter template contains variables of the form $${option_name} that will be filled in using options in the [Job] section of the configuration file. An exception to this is the $${job_name} option, which will be automatically generated by AaronJr for you. See the [Job] section configuration help for commonly used options. If you need to use additional options not listed in the help page, AaronJr will automatically use the corresponding option=value pairs defined in your [Job] section to fill out the template.

The qadapter_options.py command line script can be used to both ensure AaronJr will use the correct template as well as to list the options found within that should be set by the user in their configuration's [Job] section: 

$ qadapter_options.py NotAQueueType
Could not find NotAQueueType_qadapter.template in $AARONLIB/templates or AaronJr/templates
$ qadapter_options.py SLURM
Using $AARONLIB/templates/SLURM_qadapter.template
[Job] options found:
  queue
  nodes
  ppn
  wall
  memory

Example

The SLURM_qadapter.template is as follows: 

#!/bin/bash
#SBATCH --partition=$${queue}
#SBATCH --job-name=$${job_name}
#SBATCH --ntasks=$${nodes}
#SBATCH --cpus-per-task=$${ppn}
#SBATCH --time=$${wall}:00:00
#SBATCH --mem=$${memory}

cd $${launch_dir}
$${rocket_launch}
Thus, the [HPC] and [Job] sections should look something like this: 
[HPC]
queue_type = SLURM

[Job]
queue = batch
nodes = 1
ppn = 6
wall = 12
memory = 12GB

Using default values

Default values for these options can be set in the user's default configuration $AARONLIB/config.ini which can be overridden by options in the project configuration file. Function parsing is also available (see here for more info).

Example

Snippet from $AARONLIB/config.ini: 

[Job]
queue = batch
nodes = 1
wall = 12
memory = %{ $ppn * $nodes * 2 }GB
Snippet from small_job.ini (will set memory = 2GB): 
[Job]
ppn = 1
wall = 2
Snippet from large_job.ini (will set memory = 96GB): 
[Job]
nodes = 2
ppn = 24
wall = 24

Leaving out unneeded values

It is best to go ahead and put additional job script directives that you may not always use in the queue adapter template, rather than creating a whole new queue adapter template. If the option is left undefined (i.e.: it is not in your configuration files), the line for that job script directive is removed from the parsed template. For example, say you want to receive a notification from the SLURM queue whenever jobs finish or error out, but only for certain projects. These lines would be added to SLURM_qadapter.template: 

#SBATCH --mail-type=$${mail_type}
#SBATCH --mail_user=$${mail_user}
Now, for projects where you DO want to receive these notifications, simply define mail_type=END,FAIL and [email protected] in the [Job] section of the configuration file. For projects where you DO NOT want to receive these notifications, leave the mail_type and mail_user options out of the configuration file entirely (this means they should also not be set in your $AARONLIB/config.ini either).

Executable templates

These are used to build the scripts that will be executed when the job is running. The syntax is a little different from the queue adapter templates; the variables are of the form {{option_name}}, and missing options will cause an error to be raised.

Options to be set in your configuration file

[HPC]

  • work_dir - This is the parent directory on the HPC that AaronJr will use to store input/output files. AaronJr may create sub-directories in here for organizational purposes, the names of which will be automatically generated.
  • scratch_dir - This is the parent directory for storing temporary files created during execution of a job. AaronJr will make a sub-directory for each job, the names of which will be automatically generated.
[Job]
  • exec_type - This is used to find the appropriate executable template, named EXECTYPE.template in either $AARONLIB/templates or AaronJr/templates (e.g.: exec_type = gaussian --> AaronJr/templates/gaussian.template)
  • exec_memory - (Gaussian only) Defaults to the value of memory in the [Job] section, but we have noticed Gaussian does not alway abide by this limit, which can lead to memory allocation errors. We suggest using exec_memory = %{ $memory*8//10 } so that Gaussian is limited to ~80% the total memory requested.
Additional options may be pulled from the [Job] or [HPC] section, if present in the template. For example, many computational softwares need to be told how many parallel processes they have access to, so procs may be pulled in.

Options automatically generated by AaronJr

All

  • job_name - Generated using the structure template name and the name option set in the default/global section of the configuration file
Crest and xtb
  • cmdline - Generated using options found in the [Theory] section. You can also append your own command line options to the automatically generated string by setting this option in your configuration file.
  • optts - Set to True if [Job]type indicates a transition state should be found, False otherwise. Currently a "dummy value", since xtb does not yet support TS optimizations.
Clone this wiki locally