Developers' Corner

1. Standard variables

1.1 The NCO standards has defined a set of variables. We should not change the meaning and usage of these variables and should not create new variables for anything covered by the NCO standard variables.

1.2 NCO variables:

We try to understand these variables from our perspective. Correct us if not right

NET: A reserved variable in the NCO implementation standards, it is rrfs for the RRFS system and rtma for the 3DRTMA system.

MESH_NAME: different domain, grid configurations of a model, such as conus12km, conus3km, atl12km, etc. MESH_NAME is a keyword in rrfs-workflow, it is fixed but we have a few choices when setting up an experiment.

RUN: the top-level subcomponents of a system, for example, for GFS, RUN can be gfs, gdas, enkfgdas, ocean, etc. For RRFS, we may have rrfs, enkfrrfs, firewx, etc. This is set automatically in runtime, users don't configure it.

PDY: present day in YYYYMMDD format
cyc: cycle time in GMT hours, formatted HH

DATAROOT: Directory containing the working directory, typically $OPSROOT/tmp in production. This is usually set as the stmp directory in the development machines.

DATA: Location of the job working directory, typically $DATAROOT/$jobid. In the development configuration, DATA can be tweaked to other locations through util/sideload/launch.sh. But in J-job and ex-scripts, we should use this variable to access the current working directory for a task in runtime. DATA is removed immediately following the completion of a task in operation but at the development stage, users can set KEEPDATA=YES to keep DATA directories

HOMErrfs, USHrrfs, EXECrrfs, PARMrrfs, SCRIPTSrrfs, FIXrrfs: locations of the rrfs system directories

COMROOT: root directory for input/output data for the rrfs system.
COMIN: directory for the current model's input data, typically, $COMROOT/$NET/$model_ver/$RUN.$PDY
COMOUT: directory for current model's output data, typically, $COMROOT/$NET/$model_ver/$RUN.$PDY
COMINgfs: incoming data from GFS
COMINrap: incoming data from RAP
COMINgefs: incoming data from GEFS

KEEPDATA: specifies whether or not the working directory should be kept upon the completion of a job.

1.3 rrfs-workflow standard variables:

EXPDIR: The experiment directory which contains rrfs.xml, exp.setup and a config/ directory
VERSION: The system version number
MACHINE: The machine name, such as jet``hera, orion, hercules, all in lower cases
TAG: Right now, it is only used as a prefix to slurm job names.
CDATE: Current cycle in YYYYMMDDHH format
OBSINprepbufr: the location of incoming prepbufr files
ENS_INDEX: the ensemble index of a task. If this variable exists in runtime, it means this run is for one of the ensembles.

2. TRUE/FALSE, true/false, True/False and YES/NO

2.1 In the exp setup files and all config files, we follow the BASH rule to assign true/false to variables

eg:

export DO_IODA=true
export DO_JEDI=true

2.2 In the Python codes, we follow the Python rule to assign True/False to logic variables

eg:

 metatask = True

2.3 Use TRUE/FALSE when passing environmental variables from job cards (or rrfs.xml) to runtime scripts.

eg:

if os.getenv('DO_CHEMISTRY', 'FALSE').upper() == "TRUE":
    dcTaskEnv['USE_EXTERNAL_CHEM'] = os.getenv('USE_EXTERNAL_CHEM_ICS', 'FALSE').upper()
or
<envar><name>USE_EXTERNAL_CHEM</name><value>TRUE</value></envar>

2.4 YES/NO only applies to the KEEPDATA variable, which is an NCO rule.

2.5 In the bash scripts, we use ${var^^} to convert inputs to the upper case

We compare the converted upper cases to TRUE/FALSE or YES/NO. This makes the scripts more robust and is a preferred way in rrfs-workflow.

3. tips

3.1 export FCST_ONLY=${FCST_ONLY:-false}

If FCST_ONLY is exported before this line, it will take the predefined values; otherwise, it will take a default value of false.

3.2 Get PDY or cyc from CDATE

CDATE="2024052700"
PDY=${CDATE:0:8}  # eg: 20240527
cyc=${CDATE:8:2}  # eg: 00

3.3 Find previous or next few cycles - NDATE

NDATE should be used if one wants to find the previous or next few cycles. This is more robust than coding a date script individually.

source ${HOMErrfs}/workflow/ush/load_prod_util.sh
${NDATE} 003 2024052700  # result: 2024052703
${NDATE} -003 2024052700 # result: 2024052621

4. coordinating the operation part and the develop part

For differences between the operation and develop part, check this link

To accommodate those differences, the following measures are considered:

4.1. side loading for non-NCO tasks, such as clean, archive, graphics. They don't need J-jobs/ex-scripts and will be put under workflow/sideload.

4.2. The rocoto workflow management software does not provide some job card variables as ecflow. To compensate for this, a workflow/sideload/launch.sh script is added to mimic the ecflow behavior and provide a switch which routes a task to either J-jobs or non-NCO tasks

4.3 We use links to manage fix files (more detailed information here. In NCO implementation, do something as follows:
cp -rpL fix fix2; rm -rf fix; mv fix2 fix
will make a hard copy of fix files needed for operation

4.4. In order to separate concerns and only export required environmental variables for a task at runtime, a cascade config structure will be adopted. Resource configuration (such as ACCOUNT, QUEUE, PARTITION, RESERVATION, NODES, WALLTIME, NATIVE, MEMORY etc) are only needed in the experiment setup process and will be separated from the runtime configuration. More detailed information here.