Optional arguments

[AndersJensen-NOAA]

I have 7 optional arguments in tempo version 3.

     real(kind_phys), optional, intent(inout) :: nc(:,:)
     real(kind_phys), optional, intent(inout) :: nwfa(:,:)
     real(kind_phys), optional, intent(inout) :: nifa(:,:)
     real(kind_phys), optional, intent(in   ) :: nwfa2d(:)
     real(kind_phys), optional, intent(in   ) :: nifa2d(:)
     real(kind_phys), optional, intent(inout) :: ng(:,:)
     real(kind_phys), optional, intent(inout) :: volg(:,:)

The first 5 are controlled by ltaerosol and the next two are controlled by lthailaware. The two variable controlled by lthailaware are graupel number (ng) and graupel volume (volg). These are declared as optional in the meta data file as well.

When both ltaerosol are lthailaware are set to false and I add these print statements to tempo_run:

     if (present(nwfa2d)) write(*,*) 'nwfa2d present'
     if (present(nifa2d)) write(*,*) 'nifa2d present'
     if (present(nwfa)) write(*,*) 'nwfa present'
     if (present(nifa)) write(*,*) 'nifa present'
     if (present(nc)) write(*,*) 'nc present'
     if (present(ng)) write(*,*) 'ng present'
     if (present(volg)) write(*,*) 'volg present'

i get multiple prints of 'ng present' in the log file. note that ntgnc = -99 in the log file and graupel number is not in the field table.

Here's the build location for the caps: /scratch4/BMC/wrfruc/jensen/ufs_tempo_dev/build/UFSATM/ccpp/physics

[climbfuji]

Which compiler/version?

[AndersJensen-NOAA]

Which compiler/version?

Currently Loaded Modules:
  1) intel-oneapi-compilers/2024.2.1  15) snappy/1.2.1            29) py-mpi4py/4.0.1            43) ip/5.1.0
  2) stack-oneapi/2024.2.1            16) zstd/1.5.6              30) intel-oneapi-mkl/2024.2.1  44) sp/2.5.0
  3) intel-oneapi-mpi/2021.13.1       17) c-blosc/1.21.5          31) py-numpy/1.26.4            45) w3emc/2.10.0
  4) stack-intel-oneapi-mpi/2021.13   18) netcdf-c/4.9.2          32) py-pip/23.1.2              46) gftl/1.14.0
  5) glibc/2.34                       19) netcdf-fortran/4.6.1    33) libyaml/0.2.5              47) gftl-shared/1.9.0
  6) intel-oneapi-runtime/2024.2.1    20) parallel-netcdf/1.12.3  34) py-pyyaml/6.0.2            48) fargparse/1.8.0
  7) nghttp2/1.63.0                   21) parallelio/2.6.2        35) esmf/8.8.0                 49) yafyaml/1.4.0
  8) zlib/1.2.13                      22) pigz/2.8                36) fms/2024.02                50) pflogger/1.15.0
  9) curl/8.10.1                      23) tar/1.34                37) bacio/2.4.1                51) udunits/2.2.28
 10) cmake/3.27.9                     24) gettext/0.22.5          38) crtm-fix/2.4.0.1_emc       52) mapl/2.53.4-esmf-8.8.0
 11) libjpeg/2.1.0                    25) sqlite/3.46.0           39) git-lfs/3.5.1              53) scotch/7.0.4
 12) jasper/2.0.32                    26) util-linux-uuid/2.40.2  40) crtm/2.4.0.1               54) ufs_common
 13) libpng/1.6.37                    27) python/3.11.7           41) g2/3.5.1                   55) nccmp/1.9.1.0
 14) hdf5/1.14.3                      28) python-venv/1.0         42) g2tmpl/1.13.0              56) ufs_ursa.intelllvm

[climbfuji]

I've seen something similar in NEPTUNE with certain versions of the oneAPI compilers. It only happened when the code was optimized. That's obviously scary, but I wasn't able to reproduce it with later versions of the compiler and therefore Intel won't be able to fix it.

Can you try this, please?

     real(kind_phys), volatile, optional, intent(inout) :: nc(:,:)
     real(kind_phys), volatile, optional, intent(inout) :: nwfa(:,:)
     real(kind_phys), volatile, optional, intent(inout) :: nifa(:,:)
     real(kind_phys), volatile, optional, intent(in   ) :: nwfa2d(:)
     real(kind_phys), volatile, optional, intent(in   ) :: nifa2d(:)
     real(kind_phys), volatile, optional, intent(inout) :: ng(:,:)
     real(kind_phys), volatile, optional, intent(inout) :: volg(:,:)

Obviously, that's not ideal, but if it works then you can add a note with a link to the issue and a reminder to try again after updating to 2025.3.x or 2026.0.0 with spack-stack-2.x.y.

[AndersJensen-NOAA]

@climbfuji
I'm still working out a few small connection issues with debug mode turned on. But for my latest test, using the volatile attribute does not fix the issue.

[dustinswales]

@AndersJensen-NOAA Any optional Scheme variable needs to have an active attribute in the Host metadata. Otherwise, the CCPP framework will assume this variable is always allocated, and won't create a local pointer for temporary assignment within the Cap.

Below (snippet from call to tempo_run in your Cap) ng is optional to the Scheme, but since there is not an active attribute, it is assumed to always provided by the Host. Whereas, volg is defined with an active attribute, an the local pointer assignment works as expected.
ng=gfs_stateout%gq0(gfs_control%chunk_begin(cdata%chunk_no):gfs_control%chunk_end(cdata%chunk_no),one:gfs_control%levs,gfs_control%ntgnc), &
volg=volg_38_ptr_array(cdata%thrd_no)%p,prcp=gfs_interstitial(cdata%thrd_no)%prcpmp(gfs_control%chunk_begin(cdata%chunk_no):gfs_control%chunk_end(cdata%chunk_no)), &

Add the following to the declaration of mass_number_concentration_of_graupel_of_new_state in GFS_typedefs.meta
active = (index_of_graupel_mixing_ratio_in_tracer_concentration_array > 0)

Also, index_of_graupel_mixing_ratio_in_tracer_concentration_array is set based on the contents of the field_table (i.e. it controls the allocation status )

[AndersJensen-NOAA]

Thank you @dustinswales -- this works. I will add to change to the tempo v3 PR: #353

[climbfuji]

@AndersJensen-NOAA Any optional Scheme variable needs to have an active attribute in the Host metadata. Otherwise, the CCPP framework will assume this variable is always allocated, and won't create a local pointer for temporary assignment within the Cap.

Below (snippet from call to tempo_run in your Cap) ng is optional to the Scheme, but since there is not an active attribute, it is assumed to always provided by the Host. Whereas, volg is defined with an active attribute, an the local pointer assignment works as expected. ng=gfs_stateout%gq0(gfs_control%chunk_begin(cdata%chunk_no):gfs_control%chunk_end(cdata%chunk_no),one:gfs_control%levs,gfs_control%ntgnc), & volg=volg_38_ptr_array(cdata%thrd_no)%p,prcp=gfs_interstitial(cdata%thrd_no)%prcpmp(gfs_control%chunk_begin(cdata%chunk_no):gfs_control%chunk_end(cdata%chunk_no)), &

Add the following to the declaration of mass_number_concentration_of_graupel_of_new_state in GFS_typedefs.meta active = (index_of_graupel_mixing_ratio_in_tracer_concentration_array > 0)

Also, index_of_graupel_mixing_ratio_in_tracer_concentration_array is set based on the contents of the field_table (i.e. it controls the allocation status )

Good finding. Glad it's not a compiler bug that we have to code around (as it was the case in NEPTUNE).

[dustinswales]

@climbfuji In Capgen we should pivot with just using if (allocated(Var)) instead of if (conditional) to assign the local pointers in the Caps.
There's no mechanism in the CCPP to compare the active attribute in the Host model metadata to the source code conditional, leading to inconsistencies like we get here.