Enable SPPT with the MPAS dycore#97
Conversation
changes to build stochastic physics with mpas
…osphere model. This implementation assumes that each MPI task has only one block.
…ption to transform the pattern to a bounded interval [-1.0, 1.0] , and to include a shift of the pattern’s mean from 0.0 to 1.0. (These two computations were previously excluded for debugging purposes.)
…ted and no perturbation is applied. Added sanity check for SPPT namelist variables; if their values are not set correctly, reset do_sppt to false. Changed module name stochastic_physics_mpas to mpas_stochastic_physics.
…PT implementation. If SPPTINT = 0 perturbation pattern advances every time step; if SPPTINT = t > 0 pattern advances after t seconds.
The tapering is specified with two namelist variables, config_sppt_hgt_top1 and config_sppt_hgt_top2; the default values for the two variables are 15000 and 27000 meters respectively. Fixed some logics for the SPPT seeds generation.
…esolution exceeds the default value.
…ong with the Registry file that defines the namelist variables, was updated to allow for reading 8-byte integer values of namelist variables that are provided as character strings. Replace print statements with mpas_log_write, and wrap debug/diag print statements with #ifdef directives.
… merge master in.
…e master branch in.
…asier to merge master branch in.
… the "gsl/MPAS_stoch_physics_try_merge_stoch_master" branch. Details: * Add new variable declarations needed in latest master branch. * Remove some unneeded variable declarations. * Add new subroutines needed in latest master branch. * Add new arguments to existing subroutines and/or functions (some optional). * Add some code modifications in if-statements required by latest "master" branch. * Add new code for new optional argument "normalize". * Limit kmax in k-loop for sppt tapering in PBL. * Bug fix(es) from master branch. * Add new file macros.h that defines preprocessor macros using macros defined via the -DMY_MACRO flag to the CPPFLAGS variable in the makefile of MPAS-Model.
… as the dycore; add back variable declarations that were accidentally removed during the merge of the master branch.
… only if do_sppt is .true.; improve messages during application of SPPT perturbations.
… SPPT may perturb; clarify log messages.
…re Fortran 2008 is the standard" (from Dustin; not sure what exactly is failing), and hard-code to always use mpi_f08.
…in the fixes_for_uwm branch (need to verify with Dustin).
There was a problem hiding this comment.
It is my understanding that MPAS uses ESMF, why does this code use the non-ESMF version of the matrix multiplications?
There was a problem hiding this comment.
@NingWang325, please see Phil's question above.
There was a problem hiding this comment.
@pjpegion As far as I can tell, the ufs-community's stand-alone MPAS-Model currently does not support most ESMF routines. It seems to support only a few ESMF timing routines. A few days ago I tried building stochastic_physics under MPAS-Model with esmf_dgemm instead of dgemm, and the build failed.
We may have to switch to esmf_dgemm here anyway to get this PR merged since I think the UFS Weather Model does not support dgemm. Then, after this PR is merged and until the stand-alone MPAS-Model gets support for esmf_dgemm or some other solution is found, running stochastic_physics with MPAS-Model will first require a local change (e.g. in a separate fork and branch) to the master branch of this repo so that dgemm is used instead of esmf_dgemm.
@dustinswales probably has more insight into this.
There was a problem hiding this comment.
@gsketefian If MPAS-A can't use esmf_dgemm, could you define CESMCOUPLED in MPAS-A to use the Fortran native dgemm?
Adding || defined(MPAS) to the conditional is problematic in the UWM, since MPAS and CESMCOUPLED are conflicting.
There was a problem hiding this comment.
@dustinswales Yes, I think defining CESMCOUPLED (along with defining MPAS) during the MPAS-A build would work. Of course it would be clearer to just have the one macro (MPAS) that does it all.
Can you explain what you mean by MPAS and CESMCOUPLED being conflicting in the UWM?
There was a problem hiding this comment.
@gsketefian
#if defined(CESMCOUPLED) || defined(MPAS) does not allow for the UWM to use esmf_dgemm with MPAS dynamical core.
If MPAS is defined, but CESMCOUPLED is not, this will use dgemm, not esmf_dgemm
-DMPAS is conflicting in that it means slightly different things in this context for MPAS-in-UFS vs. MPAS-A.
In MPAS-A, you are trying to use-DMPAS to trigger the use of dgemm.
But in UFS-MPAS, we already have -DCESMCOUPLED for this purpose, and esmf_dgemm is the default for FV3/MPAS/CDEPS.
There was a problem hiding this comment.
@dustinswales Ok, I understand what you're trying to do with -DMPAS in the UFS-MPAS. I'll change all the #if defined(CESMCOUPLED) || defined(MPAS) back to #ifdef CESMCOUPLED, try building with -DCESMCOUPLED in MPAS-A, and if everything works, will push the changes.
I do have a related question. I didn't make the connection earlier, but apparently NCAR's Community Earth System Model (CESM) is based on the Earth System Modeling Framework (ESMF). Given that, why would a macro named CESMCOUPLED imply using the non-esmf version of the matrix multiplication routine, i.e. dgemm, instead of the esmf version (esmf_dgemm)? Seems backwards.
There was a problem hiding this comment.
@gsketefian I'm equally confused. I have no clue of where this backwards name came from.
There was a problem hiding this comment.
@dustinswales I cloned the ufs-weather-model app and, with claude's help, got a decent explanation of CESMCOUPLED:
The CESMCOUPLED macro answers the question, "Is this code being compiled as part of a full CESM build, or as a standalone component (e.g., in UFS)?" In a standalone/UFS build, ESMF is the host framework and provides its own BLAS wrapper (esmf_dgemm); in a CESM build, CESM's own libraries already provide a standard dgemm, so the ESMF wrapper is unnecessary.
I'm going to insert this explanation right before the first #ifdef CESMCOUPLED (unless you have an objection; let me know).
There was a problem hiding this comment.
Since there is a MPAS specific version of this file (stochastic_physics_m.F90), why does this file need changes for MPAS?
There was a problem hiding this comment.
@NingWang325, please see Phil's question above.
pjpegion
left a comment
pjpegion
left a comment
There was a problem hiding this comment.
I see there is a new file, stochastic_physics_m.F90 which looks like a copy of stocastic_physics.F90 modified to work with MPAS. Since the CMAKE_lists.txt file does not append the original stochastic_physics.F90 file, can it be changed back to its original.
Additionally, I see a new file mpas_stochastic_physics.F90 which looks like the MPAS version of the stochastic_physics wrapper that @DomHeinzeller write for the FV3 core.
We need to move towards making the stochastic physics mode model agnostic, not including more model specific subroutines. Can this file be moved to the MPAS side?
@NingWang325, can you provide feedback on the two files @pjpegion mentioned? @dustinswales, could you summarize plans for the integration of stochastic physics within the UFS? Does this effort answer @pjpegion's concerns? Thanks! |
|
@pjpegion The abbreviation "stochy" is used in many places in the code. I'm curious why there's a "y" at the end, i.e. why not just "stoch"? Do you happen to know? Thanks. |
| vfact_sppt(k)=pbl_taper(k) | ||
| enddo | ||
| endif | ||
| do k=1,MIN(7,levs) |
There was a problem hiding this comment.
@pjpegion I made this particular change, probably before we created the new file stochastic_physics_m.F90 and were still making changes in this one. At some point the build was failing around this line and I noticed this section can be made more compact using the MIN(7,levs). I can remove it if you prefer the old way.
There was a problem hiding this comment.
I am good with this change.
That was a long time ago, I don't. |
Ok. |
|
All of my concerns have been addressed. Thank you all. |
…ith the stand-alone MPAS-Model.
|
@pjpegion @gsketefian @JeffBeck-NOAA @NingWang325 All of the existing FV3 based stochastic-physics UWM regression tests have passed on Ursa. There's a new UFS MPAS RT that calls and outputs the pattern. I opened draft PRs into the UWM and UFSATM. I still need to run the full UWM RT suite, but I'm waiting on ufs-community:MPAS-A #239 to be updated. Once this is done, and the RTs pass, I will convert the PRs to ready for review. |
@dustinswales Thanks for opening those draft PRs. I'm working on addressing a couple more comments in ufs-community:MPAS-A #239. |
…(instead of simply turning off SPPT and continuing).
…v|apply] subroutines since they are always set to 0 and thus don't have any value (yet). These may be reintroduced later.
… code prints out an error message and quits.
…t SPPT are included only once at the beginning (and only brief messages are included at every time step).
… parameters and perturbations in MPAS.
6 participants
Description
This PR enables the integration of the stochastic physics scheme SPPT (but not SPP, SHUM, or SKEB) into the standalone MPAS model (
ufs-community:MPAS-Model), as described in Issue #94.This PR introduces 5 new files that add calls to initialize, advance, and apply perturbation patterns in the MPAS dynamic core files. These files are:
Makefile:A make definition file to build the stochastic physics library and interface modules.
mpas_stochastic_physics.F90:Provides interface routines to initialize, advance, and apply perturbation patterns.
Registry_stoch_physics.xml:Registry file for stochastic physics.
stochastic_physics_m.F90:Contains subroutines to initialize, advance perturbation patterns for SPPT schemes; also contains a subroutine that calculates top layers’ k indexes.
stochy_nml_rec.F90:Retrieves stochastic physics namelist records which are defined in the registry files and stored in MPAS config.
Main changes:
Makefilethat are used specifically only when building for MPAS.MPASto flag thatstochastic_physicsis being built with the MPAS dycore.STOCH_PHYS_DIAGto allow for extra diagnostics to be printed out.sppt_hgt_top1andsppt_hgt_top2to use for tapering at the top (these are the MPAS counterparts ofsppt_sigtop1andsppt_sigtop2for FV3).kinddef.F90, bring in data types specific toMPAS(e.g.RKIND).There is a companion PR in the
ufs-community'sMPAS-Modelrepo here.After this PR is merged, the
-DMPASflag must be used when building this code with the MPAS dycore.