Aquaplanet external gravity wave by jeremy-lilly · Pull Request #187 · E3SM-Project/polaris

jeremy-lilly · 2024-02-08T21:28:48Z

This PR creates a new ocean task for testing the temporal convergence rate of time-stepping schemes in MPAS-O. The case is that of a simple external gravity wave on an aquaplant. The normal velocity is initialized to zero and the layer thickness is initialized as a Gaussian bump.

The user selects a time-stepping scheme and a series of time-steps to run the test. The first listed time-step will be used to produce a reference solution that other runs will be compared to.

The PR also implements a version of the task for LTS schemes, in particular for the new FB-LTS scheme from this PR: E3SM-Project/E3SM#6224. Convergence plots produced by this task for FB-LTS are given below.

Note that the external gravity wave case was chosen so as to have as few terms in the tendencies as possible -- because of this, both the LTS and FB-LTS schemes show their "true" order of convergence, rather than the first order convergence caused by the operator splitting that both the LTS and FB-LTS codes implement.

Checklist

User's Guide has been updated
Developer's Guide has been updated
API documentation in the Developer's Guide (api.md) has any new or modified class, method and/or functions listed
Documentation has been built locally and changes look as expected
Testing comment in the PR documents testing used to verify the changes
New tests have been added to a test suite

… point the forward step to the correct graph.info file

and a few other small edits

mark-petersen · 2024-03-11T20:17:20Z

@jeremy-lilly thanks for your work on this, it looks beautiful! Could you add a documentation page to the user's guide, similar to this?
https://github.com/E3SM-Project/polaris/blob/main/docs/users_guide/ocean/tasks/inertial_gravity_wave.md
Also, it is best if the history is just simple commits on some recent branchpoint from main. It looks like you pulled from main here. Could you fix that?

mark-petersen · 2024-03-11T20:38:33Z

Trying it out:

./configure_polaris_envs.py  --conda /usr/projects/climate/mpeterse/miconda3
source load_dev_polaris_0.3.0-alpha.1_chicoma-cpu_gnu_mpich.sh
polaris list
...
  45: ocean/spherical/icos/ext_grav_wav_local_time_step
  46: ocean/spherical/icos/ext_grav_wav_global_time_step
  47: ocean/spherical/qu/ext_grav_wav_local_time_step
  48: ocean/spherical/qu/ext_grav_wav_global_time_step

polaris setup -p ~/repos/E3SM/lts-fb-jer-pr/components/mpas-ocean -w $r/230311_polaris_ext_grav_wave -n 45 46 47 48

salloc -N 1 -t 2:0:0 --qos=debug --reservation=debug --account=t24_coastal_ocean
source load_dev_polaris_0.3.0-alpha.1_chicoma-cpu_gnu_mpich.sh
cd $r/230311_polaris_ext_grav_wave
polaris serial

This worked similarly on chicoma, perlmutter, and chrysalis. I am using a compiled executable from E3SM-Project/E3SM#6224.

Output is as follows:

(dev_polaris_0.3.0-alpha.1) pm:230311_ltsfb_test_polaris$ polaris serial
ocean/spherical/icos/ext_grav_wav_local_time_step
  * step: icos_base_mesh_120km
          execution:        SUCCESS
          runtime:          0:00:40
  * step: icos_init_120km_local_time_step
          execution:        SUCCESS
          runtime:          0:00:01
  * step: icos_init_lts_120km_local_time_step
          execution:        SUCCESS
          runtime:          0:00:15
  * step: icos_forward_120km_local_time_step_10s
          execution:        SUCCESS
          runtime:          0:01:17
  * step: icos_forward_120km_local_time_step_120s
          execution:        SUCCESS
          runtime:          0:00:13
  * step: icos_forward_120km_local_time_step_240s
          execution:        SUCCESS
          runtime:          0:00:25
  * step: icos_forward_120km_local_time_step_480s
          execution:        SUCCESS
          runtime:          0:00:44
  * step: analysis
          execution:        SUCCESS
          runtime:          0:00:07
  task execution:   SUCCESS
  task runtime:     0:03:42
ocean/spherical/icos/ext_grav_wav_global_time_step
  * step: icos_base_mesh_120km
          already completed
  * step: icos_init_120kmglobal_time_step
          execution:        SUCCESS
          runtime:          0:00:00
  * step: icos_forward_120km_global_time_step_10s
          execution:        SUCCESS
          runtime:          0:01:17
  * step: icos_forward_120km_global_time_step_120s
          execution:        SUCCESS
          runtime:          0:00:17
  * step: icos_forward_120km_global_time_step_240s
          execution:        SUCCESS
          runtime:          0:00:12
  * step: icos_forward_120km_global_time_step_480s
          execution:        SUCCESS
          runtime:          0:00:15
  * step: analysis
          execution:        SUCCESS
          runtime:          0:00:07
  task execution:   SUCCESS
  task runtime:     0:02:08
ocean/spherical/qu/ext_grav_wav_local_time_step
  * step: qu_base_mesh_120km
          execution:        SUCCESS
          runtime:          0:00:17
  * step: qu_init_120km_local_time_step
          execution:        SUCCESS
          runtime:          0:00:00
  * step: qu_init_lts_120km_local_time_step
          execution:        SUCCESS
          runtime:          0:00:15
  * step: qu_forward_120km_local_time_step_10s
          execution:        SUCCESS
          runtime:          0:01:18
  * step: qu_forward_120km_local_time_step_120s
          execution:        SUCCESS
          runtime:          0:00:12
  * step: qu_forward_120km_local_time_step_240s
          execution:        SUCCESS
          runtime:          0:00:17
  * step: qu_forward_120km_local_time_step_480s
          execution:        SUCCESS
          runtime:          0:00:08
  * step: analysis
          execution:        SUCCESS
          runtime:          0:00:06
  task execution:   SUCCESS
  task runtime:     0:02:34
ocean/spherical/qu/ext_grav_wav_global_time_step
  * step: qu_base_mesh_120km
          already completed
  * step: qu_init_120kmglobal_time_step
          execution:        SUCCESS
          runtime:          0:00:00
  * step: qu_forward_120km_global_time_step_10s
          execution:        SUCCESS
          runtime:          0:01:14
  * step: qu_forward_120km_global_time_step_120s
          execution:        SUCCESS
          runtime:          0:00:11
  * step: qu_forward_120km_global_time_step_240s
          execution:        SUCCESS
          runtime:          0:00:08
  * step: qu_forward_120km_global_time_step_480s
          execution:        SUCCESS
          runtime:          0:00:07
  * step: analysis
          execution:        SUCCESS
          runtime:          0:00:04
  task execution:   SUCCESS
  task runtime:     0:01:45
Task Runtimes:
0:03:42 PASS ocean/spherical/icos/ext_grav_wav_local_time_step
0:02:08 PASS ocean/spherical/icos/ext_grav_wav_global_time_step
0:02:34 PASS ocean/spherical/qu/ext_grav_wav_local_time_step
0:01:45 PASS ocean/spherical/qu/ext_grav_wav_global_time_step
Total runtime: 0:10:17
PASS: All passed successfully!

with plots identical to those above. Beautiful!

xylar

@jeremy-lilly, this looks great! The main thing I noticed is that documentation is missing. Please ping me to review again once that gets added.

In the meantime, a few other small comments.

xylar · 2024-03-11T20:52:25Z

+            dt_btr_scaling = section.getfloat('dt_btr_scaling')
+            dt_btr = self.dt / dt_btr_scaling
+            btr_dt_str = get_time_interval_string(seconds=dt_btr *
+                                                  self.resolution)


I don't think we can make this change without also updating any test cases that already use the split time stepping to use this approach. Maybe none are so far in which case maybe this will be fine.

I didn't originally add this, and I'll admit I'm a little lost as to where it is used/needed. Maybe best to address this part when we merge/rebase to account for any cases that were added in the last year?

Okay, let's see what a rebase turns up. Hopefully, you don't need to make any framework-level changes like this one. If you do, the issue remains that we'll have to make sure other test cases aren't affected.

cbegeman

Fantastic work here! I made a few comments and suggestions. Don't feel like you have to do all the things I suggested if you don't have the time. Also, I made most of the comments before Xylar posted his, so pardon any conflicting messaging there!

I ran the non-LTS case successfully with E3SM master.

Many of the suggestions have to do with making the convergence framework changes a bit more intuitive since we originally wrote the code for space and time convergence together. It will be great to have the option to evaluate time convergence separately with your changes.

cbegeman · 2024-03-11T20:35:07Z

+                error.append(error_res)
+        else:
+            for i in range(1, len(dts)):
+                mesh_name = resolution_to_subdir(resolutions[0])


Is this implying that when dts is specified, we are evaluating convergence in time only (and all steps must use the mesh at resolutions[0])? If so, I think we should put in some safeguards so that this function cannot be used incorrectly.

You are correct -- I added a note about this to the definition of dts to the docstring. Off of the top of my head, I'm not sure how to further safeguard against this being misused, but I'm happy to add something if anyone has ideas.

cbegeman · 2024-03-11T21:07:02Z

@@ -0,0 +1,81 @@
+omega:


It can be nice to have one file forward.yaml that has the config options common to both global and local time stepping and then either/both of forward_global_time_step.yaml and forward_local_time_step.yaml that contain the options that are specific to each.

Agreed! Would you be able to point me to an example of where this is done for another test case? Thanks!

This example from a draft PR of mine is close to what I have in mind:

polaris/polaris/ocean/tasks/drying_slope/forward.py

Lines 111 to 116 in fed3cbe

self.add_yaml_file('polaris.ocean.tasks.drying_slope',

'forward.yaml')

options = dict()

if coord_type == 'single_layer':

self.add_yaml_file('polaris.ocean.config', 'single_layer.yaml')

jeremy-lilly · 2024-03-13T16:51:51Z

@xylar, @cbegeman, @mark-petersen -- thank you all for your comments and suggestions! Just wanted to let you know that I saw these and that they are on my to-do list, but I might not get to them right away. I will ping everyone when ready for a second look : ) Thanks!

xylar · 2025-05-09T09:08:35Z

@jeremy-lilly, I know you're a busy guy but I'm wondering if this is something we should revive after more than a year or if it should go into draft mode to indicate that it needs more time.

jeremy-lilly · 2025-05-09T20:53:46Z

Hey @xylar, thanks for the reminder about this! I took some time to address most of the remaining comments. I'll close the ones I think are handled and leave comments on others.

Before getting too much further, do you prefer that we rebase this PR on to the current head of main, or or do you prefer that we merge the two branches?

xylar · 2025-05-09T21:09:07Z

@jeremy-lilly, I would much prefer a rebase. Thanks for checking on that. A lot has changed in a year so let me know if you run into problems you can't quite figure out. I would imagine @cbegeman may also be up for helping out, and she would be a good resource since she wrote most of the convergence infrastructure.

Co-authored-by: Carolyn Begeman <cbegeman@lanl.gov>

cbegeman · 2025-05-10T16:00:35Z

@jeremy-lilly Yes, happy to help. Let me know if there are questions that come up that I can help answer during/after the rebase.

jeremy-lilly · 2025-05-14T19:15:33Z

@xylar, @cbegeman, I did a local rebase, but since so much has changed in the underlying Polaris infrastructure and this PR makes a lot of changes to the infrastructure, I think the easiest thing will be to close this PR and rebuild the test case on top of modern Polaris. I've started that process here, and I'll open a new PR when it is ready.

xylar · 2025-05-14T19:22:58Z

Yes, I could imagine that being easier.

jeremy-lilly and others added 20 commits January 11, 2024 16:19

First commit for external gravity wave case with fblts, still need to…

4c54bb0

… point the forward step to the correct graph.info file

Fix lts regions step, and complete region labeling routine

cadab38

graph_info is read from right folder now

cee0158

and a few other small edits

Added missing file

2637cb4

Changed number of time levels in fblts namelist

b64c87c

Fixed input read of lts regions

2367cad

Fix path for graph.info if using FB_LTS

c8fab6a

Added extra forward runs for all the necessary dts

6425017

Trying to get the analysis step to work

9438371

Fixed issue thanks to Xylar

acc02c6

Intermediate commit

52b9e7e

Stuff works

d282587

Modified forward.yaml for RK4 and dt for conv test

4714bb9

Made a lot of changes

5a95f96

Added files

6dfa0bd

Fixed issues

f4e7e8d

Small change to plot

e3a061a

Modified convergence threshold

4396252

Fix step names for runs with different dts

f999cb2

Set nVertLevels during paraview extraction of lts mesh

03c72f0

jeremy-lilly mentioned this pull request Feb 8, 2024

Implement FB-LTS Time-Stepping Scheme E3SM-Project/E3SM#6224

Merged

mark-petersen requested review from cbegeman, mark-petersen and xylar March 11, 2024 19:47

mark-petersen added ocean Related to the ocean component E3SM PR required The polaris changes won't work with the current E3SM-Project submodule and require an update labels Mar 11, 2024

xylar reviewed Mar 11, 2024

View reviewed changes

Comment thread docs/users_guide/quick_start.md

xylar reviewed Mar 11, 2024

View reviewed changes

cbegeman requested changes Mar 12, 2024

View reviewed changes

jeremy-lilly force-pushed the aquaplanet-external-gravity-wave branch from 7ce4884 to 03c72f0 Compare March 15, 2024 23:11

Add documentation for ext gravity wave case

6dd12aa

jeremy-lilly added 3 commits May 9, 2025 13:37

Cleanup config files

9a9fd6f

Cleanup convergence analysis

d4c6144

Cleanup ext grav wave

5a8dd41

Remove uneeded line

bcf4c47

Co-authored-by: Carolyn Begeman <cbegeman@lanl.gov>

jeremy-lilly closed this May 14, 2025

jeremy-lilly mentioned this pull request May 23, 2025

External gravity wave #325

Merged

5 tasks

	self.add_yaml_file('polaris.ocean.tasks.drying_slope',
	'forward.yaml')

	options = dict()
	if coord_type == 'single_layer':
	self.add_yaml_file('polaris.ocean.config', 'single_layer.yaml')

Conversation

jeremy-lilly commented Feb 8, 2024

Uh oh!

mark-petersen commented Mar 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mark-petersen commented Mar 11, 2024

Uh oh!

Uh oh!

xylar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xylar Mar 11, 2024

Choose a reason for hiding this comment

Uh oh!

jeremy-lilly May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xylar May 10, 2025

Choose a reason for hiding this comment

Uh oh!

cbegeman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cbegeman Mar 11, 2024

Choose a reason for hiding this comment

Uh oh!

jeremy-lilly May 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbegeman Mar 11, 2024

Choose a reason for hiding this comment

Uh oh!

jeremy-lilly May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cbegeman May 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeremy-lilly commented Mar 13, 2024

Uh oh!

xylar commented May 9, 2025

Uh oh!

jeremy-lilly commented May 9, 2025

Uh oh!

xylar commented May 9, 2025

Uh oh!

cbegeman commented May 10, 2025

Uh oh!

jeremy-lilly commented May 14, 2025

Uh oh!

xylar commented May 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mark-petersen commented Mar 11, 2024 •

edited

Loading

jeremy-lilly May 9, 2025 •

edited

Loading

jeremy-lilly May 9, 2025 •

edited

Loading