Set io_type in MPAS streams based on PIO_TYPENAME from the CIME settings

[dqwu]

Update the build scripts for streams.ocean and streams.seaice to set
the io_type using ocn_pio_typename or ice_pio_typename, respectively.
This resolves an issue where MPAS output files did not always
follow the PIO_TYPENAME specified in the CIME settings.

Also handle ADIOS mesh I/O types in MPAS framework.

Fixes #7494

[BFB]

[jayeshkrishna]

@dqwu : Did you also verify that this change does not negatively impact MPAS standalone builds/runs?

[dqwu]

@dqwu : Did you also verify that this change does not negatively impact MPAS standalone builds/runs?

No, maybe @jonbob can review the changes and provide some feedback.

[dqwu]

@jonbob We probably need something like the pseudocode below.
Unfortunately, we don’t currently have access to pio_netcdf_format.

if ocn_pio_typename == "pnetcdf":
    if pio_netcdf_format == "64bit_data":
        lines.append(' io_type="pnetcdf,cdf5"')
    else:
        lines.append(' io_type="pnetcdf"')
else:
    lines.append(' io_type="{}"'.format(ocn_pio_typename))

[dqwu]

I will define pio_netcdf_format in the build scripts, like:

    ocn_pio_typename  = case.get_value("OCN_PIO_TYPENAME")
    ocn_pio_netcdf_format  = case.get_value("OCN_PIO_NETCDF_FORMAT")

[dqwu]

@jonbob PIO_NETCDF_FORMAT is now taken into account to set io_type to either "pnetcdf,cdf5" or "pnetcdf" in MPAS streams.

[jonbob]

@dqwu -- I asked Matt and Mark to review this as well, since they work with the standalone models more and can test to make sure these changes don't impact their work

[matthewhoffman]

I don't see anything here that will affect standalone MALI. Until recently we did not have a compass build environment for standalone MALI that supports ADIOS, but that was resolved with a new compass env a few days ago (MPAS-Dev/compass#908), so I can test out that new environment and this PR at the same time.

[matthewhoffman]

Should we add lines like

lines.append(' io_type="{}"'.format(io_type_str))

to the MALI streams as part of this PR? MALI streams currently do not have that included, and I think all the MALI streams are written with the MPAS default io_type. It's not urgent as MALI is not active in any current production simulation campaigns, but this might be the time to do it. Let me know if you would like me to add that. (Or @dqwu you are welcome to add it if you prefer)

[xylar]

@jonbob, are you in a good position to run a test with the new high res (RRS) mesh to make sure no harm is done by removing all those:

            if ocn_grid.startswith("oRRS1") or ocn_grid.startswith("FRIS") \
                    or ocn_grid.startswith("RRSwISC6"):
dqwu marked this conversation as resolved.
                lines.append('                  io_type="pnetcdf,cdf5"')
            else:
                lines.append('                  io_type="{}"'.format(ocn_pio_typename))

conditions? As @rljacob says, this should be the default now. But let's just make sure nothing got broken.

[xylar]

The changes look reasonable to me. Thanks @dqwu for taking care of this.

I ran:

./create_test --wait --walltime=1:00:00 "e3sm_cryo_developer"

and all tests passed.

I also ran the Polaris pr test suite after building MPAS-Ocean with this branch.

All tests passed.

[jonbob]

I'm running a HR test of this PR on chrysalis:

ERS.ne120pg2_r025_RRSwISC6to18E3r5.WCYCL1850NS.chrysalis_intel

where I specify

    <entry id="PIO_TYPENAME">
      <type>char</type>
      <valid_values>netcdf,pnetcdf,netcdf4p,netcdf4c,adios,adiosc,hdf5,nothing</valid_values>
      <desc>pio io type</desc>
      <values>
        <value compclass="ATM">pnetcdf</value>
        <value compclass="CPL">adios</value>
        <value compclass="OCN">adios</value>
        <value compclass="WAV">adios</value>
        <value compclass="GLC">adios</value>
        <value compclass="ICE">adios</value>
        <value compclass="ROF">pnetcdf</value>
        <value compclass="LND">pnetcdf</value>
        <value compclass="ESP">pnetcdf</value>
        <value compclass="IAC">pnetcdf</value>
      </values>
    </entry>

The first run in that test is successful but the second one (restart) fails with:

    0: PIO: ERROR: Opening file (ERS.ne120pg2_r025_RRSwISC6to18E3r5.WCYCL1850NS.chrysalis_intel.20250725_081049_5f26uz.mpaso.hist.am.highFrequencyOutput.0001-01-01_00.00.00.nc) with iotype 2 (PIO_IOTYPE_NETCDF) failed. The low level I/O library call failed. NetCDF: Unknown file format (error num=-51), (/lcrc/group/acme/ac.jwolfe/e3sm_PR7502/externals/scorpio/src/clib/pioc_support.cpp:5457)

That stream does say it's using adios, but maybe the issue is with the append clobber mode?

<stream name="highFrequencyOutput"
        type="output"
        precision="single"
        io_type="adios"
        filename_template="ERS.ne120pg2_r025_RRSwISC6to18E3r5.WCYCL1850NS.chrysalis_intel.20250725_081049_5f26uz.mpaso.hist.am.highFrequencyOutput.$Y-$M-$D_$h.$m.$s.nc"
        filename_interval="00-01-00_00:00:00"
        reference_time="01-01-01_00:00:00"
        output_interval="00-00-05_00:00:00"
        clobber_mode="append"
        packages="highFrequencyOutputAMPKG">

[rljacob]

Its certainly the attempt to append that history file in the second run. You could get around by disabling that file or forcing it to close at the end of simulated day. You would need to do that for any other history that gets appended over the course of the ERS test. Or you could try the SCORPIO option that allows history to stay in pnetcdf format.

[mark-petersen]

Tested stand-alone MPAS-O with gnu (debug and optimized) on perlmutter. Passes compass nightly test suite for all cases.

[dqwu]

@jonbob
In SCORPIO, you can override the default ADIOS type with PnetCDF for files matched by a regular expression. To force history files whose names contain substrings such as .eam.h, .elm.h, or .hist to use PnetCDF instead of ADIOS, add the following line to your job run script:
./xmlchange PIO_CONFIG_OPTS="-DSPIO_OVERRIDE_ADIOS_WITH_PNETCDF_FNAME_REGEX='.*(\.eam\.h|\.elm\.h|\.hist).*'"

[jonbob]

Thanks, @dqwu. Can you tell me what should work for an HR test? Do I expect atm and lnd to be OK with adios? Or should I only be testing ocn and ice?

[rljacob]

Run the ERS you were trying but with that additional xmlchange. You should be able to apply it in the testcasedir and then just do "./case.submit". For that test, there is no land or atm history output but if there was, you would still need to add that xmlchange. ADIOS-format restarts work fine for all models. It also works for all output files provided they never need appending (but most of E3SM's output files need appending).

[jonbob]

ERS.ne120pg2_r025_RRSwISC6to18E3r5.WCYCL1850NS.chrysalis_intel passed with

    <entry id="PIO_TYPENAME">
      <type>char</type>
      <valid_values>netcdf,pnetcdf,netcdf4p,netcdf4c,adios,adiosc,hdf5,nothing</valid_values>
      <desc>pio io type</desc>
      <values>
        <value compclass="ATM">pnetcdf</value>
        <value compclass="CPL">adios</value>
        <value compclass="OCN">adios</value>
        <value compclass="WAV">pnetcdf</value>
        <value compclass="GLC">pnetcdf</value>
        <value compclass="ICE">adios</value>
        <value compclass="ROF">pnetcdf</value>
        <value compclass="LND">pnetcdf</value>
        <value compclass="ESP">pnetcdf</value>
        <value compclass="IAC">pnetcdf</value>
      </values>
    </entry>

and changing the highFrequencyOutput clobber_mode to "truncate".

[jonbob]

Approved based on testing and visual inspection

[rljacob]

and changing the highFrequencyOutput clobber_mode to "truncate".

cool. Didn't know you could do that.

PIO_TYPENAME sets the type for all output--restart and history. The "-DSPIO_OVERRIDE_ADIOS_WITH_PNETCDF_FNAME_REGEX" option will override PIO_TYPENAME and force it to PNetCDF for whatever matches the regex. We plan to also add a PIO_TYPENAME_RESTART so you can specify history and restart types directly as you did above which would be more robust then a regex.

[matthewhoffman]

@dqwu , should we make corresponding changes for the MALI streams? Other than that question, I don't have any concerns about this PR from the MALI side.

[rljacob]

I'd say "yes" for MALI just to get this issue fixed across MPAS.

[jonbob]

@matthewhoffman and @dqwu -- who should take that on? It doesn't look too difficult if it makes sense for me to do it

[rljacob]

Please do, @jonbob

[matthewhoffman]

@jonbob , that would be great if you're happy to do it!

[rljacob]

Note: this will be ready to merge next week when Jon is back.

[jonbob]

Passes:

• SMS_D_Ld1.ne30pg2_r05_IcoswISC30E3r5.WCYCL1850.chrysalis_intel.allactive-wcprod
• ERS_Ld5.TL319_oQU240wLI_ais8to30.MPAS_LISIO_JRA1p5.chrysalis_intel.mpaso-ocn_glcshelf

Also passes the last test with PIO_TYPENAME set to "adios" for cpl, ocn, glc and ice (and changing the ocn highFrequencyOutput clobber_mode to "truncate")

Merged to next

[jonbob]

merged to master