PIO errors with Gnu on many machines

[xylar]

We are seeing errors like:

Assertion failed...
PIO: FATAL ERROR: Aborting... invalid IO type (/gpfs/fs1/home/ac.xylar/e3sm_work/polaris/update-to-0.10.0-alpha.1/e3sm_submodules/Omega/externals/scorpio/src/clib/pio_lists.cpp: 120)

on various machines with various Polaris tests, always with some version of the Gnu compilers:

• chrysalis gnu:
  • icos/rotation_2d
• frontier craygnu:
  • manufactured_solution/convergence_both/default
• frontier craygnu-mphipcc
  • manufactured_solution/convergence_both/default
  • manufactured_solution/convergence_both/del2
  • manufactured_solution/convergence_both/default
  • icos/rotation_2d

See E3SM-Project/polaris#412 and E3SM-Project/polaris#420

We are also seeing hanging on Perlmutter that may or may not be the same issue: E3SM-Project/polaris#396

[amametjanov]

@xylar can you try switching from NETCDF4C to PNETCDF as the default IO type (as here) to see if it fixes polaris errors.
There are some CTests errors with 64-bit ints, which i was working on (WIP), but netcdf libs on some machines might not support NETCDF4C.

[xylar]

Unfortunately, #325 did not fix this issue, though it reduced the frequency.

The only compilers where I continue to see the issue are craygnu-mphipcc and crayamd-mphipcc on Frontier. Which tests have the issue changes from one omega_pr run to the next.