E3SM ne4pg2 eamxx prod test fails with HDF5 iotypes

[jayeshkrishna]

The E3SM test, SMS_Ln5.ne4pg2_ne4pg2.F2010-SCREAMv1.MACHINE_COMPILER.eamxx-prod (e.g. SMS_Ln5.ne4pg2_ne4pg2.F2010-SCREAMv1.anlgce-ub22_gnu.eamxx-prod), fails with HDF5 I/O types for SCORPIO master & SCORPIO v1.9.2

The test fails with PIO_IOTYPE_HDF5 (./xmlchange PIO_TYPENAME=hdf5) and PIO_IOTYPE_HDF5C (./xmlchange PIO_TYPENAME=hdf5c) I/O types. The test fails using SCORPIO v1.9.2 with the following error message,

[0] PIO: FATAL ERROR: Aborting... An error occured, Ending the define mode for file (SMS_Ln5.ne4pg2_ne4pg2.F2010-SCREAMv1.anlgce-ub22_gnu.eamxx-prod.20260416_144104_zu2txp.scream.1hourlyINST_native.h.INSTANT.nhours_x1.1994-10-01-00000.nc, ncid=29) using HDF5 iotype failed. The low level (HDF5) I/O library call failed to attach a dimension scale (for dimension ilev) to 0th dimension of a dataset (for variable hyai). HDF5 API failed. Unknown error occurred when calling an HDF5 API (err=-700). Aborting since the error handler was set to PIO_INTERNAL_ERROR... (/scratch/jayesh/E3SM_master/externals/scorpio/src/clib/pioc_support.cpp: 7413)
[0] HDF5-DIAG: Error detected in HDF5 (1.14.6) MPI-process 0:
[0]   #000: /nfs/gce/projects/climate/software/linux-ubuntu22.04-x86_64/hdf5/1.14.6/src/src/H5O.c line 1077 in H5Oget_info3(): invalid location identifier
[0]     major: Invalid arguments to routine
[0]     minor: Inappropriate type
[0]   #001: /nfs/gce/projects/climate/software/linux-ubuntu22.04-x86_64/hdf5/1.14.6/src/src/H5VLint.c line 1756 in H5VL_vol_object(): invalid identifier type to function
[0]     major: Invalid arguments to routine
[0]     minor: Inappropriate type

EAMXX changes to use HDF5 iotypes is being added in E3SM-Project/E3SM#8269
To recreate the issue change the eamxx prod E3SM test mod to use hdf5 for I/O type,

diff --git a/components/eamxx/cime_config/testdefs/testmods_dirs/eamxx/prod/shell_commands b/components/eamxx/cime_config/testdefs/testmods_dirs/eamxx/prod/shell_commands
index 1f6d37e6b5..54f19edbaf 100644
--- a/components/eamxx/cime_config/testdefs/testmods_dirs/eamxx/prod/shell_commands
+++ b/components/eamxx/cime_config/testdefs/testmods_dirs/eamxx/prod/shell_commands
@@ -13,7 +13,8 @@ case_name=$(./xmlquery --value CASE)
 
 # For big data
 ./xmlchange PIO_NETCDF_FORMAT="64bit_data"
-./xmlchange PIO_TYPENAME=pnetcdf #adios #,PIO_TYPENAME_ATM=adios
+./xmlchange PIO_TYPENAME=hdf5 #adios #,PIO_TYPENAME_ATM=adios
 ./xmlchange PIO_REARRANGER=1  # use PIO_REARRANGER=3, for ADIOS; PIO_REARRANGER=1 for pnetcdf
 
 # Turn on cosp and set frequency
@@ -100,6 +101,9 @@ for file in ${output_yaml_files[@]}; do
     # Replace horiz remap file
     sed -i "s|horiz_remap_file:.*|horiz_remap_file: ${mapfile}|" ./$(basename ${file})
 
+    # Set iotype to hdf5
+    sed -i "s|iotype:.*|iotype: hdf5|" ./$(basename ${file})
+
     # Replace all filename prefixes so that st_archive works...
     sed -i "s|eamxx_output.decadal|${case_name}.scream|" ./$(basename ${file})
 done

Thanks @AaronDonahue for finding this issue during testing.