E3SM api 27 restart issue

[rgknox]

E3SM-fates does not pass restart tests with API 27, namely:

ERS_Ld20.f45_f45.IELMFATES.pm-cpu_gnu (perlmutter)

The changes are relegated to the two history diagnostics that use the heterotrophic respiration boundary condition passed in from E3SM.

I'm currently testing some fixes, the first approach is to save the column/site level heterotrophic respiration in a fates structure, instead of setting the diagnostics from the bc_in structure.

[rosiealice]

@mvertens will kick off a runs to spin up the model and look at the results next week (Norwegian computers just changed their configurations and @mvdebolskiy is mostly trying to fix compiling the model on that...)

[glemieux]

@rgknox this appears to be cropping back up in testing the expanded fates test list for E3SM-Project/E3SM#6018 for some reason.

[glemieux]

I determined that this came up via E3SM-Project/E3SM#6018 in stead of E3SM-Project/E3SM#5555 because the former includes an update to make the test output double precision (hist_ndens=1). I confirmed this by testing E3SM-Project/E3SM#5555 (which includes the TOT_HET_RESP fix from #1114) twice, once with the default single precision and once with double precision. Sure enough the single precision passes, but the double fails. The differences I'm seeing are on the order of E-28 (normalized to E-17). This crops up with both gnu and intel compilers, regardless of debug mode.

@rgknox and I discussed this and our short term plan is to reopen this issue and revert back to single precision for E3SM-Project/E3SM#6018 for the time being.