Can a run be continued after rebuilding with DEBUG=TRUE?

[jennykowalcz]

Hi all, Sorry if this is documented somewhere and I missed it. Is it not possible to continue a run after rebuilding with DEBUG=TRUE? I am also unable to start a hybrid run with DEBUG=TRUE when the RUN_REFCASE was run with DEBUG=FALSE

[ckoven]

I have certainly done the first of these, so I think it should be possible.

[jennykowalcz]

Okay, thanks Charlie, I must be having some other kind of problem then.

[ekluzek]

The answer is absolutely you can.

It is certainly a valid thing to do. And there is no reason this would NOT work. There's only a tiny amount of code that is triggered when DEBUG is on, verses off. The only thing there that I can think of would be if it were dying in a SHR_ASSERT statement.

The other thing is that the DEBUG on version, may be dying because of for example a floating point exception that the original case didn't see. So it's exposing a problem you didn't see in the previous segment when DEBUG was off. So it's dying because of a legitimate error. It could be for example that the restart file you are starting up from has NaN's in some fields that are creating problems.

DEBUG off, does hide legitimate problems, so when you bring in code changes it's always good to run with DEBUG==TRUE in order to catch these issues. We do that with software testing of the standard code, but it might be something in the changes you have or the specific configuration you are running.

The other caveat is that answers are likely to be different by roundoff level when you change the DEBUG flag. So if you do this sort of thing, and needed to replicate the simulation, you'd need to know which segment was run with which DEBUG flag setting.

[jennykowalcz]

Thank you @ekluzek , this is very helpful! The error I get when I try to continue the run after rebuilding with DEBUG=TRUE says 0: forrtl: error (65): floating invalid so maybe this a case of the example you mentioned about a floating point exception. If I rebuild the case again with DEBUG=FALSE I can continue the run without error. I will have to dig a little deeper here...

[JessicaNeedham]

There is a known issue with DEBUG=TRUE in E3SM right now : E3SM-Project/E3SM#7049 which sounds like what you are seeing.

There is a PR to fix it which looks like it will be merged soon: E3SM-Project/E3SM#7059

[jennykowalcz]

@JessicaNeedham you are my hero 🎉🎉🎉