Python imports suddenly not working in development environment

[forsyth2]

I was running zppy fine yesterday (on Chrysalis), but today ran into the following:

$ cd zppy
$ python tests/integration/utils.py
Traceback (most recent call last):
  File "tests/integration/utils.py", line 7, in <module>
    from mache import MachineInfo
ModuleNotFoundError: No module named 'mache'

This confused me because I had just run this step yesterday without problem. I tried creating a new development environment and running again:

$ conda clean --all
$ conda env create -f conda/dev.yml -n zppy_dev20230120
$ conda activate zppy_dev20230120
$ pip install .
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
[...]
$ python tests/integration/utils.py
Traceback (most recent call last):
  File "tests/integration/utils.py", line 7, in <module>
    from mache import MachineInfo
ModuleNotFoundError: No module named 'mache'

In a new terminal tab, I ran:

$ source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
$ python tests/integration/utils.py

That was successful, so I returned to the previous terminal tab (since I needed the development environment, not Unified, to run the next step):

$ zppy -c tests/integration/generated/test_bundles_chrysalis.cfg
Traceback (most recent call last):
  File "/home/ac.forsyth2/miniconda3/bin/zppy", line 5, in <module>
    from zppy.__main__ import main
  File "/home/ac.forsyth2/miniconda3/lib/python3.8/site-packages/zppy/__main__.py", line 6, in <module>
    from configobj import ConfigObj
ModuleNotFoundError: No module named 'configobj'
$ zppy -c tests/integration/generated/test_complete_run_chrysalis.cfg 
Traceback (most recent call last):
  File "/home/ac.forsyth2/miniconda3/bin/zppy", line 5, in <module>
    from zppy.__main__ import main
  File "/home/ac.forsyth2/miniconda3/lib/python3.8/site-packages/zppy/__main__.py", line 6, in <module>
    from configobj import ConfigObj
ModuleNotFoundError: No module named 'configobj'

Note I ran the latter command yesterday with no problem.

So... what happened? Why can Python suddenly not find certain modules I was using yesterday?

[forsyth2]

@tomvothecoder @xylar Do you see anything obviously wrong? Am I missing some step I normally do??? I'm following the testing directions at https://github.com/E3SM-Project/zppy/blob/main/tests/integration/generated/directions_chrysalis.md and the dev-environment directions from https://e3sm-project.github.io/zppy/_build/html/main/getting_started.html#b-development-environment

[forsyth2]

I just tried on Perlmutter and was able to successfully launch jobs via zppy using the Perlmutter-equivalent of the above commands.

[xylar]

@forsyth2, I think you must be activating a different conda environment in your .bashrc or similar. Or something like that. Make sure you're not initializing conda and/or mamba in your .bashrc.

[forsyth2]

I couldn't find anything like that in my .bashrc. However, I just opened a new terminal, logged into Chrysalis, and ran the commands successfully. I don't see how anything was different between the two runs. But I suppose it's working now...

[xylar]

@forsyth2, if you loaded a different conda environment, a batch submission might remember that. So it's basically best to login fresh if something like this happens, as you discovered.

[forsyth2]

Good to know, thanks!

[forsyth2]

(Although, the python tests/integration/utils.py line was failing before and that didn't involve a batch submission).