more binary info in finished_config

[chrisdane]

Is your feature request related to a problem? Please describe.
In the following run I had to change the echam binary from model year 1873 onward:

# on levante
cd /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2
for y in {1871..1875}; do # 5 example years
    echo $y
    ll run_${y}0101-${y}1231/bin/echam/echam6
    stat run_${y}0101-${y}1231/bin/echam/echam6 | grep Modify:
done
1871
-rwxr-xr-x 1 a270073 ba1103 49M Jul  4  2022 run_18710101-18711231/bin/echam/echam6*
Modify: 2022-07-04 16:06:16.000000000 +0200
1872
-rwxr-xr-x 1 a270073 ba1103 49M Jul  4  2022 run_18720101-18721231/bin/echam/echam6*
Modify: 2022-07-04 16:06:16.000000000 +0200
1873
-rwxr-xr-x 1 a270073 ba1103 48M Jan  4 10:40 run_18730101-18731231/bin/echam/echam6*
Modify: 2023-01-04 10:40:14.000000000 +0100
1874
-rwxr-xr-x 1 a270073 ba1103 48M Jan  4 10:40 run_18740101-18741231/bin/echam/echam6*
Modify: 2023-01-04 10:40:14.000000000 +0100
1875
-rwxr-xr-x 1 a270073 ba1103 48M Jan  4 10:40 run_18750101-18751231/bin/echam/echam6*
Modify: 2023-01-04 10:40:14.000000000 +0100

My problem is that I cannot trace back that information with the help of finished_config if the run_-dirs are removed; neither the echam_bin-entries:

for y in {1871..1875}; do
    echo $y
    grep "        echam_bin: " config/esm-piControl2_finished_config.yaml_${y}0101-${y}1231
    # bin_intermediate
    # bin_sources
    # bin_targets
done
1871
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/run_18710101-18711231/bin/echam/echam6
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/bin/echam//echam6
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/run_18710101-18711231/work/echam6
1872
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/run_18720101-18721231/bin/echam/echam6
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/bin/echam//echam6
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/run_18720101-18721231/work/echam6
1873
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/run_18730101-18731231/bin/echam/echam6
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/bin/echam//echam6
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/run_18730101-18731231/work/echam6
1874
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/run_18740101-18741231/bin/echam/echam6
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/bin/echam//echam6
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/run_18740101-18741231/work/echam6
1875
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/run_18750101-18751231/bin/echam/echam6
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/bin/echam//echam6
        echam_bin: /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/run_18750101-18751231/work/echam6

nor the bin_dir-entries from the echam section help:

for y in {1871..1875}; do
    echo $y
    grep "    bin_dir: " config/esm-piControl2_finished_config.yaml_${y}0101-${y}1231 | head -1 # first `bin_dir` is from echam
done
1871
    bin_dir: /home/a/a270073/esm/awicm-1.0-recom/bin
1872
    bin_dir: /home/a/a270073/esm/awicm-1.0-recom/bin
1873
    bin_dir: /home/a/a270073/esm/awicm-1.0-recom_jsbach_cover_err/bin
1874
    bin_dir: /home/a/a270073/esm/awicm-1.0-recom/bin
1875
    bin_dir: /home/a/a270073/esm/awicm-1.0-recom/bin

The bin_dir-entry does provide the changed model path for the year 1873. But not for the subsequent years although the new echam bin was used since then (see the dates of the echam bins above).

Describe the solution you'd like
Provide modify date and size of all utilized binaries in the finished_config.

Thanks a lot for consideration and cheers!
Chris

[pgierz]

Maybe we can implement this into the saved info provided in the new filedicts machinery once it is ready. What do you think @nwieters @mandresm?

[mandresm]

Maybe we can implement this into the saved info provided in the new filedicts machinery once it is ready. What do you think @nwieters @mandresm?

Agreed, I'm adding this to that project.

The only thing I don't understand here is this:

for y in {1871..1875}; do
    echo $y
    grep "    bin_dir: " config/esm-piControl2_finished_config.yaml_${y}0101-${y}1231 | head -1 # first `bin_dir` is from echam
done
1871
    bin_dir: /home/a/a270073/esm/awicm-1.0-recom/bin
1872
    bin_dir: /home/a/a270073/esm/awicm-1.0-recom/bin
1873
    bin_dir: /home/a/a270073/esm/awicm-1.0-recom_jsbach_cover_err/bin
1874
    bin_dir: /home/a/a270073/esm/awicm-1.0-recom/bin
1875
    bin_dir: /home/a/a270073/esm/awicm-1.0-recom/bin

in particular the years 1874 and 1875. @chrisdane, can you describe how did you switch your binary on the 1873 so that we can reproduce this behavior?

[chrisdane]

I changed the model_dir in the general section of the runscript and resubmitted the job:

$ cd /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2
$ for y in {1871..1875}; do # 5 example years
    echo $y
    grep "    model_dir: " run_${y}0101-${y}1231/scripts/awicm_recom_*.run.yaml
done
1871
    model_dir: "/home/a/a270073/esm/awicm-1.0-recom"
1872
    model_dir: "/home/a/a270073/esm/awicm-1.0-recom"
1873
    model_dir: "/home/a/a270073/esm/awicm-1.0-recom_jsbach_cover_err"
1874
    model_dir: "/home/a/a270073/esm/awicm-1.0-recom"
1875
    model_dir: "/home/a/a270073/esm/awicm-1.0-recom"

Sorry I also don't understand why the new model_dir was not kept for further jobs (the binary was; see above!) :/

[mandresm]

When resubmitting with the new binary, did you use the option --update-filetypes bin?

[mandresm]

Are you 100% sure that the binary in 1873 is identical to the new binary and not the old one?

[pgierz]

Did you give the job a new name?

[mandresm]

Did you give the job a new name?

I don't think that's the case, I'm getting that from having a look at the paths of the first post he made. I'm still puzzled that model dir is only update for the year 1873.

One observation: the runscript here /work/ba1103/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-piControl2/scripts/awicm_recom_levante_esm-piControl.run.yaml does have the old model_name. That is not the runscript you need to modify, but the one that gets updated when you run esm_runscript with your runscript. What this yaml file is telling me is that your experiment was not updated with the new model_dir, which I don't see how it is possible. Can you point me at the original runscript that you use as input for esm_runscripts?