Test UW CALIOP operator

[weiwilliam]

@SarahLu-NOAA
You can follow the README under the repo below; it looks clear.
https://github.com/WillemMarais/jpss-simcaliop

Data on S4:
GEFS-Aerosols: /ships19/aqda/lenzen/JPSS/GEFS_AER
Sample data under your path: /data/users/clu/OBS_for_IODA/CALIPSO

[SarahLu-NOAA]

Unable to clone the repo following ReadMe.

git clone git@github.com:WillemMarais/jpss-simcaliop.git
Cloning into 'jpss-simcaliop'...
Warning: Permanently added the ECDSA host key for IP address '140.82.116.3' to the list of known hosts.
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Instead, I clone the repo via url

git clone https://github.com/WillemMarais/jpss-simcaliop.git

[SarahLu-NOAA]

Encounter the same 'publickey' error msg when I run

git submodule'

The errors are solved when I add public key at Derecho to my github account

[SarahLu-NOAA]

At Derecho, when I run

aeroextsim/gefs/compileaod.bash

I run into the following error msgs:
license_intel module is unknown, and ncarenv/24.12 is not unloaded

Will try S4 for now, fyi. @weiwilliam

[SarahLu-NOAA]

@weiwilliam
Here is a dumb question.
At S4, I run

conda create -n jpss-simcaliop "python < 3.11"'

it returns

-bash: conda: command not found

What are the minimal steps needed for me run conda commands?

[SarahLu-NOAA]

Neither S4-submit and Derecho can find license_intel (specified in compileaod.bash)

[SarahLu-NOAA]

@weiwilliam
At S4, I am able to create a conda environment after

module load miniconda

I modified /data/users/clu/CALIPSO/jpss-simcaliop/aeroextsim/gefs/compileaod.bash to use whatever modules that I can find at S4 (by using module avail), I'm able to compile the code. Let's see whether the exec actually works. I'll also revise compileaod.bash for Derecho.

[weiwilliam]

@SarahLu-NOAA I can compile the code on Derecho and am running the example case.
Willem said it would take 30 minutes in the README; let's see how it goes.

[SarahLu-NOAA]

Thanks for testing it at Derecho. Could you please share your compile script (i.e. module loaded). My test run with sample case failed with the error related to idl

[weiwilliam]

Please use my fork instead: weiwilliam/jpss-simcaliop
Every step is still the same as shown in README.md besides source modulefiles/setup_derecho_intel.sh first.
In this way, you can create your jpss-simcaliop env.
I also added the run_simcaliop.sh, which can be submitted via qsub, but you need to update the paths to your folders.

I only transferred 3 GEFS files to Derecho.
All the GEFS files on S4 take 4TB.

[SarahLu-NOAA]

Thanks, @weiwilliam
Just to confirm that I should source modulefiles/setup_derecho_intel.sh before conda activate jpss-simcaliop`

[SarahLu-NOAA]

I'm able to run your jpss-simcaliop/run_simcaliop.sh with one minor edit:
SCRATCH=/glade/derecho/scratch/clu

The run, as expected, is successfully.
You use the same file name ${L1_FILE} for both input and output.

What could be logical next steps??

[weiwilliam]

Yes, Willem uses the same file name but replaces the variable with the simulated result.
Based on the outputs, I am considering using the L1 simulation from this operator and developing L2 operator in UFO/CRTM, because

• The AOD output in NetCDF format is on model levels, I am not sure about the .dat file.
• L1 UFO/CRTM development may take longer.

I'm developing L2 converter and then L1 converter.
The L1 converter can be tweaked a little bit to read the L1 input and output here and create the IODA file with hofx group.

[weiwilliam]

Evaluation and monitoring with UW operator:

• A converter to merge input (ObsValue) and output (hofx) from the UW operator to paired IODA file for METplus
• What is the matrix for CALIOP-like observation?
• METplus evaluation (profile) based on vertical stratification and time window
• Monitoring Python scripts (below)

Stand-alone evaluation:

• 2019 sample data (GEFS-Aerosols + CALIOP on Derecho), case selected by HU.
• 2022 June (CALIOP data by HU), GEFS-Aerosols from EMC (@SarahLu-NOAA)
• Python visualization script to compare Level 1 input and output (HU)

[SarahLu-NOAA]

@weiwilliam please provide the info on what routines to review for aerosol mixing ratio to ext & backscatter and from backscatter to CALIOP attenuated backscatter