Make `run_cwl_to_stac` job result downloadable

[EmileSonneveld]

Right now, process graphs that use run_cwl_to_stac allow to export raw stac to a workspace, but do have a valid stac result themself

stac_resource = StacResource(
    graph=PGNode(
        process_id="run_cwl_to_stac",
        arguments={
            "cwl_url": "https://raw.githubusercontent.com/cloudinsar/s1-workflows/refs/heads/main/cwl/sar_slc_preprocessing.cwl",
            "context": input_dict_belgium_vv_master_outside_preprocessing,
        },
    ),
    connection=connection,
)

Ideally, their own job results should be usable like a normal batch job
#70 (comment)

[jdries]

@EmileSonneveld I have trouble understanding the issue description. (What is 'raw stac' vs 'valid stac' anyway?)
Please clarify so that it also becomes more clear what needs to be solved and what required effort is.

[EmileSonneveld]

With the raw STAC, I meant the direct output from the CWL. However, when we try to download those results trough the web-app, it raises an error, because some relative paths that are not resolved correctly.
This is difficult to test, because it needs to be deployed to test, or heavily mocked

[EmileSonneveld]

Wrong URL in metadata:

[EmileSonneveld]

On staging it is possible to download the raw CWL results now.
I am testing if it is already possible to use those using "image-name" on prod, but ran in some issues for the apex force cwl.
Insar CWL works fine on staging.

[clausmichele]

So should I run run_cwl_to_stac in combination with save_result or on its own?

[EmileSonneveld]

run_cwl_to_stac is to be used on its own, save_results would make its own STAC in the openEO way.
The ticket for the bypass that allows the run_udf interface to give raw results is here: #70

[clausmichele]

@EmileSonneveld I just tried but it failed, the logs do not contain useful info.

job_id = j-2603170911404a0db967437eefef1818

import openeo
from openeo.rest.stac_resource import StacResource
from openeo.internal.graph_building import PGNode

url = "https://openeo-staging.dataspace.copernicus.eu"
connection = openeo.connect(url).authenticate_oidc()

burst_id = 249435
polarization = "vv"
sub_swath = "IW2"
params = {
    "burst_id": burst_id,
    "polarization":  polarization,
    "sub_swath": sub_swath,
    "InSAR_pairs": [["2024-08-21", "2024-09-02"]],
    "coherence_window_rg":10,
    "coherence_window_az":2,
    "n_rg_looks":4,
    "n_az_looks":1
}

stac_resource = StacResource(
graph=PGNode(
    process_id="run_cwl_to_stac",
    arguments={
        "cwl_url": "https://raw.githubusercontent.com/cloudinsar/s1-workflows/refs/heads/keep_snap_metadata/cwl/sar_interferogram.cwl",
        "context": params,
    },
),
connection=connection,
)


merge_prefix = f"/interferogram/{burst_id}/{sub_swath}/20240821_20240902_custom_image"
job = stac_resource.create_job(title=merge_prefix,job_options={"python-memory": "4000m"})
job.start_job()

[EmileSonneveld]

I see this one in the logs: Exception: Process returned error status code: -9. This would come from one of the early exec_proc() calls

It comes from a gpt command, it has some more logs here:

SNAP logs

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.esa.snap.runtime.Engine (file:/usr/local/esa-snap/snap/modules/ext/org.esa.snap.snap-core/org-esa-snap/snap-runtime.jar) to method java.lang.ClassLoader.initializePath(java.lang.String)
WARNING: Please consider reporting this to the maintainers of org.esa.snap.runtime.Engine
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
WARNING: org.esa.snap.core.util.EngineVersionCheckActivator: A new SNAP version is available for download.
Currently installed 11.0.0, available is 13.0.0.
Please visit http://step.esa.int

Executing processing graph
INFO: eu.esa.sar.commons.io.ImageIOFile: Using FileCacheImageInputStream
INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/orbits/Sentinel-1/POEORB/S1A/2024/08/S1A_OPER_AUX_POEORB_OPOD_20240910T070624_V20240820T225942_20240822T005942.EOF.zip
INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/orbits/Sentinel-1/POEORB/S1A/2024/09/S1A_OPER_AUX_POEORB_OPOD_20240922T070612_V20240901T225942_20240903T005942.EOF.zip
WARNING: org.jlinda.core.Baseline: Max. error bperp modeling at 3D datapoints: 7.190966079798463m
WARNING: org.jlinda.core.Baseline: Max. error bperp modeling at 3D datapoints: 7.190344941442169m
....11%....21%.Copernicus_DSM_COG_10_N46_00_E010_00_DEM.tif
INFO: org.esa.snap.engine_utilities.download.DownloadableContentImpl: http retrieving http://step.esa.int/auxdata/dem/egm96/ww15mgh_b.zip
Copernicus_DSM_COG_10_N45_00_E010_00_DEM.tif
Copernicus_DSM_COG_10_N45_00_E011_00_DEM.tif
Copernicus_DSM_COG_10_N45_00_E012_00_DEM.tif
Copernicus_DSM_COG_10_N46_00_E011_00_DEM.tif
Copernicus_DSM_COG_10_N46_00_E012_00_DEM.tif
...31%

Maybe it is an oom. Does it happen when re-running?

[EmileSonneveld]

I ran the j-2603170911404a0db967437eefef1818 job again, without job options, and it got trough without errors.

[clausmichele]

@EmileSonneveld I tried again with the latest CWL and image and it worked!

import openeo
from openeo.rest.stac_resource import StacResource
from openeo.internal.graph_building import PGNode

url = "https://openeo-staging.dataspace.copernicus.eu"
connection = openeo.connect(url).authenticate_oidc()

burst_id = 249435
polarization = "vv"
sub_swath = "IW2"
params = {
    "burst_id": burst_id,
    "polarization":  polarization,
    "sub_swath": sub_swath,
    "InSAR_pairs": [["2024-08-21", "2024-09-02"]],
    "coherence_window_rg":10,
    "coherence_window_az":2,
    "n_rg_looks":4,
    "n_az_looks":1
}

stac_resource = StacResource(
graph=PGNode(
    process_id="run_cwl_to_stac",
    arguments={
        "cwl_url": "https://raw.githubusercontent.com/cloudinsar/s1-workflows/refs/heads/main/cwl/sar_interferogram.cwl",
        "context": params,
    },
),
connection=connection,
)


merge_prefix = f"/interferogram/{burst_id}/{sub_swath}/20240821_20240902_latest_image"
job = stac_resource.create_job(title=merge_prefix)