sar_coherence + merge_cubes returns only spatial area falling in first cube bbox

[clausmichele]

I tried to merge two Sentinel-1 coherence bursts but the result is not satisfying.

The issue is that merge_cubes discards the data which is not within the bbox of the first cube. @jdries is this behaviour expected? Is there a way to do the "real" merge?

cube1.merge_cubes(cube2) results in:

cube2.merge_cubes(cube1) results in:

[jdries]

these are both cubes without time dimension? Might be a bug in that case...

[clausmichele]

@jdries an update on this:

1. Yes they have a temporal label which is the same
2. I tried dropping the temporal dim before merge_cubes, but the result is the same
3. Moreover, I can't use "mean" as overlap resolver, why? "Unexpected error during 'merge_cubes': java.lang.UnsupportedOperationException: The operator: mean is not supported when merging cubes. Supported operators are: Set(subtract, min, max, xor, multiply, add, product, divide, or, sum, and)

Here is the MWE to reproduce the issue:

import openeo
import numpy as np

url = "https://openeo-staging.dataspace.copernicus.eu"
connection = openeo.connect(url).authenticate_oidc()

datacube_0 = connection.datacube_from_process(
    process_id="insar_coherence",
    InSAR_pairs=[
        ["2024-08-21", "2024-09-02"]
    ],
    burst_id=249435,
    polarization="vv",
    sub_swath="IW2",
    coherence_window_rg=21,
    coherence_window_az=5,
).drop_dimension("t")

datacube_1 = connection.datacube_from_process(
    process_id="insar_coherence",
    InSAR_pairs=[
        ["2024-08-21", "2024-09-02"]
    ],
    burst_id=249436,
    polarization="vv",
    sub_swath="IW2",
    coherence_window_rg=21,
    coherence_window_az=5,
).drop_dimension("t")

datacube = datacube_0.merge_cubes(datacube_1,overlap_resolver="max").save_result(format='netCDF')

job = datacube.create_job(title="merged_bursts_coherence_netcdf",job_options={"python-memory": "4000m"})
job.start_job()

[clausmichele]

Same example but using load_stac for faster testing (works until the signed urls are valid):

{
  "process_graph": {
    "loadstac1": {
      "process_id": "load_stac",
      "arguments": {
        "url": "https://openeo-staging.dataspace.copernicus.eu/openeo/1.2/jobs/j-25110615345041a3acdc1a1bf17dabac/results/MGVkMGZhNTEtMGNmMC00YzI1LWE5NWYtNGY3MDA0NGVjYWYy/cdb339ba86da22c9e830573b9794b705?expires=1763111283"
      }
    },
    "dropdimension1": {
      "process_id": "drop_dimension",
      "arguments": {
        "data": {
          "from_node": "loadstac1"
        },
        "name": "t"
      }
    },
    "loadstac2": {
      "process_id": "load_stac",
      "arguments": {
        "url": "https://openeo-staging.dataspace.copernicus.eu/openeo/1.2/jobs/j-2511061532264c99b140b2e790a936b7/results/MGVkMGZhNTEtMGNmMC00YzI1LWE5NWYtNGY3MDA0NGVjYWYy/a110e5799d984026a70c7fcf5710a181?expires=1763111282"
      }
    },
    "dropdimension2": {
      "process_id": "drop_dimension",
      "arguments": {
        "data": {
          "from_node": "loadstac2"
        },
        "name": "t"
      }
    },
    "mergecubes1": {
      "process_id": "merge_cubes",
      "arguments": {
        "cube1": {
          "from_node": "dropdimension1"
        },
        "cube2": {
          "from_node": "dropdimension2"
        },
        "overlap_resolver": {
          "process_graph": {
            "max1": {
              "process_id": "max",
              "arguments": {
                "data": [
                  {
                    "from_parameter": "x"
                  },
                  {
                    "from_parameter": "y"
                  }
                ]
              },
              "result": true
            }
          }
        }
      }
    },
    "saveresult1": {
      "process_id": "save_result",
      "arguments": {
        "data": {
          "from_node": "mergecubes1"
        },
        "format": "netCDF",
        "options": {}
      },
      "result": true
    }
  }
}

[clausmichele]

@jdries are there news related to this?

[jdries]

By design, the code does look like this should be supported, so it has to be a more specific bug.
I'm running the MWE to get a job id.

[jdries]

Ok, so because of these custom processes, an important part in our graph preprocessing, the 'global extent' is simply not working as it normally should. This is resulting in negative spatial keys, which causes data to be effectively filtered out before merge cubes.

I'm only not yet sure why it also wouldn't not work when using load_stac directly.

Possible workaround: add filter_bbox

Relevant logs:

Dry run extracted these source constraints: [(('load_stac', ('dummy', (), ())), {}), (('load_stac', ('dummy', (), ())), {})]

global_extent fallback: layer_native_extent=BoundingBox(west=627330.231, south=5163013.793, east=726570.231, north=5203293.793, crs='EPSG:32632')

Loading https://s3.waw4-1.cloudferro.com/calrissian/pvc-91dd9cc8-952b-4f4a-abf9-77a9e32beb58/j-2512041253404309a3-cal-cwl-d967dc08/S1_2images_collection.json with params DataCubeParameters(128, {}, FloatingLayoutScheme, ByDay, None, None, NearestNeighbor, 0.0, 0.0) and bands coh_VV initial layout: LayoutDefinition(Extent(627320.0, 5162340.0, 727160.0, 5203300.0),CellSize(20.0,20.0),39x16 tiles,4992x2048 pixels)

global_extent fallback: layer_native_extent=BoundingBox(west=623051.18, south=5181536.401, east=723651.18, north=5221536.401, crs='EPSG:32632')
Loading https://s3.waw4-1.cloudferro.com/calrissian/pvc-91dd9cc8-952b-4f4a-abf9-77a9e32beb58/j-2512041253404309a3-cal-cwl-de98ee2a/S1_2images_collection.json with params DataCubeParameters(128, {}, FloatingLayoutScheme, ByDay, None, None, NearestNeighbor, 0.0, 0.0) and bands coh_VV initial layout: LayoutDefinition(Extent(623040.0, 5180580.0, 725440.0, 5221540.0),CellSize(20.0,20.0),40x16 tiles,5120x2048 pixels)

Keybounds after preemptive filtering: KeyBounds(SpatialKey(0,0),SpatialKey(37,8))
Keybounds before preemptive filtering: KeyBounds(SpatialKey(-2,-8),SpatialKey(37,8))
Merging cubes with partitioners: None - Some(SpacePartitioner(KeyBounds(SpatialKey(0,0),SpatialKey(39,15)))) - many band case detected: false
Creating ConfigurableSpatialPartitionerReduceZ(9) based on tile size: 16384, band count: 11, cell type bits: 32, tileSizeInMb: 0.6875

[jdries]

@EmileSonneveld I pushed a commit for this one, so that if an extra filter_bbox is used, it might work.
To follow up if that actually works and doesn't break other things.

[jdries]

My fix seems to make the filter_bbox workaround work, 2 cubes are being loaded into same layout:

Loading https://s3.waw4-1.cloudferro.com/calri...S1_2images_collection.json with params DataCubeParameters(128, {}, FloatingLayoutScheme, ByDay, None, None, NearestNeighbor, 0.0, 0.0) and bands coh_VV initial layout: LayoutDefinition(Extent(627320.0, 5151860.0, 727160.0, 5213300.0),CellSize(20.0,20.0),39x24 tiles,4992x3072 pixels)

Loading https://s3.waw4-1.cloudferro.com/calr.../S1_2images_collection.json with params DataCubeParameters(128, {}, FloatingLayoutScheme, ByDay, None, None, NearestNeighbor, 0.0, 0.0) and bands coh_VV initial layout: LayoutDefinition(Extent(627320.0, 5151860.0, 727160.0, 5213300.0),CellSize(20.0,20.0),39x24 tiles,4992x3072 pixels)

[jdries]

@clausmichele for the demo, to have a working solution, please add an extra filter_bbox with your desired target area.
For a better fix, the linked issue about regridding needs to be solved, but will take more time.

[clausmichele]

If I merge 3 bursts, do I add a single filter_bbox at the end or two in a series?

[jdries]

A single filter_bbox with the union of everything is what we need!