Skip to content

Commit 5bdf7b9

Browse files
committed
Upgrade Azure Batch SDK
1 parent aa133e4 commit 5bdf7b9

22 files changed

Lines changed: 450 additions & 537 deletions

‎Dockerfile.stacapi‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
1-
FROM python:3.8-slim
1+
FROM mcr.microsoft.com/azurelinux/base/python:3.12
22

3-
4-
ENV CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
3+
RUN \
4+
tdnf update -y && \
5+
tdnf ca-certificates curl -y
56

67
ENV PATH=$PATH:/install/bin
78

‎pctasks/cli/requirements.txt‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -211,15 +211,12 @@ pytz==2025.2
211211
# via planetary-computer
212212
# from https://pypi.org/simple
213213
pyyaml==6.0.2
214-
# via
215-
# pctasks-core
216-
# stac-validator
214+
# via pctasks-core
217215
# from https://pypi.org/simple
218216
referencing==0.36.2
219217
# via
220218
# jsonschema
221219
# jsonschema-specifications
222-
# stac-validator
223220
# from https://pypi.org/simple
224221
requests==2.32.4
225222
# via
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
{
2+
"id": "deltaresfloods-deltares-floods-ingest-collection",
3+
"definition": {
4+
"id": "deltaresfloods-deltares-floods-ingest-collection",
5+
"name":"Ingest Collection:
6+
deltares-floods","dataset":"deltaresfloods/deltares-floods","jobs":{"ingest-collection":{"id":"ingest-collection","tasks":[{"id":"ingest-collection","image_key":"ingest","task":"pctasks.ingest_task.task:ingest_task","args":{"content":{"id":"deltares-floods","type":"Collection","links":[{"rel":"documentation","href":"https: //ai4edatasetspublicassets.blob.core.windows.net/assets/a
7+
od_docs/11206409-003-ZWS-0003_v0.1-Planetary-Computer-Deltares-global-flood-docs.pdf","type":"application/pdf","title":"User Guide"},{"rel":"license","href":"https: //cdla.dev/permissive-1-0/","type":"text/html","title":"Community Data License Agreement - Permissive, Version 1.0"}],"title":"Deltares Global Flood
8+
Maps","assets":{"thumbnail":{"href":"https: //ai4edatasetspublicassets.azureedge.net/assets/pc_thumbnails/additional_datasets/deltares-flood.png","type":"image/png","roles":["thumbnail"],"title":"Thumbnail"}},"extent":{"spatial":{"bbox":[[-180,90,180,-90]]},"temporal":{"interval":[["2018-01-01T00:00:00Z","2018-12-31T00:00:00Z"],["2050-01-01T00:00:00Z","2050-12-31T00:00:00Z"]]}},"
9+
license":"CDLA-Permissive-1.0","keywords":["Deltares","Flood","Sea level
10+
rise","Water","Global"],"providers":[{"url":"https: //www.deltares.nl/en/","name":"Deltares","roles":["producer"]},{"url":"https://planetarycomputer.microsoft.com/","name":"Microsoft","roles":["host"]}],"summaries":{"deltares:dem_name":["NASADEM","MERITDEM","LIDAR"],"deltares:resolution":["90m","1km","5km"],"deltares:return_period":[0,2,5,10,25,50,100,250],"deltares:sea_level_yea
11+
r":[2018,2050]},"description":"[Deltares
12+
](https: //www.deltares.nl/en/) has produced inundation maps of flood depth using a model that takes into account water level attenuation and is forced by sea level. At the coastline, the model is forced by extreme water levels containing surge and tide from GTSMip6. The water level at the coastline is extended landwards to all areas that
13+
are hydrodynamically connected to the coast following a ‘bathtub’ like approach and calculates the flood depth as the difference between the water level and the topography. Unlike a simple 'bathtub' model, this model attenuates the water level over land with a maximum attenuation factor of 0.5 m km-1. The attenuation factor simulates the dampening of the flood levels due to the
14+
roughness over land.\n\nIn its current version, the model does not account for varying roughness over land and permanent water bodies such as rivers and lakes, and it does not account for the compound effects of waves, rainfall, and river discharge on coastal flooding. It also does not include the mitigating effect of coastal flood protection. Flood extents must thus be
15+
interpreted as the area that is potentially exposed to flooding without coastal protection.\n\nSee the complete (https: //ai4edatasetspublicassets.blob.core.windows.net/assets/aod_docs/11206409-003-ZWS-0003_v0.1-Planetary-Computer-Deltares-global-flood-docs.pdf) for more information.\n\n## Digital elevation models (DEMs)\n\nThis documentation will refer to three DEMs:\n\n*
16+
`NASADEM` is the SRTM-derived [NASADEM
17+
](https: //planetarycomputer.microsoft.com/dataset/nasadem) product.\n* `MERITDEM` is the [Multi-Error-Removed Improved Terrain DEM](http://hydro.iis.u-tokyo.ac.jp/~yamadai/MERIT_DEM/), derived from SRTM and AW3D.\n* `LIDAR` is the [Global LiDAR Lowland DTM (GLL_DTM_v1)](https://data.mendeley.com/datasets/v5x4vpnzds/1).\n\n## Global
18+
datasets\n\nThis collection includes multiple global flood datasets derived from three different DEMs (`NASA`, `MERIT`, and `LIDAR`) and at different resolutions. Not all DEMs have all resolutions:\n\n* `NASADEM` and `MERITDEM` are available at `90m` and `1km` resolutions\n* `LIDAR` is available at `5km` resolution\n*\n## Historic event datasets\n\nThis collection also includes
19+
historical storm event data files that follow similar DEM and resolution conventions. Not all storms events are available for each DEM and resolution combination, but generally follow the format of:\n\n`events/[DEM
20+
]_-wm_final/__masked.nc`\n\nFor example, a flood map for the MERITDEM-derived 90m flood data for the \"Omar\" storm in 2008 is available
21+
at:\n\n<https: //deltaresfloodssa.blob.core.windows.net/floods/v2021.06/events/MERITDEM_90m-wm_final/Omar_2008_masked.nc>\n\n## Contact\n\nFor questions about this dataset, contact [`aiforearthdatasets@microsoft.com`](mailto:aiforearthdatasets@microsoft.com?subject=deltares-floods%20question).","item_assets":{"data":{"type":"application/x-netcdf","roles":["data"],"title":"Flood
22+
Map","description":"Inundation maps of flood depth using a model that takes into account water level attenuation and is forced by sea level."},"index":{"type":"application/json","roles":["index"],"title":"Index file","description":"Kerchunk index file."}},"stac_version":"1.0.0","cube:variables":{"inun":{"type":"data","unit":"m","attrs":{"units":"m","long_name":"Coastal
23+
flooding","standard_name":"water_surface_height_above_reference_datum"},"dimensions":["time","lat","lon"],"description":"Coastal flooding"},"projection":{"type":"data","attrs":{"EPSG_code":"EPSG: 4326","long_name":"wgs84","proj4_params":"+proj=longlat +ellps=WGS84 +datum=WGS84
24+
+no_defs","grid_mapping_name":"latitude_longitude"},"shape":[],"dimensions":[],"description":"wgs84"}},"msft:container":"floods","msft:region":"westeurope","cube:dimensions":{"lat":{"axis":"y","type":"spatial","extent":[-89.97500000000002,89.975],"description":"latitude","reference_system":{"name":"undefined","type":"GeographicCRS","datum":{"id":{"code":6326,"authority":"EPSG"},"name": "World Geodetic System 1984",
25+
"type": "GeodeticReferenceFrame",
26+
"ellipsoid": {
27+
"name":"WGS 84","semi_major_axis":6378137,"inverse_flattening":298.257223563}},"$schema":"https: //proj.org/schemas/v0.4/projjson.schema.json","coordinate_system":{"axis":[{"name":"Longitude","unit":"degree","direction":"east","abbreviation":"lon"},{"name":"Latitude","unit":"degree","direction":"north","abbreviation":"lat"}],"subtype":"ellipsoidal"}}},"lon":{"axis":"x","type":"spatial","exte
28+
nt":[-179.975,179.97500000000005],"description":"longitude","reference_system":{"name":"undefined","type":"GeographicCRS","datum":{"id":{"code":6326,"authority":"EPSG"},"name":"World Geodetic System 1984","type":"GeodeticReferenceFrame","ellipsoid":{"name":"WGS
29+
84","semi_major_axis":6378137,"inverse_flattening":298.257223563}},"$schema":"https: //proj.org/schemas/v0.4/projjson.schema.json","coordinate_system":{"axis":[{"name":"Longitude","unit":"degree","direction":"east","abbreviation":"lon"},{"name":"Latitude","unit":"degree","direction":"north","abbreviation":"lat"}],"subtype":"ellipsoidal"}}},"time":{"type":"temporal","extent":["201
30+
0-01-01T00: 00: 00Z","2010-01-01T00: 00: 00Z"],"description":"time"}},"stac_extensions":["https: //stac-extensions.github.io/datacube/v2.0.0/schema.json","https://stac-extensions.github.io/item-assets/v1.0.0/schema.json"],"msft:storage_account":"deltaresfloodssa","msft:short_description":"Global estimates of coastal inundation under various sea level rise conditions and return
31+
periods."},"options":{"insert_group_size":5000,"insert_only":false}},"environment":{},"schema_version":"1.0.0"}]}},"is_streaming":false,"schema_version":"1.0.0"}}

‎pctasks/client/requirements.txt‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -225,15 +225,12 @@ pytz==2025.2
225225
# via planetary-computer
226226
# from https://pypi.org/simple
227227
pyyaml==6.0.2
228-
# via
229-
# pctasks-core
230-
# stac-validator
228+
# via pctasks-core
231229
# from https://pypi.org/simple
232230
referencing==0.36.2
233231
# via
234232
# jsonschema
235233
# jsonschema-specifications
236-
# stac-validator
237234
# from https://pypi.org/simple
238235
requests==2.32.4
239236
# via

‎pctasks/core/requirements.txt‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -208,15 +208,12 @@ pytz==2025.2
208208
# via planetary-computer
209209
# from https://pypi.org/simple
210210
pyyaml==6.0.2
211-
# via
212-
# pctasks-core (./pctasks/core/pyproject.toml)
213-
# stac-validator
211+
# via pctasks-core (./pctasks/core/pyproject.toml)
214212
# from https://pypi.org/simple
215213
referencing==0.36.2
216214
# via
217215
# jsonschema
218216
# jsonschema-specifications
219-
# stac-validator
220217
# from https://pypi.org/simple
221218
requests==2.32.4
222219
# via

‎pctasks/dataset/requirements.txt‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -245,15 +245,12 @@ pytz==2025.2
245245
# via planetary-computer
246246
# from https://pypi.org/simple
247247
pyyaml==6.0.2
248-
# via
249-
# pctasks-core
250-
# stac-validator
248+
# via pctasks-core
251249
# from https://pypi.org/simple
252250
referencing==0.36.2
253251
# via
254252
# jsonschema
255253
# jsonschema-specifications
256-
# stac-validator
257254
# from https://pypi.org/simple
258255
requests==2.32.4
259256
# via

‎pctasks/dev/requirements.txt‎

Lines changed: 6 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,5 @@
11
# This file was autogenerated by uv via the following command:
22
# uv pip compile --system --emit-index-annotation ./pctasks/dev/pyproject.toml -o pctasks/dev/requirements.txt
3-
adal==1.2.7
4-
# via msrestazure
5-
# from https://pypi.org/simple
63
aiohappyeyeballs==2.6.1
74
# via aiohttp
85
# from https://pypi.org/simple
@@ -24,21 +21,18 @@ attrs==25.3.0
2421
# jsonschema
2522
# referencing
2623
# from https://pypi.org/simple
27-
azure-batch==11.0.0
24+
azure-batch==15.0.0b2
2825
# via pctasks-run
2926
# from https://pypi.org/simple
30-
azure-common==1.1.28
31-
# via azure-batch
32-
# from https://pypi.org/simple
3327
azure-core==1.35.0
3428
# via
29+
# azure-batch
3530
# azure-cosmos
3631
# azure-data-tables
3732
# azure-identity
3833
# azure-keyvault-secrets
3934
# azure-storage-blob
4035
# azure-storage-queue
41-
# msrest
4236
# opencensus-ext-azure
4337
# from https://pypi.org/simple
4438
azure-cosmos==4.9.0
@@ -67,7 +61,6 @@ cachetools==5.3.3
6761
certifi==2025.8.3
6862
# via
6963
# kubernetes
70-
# msrest
7164
# requests
7265
# from https://pypi.org/simple
7366
cffi==1.17.1
@@ -84,7 +77,6 @@ click==8.2.1
8477
# from https://pypi.org/simple
8578
cryptography==45.0.6
8679
# via
87-
# adal
8880
# azure-identity
8981
# azure-storage-blob
9082
# azure-storage-queue
@@ -118,11 +110,11 @@ idna==3.10
118110
# from https://pypi.org/simple
119111
isodate==0.7.2
120112
# via
113+
# azure-batch
121114
# azure-data-tables
122115
# azure-keyvault-secrets
123116
# azure-storage-blob
124117
# azure-storage-queue
125-
# msrest
126118
# from https://pypi.org/simple
127119
jinja2==3.0.3
128120
# via pctasks-ingest
@@ -159,14 +151,6 @@ msal==1.33.0
159151
msal-extensions==1.3.1
160152
# via azure-identity
161153
# from https://pypi.org/simple
162-
msrest==0.7.1
163-
# via
164-
# azure-batch
165-
# msrestazure
166-
# from https://pypi.org/simple
167-
msrestazure==0.6.4.post1
168-
# via azure-batch
169-
# from https://pypi.org/simple
170154
multidict==6.6.4
171155
# via
172156
# aiohttp
@@ -272,9 +256,7 @@ pygments==2.19.2
272256
# via rich
273257
# from https://pypi.org/simple
274258
pyjwt==2.10.1
275-
# via
276-
# adal
277-
# msal
259+
# via msal
278260
# from https://pypi.org/simple
279261
pystac==1.10.1
280262
# via
@@ -287,7 +269,6 @@ pystac-client==0.9.0
287269
# from https://pypi.org/simple
288270
python-dateutil==2.8.2
289271
# via
290-
# adal
291272
# argo-workflows
292273
# kubernetes
293274
# pctasks-core
@@ -307,32 +288,26 @@ pyyaml==6.0.2
307288
# via
308289
# kubernetes
309290
# pctasks-core
310-
# stac-validator
311291
# from https://pypi.org/simple
312292
referencing==0.36.2
313293
# via
314294
# jsonschema
315295
# jsonschema-specifications
316-
# stac-validator
317296
# from https://pypi.org/simple
318297
requests==2.32.4
319298
# via
320-
# adal
321299
# azure-core
322300
# google-api-core
323301
# kubernetes
324302
# msal
325-
# msrest
326303
# opencensus-ext-azure
327304
# planetary-computer
328305
# pystac-client
329306
# requests-oauthlib
330307
# stac-validator
331308
# from https://pypi.org/simple
332309
requests-oauthlib==2.0.0
333-
# via
334-
# kubernetes
335-
# msrest
310+
# via kubernetes
336311
# from https://pypi.org/simple
337312
rich==14.1.0
338313
# via pctasks-client
@@ -349,7 +324,6 @@ six==1.17.0
349324
# via
350325
# azure-core
351326
# kubernetes
352-
# msrestazure
353327
# opencensus
354328
# python-dateutil
355329
# from https://pypi.org/simple
@@ -362,6 +336,7 @@ strictyaml==1.7.3
362336
typing-extensions==4.14.1
363337
# via
364338
# aiosignal
339+
# azure-batch
365340
# azure-core
366341
# azure-cosmos
367342
# azure-data-tables

‎pctasks/ingest/requirements.txt‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -236,15 +236,12 @@ pytz==2025.2
236236
# via planetary-computer
237237
# from https://pypi.org/simple
238238
pyyaml==6.0.2
239-
# via
240-
# pctasks-core
241-
# stac-validator
239+
# via pctasks-core
242240
# from https://pypi.org/simple
243241
referencing==0.36.2
244242
# via
245243
# jsonschema
246244
# jsonschema-specifications
247-
# stac-validator
248245
# from https://pypi.org/simple
249246
requests==2.32.4
250247
# via

‎pctasks/ingest_task/requirements.txt‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -273,15 +273,12 @@ pytz==2025.2
273273
# via planetary-computer
274274
# from https://pypi.org/simple
275275
pyyaml==6.0.2
276-
# via
277-
# pctasks-core
278-
# stac-validator
276+
# via pctasks-core
279277
# from https://pypi.org/simple
280278
referencing==0.36.2
281279
# via
282280
# jsonschema
283281
# jsonschema-specifications
284-
# stac-validator
285282
# from https://pypi.org/simple
286283
requests==2.32.4
287284
# via

‎pctasks/notify/requirements.txt‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -210,15 +210,12 @@ pytz==2025.2
210210
# via planetary-computer
211211
# from https://pypi.org/simple
212212
pyyaml==6.0.2
213-
# via
214-
# pctasks-core
215-
# stac-validator
213+
# via pctasks-core
216214
# from https://pypi.org/simple
217215
referencing==0.36.2
218216
# via
219217
# jsonschema
220218
# jsonschema-specifications
221-
# stac-validator
222219
# from https://pypi.org/simple
223220
requests==2.32.4
224221
# via

0 commit comments

Comments
 (0)