Add BII v2.1.1 (De Palma et al. 2024) as second model in calcValidBII - #38
Merged
Conversation
Phase 1.1 of extending the BII validation benchmark to 2020.
The NHM data portal is behind Cloudflare; the CKAN resource URL
returns HTTP 403 to non-browser clients. Uses the portal's
/downloads/direct/<sha>.zip endpoint (content-addressable, stable
while the file content is unchanged). Falls back to a clear
manual-download instruction if the URL becomes unreachable.
Smoke-tested end-to-end: 43.8 MB download, nested unzip, 5 GeoTIFFs
(bii-{2000,2005,2010,2015,2020}_v2-1-1.tif) + LICENSE/README in the
source folder.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1.2 of extending the BII validation benchmark to 2020. Returns a cellular magclass on the 67420-cell lpjcell grid (x.y.iso naming, fraction 0-1) for 2000, 2005, 2010, 2015, 2020. Country aggregation is left to calcValidBII, matching readMehta2024 / readSoilCarbonDebt. Pipeline: terra::aggregate(fact=6, fun=mean) brings 5-arcmin to 30-arcmin (unweighted mean -- intra-block cell-area variation <1% even at 80 deg lat); terra::project pins to the canonical 0.5-deg grid (defensive against future v2.x raster origin shifts); terra::extract at the 67420 canonical coords from mstools::toolGetMappingCoord2Country; divide by 100 for unit-parity with readBII (Phillips fraction). Data sub-dim named 'variable' (not the cellular-read default 'data') so calcValidBII can use getNames(x, dim='variable') <- ... uniformly on both v1 and v2 inputs. Smoke-tested against the actual v2.1.1 rasters: BRA 2010 v2 = 0.65, DEU = 0.28, IDN = 0.60 after country aggregation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…a, bump 2.68.0
Phase 1.3 of extending the BII validation benchmark to 2020.
Aggregates cellular biiV2 (from readBIIv2) to country level via the
canonical pik-piam NA-handling idiom (toolNAreplace masks both value
AND weight to 0 where either is NA; toolAggregate then dispatches
without NA propagation), then mbinds with country-level Phillips
under variable 'Biodiversity|BII' with distinct model labels:
- 'Phillips et al' (1970-2014, decadal pre-2000 + annual 2000-2014)
- 'De Palma et al 2024' (2000-2020, 5-yearly)
DESCRIPTION: Version 2.67.0 -> 2.68.0, Imports += terra.
Three implementation notes:
1. madrat::toolAggregate weighted path does x * weight internally,
so NAs in x propagate; without pre-cleaning, ~90 of 234 countries
silently zero out (one NA cell per country is enough). toolNAreplace
is the canonical fix (cf. mrcommons::calcLanduseIntensity:95,
mrvalidation::calcValidTau:42). zeroWeight='setNA' returns NA for
countries without lpjcell coverage rather than 0.
2. magclass parses dots in dim entries as sub-dim separators, so
nm='De Palma et al v2.1.1' fragments dim 3 into 5 sub-dims.
The bare label 'De Palma et al 2024' mirrors 'Phillips et al';
v2.1.1 lineage is in the source name.
3. mbind unions one differing dim at a time. v1 covers 1970-2014,
v2 covers 2000-2020 -- both temporal and data dims differ. Pad
both to the year union with NA first.
End-to-end verified over HPC fuse-t mount: calcOutput('ValidBII')
in 2.2 min; 249 ISO x 20 year x 2 model magclass; BRA 2010 v2 = 0.6549,
DEU = 0.2800, IDN = 0.6048 (within 1% of local terra::cellSize preview).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e/bii-v2 # Conflicts: # .buildlibrary # .github/workflows/check.yaml # CITATION.cff # DESCRIPTION # README.md
pvjeetze
requested changes
May 28, 2026
…ps download, bump 2.69.1 Address #38 review: - Collapse readBIIv2/downloadBIIv2 into readBII/downloadBII with a dataset-selector subtype ("Phillips" default / "DePalma"). - Phillips downloads once via the NHM /downloads/direct content-addressable endpoint (both NHM resource URLs are Cloudflare-blocked for non-browser clients), with an .rds fallback for older source folders; De Palma reads the v2.1.1 rasters. - calcValidBII gains a datasource arg ("Phillips"/"DePalma") returning one source; fullVALIDATION calls it once per source, so a missing De Palma (a pre-staged Limited Release) does not drop Phillips. - @author set to F. Humpenoeder only on the new/changed functions. - Docstring records the verified v1->v2 coefficient-vs-land-use decomposition (coefficient term ~0.17 of the ~0.20 global gap, dominant in every H12 region; land-use/forest ~0.03).
Contributor
|
Thanks for the fixes! I missed that De Palma is diverging by such a margin, also from Philipps. This does explain that different coefficients drive the diverging trajectories. The original differences in the validation of the FSEC paper were primarily driven by the different forest area. Thanks for the clarification! Ready to merge in my view. |
Contributor
|
Also ready from my perspective. Thank you for this work Florian! |
mscrawford
approved these changes
May 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds De Palma et al. 2024 (NHM BII v2.1.1) as a second historical BII source alongside Phillips et al. 2021, extending historical BII coverage to 2020. Bumps to 2.69.1.
readBII/downloadBIItake asubtypeto select the dataset ("Phillips"default /"DePalma"). Both download via the NHM/downloads/direct/<sha>.zipendpoint (the CKAN URLs hit Cloudflare 403).calcValidBII(datasource = "Phillips"|"DePalma")returns one source undervariable = "Biodiversity|BII";fullVALIDATIONcalls it once per source:Phillips et al— decadal 1970-1990, annual 2000-2014De Palma et al 2024— 5-yearly 2000-2020 (cellular lpjcell grid → ISO viatoolNAreplace+toolAggregate(to="iso", zeroWeight="setNA"))The two series use different methodologies and are not on the same absolute scale, so they are kept as separate model labels (not spliced).
Verification
make buildclean (R CMD check, lintr, tests). Both datasources run end-to-end on the HPC mount: global land-area-weighted BII 2010 ≈ 0.77 (Phillips) / 0.60 (De Palma).Methodological note (in
@description)De Palma 2024 uses a stricter primary-vegetation baseline, logit transform, and balanced Bray-Curtis similarity, vs. Phillips's all-primary baseline, log transform, and asymmetric Jaccard — so its absolute BII is markedly lower, with a non-uniform gap. MAgPIE's BII (
modules/44_biodiversity, Leclère et al. 2020 coefficients) is on the Phillips scale. A 2010 decomposition of the MAgPIE–De Palma gap (−0.20): the PREDICTS coefficient change accounts for −0.17 and the land-use/forest difference (MAgPIE's LUH3 + FAO/FRA vs. the LUH2 behind De Palma) for +0.03 — so coefficients dominate (in all 12 H12 regions), with forest area a real but secondary factor.