fix(spatialize): stop gridded output carrying two disagreeing territorial keys - #653
Merged
Conversation
`build_gridded_landuse()` and `build_gridded_livestock()` allocate on the raw reporting codes their `country_areas`/`country_grid` are keyed on, while whep's polity-keyed national tables are aggregated on `polity_area_code`. A reporting code that is not itself a bucket therefore left every output row carrying two territorial keys that disagree -- `area_code = 276` beside `polity_area_code = 206` -- so whether a consumer joined on one or the other decided whether Sudan existed in its result. Measured on the deployed pins: `country_grid` holds 831 such cells under 276 Sudan and 277 South Sudan; a scoped 2020 run put 13,447 crop rows / 21.89 Mha and 3,671 livestock rows / 230.7 M head on a key `get_primary_production()` does not publish. `area_key` selects the vocabulary. `"grid"` is the default, reproduces today's output bit-for-bit, and now warns naming the codes that cannot join. `"polity_area"` re-keys before the polity columns are attached, so the two keys agree in every row; the raw code is carried, not replaced, as `grid_area_code`, the shape `build_cell_polity()` adopted in whep#579. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # NEWS.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
build_gridded_landuse()/build_gridded_livestock()allocate from anational table keyed on
area_codeand into acountry_gridkeyed thesame way, so both sides speak the raw reporting vocabulary the grid was
rasterized in. whep's polity-keyed national tables are aggregated on
polity_area_codeinstead, and a reporting code need not equal its bucket.Both pins are internally consistent, so nothing warned and nothing was
dropped inside the engine. The mismatch is at the output boundary: the row's
own
area_codestayed276while the.add_reporting_polity_columns()callcorrectly added
polity_area_code = 206beside it. Every such row carriedtwo territorial keys that disagree, and which one a consumer picked decided
whether Sudan and South Sudan existed in its result.
Evidence it reproduced BEFORE the change
At
origin/main(bd225575), against the deployedWHEP_*pins.The claim is half stale, and I re-measured rather than trusting it. #582
lists 8 off-bucket grid codes covering 935 cells and 12 in
country_areas.Six of those — 212 Syria, 154 North Macedonia, 61 Equatorial Guinea, 153 New
Caledonia, 209 Eswatini, 299 Palestine — plus 69 French Guiana and the four
FAO "Other" region aggregates are polity_area_codes today, because #628
gave the Rest-of-World members their own published codes. What survives:
country_grid.parquet(58,795 cells, 178 codes)276,277country_areas.parquet(1,032,750 rows, 198 codes)276,277The other half of the issue's premise is confirmed, and it is the one that
matters.
get_primary_production(years = 2020)on today's main returnsarea_codeidentical topolity_area_code(208 codes):206is present,276and277are not. So a gridded row keyed276joins nothing.Driving the real engine over the real pins, scoped to the Sudan/South Sudan
cells in 2020:
What I changed
area_key = c("grid", "polity_area")on both engines —config$area_keyforbuild_gridded_landuse()(which caps its signature at five arguments), atop-level argument for
build_gridded_livestock(), and anoverrideskey onrun_spatialize()so the option reaches the production entry point."grid"— the default, reproduces today's output bit-for-bit, and nowemits the diagnostic today's silence hides, naming the codes and cells that
cannot join. It reuses
.warn_cell_polity_off_bucket(), so there is onemessage for this failure in the package, not two.
"polity_area"— resolves each code to its bucket throughpolity_area_crosswalkbefore the polity columns are attached, soarea_codeandpolity_area_codeagree in every row. It respectsoptions(whep.unfold_rest_of_world).I followed #579's precedent literally: the raw code is carried, not
replaced. Under
"polity_area"the output gainsgrid_area_codeholdingthe code the engine allocated on, joined with
+where two reporting areas ofone bucket meet in a cell and their rows collapse — picking a winner there
would be the silent half of the same problem. A code the crosswalk does not
carry keeps its own code, so a gap stays visible instead of becoming an
NAkey.
I read
edu/polycell-spatial-support/ #619 first. It touches these two filesbut only the
cell_area_fraccontract, the compartment join columns and thecapacity-breach report — it does not touch the area vocabulary, and its
own body argues for exactly this shape ("#579's
grid_area_codeas an addedcolumn was the right shape"). My fixtures supply
cell_area_fracexplicitlyso they survive #619's stricter grid contract. Conflicts should be limited to
the roxygen block and the one line before
.add_reporting_polity_columns().How I verified
Gates, all from
Rscript --vanilla:air format .(binary, 0.10.0) ·devtools::document()·lintr::lint_package()0 lints ·rcmdcheck(--no-tests)0 errors / 0 warnings / 0 notes ·devtools::test()FAIL 0 | SKIP 8 | PASS 6417, with 10 newtest_thatblocks · pkgdown
commcheck clean (no new documented topics).Numbers, on the real pins.
area_key = "grid"isidentical()to thepre-change output of the same call.
area_key = "polity_area":manure_n_mgdiff 0276, 277→ none276, 277→ none206Structural checks, because "no number moved" is not sufficient (#561/#563).
Rows added/removed: 0 —
country_gridis a centroid crosswalk, soSudan's 629 cells and South Sudan's 202 are disjoint and nothing merges on
this pin; the merge path is exercised on a fixture instead. Keys: 2 removed
(
276,277), 1 added (206); one column added (grid_area_code), noneremoved. One
area_code→ onearealabel:206→SUD-1956-2011/"Sudan (1956-2011)", one pair for one code — andget_primary_production(years = 2020)'s own206rows carry that samepolity code and name, so the re-keyed grid agrees with the national table on
identity as well as on the key.
The new guards are load-bearing — proved by planting three mutants and
watching them die, then restoring and re-confirming green:
"polity_area"silently behaves like"grid"(the defect itself):13 assertions fail across both engines.
(
grid_area_codeexpected"276+277").Suite restored to
FAIL 0 | PASS 216on the spatialize files after each.Moves published values
No, unless
area_key = "polity_area"is asked for. The default path isidentical()to before; the only user-visible change on it is a new warning.Under the opt-in, 831 cells move off a key no national table carries.
What I deliberately did not do
country_grid.parquet,country_areas.parquet) is @eduaguilera's sequencing and collides withReplace the two cell-polity crosswalks with the polycell, and move the area denominators onto it #619; resolving at the engine boundary works against pins as they stand.
901/903/904/906.They are not off-bucket — they are their own
polity_area_codes — so thisinvariant cannot see them, and Gridded land-use/livestock output carries two disagreeing territorial keys: raw area_code next to its own polity_area_code #582 says explicitly they "need a separate
call". Filed as a follow-up rather than decided here.
country_areas.parquetandits
.prod_cache.parquetdate from 2026-06-09 and predate Model the Rest-of-World reporting members in their own right (#459) #628, so the pin'svocabulary lags production's by 21 areas. Filed separately.
.warn_unallocated_crops()'s zero-cell half (Feed crop/livestock spatialize the fractional cell-coverage grid, not the centroid grid #461).The open decision
Should
"polity_area"become the default? I defaulted to the status quobecause that is what #579 did for
build_cell_polity()one release earlier,and because flipping moves published values for anyone joining gridded output
on
area_code. But under"grid"the acceptance criterion "no output rowcarries an
area_codethat is not apolity_area_code" holds only when theoption is passed — the default merely stops being silent about it. The
argument for flipping: the movement is fully measured, mass- and
row-conserving, confined to Sudan/South Sudan, and moves those rows from a key
that matches nothing to one that matches the national table. The argument
against:
run_spatialize()writes the published parquets, and a consumerpinned on
276would silently start seeing206.@eduaguilera / @lbm364dl — one line in
.landuse_config_defaults()and.spatialize_presets()either way.Closes #582
Part of the polity migration epic #458.
🤖 Generated with Claude Code