Skip to content

fix(spatialize): stop gridded output carrying two disagreeing territorial keys - #653

Merged
lbm364dl merged 2 commits into
mainfrom
polity/582
Aug 7, 2026
Merged

fix(spatialize): stop gridded output carrying two disagreeing territorial keys#653
lbm364dl merged 2 commits into
mainfrom
polity/582

Conversation

@lbm364dl

@lbm364dl lbm364dl commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

What was wrong

build_gridded_landuse() / build_gridded_livestock() allocate from a
national table keyed on area_code and into a country_grid keyed the
same way, so both sides speak the raw reporting vocabulary the grid was
rasterized in. whep's polity-keyed national tables are aggregated on
polity_area_code instead, and a reporting code need not equal its bucket.

Both pins are internally consistent, so nothing warned and nothing was
dropped inside the engine. The mismatch is at the output boundary: the row's
own area_code stayed 276 while the .add_reporting_polity_columns() call
correctly added polity_area_code = 206 beside it. Every such row carried
two territorial keys that disagree
, and which one a consumer picked decided
whether Sudan and South Sudan existed in its result.

Evidence it reproduced BEFORE the change

At origin/main (bd225575), against the deployed WHEP_* pins.

The claim is half stale, and I re-measured rather than trusting it. #582
lists 8 off-bucket grid codes covering 935 cells and 12 in country_areas.
Six of those — 212 Syria, 154 North Macedonia, 61 Equatorial Guinea, 153 New
Caledonia, 209 Eswatini, 299 Palestine — plus 69 French Guiana and the four
FAO "Other" region aggregates are polity_area_codes today, because #628
gave the Rest-of-World members their own published codes. What survives:

table off-bucket codes extent
country_grid.parquet (58,795 cells, 178 codes) 276, 277 831 cells, 1.41% of the grid
country_areas.parquet (1,032,750 rows, 198 codes) 276, 277 866,712,208 ha, 0.639% of harvested area

The other half of the issue's premise is confirmed, and it is the one that
matters. get_primary_production(years = 2020) on today's main returns
area_code identical to polity_area_code (208 codes): 206 is present,
276 and 277 are not. So a gridded row keyed 276 joins nothing.

Driving the real engine over the real pins, scoped to the Sudan/South Sudan
cells in 2020:

build_gridded_landuse():   13,447 rows, 21,894,526 ha
  rows whose area_code != polity_area_code:  13,447 of 13,447
  rows joinable to national area_code 206:        0 of 13,447
build_gridded_livestock():  3,671 rows, 230,685,264 head
  rows whose area_code != polity_area_code:   3,671 of  3,671

What I changed

area_key = c("grid", "polity_area") on both engines — config$area_key for
build_gridded_landuse() (which caps its signature at five arguments), a
top-level argument for build_gridded_livestock(), and an overrides key on
run_spatialize() so the option reaches the production entry point.

  • "grid" — the default, reproduces today's output bit-for-bit, and now
    emits the diagnostic today's silence hides, naming the codes and cells that
    cannot join. It reuses .warn_cell_polity_off_bucket(), so there is one
    message for this failure in the package, not two.
  • "polity_area" — resolves each code to its bucket through
    polity_area_crosswalk before the polity columns are attached, so
    area_code and polity_area_code agree in every row. It respects
    options(whep.unfold_rest_of_world).

I followed #579's precedent literally: the raw code is carried, not
replaced.
Under "polity_area" the output gains grid_area_code holding
the code the engine allocated on, joined with + where two reporting areas of
one bucket meet in a cell and their rows collapse — picking a winner there
would be the silent half of the same problem. A code the crosswalk does not
carry keeps its own code, so a gap stays visible instead of becoming an NA
key.

I read edu/polycell-spatial-support / #619 first. It touches these two files
but only the cell_area_frac contract, the compartment join columns and the
capacity-breach report — it does not touch the area vocabulary, and its
own body argues for exactly this shape ("#579's grid_area_code as an added
column was the right shape"). My fixtures supply cell_area_frac explicitly
so they survive #619's stricter grid contract. Conflicts should be limited to
the roxygen block and the one line before .add_reporting_polity_columns().

How I verified

Gates, all from Rscript --vanilla: air format . (binary, 0.10.0) ·
devtools::document() · lintr::lint_package() 0 lints ·
rcmdcheck(--no-tests) 0 errors / 0 warnings / 0 notes ·
devtools::test() FAIL 0 | SKIP 8 | PASS 6417, with 10 new test_that
blocks · pkgdown comm check clean (no new documented topics).

Numbers, on the real pins. area_key = "grid" is identical() to the
pre-change output of the same call. area_key = "polity_area":

landuse livestock
rows 13,447 → 13,447 3,671 → 3,671
mass 21,894,526 ha → 21,894,526 ha (diff 0) 230,685,264 head → 230,685,264 (diff 0); manure_n_mg diff 0
disagreeing keys 13,447 → 0 3,671 → 0
off-bucket codes 276, 277none 276, 277none
joinable to national 206 0 → 13,447

Structural checks, because "no number moved" is not sufficient (#561/#563).
Rows added/removed: 0country_grid is a centroid crosswalk, so
Sudan's 629 cells and South Sudan's 202 are disjoint and nothing merges on
this pin; the merge path is exercised on a fixture instead. Keys: 2 removed
(276, 277), 1 added (206); one column added (grid_area_code), none
removed. One area_code → one area label: 206
SUD-1956-2011 / "Sudan (1956-2011)", one pair for one code — and
get_primary_production(years = 2020)'s own 206 rows carry that same
polity code and name, so the re-keyed grid agrees with the national table on
identity as well as on the key.

The new guards are load-bearing — proved by planting three mutants and
watching them die, then restoring and re-confirming green:

  1. "polity_area" silently behaves like "grid" (the defect itself):
    13 assertions fail across both engines.
  2. The fold picks the first raw code instead of joining both: 2 fail
    (grid_area_code expected "276+277").
  3. The default stops warning: 2 fail.

Suite restored to FAIL 0 | PASS 216 on the spatialize files after each.

Moves published values

No, unless area_key = "polity_area" is asked for. The default path is
identical() to before; the only user-visible change on it is a new warning.
Under the opt-in, 831 cells move off a key no national table carries.

What I deliberately did not do

The open decision

Should "polity_area" become the default? I defaulted to the status quo
because that is what #579 did for build_cell_polity() one release earlier,
and because flipping moves published values for anyone joining gridded output
on area_code. But under "grid" the acceptance criterion "no output row
carries an area_code that is not a polity_area_code" holds only when the
option is passed — the default merely stops being silent about it. The
argument for flipping: the movement is fully measured, mass- and
row-conserving, confined to Sudan/South Sudan, and moves those rows from a key
that matches nothing to one that matches the national table. The argument
against: run_spatialize() writes the published parquets, and a consumer
pinned on 276 would silently start seeing 206.

@eduaguilera / @lbm364dl — one line in .landuse_config_defaults() and
.spatialize_presets() either way.

Closes #582
Part of the polity migration epic #458.

🤖 Generated with Claude Code

lbm364dl and others added 2 commits August 7, 2026 13:45
`build_gridded_landuse()` and `build_gridded_livestock()` allocate on the
raw reporting codes their `country_areas`/`country_grid` are keyed on, while
whep's polity-keyed national tables are aggregated on `polity_area_code`. A
reporting code that is not itself a bucket therefore left every output row
carrying two territorial keys that disagree -- `area_code = 276` beside
`polity_area_code = 206` -- so whether a consumer joined on one or the other
decided whether Sudan existed in its result.

Measured on the deployed pins: `country_grid` holds 831 such cells under 276
Sudan and 277 South Sudan; a scoped 2020 run put 13,447 crop rows / 21.89 Mha
and 3,671 livestock rows / 230.7 M head on a key `get_primary_production()`
does not publish.

`area_key` selects the vocabulary. `"grid"` is the default, reproduces
today's output bit-for-bit, and now warns naming the codes that cannot join.
`"polity_area"` re-keys before the polity columns are attached, so the two
keys agree in every row; the raw code is carried, not replaced, as
`grid_area_code`, the shape `build_cell_polity()` adopted in whep#579.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lbm364dl
lbm364dl merged commit d2e3e3c into main Aug 7, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gridded land-use/livestock output carries two disagreeing territorial keys: raw area_code next to its own polity_area_code

1 participant