Getting v7 running with gcambreakout

[dobrien13]

Good morning GCAM team,

I'm trying to get the United Kingdom broken out from the EU in GCAM v7. I've used the gcambreakout package, creating a new region called United Kingdom (ID 33) with only the UK pulled out from EU-15 (ID 13). This runs successfully.

Now, when running driver() in GCAM Data System v5.1, which worked before the region breakout, I get the following error. Any advice on this would be really helpful.

.
.
.
[1] "module_energy_L154.transportation_UCD"
[1] "- make 79.01"
[1] "module_energy_L2011.ff_ALL_R_C_Y"
[1] "- make 0.30"
[1] "module_energy_L202.Ccoef"
[1] "- make 0.02"
[1] "module_energy_L210.resources"
Error in mutate() at ei-gcamdata/R/utils-data.R:32:3:
ℹ In argument: environCost = if_else(resource == "coal" & region %in% L210.low_reg, 0, environCost).
Caused by error in if_else():
! false must be a vector, not NULL.
Run rlang::last_trace() to see where the error occurred.

[Update]
This error is still occurring even after I've reverted the changes to the input data. Could it be something to do with the R packages that needed to be updated for gcambreakout?

[pkyle]

This is a bug in driver() in GCAM 7 (I think driver_drake() might work OK) that will be corrected in the next release; until then you can just either merge or cherry-pick a couple of commits from my forked repo. In the commands below, I'm just naming my forked repo github_pkyle but you could name it whatever you want:

git remote add github_pkyle https://github.com/pkyle/gcam-core.git
git fetch github_pkyle
git merge gpk/bugfix/gcam7_limitsfix

Or alternatively:

git remote add github_pkyle https://github.com/pkyle/gcam-core.git
git fetch github_pkyle
git cherry-pick f3377fc
git cherry-pick 9f40004

[dobrien13]

Thanks a lot -- pulling the modified zenergy_L210.resources.R file from your repo did the trick. Unfortunately, on the UK-broken-out branch, I'm now getting this issue in driver_drake(). Any ideas?

▶ target L162.bio_YieldRate_R_Y_GLU_irr
▶ target module_water_L171.desalination
✖ fail module_water_L171.desalination
Error: target module_water_L171.desalination failed.
diagnose(module_water_L171.desalination)$error$message:
left_join_no_match: NA values in new data columns
diagnose(module_water_L171.desalination)$error$calls:
gcamdata:::module_water_L171.desalination("MAKE", c(common.iso_GCAM_regID,
aglu.LDS.Land_type_area_ha, water.A71.globaltech_coef, water.AusNWC_desal_techs,
water.EFW_mapping, water.aquastat_ctry, water.basin_to_country_mapping,
water.DesalData_capacity_basin, water.FAO_desal_AQUASTAT,
water.FAO_desal_missing_AQUASTAT, water.nonirrigation_withdrawal,
L1011.en_bal_EJ_R_Si_Fi_Yh))
full_join(L171.out_km3_R_desal_tech_Yh, select(EFW_mapping_desal,
sector, fuel, technology), by = "technology") %>% left_join_error_no_match(L171.desal_fuel_shares,
by = c("GCAM_region_ID", "fuel", "technology", "year")) %>%
mutate(desal_km3 = value * share) %>% select(GCAM_region_ID,
sector, fuel, technology, year, "desal_km3")
select(., GCAM_region_

[pkyle]

I've heard of this error happening when you break out a country in the country-to-region mapping files inst/extdata/common/*.csv but don't have the underlying country-level energy data from the IEA Energy Balances. That's a proprietary dataset and as such can't be distributed. Instead gcamdata contains a binary file data/PREBUILT_DATA.rda with all necessary energy (and emissions) data pre-aggregated by GCAM model region. This specific error just happens to be the first of many such errors that would occur as the model tries to create the additional region, but without any underlying data on energy production and consumption.
Because of the proprietary data issue, external (i.e. outside of PNNL/UMd) users don't have the capability to modify the default country-to-model-region mapping unless they purchase the IEA World Energy Balances.

[dobrien13]

That did it -- we do have the IEA data, and I had just forgotten to add it into the extdata/energy folder.

Next update (thanks for sticking with me through these issues...) -- I'm trying two different ways to get this running and both are stuck. The first, using the gcambreakout function, is stuck on the following:
▶ target module_energy_L144.building_det_en
▶ target module_aglu_resbio_input_IRR_MGMT_xml
▶ target module_aglu_L125.LC_tot
✖ fail module_aglu_L125.LC_tot
Error: target module_aglu_L125.LC_tot failed.
diagnose(module_aglu_L125.LC_tot)$error$message:
ERROR: Interannual fluctuation in global land cover exceeds tolerance threshold of 0.005
diagnose(module_aglu_L125.LC_tot)$error$calls:
gcamdata:::module_aglu_L125.LC_tot("MAKE", c(L120.LC_bm2_R_UrbanLand_Yh_GLU,
L120.LC_bm2_R_Tundra_Yh_GLU, L120.LC_bm2_R_RckIceDsrt_Yh_GLU,
L122.LC_bm2_R_HarvCropLand_Yh_GLU, L122.LC_bm2_R_OtherArableLand_Yh_GLU,
L123.LC_bm2_R_MgdPast_Yh_GLU, L123.LC_bm2_R_MgdFor_Yh_GLU,
L124.LC_bm2_R_Shrub_Yh_GLU_adj, L124.LC_bm2_R_Grass_Yh_GLU_adj,
L124.LC_bm2_R_UnMgdPast_Yh_GLU_adj, L124.LC_bm2_R_UnMgdFor_Yh_GLU_adj))
stop("ERROR: Interannual fluctuation in global land cover exceeds tolerance threshold of ",
aglu.LAND_TOLERANCE)

where the tibble of interest looks like:

A tibble: 2,723 × 6

Groups: GCAM_region_ID, GLU [180]

GCAM_region_ID GLU year value change_rate change

1 1 GLU007 2007 1264. 0.994 -7.70
2 1 GLU023 2007 130. 0.994 -0.834
3 1 GLU027 1975 0.180 0.994 -0.00116
4 1 GLU027 1976 0.182 1.01 0.00174
5 1 GLU027 1977 0.184 1.01 0.00233
6 1 GLU027 1978 0.187 1.02 0.00287
7 1 GLU027 1979 0.188 1.01 0.00152
8 1 GLU027 1980 0.189 1.01 0.00127
9 1 GLU027 1981 0.191 1.01 0.00102
10 1 GLU027 1982 0.192 1.01 0.00159

ℹ 2,713 more rows

ℹ Use print(n = ...) to see more rows

---

The second methodology I'm trying was to manually edit the files listed in gcambreakout and work through the files as needed. On that branch, I'm stuck at the following:

▶ target module_gcamusa_L101.EIA_SEDS
▶ target module_energy_L1011.ff_GrossTrade
✖ fail module_energy_L1011.ff_GrossTrade
Error: target module_energy_L1011.ff_GrossTrade failed.
diagnose(module_energy_L1011.ff_GrossTrade)$error$message:
left_join_no_match: NA values in new data columns
diagnose(module_energy_L1011.ff_GrossTrade)$error$calls:
gcamdata:::module_energy_L1011.ff_GrossTrade("MAKE", c(common.iso_GCAM_regID,
common.GCAM_region_names, emissions.A_PrimaryFuelCCoef, energy.fuel_carbon_content,
energy.mappings.comtrade_countrycode_ISO, energy.mappings.comtrade_commodity_code,
energy.mappings.comtrade_trade_flow, energy.comtrade_ff_trade,
energy.GCAM_region_pipeline_bloc_import, energy.GCAM_region_pipeline_bloc_export))
L1011.comtrade_ff_BiTrade_y_ctry_item_FULL %>% filter(GCAM_Commodity_traded ==
"gas pipeline") %>% group_by(GCAM_region_ID_exporter = reporter_GCAM_region_ID,
GCAM_region_ID_importer = partner_GCAM_region_ID, year, GCAM_Commodity,
GCAM_Commodity_traded) %>% summarise(value = sum(value)) %>%
ungroup() %>% left_join_error_no_match(GCAM_regio

I've seen a similar issue before and believe it had to do something with dbl vs int datatypes in a column but am not positive. Thanks for all the help; appreciate you!

[pkyle]

So I don't know what's going on with the inter-annual fluctuation in land areas; the fact that it's not happening in your manual breakout method means it's probably something to do with the automated breakout not working correctly. It's not something I'd advise spending much time digging into.
The error messages on the second method indicate that perhaps your new model region isn't being added to a gas trade region, or some other mapping file used by the fossil fuel trade code. The gcambreakout package isn't officially maintained as part of gcam-core, and probably just hasn't been updated since gas trade was added. I'd say just check all of the mappings files in the dependencies of that code module_energy_L1011.ff_GrossTrade, and make sure your new region is mapped appropriately to a trading block. Running that code in debug can help to work through it, but note that it's possible the issue is upstream of that code too.

[dobrien13]

Hi @pkyle -- thank you for the help so far! Moving along steadily here with debugging. I'm on to the ceds emissions module and have a relatively vague run error (see below). I'm having difficulty diagnosing as there are so many left_joins in this module. I looked through each of the csv input files required for the module and saw no issues, so it could be either due to a preceding module or due to the code. I read online that "Users that want to build using CEDS raw data, for example to build for different regional aggregations, will need to generate CEDS data using the open-source CEDS system." Is that what's going on here - the CEDS system looks out of date relative to v7 -- any issue there?

Thank you, always!

▶ target elec_segments_water_USA.xml
▶ target module_emissions_L112.ceds_ghg_en_R_S_T_Y
✖ fail module_emissions_L112.ceds_ghg_en_R_S_T_Y
Error: target module_emissions_L112.ceds_ghg_en_R_S_T_Y failed.
diagnose(module_emissions_L112.ceds_ghg_en_R_S_T_Y)$error$message:
left_join_no_match: NA values in new data columns
diagnose(module_emissions_L112.ceds_ghg_en_R_S_T_Y)$error$calls:
gcamdata:::module_emissions_L112.ceds_ghg_en_R_S_T_Y("MAKE",
c(common.GCAM_region_names, common.iso_GCAM_regID, emissions.mappings.CEDS_sector_tech_proc,
emissions.mappings.CEDS_sector_tech_proc_revised, energy.mappings.UCD_techs,
emissions.mappings.UCD_techs_emissions_revised, energy.calibrated_techs,
energy.calibrated_techs_bld_det, emissions.mappings.Trn_subsector,
emissions.mappings.Trn_subsector_revised, emissions.CEDS.CEDS_sector_tech_combustion,
emissions.CEDS.CEDS_sector_tech_combustion_revised, emissions.EPA_FCCC_IndProc_2005,
emissions.mappings.calibrated_outresources, emissions.CEDS.gains_iso_sector_emissions,
emissions.CEDS.gains_iso_fuel_emissions, emissions.IP

[pkyle]

My assumption would be that the CEDS emissions files (e.g., extdata/emissions/CEDS/BC_total_CEDS_emissions.csv) need to be added to the workspace. Without these 10 files, which aren't distributable due to their close association with the proprietary IEA World Energy Balances, the emissions data processing will rely on the PREBUILT_DATA (in this case, the outputs of module_emissions_L102.nonco2_ceds_R_S_Y) that are pre-aggregated to region and distributed with the release.

[Kofimoley]

@dobrien13 Following your exchanges with @pkyle I was hoping to receive some assistance on gcam breakout as well.

We have acquired the energy balance and CEDS data (placed them in the required folders) and successfully broken out a couple of countries. However, during the rebuild of the XML files using the gcamdata system, I keep running into the same error, and I think it has to do with some of our new countries' region ID not being picked up (I could be wrong).

The following is the error I face at the moment so I have not been able to proceed beyond this point. I will be very grateful to receive some assistance here. Thank you.

[pkyle]

Just looking at where the crash happened, my guess is that one of the new regions does not have one of the land use types that the code is assuming to be present. For instance if you're breaking out small island nations there might not be any pasture, and I think I've also crashed in this file when breaking out Uruguay, which has no forested lands in the inventories, but has some logging production. Really the best way to diagnose it is to step into debug mode and execute the code manually, examining where the missing values are coming from. If you temporarily replace left_join_error_no_match() with left_join() where it's crashing, then you will be able to see where exactly the missing values are coming in.

x<-driver_drake(stop_before = "module_aglu_L123.LC_R_MgdPastFor_Yh_GLU")
debug(module_aglu_L123.LC_R_MgdPastFor_Yh_GLU)
module_aglu_L123.LC_R_MgdPastFor_Yh_GLU("MAKE", x)

[dobrien13]

Hi both @pkyle and @Kofimoley --

This is one place driver_drake has caught before for us, and in the past, I believe you pointed me towards a solution, Page. My memory fails me and our org deletes emails after 12 months so I can't find it. Maybe it has to do with this branch? In any case, when running with broken out regions, we now stop the data system before the managed forest chunk. For v6 we use the following process:

5. Follow the steps below:
   a. Before breaking out a region set the constant in R/constants.R aglu.USE_BEFORE_ADJUST_FOREST_FILE to FALSE
   b. Now run up to chunk - module_aglu_LB123.LC_R_MgdPastFor_Yh_GLU. This can be done using driver(stop_after = "module_aglu_LB123.LC_R_MgdPastFor_Yh_GLU")
   c. This will generate the file- L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust in the gcamdata/outputs/ folder
   d. Copy the contents of this file (generated in step 3) to a file gcamdata/inst/extdata/aglu/LDS/L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust . Just copy the contents and don't replace the whole file i.e. make sure the metadata does not change (you should see what is meant here when you open the two files)
   e. Set constant aglu.USE_BEFORE_ADJUST_FOREST_FILE to TRUE R/constants.R
   f. After this , reload gcamdata- devtools::load_all()
   g. Now run driver or driver_drake() as usual

For v7, this:
6. Follow the steps below:
a. Before breaking out a region set the constant in R/constants.R aglu.USE_BEFORE_ADJUST_FOREST_FILE to FALSE
b. Now run up to chunk - module_aglu_LB123.LC_R_MgdPastFor_Yh_GLU. This can be done using driver(stop_after = "module_aglu_L123.LC_R_MgdPastFor_Yh_GLU", write_outputs = FALSE, write_xml = TRUE)
The latter two listed arguments keep your computer from running out of memory while writing the thousands of CSV outputs of driver() as it runs.
c. This will generate the file- L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust in the gcamdata/outputs/ folder
i. In the “v7” project, we have adapted to GCAM v7.0, and instead, a file named “L123beforeadjust.csv” is dropped into the main ei-gcamdata folder. Copy that data into the next step instead.
d. Copy the contents of this file (generated in step 3) to a file gcamdata/inst/extdata/aglu/LDS/L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust . Just copy the contents and don't replace the whole file i.e. make sure the metadata does not change (you should see what is meant here when you open the two files)
e. Set constant aglu.USE_BEFORE_ADJUST_FOREST_FILE to TRUE R/constants.R
f. After this, rebuild gcamdata and run devtools::load_all()
g. Now run driver() or driver(write_outputs = FALSE, write_xml = TRUE) as usual; driver() should run without issue and it should run through every file in the data system.

Note there are some customizations here as we run our own private version of the data system, but it should be generalizable if this is the right fix! @pkyle please correct me if this is a wrong, totally unrelated bug!!

[pkyle]

Oh right, thanks @dobrien13 , I totally forgot about that issue! Yeah I guess this was an issue in one of our releases a few years ago. Your notes are much more detailed than anything I have on hand, though in the gcamdata issue tracker @kanishkan91 did post some instructions JGCRI/gcamdata#1240

[Kofimoley]

Thank you @dobrien13 & @pkyle for the super detailed and helpful feedback.

I followed the steps as described but in the last minute the following error appeared and as a result the file- "L123.LC_bm2_R_MgdFor_Yh_GLU_beforeadjust" was not generated. I went over the steps thrice just to be sure but it kept failing with same error. Would be happy to receive your assistance once again. Thank you.