Conversation
The key is registered in ~/.Renviron and lifts the OpenAlex daily budget from $0.10 to $1, but openalexR only sends it as an HTTP header the usage-priced API may ignore. Map it to openalexR.apikey for the oa_fetch calls and attach it as an api_key query parameter in oa_group_top200(), the route that registers reliably. The script still runs keyless when the variable is absent. Assisted-by: Claude claude-fable-5
Whole-journal scrape of three IWA journals whose 1996-2026 WASH share crossed the #18 threshold: Journal of Water and Health (2013 articles), AQUA (1819), and Water Supply (4884). Every article is captured, not only WASH-matched ones, so data-availability practice can be compared within each journal between matched and non-matched articles. data-raw/iwa_scraping.R generalizes washdev_scraping.R into a parameterized IWA scraper: it discovers issues from the site's own year/issue widgets, paces requests to stay under the Silverchair throttle, and survives headless-Chrome crashes by respawning the session and retrying the issue. run_iwa_scrapes.R chains the three journals sequentially with a cooldown between them; each is resumable. pull_matched_works.R pulls the OpenAlex WASH-matched work lists per journal (matched-works-*.csv) to supply a per-article wash_matched flag at processing time. das_platform_mapping.csv maps the 46 filtered-slice journals to publishers for the deferred #35 survey. Raw scrape outputs and issue manifests are committed for reproducibility, matching the washdev/ploswater precedent. Refs #32, #33, #34 Assisted-by: Claude claude-fable-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
data-raw/iwa_scraping.R, a parameterized IWA scraper generalized fromwashdev_scraping.R: it discovers issues from the site own year/issue widgets, paces requests under the Silverchair throttle, and survives headless-Chrome crashes by respawning the session and retrying the issue.run_iwa_scrapes.Rchains the three journals sequentially with a cooldown between them; each is resumable.pull_matched_works.Randmatched-works-*.csv(OpenAlex WASH-matched work lists per journal) to supply a per-articlewash_matchedflag at processing time, plusdas_platform_mapping.csvmapping the 46 filtered-slice journals to publishers for the deferred Manual DAS-accessibility check for the 46 filtered-slice journals #35 survey.OPENALEX_API_KEYintoharvest_corpus.R(lifts the OpenAlex daily budget from $0.10 to $1).Test plan