Conversation
- Manual supplement fix-ups now index uncnewsletter$paperid instead of washdev$paperid, so the curated values land on the intended rows - The correspondence author country is cleaned into its own column instead of overwriting first_author_affiliation_country - Repair Mac Roman bytes in raw washdev.csv cells at read time (openxlsx refused to export the invalid UTF-8) - Regenerate data/*.rda and inst/extdata exports Closes #13 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the Python scrapers in inst/python/ with data-raw/washdev_scraping.R: - chromote drives headless Chrome past the Cloudflare challenge (a plain user agent override suffices); rvest/xml2 parse the pages - Incremental: reads data-raw/washdev.csv, scrapes only newer issues, and checkpoints the CSV after every issue - Collects a new doi column from the citation_doi meta tag - Mixed supplement types are recorded as ' & '-joined lists instead of 'misc', removing the manual repair step for new rows - Fixes the unreachable-code bug in the Python get_issues() Validated field by field against the existing CSV row for paperid 93019 (all 20 metadata fields identical) and against the volume 13 issue 1 table of contents (6/6 paperids). Closes #11 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The washdev scraper is now R (data-raw/washdev_scraping.R, #11) and PLOS Water arrives via the API (data-raw/ploswater.R, #14), so the Python stack is retired: - 17 MB chromedriver binary plus its license files - washdev_scraping.py (expired hardcoded cookies, Cloudflare-blocked) and washdev_scraping_selenium.py (superseded by the R port) - uncnewsletter_scraping.py (candidate-URL collector for a newsletter that ceased publication in May 2024; recoverable from git history) - data_processing.ipynb (superseded by data-raw/data_processing.R) - empty requirements.txt Also narrows .Rbuildignore: the ^inst rule excluded all of inst/ from the built package, including inst/CITATION and inst/extdata. Closes #17 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
data-raw/ploswater.R downloads all PLOS Water articles (436 since volume 1, 2022) through the public search API and each article's JATS XML, pure R with httr2 and xml2, no scraping: - DAS text plus das_repo_url (URLs and dataset DOIs in the statement) and das_repo_name (recognized repositories: zenodo, dryad, figshare, osf, github, dataverse, and others) - Supplementary materials with per-file types and stable download links (no expiring CDN signatures, unlike #10) - First and correspondence author with affiliation, country, ORCID - Subject terms as keywords (PLOS XML carries no author keywords) - Resumable: skips DOIs already in data-raw/ploswater.csv All 333 research articles carry a DAS; 177 articles link out to a data location. Refs #14 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
washdev: 241 new articles from volume 14 (2024) through volume 16 issue 6 (June 2026), now 1173 observations. All new rows carry DOIs. ploswater: harmonized to the shared schema with paperid = DOI and rule-based das_type classification; 320 of 333 statements classify by rule, 13 stay NA pending review. Unresolved values go to review files instead of hand edits, per the reproducibility rule in #12: - data-raw/washdev-das-review.csv (14) and washdev-country-review.csv (33) - data-raw/ploswater-das-review.csv (13) and ploswater-country-review.csv (8) Refs #12, #15 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- R/ploswater.R roxygen documentation and man/ploswater.Rd - Dictionary rows for all 32 ploswater variables; washdev country variable names corrected from *_affiliation_region to *_affiliation_country; uncnewsletter issue_url entry fixed; doi and url_source documented for washdev - README: three-dataset overview, ploswater section, dynamic washdev coverage line, uncnewsletter frozen-source note; rebuilt README.md - DESCRIPTION: version 0.2.0, three datasets in the description - NEWS.md entry for 0.2.0 - .Rbuildignore: replace the overly broad ^inst rule (this edit was described in the #17 cleanup commit but not included in it) R CMD check: 0 errors, 0 warnings, 0 notes. Refs #16 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ploswater: all 436 PLOS Water articles (2022 to July 2026) via the public PLOS API, including where data behind each paper is stored (das_repo_url,das_repo_name) and per-file supplement types (New data source: PLOS Water via the PLOS API (pure R, no scraping) #14, Harmonize ploswater into the shared schema, including das_type classification #15)washdevupdated from 932 to 1173 observations, volume 1 (2011) through volume 16 issue 6 (June 2026), with a newdoicolumn for the 241 newly scraped rows (Update washdev with volumes 14-16 (2024 to mid-2026) #12)inst/python/(17 MB chromedriver included) was removed (Port washdev scraper from Python/Selenium to R with incremental updates #11, Clean up Python tooling in inst/python/ #17)uncnewsletterregenerated (Two bugs in data-raw/data_processing.R: wrong indexing and overwritten first-author country #13).Rbuildignoreexcluding all ofinst/(built packages were missinginst/CITATIONandinst/extdata)data-raw/*-review.csvawaiting decisions on Update washdev with volumes 14-16 (2024 to mid-2026) #12 and Harmonize ploswater into the shared schema, including das_type classification #15; nothing was fixed by handCloses #11, #13, #14, #17. Refs #12, #15, #16 (open until the review decisions on the das_type and country files land).
Test plan
devtools::check()passes with 0 errors, 0 warnings, 0 notesdata/loads three datasets with expected dimensions: washdev 1173 rows, uncnewsletter 204, ploswater 436data-raw/dictionary.csvREADME.mdis current withREADME.Rmdand documents all three datasets