Conversation
Scaffolds the WASH & FSM bibliometric corpus agreed in #18: a keyword list grouped into blocks, a two-table schema (works + authorships) that captures author position and institution country for the planned authorship-equity follow-up, and an OpenAlex harvest skeleton whose intermediate outputs answer the journal-selection question in #18. Refs #18 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MWoEfPvd291T6sbRAxGqP9
Implements the plan from data-raw/plan-issue-28-datapapers.md: - data-raw/01_datapapers_acquire.R: harvest candidate WASH data papers from Crossref (by journal ISSN) and Europe PMC (F1000-platform journals) for Scientific Data, Data in Brief, Gates Open Research, F1000Research, GigaScience, GigaByte, and Data (MDPI), recording retrieval_date and query_term per row, deduplicated on DOI, written as a committed snapshot data-raw/datapapers_raw.csv. - data-raw/02_datapapers_screen.R: auto-relevance flag plus an append-only screening decision sheet keyed on DOI; human decisions are never overwritten, replacing interactive inspection and hard-coded ID vectors. - data-raw/03_datapapers_process.R: harmonise included papers to the washdev/uncnewsletter schema plus data-paper-specific columns (doi, data_repo_url, data_repo, license, related_paper_doi), clean countries via a committed fixes sheet, export rda/csv/xlsx. - data-raw/helpers.R: shared collapse_list_col(), country cleaning, journal and search-term lists; data_processing.R now sources it. - data-raw/README.md: run order and provenance for all three sources. - Package integration: R/datapapers.R roxygen docs, dictionary.csv rows, README.Rmd sections, NEWS.md entry, DESCRIPTION dev version bump and Suggests: rcrossref, europepmc. The harvest itself (step 01) requires network access to api.crossref.org / Europe PMC and must be run by a maintainer; the snapshot, screening sheet, data object, exports, Rd files, and README re-knit follow from the documented run order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BpiR3vepy87ibd5KHytLoh
…lysis-xc5fpn Add washbiblio corpus staging (keywords, schema, harvest skeleton)
Resolves conflicts against the 0.2.0 release: version set to 0.2.0.9000, datapapers NEWS entry moved to a development-version section, README and dictionary keep both ploswater and datapapers, data_processing.R keeps the encoding helper and sources the shared helpers.R. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes harvest_corpus.R steps 1-5 and commits the three outputs: researcher_ranking.csv (top 200 authors, 178 retained after the low-share false-positive screen), venue_ranking.csv (302 venues ranked by keyword match and by retained-researcher output), and journal_wash_share.csv (50 journals with real 1996-2026 WASH shares and the issue #18 decision per journal; 4 whole_journal, 46 filtered_slice_conditional). Matching uses title_and_abstract.search rather than fulltext search, the two big group_bys go through a direct GET to stay within the OpenAlex usage-priced request budget, every network step is cached under cache/, and an approximate-share screen adds small dedicated journals that the volume-dominated rank cutoff would miss. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3760 unique candidates from Crossref (Scientific Data, Data in Brief, GigaScience, GigaByte, Data) and Europe PMC (F1000Research, Gates Open Research), harvested 2026-07-23. The screening sheet prefills auto_relevant (200 TRUE) from title/abstract keyword match; include and reason await manual decisions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scoping sheet and rubric for thirteen platforms (data-raw/platforms/), seeded with mWater and Project W as the paper's focus and screening the long-list (WPdx, JMP, GLAAS, SDG 6, DHS, MICS, World Bank Microdata, IBNET, HDX, Akvo, KoboToolbox) into in/borderline/out of scope. Every fact carries a source_url and accessed_date (2026-07-23), with 'not found' recorded literally where a public page did not state a fact. scan_platform_mentions.R searches the das text and repository links of washdev, uncnewsletter, and ploswater (and datapapers once built) for each platform. Finding: across 1,782 papers, neither mWater nor Project W is mentioned once; only general platforms appear (DHS 12, HDX 3, MICS 2, IBNET 1). The three intro paragraphs for this are added to the vignette. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-researched the launch year, access model, and licensing that the public catalogue pages did not state. Verified via the Wayback Machine and Aquaya's own milestones and terms pages: Project W is a 2022 pilot still in beta, access is waitlist/invite-based behind a sign-in, and the terms of use grant only personal non-commercial use with no open license on the indexed datasets. Updated platforms.csv, rubric.csv, and the vignette paragraph. An exact launch date and whether an API or DOIs exist behind the login remain not found without an account. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
343 washdev supp_url values were pre-signed Silverchair CDN links that expired in January 2024, and 3 uncnewsletter paper_url values were Google Scholar alert redirects. Both are now canonical: the Silverchair links become https://doi.org/<doi> URLs (the DOI is recovered from the link path), and the Scholar redirects are decoded to their target URLs. The high-entropy Signature and scisig tokens that tripped secret scanners are gone. No re-collection was needed. Two reproducible helpers in data-raw/helpers.R do the rewrites: canonicalize_silverchair_url() and decode_scholar_redirect(). Only washdev$supp_url and uncnewsletter$paper_url changed; row counts and all other columns are identical, and ploswater is untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The R scraper only collects doi for newly scraped washdev rows; 932 legacy rows and all uncnewsletter rows had none. backfill_dois.R fills them from Crossref: washdev by volume + issue + normalised title against the journal's two ISSNs (897 matched, 35 for review), uncnewsletter by per-row bibliographic title search accepted only above a 0.6 trigram-similarity threshold (167 of 173 matched, 6 near-misses for review). Crossref intermittently answers 429 even at polite spacing, so the lookup retries with backoff; without it 39 rows landed in the review file as false no-hits. The matches live in committed backfill CSVs that data_processing.R joins back in, so the fill is reproducible and every unmatched row stays visible in the review CSVs rather than hidden in a hand-edit. washdev now carries a DOI on 1138 of 1173 rows, uncnewsletter on 167 of 173. The doi column is documented in the dictionary and roxygen docs for both datasets. ploswater artifacts changed only by re-serialisation during the dataset rebuild. Refs #20 Assisted-by: Claude claude-fable-5
score_fair() scores each paper 0-2 on Findable, Accessible, Interoperable, and Reusable from the DAS and supplement fields the datasets already carry, with fair_total their sum (0-8). The rubric is spelled out in the function's roxygen so the scoring stays reproducible and open to criticism; Reusable falls back to the repository signal as a lower bound until a license column exists. data-raw/fair_scores.R applies it across the three datasets and writes fair-scores-summary.csv with per-year means. Headline means of 8: washdev 1.72, uncnewsletter 2.13, ploswater 4.07, consistent with PLOS Water's mandatory-DAS policy. First exported function of the package: dplyr joins Imports and NAMESPACE gains export(score_fair). Refs #19 Assisted-by: Claude claude-fable-5
make_datapapers_worklist.R builds a one-off review file of the 200 auto-relevant candidates with matched_terms columns showing which WASH search terms hit and where (title vs abstract), so genuine WASH data papers separate from incidental matches like "wastewater of a gold mine" during manual screening. Title-only matches without an abstract, the weakest signal, sort to the top. The script never writes datapapers_screening.csv; the screening sheet stays the decision record. R/datapapers.R now documents the dataset via @name/NULL instead of the string sentinel, so devtools::document() runs before data/datapapers.rda exists. The dataset itself lands once the screening sheet is filled and 03_datapapers_process.R runs. Refs #28 Assisted-by: Claude claude-fable-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan