Skip to content

Screen datapapers candidates and build the dataset - #30

Merged
larnsce merged 4 commits into
mainfrom
dev
Jul 23, 2026
Merged

Screen datapapers candidates and build the dataset#30
larnsce merged 4 commits into
mainfrom
dev

Conversation

@larnsce

@larnsce larnsce commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a self-contained HTML review app for screening the datapapers candidates (search-term highlighting in title and abstract, keyboard-driven include/exclude with reasons, progress persisted in the browser, CSV export), plus the generator that rebuilds it from the worklist and the apply script that merges exported decisions into the committed screening sheet keyed on DOI.
  • Records the screening outcome: the 45 auto-relevant candidates with an abstract were reviewed (8 included, 37 excluded, each with a reason). The 155 title-only candidates were deliberately left unreviewed and stay include = NA, which keeps them out of builds while the sheet stays honest about what was never judged.
  • Builds the datapapers dataset (8 papers, 2018-2025, in Data, GigaByte, and Scientific Data) with CSV/XLSX exports, re-knits the README with all four datasets rendering for the first time, and removes six stale README figure PNGs orphaned by chunk renumbering. Merging closes Identify wash data that's in data papers #28.

Test plan

  • data/datapapers.rda loads with 8 rows and the documented columns
  • Screening sheet totals match the decisions export (8 included, 37 excluded, 3715 pending)
  • README.md shows the datapapers section with row count and download links
  • devtools::check() reports no errors

larnsce added 4 commits July 23, 2026 22:54
Screening 200 auto-relevant candidates in a raw CSV is slow and
error-prone. The review app is a single self-contained HTML file: cards
with metadata, DOI link, and WASH search terms highlighted in title and
abstract, keyboard-driven include/exclude with reasons, progress
persisted in the browser's localStorage, and a CSV export of decisions.

make_datapapers_review_app.R builds the app from the worklist and the
committed template, dropping DOIs already decided in the screening
sheet, so rebuilding after each applied batch yields the remaining
queue. apply_datapapers_decisions.R merges the exported decisions into
data-raw/datapapers_screening.csv keyed on DOI; the sheet stays the
committed decision record. The committed datapapers_review.html is the
instrument as last built (155 pending title-only candidates).

Refs #28

Assisted-by: Claude claude-fable-5
The 45 auto-relevant candidates with an abstract were reviewed in the
HTML review app: 8 included, 37 excluded, each with a reason.
datapapers_decisions.csv is the app's export; the apply script merged
it into the screening sheet keyed on DOI.

The 155 title-only candidates were deliberately left unreviewed and
stay include = NA in the sheet: pending rows are excluded from builds,
and the sheet remains honest about what was never judged rather than
recording blanket exclusions.

Refs #28

Human-authored: true
Assisted-by: Claude claude-fable-5
The 8 included papers become data/datapapers.rda plus the CSV/XLSX
exports, processed by 03_datapapers_process.R from the committed
screening sheet. One affiliation needed the country fixes sheet
(Universidade Jean Piaget de Cabo Verde is Cabo Verde). All 8 rows
have data_repo_url NA because the harvest carried no dataset-relation
metadata; collecting those links from the papers is where issue #27
picks up.

README.Rmd renders with all four datasets for the first time. The
chunk renumbering orphaned the previous README figure PNGs, so the six
stale ones are removed; the pkgdown docs/ site keeps its own copies
until its next rebuild. devtools::check() passes with 0 errors,
0 warnings, 0 notes.

Closes #28

Assisted-by: Claude claude-fable-5
The generated datapapers_review.html is a build artifact: it embeds
only the candidates still pending at build time, so it goes stale
after every applied decision batch. Screening is done, and the file
is rebuildable at any time from the committed template via
make_datapapers_review_app.R, so the repo keeps the method and drops
the artifact.

Refs #28

Assisted-by: Claude claude-fable-5
@larnsce
larnsce merged commit ac817e1 into main Jul 23, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Identify wash data that's in data papers

1 participant