Harvests, DOI backfill, FAIR rubric, and platforms evaluation

DESCRIPTION

@@ -1,6 +1,6 @@
 Package: washopenresearch
 Title: Dataset about open research data information in Water, Sanitation, and Hygiene
-Version: 0.2.0
+Version: 0.2.0.9000
 Authors@R: c(
     person("Mian", "Zhong", , "mzhong@ethz.ch", role = c("aut", "cre"),
            comment = c(ORCID = "0009-0009-4546-7214")),
@@ -13,8 +13,13 @@ Description: The goal of washopenresearch is to provide an overview of open rese
 License: CC BY 4.0
 Encoding: UTF-8
 Roxygen: list(markdown = TRUE)
-Depends: 
+Depends:
     R (>= 3.5)
+Imports:
+    dplyr
+Suggests:
+    europepmc,
+    rcrossref
 LazyData: true
 Config/Needs/website: rmarkdown
 Date: 2026-07-08

NAMESPACE

@@ -1,2 +1,3 @@
 # Generated by roxygen2: do not edit by hand
 
+export(score_fair)

NEWS.md

@@ -1,3 +1,32 @@
+# washopenresearch (development version)
+
+## New features
+
+- New scripted acquisition pipeline for a fourth dataset, `datapapers`, covering
+  WASH-related data papers in seven dedicated data journals (Scientific Data,
+  Data in Brief, Gates Open Research, F1000Research, GigaScience, GigaByte,
+  and Data (MDPI)) (#28). The pipeline lives in
+  `data-raw/01_datapapers_acquire.R` (Crossref/Europe PMC harvest with a
+  committed raw snapshot), `data-raw/02_datapapers_screen.R` (relevance
+  screening captured in a committed decision sheet keyed on DOI), and
+  `data-raw/03_datapapers_process.R` (harmonisation to the shared schema and
+  export). The dataset itself is added once the first harvest and screening
+  round are complete.
+
+## Minor improvements and fixes
+
+- Expired pre-signed CDN links in `washdev$supp_url` are rewritten to stable DOI
+  URLs, and Google Scholar alert redirects in `uncnewsletter$paper_url` are
+  decoded to their target URLs (#10). The 343 Silverchair links carried a
+  January 2024 expiry, and the high-entropy signature tokens tripped secret
+  scanners; the article DOI is recovered from the link path, so no re-collection
+  is needed. Two helpers in `data-raw/helpers.R`, `canonicalize_silverchair_url()`
+  and `decode_scholar_redirect()`, do the rewrites reproducibly.
+- The list-column collapsing helper and shared country-cleaning steps moved to
+  `data-raw/helpers.R`, sourced by all processing scripts.
+- `data-raw/README.md` documents the run order and provenance of every
+  committed snapshot and decision sheet.
+
 # washopenresearch 0.2.0
 
 ## New features

R/datapapers.R

@@ -0,0 +1,41 @@
+#' Dataset about WASH data papers published in dedicated data journals
+#'
+#' Candidate WASH-related data papers harvested from Crossref and Europe PMC
+#' for seven data journals (Scientific Data, Data in Brief, Gates Open
+#' Research, F1000Research, GigaScience, GigaByte, and Data), screened for
+#' relevance with a committed decision sheet. Data papers describe a shared
+#' dataset, so the repository link takes the role that the data availability
+#' statement variables play in `washdev` and `uncnewsletter`.
+#'
+#' @format ## `datapapers`
+#'
+#' \describe{
+#'   \item{paperid}{ID number of the paper within this dataset}
+#'   \item{doi}{DOI of the data paper}
+#'   \item{paper_url}{Official url of the paper (DOI resolver link)}
+#'   \item{url_source}{Publisher website of the paper}
+#'   \item{journal}{Full name of the journal}
+#'   \item{title}{Title of the paper}
+#'   \item{published_year}{Year of publication}
+#'   \item{num_authors}{Number of the authors}
+#'   \item{first_author_name}{Name of the first author}
+#'   \item{first_author_affiliation}{Academic affiliation of the first author}
+#'   \item{first_author_affiliation_country}{Country of the first author parsed from first_author_affiliation variable encoded with United Nations names}
+#'   \item{data_repo_url}{Website urls of the repository holding the dataset the paper describes, separated by "; " when there are multiple}
+#'   \item{data_repo}{Name of the data repository (e.g. Zenodo, Dryad, Figshare, OSF, Dataverse) parsed from data_repo_url}
+#'   \item{license}{License url of the paper from Crossref metadata}
+#'   \item{related_paper_doi}{DOI of a linked research article, if any (e.g. Data in Brief co-submissions), separated by "; " when there are multiple}
+#'   \item{abstract}{Abstract of the paper as provided by the metadata source}
+#'   \item{query_term}{WASH search term(s) that retrieved the paper, separated by "; "}
+#'   \item{retrieval_date}{Date the paper metadata was harvested from the API}
+#' }
+#' @source Crossref (<https://api.crossref.org>) and Europe PMC
+#'   (<https://europepmc.org>); see `data-raw/README.md` for the pipeline.
+#'
+#' @docType data
+#' @keywords datasets
+#' @name datapapers
+# Documented via @name/NULL rather than the "datapapers" string sentinel so
+# devtools::document() runs before data/datapapers.rda exists; the dataset
+# lands once the issue #28 screening sheet is filled and processed.
+NULL

R/score_fair.R

@@ -0,0 +1,127 @@
+#' Score a paper's data management against the FAIR principles
+#'
+#' Scores each paper on the four FAIR dimensions (Findable, Accessible,
+#' Interoperable, Reusable) from the data-sharing fields the package already
+#' records. Each dimension is scored 0, 1, or 2, and `fair_total` is their sum
+#' (0 to 8). The rubric is deliberately simple and documented so the scoring is
+#' reproducible and open to criticism (issue #19); it is a screening instrument,
+#' not a certified FAIR assessment.
+#'
+#' @details
+#' The four dimensions are scored as follows.
+#'
+#' \strong{Findable} (persistent identifier and registered location):
+#' \itemize{
+#'   \item 2: the data is in a registered repository (`das_repo_name` is set)
+#'     and a repository link or dataset DOI is present.
+#'   \item 1: a data location is stated (`das_type` is "available in online
+#'     repository", or a `das_repo_url` is present) but without a recognised
+#'     repository, or the data is in the paper or its supplement.
+#'   \item 0: no data location (no DAS, "on request", or "not shareable").
+#' }
+#'
+#' \strong{Accessible} (can a reader get the data without a barrier):
+#' \itemize{
+#'   \item 2: a repository or supplement link is present (`das_repo_url` or
+#'     `supp_url`), so the data is directly retrievable.
+#'   \item 1: the data is stated to be in the paper or supplement but no link is
+#'     recorded.
+#'   \item 0: "on request", "not shareable", or no DAS.
+#' }
+#'
+#' \strong{Interoperable} (open, machine-readable shared formats), from
+#' `supp_file_type`:
+#' \itemize{
+#'   \item 2: any open machine-readable format (csv, txt, tsv, json, xml).
+#'   \item 1: structured but proprietary formats only (xlsx, docx, sav, dta).
+#'   \item 0: unstructured only (pdf, images), or no shared files.
+#' }
+#'
+#' \strong{Reusable} (license and repository metadata). Because a license column
+#' is not yet collected for these datasets, this dimension is scored from the
+#' repository signal as a lower bound:
+#' \itemize{
+#'   \item 2: data in a recognised repository (`das_repo_name` set), which
+#'     normally carries a license and rich metadata.
+#'   \item 1: a data location is stated but not in a recognised repository.
+#'   \item 0: no shared data.
+#' }
+#' When a `license` column is added (see issue #19), raise this dimension to use
+#' it directly.
+#'
+#' Note that Accessible scores the shared files, not the authors' intent. A
+#' paper whose `das_type` is "on request" or "not shareable" can still score 2
+#' on Accessible if it ships a supplement with a `supp_url`, because that
+#' supplement is directly retrievable. The `das_type` value stays visible
+#' alongside the score, so a restricted-data paper that still shares a
+#' supplement is distinguishable from a fully open one.
+#'
+#' @param data A data frame with the columns `has_das`, `das_type`,
+#'   `das_repo_url`, `das_repo_name`, and `supp_file_type`. The washdev,
+#'   uncnewsletter, and ploswater datasets all carry these. `das_repo_name` is
+#'   optional; when absent, the repository signal falls back to `das_repo_url`.
+#'
+#' @return `data` with five integer columns added: `fair_findable`,
+#'   `fair_accessible`, `fair_interoperable`, `fair_reusable`, and `fair_total`.
+#'
+#' @examples
+#' scored <- score_fair(ploswater)
+#' table(scored$fair_total)
+#'
+#' @export
+score_fair <- function(data) {
+  needed <- c("has_das", "das_type", "supp_file_type")
+  missing <- setdiff(needed, names(data))
+  if (length(missing)) {
+    stop("score_fair() needs column(s): ", paste(missing, collapse = ", "),
+         call. = FALSE)
+  }
+
+  das_type <- as.character(data$das_type)
+  repo_url <- if ("das_repo_url" %in% names(data)) data$das_repo_url else NA_character_
+  repo_name <- if ("das_repo_name" %in% names(data)) data$das_repo_name else NA_character_
+  supp_url <- if ("supp_url" %in% names(data)) data$supp_url else NA_character_
+  supp_type <- data$supp_file_type
+
+  has_repo_name <- !is.na(repo_name) & nzchar(repo_name)
+  has_repo_url <- !is.na(repo_url) & nzchar(repo_url)
+  has_supp_url <- !is.na(supp_url) & nzchar(supp_url)
+  in_repo_das <- !is.na(das_type) & das_type == "available in online repository"
+  in_paper_das <- !is.na(das_type) & das_type == "in paper"
+  barrier_das <- !is.na(das_type) & das_type %in% c("on request", "not shareable")
+
+  findable <- dplyr::case_when(
+    has_repo_name & (has_repo_url | in_repo_das) ~ 2L,
+    in_repo_das | has_repo_url | in_paper_das ~ 1L,
+    TRUE ~ 0L
+  )
+
+  accessible <- dplyr::case_when(
+    has_repo_url | has_supp_url ~ 2L,
+    in_paper_das ~ 1L,
+    barrier_das | is.na(das_type) ~ 0L,
+    TRUE ~ 0L
+  )
+
+  open_fmt <- "\\b(csv|txt|tsv|json|xml)\\b"
+  struct_fmt <- "\\b(xlsx|xls|docx|doc|sav|dta|rds|parquet)\\b"
+  supp_lc <- tolower(ifelse(is.na(supp_type), "", supp_type))
+  interoperable <- dplyr::case_when(
+    grepl(open_fmt, supp_lc) ~ 2L,
+    grepl(struct_fmt, supp_lc) ~ 1L,
+    TRUE ~ 0L
+  )
+
+  reusable <- dplyr::case_when(
+    has_repo_name ~ 2L,
+    in_repo_das | has_repo_url | in_paper_das ~ 1L,
+    TRUE ~ 0L
+  )
+
+  data$fair_findable <- findable
+  data$fair_accessible <- accessible
+  data$fair_interoperable <- interoperable
+  data$fair_reusable <- reusable
+  data$fair_total <- findable + accessible + interoperable + reusable
+  data
+}

R/uncnewsletter.R

@@ -3,7 +3,6 @@
 #' @format ## `uncnewsletter`
 #'
 #' \describe{
-#'   \item{url_source}{Publisher website of the paper}
 #'   \item{paperid}{ID number of the paper on the journal website}
 #'   \item{issue_url}{Volume number of the journal}
 #'   \item{paper_url}{Official website url of the paper}
@@ -31,5 +30,6 @@
 #'   \item{das_type}{Type of the data availability statement including in paper(data in full paper scope like supplementary material or appendix or main content) on request(data available on request to the authors) available in online repository(data is shared in a public online repository) not shareable(data is not shareable). NA if it does not have a data availability statement.}
 #'   \item{das_repo_url}{Website urls of the data if the relevant data of the paper is shared on a public repository, separated by "; " when there are multiple}
 #'   \item{keywords}{Keywords of the paper, separated by "; "}
+#'   \item{doi}{DOI of the paper, backfilled via a Crossref title search (issue #20); NA where no match cleared the title-similarity threshold.}
 #' }
 "uncnewsletter"

R/washdev.R

@@ -31,6 +31,6 @@
 #'   \item{das_repo_url}{Website urls of the data if the relevant data of the paper is shared on a public repository, separated by "; " when there are multiple}
 #'   \item{keywords}{Keywords of the paper, separated by "; "}
 #'   \item{url_source}{Publisher website of the paper}
-#'   \item{doi}{DOI of the paper. Collected since the R port of the scraper; NA for articles scraped earlier, to be backfilled via Crossref.}
+#'   \item{doi}{DOI of the paper. Collected by the R scraper for recent articles and backfilled via Crossref for legacy rows (issue #20); NA where no Crossref match was found.}
 #' }
 "washdev"

README.Rmd

@@ -38,7 +38,7 @@ library(wordcloud2)
 
 The goal of washopenresearch is to provide an overview of open research
 data related to Water Sanitation and Hygiene (WASH). The current version
-contains three datasets from the following sources:
+contains four datasets from the following sources:
 
 -   `washdev`: Open access journal [*Journal of Water, Sanitation and
     Hygiene for Development*](https://iwaponline.com/washdev)
@@ -47,6 +47,16 @@ contains three datasets from the following sources:
     News](https://waterinstitute.unc.edu/our-work/nc-water-news-newsletter)
 -   `ploswater`: Open access journal [*PLOS
     Water*](https://journals.plos.org/water/)
+-   `datapapers`: WASH-related data papers in seven dedicated data
+    journals ([Scientific
+    Data](https://www.nature.com/sdata/), [Data in
+    Brief](https://www.sciencedirect.com/journal/data-in-brief), [Gates
+    Open Research](https://gatesopenresearch.org),
+    [F1000Research](https://f1000research.com),
+    [GigaScience](https://academic.oup.com/gigascience),
+    [GigaByte](https://gigabytejournal.com), and
+    [Data](https://www.mdpi.com/journal/data)), harvested from Crossref
+    and Europe PMC
 
 ![Word cloud of the most frequent keywords in articles of the Journal of Water, Sanitation and Hygiene for Development, with water, sanitation, and hygiene appearing largest](man/figures/washdev_wordcloud.png){width="515"}
 
@@ -81,12 +91,12 @@ read_csv("data-raw/dictionary.csv") |>
 
 ## Data
 
-The package provides access to three datasets `washdev`,
-`uncnewsletter`, and `ploswater`. Each dataset collects information on
-scientific articles about (1) article metadata (e.g. title, first
-author, correspondence author), (2) supplementary material information,
-(3) data availability statement, and (4) semantic information (e.g.
-keywords).
+The package provides access to four datasets `washdev`,
+`uncnewsletter`, `ploswater`, and `datapapers`. Each dataset collects
+information on scientific articles about (1) article metadata (e.g.
+title, first author, correspondence author), (2) supplementary material
+information, (3) data availability statement or linked data repository,
+and (4) semantic information (e.g. keywords or abstract).
 
 ```{r}
 library(washopenresearch)
@@ -174,6 +184,34 @@ readr::read_csv("data-raw/dictionary.csv") |>
   kableExtra::scroll_box(height = "200px")
 ```
 
+### datapapers
+
+The dataset `datapapers` contains WASH-related data papers published in
+seven dedicated data journals, identified from Crossref and Europe PMC
+metadata and screened for relevance (see `data-raw/README.md` for the
+pipeline). It has `r nrow(datapapers)` observations. Because a data
+paper exists to describe a shared dataset, `data_repo_url` and
+`data_repo` take the role that the data availability statement variables
+play in the other two datasets.
+
+```{r}
+datapapers |>
+  head(3) |>
+  gt::gt() |>
+  gt::as_raw_html()
+```
+
+For an overview of the variable descriptions, see the following table.
+
+```{r echo=FALSE, message=FALSE, warning=FALSE}
+readr::read_csv("data-raw/dictionary.csv") |>
+  dplyr::filter(file_name == "datapapers.rda") |>
+  dplyr::select(variable_name:description) |>
+  knitr::kable() |>
+  kableExtra::kable_styling("striped") |>
+  kableExtra::scroll_box(height = "200px")
+```
+
 ## Example
 
 ### washdev
@@ -261,6 +299,20 @@ uncnewsletter |>
    theme_classic()
 ```
 
+### datapapers
+
+1.  How many papers per journal, and how many resolve to a data
+    repository?
+
+```{r}
+datapapers |>
+  group_by(journal) |>
+  summarise(papers = n(),
+            with_repository_link = sum(!is.na(data_repo_url))) |>
+  arrange(desc(papers)) |>
+  knitr::kable()
+```
+
 ## Method
 
 We describe the raw data collection procedure of each dataset in this
@@ -286,6 +338,17 @@ publication's html file using the publication url. The retrieval is
 rule-based to find the relevant fields (e.g. supplementary materials)
 and extract the value.
 
+### datapapers
+
+The collection of `datapapers` is fully scripted in R. Crossref is
+queried by journal ISSN and Europe PMC by journal name (for the
+F1000-platform journals) with a fixed list of WASH search terms; the
+harvest is committed as a raw snapshot with the retrieval date and
+matching query terms recorded per row. Relevance screening and country
+corrections are captured in committed CSV decision sheets keyed on DOI,
+so the pipeline runs end-to-end non-interactively. See
+`data-raw/README.md` for the run order.
+
 ### uncnewsletter
 
 The collection of `uncnewsletter` is a combination of web scraping and