Verify in-paper DAS claims and backfill datapapers repository links

.gitignore

@@ -11,3 +11,10 @@
 data-raw/*_scrape.log
 data-raw/iwa_scrapes.log
 
+# Personal notes, not part of the package
+python-scripts-lesson.md
+
+# Downloaded supplement files (issue #47 tier 3); the committed record is
+# data-raw/suppfiles/manifest.csv with sha256 checksums
+data-raw/suppfiles/files/
+

NAMESPACE

@@ -1,3 +1,4 @@
 # Generated by roxygen2: do not edit by hand
 
+export(das_in_paper_support)
 export(score_fair)

NEWS.md

@@ -2,6 +2,34 @@
 
 ## New features
 
+- New function `das_in_paper_support()` classifies how a "data in paper"
+  data availability statement is backed by the recorded supplement fields
+  (#47). The modal claim "all relevant data are included in the paper or its
+  supplementary information" splits into `"no supplement"` (the claim rests
+  on the printed tables alone), `"unstructured supplement"` (pdf or images
+  only), `"structured supplement"` (docx, xlsx and similar), and
+  `"open supplement"` (csv, txt, json, xml). Across washdev and the three
+  IWA journal snapshots (2,599 claims), 71.5% have no supplement, 26.5% a
+  structured one, 2.1% an unstructured one, and none an open format.
+  `data-raw/das_in_paper_support.R` reproduces the summary in
+  `data-raw/das-in-paper-support-summary.csv`. A follow-up cross-check
+  against OpenAlex article types (`data-raw/das_in_paper_article_types.R`,
+  committed DOI-to-type lookup) shows the no-supplement claims are almost
+  entirely substantive research articles: only 0.5% are front matter or
+  reviews, so the article-type filter proposed in #47 does not shrink the
+  population. Verifying the remaining 1,847 bare claims requires reading
+  the articles' tables.
+- Supplement content audit (#47 tier 3): the 741 supplementary files of the
+  IWA snapshots whose pre-signed CDN links were still valid were downloaded
+  (checksummed manifest in `data-raw/suppfiles/manifest.csv`; the signatures
+  lapse 2026-08-18 to 2026-08-30, so 572 further links were already dead)
+  and classified with transparent structural heuristics
+  (`data-raw/suppfiles_parse.R`: pandoc-converted docx tables, readxl/xlsx
+  sheet metrics). Of the 290 in-paper claims whose structured supplement was
+  in hand, 54% share prose only, 21% summary tables, and 20% tables shaped
+  like observations (`data-raw/das_in_paper_suppfile_audit.R`). xlsx files
+  are the exception: 25 of 29 hold observation-shaped sheets. Heuristics are
+  recorded per file and await validation against a manual sample.
 - New scripted acquisition pipeline for a fourth dataset, `datapapers`, covering
   WASH-related data papers in seven dedicated data journals (Scientific Data,
   Data in Brief, Gates Open Research, F1000Research, GigaScience, GigaByte,
@@ -15,6 +43,14 @@
 
 ## Minor improvements and fixes
 
+- `datapapers` now carries the repository links its papers deposit to:
+  `data_repo_url` and `data_repo` were NA for all 8 papers because the
+  Crossref relation metadata was empty and the fallback planned for #27
+  never ran. The links were verified against Crossref relations, DataCite
+  resource types, and the articles' availability sections, and are recorded
+  in `data-raw/datapapers_repo_fixes.csv`, applied during processing. Seven
+  papers use general repositories (GBIF, IEEE DataPort, Figshare, Dryad,
+  NCBI BioProject, Zenodo); none uses a WASH sector platform.
 - Expired pre-signed CDN links in `washdev$supp_url` are rewritten to stable DOI
   URLs, and Google Scholar alert redirects in `uncnewsletter$paper_url` are
   decoded to their target URLs (#10). The 343 Silverchair links carried a

R/das_in_paper_support.R

@@ -0,0 +1,90 @@
+#' Classify how a "data in paper" claim is backed by the supplement fields
+#'
+#' The stock statement "all relevant data are included in the paper or its
+#' supplementary information" is the modal data availability statement across
+#' the IWA journal snapshots. As a single category it is nearly uninformative:
+#' it covers papers whose complete dataset genuinely fits in the printed
+#' tables as well as papers whose tables hold only summary statistics
+#' (issue #47). This function scores the claim jointly with the supplement
+#' fields already recorded, so "in paper plus xlsx supplement" is
+#' distinguished from "in paper, nothing attached".
+#'
+#' @details
+#' A paper makes the in-paper claim when `das_type` is the normalized value
+#' `"in paper"` (washdev style) or contains the stock phrasing
+#' "included in the paper" / "included in the article" (the IWA journal
+#' snapshots keep the raw sentence in `das_type`). Papers without the claim
+#' get `NA`.
+#'
+#' Claims are classified by the strongest attachment that could carry the
+#' data, using the same format tiers as [score_fair()]:
+#' \itemize{
+#'   \item `"open supplement"`: an open machine-readable format is attached
+#'     (csv, txt, tsv, json, xml).
+#'   \item `"structured supplement"`: structured but proprietary formats only
+#'     (xlsx, xls, docx, doc, sav, dta, rds, parquet).
+#'   \item `"unstructured supplement"`: a supplement exists but only as pdf
+#'     or images, or its format is unknown.
+#'   \item `"no supplement"`: nothing is attached; the claim rests entirely
+#'     on the printed tables and figures.
+#' }
+#'
+#' The classification is structural: it says where the claimed data could
+#' be, not whether it is actually there. Verifying the content of printed
+#' tables and supplement files is the follow-up work in issue #47.
+#'
+#' @param data A data frame with the columns `das_type` and `supp_file_type`.
+#'   `is_supp`, `num_supp`, and `supp_url` are used when present to detect
+#'   supplements with an unknown format. The washdev, uncnewsletter, and
+#'   ploswater datasets and the IWA journal snapshots in `data-raw/` all
+#'   carry these.
+#'
+#' @return `data` with one character column added: `das_in_paper_support`.
+#'   `NA` for papers that do not make the in-paper claim.
+#'
+#' @examples
+#' classified <- das_in_paper_support(washdev)
+#' table(classified$das_in_paper_support, useNA = "ifany")
+#'
+#' @export
+das_in_paper_support <- function(data) {
+  needed <- c("das_type", "supp_file_type")
+  missing <- setdiff(needed, names(data))
+  if (length(missing)) {
+    stop("das_in_paper_support() needs column(s): ",
+         paste(missing, collapse = ", "), call. = FALSE)
+  }
+
+  das_type <- as.character(data$das_type)
+  supp_type <- tolower(ifelse(is.na(data$supp_file_type), "",
+                              data$supp_file_type))
+  supp_url <- if ("supp_url" %in% names(data)) data$supp_url else NA_character_
+  is_supp <- if ("is_supp" %in% names(data)) data$is_supp else NA
+  num_supp <- if ("num_supp" %in% names(data)) data$num_supp else NA_integer_
+
+  claim <- !is.na(das_type) & (
+    das_type == "in paper" |
+      grepl("included in the (paper|article)", das_type, ignore.case = TRUE)
+  )
+
+  # The IWA snapshots serialize supp_url as a list literal, "[]" when empty,
+  # so emptiness cannot be tested with nzchar() alone.
+  has_supp_url <- !is.na(supp_url) & nzchar(supp_url) &
+    !grepl("^\\s*\\[\\s*\\]\\s*$", supp_url)
+  has_supp <- (is_supp %in% TRUE) |
+    (!is.na(num_supp) & num_supp > 0) |
+    has_supp_url |
+    nzchar(supp_type)
+
+  open_fmt <- "\\b(csv|txt|tsv|json|xml)\\b"
+  struct_fmt <- "\\b(xlsx|xls|docx|doc|sav|dta|rds|parquet)\\b"
+
+  data$das_in_paper_support <- dplyr::case_when(
+    !claim ~ NA_character_,
+    grepl(open_fmt, supp_type) ~ "open supplement",
+    grepl(struct_fmt, supp_type) ~ "structured supplement",
+    has_supp ~ "unstructured supplement",
+    TRUE ~ "no supplement"
+  )
+  data
+}