Measure data-availability and OA practice: harvest existing open datasets before scraping

[larnsce]

Background

This issue reframes how we measure data-availability and open-access practice for the journals in scope, after discovering that several signals we planned to scrape or hand-collect already exist as open, identifier-joinable datasets and APIs.

It started as a narrow task ("extract the open-access model per publisher for the ~22 publishers behind the 46 filtered-slice journals") and grew, through a review of external sources, into a three-layer methods design. Recording it here so it can be picked up deliberately. Related: #35 (manual DAS-accessibility survey), #37 (WASH-relevance overestimation), #18 (venue mapping).

The key reframing

The original #35 premise was: manually check, per publisher platform, whether the Data Availability Statement is programmatically extractable, then build bespoke scrapers for it. Reviewing the landscape showed that data-availability detection at scale already exists and is partly published openly (DataSeer and its publisher partners). So the higher-leverage approach is harvest what exists first, scrape only the residual gap, and use the overlap to validate our own scraper (washdev/IWA method).

Three distinct questions were being conflated under "open access"; separating them is the core insight:

Layer	Question	Source	Join key	Granularity
Policy - OA	What OA route/model does the journal/publisher permit?	Jisc Open Policy Finder (ex-Sherpa Romeo)	ISSN	journal + publisher
Policy - data	What data-sharing does the journal/publisher require?	TOP Factor	ISSN	journal
Behavior - observed	What did articles actually do (DAS present, data/code shared, repository used)?	PLOS OSI, AAAS/Dryad, DataCite	DOI	article

Permitted vs required vs observed. All identifier-joinable, all with open APIs or downloads. Our bespoke DAS scraping (washdev/IWA) becomes the fallback for the behavioral layer where no open detection dataset covers a journal, not the primary method.

Sources discovered

Behavioral layer (observed practice)

• DataSeer (https://dataseer.ai/) - NLP detection of open-science indicators (data/code sharing, repositories, preprints, protocols, registrations). Its Open Science Metrics product does journal/publisher/funder-level aggregate reporting. Partners include PLOS, Taylor & Francis, AAAS/Science, IOP, Royal Society. Not open per se, but its engine output is published via partners.
• PLOS Open Science Indicators - https://plos.figshare.com/articles/dataset/PLOS_Open_Science_Indicators/21687686 (DOI 10.6084/m9.figshare.21687686). ~139k PLOS articles + ~28k non-PLOS comparators, 2018-2025, updated annually, open download. Per-article: data sharing, data repository location, code sharing, preprints, protocols, study registrations. Covers PLoS ONE + PLoS NTD directly, plus a cross-publisher comparator set.
• AAAS/Science Open Science Metrics - https://datadryad.org/dataset/doi:10.5061/dryad.zkh1893qt (DOI 10.5061/dryad.zkh1893qt). 2,680 Science articles 2021-2024, with columns Data_Generated, Data_Shared, Data_Location, Data_Sharing_Repositories, DAS presence, Code_Shared. Small, but a clean schema template for the table we want, and includes PLOS + T&F comparators.
• DataCite API - reverse links from article DOIs to deposited dataset DOIs. Finds shared data independent of whether the DAS text is scrapable; complementary signal.

Policy layers

• Jisc Open Policy Finder (https://openpolicyfinder.jisc.ac.uk/, API https://api.openpolicyfinder.jisc.ac.uk) - 2024 merger of Sherpa Romeo/Juliet/Fact. 3,503 publisher OA policies, 28,000 journal OA policies, 178 funder policies. Open API, joins by ISSN. Replaces the plan's fragile "scrape 22 publisher OA marketing pages" phase. NOTE: legacy Sherpa/Romeo endpoints die April 2026; target the new API. Caveat: its OA policy is about self-archiving/green-OA permissions and route, adjacent to but not identical to gold-vs-hybrid business model; pair with OpenAlex is_oa/is_in_doaj for the empirical model flag.
• TOP Factor (COS) - https://osf.io/kgnva/ (top-factor.csv, ~4.2 MB, ~3,200 journals scored, updated Mar 2025). Per-journal scores on 8 TOP transparency standards; the two relevant columns are Data transparency score and Data citation score. Measures required data policy, NOT OA model. Join by ISSN. Name-only match hit 19/46 of our journals (a floor); ISSN join should lift it, but some WASH-core venues (IWA titles, ACS ES&T Water) may simply not be scored.

Identifier / empirical base

• OpenAlex sources - per journal: ISSN-L, ISSNs (the join key everything else needs), host_organization (publisher), is_oa, is_in_doaj, apc_usd, works-level oa_status mix (gold/hybrid/green/bronze/closed). One GET per source; covers all 46+3 journals.

Planned coverage assessment

For the 46 filtered-slice + 3 IWA journals, joined by ISSN or DOI:

1. Pull identifiers + empirical OA flags from OpenAlex (join key + OA-model base).
2. Query Jisc Open Policy Finder by ISSN (OA policy layer). Needs an API key (user to register; store as OPENPOLICYFINDER_KEY in ~/.Renviron, matching the OPENALEX_API_KEY pattern).
3. Join TOP Factor by ISSN (data policy layer).
4. Measure coverage of PLOS OSI + AAAS/Dryad; query DataCite reverse-links (behavioral layer).
5. The residual - journals covered by no behavioral source - defines the actual bespoke-scraping scope for Manual DAS-accessibility check for the 46 filtered-slice journals #35. Where sources overlap our own scraping, use them to validate the washdev/IWA DAS detector's precision/recall (a measured-accuracy step the pipeline currently lacks).

Honest limits

• Publisher-level OA label is a simplification: for the big commercial houses (Elsevier, Springer, Wiley, T&F, ACS) the truthful answer is "hybrid portfolio"; lead with the per-journal breakdown.
• APC coverage in OpenAlex is patchy (mainly DOAJ gold); missing is not zero.
• TOP Factor and the behavioral datasets will have genuine gaps for the WASH-specific venues; report exact hit rates and name every uncovered journal rather than imply full coverage.
• Open Policy Finder's OA policy is self-archiving/route, not business model; pair with OpenAlex.

Status

Plan only; no data pulled yet. Waiting on the Open Policy Finder API key before the OA-policy layer can be queried. Other sources are pullable now.

---

Landscape research (added): tools and pipelines, not just datasets

A follow-up landscape sweep surfaced a third option the datasets-only framing missed: run open-source detection tools ourselves over full text, sitting between "harvest someone's dataset" and "build a bespoke scraper." Plus national-scale open pipelines that already solve the whole permit/require/observe problem.

Runnable open-source detection tools (the key addition)

DataSeer is commercial, but validated open equivalents exist, several as R packages (this is an R project):

• ODDPub - https://github.com/quest-bih/oddpub (QUEST/BIH Charité). R package detecting Open Data and Open Code statements. Validated vs manual screening: sensitivity 0.73, specificity 0.97 (data) / 1.00 (code). Training data: https://osf.io/yv5rx/. Closest open analog to what Manual DAS-accessibility check for the 46 filtered-slice journals #35 wanted to build; callable directly from R.
• rtransparent - https://github.com/serghiou/rtransparent. R package extracting data availability, code availability, conflict-of-interest, funding, and protocol-registration statements from PMC XML. Broader than ODDPub.
• Softcite / DataStet - https://github.com/softcite/software-mentions - ML models (GROBID-based) for software and dataset mentions in full text; power the French monitor.
• Commercial/other in the same family: SciScore, Ripeta, ContriBot, ctRegistries.
• Benchmark: "Use as Directed?" (arXiv 2507.17991) compares 7 tools (rtransparent, oddpub, ctRegistries, ContriBot, DataSeer, SciScore, LLM) across 9 rigor criteria with per-tool per-signal F1 scores. Plus a Peer Review Congress diagnostic-accuracy study on major medical journals. These give published sensitivity/specificity to pick tools by measured accuracy - the ground-truth validation the pipeline currently lacks.

National / infrastructure-scale open pipelines (build templates)

• French Open Science Monitor (BSO) - https://www.ouvrirlascience.fr/the-open-science-monitor/ - government monitor of OA + research data + code sharing, entirely open method: open data, open code on GitHub, published methodology, built only from open sources (Unpaywall, OpenAlex, DOAJ, Crossref, PubMed) + Softcite/DataStet. A complete reproducible template for our exact measurement; read before designing our own.
• OpenAIRE MONITOR - https://monitor.openaire.eu/ - dashboards over the OpenAIRE Graph (publications, datasets, software, DMPs) with data download.
• PATHOS Open Science Impact Indicator Handbook - https://handbook.pathos-project.eu/ - per-indicator (data use, code use, reproducibility, OA) recommended source/tool. Peer-reviewed version of our ad hoc source-selection; should anchor choices.
• Landscape catalog of OA dashboards: Nature Scientific Data, https://www.nature.com/articles/s41597-026-07217-z.

OA-status and repository sources beyond #38's original list

• Unpaywall - https://unpaywall.org/ - canonical per-DOI OA-status API (oa_status: gold/green/hybrid/bronze/closed), 100k calls/day, bulk snapshot, R client roadoi. Cleaner empirical OA-status signal than the OpenAlex flags; used by the French monitor. Add to the OA layer alongside OpenAlex/Open Policy Finder.
• re3data - https://www.re3data.org/ - registry of ~3,500 research-data repositories; normalizes where shared data lands (our washdev scraper captures das_repo_url but does not normalize it).
• DataCite / Google Dataset Search / DataMed - dataset-discovery layers (DataCite already listed).

Added plan step: evaluate detection tools on our own data

Trial ODDPub and rtransparent on a sample of our journals' full text (Europe PMC OA XML, publisher XML where available) and measure their accuracy against our own scraped DAS ground truth (washdev/IWA). If accuracy holds, this could replace much of the bespoke #35 scraping.

Honest caveats on the tool family

• Nearly all these detectors are validated on biomedical literature; accuracy on WASH / water-engineering journals is unverified. Measure on our scraped ground truth before trusting.
• Most rely on full-text XML, freely available only for OA content; paywalled articles remain a genuine gap that no open tool closes.

Refs #18, #35, #37