Build the harvest orchestrator with local and GitHub backends

[larnsce]

Goal

The harvest orchestrator (data-raw/harvest.R) that builds the catalog from a GitHub organisation, with two interchangeable backends so it is testable offline.

Rewritten 2026-09-04 per dev/decisions-2026-09.md. Start from data-raw/harvest.R and data-raw/harvest_helpers.R on the closed PR #15 branch.

Design

• Backends behind one source interface (read_file, read_file_media, list_files, meta): GitHub (one paginated repo list, then per repo the raw fetches of DESCRIPTION, dictionary.csv, CITATION.cff plus contents listings of data/, R/, inst/extdata; about 5 API requests per repo) and local clones (GitHub-only columns come back NA). The org is an argument with default openwashdata.
• Package detection: not archived, not in exclude_repos.csv, root DESCRIPTION with a Package field, at least one data/*.rda.
• published is TRUE when CITATION.cff carries a Zenodo DOI. reviewed is NA (see Define and ship the catalog data model #5). Dataset and variable detail is harvested for published packages only.
• Error tolerance: per-repo tryCatch; failures accumulate in a problems table written to data-raw/harvest_report.md. The run errors only if a shipped table would be empty or more than 10 percent of detected packages failed to parse.
• Value-level gates, in the run and in test-schema.R: no <U+xxxx> escapes anywhere (the run must be in a UTF-8 locale), no DOI shared by two packages, no duplicated pkg_name. Deltas against the previous committed CSVs: NA-rate increases in doi, title, description, maintainer; packages dropped; published flag changes. Deltas go in the report, not the exit code.
• On any change to the tables, bump the fourth version component in DESCRIPTION and set Date to the harvest date (desc::desc_bump_version("dev")).
• Regenerate data-raw/packages.json: every published package plus washr and owdata, hard-coded and unit tested, because a custom r-universe registry supersedes the auto-generated one and must never drop washr (Register all packages on r-universe #8). The report says when it differs from the last committed copy.

Acceptance criteria

• Local backend against all local clones: zero hard failures, problems list contains only the known pathological packages
• GitHub backend first against five repos (washmalawi, gdho, fslogisticskampala, washopenresearch, waterpointstatus), then a full run diffed against the local output; record the size of the public versus local gap once and stop counting to 70
• packages.json from any run contains washr and owdata
• The UTF-8 and duplicate-DOI gates fail on a fixture that violates them

[larnsce]

Premortem findings for this issue (advisory, non-blocking)

From the 2026-07-18 premortem (report in docs/ on branch claude/implementation-premortem-jd5udn):

The error-tolerance design (per-repo tryCatch, problems tibble, run errors only on an empty table or over 10 percent parse failures) makes silent degradation a designed-in feature. A washr or fairenough template change can introduce a dictionary format no fixture covers; the parsers correctly refuse to abort, a handful of packages half-parse into NA DOIs or garbage descriptions, the failure rate sits under the 10 percent tripwire, and the bad rows ship. The fixtures in #3 freeze the pathologies known today while the templates keep evolving.

Suggestions:

• Alert on deltas, not just absolutes: compare each run's problems tibble and per-column NA rates against the previous committed run, and surface any week-over-week increase (the committed tables make this a cheap git diff).
• Since the harvester also regenerates data-raw/packages.json, hard-code washr and owdata into that generation with a unit test asserting their presence, so the r-universe registry can never be produced without them (see the note on Register all packages on r-universe #8 about the superseded auto-registry).

---

Generated by Claude Code