Skip to content

Build hardened metadata parsers with fixtures #3

Description

@larnsce

Goal

Hardened parsers for the per-package metadata files, with a fixture suite covering every known pathology across the org. They live in R/parsers.R, take text already fetched by a backend, never touch the network, and are unit tested against tests/testthat/fixtures/.

Rewritten 2026-09-04 per dev/decisions-2026-09.md. Start from R/parsers.R and the fixtures on the closed PR #15 branch.

Inputs to parse

  1. DESCRIPTION via desc::desc(text = ): Package, Title, Description, Version, Authors@R (maintainer, ORCID, all aut/cre), License, Date, URL, and the washr 1.1.0 fields X-schema.org-keywords, X-schema.org-spatialCoverage, X-schema.org-temporalCoverage
  2. data-raw/dictionary.csv: the five standard columns; washr's own reader requires only file_name, variable_name and description, so keep that minimum
  3. CITATION.cff via cffr::cff_read() on a temp file: doi and version
  4. Roxygen data docs in R/: dataset title, description, and dimensions from the @format line
  5. The file listing of inst/extdata, for the csv and xlsx URLs in Define and ship the catalog data model #5

Known dictionary.csv pathologies

Fully quoted headers, space-padded names, trailing empty columns, extra columns (unit_type, error), a leading unnamed column, double-double-quoted headers, the deprecated two-column schema with a UTF-8 BOM, a git-lfs pointer file, latin1 encoding, and multi-class variable_type written as a deparsed vector such as c("POSIXct", "POSIXt") by washr releases before 1.1.0.

Parser behaviour: read as text, detect git-lfs pointers and refetch through the media endpoint in the GitHub backend, convert latin1 to UTF-8, strip the BOM, normalise header names, drop unnamed and empty columns, keep only the five standard columns, map the two-column schema to the package's single dataset when it has exactly one, normalise variable_type to its first class. Unrecoverable failure lands in the problems table.

Known @Format variants

"A tibble with N rows and M variables", a trailing colon after the tag, #'@format without a space, reversed order with a thousands separator. Anything unrecognised returns NA dimensions.

Acceptance criteria

  • Every fixture parses to the normalised schema or lands in the problems table with a reason
  • No parser call can abort a harvest run
  • A DESCRIPTION written by washr::update_description() and a CITATION.cff written by washr::update_citation() round-trip without a problems row
  • Non-ASCII text survives: a fixture with ö and é parses to the same characters, never to <U+00F6>

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions