You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hardened parsers for the per-package metadata files, with a fixture suite covering every known pathology across the org. They live in R/parsers.R, take text already fetched by a backend, never touch the network, and are unit tested against tests/testthat/fixtures/.
Rewritten 2026-09-04 per dev/decisions-2026-09.md. Start from R/parsers.R and the fixtures on the closed PR #15 branch.
Inputs to parse
DESCRIPTION via desc::desc(text = ): Package, Title, Description, Version, Authors@R (maintainer, ORCID, all aut/cre), License, Date, URL, and the washr 1.1.0 fields X-schema.org-keywords, X-schema.org-spatialCoverage, X-schema.org-temporalCoverage
data-raw/dictionary.csv: the five standard columns; washr's own reader requires only file_name, variable_name and description, so keep that minimum
CITATION.cff via cffr::cff_read() on a temp file: doi and version
Roxygen data docs in R/: dataset title, description, and dimensions from the @format line
Fully quoted headers, space-padded names, trailing empty columns, extra columns (unit_type, error), a leading unnamed column, double-double-quoted headers, the deprecated two-column schema with a UTF-8 BOM, a git-lfs pointer file, latin1 encoding, and multi-class variable_type written as a deparsed vector such as c("POSIXct", "POSIXt") by washr releases before 1.1.0.
Parser behaviour: read as text, detect git-lfs pointers and refetch through the media endpoint in the GitHub backend, convert latin1 to UTF-8, strip the BOM, normalise header names, drop unnamed and empty columns, keep only the five standard columns, map the two-column schema to the package's single dataset when it has exactly one, normalise variable_type to its first class. Unrecoverable failure lands in the problems table.
"A tibble with N rows and M variables", a trailing colon after the tag, #'@format without a space, reversed order with a thousands separator. Anything unrecognised returns NA dimensions.
Acceptance criteria
Every fixture parses to the normalised schema or lands in the problems table with a reason
No parser call can abort a harvest run
A DESCRIPTION written by washr::update_description() and a CITATION.cff written by washr::update_citation() round-trip without a problems row
Non-ASCII text survives: a fixture with ö and é parses to the same characters, never to <U+00F6>
Goal
Hardened parsers for the per-package metadata files, with a fixture suite covering every known pathology across the org. They live in
R/parsers.R, take text already fetched by a backend, never touch the network, and are unit tested againsttests/testthat/fixtures/.Rewritten 2026-09-04 per dev/decisions-2026-09.md. Start from
R/parsers.Rand the fixtures on the closed PR #15 branch.Inputs to parse
desc::desc(text = ): Package, Title, Description, Version, Authors@R (maintainer, ORCID, all aut/cre), License, Date, URL, and the washr 1.1.0 fieldsX-schema.org-keywords,X-schema.org-spatialCoverage,X-schema.org-temporalCoveragedata-raw/dictionary.csv: the five standard columns; washr's own reader requires only file_name, variable_name and description, so keep that minimumcffr::cff_read()on a temp file: doi and versionR/: dataset title, description, and dimensions from the@formatlineinst/extdata, for the csv and xlsx URLs in Define and ship the catalog data model #5Known dictionary.csv pathologies
Fully quoted headers, space-padded names, trailing empty columns, extra columns (unit_type, error), a leading unnamed column, double-double-quoted headers, the deprecated two-column schema with a UTF-8 BOM, a git-lfs pointer file, latin1 encoding, and multi-class
variable_typewritten as a deparsed vector such asc("POSIXct", "POSIXt")by washr releases before 1.1.0.Parser behaviour: read as text, detect git-lfs pointers and refetch through the media endpoint in the GitHub backend, convert latin1 to UTF-8, strip the BOM, normalise header names, drop unnamed and empty columns, keep only the five standard columns, map the two-column schema to the package's single dataset when it has exactly one, normalise
variable_typeto its first class. Unrecoverable failure lands in the problems table.Known @Format variants
"A tibble with N rows and M variables", a trailing colon after the tag,
#'@formatwithout a space, reversed order with a thousands separator. Anything unrecognised returns NA dimensions.Acceptance criteria
washr::update_description()and a CITATION.cff written bywashr::update_citation()round-trip without a problems rowöandéparses to the same characters, never to<U+00F6>