Skip to content

update_access() breaks for multi-dataset packages: repo URL from dataset name, recycled encodingFormat, assumed csv+xlsx pairs #100

Description

@larnsce

Summary

update_access() (R/update_access.R, currently on the 1.0.2.9000 dev source) produces wrong rows in data/metadata/access.csv for any package with more than one dataset, and for datasets whose export is not a csv+xlsx pair. It only works by accident for single-dataset packages whose dataset name equals the package name.

Found while converting jmpdata (two datasets: jmpraw, jmpindicators; the indicators export is jmpindicators.csv.gz only).

Observed

With a dictionary listing jmpraw.rda and jmpindicators.rda, update_access() wrote:

fileName,name,contentUrl,encodingFormat
jmpindicators.csv,jmpindicators,https://raw.githubusercontent.com/openwashdata/jmpindicators/main/inst/extdata/jmpindicators.csv,csv
jmpraw.csv,jmpraw,https://raw.githubusercontent.com/openwashdata/jmpraw/main/inst/extdata/jmpraw.csv,ooxml
jmpindicators.xlsx,jmpindicators,https://raw.githubusercontent.com/openwashdata/jmpindicators/main/inst/extdata/jmpindicators.xlsx,csv
jmpraw.xlsx,jmpraw,https://raw.githubusercontent.com/openwashdata/jmpraw/main/inst/extdata/jmpraw.xlsx,ooxml

Three problems:

  1. The repo segment of every URL is the dataset name (openwashdata/jmpraw, openwashdata/jmpindicators), not the package repo (openwashdata/jmpdata). All four URLs 404.
  2. encodingFormat is misaligned: jmpraw.csv is labelled ooxml and jmpindicators.xlsx is labelled csv.
  3. jmpindicators.xlsx does not exist; the dataset ships as jmpindicators.csv.gz (about 38 MB uncompressed, so no xlsx). The function invents rows instead of reading inst/extdata/.

Root cause (R/update_access.R)

  • Lines 35-36 build contentUrl with paste0(".../openwashdata/", file_name, "/main/..."), so the dataset name stands in for the repo name. Single-dataset packages named after their dataset mask this.
  • Line 32 builds fileName as c(paste0(file_name, ".csv"), paste0(file_name, ".xlsx")) (length 2n), while line 38 sets encodingFormat = c("csv", "ooxml") (length 2). R recycles the short vector, which aligns correctly only for n = 1. For n = 2 the formats interleave wrongly, exactly as observed.
  • The csv/xlsx pair is assumed rather than derived from the files actually present in inst/extdata/.
  • Line 18 defines an unused extdata_path hardcoding the worldhdi repo; dead code from the same single-package assumption.

Suggested fix

  • Derive the repo name from the package, for example desc::desc_get("Package") or the DESCRIPTION URL field, instead of file_name.
  • Enumerate list.files("inst/extdata"), match files to datasets by name prefix, and map extension to encodingFormat per file (csv/csv.gz to csv, xlsx to ooxml). That fixes the recycling misalignment and the invented rows in one move, and picks up compressed exports.
  • Drop the unused extdata_path.

For jmpdata, access.csv was corrected by hand in openwashdata/jmpdata@770210b, which shows the expected output.

Possibly fits the v1.1.0 milestone (consolidate the FAIR layer).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions