Categorise shared data and identify PII and other sensitive content

[larnsce]

For each paper where data is shared. Download it alongside the metadata. Create scripts to evaluate if any PII or other sensitive content is shared.

Also categorise the data sources into the types of data that was collected and shared (quant / qual, observational, sensor, laboratory, survey, etc.)

Check if categorisation and checking for PII can be done with ellmer r package and give me other more reproducible alternatives

[larnsce]

Plan: categorise shared data and screen for PII (issue #27)

Builds on the pipeline conventions from #28 (scripted acquisition, committed snapshots, human decisions in CSV decision sheets keyed on DOI/paperid).

0. Scope

Every paper with a resolvable repository link: washdev and uncnewsletter rows with das_type == "available in online repository" and a non-NA das_repo_url, plus datapapers$data_repo_url once #28 lands. All downstream artefacts key on dataset + paperid (or DOI where available).

1. Download harness — data-raw/04_shareddata_download.R

• Split the "; "-separated repository URLs, classify the repository (Zenodo, Figshare, Dryad, OSF, Dataverse, GitHub, publisher supplementary page), and fetch files plus repository metadata via the respective APIs: zen4R (Zenodo), rdryad, osfr, dataverse, the Figshare REST API, plain HTTP for direct links.
• Downloads go to data-raw/shared-data/<dataset>/<paperid>/ which is gitignored — the files are large and, per this issue's premise, potentially sensitive, so they must not be redistributed through this repo.
• What is committed is a manifest CSV: paperid, url, repo, resolved, http_status, file_name, file_type, size_bytes, sha256, download_date. The checksummed manifest is the reproducible record of what was retrieved without republishing it; broken/moved links become data (link-rot is a finding in itself).

2. File triage and text extraction

• Unpack archives; inventory formats; for tabular files (csv, xlsx, sav/dta via haven) record column names, column count, row count; extract text from pdf (pdftools) and docx (officer). Per-file records appended to the manifest.

3. Data-type categorisation

• A controlled vocabulary committed as CSV, with two axes: mode (quantitative / qualitative / mixed) and collection method (household survey, water-quality laboratory, sensor/logger, observational/spot-check, GIS/spatial, interview or FGD transcript, secondary/compiled, model or code output).
• A rules pass auto-suggests categories from file formats, column names, and the paper's title/abstract/keywords; suggestions are written to a decision sheet (doi/paperid, suggested_category, category, notes) and a human confirms — same append-only, never-overwrite pattern as the Identify wash data that's in data papers #28 screening sheet, so the manual effort is captured in git and splittable among contributors.

4. PII / sensitive-content screening — layered, deterministic first

1. Rule-based scanner (layer of record, fully reproducible). Committed regex/keyword rules: column-name heuristics (name, phone, gps, lat/lon, address, village, respondent_id, dob, …) and value patterns (emails, phone numbers, GPS coordinates and their precision, national-ID formats, birth dates). Output: per-file findings CSV with a severity class — direct identifier, quasi-identifier, sensitive attribute (health status, income, caste/ethnicity, etc.).
2. Statistical disclosure risk. sdcMicro (CRAN, the standard tool for survey microdata) on tabular files that carry quasi-identifiers: k-anonymity / sample uniqueness on combinations like age × sex × location × occupation. This catches re-identification risk that pattern matching cannot.
3. LLM triage (optional assist, never the record) — see next section.

5. Can this be done with ellmer? Yes — with caveats. Plus more reproducible alternatives

ellmer feasibility: both tasks fit ellmer well technically. Structured outputs (type_object() / type_enum()) can classify a dataset's mode and collection method from its metadata, and flag PII candidates from column names plus a small sample of values, returning tidy rows for the decision sheets. Two caveats:

• Privacy first: sending downloaded microdata to a hosted API (Claude/OpenAI/Gemini via ellmer) is itself a disclosure of exactly the content we're worried about. If hosted models are used at all, send only metadata — column names, file descriptions, the paper abstract — never raw rows.
• Reproducibility: hosted models drift and are nondeterministic. Mitigations: pin the exact model version, set temperature 0, cache every raw request/response as committed JSON, and treat all LLM output as suggestions that a human confirms in the committed decision sheet — the sheet, not the model, is the ground truth the pipeline reruns from.

More reproducible alternatives, in descending order of determinism:

Approach	Tooling	Notes
Committed rules + regex	base R / stringr, rules as CSV	Fully deterministic; the baseline layer of record
Statistical disclosure control	sdcMicro (R)	Deterministic, peer-reviewed methodology for exactly this problem
Local NER-based PII engine	Microsoft Presidio (Python, runs locally; call from inst/python/ or via reticulate)	Deterministic rules + spaCy NER, widely used, no data leaves the machine
Local LLM	ellmer::chat_ollama() or rollama with a pinned local model	Data stays local; model pinned by digest; still stochastic → same suggestion-then-confirm pattern
Hosted LLM via ellmer	chat_anthropic() etc.	Most capable, least reproducible; metadata-only inputs, cached responses

Recommendation: rules + sdcMicro as the authoritative layers; Presidio as a cross-check; ellmer (local model preferred) only as a triage layer whose suggestions are human-confirmed. Pilot all layers on ~20 papers first to compare precision/recall before scaling.

6. Outputs

• A new package dataset (e.g. shareddata): one row per paper — n_files, formats, total size, resolved/link-rot status, confirmed data categories, and PII flags as counts/severity levels only. No sensitive values, filenames-of-concern excerpts, or examples are ever committed.
• A vignette/article reporting aggregate results (share of shared datasets with disclosure risk, category breakdown).
• Responsible disclosure process: if direct identifiers are confirmed in a published dataset, the specific paper is not named in public repo artefacts; findings are tracked privately and reported to the authors/journal/repository. The process gets a short paragraph in data-raw/README.md.

Milestones

1. Download harness + committed manifest.
2. Rule-based PII scan + category auto-suggestions → decision sheets.
3. sdcMicro disclosure-risk pass on tabular files.
4. Pilot comparison of ellmer (hosted, metadata-only) vs. local LLM vs. Presidio on ~20 papers; pick the triage stack.
5. Human-confirmed sheets → shareddata dataset, vignette, disclosure process documented.

---

Generated by Claude Code

[larnsce]

Closing for now: this phase is not being pursued at the moment.

Context for a future pickup:

• The datapapers screening stopped after the 45 candidates that had an abstract (8 included, 37 excluded, PR Screen datapapers candidates and build the dataset #30). The 155 title-only candidates will not be reviewed; they stay include = NA in data-raw/datapapers_screening.csv and out of builds.
• All 8 included data papers have data_repo_url NA because the harvest carried no dataset-relation metadata. If this issue is picked up again, the first step is collecting the repository links from the papers themselves.

Reopen when screening shared data for PII and sensitive content becomes a priority.