Project: Migrate data.gov data from Harvard Innovation Lab to HDV

[sbarbosadataverse]

Objective:

• 300K datasets rescued from data.gov by Harvard Innovation Lab, to move into a collection on HDV
  Requires a Harvard faculty owner

Data details:

Our data is here, with a readme — let me know if there are parts of that we could expand:

https://source.coop/harvard-lil/gov-data

The metadata.csv.zip and metadata.jsonl.zip files have metadata about all of the datasets we collected.

We also now have a statically-hosted browser for the data, described here:

https://lil.law.harvard.edu/blog/2025/10/10/welcome-to-lil-s-data-gov-archive-search/

I’m adding our developer Chris Setzer as well — if there were other data formats that would be helpful, we’d be happy to consider.

I think we will have a great deal of CDC data, but you would have to analyze the metadata files or use the hosted browser to check what’s in there.

Thanks,
Jack

Contact:

• Cushman, Jack jcushman@law.harvard.edu

[jggautier]

Related:

• Harvest over 2 million datasets from the US Government (data.gov) #56
• https://github.com/IQSS/dataverse-HDV-Curation/issues/568

[cmbz]

2025-11-05

• This task needs a great deal of discussion, given the potential impact of loading this many datasets into production (and also hosting this many datasets).
• @landreev and @sbarbosadataverse (and others) will set up a time to discuss implications, considerations, and plans.

[jggautier]

Does this mean we shouldn't or can no longer harvest datasets from data.gov, and we should close #56? Just adding as another question maybe for @landreev, @sbarbosadataverse and others.

[landreev]

@jggautier A good rule of thumb is that any issue that's been open since before COVID should be closed quietly.
A less flippant/more serious answer is we don't know without more research. I haven't spoken to the "rescuers" of the data yet. I don't know if they meant it as a one time operation, or if the plan is to build an ongoing replication system to keep grabbing new data they make available. ... But I'd say there is a good chance we'll want to close #56 as obsolete. It was opened specifically with their OAI service in mind; and that looks like it was a short-lived experiment that's been long discontinued. Even the "2 million datasets" part is no longer true, as they re-organized their holdings into fewer combined datasets.

[landreev]

(also, let's not assume that this migration/rescue has already been done. the scale of the undertaking may prove to be a serious challenge)

[sbarbosadataverse]

@jggautier I would leave #56 open because we are only getting 300K datasets from data.gov as part of the 2025 data rescue effort.