Create a Harvesting Guide

[karenmajewicz]

Include topics such as:

• what harvesting means
• why we have to re-harvest some sites on a regular basis and how we do that.
• an overview of the scripts we use (Python, Notebooks, BeautifulSoup)
• links to script examples
  ...

[karenmajewicz]

Test out all the forms of harvesting we have listed so far:

structured APIs

• DCAT JSONs
• CKAN API
• OAI-PMH
• other???

individual metadata standards files

• ISO 19139
• FGDC
• ArcGIS 1.0 metadata
• MODS
• MARC
• other???

web scraping

• HTML
• XML standards files (like above heading)

[karenmajewicz]

read about OAI-PMH

[karenmajewicz]

read about CSW: http://opengeospatial.github.io/e-learning/cat/text/main.html

[ardumn]

DCAT: https://www.w3.org/TR/vocab-dcat-2/ and https://geobtaa.github.io/metadata/harvestGuide/#dcat-harvesting for our process
CKAN API: https://docs.ckan.org/en/2.9/user-guide.html#what-is-ckan and https://geobtaa.github.io/metadata/harvestGuide/#ckan-harvesting for our process
OAI-PMH: https://www.openarchives.org/pmh/
Metadata Standards: https://pitt.libguides.com/metadatadiscovery/metadata-standards

ISO 19139: https://www.iso.org/standard/67253.html Paper about encoding rules and how it has changed
FGDC: https://www.fgdc.gov/dataandservices/fgdcmeta#:~:text=The%20FGDC%20metadata%20standard%20is,overall%20structure%20to%20the%20standard.
ArcGIS 1.0: https://desktop.arcgis.com/en/arcmap/10.3/manage-data/metadata/the-arcgis-metadata-format.htm also gives info about ISO 19139, FGDC, and others
MODS: https://www.loc.gov/standards/mods/mods-overview.html
**MARC:**https://rdamsc.bath.ac.uk/msc/m88#:~:text=MARC%20is%20a%20standard%20and,probably%20the%20most%20widely%20used.

*Dublin Core, Astronomy Metadata, and Ecological Standard are covered in Metadata Standards link.

HTML: https://en.wikipedia.org/wiki/HTML
XML:https://www.w3.org/standards/xml/core#:~:text=What%20is%20XML%3F,more%20suitable%20for%20Web%20use.

[karenmajewicz]

Outline for ArcGIS Hub harvests