Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outreachy Moja Global Task 2 completed #73

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Outreachy Moja Global Task 2 completed #73

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ca7ab9c
Outreachy Moja Global Task 2 completed
naz-coder Oct 27, 2023
ed23d59
Geographical Analysis of the Datasets chosen
naz-coder Oct 30, 2023
5083fd9
Geographical Analysis of the Datasets chosen updated
naz-coder Oct 30, 2023
12e44ff
Visualisation Prototype for the given datasets
naz-coder Oct 30, 2023
124acfd
Visualisation Prototype for the given datasets - updated
naz-coder Oct 30, 2023
a957337
User Documentation - task 5
naz-coder Oct 30, 2023
8568334
Task 4 Updated
naz-coder Oct 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
CI_BiodiversityHotspots_Mapped.geojson
IPCC_ClimateZoneMap_Vector.geojson
TerrestrialEcoregionsoftheWorld_WWF.geojson
25 changes: 25 additions & 0 deletions .ipynb_checkpoints/README-checkpoint.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Land Sector Datasets
This Repo is bringing together datasets that can be useful for land sector management. The range of datasets is going beyond what is needed to run FLINT. For each dataset information is provided on content and licence.


## Maintainers Reviewers Ambassadors Coaches

The following people are Maintainers Reviewers Ambassadors or Coaches

<table><tr><td align="center"><a href="https://github.com/gmajan"><img src="https://avatars0.githubusercontent.com/u/8733319?v=4" width="100px;" alt="Guy Janssen"/><br /><sub><b>Guy Janssen</b></sub></a><br /><a href="#maintenance-gmajan" title="Maintenance">🚧</a></td><td align="center"><a href="https://github.com/mtbdeligt"><img src="https://avatars3.githubusercontent.com/u/16447169?v=4" width="100px;" alt="mtbdeligt"/><br /><sub><b>mtbdeligt</b></sub></a><br /><a href="https://github.com/moja-global/About-moja-global/commits?author=mtbdeligt" title="Documentation">📖</a></tr></table>

**Maintainers** review and accept proposed changes
**Reviewers** check proposed changes before they go to the Maintainers
**Ambassadors** are available to provide training related to this repository
**Coaches** are available to provide information to new contributors to this repository

## Processed datasets
FLINT ready processed vector (json) and raster/grid (tiff) data described in this repo can be downloaded from Moja Global [Land Sector Datasets](https://datasets.mojaglobal.workers.dev/) data library.

### Formats
To be FLINT ready, vector datasets must be in geoJSON vector format, with no overlapping parts or topological error such as self intersections. Code is provided to systematically process the original data sources into this format, but please note this may change the content of the data e.g. where there are two overlapping polygons with different attribution, one will arbitrarily win over the other. We have also attempted to clean up small gaps and slithers systematically, but it is still recommended that manual checking and editing is completed depending on your project requirements.

To be FLINT ready, rasters or grids must be in tiff or geoTiff format, with the same extent and resolution equal to or a multiple of the overall simulation resolution, padded to the nearest 1 degree lat/lon. These can be mosaic or tiled. We provide gdal code to resample the grids and tiles to different extents and resolutions. There are a range of continous and discrete grids available in the data library.

## Licence information
Please review individual dataset licence terms and conditions. Please note, while all care has been taken to ensure data is open data available for use, individual datasets will have their own licence conditions and citation requirements.
265 changes: 265 additions & 0 deletions Data/Administrative/.ipynb_checkpoints/WorldDatabaseProtectedAreas_WDPA-checkpoint.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,265 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Title: The World Database on Protected Areas (WDPA)\n",
"\n",
"### Description\n",
"The World Database on Protected Areas (WDPA) is the most comprehensive global database of marine and terrestrial protected areas. It is a joint project between UN Environment Programme and the International Union for Conservation of Nature (IUCN), and is managed by UN Environment Programme World Conservation Monitoring Centre (UNEP-WCMC), in collaboration with governments, non-governmental organisations, academia and industry.The WDPA is updated on a monthly basis, and can be downloaded using the button in the top right of this page.<br>\n",
"\n",
"As the data collates many different reserve types from many different sources, there are a range of topological issues including significant duplication and overlapping of reserve areas. There are over 150,000 overlapping polygons in the National cateogry alone. To simplify this dataset for use in FLINT, only National reserves are shown, excluding Marine, Regional and International reserve areas.\n",
"\n",
"### Format\n",
"<b>Extent: </b>Global coverage<br>\n",
"<b>Format</b>: polygon geoJSON .json<br>\n",
"<b>Cordinate system:</b> EPSG:4326 (WGS84)<br>\n",
"<b>Temporal Resolution: </b>2020 as at 29/12/2020. Please note, updates to this dataset are likely and will require re-processing for up to date use.<br>\n",
"<b>Size:</b> 700MB\n",
"\n",
"\n",
"### Original source\n",
"Original Source: https://www.protectedplanet.net/en/thematic-areas/wdpa Accessed 29/12/2020. Shapefile, geodatabase or feature service.\n",
"\n",
"### FLINT\n",
"This dataset has been pre-processed/checked and is suitable for use in FLINT as at 29/12/2020. Please adhere to individual dataset licence conditions and citations. Processed data can be accessed here: https://datasets.mojaglobal.workers.dev/.\n",
"\n",
"### Licence\n",
"Read and accept terms and conditions information here: https://www.protectedplanet.net/en/legal\n",
"- Credit source<br>\n",
"- Only most recent version can be used <br>\n",
"- For Commercial use use subscription via https://www.ibat-alliance.org/ <br>\n",
"- No sub-licensing, No redistribution, No distribution of derivative work, except with permission from [email protected]<br>\n",
"- No publication in downloadable format except with permission from [email protected]<br>\n",
"\n",
"See [Additional permissions](https://github.com/moja-global/Land-Sector-Datasets/blob/master/Additional-Permissions/Gmail%20-%20Thank%20you%20for%20useful%20input%20and%20follow%20up.pdf) related to this project.\n",
"\n",
"### Citation\n",
"UNEP-WCMC and IUCN (2020), Protected Planet: The World Database on Protected Areas (WDPA) [Online], September 2020, Cambridge, UK: UNEP-WCMC and IUCN. Available at: www.protectedplanet.net.\n",
"\n",
"### Original format\n",
"Global coverage, vector, shapefile<br>\n",
"Cordinate system EPSG: 4326 (WGS84)\n",
"\n",
"### Metadata\n",
"The WDPA is a joint project between UN Environment Programme and the International Union for Conservation of Nature (IUCN). The compilation and management of the WDPA is carried out by UN Environment Programme World Conservation Monitoring Centre (UNEP-WCMC), in collaboration with governments, non-governmental organisations, academia and industry. There are monthly updates of the data which are made available online through the ProtectedPlanet website where the data is both viewable and downloadable.<br>\n",
"\n",
"Data and information on the world's protected areas compiled in the WDPA are used for reporting to the Convention on Biological Diversity on progress towards reaching the Aichi Biodiversity Targets (particularly Target 11), to the UN to track progress towards the 2030 Sustainable Development Goals, to some of the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services (IPBES) core indicators, and other international assessments and reports including the Global Biodiversity Outlook, as well as for the publication of the United Nations List of Protected Areas. Every two years, UNEP-WCMC releases the Protected Planet Report on the status of the world's protected areas and recommendations on how to meet international goals and targets.<br>\n",
"\n",
"For more details on the WDPA please read through the [User Manual](http://wcmc.io/WDPA_Manual).\n",
"\n",
"### Notes\n",
"There are significant topological issues in this dataset including significant duplication and overlapping of reserve area types (National, International, Regional and Marine). Reserve areas vary from stratigh eco protection to sustainable game management and indigenous lands. To simplify this dataset for use in FLINT, only National reserves are included, excluding Marine, Regional and International reserve areas. Reserves are then split into IUCN Cateogory. Processing time is lengthy as the dataset is 200,000+ attributes. Processing this layer to suit your individual project needs is recommended.\n",
"\n",
"Gaps and overlaps are evident and can be fixed with code below. Please note, only gaps smaller than 0.1ha will be fixed as gaps larger than this could be valid.\n",
"\n",
"### Processing\n",
"Query National Reserves only and non-marine. Repair geometry, fix topologial error (remove overlaps), convert to geojson, EPSG:4326 (WGS84), remove/disable Z values."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Import arcpy module\n",
"import arcpy\n",
"import os\n",
"\n",
"# Input variables\n",
"in_folder = r\"C:\\Data\\WDPA\"\n",
"scr_folder = r\"C:\\Data\\scratch.gdb\"\n",
"out_folder = r\"C:\\Data\\json\"\n",
"field = \"NAME IS NULL or WDPA_PID = ''\" #field that will be blank on appended gaps (any field will do)\n",
"smallest = \"1000\"\n",
"\n",
"# Input variables\n",
"in_folder = r\"C:\\Data\\WDPA\"\n",
"out_folder = r\"C:\\Data\\json\"\n",
"fullfield = \"NAME\" #this needs to be a field in the original table that is fully populated\n",
"smallest = \"20000\" #smallest area to be fixed in m2 - gaps and slithers\n",
"scr = arcpy.CreateFileGDB_management(out_folder, \"scratch\")\n",
"scr_folder = os.path.join(out_folder, \"scratch.gdb\")\n",
"\n",
"# Environments\n",
"workspace = in_folder\n",
"arcpy.env.workspace = workspace\n",
"arcpy.env.outputCoordinateSystem = arcpy.SpatialReference(4326)\n",
"arcpy.env.outputZFlag = \"Disabled\"\n",
"arcpy.env.overwriteOutput = True\n",
"field = fullfield + \" IS NULL or \" + fullfield + \" = ''\"\n",
"arcpy.env.parallelProcessingFactor = \"100%\"\n",
"\n",
"# List features to process\n",
"featureclasses = arcpy.ListFeatureClasses()\n",
"print(featureclasses)\n",
"\n",
"# Repair/check topology and make FLINT ready\n",
"for fc in featureclasses:\n",
" fcname = os.path.join(os.path.splitext(fc)[0])\n",
" outjson = os.path.join(out_folder, fcname)\n",
" whereclause = \"FID_\" + fcname + \" =-1 AND AREA_GEO <= \" + smallest\n",
" print(fcname + ' processing...')\n",
" \n",
" fLayer = \"project_Layer\"\n",
" arcpy.management.MakeFeatureLayer(fc, fLayer)\n",
" geomRepair = arcpy.management.RepairGeometry(fLayer, \"DELETE_NULL\", \"OGC\")[0]\n",
" arcpy.management.DeleteIdentical(fLayer, \"Shape\")\n",
" \n",
" projectIntersect = os.path.join(scr_folder, \"projectIntersect\")\n",
" arcpy.analysis.Intersect(fLayer, projectIntersect, \"ONLY_FID\")\n",
" \n",
" projectSingle = os.path.join(scr_folder, \"projectSingle\")\n",
" arcpy.management.MultipartToSinglepart(projectIntersect, projectSingle)\n",
"\n",
" dissolveSlither = os.path.join(scr_folder, \"dissolveSlither\")\n",
" arcpy.management.Dissolve(projectSingle, dissolveSlither, None, None,\"SINGLE_PART\")\n",
" \n",
" # Take action if no overlaps\n",
" if arcpy.management.GetCount(dissolveSlither)[0] == \"0\":\n",
" print('no overlaps detected...checking for gaps...')\n",
" \n",
" projectUnion = os.path.join(scr_folder, \"projectUnion\")\n",
" arcpy.analysis.Union(fLayer,projectUnion, \"ALL\", None, \"NO_GAPS\")\n",
" \n",
" unionSingle = os.path.join(scr_folder, \"unionSingle\")\n",
" arcpy.management.MultipartToSinglepart(projectUnion, unionSingle)\n",
" \n",
" arcpy.management.DeleteIdentical(unionSingle, \"Shape\")\n",
" arcpy.management.AddGeometryAttributes(unionSingle, \"AREA_GEODESIC\", None, \"SQUARE_METERS\")\n",
" \n",
" \n",
" uniSelect = os.path.join(scr_folder, \"uniSelect\")\n",
" arcpy.analysis.Select(unionSingle, uniSelect, whereclause)\n",
" \n",
" if arcpy.management.GetCount(uniSelect)[0] == \"0\":\n",
" \n",
" # Progress report no error\n",
" print(fcname, 'No gaps and overlaps. Repairing geometry and conversion to json...')\n",
" \n",
" # Process: Repair Geometry (non-simple geometry)\n",
" geomRepair = arcpy.management.RepairGeometry(fLayer, \"DELETE_NULL\", \"OGC\")[0]\n",
"\n",
" # Process: Features To JSON\n",
" arcpy.conversion.FeaturesToJSON(fLayer, outjson, \"NOT_FORMATTED\", \"NO_Z_VALUES\", \"NO_M_VALUES\", \"GEOJSON\", \"WGS84\", \"USE_FIELD_NAME\")\n",
"\n",
" print(outjson, '.geojson complete')\n",
" \n",
" else:\n",
" # Take action if gaps\n",
" print('gaps detected')\n",
"\n",
" appendGap = arcpy.management.Append(uniSelect, fLayer, \"NO_TEST\") \n",
" selectGap = arcpy.management.SelectLayerByAttribute(fLayer, \"NEW_SELECTION\", field)\n",
"\n",
" fixedlyr = os.path.join(scr_folder, \"fixedlyr\")\n",
" arcpy.management.Eliminate(selectGap, fixedlyr, \"LENGTH\")\n",
"\n",
" # Progress report \n",
" print(fcname, 'No overlaps but gaps detected and repaired. Repairing geometry and conversion to json...')\n",
"\n",
" # Process: Repair Geometry (non-simple geometry)\n",
" geomRepair = arcpy.management.RepairGeometry(fixedlyr, \"DELETE_NULL\", \"OGC\")[0]\n",
"\n",
" # Process: Features To JSON\n",
" arcpy.conversion.FeaturesToJSON(fixedlyr, outjson, \"NOT_FORMATTED\", \"NO_Z_VALUES\", \"NO_M_VALUES\", \"GEOJSON\", \"WGS84\", \"USE_FIELD_NAME\")\n",
"\n",
"\n",
" else:\n",
" print('Overlaps detected...')\n",
" # Fix overlaps\n",
" projectErase = os.path.join(scr_folder, \"projectErase\")\n",
" arcpy.analysis.Erase(fLayer, dissolveSlither, projectErase)\n",
" \n",
" arcpy.management.Append(dissolveSlither, projectErase, \"NO_TEST\")\n",
" \n",
" selectSlither = arcpy.management.SelectLayerByAttribute(projectErase, \"NEW_SELECTION\", field)\n",
" \n",
" eliminateSlither = os.path.join(scr_folder, \"eliminateSlither\")\n",
" arcpy.management.Eliminate(selectSlither, eliminateSlither, \"LENGTH\")\n",
" \n",
" print('Overlaps detected and fixed...checking for gaps...')\n",
" \n",
" projectUnion = os.path.join(scr_folder, \"projectUnion\")\n",
" arcpy.analysis.Union(eliminateSlither, projectUnion, \"ALL\", None, \"NO_GAPS\")\n",
" \n",
" unionSingle = os.path.join(scr_folder, \"unionSingle\")\n",
" arcpy.management.MultipartToSinglepart(projectUnion, unionSingle)\n",
" arcpy.management.DeleteIdentical(unionSingle, \"Shape\")\n",
" \n",
" arcpy.management.AddGeometryAttributes(unionSingle, \"AREA_GEODESIC\", None, \"SQUARE_METERS\")\n",
" \n",
" uniSelect = os.path.join(scr_folder, \"uniSelect\")\n",
" whereUnion= \"FID_eliminateSlither = -1 AND AREA_GEO <= \" + smallest\n",
" arcpy.analysis.Select(unionSingle, uniSelect, whereUnion)\n",
" \n",
" if arcpy.management.GetCount(uniSelect)[0] == \"0\":\n",
" \n",
" # Progress report no error\n",
" print(fcname, ' No gaps detected. Repairing geometry and conversion to json...')\n",
" \n",
" # Process: Repair Geometry (non-simple geometry)\n",
" geomRepair = arcpy.management.RepairGeometry(eliminateSlither, \"DELETE_NULL\", \"OGC\")[0]\n",
"\n",
" # Process: Features To JSON\n",
" arcpy.conversion.FeaturesToJSON(eliminateSlither, outjson, \"NOT_FORMATTED\", \"NO_Z_VALUES\", \"NO_M_VALUES\", \"GEOJSON\", \"WGS84\", \"USE_FIELD_NAME\")\n",
"\n",
" print(outjson, '.geojson complete')\n",
" \n",
" else:\n",
" # Take action if gaps\n",
" appendGap = arcpy.management.Append(uniSelect, eliminateSlither, \"NO_TEST\")\n",
" \n",
" selectGap = arcpy.management.SelectLayerByAttribute(eliminateSlither, \"NEW_SELECTION\", field)\n",
" \n",
" fixedlyr = os.path.join(scr_folder, \"fixedlyr\")\n",
" \n",
" arcpy.management.Eliminate(selectGap, fixedlyr, \"LENGTH\")\n",
" \n",
" print('gaps detected and repaired')\n",
" \n",
" # Progress report\n",
" print(fcname, 'Gaps and overlaps fixed. Repairing geometry and conversion to json...')\n",
" \n",
" # Process: Repair Geometry (non-simple geometry)\n",
" geomRepair = arcpy.management.RepairGeometry(fixedlyr, \"DELETE_NULL\", \"OGC\")[0]\n",
" \n",
" \n",
" # Process: Features To JSON\n",
" arcpy.conversion.FeaturesToJSON(fixedlyr, outjson, \"NOT_FORMATTED\", \"NO_Z_VALUES\", \"NO_M_VALUES\", \"GEOJSON\", \"WGS84\", \"USE_FIELD_NAME\")\n",
" print(outjson)\n",
"\n",
" # Process: Features To JSON\n",
" arcpy.conversion.FeaturesToJSON(fixedlyr, outjson, \"NOT_FORMATTED\", \"NO_Z_VALUES\", \"NO_M_VALUES\", \"GEOJSON\", \"WGS84\", \"USE_FIELD_NAME\")\n",
"\n",
"arcpy.management.Delete(scr_folder)\n",
"\n",
"arcpy.AddMessage(\"All done!\")\n",
"print('done')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Loading