notebooks/ICESat-2_Cloud_Access/ATL06-direct-access.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e0754304-7036-4530-83ec-86cec0f9886b",
   "metadata": {
    "tags": []
   },
   "source": [
    "\n",
    "\n",
    "# **Accessing and working with ICESat-2 data in the cloud**\n",
    "\n",
    "\n",
    "<div align=\"left\"><img src='https://raw.githubusercontent.com/nsidc/NSIDC-Data-Tutorials/is2-cryo2/notebooks/ICESat-2_Cloud_Access/img/nsidc_logo.png'/></div>\n",
    "   \n",
    "\n",
    "## **1. Tutorial Overview**\n",
    "\n",
    "This notebook demonstrates searching for cloud-hosted ICESat-2 data and directly accessing Land Ice Height (ATL06) granules from an Amazon Compute Cloud (EC2) instance using the `earthaccess` package. NASA data \"in the cloud\" are stored in Amazon Web Services (AWS) Simple Storage Service (S3) Buckets. **Direct Access** is an efficient way to work with data stored in an S3 Bucket when you are working in the cloud. Cloud-hosted granules can be opened and loaded into memory without the need to download them first. This allows you take advantage of the scalability and power of cloud computing. \n",
    "\n",
    "The Amazon Global cloud is divided into geographical regions.  To have direct access to data stored in a region, our compute instance - a virtual computer that we create to perform processing operations in place of using our own desktop or laptop - must be in the same region as the data.  This is a fundamental concept of _analysis in place_. **NASA cloud-hosted data is in Amazon Region us-west2.  So your compute instance must also be in us-west2.** If we wanted to use data stored in another region, to use direct access for that data, we would start a compute instance in that region.\n",
    "\n",
    "As an example data  collection, we use ICESat-2 Land Ice Height (ATL06) over the Juneau Icefield, AK, for March 2003. ICESat-2 data granules, including ATL06, are stored in HDF5 format. We demonstrate how to open an HDF5 granule and access data variables using `xarray`. Land Ice Heights are then plotted using `hvplot`. \n",
    "\n",
    "`earthaccess` is a community-developed Python package developed by Luis Lopez (NSIDC developer) to allow easy search of the NASA Common Metadata Repository (CMR) and download of NASA data collections.  It can be used for programmatic search and access for both _on-premises-hosted_ and _cloud-hosted_ data. It manages authenticating using Earthdata Login credentials which are then used to obtain the S3 tokens that are needed for S3 direct access. https://github.com/nsidc/earthaccess\n",
    "\n",
    "\n",
    "### **Credits**\n",
    "\n",
    "The notebook was created by Andy Barrett, NSIDC, updated by Jennifer Roebuck and Amy Steiker, NSIDC, and is based on notebooks developed by Luis Lopez and Mikala Beig, NSIDC.\n",
    "\n",
    "For questions regarding the notebook, or to report problems, please create a new issue in the [NSIDC-Data-Tutorials repo](https://github.com/nsidc/NSIDC-Data-Tutorials/issues).\n",
    "\n",
    "### **Learning Objectives**\n",
    "\n",
    "By the end of this demonstration you will be able to:  \n",
    "1. use `earthaccess` to search for ICESat-2 data using spatial and temporal filters and explore search results;  \n",
    "2. open data granules using direct access to the ICESat-2 S3 bucket;  \n",
    "3. load HDF5 data into an `xarray.Datatree` object;\n",
    "4. visualize the land ice heights using `hvplot`.  \n",
    "\n",
    "### **Prerequisites**\n",
    "\n",
    "1. An EC2 instance in the us-west-2 region. **NASA cloud-hosted data is in Amazon Region us-west2. So you also need an EC2 instance in the us-west-2 region.** An EC2 instance is a virtual computer that you create to perform processing operations in place of using your own desktop or laptop.  Details on how to set up an instance can be found [here](https://nsidc.org/data/user-resources/help-center/nasa-earthdata-cloud-data-access-guide#anchor-1).\n",
    "2. An Earthdata Login is required for data access. If you don't have one, you can register for one [here](https://urs.earthdata.nasa.gov/).\n",
    "3. A .netrc file, that contains your Earthdata Login credentials, in your home directory. The current recommended practice for authentication is to create a .netrc file in your home directory following [these instructions](https://nsidc.org/support/how/how-do-i-programmatically-request-data-services) (Step 1) and to use the .netrc file for authentication when required for data access during the tutorial.\n",
    "4. The *nsidc-tutorials* environment is setup and activated. This [README](https://github.com/nsidc/NSIDC-Data-Tutorials/blob/main/README.md) has setup instructions.\n",
    "\n",
    "### **Example of end product** \n",
    "At the end of this tutorial, the following figure will be generated:\n",
    "       \n",
    "![ATL06 land ice heights](./img/atl06_example_end_product.png)\n",
    "### **Time requirement**\n",
    "\n",
    "Allow approximately 20 minutes to complete this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "816f31af",
   "metadata": {},
   "source": [
    "## **2. Tutorial steps**\n",
    "\n",
    "## Import Packages\n",
    "\n",
    "The first step in any `python` script or notebook is to import packages.  This tutorial requires the following packages:\n",
    "- `earthaccess`, which enables Earthdata Login authentication and retrieves AWS credentials; enables collection and granule searches; and S3 access;\n",
    "- `xarray`, used to load data;\n",
    "- `hvplot`, used to visualize land ice height data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3b4c9e6",
   "metadata": {},
   "source": [
    "We are going to import the whole `earthaccess` package.\n",
    "\n",
    "We will also import the whole `xarray` package but use a standard short name `xr`, using the `import <package> as <short_name>` syntax.  We could use anything for a short name but `xr` is an accepted standard that most `xarray` users are familiar with.\n",
    "\n",
    "We only need the `xarray` module from `hvplot` so we import that using the `import <package>.<module>` syntax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b00b7a6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For searching NASA data\n",
    "import earthaccess\n",
    "\n",
    "# For reading data, analysis and plotting\n",
    "import xarray as xr\n",
    "import hvplot.xarray\n",
    "import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15ae2994",
   "metadata": {},
   "source": [
    "## Authenticate\n",
    "\n",
    "The first step is to get the correct authentication that will allow us to get _cloud-hosted_ ICESat-2 data.  This is all done through Earthdata Login.  The `login` method also gets the correct AWS credentials.\n",
    "\n",
    "Login requires your Earthdata Login username and password. The `login` method will automatically search for these credentials as environment variables or in a `.netrc` file, and if those aren't available it will prompt us to enter our username and password. We use a `.netrc` strategy. A `.netrc` file is a text file located in our home directory that contains login information for remote machines.  If we don't have a `.netrc` file, `login` can create one for us.\n",
    "\n",
    "```\n",
    "earthaccess.login(strategy='interactive', persist=True)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37d6a667",
   "metadata": {},
   "outputs": [],
   "source": [
    "auth = earthaccess.login()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28d7b582",
   "metadata": {},
   "source": [
    "## Search for ICESat-2 Collections\n",
    "\n",
    "`earthaccess` leverages the Common Metadata Repository (CMR) API to search for collections and granules.  [Earthdata Search](https://search.earthdata.nasa.gov/search) also uses the CMR API.\n",
    "\n",
    "We can use the `search_datasets` method to search for ICESat-2 collections by setting `keyword='ICESat-2'`.\n",
    "\n",
    "This will display the number of data collections (data sets) that meet this search criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e80e935",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "Query = earthaccess.search_datasets(keyword = 'ICESat-2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3957627",
   "metadata": {},
   "source": [
    "Several dozen collections with the keyword ICESat-2 are returned in the Query object.\n",
    "\n",
    "The `search_datasets` method returns a python list of `DataCollection` objects. We can view the metadata for each collection in long form by passing a `DataCollection` object to print or as a summary using the `summary` method. We can also use the `pprint` function to Pretty Print each object.\n",
    "\n",
    "We will do this for the first 10 results (objects)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f54b13d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "for collection in Query[:10]:\n",
    "    pprint.pprint(collection.summary(), sort_dicts=True, indent=4)\n",
    "    print('')\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b88357e5",
   "metadata": {},
   "source": [
    "For each collection, `summary` returns a subset of fields from the collection metadata and the Unified Metadata Model (UMM):\n",
    "- `concept-id` is a unique id for the collection. It consists of an alphanumeric code and the provider-id specific to the DAAC (Distributed Active Archive Center). You can use the `concept_id` to search for data granules.\n",
    "- `short_name` is a quick way of referring to a collection (instead of using the full title). It can be found on the collection landing page underneath the collection title after 'DATA SET ID'. See the table below for a list of the shortnames for ICESat-2 collections.\n",
    "- `version` is the version of each collection.\n",
    "- `file-type` gives information about the file format of the collection granules.\n",
    "- `get-data` is a collection of URLs that can be used to access the data, collection landing pages and data tools. \n",
    "- `cloud-info` this is for cloud-hosted data and provides additional information about the location of the S3 bucket that holds the data and where to get temporary AWS S3 credentials to access the S3 buckets. `earthaccess` handles these credentials and the links to the S3 buckets, so in general you won't need to worry about this information. \n",
    "\n",
    "For the ICESat-2 search results, within the concept-id, there is a provider-id; `NSIDC_ECS` and `NSIDC_CPRD`. `NSIDC_ECS` which is for the _on-prem_ collections and `NSIDC_CPRD` is for the _cloud-hosted_ collections. \n",
    "\n",
    "For ICESat-2, `ShortNames` are generally how different products are referred to.\n",
    "\n",
    "| ShortName | Product Description |\n",
    "|:-----------:|:---------------------|\n",
    "| ATL03 | ATLAS/ICESat-2 L2A Global Geolocated Photon Data |\n",
    "| ATL06 | ATLAS/ICESat-2 L3A Land Ice Height |\n",
    "| ATL07 | ATLAS/ICESat-2 L3A Sea Ice Height |\n",
    "| ATL08 | ATLAS/ICESat-2 L3A Land and Vegetation Height |\n",
    "| ATL09 | ATLAS/ICESat-2 L3A Calibrated Backscatter Profiles and Atmospheric Layer Characteristics |\n",
    "| ATL10 | ATLAS/ICESat-2 L3A Sea Ice Freeboard |\n",
    "| ATL11 | ATLAS/ICESat-2 L3B Slope-Corrected Land Ice Height Time Series |\n",
    "| ATL12 | ATLAS/ICESat-2 L3A Ocean Surface Height |\n",
    "| ATL13 | ATLAS/ICESat-2 L3A Along Track Inland Surface Water Data |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc62d6f6",
   "metadata": {},
   "source": [
    "### Search for cloud-hosted data\n",
    "For most collections, to search for only data in the cloud, the `cloud_hosted` method can be used. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "322d78c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "Query = earthaccess.search_datasets(\n",
    "    keyword = 'ICESat-2',\n",
    "    cloud_hosted = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8df10797",
   "metadata": {},
   "source": [
    "## Search a data set using spatial and temporal filters \n",
    "\n",
    "We can use the `search_data` method to search for granules within a data set by location and time using spatial and temporal filters. In this example, we will search for data granules from the ATL06 verison 006 cloud-hosted data set over the Juneau Icefield, AK, for March and April 2020.\n",
    "\n",
    "The temporal range is identified with standard date strings, and latitude-longitude corners of a bounding box is specified.  Polygons and points, as well as shapefiles can also be specified.\n",
    "\n",
    "This will display the number of granules that match our search. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fba5c34",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = earthaccess.search_data(\n",
    "    short_name = 'ATL06',\n",
    "    version = '006',\n",
    "    cloud_hosted = True,\n",
    "    bounding_box = (-134.7,58.9,-133.9,59.2),\n",
    "    temporal = ('2020-03-01','2020-04-30'),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7bc1b37",
   "metadata": {},
   "source": [
    "To display the rendered metadata, including the download link, granule size and two images, we will use `display`. In the example below, all 4 results are shown. \n",
    "\n",
    "The download link is `https` and can be used download the granule to your local machine.  This is similar to downloading data located _on-premises_ but in this case the data are coming from the Earthdata Cloud.  For NASA data in the Earthdata Cloud, there is no charge to the user for egress from AWS Cloud servers.  This may not be the case for other data in the cloud.\n",
    "\n",
    "Note the `[None, None, None, None]` that is displayed at the end can be ignored; it has no meaning in relation to the metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a04370d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "[display(r) for r in results]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "810da59e",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Use Direct-Access to open, load and display data stored on S3\n",
    "\n",
    "Direct-access to data from an S3 bucket is a two step process.  First, the files are opened using the `open` method.  The `auth` object created at the start of the notebook is used to provide Earthdata Login authentication and AWS credentials. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e50bf87d-1c83-42c7-b645-3948b15b7675",
   "metadata": {},
   "outputs": [],
   "source": [
    "files = earthaccess.open(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cecdf984-ce9f-41c0-946b-3a0fa1ce40bc",
   "metadata": {},
   "source": [
    "The next step is to load the data. `xarray.DataTree` objects allow us to work with hierarchical data structures and file formats such as HDF5, Zarr and NetCDF4 with groups. \n",
    "\n",
    "We use `xr.open_datatree` to open the ATL06 data. We add the `phony_dims=\"sort\"` option because data variables in several groups including `ancillary_data` do not have any assigned dimension scales. `xarray` names dimensions `phony_dim0`, `phony_dim1`, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "013136a2-80fd-4625-aca8-1a64775c9593",
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = xr.open_datatree(files[1], phony_dims='sort')\n",
    "dt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56677586-a2cd-4ddb-8e57-457ced954331",
   "metadata": {},
   "source": [
    "We can see from the representation of the `xarray.DataTree` object `dt` that there are ten groups in the top, or \"root\", level. Clicking on Groups reveals various Metadata and Ancillary data groups as well as groups representing each of the left and right beam pairs from the ICESat-2 ATLAS instrument*. We can also see that there are no dimensions, coordinates, or data variables in the root group. Reading the data into `numpy` arrays or a `pandas.Dataframe` could be an alternative method to using `xarray.Datatree`. However, each granule (file) would have to be read first using a package that reads HDF5 files such as `h5py`. `xarray` does this all under-the-hood in a single line.\n",
    "\n",
    "*ICESat-2 measures photon returns from 3 beam pairs numbered 1, 2 and 3 that each consist of a left and a right beam. In this case, we are interested in plotting the left ground track (gt) of beam pair 1. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1282ce34",
   "metadata": {},
   "source": [
    "`hvplot` is an interactive plotting tool that is useful for exploring data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be7386c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "dt['/gt1l/land_ice_segments/h_li'].hvplot(kind='scatter', s=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a4335c8",
   "metadata": {},
   "source": [
    "## **3. Learning outcomes recap**\n",
    "\n",
    "We have learned how to:\n",
    "1. use `earthaccess` to search for ICESat-2 data using spatial and temporal filters and explore search results;\n",
    "2. open data granules using direct access to the ICESat-2 S3 bucket;\n",
    "3. load a HDF5 group into an xarray.Datatree;\n",
    "4. visualize the land ice heights using hvplot."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "317ea6bd",
   "metadata": {},
   "source": [
    "## **4. Additional resources**\n",
    "\n",
    "For general information about NSIDC DAAC data in the Earthdata Cloud: \n",
    "\n",
    "[FAQs About NSIDC DAAC's Earthdata Cloud Migration](https://nsidc.org/data/user-resources/help-center/faqs-about-nsidc-daacs-earthdata-cloud-migration)\n",
    "\n",
    "[NASA Earthdata Cloud Data Access Guide](https://nsidc.org/data/user-resources/help-center/nasa-earthdata-cloud-data-access-guide)\n",
    "\n",
    "Additional tutorials and How Tos:\n",
    "\n",
    "[NASA Earthdata Cloud Cookbook](https://nasa-openscapes.github.io/earthdata-cloud-cookbook/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a67c7eb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}