Skip to content

The code for the database creation. #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rbrink1 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

The code for the database creation. #37

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions rbrink1_MongoDB.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/usr/bin/env python
# coding: utf-8

# In[1]:


import pprint
import re
import pymongo, json

pp = pprint.PrettyPrinter(indent=1,width=65)

client = pymongo.MongoClient (host="da1.eecs.utk.edu")
db = client ['fdac19mp2']
coll = db ['cmelton3']
# for each dataset
#coll.insert_one ( { 'topic':'student debt', 'first coll.insert_one ( { 'topic':'git URLscoll.insert_one ( { 'topic':'git URLs', 'first dataset': 'largest projects', 'license': 'NA', 'description': 'The list of projects on github with the largest number of starts', 'urls': [ 'url1', 'url2' ] } )
#coll.insert ( { 'topic':'YourTopic', 'title': 'Data title', 'license': 'license', 'description': 'Brief data description', 'urls': [ 'url1', 'url2', ... ] } )


coll.insert ( { 'topic':'TSI', 'title': 'NOAA Climate Data Record (CDR) of Total Solar Irradiance (TSI), NRLTSI Version 2', 'license': 'https://data.nodc.noaa.gov/cgi-bin/iso?id=gov.noaa.ncdc:C00828', 'description': 'This Climate Data Record (CDR) contains total solar irradiance (TSI) as a function of time created with the Naval Research Laboratory model for spectral and total irradiance (version 2). Total solar irradiance is the total, spectrally integrated energy input to the top of the Earth atmosphere, at a standard distance of one Astronomical Unit from the Sun. Its units are W per m2. The dataset was created by Judith Lean (Space Science Division, Naval Research Laboratory), Odele Coddington and Peter Pilewskie (Laboratory for Atmospheric and Space Science, University of Colorado). The daily- and monthly-averaged TSI data range from 1882 to the present, and annual-averaged TSI data begin in 1610. The data file format is netCDF-4 following CF metadata conventions. The dataset is accompanied by algorithm documentation, data flow diagram and source code for the NOAA CDR Program.', 'urls': [ 'https://data.nodc.noaa.gov/cgi-bin/iso?id=gov.noaa.ncdc:C00828' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'National Solar Radiation Database (NSRDB) Station Data Output for 1991 to 2010', 'license': 'https://data.nodc.noaa.gov/cgi-bin/iso?id=gov.noaa.ncdc:C00843', 'description': 'The National Solar Radiation Database (NSRDB) was produced by the National Renewable Energy Laboratory under the U.S. DOE Office of Energy Efficiency and Renewable Energy. The NSRDB update is a collection of hourly values of the three most common measurements of solar radiation (i.e., global horizontal, direct normal, and diffuse horizontal) over a period of time adequate to establish means and extremes and at a sufficient number or locations to represent regional solar radiation climates. Nearly all of the solar data in the NSRDB are modeled, and only 40 sites have measured solar data - none of them with a complete period of record. Because of the data-filling methods used to accomplish the goal of serial completeness, NSRDB meteorological data are not suitable for climatological work. The meteorological fields in the NSRDB should be used only as ancillary data for solar deployment and sizing applications. Filled/interpolated meteorological data should not be used for climatic applications. (All such data are flagged.) ', 'urls': [ 'https://data.nodc.noaa.gov/cgi-bin/iso?id=gov.noaa.ncdc:C00843' ] } )
coll.insert ( { 'topic':'TSI', 'title': 'TOTAL SOLAR IRRADIANCE (TSI) DATASETS: AN OVERVIEW', 'license': 'https://climatedataguide.ucar.edu/climate-data/total-solar-irradiance-tsi-datasets-overview', 'description': 'The solar radiation arriving at Earth (once known as the “solar constant”, now usually referred to as Total Solar Irradiance (TSI)), is the most fundamental of climate parameters as it indicates the totality of the energy driving the climate system. All climate models need to prescribe a value for it, either explicitly or implicitly, but its measurement with the precision and stability needed for climate studies has proved challenging. -From Expert Guidance by Drs. Joanna Haigh and William Ball; please see the "Expert Guidance" tab for more.', 'urls': [ 'https://climatedataguide.ucar.edu/climate-data/total-solar-irradiance-tsi-datasets-overview' ] } )
coll.insert ( { 'topic':'Surface Radiation', 'title': 'SURFACE SOLAR RADIATION DATA SET - HELIOSAT (SARAH) -EDITION 1', 'license': 'https://climatedataguide.ucar.edu/climate-data/surface-solar-radiation-data-set-heliosat-sarah-edition-1', 'description': 'The Surface Solar Radiation Data Set - Heliosat (SARAH) is a satellite-based climatology of the solar surface irradiance (SIS), the surface direct normalized irradiance (DNI) and the effective cloud albedo (CAL) derived from satellite observations of the visible channels of the MVIRI and SEVIRI instruments onboard the geostationary Meteosat satellites. The data are available from 1983 to 2013 and cover the region ±65° longitude and ±65° latitude. The products are available as monthly, daily, and hourly averages on a regular latitude/longitude grid with a spatial resolution of 0.05° x 0.05°. The data set is complemented with a comprehensive documentation of the applied algorithms as well as validation studies and user guidance.', 'urls': [ 'https://climatedataguide.ucar.edu/climate-data/surface-solar-radiation-data-set-heliosat-sarah-edition-1' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'Surface Meteorology and Solar Energy', 'license': 'https://www.usa.gov/government-works', 'description': 'Surface Meteorology and Solar Energy data - over 200 satellite-derived meteorology and solar energy parameters, monthly averaged from 22 years of data, gridded global solar and meteorological data', 'urls': [ 'https://catalog.data.gov/dataset/surface-meteorology-and-solar-energy' ] } )
coll.insert ( { 'topic':'Solar Irradiance', 'title': 'Solar Indices - Solar Irradiance', 'license': 'https://www.usa.gov/government-works', 'description': 'Collection includes a variety of indices related to solar activity contributed by a number of national and private solar observatories located worldwide. This metadata record is currently under construction.[SolarIndices A]', 'urls': [ 'https://catalog.data.gov/dataset/solar-indices-solar-irradiance' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'National Solar Radiation Database (NSRDB) Station Data Output for 1991 to 2010', 'license': 'https://cmr.earthdata.nasa.gov/search/concepts/C1214565688-NOAA_NCEI', 'description': 'The National Solar Radiation Database (NSRDB) was produced by the National Renewable Energy Laboratory under the U.S. DOE Office of Energy Efficiency and Renewable Energy. The NSRDB update is a collection of hourly values of the three most common measurements of solar radiation (i.e., global horizontal, direct normal, and diffuse horizontal) over a period of time adequate to establish means and extremes and at a sufficient number or locations to represent regional solar radiation climates. Nearly all of the solar data in the NSRDB are modeled, and only 40 sites have measured solar data - none of them with a complete period of record. Because of the data-filling methods used to accomplish the goal of serial completeness, NSRDB meteorological data are not suitable for climatological work. The meteorological fields in the NSRDB should be used only as ancillary data for solar deployment and sizing applications. Filled/interpolated meteorological data should not be used for climatic applications. (All such data are flagged.) The serially complete hourly data provided in the NSRDB update are provided in two output formats: 1) ground-based solar and meteorological dataset, and 2) 10 km gridded output produced by the SUNY model. The 1991-2010 NSRDB is an update of the 1991-2005 NSRDB released in 2006 and archived at NCDC. The updated NSRDB dataset an hourly ground-based data set of solar and meteorological fields for 1454 stations. The primary provider for ground-based data is NCDC, which are stored as site-year files in comma-separated value (CSV) American Standard Code for Information Interchange (ASCII) format. Station identification numbers use the six-digit United States Air Force (USAF) station ID numbering scheme. The measured solar radiation data came from multiple sources, including: Atmospheric Radiation Measurement Program, Department of Energy Florida Solar Energy Center, State of Florida Integrated Surface Irradiance Study and Surface Radiation Budget Measurement Networks, National Oceanic and Atmospheric Administration Air Resources Laboratory and Earth System Research Laboratory Global Monitoring Division Measurement and Instrumentation Data Center, National Renewable Energy Laboratory University of Oregon Solar Radiation Monitoring Laboratory Network University of Texas Solar Energy Laboratory. All meteorological data were provided by the National Climatic Data Center from its Integrated Surface Hourly Database (ISD) product. The NSRDB Statistics Files hold summary statistics for all Class I and Class II stations. The Daily Statistics provide monthly and annual averages of solar radiation and several meteorological parameters for both annual and a 20 year roll-up. The Hourly Statistics provide average diurnal profiles by hour for each station year for each solar parameter. The Persistence Statistics provide multiple levels of persistence for up to 30 days for each station for each solar parameter. These Summary Statistics files are documented in the NSRDB Users Manual.', 'urls': [ 'https://cmr.earthdata.nasa.gov/search/concepts/C1214565688-NOAA_NCEI' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Eugene, OR.The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=44.453388800301774%2C-120.0146484375&zL=7' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'SantaFe, NM.The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=32.89803818160521%2C-97.6904296875&zL=5' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Madison, WI.The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=41.00477542222947%2C-86.8798828125&zL=6' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Austin, TX.The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=30.888083515609047%2C-95.240478515625&zL=7' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Portland, Maine.The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=43.83848910616803%2C-72.1142578125&zL=7' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Nashville, TN.The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=36.24870331653198%2C-86.396484375&zL=7' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Nashville, TN.The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=36.24870331653198%2C-86.396484375&zL=7' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Denver, CO.The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=38.40194908237822%2C-102.59033203125&zL=6' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Little Rock, AR. The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=35.074964853989556%2C-90.10986328125&zL=6' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Boise, ID. The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=42.08191667830631%2C-104.7216796875&zL=5' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'Solar Radio', 'license': 'http://datadiscoverystudio.org/geoportal/rest/metadata/item/9d2eee61e0c94b0d8a29af47e2bc78e7/html', 'description': 'Scientists monitor the structure of the solar corona, the outer most regions of the Suns atmosphere, using radio waves (100???s of MHz to 10???s of GHz). Variations in the radiowave spectrum reveal characteristics of the corona and upper chromosphere in terms of altitude profile for the local plasma temperature, density and magnetic field. Typically, the lower the frequency then the higher the height of origin. The frequency, like the solar electron density, decreases uniformly outwards with 245 MHz originating high in the corona whereas 15,400 MHz originates in the low corona. Radio bursts are associated with solar flares. The delay at Earth of the different radio frequencies during burst events is due to the outward movement of the source. Bursts can have temperatures of 10xE12 degrees Kelvin. Large bursts last 10 to 20 minutes on average. Longer radio noise storms of persistent and variable high levels of radiation originate in sunspot groups, areas of large, intense magnetic fields. These storms are strongly circularly polarized due to the intense magnetic fields. The microwave wavelength 2800 MHz daily radio flux correlates highly with the daily sunspot number and the two databases are used interchangeably. The 2800 MHz, or 10.7 cm, responds to the same conditions that produce changes in the visible and X-ray wavelengths. Schmahl and Kundu (1995) find that the solar radio fluxes in the spectral range 1000-9400 MHz correlate well with the total solar irradiance. The intermediate frequencies (at 2800 and 3750 MHz) are produced mainly by free-free gyroresonance emission from sunspot structures, while 1000 and 9400 MHz flux are produced mainly by free-free processes from structures associated with plages. They can distinguish plage-associated emission from spot-associated emission in the time series of microwave flux, both contributing opposing effects on the total solar irradiance.', 'urls': [ 'http://datadiscoverystudio.org/geoportal/rest/metadata/item/9d2eee61e0c94b0d8a29af47e2bc78e7/html' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Springfeild, MO. The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=38.12591462924157%2C-90.28564453124999&zL=6' ] } )
coll.insert ( { 'topic':'Solar Radiation', 'title': 'NREL National Solar Radiation Database (NSRDB)', 'license': 'https://nsrdb.nrel.gov/nsrdb-viewer', 'description': 'Tampa, FL. The NSRDB Viewer is an interactive Web-based GIS application for visualizing, examining, and downloading complete solar resource data, including TMYs.Both U.S. and international data sets are available in the NSRDB Viewer', 'urls': [ 'https://maps.nrel.gov/nsrdb-viewer/?aL=UdPEX9%255Bv%255D%3Dt%26f69KzE%255Bv%255D%3Dt%26f69KzE%255Bd%255D%3D1&bL=clight&cE=0&lR=0&mC=31.70947636001935%2C-83.8916015625&zL=5' ] } )

# In[3]:


import pprint
import pymongo, json
client = pymongo.MongoClient (host="da1.eecs.utk.edu")
db = client ['fdac19mp2']
coll = db ['audris']
pp = pprint.PrettyPrinter(indent=1,width=65)
for r in coll. find():
print(pp .pformat (r))


# In[ ]: