Collecting_Jobs_data_Using_API-Questions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Collecting Job Data Using APIs**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **45 to 60** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After completing this lab, you will be able to:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Collect job data from Jobs API\n",
    "*   Store the collected data into an excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "><strong>Note: Before starting with the assignment make sure to read all the instructions and then move ahead with the coding part.</strong>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instructions\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the actual lab, firstly you need to click on the [Jobs_API](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/Jobs_API.ipynb) notebook link. The file contains flask code which is required to run the Jobs API data.\n",
    "\n",
    "Now, to run the code in the file that opens up follow the below steps.\n",
    "\n",
    "Step1: Download the file. \n",
    "\n",
    "Step2: Upload it on the IBM Watson studio. (If IBM Watson Cloud service does not work in your system, follow the alternate Step 2 below)\n",
    "\n",
    "Step2(alternate): Upload it in your SN labs environment using the upload button which is highlighted in red in the image below:\n",
    "Remember to upload this Jobs_API file in the same folder as your current .ipynb file\n",
    "\n",
    "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/Upload.PNG\">\n",
    "\n",
    "Step3:  Run all the cells of the Jobs_API file. (Even if you receive an asterik sign after running the last cell, the code works fine.)\n",
    "\n",
    "If you want to learn more about flask, which is optional, you can click on this link [here](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/FLASK_API.md.html).\n",
    "\n",
    "Once you run the flask code, you can start with your assignment.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Used in this Assignment\n",
    "\n",
    "The dataset used in this lab comes from the following source: https://www.kaggle.com/promptcloud/jobs-on-naukricom under the under a **Public Domain license**.\n",
    "\n",
    "> Note: We are using a modified subset of that dataset for the lab, so to follow the lab instructions successfully please use the dataset provided with the lab, rather than the dataset from the original source.\n",
    "\n",
    "The original dataset is a csv. We have converted the csv to json as per the requirement of the lab.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Warm-Up Exercise\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you attempt the actual lab, here is a fully solved warmup exercise that will help you to learn how to access an API.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using an API, let us find out who currently are on the International Space Station (ISS).<br> The API at [http://api.open-notify.org/astros.json](http://api.open-notify.org/astros.json?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2021-01-01&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ) gives us the information of astronauts currently on ISS in json format.<br>\n",
    "You can read more about this API at [http://open-notify.org/Open-Notify-API/People-In-Space/](http://open-notify.org/Open-Notify-API/People-In-Space?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2021-01-01&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests # you need this module to make an API call\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_url = \"http://api.open-notify.org/astros.json\" # this url gives use the astronaut data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(api_url) # Call the API using the get method and store the\n",
    "                                # output of the API call in a variable called response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "if response.ok:             # if all is well() no errors, no network timeouts)\n",
    "    data = response.json()  # store the result in json format in a variable called data\n",
    "                            # the variable data is of type dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'message': 'success', 'people': [{'name': 'Sergey Prokopyev', 'craft': 'ISS'}, {'name': 'Dmitry Petelin', 'craft': 'ISS'}, {'name': 'Frank Rubio', 'craft': 'ISS'}, {'name': 'Nicole Mann', 'craft': 'ISS'}, {'name': 'Josh Cassada', 'craft': 'ISS'}, {'name': 'Koichi Wakata', 'craft': 'ISS'}, {'name': 'Anna Kikina', 'craft': 'ISS'}, {'name': 'Fei Junlong', 'craft': 'Shenzhou 15'}, {'name': 'Deng Qingming', 'craft': 'Shenzhou 15'}, {'name': 'Zhang Lu', 'craft': 'Shenzhou 15'}], 'number': 10}\n"
     ]
    }
   ],
   "source": [
    "print(data)   # print the data just to check the output or for debugging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the number of astronauts currently on ISS.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n"
     ]
    }
   ],
   "source": [
    "print(data.get('number'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the names of the astronauts currently on ISS.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 10 astronauts on ISS\n",
      "And their names are :\n",
      "Sergey Prokopyev\n",
      "Dmitry Petelin\n",
      "Frank Rubio\n",
      "Nicole Mann\n",
      "Josh Cassada\n",
      "Koichi Wakata\n",
      "Anna Kikina\n",
      "Fei Junlong\n",
      "Deng Qingming\n",
      "Zhang Lu\n"
     ]
    }
   ],
   "source": [
    "astronauts = data.get('people')\n",
    "print(\"There are {} astronauts on ISS\".format(len(astronauts)))\n",
    "print(\"And their names are :\")\n",
    "for astronaut in astronauts:\n",
    "    print(astronaut.get('name'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hope the warmup was helpful. Good luck with your next lab!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab: Collect Jobs Data using Jobs API\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objective: Determine the number of jobs currently open for various technologies  and for various locations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collect the number of job postings for the following locations using the API:\n",
    "\n",
    "* Los Angeles\n",
    "* New York\n",
    "* San Francisco\n",
    "* Washington DC\n",
    "* Seattle\n",
    "* Austin\n",
    "* Detroit\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import required libraries\n",
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write a function to get the number of jobs for the Python technology.<br>\n",
    "> Note: While using the lab you need to pass the **payload** information for the **params** attribute in the form of **key** **value** pairs.\n",
    "  Refer the ungraded **rest api lab** in the course **Python for Data Science, AI & Development**  <a href=\"https://www.coursera.org/learn/python-for-applied-data-science-ai/ungradedLti/P6sW8/hands-on-lab-access-rest-apis-request-http?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01\">link</a>\n",
    "  \n",
    " ##### The keys in the json are \n",
    " * Job Title\n",
    " \n",
    " * Job Experience Required\n",
    " \n",
    " * Key Skills\n",
    " \n",
    " * Role Category\n",
    " \n",
    " * Location\n",
    " \n",
    " * Functional Area\n",
    " \n",
    " * Industry\n",
    " \n",
    " * Role \n",
    " \n",
    "You can also view  the json file contents  from the following <a href = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/jobs.json\">json</a> URL.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_url=\"http://127.0.0.1:5000/data\"\n",
    "def get_number_of_jobs_T(technology):\n",
    "    response = requests.get(api_url, params={'Key Skills':technology})\n",
    "    if response.status_code == 200:\n",
    "        json_result = response.json()\n",
    "        number_of_jobs = len(json_result)\n",
    "        return technology, len(json_result)\n",
    "    else:\n",
    "        return f\"ERROR: {location}\", 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling the function for Python and checking if it works.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Python', 1173)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_number_of_jobs_T(\"Python\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write a function to find number of jobs in US for a location of your choice\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_number_of_jobs_L(location):\n",
    "    response = requests.get(api_url, params={'Location':location})\n",
    "    if response.status_code == 200:\n",
    "        json_result = response.json()\n",
    "        number_of_jobs = len(json_result)\n",
    "        return location,number_of_jobs\n",
    "    else:\n",
    "        return f\"ERROR: {location}\", 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call the function for Los Angeles and check if it is working.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Los Angeles', 640)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_number_of_jobs_L('Los Angeles')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store the results in an excel file\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call the API for all the given technologies above and write the results in an excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you do not know how create excel file using python, double click here for **hints**.\n",
    "\n",
    "<!--\n",
    "\n",
    "from openpyxl import Workbook        # import Workbook class from module openpyxl\n",
    "wb=Workbook()                        # create a workbook object\n",
    "ws=wb.active                         # use the active worksheet\n",
    "ws.append(['Country','Continent'])   # add a row with two columns 'Country' and 'Continent'\n",
    "ws.append(['Eygpt','Africa'])        # add a row with two columns 'Egypt' and 'Africa'\n",
    "ws.append(['India','Asia'])          # add another row\n",
    "ws.append(['France','Europe'])       # add another row\n",
    "wb.save(\"countries.xlsx\")            # save the workbook into a file called countries.xlsx\n",
    "\n",
    "\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a python list of all locations for which you need to find the number of jobs postings.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "locations = [\"Los Angeles\", \"New York\", \"San Francisco\", \"Washington DC\", \"Seattle\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import libraries required to create excel spreadsheet\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openpyxl import Workbook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a workbook and select the active worksheet\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "wb = Workbook()\n",
    "ws = wb.active"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the number of jobs postings for each of the location in the above list.\n",
    "Write the Location name and the number of jobs postings into the excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Los Angeles', 640),\n",
       " ('New York', 3226),\n",
       " ('San Francisco', 435),\n",
       " ('Washington DC', 5316),\n",
       " ('Seattle', 3375)]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "job_location_data = [get_number_of_jobs_L(loc) for loc in locations]\n",
    "\n",
    "for location, number_of_jobs in job_location_data:\n",
    "    ws.append([location, number_of_jobs])\n",
    "\n",
    "job_location_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save into an excel spreadsheet named 'job-postings.xlsx'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "wb.save('job-postings.xlsx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In the similar way, you can try for below given technologies and results  can be stored in an excel sheet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collect the number of job postings for the following languages using the API:\n",
    "\n",
    "*   C\n",
    "*   C#\n",
    "*   C++\n",
    "*   Java\n",
    "*   JavaScript\n",
    "*   Python\n",
    "*   Scala\n",
    "*   Oracle\n",
    "*   SQL Server\n",
    "*   MySQL Server\n",
    "*   PostgreSQL\n",
    "*   MongoDB\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C 13498\n",
      "C# 333\n",
      "C++ 305\n",
      "Java 2609\n",
      "JavaScript 355\n",
      "Python 1173\n",
      "Scala 33\n",
      "Oracle 784\n",
      "SQL Server 250\n",
      "MySQL Server 0\n",
      "PostgreSQL 10\n",
      "MongoDB 174\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('C', 13498),\n",
       " ('C#', 333),\n",
       " ('C++', 305),\n",
       " ('Java', 2609),\n",
       " ('JavaScript', 355),\n",
       " ('Python', 1173),\n",
       " ('Scala', 33),\n",
       " ('Oracle', 784),\n",
       " ('SQL Server', 250),\n",
       " ('MySQL Server', 0),\n",
       " ('PostgreSQL', 10),\n",
       " ('MongoDB', 174)]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "technologies = ['C', 'C#', 'C++', 'Java', 'JavaScript', 'Python', 'Scala', 'Oracle', 'SQL Server', 'MySQL Server', 'PostgreSQL', 'MongoDB']\n",
    "\n",
    "job_tech_data = [get_number_of_jobs_T(tech) for tech in technologies]\n",
    "\n",
    "for tech, number_of_jobs in job_tech_data:\n",
    "    print(tech, number_of_jobs)\n",
    "\n",
    "job_tech_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Author\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ayushi Jain\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rav Ahuja\n",
    "\n",
    "Lakshmi Holla\n",
    "\n",
    "Malika\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Date (YYYY-MM-DD) | Version | Changed By        | Change Description                 |\n",
    "| ----------------- | ------- | ----------------- | ---------------------------------- | \n",
    "| 2022-01-19        | 0.3     | Lakshmi Holla        | Added changes in the markdown      |\n",
    "| 2021-06-25        | 0.2     | Malika            | Updated GitHub job json link       |\n",
    "| 2020-10-17        | 0.1     | Ramesh Sannareddy | Created initial version of the lab |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2022 IBM Corporation. All rights reserved. \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}