03-data-types-and-format.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Data Types and Formats**\n",
    "\n",
    "# **Types of Data**\n",
    "\n",
    "How information is stored in a\n",
    "DataFrame or a python object affects what we can do with it and the outputs of\n",
    "calculations as well. There are two main types of data that we're explore in\n",
    "this lesson: numeric and text data types.\n",
    "\n",
    "## **Numeric Data Types**\n",
    "\n",
    "Numeric data types include integers and floats. A **floating point** (known as a\n",
    "float) number has decimal points even if that decimal point value is 0. For\n",
    "example: 1.13, 2.0 1234.345. If we have a column that contains both integers and\n",
    "floating point numbers, Pandas will assign the entire column to the float data\n",
    "type so the decimal points are not lost.\n",
    "\n",
    "An **integer** will never have a decimal point. Thus if we wanted to store 1.13 as\n",
    "an integer it would be stored as 1. Similarly, 1234.345 would be stored as 1234. You\n",
    "will often see the data type `Int64` in python which stands for 64 bit integer. The 64\n",
    "simply refers to the memory allocated to store data in each cell which effectively\n",
    "relates to how many digits it can store in each \"cell\". Allocating space ahead of time\n",
    "allows computers to optimize storage and processing efficiency.\n",
    "\n",
    "## **Text Data Type**\n",
    "\n",
    "Text data type is known as Strings in Python, or Objects in Pandas. Strings can\n",
    "contain numbers and / or characters. For example, a string might be a word, a\n",
    "sentence, or several sentences. A Pandas object might also be a plot name like\n",
    "'plot1'. A string can also contain or consist of numbers. For instance, '1234'\n",
    "could be stored as a string. As could '10.23'. However **strings that contain\n",
    "numbers can not be used for mathematical operations**!\n",
    "\n",
    "Pandas and base Python use slightly different names for data types. More on this\n",
    "is in the table below:\n",
    "\n",
    "| Pandas Type | Native Python Type | Description |\n",
    "|-------------|--------------------|-------------|\n",
    "| object | string | The most general dtype. Will be assigned to your column if column has mixed types (numbers and strings). |\n",
    "| int64  | int | Numeric characters. 64 refers to the memory allocated to hold this character. |\n",
    "| float64 | float | Numeric characters with decimals. If a column contains numbers and NaNs(see below), pandas will default to float64, in case your missing value has a decimal. |\n",
    "| datetime64, timedelta[ns] | N/A (but see the [datetime] module in Python's standard library) | Values meant to hold time data. Look into these for time series experiments. |\n",
    "[datetime]: http://doc.python.org/2/library/datetime.html\n",
    "\n",
    "## **Checking the format of our data**\n",
    "\n",
    "Now that we're armed with a basic understanding of numeric and text data\n",
    "types, let's explore the format of our survey data. We'll be working with the\n",
    "same `surveys.csv` dataset that we've used in previous lessons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "# note that pd.read_csv is used because we imported pandas as pd\n",
    "surveys_df = pd.read_csv(\"data/surveys.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that we can check the type of an object like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(surveys_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's look at the structure of our surveys data. In pandas, we can check\n",
    "the type of one column in a DataFrame using the syntax\n",
    "`dataFrameName[column_name].dtype`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "surveys_df['sex'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A type 'O' just stands for \"object\" which in Pandas' world is a string\n",
    "(text)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "surveys_df['record_id'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The type `int64` tells us that python is storing each value within this column\n",
    "as a 64 bit integer. We can use the `dat.dtypes` command to view the data type\n",
    "for each column in a DataFrame (all at once)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "record_id            int64\n",
       "month                int64\n",
       "day                  int64\n",
       "year                 int64\n",
       "plot_id              int64\n",
       "species_id          object\n",
       "sex                 object\n",
       "hindfoot_length    float64\n",
       "weight             float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "surveys_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that most of the columns in our Survey data are of type `int64`. This means\n",
    "that they are 64 bit integers. But the weight column is a floating point value\n",
    "which means it contains decimals. The `species_id` and `sex` columns are objects which\n",
    "means they contain strings.\n",
    "\n",
    "## **Working With Integers and Floats**\n",
    "\n",
    "So we've learned that computers store numbers in one of two ways: as integers or\n",
    "as floating-point numbers (or floats). Integers are the numbers we usually count\n",
    "with. Floats have fractional parts (decimal places).  Let's next consider how\n",
    "the data type can impact mathematical operations on our data. Addition,\n",
    "subtraction, division and multiplication work on floats and integers as we'd expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "20\n"
     ]
    }
   ],
   "source": [
    "print(5+5)\n",
    "\n",
    "print(24-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we divide one integer by another, we get a float.\n",
    "The result on python 3 is different than in python 2, where the result is an\n",
    "integer (integer division)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5555555555555556\n",
      "3.3333333333333335\n"
     ]
    }
   ],
   "source": [
    "print(5/9)\n",
    "\n",
    "print(10/3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also convert a floating point number to an integer or an integer to\n",
    "floating point number. Notice that Python by default rounds down when it\n",
    "converts from floating point to integer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert a to integer\n",
    "a = 7.83\n",
    "int(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert to float\n",
    "b = 7\n",
    "float(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Working With Our Survey Data**\n",
    "Getting back to our data, we can modify the format of values within our data, if\n",
    "we want. For instance, we could convert the `record_id` field to floating point\n",
    "values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert the record_id field from an integer to a float\n",
    "surveys_df['record_id'] = surveys_df['record_id'].astype('float64')\n",
    "surveys_df['record_id'].dtype\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## **Challenge - Changing Types**\n",
    ">\n",
    "> Try converting the column `plot_id` to floats using\n",
    ">\n",
    "> ```python\n",
    "> surveys_df.plot_id.astype(\"float\")\n",
    "> ```\n",
    ">\n",
    "> Next try converting `weight` to an integer. What goes wrong here? What is Pandas telling you?\n",
    "> We will talk about some solutions to this later.\n",
    "\n",
    "## **Missing Data Values - NaN**\n",
    "\n",
    "What happened in the last challenge activity? Notice that this throws a value error:\n",
    "`ValueError: Cannot convert NA to integer`. If we look at the `weight` column in the surveys\n",
    "data we notice that there are NaN (**N**ot **a** **N**umber) values. *NaN* values are undefined\n",
    "values that cannot be represented mathematically. Pandas, for example, will read\n",
    "an empty cell in a CSV or Excel sheet as a NaN. NaNs have some desirable properties: if we\n",
    "were to average the `weight` column without replacing our NaNs, Python would know to skip\n",
    "over those cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42.672428212991356"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "surveys_df['weight'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dealing with missing data values is always a challenge. It's sometimes hard to\n",
    "know why values are missing - was it because of a data entry error? Or data that\n",
    "someone was unable to collect? Should the value be 0? We need to know how\n",
    "missing values are represented in the dataset in order to make good decisions.\n",
    "If we're lucky, we have some metadata that will tell us more about how null\n",
    "values were handled.\n",
    "\n",
    "For instance, in some disciplines, like Remote Sensing, missing data values are\n",
    "often defined as -9999. Having a bunch of -9999 values in your data could really\n",
    "alter numeric calculations. Often in spreadsheets, cells are left empty where no\n",
    "data are available. Pandas will, by default, replace those missing values with\n",
    "NaN. However it is good practice to get in the habit of intentionally marking\n",
    "cells that have no data, with a no data value! That way there are no questions\n",
    "in the future when you (or someone else) explores your data.\n",
    "\n",
    "### **Where Are the NaN's?**\n",
    "\n",
    "Let's explore the NaN values in our data a bit further. Using the tools we\n",
    "learned in lesson 02, we can figure out how many rows contain NaN values for\n",
    "weight. We can also create a new subset from our data that only contains rows\n",
    "with weight values > 0 (ie select meaningful weight values):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3266"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(surveys_df[pd.isnull(surveys_df.weight)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32283"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many rows have weight values?\n",
    "len(surveys_df[surveys_df.weight> 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can replace all NaN values with zeroes using the `.fillna()` method (after\n",
    "making a copy of the data so we don't lose our work):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df1 = surveys_df.copy()\n",
    "# fill all NaN values with 0\n",
    "df1['weight'] = df1['weight'].fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However NaN and 0 yield different analysis results. The mean value when NaN\n",
    "values are replaced with 0 is different from when NaN values are simply thrown\n",
    "out or ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38.751976145601844"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1['weight'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can fill NaN values with any value that we chose. The code below fills all\n",
    "NaN values with a mean for all weight values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    " df1['weight'] = surveys_df['weight'].fillna(surveys_df['weight'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also chose to create a subset of our data, only keeping rows that do\n",
    "not contain NaN values.\n",
    "\n",
    "The point is to make conscious decisions about how to manage missing data. This\n",
    "is where we think about how our data will be used and how these values will\n",
    "impact the scientific conclusions made from the data.\n",
    "\n",
    "Python gives us all of the tools that we need to account for these issues. We\n",
    "just need to be cautious about how the decisions that we make impact scientific\n",
    "results.\n",
    "\n",
    "> ## **Challenge - Counting**\n",
    "> Count the number of missing values per column. Hint: The method .count() gives you\n",
    "> the number of non-NA observations per column. Try looking to the .isnull() method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Writing Out Data to CSV**\n",
    "\n",
    "We've learned about using manipulating data to get desired outputs. But we've also discussed\n",
    "keeping data that has been manipulated separate from our raw data. Something we might be interested\n",
    "in doing is working with only the columns that have full data. First, let's reload the data so\n",
    "we're not mixing up all of our previous manipulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_df = pd.read_csv(\"data/surveys.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's drop all the rows that contain missing values. We will use the command `drop_na`.\n",
    "By default, dropna removes columns that contain missing data for even just one row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_na = surveys_df.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you now type ```drop_na```, you should observe that the resulting DataFrame has 30676 rows\n",
    "and 9 columns, much smaller than the 35549 row original.\n",
    "\n",
    "We can now use the `to_csv` command to do export a DataFrame in CSV format. Note that the code\n",
    "below will by default save the data into the current working directory. We can\n",
    "save it to a different folder by adding the foldername and a slash to the file\n",
    "`df1.to_csv('foldername/out.csv')`. We use the 'index=False' so that\n",
    "pandas doesn't include the index number for each line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Write DataFrame to CSV\n",
    "df_na.to_csv('data_output/surveys_complete.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use this data file later in the workshop. Check out your working directory to make\n",
    "sure the CSV wrote out properly, and that you can open it! If you want, try to bring it\n",
    "back into python to make sure it imports properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}