Parsing_Text_Files_XML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parsing_Text_Files_XML\n",
    "#### Author: Armin Berger\n",
    "\n",
    "Date: 23/08/2020\n",
    "\n",
    "Version: 1.0\n",
    "\n",
    "Environment: Python 2.7.11 and Jupyter notebook\n",
    "\n",
    "Libraries used: \n",
    "* re (for regular expression, included in Anaconda Python 2.7)\n",
    "* os (library used for interacting with operating systems, included in Anaconda Python 2.7) \n",
    "* langid (library used for language identification, included in Anaconda Python 2.7) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.  Import libraries \n",
    "\n",
    "Firstly, we will install and import all the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# installing langid library, in case it is not installed \n",
    "pip install langid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing the three libraries required for this project (re, os,and langid)\n",
    "\n",
    "import re # used for regular expression\n",
    "\n",
    "import os # used for reading in multiple file from the local os \n",
    "\n",
    "import langid # used to check whether a tweet is in english \n",
    "\n",
    "# import langid # used for identify only tweets in this case (does language identification in general)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Check for the path of the current working directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we need to know the location of the current working directory in order to read in the folder containing the\n",
    "# twitter txt files\n",
    "\n",
    "# Get the current working directory (cwd) in case it is needed for reading in files\n",
    "cwd = os.getcwd()\n",
    "\n",
    "# print current working directory\n",
    "cwb\n",
    "\n",
    "# Get all the files in that directory\n",
    "files = sorted(os.walk('26255367'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check files names that were read in\n",
    "files[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Parse the txt files, extract  and save the main three data poinst of each tweet\n",
    "\n",
    "Since the text data is only semi structred we need to find ways in oder to only extract the information we need. This is largely achieved throught the use of regular expressions. For this purpose I came up with three sets of regular expressions to extract the id, text, and the date created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## NOTE: this code block runs for a while but creates the desired output (please don't restart it, just let it run)\n",
    "\n",
    "# we create a set of global data structures to save the processed tweet data\n",
    "new_tweets = []       # list that holds dicts of tweets\n",
    "\n",
    "# loop through all files\n",
    "for entry in files[0][2]:\n",
    "    \n",
    "    # we read in the txt data using a combination of the open() and the read() method\n",
    "    # this generates a large string which is saved as the variable tweets\n",
    "    # tweets = open('/Users/arminberger/Desktop/Code/fit5196_code/assign_1/26255367_txt/' + entry).read()\n",
    "    with open('26255367/'+entry, encoding = 'latin-1') as tweets:\n",
    "        \n",
    "        # read in the tweet\n",
    "        tweets = tweets.read()\n",
    "        \n",
    "        # we split the large string named tweets into actual individual tweets \n",
    "        # since each tweet is enclosed in {} and seperated by a comma , we can use regular expression to split the \n",
    "        # large string into individual tweets and save the as list named 'single_tweets'\n",
    "        single_tweets = re.split(r'},{', tweets)\n",
    "\n",
    "        # iterate through the single_tweet list dict by dict using a for loop\n",
    "        for item in single_tweets:\n",
    "\n",
    "            ## FIRST ##, extract the text\n",
    "            # valid tweets for xml file must have three elements - the tweet, created date and time, and the user ID. \n",
    "            # tweet_dict is created in order to save the tweet and the relevant information \n",
    "            tweet_text = re.search(r'\"text\":\"(.+?)\"',item)\n",
    "            tweet_date = re.search(r'\"created_at\":\"(.+?)\",\"id\":',item)\n",
    "            tweet_id = re.search(r'\"id\":\"([0-9]{19})\"',item)\n",
    "            \n",
    "            # checking that tweet text, date and id is not None\n",
    "            if tweet_text is not None and tweet_date is not None and tweet_id is not None:\n",
    "                \n",
    "                # dict to save individual tweets\n",
    "                tweet_dict = {}\n",
    "                \n",
    "                # get the first match in tweet_text \n",
    "                actual_text = tweet_text.group(1)\n",
    "                \n",
    "                # if the tweet ends with '\\', .decode('unicode_escape') raises an error. \n",
    "                # therefore, the backslash at the end of any tweets is eliminated.\n",
    "                if re.search(r'.*\\\\$', actual_text) is not None:\n",
    "                    \n",
    "                    # remove backslash at the end using .replace()\n",
    "                    actual_text = actual_text.replace('\\\\','')\n",
    "\n",
    "                # the strings with escaped sequence should be corrected into unescaped sequence\n",
    "                # to decode and encode the surrogate pairs seperately from other escaped sequence, \n",
    "                # .decode('unicode_escape') is performed seperately based on whether a string contains either surrogate pairs, \n",
    "                # \\n or both \n",
    "                # if the string contains '\\n' with escaped sequence but no emoticons \n",
    "                if re.search(r'\\\\\\\\n',actual_text) is not None and re.search(r'\\\\uD', actual_text) is None: \n",
    "                    \n",
    "                    # directly unescape sequence\n",
    "                    actual_text = actual_text.encode('utf-8').decode('unicode_escape')\n",
    "\n",
    "                # if the string contains both the emoticons and \\n OR the string contains only emoticons\n",
    "                if (re.search(r'\\\\\\\\n',actual_text) is not None and re.search(r'\\\\uD', actual_text) is not None)\\\n",
    "                or (re.search(r'\\\\\\\\n',actual_text) is None and re.search(r'\\\\uD', actual_text) is not None): \n",
    "                \n",
    "                    # unescape backslash and decode the emoticons \n",
    "                    actual_text = actual_text.encode('utf-8').decode('unicode_escape').encode(\"utf-16\", \"surrogatepass\").decode(\"utf-16\")\n",
    "                \n",
    "                # after decoding and encoding the strings, the tweet is written into the dictionary \n",
    "                tweet_dict['text']= str(actual_text)\n",
    "                \n",
    "\n",
    "                ## SECOND ##, extract the date/time\n",
    "                actual_date = tweet_date.group(1)\n",
    "                \n",
    "                # assign the extracted date to the key 'date' and add it to the tweet_dict\n",
    "                tweet_dict['date']= str(actual_date)\n",
    "\n",
    "                ## THIRD ##, extract the ID\n",
    "                actual_id = tweet_id.group(1)\n",
    "                \n",
    "                # then we assign the extracted id to the key id and add it to the tweet_dict\n",
    "                tweet_dict['id']= str(actual_id)\n",
    "\n",
    "                # the structured tweet_dict is appended into new_tweets\n",
    "                new_tweets.append(tweet_dict)\n",
    "\n",
    "# filter out the non-English tweet using list comprehension \n",
    "new_tweets = [dic for dic in new_tweets if dic['text'] !='' and langid.classify(dic['text'])[0]=='en']\n",
    "\n",
    "# the list of non-duplicate tweets \n",
    "unique_tweets = []\n",
    "\n",
    "# loop through new_tweets\n",
    "for item in new_tweets: \n",
    "    \n",
    "    # in case tweet is not in unique_tweets\n",
    "    if item not in unique_tweets: \n",
    "        \n",
    "        # append tweet\n",
    "        unique_tweets.append(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Save the all unique tweets in an xml format \n",
    "\n",
    "In the last step of task one we need to save all the extracted tweets in an xml format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an xml file to write to\n",
    "tweet_file = open('<26255367>.xml','w')\n",
    "\n",
    "# the XML file starts with the version of XML and its character encoding \n",
    "# the root tag of the XML file is <data>\n",
    "# all the tweets are enclosed within this tag \n",
    "# the tag 'data' contains <tweets> tag, whose attribute is 'date'\n",
    "tweet_file.write('''<?xml version=\"1.0\" encoding=\"UTF-8\"?>\\n<data>\\n''')\n",
    "\n",
    "# looping through the sorted list of unique_tweets \n",
    "# the unique_tweets is sorted based on the created date of every tweet\n",
    "for i in range(len(sorted(unique_tweets, key = lambda x: x['date']))): \n",
    "    \n",
    "    # extract the created date of the tweet \n",
    "    date = unique_tweets[i]['date'].split('T')[0]\n",
    "    \n",
    "    if i >0:\n",
    "        \n",
    "        # extract the created date of the tweet before this loop \n",
    "        former_date = unique_tweets[i-1]['date'].split('T')[0]\n",
    "        \n",
    "        # if the created date of the tweet differs from the one before, \n",
    "        if date != former_date: \n",
    "            \n",
    "            # a new <tweets> tag is created followed by the date attribute \n",
    "            tweet_file.write('''</tweets>\\n<tweets date=\"{}\">\\n'''.format(date))\n",
    "    \n",
    "    else:\n",
    "        \n",
    "        # the first tweet of the list unique_tweets does not have any former date to compare \n",
    "        # therefore the tweet starts off with a new tag <tweets>\n",
    "        tweet_file.write('''<tweets date=\"{}\">\\n'''.format(date))\n",
    "    \n",
    "    # write in the actual tweet text into <tweet> tag, whose attribute is the id of the tweet \n",
    "    tweet_file.write('''<tweet id=\"{}\">{}</tweet>\\n'''.format(unique_tweets[i]['id'], unique_tweets[i]['text']))\n",
    "\n",
    "# finish off the XML file by close the last <tweets> tag and the <data> tag \n",
    "tweet_file.write('''</tweets>\\n</data>''')\n",
    "\n",
    "# we close the xml file after writing to it\n",
    "tweet_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, I have been able to read in the provided semi-structed textfile data, extract the required information, and write that information to an xml file. The proceedure can be broken down into the following steps:\n",
    "\n",
    "- Firstly, we imported the required libraries re, os,and langid\n",
    "    - re was required in order to use regular expressions\n",
    "    - os was required to open and write to text and xml files\n",
    "    - landid was required to be able to determine whether a tweet is in english or not\n",
    "    \n",
    "- Secondly, we parsed the text file and turned its content into one long continous string \n",
    "\n",
    "- Thridly, we extracted and saved the main three data poinst of each tweet, which are the tweets unique id, its text and it's date/time of creation\n",
    "    - the extraction of the required information happen in two main steps\n",
    "    - first, the long string was split into individual tweets and saved as a list of strings\n",
    "    - second, list of strings was iterated through and within each tweet id, text, and date were extracted with the help of regular expression]\n",
    "    - fourth, each id, txt, and date data point was added as a value to a dict\n",
    "    - lastly, it was checked whether a certain tweet has aready been saved earlier. This was done by turning the dict into a tuple and checking whether it was already present in the dict. If that was not the case the dict was appended to the list of tweets \n",
    "\n",
    "- Fourthly, we created a new xml file and wrote all the data contained in the tweet list to the xml file\n",
    "\n",
    "\n",
    "The main finding of this task for me was that, despite every tweet being largely different to another in terms of lenght and the characters it contains, the fact that the tweet data was semi-structered already made it easier to seperate each tweet from another and extract the important information. I learned the importance of looking for repreating patterns within data to guide the wrangleing process."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}