Regular Expressions Examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regular Expressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#this is an inbuilt package which is used in python to work with Regular Expressions\n",
    "import re\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This **re** module offers a set of functions that allow to search for a string to match. This Notebook will contain code showing examples from the  **Speech and Language Processing** Book that we're going through at AMMI.\n",
    "\n",
    "AT the end of the notebook we try to build a very simple Eliza chatbot that uses pattern matching to recognize phrases and give outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to use  the following functions in **re**:\n",
    "- re.findall  - Returns a list containing all matches\n",
    "- re.search - Returns a match object if there is a match in the string\n",
    "- re.sub - Replaces one or many matches with a string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Regular Expression Patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True!\n"
     ]
    }
   ],
   "source": [
    "#searching for a simple sequence of characters\n",
    "#matches name 'woodchuck in text'\n",
    "txt = 'woodchuck'\n",
    "regexp = re.compile(r'woodchuck')\n",
    "\n",
    "if regexp.search(txt):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('Not in text')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not in text\n"
     ]
    }
   ],
   "source": [
    "#search words for both capital and small starting letter\n",
    "#this gives false since the word we are searching of starts with a capital W\n",
    "#Regex is case sensistive\n",
    "regexp = re.compile(r'Woodchuck')\n",
    "\n",
    "if regexp.search(txt):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('Not in text')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True!\n"
     ]
    }
   ],
   "source": [
    "#to solve the above problem\n",
    "regexp = re.compile(r'[wW]oodchuck')\n",
    "\n",
    "if regexp.search(txt):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('Not in text')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I', 'I']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#finding a sequence of  capital letters\n",
    "txt = 'I am so and so and I am 26 years old'\n",
    "regexp = re.compile(r'[A-Z]')\n",
    "\n",
    "regexp.findall(txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a',\n",
       " 'm',\n",
       " 's',\n",
       " 'o',\n",
       " 'a',\n",
       " 'n',\n",
       " 'd',\n",
       " 's',\n",
       " 'o',\n",
       " 'a',\n",
       " 'n',\n",
       " 'd',\n",
       " 'a',\n",
       " 'm',\n",
       " 'y',\n",
       " 'e',\n",
       " 'a',\n",
       " 'r',\n",
       " 's',\n",
       " 'o',\n",
       " 'l',\n",
       " 'd']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#finding a sequence of small letters\n",
    "regexp = re.compile(r'[a-z]')\n",
    "\n",
    "regexp.findall(txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2', '6']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#finding a sequence of single digits\n",
    "regexp = re.compile(r'[0-9]')\n",
    "\n",
    "regexp.findall(txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I', ' ', ' ', ' ', ' ', ' ', ' ', 'I', ' ', ' ', '2', '6', ' ', ' ']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#specify what a single charachter cannot be using ^\n",
    "# ^ needs to be the first symbol after the first open bracket\n",
    "regexp = re.compile(r'[^a-z]')\n",
    "regexp.findall(txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not in text\n"
     ]
    }
   ],
   "source": [
    "#matching the charachter before or nothing\n",
    "#looking for woodchucks. will not find it because of the additional s\n",
    "txt = 'woodchuck'\n",
    "regexp = re.compile(r'[wW]oodchucks')\n",
    "\n",
    "if regexp.search(txt):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('Not in text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "? is used to match zero or one instances of the previous character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not in text\n"
     ]
    }
   ],
   "source": [
    "#we can solve the above by adding the ?. \n",
    "regexp = re.compile(r'[wW]oodchucks?')\n",
    "\n",
    "if regexp.search(txt):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('Not in text')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "colour\n",
      "True!\n"
     ]
    }
   ],
   "source": [
    "#another example using color\n",
    "txt = input()\n",
    "regexp = re.compile(r'[cC]olou?r')\n",
    "\n",
    "if regexp.search(txt):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('Not in text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "using **Kleen *** which means matching zero or more occurences of the immediate previous character "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not in text\n"
     ]
    }
   ],
   "source": [
    "#searching for zero or more a or b's \n",
    "#txt = input()\n",
    "txt2 = 'Tund'\n",
    "regexp = re.compile(r'[e][e]*')\n",
    "\n",
    "if regexp.search(txt2):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('Not in text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Kleen +**\n",
    "specifying at least one of the charachters you use Kleen + so that you don't have to specify regular expression for digits twice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True!\n"
     ]
    }
   ],
   "source": [
    "#txt = input()\n",
    "txt2 = '2'\n",
    "regexp = re.compile(r'[0-9]+') #specifies a sequence of digits\n",
    "\n",
    "if regexp.search(txt2):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('Not in text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matching any single character between a word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example\n",
    "txt2 = 'begn'\n",
    "regexp = re.compile(r'beg.n') #\n",
    "\n",
    "if regexp.search(txt2):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('Not in text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding any line in which a particular word appears twice we use .* together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True!\n"
     ]
    }
   ],
   "source": [
    "# Example\n",
    "#try removing the word that is repeated to see how it works\n",
    "txt2 = 'my name name is she'\n",
    "regexp = re.compile(r'name.*name') #specifies a sequence of digits\n",
    "\n",
    "if regexp.search(txt2):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('Not ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Anchors** are special characters that anchor regular expressions to particular places in a string most popular are ^ and $.\n",
    "\n",
    "Example1:\n",
    "'^The' matches word The only at the start of a line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The is not at the start of the line \n"
     ]
    }
   ],
   "source": [
    "#Example 1\n",
    "txt2 = 'I am the girl is here'\n",
    "regexp = re.compile(r'^The')\n",
    "\n",
    "if regexp.search(txt2):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('The is not at the start of the line ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True!\n"
     ]
    }
   ],
   "source": [
    "#Example 2\n",
    "txt2 = 'The girl is here'\n",
    "regexp = re.compile(r'^The')\n",
    "\n",
    "if regexp.search(txt2):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('The is not at the start of the line')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NB**:\n",
    "The ^ (caret) character has **three** uses:\n",
    "\n",
    "- To match the start of a line ^The\n",
    "- To indicate a negation inside square brackets \\[^0-9\\] finds everything except numerical characters\n",
    "- And just to mean a caret \\[a^\\] finds a^"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 2: The $ sign matches the end of the line. so example ' $' is a useful pattern for matching a space at the end of the line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The dog']"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt2 = 'The dog.is here'\n",
    "regexp = re.compile(r'^The dog')\n",
    "\n",
    "regexp.findall(txt2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The dog.']"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt2 = 'The dog.'\n",
    "regexp = re.compile(r'^The dog\\.$') \n",
    "# the backslash \\ here is used to mean the character . not the wildcard . that we talked about earlier\n",
    "\n",
    "regexp.findall(txt2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two other **Anchors** used:\n",
    "\\b matches a word boundary\n",
    "\\B matches non-boundary\n",
    "\n",
    "Example below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "txt1 = 'The girl is one that goes to school there'\n",
    "txt2 = 'The girl is the one that goes to school there'\n",
    "regexp = re.compile(r'\\bthe\\b')\n",
    "\n",
    "if regexp.search(txt2):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('No the with boundaries')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True!\n"
     ]
    }
   ],
   "source": [
    "#this will match a string with the number 25. try with another number like 325, it wont match\n",
    "txt2 = 'The girl is the one that is 25 years old'\n",
    "regexp = re.compile(r'\\b25\\b')\n",
    "\n",
    "if regexp.search(txt2):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('25 is not in the text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Disjunction, Grouping and Precedence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **disjunction** operator is also known as the **pipe** | used when we want to match either one word or the other"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True!\n"
     ]
    }
   ],
   "source": [
    "txt2 = 'There is a dog in the house'\n",
    "regexp = re.compile(r'cat|dog')\n",
    "\n",
    "if regexp.search(txt2):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('No cat or dog in the text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make the disjunction operator apply only to a specific pattern, we need to use the parenthesis operators ( )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True!\n"
     ]
    }
   ],
   "source": [
    "#Example we want to match both puppy and puppies\n",
    "txt2 = 'There is a puppies in the house'\n",
    "regexp = re.compile(r'pupp(y|ies)')\n",
    "\n",
    "if regexp.search(txt2):\n",
    "    print('True!')\n",
    "else:\n",
    "    print('No cat or dog in the text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we want to match repeated instances of a string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Column 1 ', 'Column 2 ', 'Column 3']"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt2 = 'Column 1 Column 2 Column 3'\n",
    "#matches the word Column followed by a number and optional spaces & the \n",
    "#whole pattern repeated zero or more times\n",
    "regexp = re.compile(r'(Column [0-9]+ *)')\n",
    "\n",
    "regexp.findall(txt2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**greedy** vs **non-greedy** matching\n",
    "\n",
    "*read more on this*\n",
    "\n",
    "The operator *? is a Kleene star that matches as little text as\n",
    "possible. The operator +? is a Kleene plus that matches as little text as possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num = '45'\n",
    "re.findall('[5-9][0-9]{2}|[1-9][0-9]{3}[0-9]*',num)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Substitutions, Capture Groups and Eliza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In python, the sub function is used in replacing text in a string.\n",
    "It takes up 3 arguements:\n",
    "- The text to replace with\n",
    "- The text to replace in\n",
    "- The maximum number of substitutions to make\n",
    "\n",
    "sub returns a string consisting of the given text with the substitution(s) made"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hey\n"
     ]
    }
   ],
   "source": [
    "print (re.sub(r\"(.*)\\1\", r\"\\1\", \"HeyHey\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "#example 1\n",
    "mystring = 'There are 35 boxes'\n",
    "pattern = re.compile(r'/([0-9]+)/')\n",
    "newstring = pattern.sub(r\"/<\\1>/\", mystring)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "newstr2 = re.sub(r'([0-9]+)', r'<\\1>', mystring)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'There are <35> boxes'"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newstr2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "mystring2 = 'the Xer they were, the Xer they will be'\n",
    "newstring = pattern.sub(r'the (.*)er they were, the \\1er they will be', mystring2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the Xer they were, the Xer they will be'"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newstring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I AM SORRY TO HEAR YOU ARE depressed'"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user1 = \"I'm depressed\"\n",
    "pattern = re.compile(r\"I'm (depressed|sad)\")\n",
    "pattern.sub(r\"I AM SORRY TO HEAR YOU ARE \\1\", user1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I AM SORRY TO HEAR YOU ARE cate'"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user2 = \"my name is cate\"\n",
    "pattern = re.compile(r\"my name is (.*)\")\n",
    "pattern.sub(r\"I AM SORRY TO HEAR YOU ARE \\1\", user2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "newstring = pattern.sub('s/.*I am (.*).*/Hi  $1/', user2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}