006_Python_RegEx.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<small><small><i>\n",
    "All the IPython Notebooks in **Python Advanced Topics** lecture series by Dr. Milaan Parmar are available @ **[GitHub](https://github.com/milaan9/07_Python_Advanced_Topics)**\n",
    "</i></small></small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python RegEx\n",
    "\n",
    "In this class, you will learn about regular expressions (RegEx), and use Python's re module to work with RegEx (with the help of examples)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **Reg**ular **Ex**pression (RegEx) is a sequence of characters that defines a search pattern. For example,\n",
    "\n",
    "```python\n",
    ">>> ^a...s$\n",
    "```\n",
    "\n",
    "The above code defines a RegEx pattern. The pattern is: **any five letter string starting with `a` and ending with `s`**.\n",
    "\n",
    "A pattern defined using RegEx can be used to match against a string.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|               |  **`abs`** | No match | \n",
    "|               |  **`alias`** | Match |\n",
    "| **`^a...s$`** |  **`abyss`** | Match |\n",
    "|               |  **`Alias`** | No match |\n",
    "|               |  **`An abacus`** | No match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python has a module named **`re`** to work with RegEx. Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:12.328032Z",
     "start_time": "2021-10-19T12:31:12.315374Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search successful.\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "pattern = '^a...s$'\n",
    "test_string = 'abyss'\n",
    "result = re.match(pattern, test_string)\n",
    "\n",
    "if result:\n",
    "    print(\"Search successful.\")\n",
    "else:\n",
    "    print(\"Search unsuccessful!\")\t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we used **`re.match()`** function to search **`pattern`** within the **`test_string`**. The method returns a match object if the search is successful. If not, it returns **`None`**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are other several functions defined in the **`re`** module to work with RegEx. Before we explore that, let's learn about regular expressions themselves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify Pattern Using RegEx\n",
    "\n",
    "To specify regular expressions, metacharacters are used. In the above example, **`^`** and **`$`** are metacharacters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MetaCharacters\n",
    "\n",
    "Metacharacters are characters that are interpreted in a special way by a RegEx engine. Here's a list of metacharacters:\n",
    "\n",
    "###  **`[] . ^ $ * + ? {} () \\ |`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. `[]` - Square brackets\n",
    "\n",
    "Square brackets specifies a set of characters you wish to match.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|             |  **`a`** | 1 match | \n",
    "|             |  **`ac`** | 2 matches |\n",
    "| **`[abc]`** |  **`Hey Jude`** | No match |\n",
    "|             |  **`abc de ca`** | 5 matches |\n",
    "\n",
    "Here, **`[abc]`** will match if the string you are trying to match contains any of the **`a`**, **`b`** or **`c`**.\n",
    "\n",
    "You can also specify a range of characters using **`-`** inside square brackets.\n",
    "\n",
    "* **`[a-c]`** means, a or b or c\n",
    "* **`[a-z]`** means, any letter from a to z\n",
    "* **`[A-Z]`** means, any character from A to Z\n",
    "* **`[0-3]`** means, 0 or 1 or 2 or 3\n",
    "* **`[0-9]`** means any number from 0 to 9\n",
    "* **`[A-Za-z0-9]`** any single character, that is a to z, A to Z or 0 to 9\n",
    "\n",
    "You can complement (invert) the character set by using caret **`^`** symbol at the start of a square-bracket.\n",
    "\n",
    "* **`[^abc]`** means any character except **`a`** or **`b`** or **`c`**.\n",
    "* **`[^0-9]`** means any non-digit character."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. `.` - Period\n",
    "\n",
    "A period matches any single character (except newline **`'\\n'`**).\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|          |  **`a`** | No match | \n",
    "|          |  **`ac`** | 1 match |\n",
    "| **`..`** |  **`acd`** | 1 match |\n",
    "|          |  **`acde`** | 2 matches (contains 4 characters) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. `^` - Caret\n",
    "\n",
    "The caret symbol **`^`** is used to check if a string **starts with** a certain character.\n",
    "\n",
    "  * **`r'^substring'`** eg **`r'^love'`**, a sentence that starts with a word love\n",
    "  * **`r'[^abc]`** means not a, not b, not c.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|          |  **`a`** | 1 match | \n",
    "| **`^a`** |  **`abc`** | 1 match |\n",
    "|          |  **`bac`** | No match |\n",
    "|----------|------------|------------------------------------------------------------|\n",
    "| **`^a`** |  **`abc`** | 1 match |\n",
    "|          |  **`acd`** | No match (starts with **`a`** but not followed by **`b`**) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. `$` - Dollar\n",
    "\n",
    "The dollar symbol **`$`** is used to check if a string **ends with** a certain character.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|          |  **`a`** | 1 match | \n",
    "| **`a$`** |  **`formula`** | 1 match |\n",
    "|          |  **`cab`** | No result |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. `*` - Star\n",
    "\n",
    "The star symbol **`*`** matches **zero or more occurrences** of the pattern left to it.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|            |  **`mn`** | 1 match | \n",
    "|            |  **`man`** | 1 match |\n",
    "| **`ma*n`** |  **`maaan`** | 1 match |\n",
    "|            |  **`main`** | No match (**`a`** is not followed by **`n`**) |\n",
    "|            |  **`woman`** | 1 match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. `+` - Plus\n",
    "\n",
    "The plus symbol **`+`** matches **zero or more occurrences** of the pattern left to it.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|            |  **`mn`** | No match (no **`a`** character) | \n",
    "|            |  **`man`** | 1 match |\n",
    "| **`ma+n`** |  **`maaan`** | 1 match |\n",
    "|            |  **`main`** | No match (**`a`** is not followed by **`n`**) |\n",
    "|            |  **`woman`** | 1 match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. `?` - Question Mark\n",
    "\n",
    "The question mark symbol **`?`** matches **zero or more occurrences** of the pattern left to it.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|            |  **`mn`** | 1 match | \n",
    "|            |  **`man`** | 1 match |\n",
    "| **`ma?n`** |  **`maaan`** | No match (more than one **`a`** character) |\n",
    "|            |  **`main`** | No match (**`a`** is not followed by **`n`**) |\n",
    "|            |  **`woman`** | 1 match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. `{}` - Braces\n",
    "\n",
    "We can specify the length of the substring we are looking for in a text, using a curly brackets **`{}`**.\n",
    "\n",
    "Consider this code: **`{n,m}`**. This means at least **`n`**, and at most **`m`** repetitions of the pattern left to it.\n",
    "\n",
    "* **`{3}`**: Exactly 3 characters\n",
    "* **`{3,}`**: At least 3 characters\n",
    "* **`{3,8}`**: 3 to 8 characters\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|              |  **`abc dat`** | No match | \n",
    "|              |  **`abc daat`** | 1 match (at **`daat`**) |\n",
    "| **`a{2,3}`** |  **`aabc daaat`** | 2 matches (at **`aabc`** and **`daaat`**) |\n",
    "|              |  **`aabc daaaat`** | 2 matches (at **`aabc`** and **`daaaat`**) |\n",
    "\n",
    "Let's try one more example. This RegEx **`[0-9]{2, 4}`** matches at least 2 digits but not more than 4 digits\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|                  |  **`ab123csde`** | 1 match (match at **`ab123csde`**) | \n",
    "| **`[0-9]{2,4}`** |  **`12 and 345673`** | 3 matches (**`12`**, **`3456`**, **`73`**) |\n",
    "|                  |  **`1 and 2`** | No match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. `|` - Alternation\n",
    "\n",
    "Vertical bar **`|`** is used for alternation (**`or`** operator).\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|           |  **`cde`** | No match | \n",
    "| **`a\\|b`** |  **`ade`** | 1 match (match at **`ade`**) |\n",
    "|           |  **`acdbea`** | 3 matches (at **`acdbea`**) |\n",
    "\n",
    "Here, **`a|b`** match any string that contains either **`a`** or **`b`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. `()` - Group\n",
    "\n",
    "Parentheses **`()`** is used to group sub-patterns. For example, **`(a|b|c)xz`** match any string that matches either **`a`** or **`b`** or **`c`**. followed by **`xz`**.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|                 |  **`ab xz`** | No match | \n",
    "| **`(a\\|b\\|c)xz`** |  **`abxz`** | 1 match (match at **`abxz`**) |\n",
    "|                 |  **`axz cabxz`** | 2 matches (at **`axz cabxz`**) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. `\\` - Backslash\n",
    "\n",
    "Backlash **`\\`** is used to escape various characters including all metacharacters. For example,\n",
    "\n",
    "**`\\$a`** match if a string contains **`$`** followed by **`a`**. Here, **`$`** is not interpreted by a RegEx engine in a special way.\n",
    "\n",
    "If you are unsure if a character has special meaning or not, you can put **`\\`** in front of it. This makes sure the character is not treated in a special way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Special Sequences\n",
    "\n",
    "Special sequences make commonly used patterns easier to write. Here's a list of special sequences:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `\\A` - Matches if the specified characters are at the start of a string.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "| **`\\Athe`** |  **`the sun`** | Match | \n",
    "|             |  **`In the sun`** | No match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `\\b` - Matches if the specified characters are at the beginning or end of a word.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|             |  **`football`** | Match | \n",
    "| **`\\bfoo`** |  **`a football`** | Match |\n",
    "|             |  **`afootball`** | No match |\n",
    "|-------------|------------|-----------|\n",
    "| **`foo\\b`** |  **`the foo`** | Match |\n",
    "|             |  **`the afoo test`** | Match |\n",
    "|             |  **`the afootest`** | No match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `\\B` - Opposite of `\\b`. Matches if the specified characters are not at the beginning or end of a word.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|             |  **`football`** | No match | \n",
    "| **`\\Bfoo`** |  **`a football`** | No match |\n",
    "|             |  **`afootball`** | Match |\n",
    "|-------------|------------|-----------|\n",
    "| **`foo\\B`** |  **`the foo`** | No match |\n",
    "|             |  **`the afoo test`** | No match |\n",
    "|             |  **`the afootest`** | Match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `\\d` - Matches any non-decimal digit. Equivalent to `[^0-9]`\n",
    "\n",
    "- **`\\d`** means: match where the string contains digits (numbers from 0-9)\n",
    "- **`\\D`** means: match where the string does not contain digits\n",
    "    \n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "| **`\\d`** |  **`1ab34\"50`** | 3 matches (at **`1ab34\"50`**) | \n",
    "|          |  **`1345`** | No match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `\\s` - Matches where a string contains any whitespace character. Equivalent to `[ \\t\\n\\r\\f\\v]`.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "| **`\\s`** |  **`Python RegEx`** | 1 match | \n",
    "|          |  **`PythonRegEx`** | No match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `\\S` - Matches where a string contains any non-whitespace character. Equivalent to `[^ \\t\\n\\r\\f\\v]`.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "| **`\\S`** |  **`a b`** | 2 matches (at  **`a b`**) | \n",
    "|          |  **` `** | No match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `\\w` - Matches any alphanumeric character (digits and alphabets). Equivalent to `[a-zA-Z0-9_]`. By the way, underscore `_` is also considered an alphanumeric character.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "| **`\\w`** |  **`12&\": ;c`** | 3 matches (at **`12&\": ;c`**) | \n",
    "|          |  **`%\"> !`** | No match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `\\W` - Matches any non-alphanumeric character. Equivalent to `[^a-zA-Z0-9_]`.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "| **`\\W`** |  **`1a2%c`** | 1 match (at **`1a2%c`**) | \n",
    "|          |  **`Python`** | No match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `\\Z` - Matches if the specified characters are at the end of a string.\n",
    "\n",
    "| Expression | String | Matched? | \n",
    "|:----| :--- |:--- |\n",
    "|                |  **`I like Python`** | 1 match |\n",
    "| **`Python\\Z`** |  **`I like Python Programming`** | No match | \n",
    "|                |  **`Python is fun.`** | No match |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Summary - MetaCharacters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "<div>\n",
    "<img src=\"img/regex.png\" width=\"1000\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    ">**Tip**: To build and test regular expressions, you can use RegEx tester tools such as **[regex101](https://regex101.com/)**. This tool not only helps you in creating regular expressions, but it also helps you learn it.\n",
    "\n",
    "Now you understand the basics of RegEx, let's discuss how to use RegEx in your Python code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "A **Reg**ular **Ex**pression or **RegEx** is a special text string that helps to find patterns in data. A RegEx can be used to check if some pattern exists in a different data type. To use RegEx in python first we should import the RegEx module which is called **`re`**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Python RegEx Methods\n",
    "\n",
    "To find a pattern we use different set of **`re`** character sets that allows to search for a match in a string.\n",
    "\n",
    "* **`re.findall`**: Returns a list containing all matches\n",
    "* **`re.split`**:\tTakes a string, splits it at the match points, returns a list\n",
    "* **`re.sub`**:  Replaces one or many matches within a string\n",
    "* **`re.search`**: Returns a match object if there is one anywhere in the string, including multiline strings.\n",
    "* **`re.match()`**: searches only in the beginning of the first line of the string and returns matched objects if found, else returns **`None`**.\n",
    "\n",
    "To use it, we need to import the module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:12.437103Z",
     "start_time": "2021-10-19T12:31:12.360170Z"
    },
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The module defines several functions and constants to work with RegEx."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 1. `re.findall()`\n",
    "\n",
    "The **`re.findall()`** method returns a list of strings containing all matches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:12.545683Z",
     "start_time": "2021-10-19T12:31:12.440996Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['12', '89', '34']\n"
     ]
    }
   ],
   "source": [
    "# Example 1: re.findall()\n",
    "\n",
    "# Program to extract numbers from a string\n",
    "\n",
    "import re\n",
    "\n",
    "string = 'hello 12 hi 89. Howdy 34'\n",
    "pattern = '\\d+'\n",
    "\n",
    "result = re.findall(pattern, string) \n",
    "print(result)\n",
    "\n",
    "# Output: ['12', '89', '34']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "If the pattern is not found, **`re.findall()`** returns an empty list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:12.639170Z",
     "start_time": "2021-10-19T12:31:12.550551Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['language', 'language']\n"
     ]
    }
   ],
   "source": [
    "# Example 2: re.findall()\n",
    "\n",
    "txt = '''Python is the most beautiful language that a human being has ever created.\n",
    "I recommend python for a first programming language'''\n",
    "\n",
    "# It return a list\n",
    "matches = re.findall('language', txt, re.I)\n",
    "print(matches)  # ['language', 'language']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "As you can see, the word **`language`** was found two times in the string. Let us practice some more.\n",
    "Now we will look for both **`Python`** and **`python`** words in the string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:12.732170Z",
     "start_time": "2021-10-19T12:31:12.643064Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Python', 'python']\n"
     ]
    }
   ],
   "source": [
    "# Example 3: re.findall()\n",
    "\n",
    "txt = '''Python is the most beautiful language that a human being has ever created.\n",
    "I recommend python for a first programming language'''\n",
    "\n",
    "# It returns list\n",
    "matches = re.findall('python', txt, re.I)\n",
    "print(matches)  # ['Python', 'python']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Since we are using **`re.I`** both lowercase and uppercase letters are included. If we do not have the **`re.I`** flag, then we will have to write our pattern differently. Let us check it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:12.856333Z",
     "start_time": "2021-10-19T12:31:12.735091Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Python', 'python']\n",
      "['Python', 'python']\n"
     ]
    }
   ],
   "source": [
    "# Example 4: re.findall()\n",
    "\n",
    "txt = '''Python is the most beautiful language that a human being has ever created.\n",
    "I recommend python for a first programming language'''\n",
    "\n",
    "matches = re.findall('Python|python', txt)\n",
    "print(matches)  # ['Python', 'python']\n",
    "\n",
    "matches = re.findall('[Pp]ython', txt)\n",
    "print(matches)  # ['Python', 'python']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 2. `re.split()`\n",
    "\n",
    "The **`re.split`** method splits the string where there is a match and returns a list of strings where the splits have occurred."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:12.996076Z",
     "start_time": "2021-10-19T12:31:12.863149Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I am teacher and  I love teaching.', 'There is nothing as rewarding as educating and empowering people.', 'I found teaching more interesting than any other jobs.', 'Does this motivate you to be a teacher?']\n"
     ]
    }
   ],
   "source": [
    "# Example 1: re.split()\n",
    "\n",
    "txt = '''I am teacher and  I love teaching.\n",
    "There is nothing as rewarding as educating and empowering people.\n",
    "I found teaching more interesting than any other jobs.\n",
    "Does this motivate you to be a teacher?'''\n",
    "print(re.split('\\n', txt)) # splitting using \\n - end of line symbol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:13.135822Z",
     "start_time": "2021-10-19T12:31:12.998997Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Twelve:', ' Eighty nine:', '.']\n"
     ]
    }
   ],
   "source": [
    "# Example 2: re.split()\n",
    "\n",
    "import re\n",
    "\n",
    "string = 'Twelve:12 Eighty nine:89.'\n",
    "pattern = '\\d+'\n",
    "\n",
    "result = re.split(pattern, string) \n",
    "print(result)\n",
    "\n",
    "# Output: ['Twelve:', ' Eighty nine:', '.']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "If the pattern is not found, **`re.split`** returns a list containing the original string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "You can pass **`maxsplit`** argument to the **`re.split`** method. It's the maximum number of splits that will occur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:13.314030Z",
     "start_time": "2021-10-19T12:31:13.141663Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Twelve:', ' Eighty nine:89 Nine:9.']\n"
     ]
    }
   ],
   "source": [
    "# Example 3: re.split()\n",
    "\n",
    "import re\n",
    "\n",
    "string = 'Twelve:12 Eighty nine:89 Nine:9.'\n",
    "pattern = '\\d+'\n",
    "\n",
    "# maxsplit = 1\n",
    "# split only at the first occurrence\n",
    "result = re.split(pattern, string, 1) \n",
    "print(result)\n",
    "\n",
    "# Output: ['Twelve:', ' Eighty nine:89 Nine:9.']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "By the way, the default value of **`maxsplit`** is 0; meaning all possible splits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 3. `re.sub()`\n",
    "\n",
    "**Syntax:**\n",
    "\n",
    "```python\n",
    "re.sub(pattern, replace, string)\n",
    "```\n",
    "\n",
    "The method returns a string where matched occurrences are replaced with the content of **`replace`** variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:13.431862Z",
     "start_time": "2021-10-19T12:31:13.327176Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "abc12de23f456\n"
     ]
    }
   ],
   "source": [
    "# Example 1: re.sub()\n",
    "\n",
    "# Program to remove all whitespaces\n",
    "import re\n",
    "\n",
    "# multiline string\n",
    "string = 'abc 12\\\n",
    "de 23 \\n f45 6'\n",
    "\n",
    "# matches all whitespace characters\n",
    "pattern = '\\s+'\n",
    "\n",
    "# empty string\n",
    "replace = ''\n",
    "\n",
    "new_string = re.sub(pattern, replace, string) \n",
    "print(new_string)\n",
    "\n",
    "# Output: abc12de23f456"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "If the pattern is not found, **`re.sub()`** returns the original string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "You can pass **`count`** as a fourth parameter to the **`re.sub()`** method. If omited, it results to 0. This will replace all occurrences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:13.555048Z",
     "start_time": "2021-10-19T12:31:13.434784Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "abc12de 23 \n",
      " f45 6\n"
     ]
    }
   ],
   "source": [
    "# Example 2: re.sub()\n",
    "\n",
    "import re\n",
    "\n",
    "# multiline string\n",
    "string = 'abc 12\\\n",
    "de 23 \\n f45 6'\n",
    "\n",
    "# matches all whitespace characters\n",
    "pattern = '\\s+'\n",
    "replace = ''\n",
    "\n",
    "new_string = re.sub(r'\\s+', replace, string, 1) \n",
    "print(new_string)\n",
    "\n",
    "# Output:\n",
    "# abc12de 23\n",
    "# f45 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:13.664605Z",
     "start_time": "2021-10-19T12:31:13.557971Z"
    },
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "JavaScript is the most beautiful language that a human being has ever created.\n",
      "I recommend JavaScript for a first programming language\n",
      "JavaScript is the most beautiful language that a human being has ever created.\n",
      "I recommend JavaScript for a first programming language\n"
     ]
    }
   ],
   "source": [
    "# Example 3: re.sub()\n",
    "\n",
    "txt = '''Python is the most beautiful language that a human being has ever created.\n",
    "I recommend python for a first programming language'''\n",
    "\n",
    "match_replaced = re.sub('Python|python', 'JavaScript', txt, re.I)\n",
    "print(match_replaced)  # JavaScript is the most beautiful language that a human being has ever created.\n",
    "# OR\n",
    "match_replaced = re.sub('[Pp]ython', 'JavaScript', txt, re.I)\n",
    "print(match_replaced)  # JavaScript is the most beautiful language that a human being has ever created.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let us add one more example. The following string is really hard to read unless we remove the **`%`** symbol. Replacing the **`%`** with an empty string will clean the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:13.773672Z",
     "start_time": "2021-10-19T12:31:13.667528Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I am teacher and  I love teaching. \n",
      "There is nothing as rewarding as educating and empowering people.\n",
      "I found teaching more interesting than any other jobs. \n",
      "Does this motivate you to be a teacher?\n"
     ]
    }
   ],
   "source": [
    "# Example 4: re.sub()\n",
    "\n",
    "txt = '''%I a%m te%%a%%che%r% a%n%d %% I l%o%ve te%ach%ing. \n",
    "T%he%re i%s n%o%th%ing as r%ewarding a%s e%duc%at%i%ng a%n%d e%m%p%ow%er%ing p%e%o%ple.\n",
    "I fo%und te%a%ching m%ore i%n%t%er%%es%ting t%h%an any other %jobs. \n",
    "D%o%es thi%s m%ot%iv%a%te %y%o%u to b%e a t%e%a%cher?'''\n",
    "\n",
    "matches = re.sub('%', '', txt)\n",
    "print(matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 4. `re.subn()`\n",
    "\n",
    "The **`re.subn()`** is similar to **`re.sub()`** expect it returns a tuple of 2 items containing the new string and the number of substitutions made."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:13.882253Z",
     "start_time": "2021-10-19T12:31:13.778545Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('abc12de23f456', 4)\n"
     ]
    }
   ],
   "source": [
    "# Example 1: re.subn()\n",
    "\n",
    "# Program to remove all whitespaces\n",
    "import re\n",
    "\n",
    "# multiline string\n",
    "string = 'abc 12\\\n",
    "de 23 \\n f45 6'\n",
    "\n",
    "# matches all whitespace characters\n",
    "pattern = '\\s+'\n",
    "\n",
    "# empty string\n",
    "replace = ''\n",
    "\n",
    "new_string = re.subn(pattern, replace, string) \n",
    "print(new_string)\n",
    "\n",
    "# Output: ('abc12de23f456', 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 5. `re.search()`\n",
    "\n",
    "The **`re.search()`** method takes two arguments: a pattern and a string. The method looks for the first location where the RegEx pattern produces a match with the string.\n",
    "\n",
    "If the search is successful, **`re.search()`** returns a match object; if not, it returns **`None`**.\n",
    "\n",
    "```python\n",
    "match = re.search(pattern, str)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:14.006906Z",
     "start_time": "2021-10-19T12:31:13.885175Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pattern found inside the string\n"
     ]
    }
   ],
   "source": [
    "# Example 1: re.search()\n",
    "\n",
    "import re\n",
    "\n",
    "string = \"Python is fun\"\n",
    "\n",
    "# check if 'Python' is at the beginning\n",
    "match = re.search('\\APython', string)\n",
    "\n",
    "if match:\n",
    "    print(\"pattern found inside the string\")\n",
    "else:\n",
    "    print(\"pattern not found\")  \n",
    "\n",
    "# Output: pattern found inside the string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Here, **`match`** contains a match object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:14.133017Z",
     "start_time": "2021-10-19T12:31:14.010801Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(100, 105), match='first'>\n",
      "(100, 105)\n",
      "100 105\n",
      "first\n"
     ]
    }
   ],
   "source": [
    "# Example 2: re.search()\n",
    "\n",
    "import re\n",
    "\n",
    "txt = '''Python is the most beautiful language that a human being has ever created.\n",
    "I recommend python for a first programming language'''\n",
    "\n",
    "# It returns an object with span and match\n",
    "match = re.search('first', txt, re.I)\n",
    "print(match)  # <re.Match object; span=(100, 105), match='first'>\n",
    "# We can get the starting and ending position of the match as tuple using span\n",
    "span = match.span()\n",
    "print(span)     # (100, 105)\n",
    "# Lets find the start and stop position from the span\n",
    "start, end = span\n",
    "print(start, end)  # 100 105\n",
    "substring = txt[start:end]\n",
    "print(substring)       # first"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Match object\n",
    "\n",
    "You can get methods and attributes of a match object using **[dir()](https://github.com/milaan9/04_Python_Functions/blob/main/002_Python_Functions_Built_in/016_Python_dir().ipynb)** function.\n",
    "\n",
    "Some of the commonly used methods and attributes of match objects are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:14.177809Z",
     "start_time": "2021-10-19T12:31:14.141780Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(0, 15), match='I love to teach'>\n",
      "(0, 15)\n",
      "0 15\n",
      "I love to teach\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "txt = 'I love to teach python and javaScript'\n",
    "# It returns an object with span, and match\n",
    "match = re.match('I love to teach', txt, re.I)\n",
    "print(match)  # <re.Match object; span=(0, 15), match='I love to teach'>\n",
    "# We can get the starting and ending position of the match as tuple using span\n",
    "span = match.span()\n",
    "print(span)     # (0, 15)\n",
    "# Lets find the start and stop position from the span\n",
    "start, end = span\n",
    "print(start, end)  # 0, 15\n",
    "substring = txt[start:end]\n",
    "print(substring)       # I love to teach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "As you can see from the example above, the pattern we are looking for (or the substring we are looking for) is **`I love to teach`**. The match function returns an object **only** if the text starts with the pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:14.209464Z",
     "start_time": "2021-10-19T12:31:14.185601Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "txt = 'I love to teach python and javaScript'\n",
    "match = re.match('I like to teach', txt, re.I)\n",
    "print(match)  # None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The string does not string with **`I like to teach`**, therefore there was no match and the match method returned **`None`**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 1. `match.group()`\n",
    "\n",
    "The **`group()`** method returns the part of the string where there is a match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:14.393998Z",
     "start_time": "2021-10-19T12:31:14.213355Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "801 35\n"
     ]
    }
   ],
   "source": [
    "# Example 6: Match object\n",
    "\n",
    "import re\n",
    "\n",
    "string = '39801 356, 2102 1111'\n",
    "\n",
    "# Three digit number followed by space followed by two digit number\n",
    "pattern = '(\\d{3}) (\\d{2})'\n",
    "\n",
    "# match variable contains a Match object.\n",
    "match = re.search(pattern, string) \n",
    "\n",
    "if match:\n",
    "    print(match.group())\n",
    "else:\n",
    "    print(\"pattern not found\")\n",
    "\n",
    "# Output: 801 35"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Here, match variable contains a match object.\n",
    "\n",
    "Our pattern **`(\\d{3}) (\\d{2})`** has two subgroups **`(\\d{3})`** and **`(\\d{2})`**. You can get the part of the string of these parenthesized subgroups. Here's how:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:14.518650Z",
     "start_time": "2021-10-19T12:31:14.396921Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'801'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.group(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:14.643787Z",
     "start_time": "2021-10-19T12:31:14.523520Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'35'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.group(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:14.767947Z",
     "start_time": "2021-10-19T12:31:14.646708Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('801', '35')"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.group(1, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:14.892597Z",
     "start_time": "2021-10-19T12:31:14.772818Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('801', '35')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.groups()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 2. `match.start(), match.end() and match.span()`\n",
    "\n",
    "The **`start()`** function returns the index of the start of the matched substring. Similarly, **`end()`** returns the end index of the matched substring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:14.986572Z",
     "start_time": "2021-10-19T12:31:14.903798Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:15.173059Z",
     "start_time": "2021-10-19T12:31:14.991441Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The **`span()`** function returns a tuple containing start and end index of the matched part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:15.357112Z",
     "start_time": "2021-10-19T12:31:15.175979Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 8)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.span()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 3. `match.re and match.string`\n",
    "\n",
    "The **`re`** attribute of a matched object returns a regular expression object. Similarly, **`string`** attribute returns the passed string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:15.495395Z",
     "start_time": "2021-10-19T12:31:15.360033Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "re.compile(r'(\\d{3}) (\\d{2})', re.UNICODE)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:15.603977Z",
     "start_time": "2021-10-19T12:31:15.502211Z"
    },
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'39801 356, 2102 1111'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "match.string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We have covered all commonly used methods defined in the **`re`** module.\n",
    "\n",
    "If you want to learn more, visit **[Python 3 re module](https://docs.python.org/3/library/re.html)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using `r` prefix before RegEx\n",
    "\n",
    "When **`r`** or **`R`** prefix is used before a regular expression, it means raw string. For example, **`'\\n'`** is a new line whereas **`r'\\n'`** means two characters: a backslash **`\\`** followed by **`n`**.\n",
    "\n",
    "Backlash **`\\`** is used to escape various characters including all metacharacters. However, using **`r`** prefix makes **`\\`** treat as a normal character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:15.712558Z",
     "start_time": "2021-10-19T12:31:15.606897Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\\n', '\\r']\n"
     ]
    }
   ],
   "source": [
    "# Example 7: Raw string using r prefix\n",
    "\n",
    "import re\n",
    "\n",
    "string = '\\n and \\r are escape sequences.'\n",
    "\n",
    "result = re.findall(r'[\\n\\r]', string) \n",
    "print(result)\n",
    "\n",
    "# Output: ['\\n', '\\r']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example of RegEx with Metacharacters\n",
    "\n",
    "Let us use examples to clarify the meta characters with RegEx methods:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Square Brackets `[]`\n",
    "\n",
    "Let us use square bracket to include lower and upper case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:15.867882Z",
     "start_time": "2021-10-19T12:31:15.715479Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Apple', 'apple']\n"
     ]
    }
   ],
   "source": [
    "# Example 1:\n",
    "\n",
    "regex_pattern = r'[Aa]pple' # this square bracket mean either A or a\n",
    "txt = 'Apple and banana are fruits. An old cliche says an apple a day a doctor way has been replaced by a banana a day keeps the doctor far far away.'\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['Apple', 'apple']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to look for the both apple and banana, we write the pattern as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:16.054367Z",
     "start_time": "2021-10-19T12:31:15.881516Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Apple', 'banana', 'apple', 'banana']\n"
     ]
    }
   ],
   "source": [
    "# Example 2:\n",
    "\n",
    "regex_pattern = r'[Aa]pple|[Bb]anana' # this square bracket means either A or a\n",
    "txt = 'Apple and banana are fruits. An old cliche says an apple a day a doctor way has been replaced by a banana a day keeps the doctor far far away.'\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['Apple', 'banana', 'apple', 'banana']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the square brackets **`[]`** and or operator **`|`** , we manage to extract Apple, apple, Banana"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Escape character `\\`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:16.163438Z",
     "start_time": "2021-10-19T12:31:16.058264Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['8', '1', '9', '4', '2', '1', '4', '2', '0', '1', '8', '7', '6']\n"
     ]
    }
   ],
   "source": [
    "# Example 1:\n",
    "\n",
    "regex_pattern = r'\\d'  # d is a special character which means digits\n",
    "txt = \"Hawking born on 8 January 1942 and died on 14 March 2018 Einstein's birth anniversary (Pi-Day) and both died at 76\"\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['8', '1', '9', '4', '2', '1', '4', '2', '0', '1', '8', '7', '6'] - this is not what we want"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One or more times `+`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:16.382549Z",
     "start_time": "2021-10-19T12:31:16.171716Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['8', '1942', '14', '2018', '76']\n"
     ]
    }
   ],
   "source": [
    "# Example 1:\n",
    "\n",
    "regex_pattern = r'\\d+'  # d is a special character which means digits, + mean one or more times\n",
    "txt = \"Hawking born on 8 January 1942 and died on 14 March 2018 Einstein's birth anniversary (Pi-Day) and both died at 76\"\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['8', '1942', '14', '2018', '76'] - this is better!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Period `.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:16.659111Z",
     "start_time": "2021-10-19T12:31:16.394236Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['an', 'an', 'an', 'a ', 'ar']\n"
     ]
    }
   ],
   "source": [
    "# Example 1:\n",
    "\n",
    "regex_pattern = r'[a].'  # this square bracket means a and . means any character except new line\n",
    "txt = '''Apple and Banana are fruits'''\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['an', 'an', 'an', 'a ', 'ar']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:16.860209Z",
     "start_time": "2021-10-19T12:31:16.662036Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['and Banana are fruits']\n"
     ]
    }
   ],
   "source": [
    "# Example 2: [] with +\n",
    "\n",
    "regex_pattern = r'[a].+'  # . any character, + any character one or more times \n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['and Banana are fruits']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Zero or more times `*`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:17.014072Z",
     "start_time": "2021-10-19T12:31:16.864104Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['and Banana are fruits']\n"
     ]
    }
   ],
   "source": [
    "# Example 1:\n",
    "\n",
    "regex_pattern = r'[a].*'  # . any character, * any character zero or more times \n",
    "txt = '''Apple and Banana are fruits'''\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['and Banana are fruits']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Zero or one time `?`\n",
    "\n",
    "Zero or one time. The pattern may not occur or it may occur once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:17.138727Z",
     "start_time": "2021-10-19T12:31:17.016994Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['e-mail', 'email', 'Email', 'E-mail']\n"
     ]
    }
   ],
   "source": [
    "# Example 1:\n",
    "\n",
    "txt = '''I am not sure if there is a convention how to write the word e-mail.\n",
    "Some people write it as email others may write it as Email or E-mail.'''\n",
    "regex_pattern = r'[Ee]-?mail'  # ? means here that '-' is optional\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['e-mail', 'email', 'Email', 'E-mail']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quantifier `{}`\n",
    "\n",
    "We can specify the length of the substring we are looking for in a text, using a curly brackets **`{}`**. Let us imagine, we are interested in a substring with a length of 4 characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:17.337869Z",
     "start_time": "2021-10-19T12:31:17.149921Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['1942', '2018']\n"
     ]
    }
   ],
   "source": [
    "# Example 1:\n",
    "\n",
    "txt = \"Hawking born on 8 January 1942 and died on 14 March 2018 Einstein's birth anniversary (Pi-Day) and both died at 76\"\n",
    "regex_pattern = r'\\d{4}'  # exactly four times\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['1942', '2018']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:17.507799Z",
     "start_time": "2021-10-19T12:31:17.340791Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['8', '1942', '14', '2018', '76']\n"
     ]
    }
   ],
   "source": [
    "# Example 2:\n",
    "\n",
    "txt = \"Hawking born on 8 January 1942 and died on 14 March 2018 Einstein's birth anniversary (Pi-Day) and both died at 76\"\n",
    "regex_pattern = r'\\d{1,4}'   # 1 to 4\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['8', '1942', '76', '14', '2018', '76']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cart `^`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:17.630989Z",
     "start_time": "2021-10-19T12:31:17.510723Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hawking']\n"
     ]
    }
   ],
   "source": [
    "# Example 1: Starts with\n",
    "\n",
    "txt = \"Hawking born on 8 January 1942 and died on 14 March 2018 Einstein's birth anniversary (Pi-Day) and both died at 76\"\n",
    "regex_pattern = r'^Hawking'  # ^ means starts with\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['Hawking']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-19T12:31:17.817477Z",
     "start_time": "2021-10-19T12:31:17.633911Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['8', '1942', '14', '2018', \"'\", '(', '-', ')', '76']\n"
     ]
    }
   ],
   "source": [
    "# Example 2: Negation\n",
    "\n",
    "txt = \"Hawking born on 8 January 1942 and died on 14 March 2018 Einstein's birth anniversary (Pi-Day) and both died at 76\"\n",
    "regex_pattern = r'[^A-Za-z ]+'  # ^ in set character means negation, not A to Z, not a to z, no space\n",
    "matches = re.findall(regex_pattern, txt)\n",
    "print(matches)  # ['8', '1942', '14', '2018', \"'\", '(', '-', ')', '76']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 💻 Exercises ➞ <span class='label label-default'>RegEx</span>\n",
    "\n",
    "### Exercises ➞ <span class='label label-default'>Level 1</span>\n",
    "\n",
    " 1. What is the most frequent word in the following paragraph?\n",
    "    - ```py\n",
    "paragraph = 'I love teaching. If you do not love teaching what else can you love. I love Python if you do not love something which can give you all the capabilities to develop an application what else can you love.\n",
    "    ```\n",
    "\n",
    "   - ```sy\n",
    "    [\n",
    "    (6, 'love'),\n",
    "    (5, 'you'),\n",
    "    (3, 'can'),\n",
    "    (2, 'what'),\n",
    "    (2, 'teaching'),\n",
    "    (2, 'not'),\n",
    "    (2, 'else'),\n",
    "    (2, 'do'),\n",
    "    (2, 'I'),\n",
    "    (1, 'which'),\n",
    "    (1, 'to'),\n",
    "    (1, 'the'),\n",
    "    (1, 'something'),\n",
    "    (1, 'if'),\n",
    "    (1, 'give'),\n",
    "    (1, 'develop'),\n",
    "    (1, 'capabilities'),\n",
    "    (1, 'application'),\n",
    "    (1, 'an'),\n",
    "    (1, 'all'),\n",
    "    (1, 'Python'),\n",
    "    (1, 'If')\n",
    "    ]\n",
    "    ```\n",
    "\n",
    "2. The position of some particles on the horizontal x-axis are -12, -4, -3 and -1 in the negative direction, 0 at origin, 4 and 8 in the positive direction. Extract these numbers from this whole text and find the distance between the two furthest particles.\n",
    "\n",
    "    - ```py\n",
    "points = ['-1', '2', '-4', '-3', '-1', '0', '4', '8']\n",
    "sorted_points =  [-4, -3, -1, -1, 0, 2, 4, 8]\n",
    "distance = 8 -(-4) # 12\n",
    "    ```\n",
    "\n",
    "### Exercises ➞ <span class='label label-default'>Level 2</span>\n",
    "\n",
    "1. Write a pattern which identifies if a string is a valid python variable\n",
    "\n",
    "    - ```py\n",
    "    is_valid_variable('first_name')  # True\n",
    "    is_valid_variable('first-name')  # False\n",
    "    is_valid_variable('1first_name') # False\n",
    "    is_valid_variable('firstname')   # True\n",
    "    ```\n",
    "\n",
    "\n",
    "### Exercises ➞ <span class='label label-default'>Level 3</span>\n",
    "\n",
    "1. Clean the following text. After cleaning, count three most frequent words in the string.\n",
    "\n",
    "    - ```py\n",
    "    sentence = '''%I $am@% a %tea@cher%, &and& I lo%#ve %tea@ching%;. There $is nothing; &as& mo@re rewarding as educa@ting &and& @emp%o@wering peo@ple. ;I found tea@ching m%o@re interesting tha@n any other %jo@bs. %Do@es thi%s mo@tivate yo@u to be a tea@cher!?'''\n",
    "\n",
    "    print(clean_text(sentence));\n",
    "    I am a teacher and I love teaching There is nothing as more rewarding as educating and empowering people I found teaching more interesting than any other jobs Does this motivate you to be a teacher\n",
    "    print(most_frequent_words(cleaned_text)) # [(3, 'I'), (2, 'teaching'), (2, 'teacher')]\n",
    "    ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}