crawl_test_no_framework.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "crawl_test_no_framework.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/soiqualang/colab/blob/master/crawl_test_no_framework.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PYHddEUY3-eM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import requests"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CpJiVMp84fxJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "r=requests.get('https://www.cleveland.com/metro/2017/12/case_western_reserve_university_president_barbara_snyders_base_salary_and_bonus_pay_tops_among_private_colleges_in_ohio.html')\n",
        "c=r.content"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g_epaSra4nkg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from bs4 import BeautifulSoup"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f_Viuvc64p2Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "soup=BeautifulSoup(c)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KREz_lbN57V0",
        "colab_type": "text"
      },
      "source": [
        "# Find Element"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "liuNyiv25UPy",
        "colab_type": "code",
        "outputId": "b85058e0-5945-425e-9e45-b070e06a1bb1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "main_content=soup.find('div',attrs={'class':'entry-content'})\n",
        "print(main_content)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<div class=\"entry-content\"> <p class=\"article__paragraph\">CLEVELAND, Ohio - <a href=\"http://www.case.edu/\">Case Western Reserve University </a>President Barbara Snyder's $700,312 base pay and $175,000 bonus in 2015 again made her the highest paid private college leader in Ohio, according to an annual report.</p><p class=\"article__paragraph\">Nationally, Snyder's total compensation of $1.154 million, which can include deferred compensation and other pay, ranked 41st out of 541 presidents of private colleges according to the report by the <a href=\"http://chronicle.com/\">Chronicle of Higher Education.</a></p> <div class=\"ad ad--in-article\" id=\"in-article_AOJSWF2SO5F6NLAFYE77J7FAPI\"></div> <p class=\"article__paragraph\">She is among 58 presidents to earn $1 million or more. The previous year, 39 private college presidents earned more than $1 million.</p><p class=\"article__paragraph\">The Chronicle's <a href=\"https://www.chronicle.com/interactives/executive-compensation#id=table_private_2015\">compensation tables </a>come from a review of federal tax documents from the 2015 calendar year, the most recent available.</p><p class=\"article__paragraph\">Former University of Dayton President Daniel Curran, who left in 2016, ranked eighth in the U.S. with a total compensation of $2.44 million, but $1.66 million of that was in other pay, which was likely deferred compensation awarded after he announced his retirement. His total pay in 2014 was $771,354.</p><p class=\"article__paragraph\">Following is total compensation for other presidents at private colleges in Ohio in 2015:</p><ul class=\"article__unordered-list\"> <li class=\"article__unordered-list-item\">Grant Cornwell, College of Wooster (left in 2015): $911,651</li> <li class=\"article__unordered-list-item\">Marvin Krislov, Oberlin College (left in 2016):  $829,913</li> <li class=\"article__unordered-list-item\">Mark Roosevelt, Antioch College, (left in 2015): $507,672</li> <li class=\"article__unordered-list-item\">Laurie Joyner, Wittenberg University (left in 2015): $463,504</li> <li class=\"article__unordered-list-item\">Richard Giese, University of Mount Union (left in 2015): $453,800</li> <li class=\"article__unordered-list-item\">Sean Decatur,Kenyon College: $451,698</li> <li class=\"article__unordered-list-item\">Adam Weinberg, Denison University: $435,322</li> <li class=\"article__unordered-list-item\">Daniel Dibiasio, Ohio Northern University: $414,716</li> <li class=\"article__unordered-list-item\">Denvy Bowman, Capital University (left in 2016): $388,570</li> <li class=\"article__unordered-list-item\">Anne Steele, Muskingum University (left in 2016): $384,233</li> <li class=\"article__unordered-list-item\">Kathy Krendl, Otterbein University:  $378,035</li> <li class=\"article__unordered-list-item\">Rockwell Jones, Ohio Wesleyan University: $366,625</li> <li class=\"article__unordered-list-item\">Robert Helmer, Baldwin Wallace University: $365,616</li> <li class=\"article__unordered-list-item\">Robert Huntington, Heidelberg University: $300,005</li> <li class=\"article__unordered-list-item\">Lori Varlotta, Hiram College: $293,336</li> <li class=\"article__unordered-list-item\">Joseph Bruno, Marietta College (left in 2016): $288,295</li> <li class=\"article__unordered-list-item\">W. Richard Merriman Jr., University of Mount Union (started in June 2015): $221,761</li> </ul> <p class=\"article__paragraph\">The Rev. Robert Niehoff, president of John Carroll University (retired in 2017), Rev. Sean Sheridan of  Franciscan University of Steubenville and Rev. Michael Graham Xavier University do not take salaries.</p><p class=\"article__paragraph\">See the full list of presidents at <a href=\"http://www.chronicle.com/compensation\">chronicle.com/compensation.</a></p><p class=\"article__paragraph\">The Chronicle's list of pay for public college presidents was released in July.</p><div class=\"rawhtml\"><a href=\"http://www.cleveland.com/metro/index.ssf/2017/06/ohio_state_university_presiden_6.html\">Ohio State University President Michael Drake is 7th highest paid U.S. public college president</a></div> <p class=\"article__paragraph\">Ohio State President Michael Drake, with a total compensation of $1.03 million, was one of eight presidents who earned $1 million or more in the 2015-16 fiscal year.</p> <div class=\"ad ad--in-article\" id=\"in-article_FXUGCLQM3VGPHBZIYHU7QYIX7M\"></div> <p class=\"article__paragraph\">But when compared to Ohio State's 2014 annual $5.2 billion operating budget, Drake's pay per million in expenses is $199 -- among the lowest for presidents.</p><p class=\"article__paragraph\">By comparison, Cleveland State University President Ronald Berkman, whose total compensation was $641,817, oversaw an annual operating budget of $300.3 million in 2014. His pay per million in expenses was $2,127.</p><p class=\"article__paragraph\">Kent State University President Beverly Warren's pay per million in expenses was $1,006. Her total compensation was $560,659 and she oversaw an annual operating budget of $557.4 million in 2014.</p><p class=\"article__paragraph\">The<a href=\"http://www.chronicle.com/interactives/executive-compensation\"> Chronicle's analysis </a>was based on 254 presidents from 221 colleges.</p><p class=\"article__paragraph\">According to the Chronicle:</p><ul class=\"article__unordered-list\"> <li class=\"article__unordered-list-item\">The average pay of private college presidents who served the full year in 2015 was $569,932, up 9 percent over presidents who served a full year in 2014.</li> <li class=\"article__unordered-list-item\">Nine of the 10 highest-paid chief executives earned more from bonuses and deferred compensation than they did from base salaries.</li> <li class=\"article__unordered-list-item\">The analysis is based on the latest available federal tax filings of the 500 private, nonprofit colleges with the largest endowments. The data include compensation figures for 568 presidents who served at 500 institutions for all or part of 2015. There were 352 presidents who served all of 2014 and 2015.</li> <li class=\"article__unordered-list-item\">The number of presidents earning over $1 million is attributed, in part, to a market where presidents are negotiating more deferred compensation and bonus packages before they take the job.</li> <li class=\"article__unordered-list-item\">The top two earners, Wake Forest University President Nathan Hatch ($4 million) and former Emory University President James Wagner ($3.5 million) were each paid more than $2 million in deferred compensation in 2015.</li> </ul> <p class=\"article__paragraph\">The Chronicle also compares pay to presidents in context of their colleges, including looking at each institution's expenses, average tuition and fees and professor salaries.</p></div>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WkmtiSsk6igx",
        "colab_type": "text"
      },
      "source": [
        "# Extract info"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WEtiSe445X2A",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "content=main_content.find('ul').text"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vozWKV8m8-gv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pprint"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PPqcMgyI9Ih4",
        "colab_type": "code",
        "outputId": "db8f7455-bc49-4717-b851-14cf20fc879c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        }
      },
      "source": [
        "pprint.pprint(content)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(' Grant Cornwell, College of Wooster (left in 2015): $911,651 Marvin Krislov, '\n",
            " 'Oberlin College (left in 2016): \\xa0$829,913 Mark Roosevelt, Antioch '\n",
            " 'College, (left in 2015): $507,672 Laurie Joyner, Wittenberg University (left '\n",
            " 'in 2015): $463,504 Richard Giese, University of Mount Union (left in 2015): '\n",
            " '$453,800 Sean Decatur,Kenyon College: $451,698 Adam Weinberg, Denison '\n",
            " 'University: $435,322 Daniel Dibiasio, Ohio Northern University: $414,716 '\n",
            " 'Denvy Bowman, Capital University (left in 2016): $388,570 Anne Steele, '\n",
            " 'Muskingum University (left in 2016): $384,233 Kathy Krendl, Otterbein '\n",
            " 'University: \\xa0$378,035 Rockwell Jones, Ohio Wesleyan University: $366,625 '\n",
            " 'Robert Helmer, Baldwin Wallace University: $365,616 Robert Huntington, '\n",
            " 'Heidelberg University: $300,005 Lori Varlotta, Hiram College: $293,336 '\n",
            " 'Joseph Bruno, Marietta College (left in 2016): $288,295 W. Richard Merriman '\n",
            " 'Jr., University of Mount Union (started in June 2015): $221,761 ')\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jGy5LC0Z-i44",
        "colab_type": "text"
      },
      "source": [
        "# Tách chuỗi Re"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CaPewSmx9j1o",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RKcM-tkN-mq3",
        "colab_type": "code",
        "outputId": "78801005-7f4e-4d14-a3c7-ad8870a22ab2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "name_pattern = re.compile(r'^([A-Z]{1}.+?)(?:,)', flags = re.M)\n",
        "name_pattern.findall(content)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oLSK0fZN-pL3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "names = name_pattern.findall(content)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aOQZY0eR-rY3",
        "colab_type": "code",
        "outputId": "f9ca156d-d44e-4348-c850-b95c030c5a3b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(names)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Besp_BXI-tXv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}