Step2_Find_Subreddits.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": "### Step 2: Find Subreddits\nThis step runs combinations of the NLP category words as well as the Wiki term through the Reddit subreddit search API to find subreddits that are likely to contain the words. After pulling back many possibilities for subreddits, it uses cosine similarity between the potential subreddit (using a combination of name, title, and description) and the wiki categories to select the \"best\" subreddits to interrogate for information on our Next Big Thing.",
   "metadata": {
    "cell_id": "a7d0af07a36f490583e420c83172cb38",
    "tags": [],
    "deepnote_cell_type": "markdown",
    "deepnote_cell_height": 167.5625
   }
  },
  {
   "cell_type": "code",
   "metadata": {
    "tags": [],
    "deepnote_to_be_reexecuted": false,
    "source_hash": "31ae4d42",
    "execution_start": 1650825288237,
    "execution_millis": 150,
    "cell_id": "14840197-9846-4755-96fb-0a4c95f8eaca",
    "deepnote_cell_type": "code",
    "deepnote_cell_height": 153
   },
   "source": "import numpy as np \nimport pandas as pd\nimport praw \nfrom datetime import datetime, timezone, timedelta, date\nimport pickle",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {
    "tags": [],
    "deepnote_to_be_reexecuted": false,
    "source_hash": "5e8d6b02",
    "execution_start": 1650825288391,
    "execution_millis": 3,
    "cell_id": "00001-8c49cd3a-5d98-462f-a3c5-dffc0a6c7ab0",
    "deepnote_cell_type": "code",
    "deepnote_cell_height": 171
   },
   "source": "### reddit and app credentials\nREDDIT_USERNAME = 'nbt_capstone '\nAPP_ID = '38HKH06bLOdSlpZIVtO5-w'\nAPP_SECRET = 'dmU0nd_1tYIDn9AldwjEDgYU-hj-Lw' \nAPP_NAME = 'nextbigthing'\nPASSWORD = 'Capstone698!'",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {
    "tags": [],
    "deepnote_to_be_reexecuted": false,
    "source_hash": "75f30b9f",
    "execution_start": 1650825288396,
    "execution_millis": 48,
    "cell_id": "00002-aa630991-0b70-41b9-8bcc-8a8bed72611c",
    "deepnote_cell_type": "code",
    "deepnote_cell_height": 171
   },
   "source": "#Set up the Reddit API in python\nreddit = praw.Reddit(client_id=APP_ID,\n                     client_secret=APP_SECRET,\n                     user_agent=REDDIT_USERNAME)\n\n",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {
    "cell_id": "55f9a929-25bb-43c2-95f3-9f3f485e9e56",
    "tags": [],
    "deepnote_to_be_reexecuted": false,
    "source_hash": "332c8bf",
    "execution_start": 1650825288490,
    "execution_millis": 14075,
    "deepnote_cell_type": "code",
    "deepnote_cell_height": 171
   },
   "source": "#Set up the language model\n\n# https://www.sbert.net/docs/pretrained_models.html\nfrom sentence_transformers import SentenceTransformer, util\n\nmodel = SentenceTransformer('all-MiniLM-L6-v2')",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {
    "cell_id": "5eaca369-2388-44cf-b1cb-e3a2c3888690",
    "tags": [],
    "deepnote_to_be_reexecuted": false,
    "source_hash": "278458b0",
    "execution_start": 1650825302570,
    "execution_millis": 4,
    "deepnote_cell_type": "code",
    "deepnote_cell_height": 441
   },
   "source": "def get_search_terms(nlp_category):\n    '''This function creates single search terms by using the last word provided and adding one word to\n       the front of this term until there are no words left.\n       So, ['American','financial','corporation'] would result in 3 search terms: \n       \"corporation\", \"financial corporation\", and \"American financial corporation\"\n       \n       Input:\n       nlp_category - a list of one or more comma separated single words\n       \n       Output:\n       terms - a list of one or more comma separated multi-words'''\n\n    terms = []\n    for i, word in enumerate(reversed(nlp_category)):\n        if i == 0:\n            terms.append(nlp_category[-1])\n        else:\n            terms.append(word + ' ' + terms[i-1])\n        \n\n    return terms",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {
    "tags": [],
    "deepnote_to_be_reexecuted": false,
    "source_hash": "aee2da33",
    "execution_start": 1650825302580,
    "execution_millis": 6,
    "cell_id": "00003-aa3dd601-2e34-4e95-9469-fae8b3b499bc",
    "deepnote_cell_type": "code",
    "deepnote_cell_height": 549
   },
   "source": "def search_subreddits(search_term):\n    '''This function uses the reddit search method to search subreddits for a search term\n    Input: \n    search_term - a string value\n\n    Output: \n    subreddit_details - a list containing a list for each subreddit found. The inner list\n                                 contains the details of the subreddit: name, title, description,\n                                 count of subscribers, and over18 flag\n    '''\n\n    #Subreddits are searched by both title and description for keyword\n    subreddits = reddit.subreddits\n    \n    subreddits_result = subreddits.search(search_term)\n    subreddit_details = []\n    \n    for subreddit in subreddits_result:\n        subreddit_details.append([subreddit.display_name, \n                                subreddit.title,\n                                subreddit.public_description,\n                                subreddit.subscribers,\n                                subreddit.over18])\n\n    \n        \n    return subreddit_details",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {
    "tags": [],
    "deepnote_to_be_reexecuted": false,
    "source_hash": "9b318167",
    "execution_start": 1650825302586,
    "execution_millis": 8,
    "cell_id": "00004-1fe0a3d2-341b-4172-b569-1605336146d6",
    "deepnote_table_state": {
     "pageSize": 10,
     "pageIndex": 2,
     "filters": [],
     "sortBy": []
    },
    "deepnote_table_loading": false,
    "deepnote_cell_type": "code",
    "deepnote_cell_height": 1035
   },
   "source": "def run_subreddit_search_options(orig_entry,nlp_category,wiki_categories):\n    '''Search reddit using multiple methods to return relevant subreddits\n    Inputs: \n    orig_entry - Wikipedia term for string entered by user\n    nlp_category - list containing nlp category words for orig_entry\n    wiki_categories - Wikipedia categories for the string entered by user\n\n    Output: \n    subreddit_details_df - dataframe containing all subreddits found and their details,\n                            including name, title, description, subsriber count, and over18 flag\n    '''\n\n    #print('Wikipedia term is: ',orig_entry)\n    #print('nlp_category is: ',nlp_category)\n    #print('******************************')\n\n    #Use search on the the collection of words in the nlp_category\n    # print('Using search endpoint on terms in the nlp_category: ')\n    search_nlp_category_list = []\n\n    nlp_category_terms = get_search_terms(nlp_category)\n    \n    for term in nlp_category_terms:\n        \n        term_subreddits = search_subreddits(term)\n        search_nlp_category_list.append(term_subreddits)\n        # print('term is: ',term)\n        # print('term subreddits are: ',[subreddit[0] for subreddit in term_subreddits])\n    \n    #Flatten the results\n    search_nlp_category_list = [item for sublist in search_nlp_category_list for item in sublist]\n    # print('Count: ',len(search_nlp_category_list))\n\n    #Use search on the original entry\n    search_entry_list = search_subreddits(orig_entry)\n    # print('Using search endpoint on original entry: ')\n    # print('original entry subreddits are: ',[subreddit[0] for subreddit in search_entry_list])\n    # print('Count: ',len(search_entry_list))\n    # print()\n\n    #Put the results together \n    full_subreddit_list = search_nlp_category_list + search_entry_list\n    # full_subreddit_list = search_nlp_category_list + search_entry_list + search_wiki_categories_list\n\n    #Add results to dataframe\n    subreddit_details_df = pd.DataFrame(full_subreddit_list,\n        columns=['display_name', 'title', 'description', 'subscriber_count', 'over18'])\n\n    #print('Count of Unique Subreddits: ',subreddit_details_df['display_name'].nunique())\n    #print(subreddit_details_df['display_name'].unique())\n\n\n    return subreddit_details_df\n",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {
    "cell_id": "bf26b24a-04cd-4b15-985d-f173fa990436",
    "tags": [],
    "deepnote_to_be_reexecuted": false,
    "source_hash": "96568652",
    "execution_start": 1650825302595,
    "execution_millis": 62,
    "deepnote_cell_type": "code",
    "deepnote_cell_height": 621
   },
   "source": "def get_subreddit_by_cos_sim_categories(subreddits, wiki_categories, subreddit_details):\n    '''Creates cosine similarity scores for a list of subreddit with details against a list of \n       wikipedia categories\n    \n    Inputs: \n    subreddits - list of subreddits\n    wiki_categories - wikipedia categories for the NBT original entry\n    subreddit_details - concatenation of subreddit display name, title,and public description\n            \n    Output: \n    avg_df - a dataframe containing each subreddit and its average cosine similarity score\n            over the list of wikipedia categories'''\n    cosine_result = []\n\n    subreddits_embeddings = model.encode(subreddit_details)\n\n    for wiki_category in wiki_categories: \n        category_embedding= model.encode(wiki_category)\n\n        for i, subreddit in enumerate(subreddit_details):\n            cos_similarity = util.pytorch_cos_sim(category_embedding, subreddits_embeddings[i]).numpy()[0][0]\n            cosine_result.append([wiki_category, subreddits[i],subreddit,cos_similarity])\n\n    cos_df = pd.DataFrame(cosine_result,columns=['wiki_category','subreddit','subreddit_details','cosine_similarity'])\n    \n    avg_df = cos_df.groupby(['subreddit','subreddit_details']).mean().reset_index()\n    avg_df = avg_df.rename(columns={'cosine_similarity':'avg_cosine_similarity'})\n    avg_df['subreddit_details_wiki_categories_rank'] = avg_df['avg_cosine_similarity'].rank(method='max',ascending=False)\n    \n    return avg_df\n",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "metadata": {
    "cell_id": "58cce812-76dd-4580-91b6-68ea8a0177f5",
    "tags": [],
    "deepnote_to_be_reexecuted": false,
    "source_hash": "e93a9a00",
    "execution_start": 1650825302674,
    "execution_millis": 46,
    "owner_user_id": "d1c3db5a-4505-48be-99b3-4a4061f5f711",
    "deepnote_cell_type": "code",
    "deepnote_cell_height": 1917
   },
   "source": "def get_subreddits(orig_entry, nlp_category, wiki_categories, num_subreddits, min_subreddit_subscribers,mvp_flag):\n    '''Identify subreddits most likely to contain information about siblings of an entered string \n       (orig_entry) using subreddit search methods and cosine similarity between the subreddit details\n       and wikpedia categories. \n       \n        Inputs: \n        orig_entry - Wikipedia term for string entered by user\n        nlp_category - list containing nlp category words for orig_entry\n        wiki_categories - list of filtered wikipedia categories found for the orig_entry\n        num_subreddits - number of subreddits to return\n        min_subreddit_subscribers - smallest number of subscribers a subreddit must have in order\n                                    to be included for consideration\n        mvp_flag - indicates if we are running a pre-aggregated data set for our minimum viable\n                   product\n                            \n        Output: \n        subreddits_list - list of display names for selected subreddits'''\n\n    #Set pickle file name, will either be creating it or loading it\n    orig_entry_mod = orig_entry.replace(' ','_')\n    file_name = './output_step2/subreddits_' + orig_entry_mod + '.pickle'\n\n    #If we are using a pre-aggregated data set, load its pickle file\n    if mvp_flag:\n        print('MVP mode - loading pickle file...')\n\n        #Load pickle file of relevant posts\n        try:\n            with open(file_name, 'rb') as f:\n                subreddits_list = pickle.load(f)\n        except:\n            print('Unable to find pickle file for', orig_entry)\n            subreddits_list = []\n\n    else:\n\n        #Remove 'Category:' from beginning of wikipedia categories\n        wiki_categories = [cat[9:] for cat in wiki_categories]\n\n        #Find subreddits based on api search \n        #print('Initial Reddit Search start time: ',datetime.now(timezone(timedelta(hours=-4), 'EST')).strftime('%Y-%m-%d %H:%M:%S'))\n        \n        subreddit_details_df = run_subreddit_search_options(orig_entry, nlp_category, wiki_categories)\n\n        #print('Initial Reddit Search end time: ',datetime.now(timezone(timedelta(hours=-4), 'EST')).strftime('%Y-%m-%d %H:%M:%S'))\n\n        #Remove any duplicate subreddits, keep last because it was the most recent query\n        subreddit_details_df = subreddit_details_df.drop_duplicates(subset = 'display_name',keep = 'last')\n\n        #Remove subreddits that are nsfw and have less than a certain number of subscribers\n        #print('Original number of subreddits: ',len(subreddit_details_df))\n        subreddit_details_filtered_df = \\\n            subreddit_details_df[(subreddit_details_df['subscriber_count'] >= min_subreddit_subscribers) & \n            (subreddit_details_df['over18']==False)]\n        #print('New # of subreddits after filtering by nsfw and subscriber count: ',len(subreddit_details_filtered_df))\n\n        #Remove subreddits that have not had a recent post\n        current_subreddit_list = list(subreddit_details_filtered_df['display_name'])\n        last_post_dates = []\n        for subreddit in current_subreddit_list:\n            last_post = []\n            for post in reddit.subreddit(subreddit).new(limit = 1):\n                last_post.append(subreddit)\n                last_post.append(datetime.fromtimestamp(post.created_utc).date())\n            last_post_dates.append(last_post)\n        \n        last_post_df = pd.DataFrame(last_post_dates,columns=['subreddit','last_post'])\n        \n        last_post_df['active'] = last_post_df['last_post'] + timedelta(days=3) > date.today()\n        last_post_df = last_post_df[last_post_df['active']==True]\n        subreddit_details_filtered_df = subreddit_details_filtered_df.merge(last_post_df, \n                            how = 'inner', left_on = 'display_name', right_on = 'subreddit')\n\n        #print('New # of subreddits after filtering out those with no recent posts: ',len(subreddit_details_filtered_df))\n\n        #Cosine Similarity to choose best subreddits\n        #print('Filtering and cos sim final selection start time: ',datetime.now(timezone(timedelta(hours=-4), 'EST')).strftime('%Y-%m-%d %H:%M:%S'))\n        \n        #Prep data for cosine similarity by concatenating the subreddit name, title, and description\n        subreddit_details_filtered_df['full_details'] = subreddit_details_filtered_df['display_name'] + \\\n            '.' + subreddit_details_filtered_df['title'] + '.' + subreddit_details_filtered_df['description']\n        \n        subreddit_details = list(subreddit_details_filtered_df['full_details'])\n        subreddits = list(subreddit_details_filtered_df['display_name'])\n        \n        #Run cosine similarity to rank results\n        cos_sim_result_df = get_subreddit_by_cos_sim_categories(subreddits, wiki_categories, \n                                                                            subreddit_details)\n        \n        subreddits_result_df = cos_sim_result_df.nsmallest(num_subreddits,'subreddit_details_wiki_categories_rank')[['subreddit','subreddit_details','subreddit_details_wiki_categories_rank']]\n        \n        subreddits_list = list(subreddits_result_df['subreddit'])\n\n        #print('Filtering and cos sim final selection end time: ',datetime.now(timezone(timedelta(hours=-4), 'EST')).strftime('%Y-%m-%d %H:%M:%S'))\n    \n        #Create pickle file from our subreddits\n       \n        with open (file_name, 'wb') as f:\n            pickle.dump(subreddits_list, f)\n\n\n    return subreddits_list\n",
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": "<a style='text-decoration:none;line-height:16px;display:flex;color:#5B5B62;padding:10px;justify-content:end;' href='https://deepnote.com?utm_source=created-in-deepnote-cell&projectId=90b052a7-f47d-474e-888f-9345355cfd9a' target=\"_blank\">\n<img alt='Created in deepnote.com' style='display:inline;max-height:16px;margin:0px;margin-right:7.5px;' src='data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHN2ZyB3aWR0aD0iODBweCIgaGVpZ2h0PSI4MHB4IiB2aWV3Qm94PSIwIDAgODAgODAiIHZlcnNpb249IjEuMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayI+CiAgICA8IS0tIEdlbmVyYXRvcjogU2tldGNoIDU0LjEgKDc2NDkwKSAtIGh0dHBzOi8vc2tldGNoYXBwLmNvbSAtLT4KICAgIDx0aXRsZT5Hcm91cCAzPC90aXRsZT4KICAgIDxkZXNjPkNyZWF0ZWQgd2l0aCBTa2V0Y2guPC9kZXNjPgogICAgPGcgaWQ9IkxhbmRpbmciIHN0cm9rZT0ibm9uZSIgc3Ryb2tlLXdpZHRoPSIxIiBmaWxsPSJub25lIiBmaWxsLXJ1bGU9ImV2ZW5vZGQiPgogICAgICAgIDxnIGlkPSJBcnRib2FyZCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTEyMzUuMDAwMDAwLCAtNzkuMDAwMDAwKSI+CiAgICAgICAgICAgIDxnIGlkPSJHcm91cC0zIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMjM1LjAwMDAwMCwgNzkuMDAwMDAwKSI+CiAgICAgICAgICAgICAgICA8cG9seWdvbiBpZD0iUGF0aC0yMCIgZmlsbD0iIzAyNjVCNCIgcG9pbnRzPSIyLjM3NjIzNzYyIDgwIDM4LjA0NzY2NjcgODAgNTcuODIxNzgyMiA3My44MDU3NTkyIDU3LjgyMTc4MjIgMzIuNzU5MjczOSAzOS4xNDAyMjc4IDMxLjY4MzE2ODMiPjwvcG9seWdvbj4KICAgICAgICAgICAgICAgIDxwYXRoIGQ9Ik0zNS4wMDc3MTgsODAgQzQyLjkwNjIwMDcsNzYuNDU0OTM1OCA0Ny41NjQ5MTY3LDcxLjU0MjI2NzEgNDguOTgzODY2LDY1LjI2MTk5MzkgQzUxLjExMjI4OTksNTUuODQxNTg0MiA0MS42NzcxNzk1LDQ5LjIxMjIyODQgMjUuNjIzOTg0Niw0OS4yMTIyMjg0IEMyNS40ODQ5Mjg5LDQ5LjEyNjg0NDggMjkuODI2MTI5Niw0My4yODM4MjQ4IDM4LjY0NzU4NjksMzEuNjgzMTY4MyBMNzIuODcxMjg3MSwzMi41NTQ0MjUgTDY1LjI4MDk3Myw2Ny42NzYzNDIxIEw1MS4xMTIyODk5LDc3LjM3NjE0NCBMMzUuMDA3NzE4LDgwIFoiIGlkPSJQYXRoLTIyIiBmaWxsPSIjMDAyODY4Ij48L3BhdGg+CiAgICAgICAgICAgICAgICA8cGF0aCBkPSJNMCwzNy43MzA0NDA1IEwyNy4xMTQ1MzcsMC4yNTcxMTE0MzYgQzYyLjM3MTUxMjMsLTEuOTkwNzE3MDEgODAsMTAuNTAwMzkyNyA4MCwzNy43MzA0NDA1IEM4MCw2NC45NjA0ODgyIDY0Ljc3NjUwMzgsNzkuMDUwMzQxNCAzNC4zMjk1MTEzLDgwIEM0Ny4wNTUzNDg5LDc3LjU2NzA4MDggNTMuNDE4MjY3Nyw3MC4zMTM2MTAzIDUzLjQxODI2NzcsNTguMjM5NTg4NSBDNTMuNDE4MjY3Nyw0MC4xMjg1NTU3IDM2LjMwMzk1NDQsMzcuNzMwNDQwNSAyNS4yMjc0MTcsMzcuNzMwNDQwNSBDMTcuODQzMDU4NiwzNy43MzA0NDA1IDkuNDMzOTE5NjYsMzcuNzMwNDQwNSAwLDM3LjczMDQ0MDUgWiIgaWQ9IlBhdGgtMTkiIGZpbGw9IiMzNzkzRUYiPjwvcGF0aD4KICAgICAgICAgICAgPC9nPgogICAgICAgIDwvZz4KICAgIDwvZz4KPC9zdmc+' > </img>\nCreated in <span style='font-weight:600;margin-left:4px;'>Deepnote</span></a>",
   "metadata": {
    "tags": [],
    "created_in_deepnote_cell": true,
    "deepnote_cell_type": "markdown"
   }
  }
 ],
 "nbformat": 4,
 "nbformat_minor": 2,
 "metadata": {
  "orig_nbformat": 2,
  "deepnote": {
   "is_reactive": false
  },
  "deepnote_notebook_id": "b432badc-d5fb-440f-b21b-2c8508969dad",
  "deepnote_execution_queue": []
 }
}