Machine Learning with Text in scikit-learn (ClickBait Detection).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning with Text in scikit-learn  \n",
    "\n",
    "Forked from [justmarkham/pycon-2016-tutorial](http://github.com/justmarkham/pycon-2016-tutorial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agenda\n",
    "\n",
    "1. Model building in scikit-learn (refresher)\n",
    "2. Representing text as numerical data\n",
    "3. Reading a text-based dataset into pandas\n",
    "4. Vectorizing our dataset\n",
    "5. Building and evaluating a model\n",
    "6. Comparing models\n",
    "7. Examining a model for further insight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Model building in scikit-learn (refresher)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the iris dataset as an example\n",
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# store the feature matrix (X) and response vector (y)\n",
    "X = iris.data\n",
    "y = iris.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**\"Features\"** are also known as predictors, inputs, or attributes. The **\"response\"** is also known as the target, label, or output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150, 4)\n",
      "(150,)\n"
     ]
    }
   ],
   "source": [
    "# check the shapes of X and y\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**\"Observations\"** are also known as samples, instances, or records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)\n",
       "0                5.1               3.5                1.4               0.2\n",
       "1                4.9               3.0                1.4               0.2\n",
       "2                4.7               3.2                1.3               0.2\n",
       "3                4.6               3.1                1.5               0.2\n",
       "4                5.0               3.6                1.4               0.2"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine the first 5 rows of the feature matrix (including the feature names)\n",
    "import pandas as pd\n",
    "pd.DataFrame(X, columns=iris.feature_names).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
      " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n"
     ]
    }
   ],
   "source": [
    "# examine the response vector\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to **build a model**, the features must be **numeric**, and every observation must have the **same features in the same order**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import the class\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# instantiate the model (with the default parameters)\n",
    "knn = KNeighborsClassifier()\n",
    "\n",
    "# fit the model with data (occurs in-place)\n",
    "knn.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to **make a prediction**, the new observation must have the **same features as the training observations**, both in number and meaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# predict the response for a new observation\n",
    "knn.predict([[3, 5, 4, 2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Representing text as numerical data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# example text for model training (Article Titles)\n",
    "simple_train = ['9 Shape Tips for Wedding season', 'Barack leaves office in January']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the [scikit-learn documentation](http://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction):\n",
    "\n",
    "> Text Analysis is a major application field for machine learning algorithms. However the raw data, a sequence of symbols cannot be fed directly to the algorithms themselves as most of them expect **numerical feature vectors with a fixed size** rather than the **raw text documents with variable length**.\n",
    "\n",
    "We will use [CountVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) to \"convert text into a matrix of token counts\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import and instantiate CountVectorizer (with the default parameters)\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vect = CountVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=None, vocabulary=None)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# learn the 'vocabulary' of the training data (occurs in-place)\n",
    "vect.fit(simple_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['barack',\n",
       " 'for',\n",
       " 'in',\n",
       " 'january',\n",
       " 'leaves',\n",
       " 'office',\n",
       " 'season',\n",
       " 'shape',\n",
       " 'tips',\n",
       " 'wedding']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine the fitted vocabulary\n",
    "vect.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<2x10 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 10 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# transform training data into a 'document-term matrix'\n",
    "simple_train_dtm = vect.transform(simple_train)\n",
    "simple_train_dtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 0, 0, 0, 0, 1, 1, 1, 1],\n",
       "       [1, 0, 1, 1, 1, 1, 0, 0, 0, 0]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert sparse matrix to a dense matrix\n",
    "simple_train_dtm.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>barack</th>\n",
       "      <th>for</th>\n",
       "      <th>in</th>\n",
       "      <th>january</th>\n",
       "      <th>leaves</th>\n",
       "      <th>office</th>\n",
       "      <th>season</th>\n",
       "      <th>shape</th>\n",
       "      <th>tips</th>\n",
       "      <th>wedding</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   barack  for  in  january  leaves  office  season  shape  tips  wedding\n",
       "0       0    1   0        0       0       0       1      1     1        1\n",
       "1       1    0   1        1       1       1       0      0     0        0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine the vocabulary and document-term matrix together\n",
    "pd.DataFrame(simple_train_dtm.toarray(), columns=vect.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the [scikit-learn documentation](http://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction):\n",
    "\n",
    "> In this scheme, features and samples are defined as follows:\n",
    "\n",
    "> - Each individual token occurrence frequency (normalized or not) is treated as a **feature**.\n",
    "> - The vector of all the token frequencies for a given document is considered a multivariate **sample**.\n",
    "\n",
    "> A **corpus of documents** can thus be represented by a matrix with **one row per document** and **one column per token** (e.g. word) occurring in the corpus.\n",
    "\n",
    "> We call **vectorization** the general process of turning a collection of text documents into numerical feature vectors. This specific strategy (tokenization, counting and normalization) is called the **Bag of Words** or \"Bag of n-grams\" representation. Documents are described by word occurrences while completely ignoring the relative position information of the words in the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "scipy.sparse.csr.csr_matrix"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the type of the document-term matrix\n",
    "type(simple_train_dtm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 1)\t1\n",
      "  (0, 6)\t1\n",
      "  (0, 7)\t1\n",
      "  (0, 8)\t1\n",
      "  (0, 9)\t1\n",
      "  (1, 0)\t1\n",
      "  (1, 2)\t1\n",
      "  (1, 3)\t1\n",
      "  (1, 4)\t1\n",
      "  (1, 5)\t1\n"
     ]
    }
   ],
   "source": [
    "# examine the sparse matrix contents\n",
    "print(simple_train_dtm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the [scikit-learn documentation](http://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction):\n",
    "\n",
    "> As most documents will typically use a very small subset of the words used in the corpus, the resulting matrix will have **many feature values that are zeros** (typically more than 99% of them).\n",
    "\n",
    "> For instance, a collection of 10,000 short text documents (such as emails) will use a vocabulary with a size in the order of 100,000 unique words in total while each document will use 100 to 1000 unique words individually.\n",
    "\n",
    "> In order to be able to **store such a matrix in memory** but also to **speed up operations**, implementations will typically use a **sparse representation** such as the implementations available in the `scipy.sparse` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# example text for model testing\n",
    "simple_test = [\"5 tips and tricks for college\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to **make a prediction**, the new observation must have the **same features as the training observations**, both in number and meaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 0, 0, 0, 0, 0, 0, 1, 0]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# transform testing data into a document-term matrix (using existing vocabulary)\n",
    "simple_test_dtm = vect.transform(simple_test)\n",
    "simple_test_dtm.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>barack</th>\n",
       "      <th>for</th>\n",
       "      <th>in</th>\n",
       "      <th>january</th>\n",
       "      <th>leaves</th>\n",
       "      <th>office</th>\n",
       "      <th>season</th>\n",
       "      <th>shape</th>\n",
       "      <th>tips</th>\n",
       "      <th>wedding</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   barack  for  in  january  leaves  office  season  shape  tips  wedding\n",
       "0       0    1   0        0       0       0       0      0     1        0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine the vocabulary and document-term matrix together\n",
    "pd.DataFrame(simple_test_dtm.toarray(), columns=vect.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Summary:**\n",
    "\n",
    "- `vect.fit(train)` **learns the vocabulary** of the training data\n",
    "- `vect.transform(train)` uses the **fitted vocabulary** to build a document-term matrix from the training data\n",
    "- `vect.transform(test)` uses the **fitted vocabulary** to build a document-term matrix from the testing data (and **ignores tokens** it hasn't seen before)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Reading a text-based dataset into pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# read file into pandas using a relative path\n",
    "path = 'data/links.csv'\n",
    "links = pd.read_csv(path, header=0 ,names=['label', 'title'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9500, 2)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine the shape\n",
    "links.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Joseph Schooling beats Michael Phelps to claim...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Bill Clinton: Email controversy is the 'bigges...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Hacker releases cell phone numbers, personal e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Lionel Messi announces Argentina return</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Fighting the male biological clock by banking ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>genuine</td>\n",
       "      <td>The face of the Olympics will never look the same</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Trump: If Clinton wins Pennsylvania, she cheated</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Texas baby found dead after nine hours in hot car</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Malawi is moving 500 elephants across the country</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Thomas Gibson fired from 'Criminal Minds' afte...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     label                                              title\n",
       "0  genuine  Joseph Schooling beats Michael Phelps to claim...\n",
       "1  genuine  Bill Clinton: Email controversy is the 'bigges...\n",
       "2  genuine  Hacker releases cell phone numbers, personal e...\n",
       "3  genuine            Lionel Messi announces Argentina return\n",
       "4  genuine  Fighting the male biological clock by banking ...\n",
       "5  genuine  The face of the Olympics will never look the same\n",
       "6  genuine   Trump: If Clinton wins Pennsylvania, she cheated\n",
       "7  genuine  Texas baby found dead after nine hours in hot car\n",
       "8  genuine  Malawi is moving 500 elephants across the country\n",
       "9  genuine  Thomas Gibson fired from 'Criminal Minds' afte..."
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine the first 10 rows\n",
    "links.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "genuine      6657\n",
       "clickbait    2843\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine the class distribution\n",
    "links.label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# convert label to a numerical variable\n",
    "links['label_num'] = links.label.map({'genuine':0, 'clickbait':1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>title</th>\n",
       "      <th>label_num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Joseph Schooling beats Michael Phelps to claim...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Bill Clinton: Email controversy is the 'bigges...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Hacker releases cell phone numbers, personal e...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Lionel Messi announces Argentina return</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>genuine</td>\n",
       "      <td>Fighting the male biological clock by banking ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     label                                              title  label_num\n",
       "0  genuine  Joseph Schooling beats Michael Phelps to claim...          0\n",
       "1  genuine  Bill Clinton: Email controversy is the 'bigges...          0\n",
       "2  genuine  Hacker releases cell phone numbers, personal e...          0\n",
       "3  genuine            Lionel Messi announces Argentina return          0\n",
       "4  genuine  Fighting the male biological clock by banking ...          0"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check that the conversion worked\n",
    "links.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>title</th>\n",
       "      <th>label_num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9495</th>\n",
       "      <td>clickbait</td>\n",
       "      <td>Upworthy Video</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9496</th>\n",
       "      <td>clickbait</td>\n",
       "      <td>The things some adopted kids are afraid to tal...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9497</th>\n",
       "      <td>clickbait</td>\n",
       "      <td>Playgrounds for senior citizens? Genius idea.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9498</th>\n",
       "      <td>clickbait</td>\n",
       "      <td>What happens when this 7-year-old elephant reu...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9499</th>\n",
       "      <td>clickbait</td>\n",
       "      <td>Upworthy Video</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          label                                              title  label_num\n",
       "9495  clickbait                                     Upworthy Video          1\n",
       "9496  clickbait  The things some adopted kids are afraid to tal...          1\n",
       "9497  clickbait      Playgrounds for senior citizens? Genius idea.          1\n",
       "9498  clickbait  What happens when this 7-year-old elephant reu...          1\n",
       "9499  clickbait                                     Upworthy Video          1"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links.tail(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150, 4)\n",
      "(150,)\n"
     ]
    }
   ],
   "source": [
    "# how to define X and y (from the iris data) for use with a MODEL\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(9500,)\n",
      "(9500,)\n"
     ]
    }
   ],
   "source": [
    "# how to define X and y (from the SMS data) for use with COUNTVECTORIZER\n",
    "X = links.title\n",
    "y = links.label_num\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(7125,)\n",
      "(2375,)\n",
      "(7125,)\n",
      "(2375,)\n"
     ]
    }
   ],
   "source": [
    "# split X and y into training and testing sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Vectorizing our dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# instantiate the vectorizer\n",
    "vect = CountVectorizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# learn training data vocabulary, then use it to create a document-term matrix\n",
    "vect.fit(X_train)\n",
    "X_train_dtm = vect.transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# equivalently: combine fit and transform into a single step\n",
    "X_train_dtm = vect.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<7125x9662 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 61386 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine the document-term matrix\n",
    "X_train_dtm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<2375x9662 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 18916 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# transform testing data (using fitted vocabulary) into a document-term matrix\n",
    "X_test_dtm = vect.transform(X_test)\n",
    "X_test_dtm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5: Building and evaluating a model\n",
    "\n",
    "We will use [multinomial Naive Bayes](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html):\n",
    "\n",
    "> The multinomial Naive Bayes classifier is suitable for classification with **discrete features** (e.g., word counts for text classification). The multinomial distribution normally requires integer feature counts. However, in practice, fractional counts such as tf-idf may also work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import and instantiate a Multinomial Naive Bayes model\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "nb = MultinomialNB()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4 ms, sys: 0 ns, total: 4 ms\n",
      "Wall time: 3.39 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train the model using X_train_dtm (timing it with an IPython \"magic command\")\n",
    "%time nb.fit(X_train_dtm, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# make class predictions for X_test_dtm\n",
    "y_pred_class = nb.predict(X_test_dtm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.89473684210526316"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate accuracy of class predictions\n",
    "from sklearn import metrics\n",
    "metrics.accuracy_score(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0] == genuine\n"
     ]
    }
   ],
   "source": [
    "new_title = vect.transform(['President leaves office in January'])\n",
    "# new_title = vect.transform(['20 Amazing Funny Gadgets you should try'])\n",
    "pd.DataFrame(new_title.toarray(), columns=vect.get_feature_names())\n",
    "labels = {0: 'genuine', 1: 'clickbait'}\n",
    "print('{} == {}'.format(nb.predict(new_title), labels[int(nb.predict(new_title))]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1492,  162],\n",
       "       [  88,  633]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print the confusion matrix\n",
    "metrics.confusion_matrix(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4935         Think Like a Doctor: A Cough That Won't Stop\n",
       "5364    Disney Princesses Do Change Girls -- and Boys,...\n",
       "941          The last VCR will be manufactured this month\n",
       "5955                     Lessons of Hiroshima and Orlando\n",
       "473     This wireless speaker is as loud as a rock con...\n",
       "229     Bride is walked down the aisle by man who rece...\n",
       "1475    Hughley: Parenting advice from Giuliani is lik...\n",
       "3657                     The Brain That Couldn't Remember\n",
       "6386    Almost As Good As A Full Night's Sleep: 25 Dis...\n",
       "911     Kristen Bell releases first ever photos from $...\n",
       "3583            Are Smoothies Better for You Than Juices?\n",
       "5816                  The Week in Pictures: June 17, 2016\n",
       "6646    Rappers Based Lyrics on Their Credit Card Frau...\n",
       "98      Facebook will now show you ads even if you use...\n",
       "337     10 strange sports you didn't know were in the ...\n",
       "5297    You're Going to Sell Your Home. Should You Men...\n",
       "2656    Comfort dogs' from around the U.S. are providi...\n",
       "5631    How Many Calories We Burn When We Sit, Stand o...\n",
       "5366    Sharing Ideals, Friendship and, After 37 Years...\n",
       "2712           Vet extracts perfect mold of dog's stomach\n",
       "5757    Silicon Valley' Season 3, Episode 9: What Exac...\n",
       "136                           Why was this pool so green?\n",
       "1572               Here's what playing Pokémon GO is like\n",
       "5216        How Much Do We Love TV? Let Us Count the Ways\n",
       "5680    What College Sports Recruiters Can Teach Your ...\n",
       "2441          Cute photo? Nope: preschool lockdown drill.\n",
       "2506               Daughters remember ESPN's Stuart Scott\n",
       "1495    \"Jennifer Aniston isn't pregnant -- and she's ...\n",
       "5367                       Can You Get Over an Addiction?\n",
       "5119     10 Free (or Cheap) Travel Apps Worth Downloading\n",
       "                              ...                        \n",
       "5425          How Old Is Too Old to Go to a Pediatrician?\n",
       "299                Distracted driving: Parents do it, too\n",
       "5009    Allison Janney Still Cringes About That Sex Scene\n",
       "6137    Message to Graduates: Times Are Tough, but You...\n",
       "796      How did this seal end up in the ladies bathroom?\n",
       "5755              Video From a Polar Bear's Point of View\n",
       "62      Fiji just won its first Olympic medal ever, an...\n",
       "6255                        Why French Workers Are So Mad\n",
       "3801                     When Friends Buy a Home Together\n",
       "2512                          Generate your own dad jokes\n",
       "6033    A London Subway Experiment: Please Don't Walk ...\n",
       "4215              Yes, We Are Still Writing About Pokémon\n",
       "5834                             Think Less, Think Better\n",
       "5665        Mom Hair: It Exists. Now What to Do About It.\n",
       "305                20 vintage planes you can still fly in\n",
       "4514                     We're Helping Deport Kids to Die\n",
       "5147    You Are in the Sea. Your Stuff Is on the Beach...\n",
       "4462       Winona Ryder, an Emblem of '90s Cool, Grows Up\n",
       "1837                      A healthy way to break the fast\n",
       "5053    Female Elephants Follow in Their Mothers' Foot...\n",
       "4190     The Incalculable Value of Finding a Job You Love\n",
       "634     Cockroach milk: The drink you didn't know you'...\n",
       "2701    Obama: My daughters 'think it's weird' we have...\n",
       "955                       Brie Larson is 'Captain Marvel'\n",
       "3859    Suit Revived for Veteran Lifeguard, 66, Who Re...\n",
       "5143               How I Learned to Tolerate Blake Lively\n",
       "219     For his birthday, boy gives shoes to kids in need\n",
       "3485                                    Dinner, Disrupted\n",
       "5267    Neil Gaiman Delves Deep Into Norse Myths for N...\n",
       "802     Cockroach milk: The drink you didn't know you'...\n",
       "Name: title, Length: 162, dtype: object"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print message text for the false positives (geniune incorrectly classified as clickbait)\n",
    "X_test[y_test < y_pred_class]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8460    Princess Charlotte Made Her First Public Appea...\n",
       "6788    State Lawmaker's Son Dies On World's Tallest W...\n",
       "7501    Turkey's Military Says It Has Taken Control Of...\n",
       "8017    Jesse Williams Gave A Powerful Speech About Ra...\n",
       "8649          Nice Guys Can Commit Domestic Violence, Too\n",
       "6867    Let's Take A Moment To Talk About The Men's Ol...\n",
       "8780    Kenya's unique approach to rape prevention sho...\n",
       "8218    Hundreds Of Professors Sign Letter Condeming Y...\n",
       "8826    Meet Jordan, whose love of black cats helps he...\n",
       "7294     26 Photos Of Prince George Bossing It As A Royal\n",
       "9180        A letter to my mother-in-law about my 3 boys.\n",
       "6777           Oh My God, Beach Volleyball Is Magnificent\n",
       "7298    Several Dead And Injured In A German Shopping ...\n",
       "9039    Another major country lifted its gay blood-don...\n",
       "7843    Stop Trying To Rescue Baby Animals, Wildlife O...\n",
       "7864    The Official Ranking Of All Of Jessica Simpson...\n",
       "8670                              7 Times Trump Was Right\n",
       "9160    Listen to 5 immigrants explain why they have A...\n",
       "9167    Meet the giant air-sucking wall that might hel...\n",
       "9479    Scientists in Antarctica photographed a new sp...\n",
       "9135    Meet the new Iron Man, a badass black female t...\n",
       "7820    Meet #MrStealYourGrandma, The Hottest Grandpa ...\n",
       "7585    Watch us throw a massive Pokémon Go party (wit...\n",
       "9313    Democrats are staging an incredible anti-gun p...\n",
       "7943    25 Photos That Show What Obama's Presidency Me...\n",
       "8194    Tom DeLonge Says He Left Blink-182 To Investig...\n",
       "9365    Meet the first trans woman to appear on the co...\n",
       "9031    His brother was at the Istanbul airport during...\n",
       "8171    Starbucks Customers Can Now Sue The Company Ov...\n",
       "6767    This College Just Installed The First Pizza AT...\n",
       "                              ...                        \n",
       "7667    Texas Inmates Broke Out Of A Cell To Save A Ja...\n",
       "9280    Watch beachgoers rush to save a stranded great...\n",
       "8822    This story of lost Syrian refugees in a Canadi...\n",
       "7396    Melania Trump Copied Part of Her Speech From M...\n",
       "8440    People Are Lining Up To Donate Blood For The O...\n",
       "7703    This Is Everything We Know So Far About The Vi...\n",
       "9336    For a rapidly deteriorating city, welcoming th...\n",
       "8631    These Mexican Cops Counseled A Man Contemplati...\n",
       "8516    People Freaked Out Over Hillary Clinton's Twee...\n",
       "8549    Maria Sharapova Suspended From Tennis For Two ...\n",
       "8969    There was more to Third Eye Blind's RNC-relate...\n",
       "8971    Bartenders in D.C. are learning how to stop se...\n",
       "9393    LIVE VIDEO: Democrats Hold Senate Floor In Pus...\n",
       "9477                               Building the New World\n",
       "9330    Teenager brilliantly shows how to deal with bu...\n",
       "9091    I Need Justice, I Need Peace: A QTPOC Roundtab...\n",
       "8777    What politicians going positive teaches us abo...\n",
       "9437        Miami-Dade Bans Styrofoam From Parks, Beaches\n",
       "9043    A group of NBA players opened the ESPYs with a...\n",
       "8911    5 household items to have on hand in case of a...\n",
       "6943    This Picture Shows The Huge Difference In Heig...\n",
       "8411    Adele Breaks Down In Tears As She Dedicates Co...\n",
       "8522    Joe Biden Writes An Open Letter To Stanford Su...\n",
       "7267    No, Bernie Sanders Doesn't Have A Photo Of Har...\n",
       "9069    A 2016 take on a century-old map shows changes...\n",
       "9319    The new Zika vaccine could be a big win for pr...\n",
       "8381    People Are Sharing An Orlando Shooting Victim'...\n",
       "7756    Hillary Clinton Proposes Tuition-Free College ...\n",
       "9037    Black GOP Senator Talks About Being Pulled Ove...\n",
       "7148    A Young Black Girl Is Now The Smartest Hero In...\n",
       "Name: title, Length: 88, dtype: object"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print message text for the false negatives (clickbait incorrectly classified as genuine)\n",
    "X_test[y_test > y_pred_class]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Scientists in Antarctica photographed a new species of crab. It's extraordinary he even exists.\""
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# example false negative\n",
    "X_test[9479]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  2.21282792e-03,   6.82442823e-02,   9.99995069e-01, ...,\n",
       "         9.89507319e-01,   1.53084126e-03,   1.42533725e-05])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate predicted probabilities for X_test_dtm (poorly calibrated)\n",
    "y_pred_prob = nb.predict_proba(X_test_dtm)[:, 1]\n",
    "y_pred_prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.94528374033780171"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate AUC\n",
    "metrics.roc_auc_score(y_test, y_pred_prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 6: Comparing models\n",
    "\n",
    "We will compare multinomial Naive Bayes with [logistic regression](http://scikit-learn.org/stable/modules/linear_model.html#logistic-regression):\n",
    "\n",
    "> Logistic regression, despite its name, is a **linear model for classification** rather than regression. Logistic regression is also known in the literature as logit regression, maximum-entropy classification (MaxEnt) or the log-linear classifier. In this model, the probabilities describing the possible outcomes of a single trial are modeled using a logistic function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import and instantiate a logistic regression model\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 72 ms, sys: 0 ns, total: 72 ms\n",
      "Wall time: 68.7 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train the model using X_train_dtm\n",
    "%time logreg.fit(X_train_dtm, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# make class predictions for X_test_dtm\n",
    "y_pred_class = logreg.predict(X_test_dtm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.00469539,  0.05418011,  0.98822526, ...,  0.74744416,\n",
       "        0.02012881,  0.00312435])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate predicted probabilities for X_test_dtm (well calibrated)\n",
    "y_pred_prob = logreg.predict_proba(X_test_dtm)[:, 1]\n",
    "y_pred_prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.92042105263157892"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate accuracy\n",
    "metrics.accuracy_score(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.95880788304568254"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate AUC\n",
    "metrics.roc_auc_score(y_test, y_pred_prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 7: Examining a model for further insight\n",
    "\n",
    "We will examine the our **trained Naive Bayes model** to calculate the approximate **\"spamminess\" of each token**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9662"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# store the vocabulary of X_train\n",
    "X_train_tokens = vect.get_feature_names()\n",
    "len(X_train_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['000', '00s', '01pm', '06pm', '09pm', '10', '100', '101', '102', '10m', '10th', '11', '110', '112', '116', '12', '120', '125', '126', '13', '130', '130m', '13b', '13pm', '13th', '14', '140', '142', '15', '150', '15k', '16', '160', '161', '166', '17', '174', '17th', '18', '187', '18pm', '19', '1900s', '196', '1960s', '1964', '1970s', '1982', '1993', '1994']\n"
     ]
    }
   ],
   "source": [
    "# examine the first 50 tokens\n",
    "print(X_train_tokens[0:50])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['yiannopoulos', 'yield', 'yo', 'yoga', 'yogaday', 'yogurt', 'yoko', 'york', 'yorker', 'you', 'young', 'younger', 'youngest', 'your', 'youree', 'yours', 'yourself', 'yourselves', 'youth', 'youths', 'youtube', 'yulia', 'yuliya', 'yup', 'yves', 'zabar', 'zac', 'zack', 'zara', 'zealand', 'zen', 'zeppelin', 'zero', 'zika', 'zimbabwe', 'zimbabwean', 'zimmer', 'zip', 'zodiac', 'zombie', 'zombies', 'zone', 'zones', 'zoo', 'zookeeper', 'zoos', 'zootopia', 'zubabox', 'zucchini', 'zuckerberg']\n"
     ]
    }
   ],
   "source": [
    "# examine the last 50 tokens\n",
    "print(X_train_tokens[-50:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 27.,   0.,   1., ...,   0.,   3.,   4.],\n",
       "       [  3.,  11.,   0., ...,   1.,   0.,   0.]])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Naive Bayes counts the number of times each token appears in each class\n",
    "nb.feature_count_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 9662)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rows represent classes, columns represent tokens\n",
    "nb.feature_count_.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 27.,   0.,   1., ...,   0.,   3.,   4.])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of times each token appears across all HAM messages\n",
    "genuine_token_count = nb.feature_count_[0, :]\n",
    "genuine_token_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  3.,  11.,   0., ...,   1.,   0.,   0.])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of times each token appears across all SPAM messages\n",
    "clickbait_token_count = nb.feature_count_[1, :]\n",
    "clickbait_token_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clickbait</th>\n",
       "      <th>genuine</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>token</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>000</th>\n",
       "      <td>3.0</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>00s</th>\n",
       "      <td>11.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>01pm</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>06pm</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>09pm</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       clickbait  genuine\n",
       "token                    \n",
       "000          3.0     27.0\n",
       "00s         11.0      0.0\n",
       "01pm         0.0      1.0\n",
       "06pm         0.0      2.0\n",
       "09pm         0.0      1.0"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create a DataFrame of tokens with their separate ham and spam counts\n",
    "tokens = pd.DataFrame({'token':X_train_tokens, 'genuine':genuine_token_count, 'clickbait':clickbait_token_count}).set_index('token')\n",
    "tokens.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clickbait</th>\n",
       "      <th>genuine</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>token</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>torrent</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>retailers</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>85.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subjects</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>market</th>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           clickbait  genuine\n",
       "token                        \n",
       "torrent          0.0      1.0\n",
       "retailers        0.0      1.0\n",
       "19              85.0      5.0\n",
       "subjects         0.0      2.0\n",
       "market           0.0     11.0"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine 5 random DataFrame rows\n",
    "tokens.sample(5, random_state=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 5003.,  2122.])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Naive Bayes counts the number of observations in each class\n",
    "nb.class_count_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can calculate the \"clickbaitness\" of each token, we need to avoid **dividing by zero** and account for the **class imbalance**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clickbait</th>\n",
       "      <th>genuine</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>token</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>torrent</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>retailers</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>86.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subjects</th>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>market</th>\n",
       "      <td>1.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           clickbait  genuine\n",
       "token                        \n",
       "torrent          1.0      2.0\n",
       "retailers        1.0      2.0\n",
       "19              86.0      6.0\n",
       "subjects         1.0      3.0\n",
       "market           1.0     12.0"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add 1 to ham and spam counts to avoid dividing by 0\n",
    "tokens['genuine'] = tokens.genuine + 1\n",
    "tokens['clickbait'] = tokens.clickbait + 1\n",
    "tokens.sample(5, random_state=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clickbait</th>\n",
       "      <th>genuine</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>token</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>torrent</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.000400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>retailers</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.000400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.040528</td>\n",
       "      <td>0.001199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subjects</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.000600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>market</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.002399</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           clickbait   genuine\n",
       "token                         \n",
       "torrent     0.000471  0.000400\n",
       "retailers   0.000471  0.000400\n",
       "19          0.040528  0.001199\n",
       "subjects    0.000471  0.000600\n",
       "market      0.000471  0.002399"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert the ham and spam counts into frequencies\n",
    "tokens['genuine'] = tokens.genuine / nb.class_count_[0]\n",
    "tokens['clickbait'] = tokens.clickbait / nb.class_count_[1]\n",
    "tokens.sample(5, random_state=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clickbait</th>\n",
       "      <th>genuine</th>\n",
       "      <th>clickbait_ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>token</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>torrent</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.000400</td>\n",
       "      <td>1.178841</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>retailers</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.000400</td>\n",
       "      <td>1.178841</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.040528</td>\n",
       "      <td>0.001199</td>\n",
       "      <td>33.793434</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>subjects</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.000600</td>\n",
       "      <td>0.785894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>market</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.002399</td>\n",
       "      <td>0.196473</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           clickbait   genuine  clickbait_ratio\n",
       "token                                          \n",
       "torrent     0.000471  0.000400         1.178841\n",
       "retailers   0.000471  0.000400         1.178841\n",
       "19          0.040528  0.001199        33.793434\n",
       "subjects    0.000471  0.000600         0.785894\n",
       "market      0.000471  0.002399         0.196473"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate the ratio of spam-to-ham for each token\n",
    "tokens['clickbait_ratio'] = tokens.clickbait / tokens.genuine\n",
    "tokens.sample(5, random_state=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clickbait</th>\n",
       "      <th>genuine</th>\n",
       "      <th>clickbait_ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>token</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.045712</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>228.695099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>upworthy</th>\n",
       "      <td>0.039114</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>195.687559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.028275</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>141.460886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>buzzfeed</th>\n",
       "      <td>0.026861</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>134.387842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>obsessed</th>\n",
       "      <td>0.014609</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>73.088124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>laugh</th>\n",
       "      <td>0.013195</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>66.015080</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>funny</th>\n",
       "      <td>0.010368</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>51.868992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.028746</td>\n",
       "      <td>0.000600</td>\n",
       "      <td>47.939522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>af</th>\n",
       "      <td>0.009425</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>47.153629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hacks</th>\n",
       "      <td>0.008954</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>44.795947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hilarious</th>\n",
       "      <td>0.017908</td>\n",
       "      <td>0.000400</td>\n",
       "      <td>44.795947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>shit</th>\n",
       "      <td>0.008483</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>42.438266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fucking</th>\n",
       "      <td>0.008483</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>42.438266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cocoa</th>\n",
       "      <td>0.008011</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>40.080584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.015551</td>\n",
       "      <td>0.000400</td>\n",
       "      <td>38.901744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tumblr</th>\n",
       "      <td>0.007540</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>37.722903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>amazing</th>\n",
       "      <td>0.007540</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>37.722903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>products</th>\n",
       "      <td>0.014609</td>\n",
       "      <td>0.000400</td>\n",
       "      <td>36.544062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>makeup</th>\n",
       "      <td>0.014138</td>\n",
       "      <td>0.000400</td>\n",
       "      <td>35.365221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.040528</td>\n",
       "      <td>0.001199</td>\n",
       "      <td>33.793434</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perfectly</th>\n",
       "      <td>0.006126</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>30.649859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>songs</th>\n",
       "      <td>0.006126</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>30.649859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>moms</th>\n",
       "      <td>0.005655</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>28.292177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tips</th>\n",
       "      <td>0.005655</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>28.292177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>damn</th>\n",
       "      <td>0.005655</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>28.292177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>00s</th>\n",
       "      <td>0.005655</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>28.292177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tweets</th>\n",
       "      <td>0.032045</td>\n",
       "      <td>0.001199</td>\n",
       "      <td>26.720390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>genius</th>\n",
       "      <td>0.005184</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>25.934496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gorgeous</th>\n",
       "      <td>0.005184</td>\n",
       "      <td>0.000200</td>\n",
       "      <td>25.934496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.020735</td>\n",
       "      <td>0.000800</td>\n",
       "      <td>25.934496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cleveland</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.003998</td>\n",
       "      <td>0.117884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>europe</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.004197</td>\n",
       "      <td>0.112271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trump</th>\n",
       "      <td>0.012724</td>\n",
       "      <td>0.116730</td>\n",
       "      <td>0.109002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>claims</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.004397</td>\n",
       "      <td>0.107167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>leader</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.004397</td>\n",
       "      <td>0.107167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>zika</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.004397</td>\n",
       "      <td>0.107167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hillary</th>\n",
       "      <td>0.004241</td>\n",
       "      <td>0.040176</td>\n",
       "      <td>0.105568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>poll</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.004597</td>\n",
       "      <td>0.102508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>wins</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.004797</td>\n",
       "      <td>0.098237</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>attacks</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.004997</td>\n",
       "      <td>0.094307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>town</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.004997</td>\n",
       "      <td>0.094307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>protesters</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.004997</td>\n",
       "      <td>0.094307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>euro</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.004997</td>\n",
       "      <td>0.094307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>campaign</th>\n",
       "      <td>0.000943</td>\n",
       "      <td>0.010194</td>\n",
       "      <td>0.092458</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>politics</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.005197</td>\n",
       "      <td>0.090680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>clinton</th>\n",
       "      <td>0.005655</td>\n",
       "      <td>0.063562</td>\n",
       "      <td>0.088969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>international</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.005597</td>\n",
       "      <td>0.084203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recipe</th>\n",
       "      <td>0.000943</td>\n",
       "      <td>0.011193</td>\n",
       "      <td>0.084203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dies</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.005797</td>\n",
       "      <td>0.081299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>war</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.006196</td>\n",
       "      <td>0.076054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>coup</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.006196</td>\n",
       "      <td>0.076054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>democratic</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.006396</td>\n",
       "      <td>0.073678</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>york</th>\n",
       "      <td>0.001885</td>\n",
       "      <td>0.026984</td>\n",
       "      <td>0.069857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>convention</th>\n",
       "      <td>0.001414</td>\n",
       "      <td>0.020788</td>\n",
       "      <td>0.068010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>democrats</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.006996</td>\n",
       "      <td>0.067362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>briefing</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.007995</td>\n",
       "      <td>0.058942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>terror</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.008195</td>\n",
       "      <td>0.057504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cnn</th>\n",
       "      <td>0.000943</td>\n",
       "      <td>0.016990</td>\n",
       "      <td>0.055475</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>turkey</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.009794</td>\n",
       "      <td>0.048116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>isis</th>\n",
       "      <td>0.000471</td>\n",
       "      <td>0.011993</td>\n",
       "      <td>0.039295</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9662 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               clickbait   genuine  clickbait_ratio\n",
       "token                                              \n",
       "21              0.045712  0.000200       228.695099\n",
       "upworthy        0.039114  0.000200       195.687559\n",
       "23              0.028275  0.000200       141.460886\n",
       "buzzfeed        0.026861  0.000200       134.387842\n",
       "obsessed        0.014609  0.000200        73.088124\n",
       "laugh           0.013195  0.000200        66.015080\n",
       "funny           0.010368  0.000200        51.868992\n",
       "17              0.028746  0.000600        47.939522\n",
       "af              0.009425  0.000200        47.153629\n",
       "hacks           0.008954  0.000200        44.795947\n",
       "hilarious       0.017908  0.000400        44.795947\n",
       "shit            0.008483  0.000200        42.438266\n",
       "fucking         0.008483  0.000200        42.438266\n",
       "cocoa           0.008011  0.000200        40.080584\n",
       "27              0.015551  0.000400        38.901744\n",
       "tumblr          0.007540  0.000200        37.722903\n",
       "amazing         0.007540  0.000200        37.722903\n",
       "products        0.014609  0.000400        36.544062\n",
       "makeup          0.014138  0.000400        35.365221\n",
       "19              0.040528  0.001199        33.793434\n",
       "perfectly       0.006126  0.000200        30.649859\n",
       "songs           0.006126  0.000200        30.649859\n",
       "moms            0.005655  0.000200        28.292177\n",
       "tips            0.005655  0.000200        28.292177\n",
       "damn            0.005655  0.000200        28.292177\n",
       "00s             0.005655  0.000200        28.292177\n",
       "tweets          0.032045  0.001199        26.720390\n",
       "genius          0.005184  0.000200        25.934496\n",
       "gorgeous        0.005184  0.000200        25.934496\n",
       "18              0.020735  0.000800        25.934496\n",
       "...                  ...       ...              ...\n",
       "cleveland       0.000471  0.003998         0.117884\n",
       "europe          0.000471  0.004197         0.112271\n",
       "trump           0.012724  0.116730         0.109002\n",
       "claims          0.000471  0.004397         0.107167\n",
       "leader          0.000471  0.004397         0.107167\n",
       "zika            0.000471  0.004397         0.107167\n",
       "hillary         0.004241  0.040176         0.105568\n",
       "poll            0.000471  0.004597         0.102508\n",
       "wins            0.000471  0.004797         0.098237\n",
       "attacks         0.000471  0.004997         0.094307\n",
       "town            0.000471  0.004997         0.094307\n",
       "protesters      0.000471  0.004997         0.094307\n",
       "euro            0.000471  0.004997         0.094307\n",
       "campaign        0.000943  0.010194         0.092458\n",
       "politics        0.000471  0.005197         0.090680\n",
       "clinton         0.005655  0.063562         0.088969\n",
       "international   0.000471  0.005597         0.084203\n",
       "recipe          0.000943  0.011193         0.084203\n",
       "dies            0.000471  0.005797         0.081299\n",
       "war             0.000471  0.006196         0.076054\n",
       "coup            0.000471  0.006196         0.076054\n",
       "democratic      0.000471  0.006396         0.073678\n",
       "york            0.001885  0.026984         0.069857\n",
       "convention      0.001414  0.020788         0.068010\n",
       "democrats       0.000471  0.006996         0.067362\n",
       "briefing        0.000471  0.007995         0.058942\n",
       "terror          0.000471  0.008195         0.057504\n",
       "cnn             0.000943  0.016990         0.055475\n",
       "turkey          0.000471  0.009794         0.048116\n",
       "isis            0.000471  0.011993         0.039295\n",
       "\n",
       "[9662 rows x 3 columns]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# examine the DataFrame sorted by spam_ratio\n",
    "# note: use sort() instead of sort_values() for pandas 0.16.2 and earlier\n",
    "tokens.sort_values('clickbait_ratio', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "37.722902921771912"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look up the spam_ratio for a given token\n",
    "tokens.loc['amazing', 'clickbait_ratio']"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}