Book Recommendation System.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**About Book Crossing Dataset**<br>\n",
    "\n",
    "This dataset has been compiled by Cai-Nicolas Ziegler in 2004, and it comprises of three tables for users, books and ratings. Explicit ratings are expressed on a scale from 1-10 (higher values denoting higher appreciation) and implicit rating is expressed by 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Link to dataset files<br>\n",
    "http://www2.informatik.uni-freiburg.de/~cziegler/BX/ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**About this Project**\n",
    "\n",
    "This project entails building a Book Recommender System for users based on user-based and item-based collaborative filtering approaches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Making necesarry imports\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn.metrics as metrics\n",
    "import numpy as np\n",
    "from sklearn.neighbors import NearestNeighbors\n",
    "from scipy.spatial.distance import correlation\n",
    "from sklearn.metrics.pairwise import pairwise_distances\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import display, clear_output\n",
    "from contextlib import contextmanager\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "import numpy as np\n",
    "import os, sys\n",
    "import re\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Setting the current working directory\n",
    "os.chdir('D:\\Data Science\\Projects\\Book Crossing Dataset - Recommender System')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "code_show_err=false; \n",
       "function code_toggle_err() {\n",
       " if (code_show_err){\n",
       " $('div.output_stderr').hide();\n",
       " } else {\n",
       " $('div.output_stderr').show();\n",
       " }\n",
       " code_show_err = !code_show_err\n",
       "} \n",
       "$( document ).ready(code_toggle_err);\n",
       "</script>\n",
       "To toggle on/off output_stderr, click <a href=\"javascript:code_toggle_err()\">here</a>."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('''<script>\n",
    "code_show_err=false; \n",
    "function code_toggle_err() {\n",
    " if (code_show_err){\n",
    " $('div.output_stderr').hide();\n",
    " } else {\n",
    " $('div.output_stderr').show();\n",
    " }\n",
    " code_show_err = !code_show_err\n",
    "} \n",
    "$( document ).ready(code_toggle_err);\n",
    "</script>\n",
    "To toggle on/off output_stderr, click <a href=\"javascript:code_toggle_err()\">here</a>.''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Skipping line 6452: expected 8 fields, saw 9\n",
      "Skipping line 43667: expected 8 fields, saw 10\n",
      "Skipping line 51751: expected 8 fields, saw 9\n",
      "\n",
      "Skipping line 92038: expected 8 fields, saw 9\n",
      "Skipping line 104319: expected 8 fields, saw 9\n",
      "Skipping line 121768: expected 8 fields, saw 9\n",
      "\n",
      "Skipping line 144058: expected 8 fields, saw 9\n",
      "Skipping line 150789: expected 8 fields, saw 9\n",
      "Skipping line 157128: expected 8 fields, saw 9\n",
      "Skipping line 180189: expected 8 fields, saw 9\n",
      "Skipping line 185738: expected 8 fields, saw 9\n",
      "\n",
      "Skipping line 209388: expected 8 fields, saw 9\n",
      "Skipping line 220626: expected 8 fields, saw 9\n",
      "Skipping line 227933: expected 8 fields, saw 11\n",
      "Skipping line 228957: expected 8 fields, saw 10\n",
      "Skipping line 245933: expected 8 fields, saw 9\n",
      "Skipping line 251296: expected 8 fields, saw 9\n",
      "Skipping line 259941: expected 8 fields, saw 9\n",
      "Skipping line 261529: expected 8 fields, saw 9\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Loading data\n",
    "books = pd.read_csv('books.csv', sep=';', error_bad_lines=False, encoding=\"latin-1\")\n",
    "books.columns = ['ISBN', 'bookTitle', 'bookAuthor', 'yearOfPublication', 'publisher', 'imageUrlS', 'imageUrlM', 'imageUrlL']\n",
    "users = pd.read_csv('users.csv', sep=';', error_bad_lines=False, encoding=\"latin-1\")\n",
    "users.columns = ['userID', 'Location', 'Age']\n",
    "ratings = pd.read_csv('ratings.csv', sep=';', error_bad_lines=False, encoding=\"latin-1\")\n",
    "ratings.columns = ['userID', 'ISBN', 'bookRating']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(271360, 8)\n",
      "(278858, 3)\n",
      "(1149780, 3)\n"
     ]
    }
   ],
   "source": [
    "#checking shapes of the datasets\n",
    "print books.shape\n",
    "print users.shape\n",
    "print ratings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "      <th>imageUrlS</th>\n",
       "      <th>imageUrlM</th>\n",
       "      <th>imageUrlL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0195153448</td>\n",
       "      <td>Classical Mythology</td>\n",
       "      <td>Mark P. O. Morford</td>\n",
       "      <td>2002</td>\n",
       "      <td>Oxford University Press</td>\n",
       "      <td>http://images.amazon.com/images/P/0195153448.01.THUMBZZZ.jpg</td>\n",
       "      <td>http://images.amazon.com/images/P/0195153448.01.MZZZZZZZ.jpg</td>\n",
       "      <td>http://images.amazon.com/images/P/0195153448.01.LZZZZZZZ.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0002005018</td>\n",
       "      <td>Clara Callan</td>\n",
       "      <td>Richard Bruce Wright</td>\n",
       "      <td>2001</td>\n",
       "      <td>HarperFlamingo Canada</td>\n",
       "      <td>http://images.amazon.com/images/P/0002005018.01.THUMBZZZ.jpg</td>\n",
       "      <td>http://images.amazon.com/images/P/0002005018.01.MZZZZZZZ.jpg</td>\n",
       "      <td>http://images.amazon.com/images/P/0002005018.01.LZZZZZZZ.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0060973129</td>\n",
       "      <td>Decision in Normandy</td>\n",
       "      <td>Carlo D'Este</td>\n",
       "      <td>1991</td>\n",
       "      <td>HarperPerennial</td>\n",
       "      <td>http://images.amazon.com/images/P/0060973129.01.THUMBZZZ.jpg</td>\n",
       "      <td>http://images.amazon.com/images/P/0060973129.01.MZZZZZZZ.jpg</td>\n",
       "      <td>http://images.amazon.com/images/P/0060973129.01.LZZZZZZZ.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0374157065</td>\n",
       "      <td>Flu: The Story of the Great Influenza Pandemic of 1918 and the Search for the Virus That Caused It</td>\n",
       "      <td>Gina Bari Kolata</td>\n",
       "      <td>1999</td>\n",
       "      <td>Farrar Straus Giroux</td>\n",
       "      <td>http://images.amazon.com/images/P/0374157065.01.THUMBZZZ.jpg</td>\n",
       "      <td>http://images.amazon.com/images/P/0374157065.01.MZZZZZZZ.jpg</td>\n",
       "      <td>http://images.amazon.com/images/P/0374157065.01.LZZZZZZZ.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0393045218</td>\n",
       "      <td>The Mummies of Urumchi</td>\n",
       "      <td>E. J. W. Barber</td>\n",
       "      <td>1999</td>\n",
       "      <td>W. W. Norton &amp;amp; Company</td>\n",
       "      <td>http://images.amazon.com/images/P/0393045218.01.THUMBZZZ.jpg</td>\n",
       "      <td>http://images.amazon.com/images/P/0393045218.01.MZZZZZZZ.jpg</td>\n",
       "      <td>http://images.amazon.com/images/P/0393045218.01.LZZZZZZZ.jpg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         ISBN  \\\n",
       "0  0195153448   \n",
       "1  0002005018   \n",
       "2  0060973129   \n",
       "3  0374157065   \n",
       "4  0393045218   \n",
       "\n",
       "                                                                                            bookTitle  \\\n",
       "0  Classical Mythology                                                                                  \n",
       "1  Clara Callan                                                                                         \n",
       "2  Decision in Normandy                                                                                 \n",
       "3  Flu: The Story of the Great Influenza Pandemic of 1918 and the Search for the Virus That Caused It   \n",
       "4  The Mummies of Urumchi                                                                               \n",
       "\n",
       "             bookAuthor yearOfPublication                   publisher  \\\n",
       "0  Mark P. O. Morford    2002              Oxford University Press      \n",
       "1  Richard Bruce Wright  2001              HarperFlamingo Canada        \n",
       "2  Carlo D'Este          1991              HarperPerennial              \n",
       "3  Gina Bari Kolata      1999              Farrar Straus Giroux         \n",
       "4  E. J. W. Barber       1999              W. W. Norton &amp; Company   \n",
       "\n",
       "                                                      imageUrlS  \\\n",
       "0  http://images.amazon.com/images/P/0195153448.01.THUMBZZZ.jpg   \n",
       "1  http://images.amazon.com/images/P/0002005018.01.THUMBZZZ.jpg   \n",
       "2  http://images.amazon.com/images/P/0060973129.01.THUMBZZZ.jpg   \n",
       "3  http://images.amazon.com/images/P/0374157065.01.THUMBZZZ.jpg   \n",
       "4  http://images.amazon.com/images/P/0393045218.01.THUMBZZZ.jpg   \n",
       "\n",
       "                                                      imageUrlM  \\\n",
       "0  http://images.amazon.com/images/P/0195153448.01.MZZZZZZZ.jpg   \n",
       "1  http://images.amazon.com/images/P/0002005018.01.MZZZZZZZ.jpg   \n",
       "2  http://images.amazon.com/images/P/0060973129.01.MZZZZZZZ.jpg   \n",
       "3  http://images.amazon.com/images/P/0374157065.01.MZZZZZZZ.jpg   \n",
       "4  http://images.amazon.com/images/P/0393045218.01.MZZZZZZZ.jpg   \n",
       "\n",
       "                                                      imageUrlL  \n",
       "0  http://images.amazon.com/images/P/0195153448.01.LZZZZZZZ.jpg  \n",
       "1  http://images.amazon.com/images/P/0002005018.01.LZZZZZZZ.jpg  \n",
       "2  http://images.amazon.com/images/P/0060973129.01.LZZZZZZZ.jpg  \n",
       "3  http://images.amazon.com/images/P/0374157065.01.LZZZZZZZ.jpg  \n",
       "4  http://images.amazon.com/images/P/0393045218.01.LZZZZZZZ.jpg  "
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Exploring books dataset\n",
    "books.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#dropping last three columns containing image URLs which will not be required for analysis\n",
    "books.drop(['imageUrlS', 'imageUrlM', 'imageUrlL'],axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0195153448</td>\n",
       "      <td>Classical Mythology</td>\n",
       "      <td>Mark P. O. Morford</td>\n",
       "      <td>2002</td>\n",
       "      <td>Oxford University Press</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0002005018</td>\n",
       "      <td>Clara Callan</td>\n",
       "      <td>Richard Bruce Wright</td>\n",
       "      <td>2001</td>\n",
       "      <td>HarperFlamingo Canada</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0060973129</td>\n",
       "      <td>Decision in Normandy</td>\n",
       "      <td>Carlo D'Este</td>\n",
       "      <td>1991</td>\n",
       "      <td>HarperPerennial</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0374157065</td>\n",
       "      <td>Flu: The Story of the Great Influenza Pandemic of 1918 and the Search for the Virus That Caused It</td>\n",
       "      <td>Gina Bari Kolata</td>\n",
       "      <td>1999</td>\n",
       "      <td>Farrar Straus Giroux</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0393045218</td>\n",
       "      <td>The Mummies of Urumchi</td>\n",
       "      <td>E. J. W. Barber</td>\n",
       "      <td>1999</td>\n",
       "      <td>W. W. Norton &amp;amp; Company</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         ISBN  \\\n",
       "0  0195153448   \n",
       "1  0002005018   \n",
       "2  0060973129   \n",
       "3  0374157065   \n",
       "4  0393045218   \n",
       "\n",
       "                                                                                            bookTitle  \\\n",
       "0  Classical Mythology                                                                                  \n",
       "1  Clara Callan                                                                                         \n",
       "2  Decision in Normandy                                                                                 \n",
       "3  Flu: The Story of the Great Influenza Pandemic of 1918 and the Search for the Virus That Caused It   \n",
       "4  The Mummies of Urumchi                                                                               \n",
       "\n",
       "             bookAuthor yearOfPublication                   publisher  \n",
       "0  Mark P. O. Morford    2002              Oxford University Press     \n",
       "1  Richard Bruce Wright  2001              HarperFlamingo Canada       \n",
       "2  Carlo D'Este          1991              HarperPerennial             \n",
       "3  Gina Bari Kolata      1999              Farrar Straus Giroux        \n",
       "4  E. J. W. Barber       1999              W. W. Norton &amp; Company  "
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Now the books datasets looks like....\n",
    "books.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ISBN                 object\n",
       "bookTitle            object\n",
       "bookAuthor           object\n",
       "yearOfPublication    object\n",
       "publisher            object\n",
       "dtype: object"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking data types of columns\n",
    "books.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#making this setting to display full text in columns\n",
    "pd.set_option('display.max_colwidth', -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**yearOfPublication**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2002L, 2001L, 1991L, 1999L, 2000L, 1993L, 1996L, 1988L, 2004L,\n",
       "       1998L, 1994L, 2003L, 1997L, 1983L, 1979L, 1995L, 1982L, 1985L,\n",
       "       1992L, 1986L, 1978L, 1980L, 1952L, 1987L, 1990L, 1981L, 1989L,\n",
       "       1984L, 0L, 1968L, 1961L, 1958L, 1974L, 1976L, 1971L, 1977L, 1975L,\n",
       "       1965L, 1941L, 1970L, 1962L, 1973L, 1972L, 1960L, 1966L, 1920L,\n",
       "       1956L, 1959L, 1953L, 1951L, 1942L, 1963L, 1964L, 1969L, 1954L,\n",
       "       1950L, 1967L, 2005L, 1957L, 1940L, 1937L, 1955L, 1946L, 1936L,\n",
       "       1930L, 2011L, 1925L, 1948L, 1943L, 1947L, 1945L, 1923L, 2020L,\n",
       "       1939L, 1926L, 1938L, 2030L, 1911L, 1904L, 1949L, 1932L, 1928L,\n",
       "       1929L, 1927L, 1931L, 1914L, 2050L, 1934L, 1910L, 1933L, 1902L,\n",
       "       1924L, 1921L, 1900L, 2038L, 2026L, 1944L, 1917L, 1901L, 2010L,\n",
       "       1908L, 1906L, 1935L, 1806L, 2021L, u'2000', u'1995', u'1999',\n",
       "       u'2004', u'2003', u'1990', u'1994', u'1986', u'1989', u'2002',\n",
       "       u'1981', u'1993', u'1983', u'1982', u'1976', u'1991', u'1977',\n",
       "       u'1998', u'1992', u'1996', u'0', u'1997', u'2001', u'1974', u'1968',\n",
       "       u'1987', u'1984', u'1988', u'1963', u'1956', u'1970', u'1985',\n",
       "       u'1978', u'1973', u'1980', u'1979', u'1975', u'1969', u'1961',\n",
       "       u'1965', u'1939', u'1958', u'1950', u'1953', u'1966', u'1971',\n",
       "       u'1959', u'1972', u'1955', u'1957', u'1945', u'1960', u'1967',\n",
       "       u'1932', u'1924', u'1964', u'2012', u'1911', u'1927', u'1948',\n",
       "       u'1962', u'2006', u'1952', u'1940', u'1951', u'1931', u'1954',\n",
       "       u'2005', u'1930', u'1941', u'1944', u'DK Publishing Inc', u'1943',\n",
       "       u'1938', u'1900', u'1942', u'1923', u'1920', u'1933', u'Gallimard',\n",
       "       u'1909', u'1946', u'2008', u'1378', u'2030', u'1936', u'1947',\n",
       "       u'2011', u'2020', u'1919', u'1949', u'1922', u'1897', u'2024',\n",
       "       u'1376', u'1926', u'2037'], dtype=object)"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#yearOfPublication should be set as having dtype as int\n",
    "#checking the unique values of yearOfPublication\n",
    "books.yearOfPublication.unique()\n",
    "\n",
    "#as it can be seen from below that there are some incorrect entries in this field. It looks like Publisher names \n",
    "#'DK Publishing Inc' and 'Gallimard' have been incorrectly loaded as yearOfPublication in dataset due to some errors in csv file\n",
    "#Also some of the entries are strings and same years have been entered as numbers in some places"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>209538</th>\n",
       "      <td>078946697X</td>\n",
       "      <td>DK Readers: Creating the X-Men, How It All Began (Level 4: Proficient Readers)\\\";Michael Teitelbaum\"</td>\n",
       "      <td>2000</td>\n",
       "      <td>DK Publishing Inc</td>\n",
       "      <td>http://images.amazon.com/images/P/078946697X.01.THUMBZZZ.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>221678</th>\n",
       "      <td>0789466953</td>\n",
       "      <td>DK Readers: Creating the X-Men, How Comic Books Come to Life (Level 4: Proficient Readers)\\\";James Buckley\"</td>\n",
       "      <td>2000</td>\n",
       "      <td>DK Publishing Inc</td>\n",
       "      <td>http://images.amazon.com/images/P/0789466953.01.THUMBZZZ.jpg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ISBN  \\\n",
       "209538  078946697X   \n",
       "221678  0789466953   \n",
       "\n",
       "                                                                                                          bookTitle  \\\n",
       "209538  DK Readers: Creating the X-Men, How It All Began (Level 4: Proficient Readers)\\\";Michael Teitelbaum\"          \n",
       "221678  DK Readers: Creating the X-Men, How Comic Books Come to Life (Level 4: Proficient Readers)\\\";James Buckley\"   \n",
       "\n",
       "       bookAuthor  yearOfPublication  \\\n",
       "209538  2000       DK Publishing Inc   \n",
       "221678  2000       DK Publishing Inc   \n",
       "\n",
       "                                                           publisher  \n",
       "209538  http://images.amazon.com/images/P/078946697X.01.THUMBZZZ.jpg  \n",
       "221678  http://images.amazon.com/images/P/0789466953.01.THUMBZZZ.jpg  "
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#investigating the rows having 'DK Publishing Inc' as yearOfPublication\n",
    "books.loc[books.yearOfPublication == 'DK Publishing Inc',:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#From above, it is seen that bookAuthor is incorrectly loaded with bookTitle, hence making required corrections\n",
    "#ISBN '0789466953'\n",
    "books.loc[books.ISBN == '0789466953','yearOfPublication'] = 2000\n",
    "books.loc[books.ISBN == '0789466953','bookAuthor'] = \"James Buckley\"\n",
    "books.loc[books.ISBN == '0789466953','publisher'] = \"DK Publishing Inc\"\n",
    "books.loc[books.ISBN == '0789466953','bookTitle'] = \"DK Readers: Creating the X-Men, How Comic Books Come to Life (Level 4: Proficient Readers)\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#ISBN '078946697X'\n",
    "books.loc[books.ISBN == '078946697X','yearOfPublication'] = 2000\n",
    "books.loc[books.ISBN == '078946697X','bookAuthor'] = \"Michael Teitelbaum\"\n",
    "books.loc[books.ISBN == '078946697X','publisher'] = \"DK Publishing Inc\"\n",
    "books.loc[books.ISBN == '078946697X','bookTitle'] = \"DK Readers: Creating the X-Men, How It All Began (Level 4: Proficient Readers)\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>209538</th>\n",
       "      <td>078946697X</td>\n",
       "      <td>DK Readers: Creating the X-Men, How It All Began (Level 4: Proficient Readers)</td>\n",
       "      <td>Michael Teitelbaum</td>\n",
       "      <td>2000</td>\n",
       "      <td>DK Publishing Inc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>221678</th>\n",
       "      <td>0789466953</td>\n",
       "      <td>DK Readers: Creating the X-Men, How Comic Books Come to Life (Level 4: Proficient Readers)</td>\n",
       "      <td>James Buckley</td>\n",
       "      <td>2000</td>\n",
       "      <td>DK Publishing Inc</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ISBN  \\\n",
       "209538  078946697X   \n",
       "221678  0789466953   \n",
       "\n",
       "                                                                                         bookTitle  \\\n",
       "209538  DK Readers: Creating the X-Men, How It All Began (Level 4: Proficient Readers)               \n",
       "221678  DK Readers: Creating the X-Men, How Comic Books Come to Life (Level 4: Proficient Readers)   \n",
       "\n",
       "                bookAuthor yearOfPublication          publisher  \n",
       "209538  Michael Teitelbaum  2000              DK Publishing Inc  \n",
       "221678  James Buckley       2000              DK Publishing Inc  "
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#rechecking\n",
    "books.loc[(books.ISBN == '0789466953') | (books.ISBN == '078946697X'),:]\n",
    "#corrections done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>220731</th>\n",
       "      <td>2070426769</td>\n",
       "      <td>Peuple du ciel, suivi de 'Les Bergers\\\";Jean-Marie Gustave Le ClÃ?Â©zio\"</td>\n",
       "      <td>2003</td>\n",
       "      <td>Gallimard</td>\n",
       "      <td>http://images.amazon.com/images/P/2070426769.01.THUMBZZZ.jpg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ISBN  \\\n",
       "220731  2070426769   \n",
       "\n",
       "                                                                       bookTitle  \\\n",
       "220731  Peuple du ciel, suivi de 'Les Bergers\\\";Jean-Marie Gustave Le ClÃ?Â©zio\"   \n",
       "\n",
       "       bookAuthor yearOfPublication  \\\n",
       "220731  2003       Gallimard          \n",
       "\n",
       "                                                           publisher  \n",
       "220731  http://images.amazon.com/images/P/2070426769.01.THUMBZZZ.jpg  "
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#investigating the rows having 'Gallimard' as yearOfPublication\n",
    "books.loc[books.yearOfPublication == 'Gallimard',:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#making required corrections as above, keeping other fields intact\n",
    "books.loc[books.ISBN == '2070426769','yearOfPublication'] = 2003\n",
    "books.loc[books.ISBN == '2070426769','bookAuthor'] = \"Jean-Marie Gustave Le ClÃ?Â©zio\"\n",
    "books.loc[books.ISBN == '2070426769','publisher'] = \"Gallimard\"\n",
    "books.loc[books.ISBN == '2070426769','bookTitle'] = \"Peuple du ciel, suivi de 'Les Bergers\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>220731</th>\n",
       "      <td>2070426769</td>\n",
       "      <td>Peuple du ciel, suivi de 'Les Bergers</td>\n",
       "      <td>Jean-Marie Gustave Le ClÃ?Â©zio</td>\n",
       "      <td>2003</td>\n",
       "      <td>Gallimard</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ISBN                              bookTitle  \\\n",
       "220731  2070426769  Peuple du ciel, suivi de 'Les Bergers   \n",
       "\n",
       "                             bookAuthor yearOfPublication  publisher  \n",
       "220731  Jean-Marie Gustave Le ClÃ?Â©zio  2003              Gallimard  "
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#rechecking\n",
    "books.loc[books.ISBN == '2070426769',:]\n",
    "#corrections done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Correcting the dtypes of yearOfPublication\n",
    "books.yearOfPublication=pd.to_numeric(books.yearOfPublication, errors='coerce')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1376, 1378, 1806, 1897, 1900, 1901, 1902, 1904, 1906, 1908, 1909, 1910, 1911, 1914, 1917, 1919, 1920, 1921, 1922, 1923, 1924, 1925, 1926, 1927, 1928, 1929, 1930, 1931, 1932, 1933, 1934, 1935, 1936, 1937, 1938, 1939, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954, 1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962, 1963, 1964, 1965, 1966, 1967, 1968, 1969, 1970, 1971, 1972, 1973, 1974, 1975, 1976, 1977, 1978, 1979, 1980, 1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2008, 2010, 2011, 2012, 2020, 2021, 2024, 2026, 2030, 2037, 2038, 2050]\n"
     ]
    }
   ],
   "source": [
    "print sorted(books['yearOfPublication'].unique())\n",
    "#Now it can be seen that yearOfPublication has all values as integers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#However, the value 0 is invalid and as this dataset was published in 2004, I have assumed the the years after 2006 to be \n",
    "#invalid keeping some margin in case dataset was updated thereafer\n",
    "#setting invalid years as NaN\n",
    "books.loc[(books.yearOfPublication > 2006) | (books.yearOfPublication == 0),'yearOfPublication'] = np.NAN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#replacing NaNs with mean value of yearOfPublication\n",
    "books.yearOfPublication.fillna(round(books.yearOfPublication.mean()), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 201,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#rechecking\n",
    "books.yearOfPublication.isnull().sum()\n",
    "#No NaNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#resetting the dtype as int32\n",
    "books.yearOfPublication = books.yearOfPublication.astype(np.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**publisher**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>128890</th>\n",
       "      <td>193169656X</td>\n",
       "      <td>Tyrant Moon</td>\n",
       "      <td>Elaine Corvidae</td>\n",
       "      <td>2002</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129037</th>\n",
       "      <td>1931696993</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Linnea Sinclair</td>\n",
       "      <td>2001</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ISBN        bookTitle       bookAuthor  yearOfPublication  \\\n",
       "128890  193169656X  Tyrant Moon      Elaine Corvidae  2002                \n",
       "129037  1931696993  Finders Keepers  Linnea Sinclair  2001                \n",
       "\n",
       "       publisher  \n",
       "128890  NaN       \n",
       "129037  NaN       "
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#exploring 'publisher' column\n",
    "books.loc[books.publisher.isnull(),:]\n",
    "#two NaNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>128890</th>\n",
       "      <td>193169656X</td>\n",
       "      <td>Tyrant Moon</td>\n",
       "      <td>Elaine Corvidae</td>\n",
       "      <td>2002</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ISBN    bookTitle       bookAuthor  yearOfPublication publisher\n",
       "128890  193169656X  Tyrant Moon  Elaine Corvidae  2002               NaN     "
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#investigating rows having NaNs\n",
    "#Checking with rows having bookTitle as Tyrant Moon to see if we can get any clues\n",
    "books.loc[(books.bookTitle == 'Tyrant Moon'),:]\n",
    "#no clues"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10799</th>\n",
       "      <td>082177364X</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Fern Michaels</td>\n",
       "      <td>2002</td>\n",
       "      <td>Zebra Books</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42019</th>\n",
       "      <td>0070465037</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Barbara Nickolae</td>\n",
       "      <td>1989</td>\n",
       "      <td>McGraw-Hill Companies</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58264</th>\n",
       "      <td>0688118461</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Emily Rodda</td>\n",
       "      <td>1993</td>\n",
       "      <td>Harpercollins Juvenile Books</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66678</th>\n",
       "      <td>1575663236</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Fern Michaels</td>\n",
       "      <td>1998</td>\n",
       "      <td>Kensington Publishing Corporation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129037</th>\n",
       "      <td>1931696993</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Linnea Sinclair</td>\n",
       "      <td>2001</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134309</th>\n",
       "      <td>0156309505</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Will</td>\n",
       "      <td>1989</td>\n",
       "      <td>Voyager Books</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>173473</th>\n",
       "      <td>0973146907</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Sean M. Costello</td>\n",
       "      <td>2002</td>\n",
       "      <td>Red Tower Publications</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195885</th>\n",
       "      <td>0061083909</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Sharon Sala</td>\n",
       "      <td>2003</td>\n",
       "      <td>HarperTorch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>211874</th>\n",
       "      <td>0373261160</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Elizabeth Travis</td>\n",
       "      <td>1993</td>\n",
       "      <td>Worldwide Library</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ISBN        bookTitle        bookAuthor  yearOfPublication  \\\n",
       "10799   082177364X  Finders Keepers  Fern Michaels     2002                \n",
       "42019   0070465037  Finders Keepers  Barbara Nickolae  1989                \n",
       "58264   0688118461  Finders Keepers  Emily Rodda       1993                \n",
       "66678   1575663236  Finders Keepers  Fern Michaels     1998                \n",
       "129037  1931696993  Finders Keepers  Linnea Sinclair   2001                \n",
       "134309  0156309505  Finders Keepers  Will              1989                \n",
       "173473  0973146907  Finders Keepers  Sean M. Costello  2002                \n",
       "195885  0061083909  Finders Keepers  Sharon Sala       2003                \n",
       "211874  0373261160  Finders Keepers  Elizabeth Travis  1993                \n",
       "\n",
       "                                publisher  \n",
       "10799   Zebra Books                        \n",
       "42019   McGraw-Hill Companies              \n",
       "58264   Harpercollins Juvenile Books       \n",
       "66678   Kensington Publishing Corporation  \n",
       "129037  NaN                                \n",
       "134309  Voyager Books                      \n",
       "173473  Red Tower Publications             \n",
       "195885  HarperTorch                        \n",
       "211874  Worldwide Library                  "
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Checking with rows having bookTitle as Finder Keepers to see if we can get any clues\n",
    "books.loc[(books.bookTitle == 'Finders Keepers'),:]\n",
    "#all rows with different publisher and bookAuthor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>126762</th>\n",
       "      <td>1931696934</td>\n",
       "      <td>Winter's Orphans</td>\n",
       "      <td>Elaine Corvidae</td>\n",
       "      <td>2001</td>\n",
       "      <td>Novelbooks</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128890</th>\n",
       "      <td>193169656X</td>\n",
       "      <td>Tyrant Moon</td>\n",
       "      <td>Elaine Corvidae</td>\n",
       "      <td>2002</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129001</th>\n",
       "      <td>0759901880</td>\n",
       "      <td>Wolfkin</td>\n",
       "      <td>Elaine Corvidae</td>\n",
       "      <td>2001</td>\n",
       "      <td>Hard Shell Word Factory</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ISBN         bookTitle       bookAuthor  yearOfPublication  \\\n",
       "126762  1931696934  Winter's Orphans  Elaine Corvidae  2001                \n",
       "128890  193169656X  Tyrant Moon       Elaine Corvidae  2002                \n",
       "129001  0759901880  Wolfkin           Elaine Corvidae  2001                \n",
       "\n",
       "                      publisher  \n",
       "126762  Novelbooks               \n",
       "128890  NaN                      \n",
       "129001  Hard Shell Word Factory  "
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking by bookAuthor to find patterns\n",
    "books.loc[(books.bookAuthor == 'Elaine Corvidae'),:]\n",
    "#all having different publisher...no clues here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>129037</th>\n",
       "      <td>1931696993</td>\n",
       "      <td>Finders Keepers</td>\n",
       "      <td>Linnea Sinclair</td>\n",
       "      <td>2001</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ISBN        bookTitle       bookAuthor  yearOfPublication  \\\n",
       "129037  1931696993  Finders Keepers  Linnea Sinclair  2001                \n",
       "\n",
       "       publisher  \n",
       "129037  NaN       "
      ]
     },
     "execution_count": 207,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking by bookAuthor to find patterns\n",
    "books.loc[(books.bookAuthor == 'Linnea Sinclair'),:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#since there is nothing in common to infer publisher for NaNs, replacing these with 'other\n",
    "books.loc[(books.ISBN == '193169656X'),'publisher'] = 'other'\n",
    "books.loc[(books.ISBN == '1931696993'),'publisher'] = 'other'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Users**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(278858, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>Location</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>nyc, new york, usa</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>stockton, california, usa</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>moscow, yukon territory, russia</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>porto, v.n.gaia, portugal</td>\n",
       "      <td>17.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>farnborough, hants, united kingdom</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID                            Location   Age\n",
       "0  1       nyc, new york, usa                 NaN  \n",
       "1  2       stockton, california, usa           18.0\n",
       "2  3       moscow, yukon territory, russia    NaN  \n",
       "3  4       porto, v.n.gaia, portugal           17.0\n",
       "4  5       farnborough, hants, united kingdom NaN  "
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print users.shape\n",
    "users.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userID      int64  \n",
       "Location    object \n",
       "Age         float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**userID**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([     1,      2,      3, ..., 278856, 278857, 278858], dtype=int64)"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users.userID.values\n",
    "#it can be seen that these are unique"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Age**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nan, 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0, 40.0, 41.0, 42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0, 50.0, 51.0, 52.0, 53.0, 54.0, 55.0, 56.0, 57.0, 58.0, 59.0, 60.0, 61.0, 62.0, 63.0, 64.0, 65.0, 66.0, 67.0, 68.0, 69.0, 70.0, 71.0, 72.0, 73.0, 74.0, 75.0, 76.0, 77.0, 78.0, 79.0, 80.0, 81.0, 82.0, 83.0, 84.0, 85.0, 86.0, 87.0, 88.0, 89.0, 90.0, 91.0, 92.0, 93.0, 94.0, 95.0, 96.0, 97.0, 98.0, 99.0, 100.0, 101.0, 102.0, 103.0, 104.0, 105.0, 106.0, 107.0, 108.0, 109.0, 110.0, 111.0, 113.0, 114.0, 115.0, 116.0, 118.0, 119.0, 123.0, 124.0, 127.0, 128.0, 132.0, 133.0, 136.0, 137.0, 138.0, 140.0, 141.0, 143.0, 146.0, 147.0, 148.0, 151.0, 152.0, 156.0, 157.0, 159.0, 162.0, 168.0, 172.0, 175.0, 183.0, 186.0, 189.0, 199.0, 200.0, 201.0, 204.0, 207.0, 208.0, 209.0, 210.0, 212.0, 219.0, 220.0, 223.0, 226.0, 228.0, 229.0, 230.0, 231.0, 237.0, 239.0, 244.0]\n"
     ]
    }
   ],
   "source": [
    "print sorted(users.Age.unique())\n",
    "#Age column has some invalid entries like nan, 0 and very high values like 100 and above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#In my view values below 5 and above 90 do not make much sense for our book rating case...hence replacing these by NaNs\n",
    "users.loc[(users.Age > 90) | (users.Age < 5), 'Age'] = np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#replacing NaNs with mean\n",
    "users.Age = users.Age.fillna(users.Age.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#setting the data type as int\n",
    "users.Age = users.Age.astype(np.int32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90]\n"
     ]
    }
   ],
   "source": [
    "#rechecking\n",
    "print sorted(users.Age.unique())\n",
    "#looks good now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Ratings Dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1149780, 3)"
      ]
     },
     "execution_count": 217,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking shape\n",
    "ratings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "75670906880\n"
     ]
    }
   ],
   "source": [
    "#ratings dataset will have n_users*n_books entries if every user rated every item, this shows that the dataset is very sparse\n",
    "n_users = users.shape[0]\n",
    "n_books = books.shape[0]\n",
    "print n_users * n_books"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookRating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>276725</td>\n",
       "      <td>034545104X</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>276726</td>\n",
       "      <td>0155061224</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>276727</td>\n",
       "      <td>0446520802</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>276729</td>\n",
       "      <td>052165615X</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>276729</td>\n",
       "      <td>0521795028</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID        ISBN  bookRating\n",
       "0  276725  034545104X  0         \n",
       "1  276726  0155061224  5         \n",
       "2  276727  0446520802  0         \n",
       "3  276729  052165615X  3         \n",
       "4  276729  0521795028  6         "
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking first few rows...\n",
    "ratings.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  5,  3,  6,  8,  7, 10,  9,  4,  1,  2], dtype=int64)"
      ]
     },
     "execution_count": 220,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.bookRating.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#ratings dataset should have books only which exist in our books dataset, unless new books are added to books dataset\n",
    "ratings_new = ratings[ratings.ISBN.isin(books.ISBN)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1149780, 3)\n",
      "(1031136, 3)\n"
     ]
    }
   ],
   "source": [
    "print ratings.shape\n",
    "print ratings_new.shape\n",
    "#it can be seen that many rows having book ISBN not part of books dataset got dropped off"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#ratings dataset should have ratings from users which exist in users dataset, unless new users are added to users dataset\n",
    "ratings = ratings[ratings.userID.isin(users.userID)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1149780, 3)\n",
      "(1031136, 3)\n"
     ]
    }
   ],
   "source": [
    "print ratings.shape\n",
    "print ratings_new.shape\n",
    "#no new users added, hence we will go with above dataset ratings_new (1031136, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of users: 278858\n",
      "number of books: 271360\n"
     ]
    }
   ],
   "source": [
    "print \"number of users: \" + str(n_users)\n",
    "print \"number of books: \" + str(n_books)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sparsity level of Book Crossing dataset is 99.9986373416 %\n"
     ]
    }
   ],
   "source": [
    "#Sparsity of dataset in %\n",
    "sparsity=1.0-len(ratings_new)/float(n_users*n_books)\n",
    "print 'The sparsity level of Book Crossing dataset is ' +  str(sparsity*100) + ' %'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  5,  3,  6,  8,  7, 10,  9,  4,  1,  2], dtype=int64)"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#As quoted in the description of the dataset -\n",
    "#BX-Book-Ratings contains the book rating information. Ratings are either explicit, expressed on a scale from 1-10 \n",
    "#higher values denoting higher appreciation, or implicit, expressed by 0\n",
    "ratings.bookRating.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Hence segragating implicit and explict ratings datasets\n",
    "ratings_explicit = ratings_new[ratings_new.bookRating != 0]\n",
    "ratings_implicit = ratings_new[ratings_new.bookRating == 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1031136, 3)\n",
      "(383842, 3)\n",
      "(647294, 3)\n"
     ]
    }
   ],
   "source": [
    "#checking shapes\n",
    "print ratings_new.shape\n",
    "print ratings_explicit.shape\n",
    "print ratings_implicit.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf0AAAFXCAYAAACoS5cAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHsdJREFUeJzt3X9Y1fXh9/HX4Zc/OAeRpW3NUClZtoa/SGvXkXK51Gu3\n05la0K3utrViusLS8DcpkuMq0JVz2LK7LhSQ0pXXfTVbmoMMIy+uwNJs5SwVi9Do6hxEOMDn/uN7\n7VwyFU7Njwd4Px9/6YcP8DomPfkcjuc4LMuyBAAAur2QYA8AAABXBtEHAMAQRB8AAEMQfQAADEH0\nAQAwBNEHAMAQYcEeYLfaWk+wJwAAcEX16+e66HGu9AEAMATRBwDAEEQfAABDEH0AAAxB9AEAMATR\nBwDAEEQfAABDEH0AAAxB9AEAMATRBwDAEEQfAABDEH0AAAxB9AEAMES3f5U9ADBJ8b6zwZ7Qxkx3\n72BPwHm40gcAwBBEHwAAQxB9AAAMQfQBADAE0QcAwBBEHwAAQxB9AAAMQfQBADAE0QcAwBBEHwAA\nQxB9AAAMQfQBADAE0QcAwBBEHwAAQxB9AAAMQfQBADAE0QcAwBBEHwAAQxB9AAAMQfQBADAE0QcA\nwBBEHwAAQxB9AAAMQfQBADAE0QcAwBBEHwAAQxB9AAAMQfQBADAE0QcAwBBEHwAAQxB9AAAMQfQB\nADAE0QcAwBBhdn1gn8+nxYsXq7q6WiEhIcrMzFRYWJgWL14sh8OhIUOGKCMjQyEhISouLlZRUZHC\nwsKUmpqqcePG6dy5c1q0aJHOnDmjyMhIZWdnKyYmRpWVlcrKylJoaKjcbrfmz59v100AAKBbse1K\nv6SkRM3NzSoqKtK8efO0fv16rV27VmlpaSooKJBlWdqzZ49qa2uVn5+voqIibd68Wbm5uWpqalJh\nYaHi4+NVUFCgqVOnauPGjZKkjIwM5eTkqLCwUFVVVTp8+LBdNwEAgG7FtugPHjxYLS0tam1tldfr\nVVhYmA4dOqTRo0dLkpKSklRWVqaDBw9qxIgRioiIkMvlUmxsrI4cOaKKigqNHTvWf+7+/fvl9XrV\n1NSk2NhYORwOud1ulZWV2XUTAADoVmy7e793796qrq7WpEmTVFdXp7y8PB04cEAOh0OSFBkZKY/H\nI6/XK5fL5X+/yMhIeb3eNsfPP9fpdLY598SJE+3u6Nu3t8LCQm24hQDQGZ0N9oA2+vVzdXwSrhjb\nov/CCy/I7Xbr0Ucf1eeff645c+bI5/P5315fX6+oqCg5nU7V19e3Oe5yudocb+/cqKiodnfU1XWu\nLwAAMEltrSfYE4x0qW+2bLt7Pyoqyn+l3qdPHzU3N+vGG29UeXm5JKm0tFSJiYlKSEhQRUWFGhsb\n5fF4dPToUcXHx2vkyJEqKSnxnztq1Cg5nU6Fh4fr+PHjsixL+/btU2Jiol03AQCAbsVhWZZlxweu\nr6/X0qVLVVtbK5/Pp9mzZ+umm27SihUr5PP5FBcXpzVr1ig0NFTFxcXatm2bLMvSAw88oAkTJqih\noUHp6emqra1VeHi4cnJy1K9fP1VWVuqJJ55QS0uL3G63FixY0O4OvssEYJLifZ3r3s2Z7t7BnmCk\nS13p2xb9zoLoAzAJ0YcUhLv3AQBA50L0AQAwBNEHAMAQRB8AAEMQfQAADEH0AQAwBNEHAMAQRB8A\nAEMQfQAADEH0AQAwBNEHAMAQRB8AAEMQfQAADEH0AQAwBNEHAMAQRB8AAEMQfQAADEH0AQAwBNEH\nAMAQRB8AAEMQfQAADEH0AQAwRFiwBwAA0NW0bv1nsCf4hdwbH/i5Nu4AAACdCNEHAMAQRB8AAEMQ\nfQAADEH0AQAwBNEHAMAQRB8AAEMQfQAADEH0AQAwBNEHAMAQRB8AAEMQfQAADEH0AQAwBNEHAMAQ\nvLQuALTjt6VVwZ7g92zSsGBPQBfHlT4AAIYg+gAAGILoAwBgCKIPAIAhiD4AAIYg+gAAGILoAwBg\nCKIPAIAhiD4AAIYg+gAAGILoAwBgCKIPAIAhiD4AAIYg+gAAGILoAwBgCKIPAIAhwuz84Js2bdKb\nb74pn8+n5ORkjR49WosXL5bD4dCQIUOUkZGhkJAQFRcXq6ioSGFhYUpNTdW4ceN07tw5LVq0SGfO\nnFFkZKSys7MVExOjyspKZWVlKTQ0VG63W/Pnz7fzJgAAbHby1YZgT2hjwJRewZ5gG9uu9MvLy/Xe\ne++psLBQ+fn5+uKLL7R27VqlpaWpoKBAlmVpz549qq2tVX5+voqKirR582bl5uaqqalJhYWFio+P\nV0FBgaZOnaqNGzdKkjIyMpSTk6PCwkJVVVXp8OHDdt0EAAC6Fduiv2/fPsXHx2vevHl68MEHdfvt\nt+vQoUMaPXq0JCkpKUllZWU6ePCgRowYoYiICLlcLsXGxurIkSOqqKjQ2LFj/efu379fXq9XTU1N\nio2NlcPhkNvtVllZmV03AQCAbsW2u/fr6up06tQp5eXl6eTJk0pNTZVlWXI4HJKkyMhIeTweeb1e\nuVwu//tFRkbK6/W2OX7+uU6ns825J06caHdH3769FRYWasMtBIArq18/V8cn6aztO76NQDafVOe6\nez+QzTVXYEegAvt78T9si350dLTi4uIUERGhuLg49ejRQ1988YX/7fX19YqKipLT6VR9fX2b4y6X\nq83x9s6Niopqd0ddXef6AgCA76q21hPsCd8am+13sb2X+kbAtrv3R40apbfeekuWZammpkYNDQ26\n9dZbVV5eLkkqLS1VYmKiEhISVFFRocbGRnk8Hh09elTx8fEaOXKkSkpK/OeOGjVKTqdT4eHhOn78\nuCzL0r59+5SYmGjXTQAAoFux7Up/3LhxOnDggKZPny7LsrRy5UoNGDBAK1asUG5uruLi4jRhwgSF\nhoZq1qxZSklJkWVZWrBggXr06KHk5GSlp6crOTlZ4eHhysnJkSStWrVKCxcuVEtLi9xut4YNG2bX\nTQAAoFtxWJZlBXuEnbra3TQAOpffllYFe4Lfs0kdX+QU7+tcP9Kc6e7d4Tld8Z/stW795xVYEpiQ\ne+MvOHbF794HAACdC9EHAMAQRB8AAEMQfQAADEH0AQAwBNEHAMAQRB8AAEMQfQAADEH0AQAwBNEH\nAMAQRB8AAEMQfQAADEH0AQAwBNEHAMAQRB8AAEMQfQAADEH0AQAwBNEHAMAQRB8AAEMQfQAADBFQ\n9DMzMy84lp6eftnHAAAA+4S198Zly5bpxIkT+uCDD/Txxx/7jzc3N8vj8dg+DgAAXD7tRj81NVXV\n1dXKysrS/Pnz/cdDQ0N13XXX2T4OAABcPu1Gf8CAARowYIB27twpr9crj8cjy7IkSWfPnlV0dPQV\nGQkAAP577Ub/3zZt2qRNmza1ibzD4dCePXtsGwYAAC6vgKL/0ksvaffu3YqJibF7DwAAsElAj97/\nwQ9+oD59+ti9BQAA2CigK/1BgwYpJSVFY8aMUUREhP/4+Q/uAwAAnVtA0b/66qt19dVX270FAADY\nKKDoc0UPAEDXF1D0b7jhBjkcjjbH+vfvr5KSEltGAQCAyy+g6B85csT/a5/Pp927d6uystK2UQAA\n4PL71i+4Ex4erkmTJumdd96xYw8AALBJQFf6r7zyiv/XlmXp448/Vnh4uG2jAADA5RdQ9MvLy9v8\nvm/fvlq3bp0tgwAAgD0Civ7atWvl8/l07NgxtbS0aMiQIQoLC+hdAQBAJxFQuT/44AM99NBDio6O\nVmtrq06fPq0//elPGjZsmN37AADAZRJQ9NesWaN169b5I19ZWanMzEy9/PLLto4DAACXT0CP3j97\n9mybq/rhw4ersbHRtlEAAODyCyj6ffr00e7du/2/3717d5uX2QUAAJ1fQHfvZ2Zm6oEHHtCyZcv8\nx4qKimwbBaBjc/b9MdgT2njR/XCwJwDoQEBX+qWlperVq5f27t2rF198UTExMXr33Xft3gYAAC6j\ngKJfXFyswsJC9e7dWzfccIN27NihLVu22L0NAABcRgFF3+fztXkGPp6NDwCAriegn+mPHz9ec+bM\n0aRJkyRJf//733XHHXfYOgwAAFxeAUV/0aJF2rVrlw4cOKCwsDDNnj1b48ePt3sbAAC4jAJ+Lt2J\nEydq4sSJdm4BAAA2+tYvrQsAALomog8AgCGIPgAAhiD6AAAYgugDAGAIog8AgCGIPgAAhrA1+mfO\nnNFtt92mo0eP6rPPPlNycrJSUlKUkZGh1tZWSf/zvP7Tpk3TzJkztXfvXknSuXPn9Pvf/14pKSm6\n//779dVXX0mSKisrNWPGDN1zzz3asGGDndMBAOh2bIu+z+fTypUr1bNnT0nS2rVrlZaWpoKCAlmW\npT179qi2tlb5+fkqKirS5s2blZubq6amJhUWFio+Pl4FBQWaOnWqNm7cKEnKyMhQTk6OCgsLVVVV\npcOHD9s1HwCAbse26GdnZ+uee+5R//79JUmHDh3S6NGjJUlJSUkqKyvTwYMHNWLECEVERMjlcik2\nNlZHjhxRRUWFxo4d6z93//798nq9ampqUmxsrBwOh9xut8rKyuyaDwBAtxPw0/B+Gzt27FBMTIzG\njh2rZ599VpJkWZYcDockKTIyUh6PR16vVy6Xy/9+kZGR8nq9bY6ff67T6Wxz7okTJzrc0rdvb4WF\nhV7OmwfgIvr1c3V8Ev4rgf0Zn7V9x7cRyOaTargCSwIXyOaaK7AjUN/ma8+W6G/fvl0Oh0P79+/X\nhx9+qPT0dP/P5SWpvr5eUVFRcjqdqq+vb3Pc5XK1Od7euVFRUR1uqavrXF8AQHdVW+sJ9oRuryv+\nGbPZfhfbe6lvBGy5e3/r1q3asmWL8vPzNXToUGVnZyspKUnl5eWSpNLSUiUmJiohIUEVFRVqbGyU\nx+PR0aNHFR8fr5EjR6qkpMR/7qhRo+R0OhUeHq7jx4/Lsizt27dPiYmJdswHAKBbsuVK/2LS09O1\nYsUK5ebmKi4uThMmTFBoaKhmzZqllJQUWZalBQsWqEePHkpOTlZ6erqSk5MVHh6unJwcSdKqVau0\ncOFCtbS0yO12a9iwYVdqPgAAXZ7t0c/Pz/f/esuWLRe8febMmZo5c2abY7169dLTTz99wbnDhw9X\ncXHx5R8JAIABeHIeAAAMQfQBADAE0QcAwBBEHwAAQxB9AAAMQfQBADAE0QcAwBBEHwAAQxB9AAAM\nQfQBADAE0QcAwBBEHwAAQxB9AAAMQfQBADAE0QcAwBBEHwAAQxB9AAAMQfQBADAE0QcAwBBEHwAA\nQ4QFewAAc/yfkp3BntDG/73tl8GeAFxRXOkDAGAIog8AgCGIPgAAhiD6AAAYgugDAGAIog8AgCGI\nPgAAhiD6AAAYgugDAGAIog8AgCGIPgAAhiD6AAAYgugDAGAIog8AgCGIPgAAhiD6AAAYgugDAGAI\nog8AgCGIPgAAhiD6AAAYgugDAGAIog8AgCGIPgAAhiD6AAAYgugDAGAIog8AgCGIPgAAhiD6AAAY\ngugDAGAIog8AgCGIPgAAhiD6AAAYgugDAGCIMDs+qM/n09KlS1VdXa2mpialpqbq+uuv1+LFi+Vw\nODRkyBBlZGQoJCRExcXFKioqUlhYmFJTUzVu3DidO3dOixYt0pkzZxQZGans7GzFxMSosrJSWVlZ\nCg0Nldvt1vz58+2YDwBAt2TLlf7OnTsVHR2tgoICPffcc8rMzNTatWuVlpamgoICWZalPXv2qLa2\nVvn5+SoqKtLmzZuVm5urpqYmFRYWKj4+XgUFBZo6dao2btwoScrIyFBOTo4KCwtVVVWlw4cP2zEf\nAIBuyZboT5w4UQ8//LAkybIshYaG6tChQxo9erQkKSkpSWVlZTp48KBGjBihiIgIuVwuxcbG6siR\nI6qoqNDYsWP95+7fv19er1dNTU2KjY2Vw+GQ2+1WWVmZHfMBAOiWbLl7PzIyUpLk9Xr10EMPKS0t\nTdnZ2XI4HP63ezweeb1euVyuNu/n9XrbHD//XKfT2ebcEydOdLilb9/eCgsLvZw3D8BF9Ovn6vik\nTqarbQ5s71nbd3wbgWw+qYYrsCRwgWyuuQI7AvVt/h7bEn1J+vzzzzVv3jylpKRo8uTJevLJJ/1v\nq6+vV1RUlJxOp+rr69scd7lcbY63d25UVFSHO+rqOtcXANBd1dZ6gj3hW+tqm7vaXonNV8LF9l7q\nGwFb7t4/ffq05s6dq0WLFmn69OmSpBtvvFHl5eWSpNLSUiUmJiohIUEVFRVqbGyUx+PR0aNHFR8f\nr5EjR6qkpMR/7qhRo+R0OhUeHq7jx4/Lsizt27dPiYmJdswHAKBbsuVKPy8vT9988402btzofxDe\nsmXLtGbNGuXm5iouLk4TJkxQaGioZs2apZSUFFmWpQULFqhHjx5KTk5Wenq6kpOTFR4erpycHEnS\nqlWrtHDhQrW0tMjtdmvYsGF2zAcAoFuyJfrLly/X8uXLLzi+ZcuWC47NnDlTM2fObHOsV69eevrp\npy84d/jw4SouLr58QwEAMAhPzgMAgCGIPgAAhiD6AAAYgugDAGAIog8AgCGIPgAAhiD6AAAYgugD\nAGAIog8AgCGIPgAAhiD6AAAYgugDAGAIog8AgCGIPgAAhiD6AAAYgugDAGAIog8AgCGIPgAAhiD6\nAAAYgugDAGAIog8AgCGIPgAAhiD6AAAYgugDAGAIog8AgCGIPgAAhiD6AAAYgugDAGAIog8AgCGI\nPgAAhiD6AAAYgugDAGAIog8AgCHCgj0A6Cz+tHd6sCf4zRv3crAnAOiGuNIHAMAQRB8AAEMQfQAA\nDEH0AQAwBNEHAMAQRB8AAEMQfQAADMG/04ctSv7fjGBPaOO2//VSsCcAQNBxpQ8AgCGIPgAAhiD6\nAAAYgugDAGAIog8AgCGIPgAAhiD6AAAYgn+n3wWcKf7fwZ7Qxvdmbgn2BADAd8CVPgAAhiD6AAAY\nwsy7919+NdgL2po+JdgLAAAG4EofAABDdLkr/dbWVj3++OP66KOPFBERoTVr1mjgwIHBngUAQKfX\n5a70d+/eraamJm3btk2PPvqo/vCHPwR7EgAAXUKXi35FRYXGjh0rSRo+fLg++OCDIC8CAKBrcFiW\nZQV7xLexbNky3XnnnbrtttskSbfffrt2796tsLAu95MKAACuqC53pe90OlVfX+//fWtrK8EHACAA\nXS76I0eOVGlpqSSpsrJS8fHxQV4EAEDX0OXu3v/3o/f/+c9/yrIsPfHEE7ruuuuCPQsAgE6vy0Uf\nAAB8N13u7n0AAPDdEH0AAAzBw96/o6qqKj311FPKz88P9pQO+Xw+LV26VNXV1WpqalJqaqruuOOO\nYM9qV0tLi5YvX65jx47J4XBo1apVXeJBm2fOnNG0adP0/PPPd4nHmvzqV7+S0+mUJA0YMEBr164N\n8qL2bdq0SW+++aZ8Pp+Sk5M1Y8aMYE9q144dO/TXv/5VktTY2KgPP/xQb7/9tqKiooK87NJ8Pp8W\nL16s6upqhYSEKDMzs1P/XW5qatKSJUt04sQJOZ1OrVy5UoMGDQr2rEs6vx2fffaZFi9eLIfDoSFD\nhigjI0MhIfZeixP97+Avf/mLdu7cqV69egV7SkB27typ6OhoPfnkk/r66681derUTh/9vXv3SpKK\niopUXl6udevW6c9//nOQV7XP5/Np5cqV6tmzZ7CnBKSxsVGWZXWJb1wlqby8XO+9954KCwvV0NCg\n559/PtiTOjRt2jRNmzZNkrRq1SrdddddnTr4klRSUqLm5mYVFRXp7bff1vr16/XMM88Ee9YlFRcX\nq3fv3iouLta//vUvZWZmavPmzcGedVH/2Y61a9cqLS1NY8aM0cqVK7Vnzx79/Oc/t3UDd+9/B7Gx\nsZ36i+A/TZw4UQ8//LAkybIshYaGBnlRx8aPH6/MzExJ0qlTpzr9/yglKTs7W/fcc4/69+8f7CkB\nOXLkiBoaGjR37lzNnj1blZWVwZ7Urn379ik+Pl7z5s3Tgw8+qNtvvz3YkwL2/vvv65NPPtHdd98d\n7CkdGjx4sFpaWtTa2iqv19vpnwflk08+UVJSkiQpLi5OR48eDfKiS/vPdhw6dEijR4+WJCUlJams\nrMz2DZ37v2YnNWHCBJ08eTLYMwIWGRkpSfJ6vXrooYeUlpYW5EWBCQsLU3p6ut544w09/fTTwZ7T\nrh07digmJkZjx47Vs88+G+w5AenZs6fuu+8+zZgxQ59++qnuv/9+7dq1q9P+T76urk6nTp1SXl6e\nTp48qdTUVO3atUsOhyPY0zq0adMmzZs3L9gzAtK7d29VV1dr0qRJqqurU15eXrAntWvo0KHau3ev\nxo8fr6qqKtXU1KilpaVTXtz8Zzssy/L//Y2MjJTH47F9A1f6hvj88881e/ZsTZkyRZMnTw72nIBl\nZ2fr9ddf14oVK3T27Nlgz7mk7du3q6ysTLNmzdKHH36o9PR01dbWBntWuwYPHqxf/vKXcjgcGjx4\nsKKjozv15ujoaLndbkVERCguLk49evTQV199FexZHfrmm2907Ngx3XLLLcGeEpAXXnhBbrdbr7/+\nul599VUtXrxYjY2NwZ51SXfddZecTqdSUlL0xhtv6Mc//nGnDP7FnP/z+/r6+ityjybRN8Dp06c1\nd+5cLVq0SNOnTw/2nIC88sor2rRpkySpV69ecjgctj/A5b+xdetWbdmyRfn5+Ro6dKiys7PVr1+/\nYM9q18svv+x/lcqamhp5vd5OvXnUqFF66623ZFmWampq1NDQoOjo6GDP6tCBAwd06623BntGwKKi\nouRyuSRJffr0UXNzs1paWoK86tLef/993XrrrSosLNTEiRN17bXXBntSwG688UaVl5dLkkpLS5WY\nmGj75+yc9+PhssrLy9M333yjjRs3auPGjZL+5wElnfkBZ3feeaeWLFmie++9V83NzVq6dGmn3tsV\nTZ8+XUuWLFFycrIcDoeeeOKJTnvXviSNGzdOBw4c0PTp02VZllauXNklruiOHTumAQMGBHtGwH79\n619r6dKlSklJkc/n04IFC9S7d+9gz7qkgQMH6o9//KPy8vLkcrmUlZUV7EkBS09P14oVK5Sbm6u4\nuDhNmDDB9s/JM/IBAGCIznt/KQAAuKyIPgAAhiD6AAAYgugDAGAIog8AgCGIPmCQ8vJyzZo167/+\nOLNmzfL/++J/O3nypG666SZNmTLF/yRQP/vZzwJ6NsXzN02ZMuW/3gfg4jrvP8oF0OX0799fr776\nqv/3NTU1mjBhgn7xi1+0+0pt7777rv/X578/gMuL6AOGqaur03333acvv/xSCQkJysjI8L+aWmtr\nq6699lqtXr1aV111lSorK5WVlaXGxkb17dtXq1ev1sCBA/0f68yZM5ozZ47S0tJ0ww03XPC5amtr\nZVmWIiMj1dzcrMcff1wff/yxTp8+rcGDB2vDhg166qmnJEkzZszQSy+9pB/96Ef66KOP9Mwzz6im\npkafffaZqqurNWPGDKWmpsrn8ykjI0MVFRW6+uqr5XA49Lvf/U5jxoy5Yn+GQFdF9AHDnDx5Uhs2\nbNDAgQO1YMECPfvss9q2bZsKCws1YMAAPffcc1q9erWeeuopPfLII1q/fr0SEhL0t7/9TY888oi2\nb98uSfJ4PPrtb3+r+fPna/z48Tp58qS+/PJLTZkyRY2Njaqrq9NPfvITbdiwQd///vd14MABhYeH\na9u2bWptbdWcOXNUUlKi5cuXKz8/Xy+99NIFWz/66CNt3bpVHo9H48eP17333qtXX31VDQ0N2rVr\nl06dOtWlXksCCDZ+pg8YJjExUYMGDZLD4dDkyZP14osvKiEhwf9UsXfffbfeeecdffrpp4qKilJC\nQoIkadKkSTp+/Lj/lcAyMjLU3NysO++80/+x/333/muvvaYpU6bI5/P5X2jm5ptvVkpKirZu3aqs\nrCx9+umnHb6I0pgxYxQREaHvfe97io6Olsfj0dtvv63JkyfL4XDohz/8YZd6Xnsg2Ig+YJjzn1//\n/Jf2PP9Yc3OzWltbL3hfy7L8L75y//33KyYmRoWFhRecFxISoscee0xnzpzR888/L0nas2ePFi5c\nqJ49e2ratGm6+eab1dGzgPfo0cP/a4fDIcuyFBoaetFtADpG9AHDVFRU6NSpU2ptbdUrr7yi3/zm\nN6qqqvK/zve2bds0ZswYxcXF6euvv9bBgwclSa+99pquueYa/yvbDR06VBkZGdqwYYNqamou+Dxh\nYWF67LHHlJeXp9raWu3fv1+TJk3SXXfdpauuukoHDhzwfwMRGhqq5ubmgPb/9Kc/1WuvveZ/tb13\n3333gm9cAFwcP9MHDHP99ddr6dKlqq2t1S233KL77rtP119/vebPny+fz6drrrlGWVlZioiI0Lp1\n65SZmamGhgb16dNH69ata/OxBg0apHvvvVerV6/WkiVLLvhcSUlJGj58uNavX6/Zs2dr4cKF2rVr\nlyIiIjR8+HD/Nxp33HGHpkyZoh07dnS4f+bMmTpy5IgmT56sfv366ZprruEVGIEA8Sp7ALqUf/zj\nH7IsS+PGjZPH49HUqVO1fft2/z0QAC6N6APoUk6cOKHHHnvM/yDAuXPn8oQ+QICIPgAAhuCBfAAA\nGILoAwBgCKIPAIAhiD4AAIYg+gAAGILoAwBgiP8PjIYQhxu3xXEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x47a4c9e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plotting count of bookRating\n",
    "sns.countplot(data=ratings_explicit , x='bookRating')\n",
    "plt.show()\n",
    "#It can be seen that higher ratings are more common amongst users and rating 8 has been rated highest number of times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Simple Popularity Based Recommendation System**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Following books are recommended\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bookRating</th>\n",
       "      <th>ISBN</th>\n",
       "      <th>bookTitle</th>\n",
       "      <th>bookAuthor</th>\n",
       "      <th>yearOfPublication</th>\n",
       "      <th>publisher</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>408</th>\n",
       "      <td>5787</td>\n",
       "      <td>0316666343</td>\n",
       "      <td>The Lovely Bones: A Novel</td>\n",
       "      <td>Alice Sebold</td>\n",
       "      <td>2002</td>\n",
       "      <td>Little, Brown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>748</th>\n",
       "      <td>4108</td>\n",
       "      <td>0385504209</td>\n",
       "      <td>The Da Vinci Code</td>\n",
       "      <td>Dan Brown</td>\n",
       "      <td>2003</td>\n",
       "      <td>Doubleday</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>522</th>\n",
       "      <td>3134</td>\n",
       "      <td>0312195516</td>\n",
       "      <td>The Red Tent (Bestselling Backlist)</td>\n",
       "      <td>Anita Diamant</td>\n",
       "      <td>1998</td>\n",
       "      <td>Picador USA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2143</th>\n",
       "      <td>2798</td>\n",
       "      <td>059035342X</td>\n",
       "      <td>Harry Potter and the Sorcerer's Stone (Harry Potter (Paperback))</td>\n",
       "      <td>J. K. Rowling</td>\n",
       "      <td>1999</td>\n",
       "      <td>Arthur A. Levine Books</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>356</th>\n",
       "      <td>2595</td>\n",
       "      <td>0142001740</td>\n",
       "      <td>The Secret Life of Bees</td>\n",
       "      <td>Sue Monk Kidd</td>\n",
       "      <td>2003</td>\n",
       "      <td>Penguin Books</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>2551</td>\n",
       "      <td>0971880107</td>\n",
       "      <td>Wild Animus</td>\n",
       "      <td>Rich Shapero</td>\n",
       "      <td>2004</td>\n",
       "      <td>Too Far</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1105</th>\n",
       "      <td>2524</td>\n",
       "      <td>0060928336</td>\n",
       "      <td>Divine Secrets of the Ya-Ya Sisterhood: A Novel</td>\n",
       "      <td>Rebecca Wells</td>\n",
       "      <td>1997</td>\n",
       "      <td>Perennial</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>706</th>\n",
       "      <td>2402</td>\n",
       "      <td>0446672211</td>\n",
       "      <td>Where the Heart Is (Oprah's Book Club (Paperback))</td>\n",
       "      <td>Billie Letts</td>\n",
       "      <td>1998</td>\n",
       "      <td>Warner Books</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231</th>\n",
       "      <td>2219</td>\n",
       "      <td>0452282152</td>\n",
       "      <td>Girl with a Pearl Earring</td>\n",
       "      <td>Tracy Chevalier</td>\n",
       "      <td>2001</td>\n",
       "      <td>Plume Books</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>2179</td>\n",
       "      <td>0671027360</td>\n",
       "      <td>Angels &amp;amp; Demons</td>\n",
       "      <td>Dan Brown</td>\n",
       "      <td>2001</td>\n",
       "      <td>Pocket Star</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      bookRating        ISBN  \\\n",
       "408   5787        0316666343   \n",
       "748   4108        0385504209   \n",
       "522   3134        0312195516   \n",
       "2143  2798        059035342X   \n",
       "356   2595        0142001740   \n",
       "26    2551        0971880107   \n",
       "1105  2524        0060928336   \n",
       "706   2402        0446672211   \n",
       "231   2219        0452282152   \n",
       "118   2179        0671027360   \n",
       "\n",
       "                                                             bookTitle  \\\n",
       "408   The Lovely Bones: A Novel                                          \n",
       "748   The Da Vinci Code                                                  \n",
       "522   The Red Tent (Bestselling Backlist)                                \n",
       "2143  Harry Potter and the Sorcerer's Stone (Harry Potter (Paperback))   \n",
       "356   The Secret Life of Bees                                            \n",
       "26    Wild Animus                                                        \n",
       "1105  Divine Secrets of the Ya-Ya Sisterhood: A Novel                    \n",
       "706   Where the Heart Is (Oprah's Book Club (Paperback))                 \n",
       "231   Girl with a Pearl Earring                                          \n",
       "118   Angels &amp; Demons                                                \n",
       "\n",
       "           bookAuthor  yearOfPublication               publisher  \n",
       "408   Alice Sebold     2002               Little, Brown           \n",
       "748   Dan Brown        2003               Doubleday               \n",
       "522   Anita Diamant    1998               Picador USA             \n",
       "2143  J. K. Rowling    1999               Arthur A. Levine Books  \n",
       "356   Sue Monk Kidd    2003               Penguin Books           \n",
       "26    Rich Shapero     2004               Too Far                 \n",
       "1105  Rebecca Wells    1997               Perennial               \n",
       "706   Billie Letts     1998               Warner Books            \n",
       "231   Tracy Chevalier  2001               Plume Books             \n",
       "118   Dan Brown        2001               Pocket Star             "
      ]
     },
     "execution_count": 232,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#At this point , a simple popularity based recommendation system can be built based on count of user ratings for different books\n",
    "ratings_count = pd.DataFrame(ratings_explicit.groupby(['ISBN'])['bookRating'].sum())\n",
    "top10 = ratings_count.sort_values('bookRating', ascending = False).head(10)\n",
    "print \"Following books are recommended\"\n",
    "top10.merge(books, left_index = True, right_on = 'ISBN')\n",
    "\n",
    "#Given below are top 10 recommendations based on popularity. It is evident that books authored by J.K. Rowling are most popular"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Similarly segregating users who have given explicit ratings from 1-10 and those whose implicit behavior was tracked\n",
    "users_exp_ratings = users[users.userID.isin(ratings_explicit.userID)]\n",
    "users_imp_ratings = users[users.userID.isin(ratings_implicit.userID)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(278858, 3)\n",
      "(68091, 3)\n",
      "(52451, 3)\n"
     ]
    }
   ],
   "source": [
    "#checking shapes\n",
    "print users.shape\n",
    "print users_exp_ratings.shape\n",
    "print users_imp_ratings.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Collaborative Filtering Based Recommendation Systems**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#To cope up with computing power I have and to reduce the dataset size, I am considering users who have rated atleast 100 books\n",
    "#and books which have atleast 100 ratings\n",
    "counts1 = ratings_explicit['userID'].value_counts()\n",
    "ratings_explicit = ratings_explicit[ratings_explicit['userID'].isin(counts1[counts1 >= 100].index)]\n",
    "counts = ratings_explicit['bookRating'].value_counts()\n",
    "ratings_explicit = ratings_explicit[ratings_explicit['bookRating'].isin(counts[counts >= 100].index)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(449, 66574)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>ISBN</th>\n",
       "      <th>0000913154</th>\n",
       "      <th>0001046438</th>\n",
       "      <th>000104687X</th>\n",
       "      <th>0001047213</th>\n",
       "      <th>0001047973</th>\n",
       "      <th>000104799X</th>\n",
       "      <th>0001048082</th>\n",
       "      <th>0001053736</th>\n",
       "      <th>0001053744</th>\n",
       "      <th>0001055607</th>\n",
       "      <th>...</th>\n",
       "      <th>B000092Q0A</th>\n",
       "      <th>B00009EF82</th>\n",
       "      <th>B00009NDAN</th>\n",
       "      <th>B0000DYXID</th>\n",
       "      <th>B0000T6KHI</th>\n",
       "      <th>B0000VZEJQ</th>\n",
       "      <th>B0000X8HIE</th>\n",
       "      <th>B00013AX9E</th>\n",
       "      <th>B0001I1KOG</th>\n",
       "      <th>B000234N3A</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>userID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2033</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2110</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2276</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4017</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4385</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 66574 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "ISBN    0000913154  0001046438  000104687X  0001047213  0001047973  \\\n",
       "userID                                                               \n",
       "2033   NaN         NaN         NaN         NaN         NaN           \n",
       "2110   NaN         NaN         NaN         NaN         NaN           \n",
       "2276   NaN         NaN         NaN         NaN         NaN           \n",
       "4017   NaN         NaN         NaN         NaN         NaN           \n",
       "4385   NaN         NaN         NaN         NaN         NaN           \n",
       "\n",
       "ISBN    000104799X  0001048082  0001053736  0001053744  0001055607  \\\n",
       "userID                                                               \n",
       "2033   NaN         NaN         NaN         NaN         NaN           \n",
       "2110   NaN         NaN         NaN         NaN         NaN           \n",
       "2276   NaN         NaN         NaN         NaN         NaN           \n",
       "4017   NaN         NaN         NaN         NaN         NaN           \n",
       "4385   NaN         NaN         NaN         NaN         NaN           \n",
       "\n",
       "ISBN       ...      B000092Q0A  B00009EF82  B00009NDAN  B0000DYXID  \\\n",
       "userID     ...                                                       \n",
       "2033       ...     NaN         NaN         NaN         NaN           \n",
       "2110       ...     NaN         NaN         NaN         NaN           \n",
       "2276       ...     NaN         NaN         NaN         NaN           \n",
       "4017       ...     NaN         NaN         NaN         NaN           \n",
       "4385       ...     NaN         NaN         NaN         NaN           \n",
       "\n",
       "ISBN    B0000T6KHI  B0000VZEJQ  B0000X8HIE  B00013AX9E  B0001I1KOG  B000234N3A  \n",
       "userID                                                                          \n",
       "2033   NaN         NaN         NaN         NaN         NaN         NaN          \n",
       "2110   NaN         NaN         NaN         NaN         NaN         NaN          \n",
       "2276   NaN         NaN         NaN         NaN         NaN         NaN          \n",
       "4017   NaN         NaN         NaN         NaN         NaN         NaN          \n",
       "4385   NaN         NaN         NaN         NaN         NaN         NaN          \n",
       "\n",
       "[5 rows x 66574 columns]"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Generating ratings matrix from explicit ratings table\n",
    "ratings_matrix = ratings_explicit.pivot(index='userID', columns='ISBN', values='bookRating')\n",
    "userID = ratings_matrix.index\n",
    "ISBN = ratings_matrix.columns\n",
    "print(ratings_matrix.shape)\n",
    "ratings_matrix.head()\n",
    "#Notice that most of the values are NaN (undefined) implying absence of ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "449 66574\n"
     ]
    }
   ],
   "source": [
    "n_users = ratings_matrix.shape[0] #considering only those users who gave explicit ratings\n",
    "n_books = ratings_matrix.shape[1]\n",
    "print n_users, n_books"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#since NaNs cannot be handled by training algorithms, replacing these by 0, which indicates absence of ratings\n",
    "#setting data type\n",
    "ratings_matrix.fillna(0, inplace = True)\n",
    "ratings_matrix = ratings_matrix.astype(np.int32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>ISBN</th>\n",
       "      <th>0000913154</th>\n",
       "      <th>0001046438</th>\n",
       "      <th>000104687X</th>\n",
       "      <th>0001047213</th>\n",
       "      <th>0001047973</th>\n",
       "      <th>000104799X</th>\n",
       "      <th>0001048082</th>\n",
       "      <th>0001053736</th>\n",
       "      <th>0001053744</th>\n",
       "      <th>0001055607</th>\n",
       "      <th>...</th>\n",
       "      <th>B000092Q0A</th>\n",
       "      <th>B00009EF82</th>\n",
       "      <th>B00009NDAN</th>\n",
       "      <th>B0000DYXID</th>\n",
       "      <th>B0000T6KHI</th>\n",
       "      <th>B0000VZEJQ</th>\n",
       "      <th>B0000X8HIE</th>\n",
       "      <th>B00013AX9E</th>\n",
       "      <th>B0001I1KOG</th>\n",
       "      <th>B000234N3A</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>userID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2033</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2110</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2276</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4017</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4385</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 66574 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "ISBN    0000913154  0001046438  000104687X  0001047213  0001047973  \\\n",
       "userID                                                               \n",
       "2033    0           0           0           0           0            \n",
       "2110    0           0           0           0           0            \n",
       "2276    0           0           0           0           0            \n",
       "4017    0           0           0           0           0            \n",
       "4385    0           0           0           0           0            \n",
       "\n",
       "ISBN    000104799X  0001048082  0001053736  0001053744  0001055607  \\\n",
       "userID                                                               \n",
       "2033    0           0           0           0           0            \n",
       "2110    0           0           0           0           0            \n",
       "2276    0           0           0           0           0            \n",
       "4017    0           0           0           0           0            \n",
       "4385    0           0           0           0           0            \n",
       "\n",
       "ISBN       ...      B000092Q0A  B00009EF82  B00009NDAN  B0000DYXID  \\\n",
       "userID     ...                                                       \n",
       "2033       ...      0           0           0           0            \n",
       "2110       ...      0           0           0           0            \n",
       "2276       ...      0           0           0           0            \n",
       "4017       ...      0           0           0           0            \n",
       "4385       ...      0           0           0           0            \n",
       "\n",
       "ISBN    B0000T6KHI  B0000VZEJQ  B0000X8HIE  B00013AX9E  B0001I1KOG  B000234N3A  \n",
       "userID                                                                          \n",
       "2033    0           0           0           0           0           0           \n",
       "2110    0           0           0           0           0           0           \n",
       "2276    0           0           0           0           0           0           \n",
       "4017    0           0           0           0           0           0           \n",
       "4385    0           0           0           0           0           0           \n",
       "\n",
       "[5 rows x 66574 columns]"
      ]
     },
     "execution_count": 239,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#checking first few rows\n",
    "ratings_matrix.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sparsity level of Book Crossing dataset is 99.9977218411 %\n"
     ]
    }
   ],
   "source": [
    "#rechecking the sparsity\n",
    "sparsity=1.0-len(ratings_explicit)/float(users_exp_ratings.shape[0]*n_books)\n",
    "print 'The sparsity level of Book Crossing dataset is ' +  str(sparsity*100) + ' %'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Training our recommendation system**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#setting global variables\n",
    "global metric,k\n",
    "k=10\n",
    "metric='cosine'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**User-based Recommendation System**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#This function finds k similar users given the user_id and ratings matrix \n",
    "#These similarities are same as obtained via using pairwise_distances\n",
    "def findksimilarusers(user_id, ratings, metric = metric, k=k):\n",
    "    similarities=[]\n",
    "    indices=[]\n",
    "    model_knn = NearestNeighbors(metric = metric, algorithm = 'brute') \n",
    "    model_knn.fit(ratings)\n",
    "    loc = ratings.index.get_loc(user_id)\n",
    "    distances, indices = model_knn.kneighbors(ratings.iloc[loc, :].values.reshape(1, -1), n_neighbors = k+1)\n",
    "    similarities = 1-distances.flatten()\n",
    "            \n",
    "    return similarities,indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#This function predicts rating for specified user-item combination based on user-based approach\n",
    "def predict_userbased(user_id, item_id, ratings, metric = metric, k=k):\n",
    "    prediction=0\n",
    "    user_loc = ratings.index.get_loc(user_id)\n",
    "    item_loc = ratings.columns.get_loc(item_id)\n",
    "    similarities, indices=findksimilarusers(user_id, ratings,metric, k) #similar users based on cosine similarity\n",
    "    mean_rating = ratings.iloc[user_loc,:].mean() #to adjust for zero based indexing\n",
    "    sum_wt = np.sum(similarities)-1\n",
    "    product=1\n",
    "    wtd_sum = 0 \n",
    "    \n",
    "    for i in range(0, len(indices.flatten())):\n",
    "        if indices.flatten()[i] == user_loc:\n",
    "            continue;\n",
    "        else: \n",
    "            ratings_diff = ratings.iloc[indices.flatten()[i],item_loc]-np.mean(ratings.iloc[indices.flatten()[i],:])\n",
    "            product = ratings_diff * (similarities[i])\n",
    "            wtd_sum = wtd_sum + product\n",
    "    \n",
    "    #in case of very sparse datasets, using correlation metric for collaborative based approach may give negative ratings\n",
    "    #which are handled here as below\n",
    "    if prediction <= 0:\n",
    "        prediction = 1   \n",
    "    elif prediction >10:\n",
    "        prediction = 10\n",
    "    \n",
    "    prediction = int(round(mean_rating + (wtd_sum/sum_wt)))\n",
    "    print '\\nPredicted rating for user {0} -> item {1}: {2}'.format(user_id,item_id,prediction)\n",
    "\n",
    "    return prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Predicted rating for user 11676 -> item 0001056107: 2\n"
     ]
    }
   ],
   "source": [
    "predict_userbased(11676,'0001056107',ratings_matrix);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Item-based Recommendation Systems**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#This function finds k similar items given the item_id and ratings matrix\n",
    "\n",
    "def findksimilaritems(item_id, ratings, metric=metric, k=k):\n",
    "    similarities=[]\n",
    "    indices=[]\n",
    "    ratings=ratings.T\n",
    "    loc = ratings.index.get_loc(item_id)\n",
    "    model_knn = NearestNeighbors(metric = metric, algorithm = 'brute')\n",
    "    model_knn.fit(ratings)\n",
    "    \n",
    "    distances, indices = model_knn.kneighbors(ratings.iloc[loc, :].values.reshape(1, -1), n_neighbors = k+1)\n",
    "    similarities = 1-distances.flatten()\n",
    "\n",
    "    return similarities,indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "similarities,indices=findksimilaritems('0001056107',ratings_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#This function predicts the rating for specified user-item combination based on item-based approach\n",
    "def predict_itembased(user_id, item_id, ratings, metric = metric, k=k):\n",
    "    prediction= wtd_sum =0\n",
    "    user_loc = ratings.index.get_loc(user_id)\n",
    "    item_loc = ratings.columns.get_loc(item_id)\n",
    "    similarities, indices=findksimilaritems(item_id, ratings) #similar users based on correlation coefficients\n",
    "    sum_wt = np.sum(similarities)-1\n",
    "    product=1\n",
    "    for i in range(0, len(indices.flatten())):\n",
    "        if indices.flatten()[i] == item_loc:\n",
    "            continue;\n",
    "        else:\n",
    "            product = ratings.iloc[user_loc,indices.flatten()[i]] * (similarities[i])\n",
    "            wtd_sum = wtd_sum + product                              \n",
    "    prediction = int(round(wtd_sum/sum_wt))\n",
    "    \n",
    "    #in case of very sparse datasets, using correlation metric for collaborative based approach may give negative ratings\n",
    "    #which are handled here as below //code has been validated without the code snippet below, below snippet is to avoid negative\n",
    "    #predictions which might arise in case of very sparse datasets when using correlation metric\n",
    "    if prediction <= 0:\n",
    "        prediction = 1   \n",
    "    elif prediction >10:\n",
    "        prediction = 10\n",
    "\n",
    "    print '\\nPredicted rating for user {0} -> item {1}: {2}'.format(user_id,item_id,prediction)      \n",
    "    \n",
    "    return prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Predicted rating for user 11676 -> item 0001056107: 1\n"
     ]
    }
   ],
   "source": [
    "prediction = predict_itembased(11676,'0001056107',ratings_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "@contextmanager\n",
    "def suppress_stdout():\n",
    "    with open(os.devnull, \"w\") as devnull:\n",
    "        old_stdout = sys.stdout\n",
    "        sys.stdout = devnull\n",
    "        try:  \n",
    "            yield\n",
    "        finally:\n",
    "            sys.stdout = old_stdout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#This function utilizes above functions to recommend items for item/user based approach and cosine/correlation. \n",
    "#Recommendations are made if the predicted rating for an item is >= to 6,and the items have not been rated already\n",
    "def recommendItem(user_id, ratings, metric=metric):    \n",
    "    if (user_id not in ratings.index.values) or type(user_id) is not int:\n",
    "        print \"User id should be a valid integer from this list :\\n\\n {} \".format(re.sub('[\\[\\]]', '', np.array_str(ratings_matrix.index.values)))\n",
    "    else:    \n",
    "        ids = ['Item-based (correlation)','Item-based (cosine)','User-based (correlation)','User-based (cosine)']\n",
    "        select = widgets.Dropdown(options=ids, value=ids[0],description='Select approach', width='1000px')\n",
    "        def on_change(change):\n",
    "            clear_output(wait=True)\n",
    "            prediction = []            \n",
    "            if change['type'] == 'change' and change['name'] == 'value':            \n",
    "                if (select.value == 'Item-based (correlation)') | (select.value == 'User-based (correlation)') :\n",
    "                    metric = 'correlation'\n",
    "                else:                       \n",
    "                    metric = 'cosine'   \n",
    "                with suppress_stdout():\n",
    "                    if (select.value == 'Item-based (correlation)') | (select.value == 'Item-based (cosine)'):\n",
    "                        for i in range(ratings.shape[1]):\n",
    "                            if (ratings[str(ratings.columns[i])][user_id] !=0): #not rated already\n",
    "                                prediction.append(predict_itembased(user_id, str(ratings.columns[i]) ,ratings, metric))\n",
    "                            else:                    \n",
    "                                prediction.append(-1) #for already rated items\n",
    "                    else:\n",
    "                        for i in range(ratings.shape[1]):\n",
    "                            if (ratings[str(ratings.columns[i])][user_id] !=0): #not rated already\n",
    "                                prediction.append(predict_userbased(user_id, str(ratings.columns[i]) ,ratings, metric))\n",
    "                            else:                    \n",
    "                                prediction.append(-1) #for already rated items\n",
    "                prediction = pd.Series(prediction)\n",
    "                prediction = prediction.sort_values(ascending=False)\n",
    "                recommended = prediction[:10]\n",
    "                print \"As per {0} approach....Following books are recommended...\".format(select.value)\n",
    "                for i in range(len(recommended)):\n",
    "                     print \"{0}. {1}\".format(i+1,books.bookTitle[recommended.index[i]].encode('utf-8'))                        \n",
    "        select.observe(on_change)\n",
    "        display(select)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User id should be a valid integer from this list :\n",
      "\n",
      "   2033   2110   2276   4017   4385   5582   6242   6251   6543   6575\n",
      "   7286   7346   8067   8245   8681   8890  10560  11676  11993  12538\n",
      "  12824  12982  13552  13850  14422  15408  15418  16634  16795  16966\n",
      "  17950  19085  21014  23768  23872  23902  25409  25601  25981  26535\n",
      "  26544  26583  28591  28634  29259  30276  30511  30711  30735  30810\n",
      "  31315  31556  31826  32773  33145  35433  35836  35857  35859  36299\n",
      "  36554  36606  36609  36836  36907  37644  37712  37950  38023  38273\n",
      "  38281  39281  39467  40889  40943  43246  43910  46398  47316  48025\n",
      "  48494  49144  49889  51883  52199  52350  52584  52614  52917  53220\n",
      "  55187  55490  55492  56271  56399  56447  56554  56959  59172  60244\n",
      "  60337  60707  63714  63956  65258  66942  67840  68555  69078  69389\n",
      "  69697  70415  70594  70666  72352  73681  75591  75819  76151  76223\n",
      "  76499  76626  78553  78783  78834  78973  79441  81492  81560  83287\n",
      "  83637  83671  85526  85656  86189  86947  87141  87555  88283  88677\n",
      "  88693  88733  89602  91113  92652  92810  93047  93363  93629  94242\n",
      "  94347  94853  94951  95010  95359  95902  95932  96448  97754  97874\n",
      "  98391  98758 100459 100906 101209 101606 101851 102359 102647 102702\n",
      " 102967 104399 104636 105028 105517 105979 106007 107784 107951 109574\n",
      " 109901 109955 110483 110912 110934 110973 112001 113270 113519 114368\n",
      " 114868 114988 115002 115003 116599 117384 120565 122429 122793 123094\n",
      " 123608 123883 123981 125519 125774 126492 126736 127200 127359 128835\n",
      " 129074 129716 129851 130554 130571 132492 132836 133747 134434 135149\n",
      " 135265 136010 136139 136348 136382 138578 138844 140000 140358 141902\n",
      " 142524 143175 143253 143415 145449 146113 146348 147847 148199 148258\n",
      " 148744 148966 149907 149908 150979 153662 156150 156269 156300 156467\n",
      " 157247 157273 158226 158295 158433 159506 160295 162052 162639 162738\n",
      " 163759 163761 163804 163973 164096 164323 164533 164828 164905 165308\n",
      " 165319 165758 166123 166596 168047 168245 169682 170513 170634 171118\n",
      " 172030 172742 172888 173291 173415 174304 174892 177072 177432 177458\n",
      " 178522 179718 179978 180378 180651 181176 182085 182086 182993 183958\n",
      " 183995 184299 184532 185233 185384 187145 187256 187517 189139 189334\n",
      " 189835 189973 190708 190925 193458 193560 193898 194600 196077 196160\n",
      " 196502 197659 199416 200226 201290 203240 204864 205735 205943 206534\n",
      " 207782 208406 208671 209516 210485 211426 211919 212965 214786 216012\n",
      " 216444 216683 217106 217318 217740 218552 218608 219546 219683 222204\n",
      " 222296 223087 223501 224349 224525 224646 224764 225087 225199 225232\n",
      " 225595 225763 226965 227250 227447 227520 227705 229011 229329 229551\n",
      " 229741 230522 231210 232131 232945 233911 234359 234828 235105 235282\n",
      " 235935 236058 236283 236340 236757 236948 239584 239594 240144 240403\n",
      " 240543 240567 240568 241198 241666 241980 242006 242083 242409 242465\n",
      " 244627 244685 245410 245827 246311 247429 247447 248718 249894 250405\n",
      " 250709 251394 251843 251844 252695 252820 254206 254465 254899 255489\n",
      " 257204 258152 258185 258534 261105 261829 262998 264031 264082 264321\n",
      " 264525 265115 265313 265889 266056 266226 268110 268300 268932 269566\n",
      " 270713 271448 271705 273113 274061 274301 275970 277427 278418 \n"
     ]
    }
   ],
   "source": [
    "#checking for incorrect entries\n",
    "recommendItem(999999,ratings_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As per Item-based (cosine) approach....Following books are recommended...\n",
      "1. My Wicked Wicked Ways\n",
      "2. Fair Peril\n",
      "3. Wolfpointe\n",
      "4. A Nest of Ninnies\n",
      "5. A Bitter Legacy\n",
      "6. A Hymn Before Battle\n",
      "7. Thomas the Rhymer\n",
      "8. Gatherer of Clouds (Initiate Brother Duology)\n",
      "9. Wege zum Ruhm: 13 Hilfestellungen fÃ¼r junge KÃ¼nstler und 1 Warnung\n",
      "10. Love In Bloom's\n"
     ]
    }
   ],
   "source": [
    "recommendItem(4385, ratings_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As per User-based (correlation) approach....Following books are recommended...\n",
      "1. The Gift\n",
      "2. A Close Run Thing : A Novel of Wellington's Army of 1815\n",
      "3. The Romantic: A Novel\n",
      "4. Mazurka for Two Dead Men\n",
      "5. The Titanic Conspiracy: Cover-Ups and Mysteries of the World's Most Famous Sea Disaster\n",
      "6. And Never Let Her Go : Thomas Capano: The Deadly Seducer\n",
      "7. Chop Wood, Carry Water: A Guide to Finding Spiritual Fulfillment in Everyday Life\n",
      "8. WHO NEEDS GOD\n",
      "9. Lords of the White Castle\n",
      "10. Prince Charming Isn't Coming: How Women Get Smart About Money\n"
     ]
    }
   ],
   "source": [
    "recommendItem(4385, ratings_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Thanks for reading this notebook**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}