{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "view-in-github"
},
"source": [
"![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8abwZImbXkN9"
},
"source": [
"This notebook covers material from Chapters 3 of Géron, and builds on the original notebooks made [available on _Github_](https://github.com/ageron/handson-ml2)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1KBWUKJQoRbu"
},
"source": [
"# (Exercises) Classification"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_2m3XdPZoRbx"
},
"source": [
"## Notebook Setup\n",
"The following cells will set up the libraries we need to run our classification tasks, as well as load the data we need to run a classifier on the MNIST handwritten digit database."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "0xzMrm7coRby"
},
"outputs": [],
"source": [
"#@title Run This cell to import common modules. Double click on this if you want to take a look at the code 😀\n",
"'''\n",
"First, let's import a few common modules, ensure MatplotLib plots figures inline\n",
"and prepare a function to save the figures. We also check that Python 3.5 or \n",
"later is installed (although Python 2.x may work, it is deprecated so we \n",
"strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20.\n",
"\n",
"You don't need to worry about understanding everything that is written \n",
"in this section.\n",
"''';\n",
"\n",
"# Python ≥3.5 is required\n",
"import sys\n",
"assert sys.version_info >= (3, 5)\n",
"\n",
"# Is this notebook running on Colab or Kaggle?\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"\n",
"# Scikit-Learn ≥0.20 is required\n",
"import sklearn\n",
"assert sklearn.__version__ >= \"0.20\"\n",
"\n",
"# Common imports\n",
"import numpy as np\n",
"import os\n",
"\n",
"# to make this notebook's output stable across runs\n",
"rnd_seed = 42\n",
"rnd_gen = np.random.default_rng(rnd_seed)\n",
"\n",
"# To plot pretty figures\n",
"%matplotlib inline\n",
"import matplotlib as mpl\n",
"import matplotlib.pyplot as plt\n",
"mpl.rc('axes', labelsize=14)\n",
"mpl.rc('xtick', labelsize=12)\n",
"mpl.rc('ytick', labelsize=12)\n",
"\n",
"# Where to save the figures\n",
"PROJECT_ROOT_DIR = \".\"\n",
"CHAPTER_ID = \"classification\"\n",
"IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
"os.makedirs(IMAGES_PATH, exist_ok=True)\n",
"\n",
"def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
" path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
" print(\"Saving figure\", fig_id)\n",
" if tight_layout:\n",
" plt.tight_layout()\n",
" plt.savefig(path, format=fig_extension, dpi=resolution)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KbNW5dKWMIYn"
},
"source": [
"We need to load the MNIST handwritten digit dataset from OpenML. We won't be loading it as a _Pandas dataframe_, but will instead use the _Dictionary / ndrray_ representation.\n",
"\n",
"You can read more about the MNIST handwritten digit dataset [here](http://yann.lecun.com/exdb/mnist/)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ar25DAokMIgq"
},
"outputs": [],
"source": [
"# Import the function used to fetch open ML datasets from scikit-learn\n",
"from sklearn.datasets import fetch_openml\n",
"\n",
"# Fetch the dataset using the imported function\n",
"mnist = fetch_openml('mnist_784', # The identifier for the dataset \n",
" version=1, # Which version to use\n",
" as_frame=False) #Should we load it as a dataframe?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "UFIMJKXP1Mjn"
},
"outputs": [],
"source": [
"# We'll load the data (i.e., the handwritten digit images) into variable X, and \n",
"# the digit values (i.e., the corresponding numbers in the range 0 - 9) into y\n",
"X = mnist['data']\n",
"y = mnist['target'].astype(np.uint8)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ADLkHNo6M9fr"
},
"source": [
"Originally, Géron uses the entire MNIST dataset, which can take too long when training several models. As a result, we will generate and use a smaller set of data, taking care to balance it. \\\n",
"\n",
"Double click the cell below to inspect it - otherwise, you can run it and the balanced input data will be stored in x_bal, while the corresponding target data will be stored in bal_y."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "GWXIhYT7K1rg"
},
"outputs": [],
"source": [
"#@title Data Rebalancing. Double click to see more!\n",
"\n",
"# In this cell we'll be generating a smaller balanced dataset for faster \n",
"# computations. By default, we'll choose 1000 samples of each digit in the dataset\n",
"\n",
"# Here, we define how many digits within each class we'll keep\n",
"balanced_size = 1000\n",
"\n",
"# Let's define some variable names as placeholders. If the variables aren't \n",
"# initiated, we'll fully overwrite the variable with the balanced_sized list\n",
"# for the digit. Otherwise, we'll append the list.\n",
"bal_X = None\n",
"bal_y = None\n",
"\n",
"# Looping through digit types, i.e.: 0,1,2,3,4,5,6,7,8,9\n",
"for digit in np.arange(0,10):\n",
" \n",
" # find the indices where the targets in _y_ are the digit of interest\n",
" y_idxs = (y == digit)\n",
" print(f'{digit} had {y_idxs.sum()} instances in the dataset. ', end='')\n",
" \n",
" # rnd_gen.choice picks n = balanced_size indices from the set of digit images \n",
" # available. We know the truth is an array with the same number of rows\n",
" # as the subset, full of the current digit - we just need to make an array\n",
" # full of the digit. \n",
" X_subset = X[y_idxs][rnd_gen.choice(np.arange(y_idxs.sum()),(balanced_size,))]\n",
" y_subset = np.full(X_subset.shape[0],digit)\n",
"\n",
" # if bal_X is None, we'll overwrite bal_x and bal_y with the subsets.\n",
" # Otherwise, we'll append stack the X/y subsets with bal_X/bal_y\n",
" if bal_X is None:\n",
" bal_X = X_subset\n",
" bal_y = y_subset\n",
" else:\n",
" bal_X = np.vstack([bal_X, X_subset])\n",
" bal_y = np.hstack((bal_y,y_subset))\n",
" \n",
" print(f'Digit {digit} has been rebalanced.')\n",
"\n",
"# Shuffling the balanced dataset\n",
"shuffler = rnd_gen.permutation(len(bal_X))\n",
"bal_X = bal_X[shuffler]\n",
"bal_y = bal_y[shuffler]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pJWR-DAQKeHE"
},
"outputs": [],
"source": [
"# Splitting the dataset into train and test\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"# set % of data to be used for testing\n",
"test_size=.10\n",
"\n",
"# and split the dataset accordingly\n",
"X_train, X_test, y_train, y_test = train_test_split(\n",
" bal_X, # the source of X data \n",
" bal_y, # the source of y data\n",
" test_size = test_size, # the percent of data to retain as a test\n",
" random_state = rnd_seed ) # set the random seed - consistent runs are important!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BN84I4gL6ByR"
},
"outputs": [],
"source": [
"# Test that the label and image match. We're using np.random instead of our \n",
"# rnd_gen since we don't want the runs of this snippet to vary the results in\n",
"# other snippets :) Run this as many times as you want!\n",
"test = np.random.randint(0,len(bal_X))\n",
"plt.imshow(bal_X[test].reshape((28,28)), cmap='Greys')\n",
"print(f'The image should be of a(n) {bal_y[test]}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wO6vNK6eoRcb"
},
"source": [
"## Exercise 1 - An MNIST Classifier with 97% accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FoKVgpBYZCNu"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JrOUuGWbZIKx"
},
"source": [
"**Caption**: So many digits! 🔢 Can you classify them all?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "q9UOfRIKoRcc"
},
"source": [
"For this exercise, we will look for the model with the best performance using Scikit's [`Grid Search`](https://duckduckgo.com/?q=gridsearchCV), which performs a [\"round robin\"](https://en.wikipedia.org/wiki/Round-robin_tournament) search on a set of possible hyperparameters to find the model with the best performance.\n",
"\n",
"Search for Scikit's [`K-neighbors Classifier`](https://duckduckgo.com/?q=scikit+kneighbors) to find the documentation page. You will find a list of parameters that users can use when making the classifier.\n",
"\n",
"Here is the list of tasks for this exercise.\n",
"\n",
"1. Define a list of dictionaries with the set of hyperparameters that are to be tested: \n",
"\n",
" \n",
" | Hyper-Parameter | \n",
" Values to Test | \n",
"
\n",
" \n",
" | Weights | \n",
" uniform | \n",
" distance | \n",
"
\n",
" \n",
" | # of Neighbors | \n",
" 3 | \n",
" 4 | \n",
" 5 | \n",
" 6 | \n",
"
\n",
"
\n",
"\n",
"2. Instantiate the KNeighborsClassifier Model\n",
"\n",
"3. Fit the models to your training data\n",
"\n",
"4. Extract the optimal model parameters\n",
"\n",
"5. Evaluate your best model\n",
"\n",
"
Let's begin!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a_b1g55_oRcc"
},
"outputs": [],
"source": [
"# Begin by importing the KNeighersClassifier and GridSearchCV from scikit.\n",
"from sklearn.neighbors import KNeighborsClassifier\n",
"from sklearn.model_selection import GridSearchCV"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d0G-qizZNaWB"
},
"source": [
"## **Q1) Set up the parameter dictionary of your [`GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html)** "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "kv_90S_7-yG9"
},
"outputs": [],
"source": [
"#@title Hint: Using grid_search\n",
"'''\n",
"Start by reading the documentation for the `param_grid` parameter of \n",
"`GridSearchCV` (Grid Search with Cross-Validation) at the URL below\n",
"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html\n",
"\n",
"In essence, here you'll be setting up the parameter grid using the information\n",
"from the table in point 1) in the list of tasks for this section.\n",
"''';"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "IZjW2OdLNagU"
},
"outputs": [],
"source": [
"# Fill in the blanks in the code below\n",
"param_grid = {'_______':['_______', '_______'], # 1st hyperparameter and the values it can take\n",
" '_______':[_______, _______, _______, _______] # 2nd hyperparameter and the values it can take\n",
" }"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CWiDG9byNuqB"
},
"source": [
"## **Q2) Make an instance of the [KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) model***"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "OrahdF-MA2_J"
},
"outputs": [],
"source": [
"#@title Hint: Reminder of model instantiation\n",
"'''\n",
"No worries if you forgot what instantiating a class means.\n",
"Essentially, classes are like the blueprints for a house - they say how things\n",
"should be built, but you can't quite live in them! Similarly, a class defines \n",
"the attributes and methods associated with a specific type of object, and when\n",
"we say that we _instantiate_ a class what we mean is that we'll create an \n",
"instance of an object that follows the \"blueprint\" laid out by the class.\n",
"\n",
"Most often, all you do is class the class object and feed it whatever arguments\n",
"it needs to create the object we're interested in. If you read the documentation \n",
"for the kNeighborsClassifier, you'll see that it doesn't need any keyword \n",
"arguments to define a model, though there are several optional ones you can\n",
"input. For today, create a model without using any arguments in the model class.\n",
"''';"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "S9497-2HNvHW"
},
"outputs": [],
"source": [
"# Instantiate the imported model class\n",
"knn_clf = ________()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RIFSOGcsN-J_"
},
"source": [
"## **Q3) Use GridSearch's `fit` method to train your models.***\n",
"\n",
"We will define a grid search using the `knn_clf` model and `param_grid`. \n",
"We will set the number of **folds for the cross-validation to 6**, and use the most verbose setting to get the best idea of how well each cv run performs. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "ZN2njQyyYTln"
},
"outputs": [],
"source": [
"#@title Hints: Running a grid search with Scikit-Learn\n",
"'''\n",
"You can find the documentation for the sklearn implementation of grid searching\n",
"at the hyperlink below:\n",
"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html\n",
"\n",
"When you use GridSearchCV, you'll have to instantiate it and tell it the model\n",
"you'll be searching with (e.g., a logistic regression classifier), a dictionary\n",
"with the hyperparameters and hyperparameter values that you want to search \n",
"through (see example 1 below), and the number of folds to use for \n",
"cross-validation.\n",
"\n",
"Example 1: with the following dictionary:\n",
"{'solver':['newton-cg','lbfgs', 'liblinear']\n",
" 'class_weight':[{0:0.2,1:0.8}, \n",
" {0:0.5,1:0.5},\n",
" {0:0.8,1:0.2}] } \n",
"would try each of 3 solvers with each of the 3 different class weight schedules\n",
"''';"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "WUBwdWFcYSyd"
},
"outputs": [],
"source": [
"#@title Hint: Help with the .fit() method\n",
"'''\n",
"You can find the documentation for the `fit()` method is \n",
"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV.fit)\n",
"\n",
"In scikit's machine learning API, every model implements a '.fit()' method that\n",
"contains the subroutine needed to train the model using the data. As a general\n",
"rule, what the model expects is input data and -for supervised learning- target\n",
"data. \n",
"''';"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "sXxafCooN9z6"
},
"outputs": [],
"source": [
"# We define a grid search using the knn_clf model and param grid. We will set\n",
"# the number of folds for the cross-validation to 6, and use the most verbose\n",
"# setting to get the best idea of how well each cv run performs.\n",
"\n",
"\n",
"# Let's instantiate the scikit class used to do grid searches\n",
"grid_search = _______( \n",
" _______, # The model we're going to use in the search\n",
" _______, # The method dictionary of parameters we'll iterate through in the search\n",
" cv = _______, # The number of folds used to cross-validate\n",
" verbose = 3) #Let's set the verbosity to the highest level: 3"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "22P-spjDVtXG"
},
"outputs": [],
"source": [
"# Complete the code\n",
"grid_search.fit(_______,_______)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mS1jFsIAy2JK"
},
"source": [
"Now that the models have been trained, let's figure out which one is the best one. \\\n",
"\n",
"## **Q4) Fetch and print the optimal parameters and best score found with the grid search*** \n",
"🔍"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "7zUMUp4zFyE9"
},
"outputs": [],
"source": [
"#@title Hint: GridSearch Attributes\n",
"'''\n",
"GridSearch essentially trains as many models as there are combinations of \n",
"parameters in our search space. As it does so, it uses whatever score we \n",
"specified (or the default score provided by the model, e.g.: Mean Square Error)\n",
"to determine if it's the best model it has found. If so, it stores it as the \n",
"best model.\n",
"\n",
"It stands to reason, then, that both the best score and best model are stored\n",
"as attributes in the search. You can access these using the best_score_ and \n",
"best_params_ attributes\n",
"''';"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "nFNiFrmwTEco"
},
"outputs": [],
"source": [
"#Store the best parameters and best scores for the models\n",
"best_parameters = grid_search._______\n",
"best_score = grid_search._______\n",
"\n",
"# And print them out. Try to make them nice!\n",
"print(f'_______{_______}',\n",
" f'_______{_______}',\n",
" sep='\\n')\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9L_cUs03eOSJ"
},
"source": [
"\n",
"Now it's time to test your model! Let's import the [accuracy score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html?highlight=accuracy%20score#sklearn.metrics.accuracy_score) metric from scikit and see how our model fares."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-q90zRHpEpeW"
},
"source": [
"## **Q5) Use the grid_search's to predict the targets from the test dataset.** \n",
"\n",
"(we'll use this to evaluate the accuracy of your model!)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "KTnDHdWHIlq3"
},
"outputs": [],
"source": [
"#@title Hint: Importing scikit-learn metrics\n",
"'''\n",
"There are a myriad of ways to measure how well your model is working. Luckily\n",
"for us, the developers contributing to scikit-learn have implemented a large \n",
"number of them in their metrics package - check out the documentation at the\n",
"link below:\n",
"https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics\n",
"\n",
"We're working on a classification problem, so we need a metric suited for\n",
"evaluating a classification algorithm. A simple and often used metrics is the\n",
"accuracy score, which is simply the number of samples we classified correctly\n",
"divided by the total number of samples:\n",
"\n",
"accuracy = true_positives / positives\n",
"\n",
"There are many other metrics we could use! Check out some options in this\n",
"wikipedia page:\n",
"https://en.wikipedia.org/wiki/Precision_and_recall#Definition_(classification_context)\n",
"\n",
"When importing a metric from scikit-learn, we can simply write, e.g.:\n",
"from sklearn.metrics import log_loss\n",
"''';"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "6Ggdg3IIIUZS"
},
"outputs": [],
"source": [
"#@title Hint: Predicting using a trained scikit-learn model\n",
"\n",
"'''\n",
"The documentation for the `predict` method is found at the link below:\n",
"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV.predict\n",
"''';"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "t6ma4S4XoRcd"
},
"outputs": [],
"source": [
"#Import the accuracy metric\n",
"from sklearn._______ import _______"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BHNXSm4zUeVD"
},
"outputs": [],
"source": [
"#generate predictions from the best model\n",
"y_pred = grid_search.predict(______)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "igZNSwBwLTE9"
},
"outputs": [],
"source": [
"#calculate the accuracy score using the imported metric\n",
"accuracy = _____(_____, _____)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3UysftuILT0i"
},
"outputs": [],
"source": [
"# And print out the accuracy of the model\n",
"print(f'The accuracy of the model is {accuracy:.1%}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YQQAlPxQX_Ar"
},
"source": [
"Congratulations! If you've done everything right, you should have a model with an accuracy of 96.9% We're so close to 97% that we can _taste_ it.\n",
"🍲"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "A2iohebZoRcd"
},
"source": [
"Let's see if we can bridge the gap with a bit of [data augmentation](https://en.wikipedia.org/wiki/Data_augmentation), working off of the smaller mnist dataset we made in the previous part of the exercise. The type of augmentation we'll be using is _shifting_, in which the images in the dataset are shifted up/down/left/right by a number of pixels to make \"new\" datapoints. \n",
"\n",
"List of Tasks:\n",
"\n",
"6. Augment the digit dataset using shifted versions of the images\n",
"\n",
"7. Train a KneighborsClassifier model on the augmented data set using the best parameters found in Exercise 1\n",
"\n",
"8. Evaluate the accuracy of the model trained with the augmented data\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SrAeGb3o3KSj"
},
"source": [
"Let's start by importing the shift tool from SciPy. You can find the documentation for the shift function [here](https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.shift.html)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6GjQXb4IoRcd"
},
"outputs": [],
"source": [
"# We will import shift form the scipy multi-dimensional image processing tools\n",
"from scipy.ndimage.interpolation import shift"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AkjCBR6v3oAw"
},
"source": [
"And we'll define a function that shifts an image from out dataset up & down and/or left & right."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rkb749Aj3mv9"
},
"outputs": [],
"source": [
"# Using shift, we will write a function that shifts a single image left or right\n",
"def shift_image(image, dx, dy):\n",
" image = image.reshape((28, 28))\n",
" shifted_image = shift(image, [dy, dx], cval=0, mode=\"constant\")\n",
" return shifted_image.reshape([-1])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5IdkUVud5OHd"
},
"source": [
"Let's go ahead and verify that the shift_image function works as intended... That way we won't run into issues further down the line.\n",
"\n",
"## **Q6) Verify that the shift function is working as intended and plot a down-shifted and left-shifted data sample.***"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "_Z0iW1pnlNhA"
},
"outputs": [],
"source": [
"#@title Hint: Default shift directions\n",
"'''\n",
"When you think of shifts left/right and up/down, you may be used to thinking in\n",
"cartesian coordinates. That is, a shift right is a positive change in x and a \n",
"shift left is a negative change in x. This is also true in most programming \n",
"applications.\n",
"\n",
"HOWEVER, when it comes to images in programming, a shift down is associated with\n",
"an _increase_ in the y value. This is the case with scipy's shift function. \n",
"''';"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "O6TcRwXalEyN"
},
"outputs": [],
"source": [
"# Let's choose a random image from out training dataset\n",
"rnd_id = rnd_gen.integers(0,\n",
" _______.shape[0]) # dataset we're sampling\n",
"\n",
"# And load the image associated with rnd_id from the dataset\n",
"image = _______[_______]\n",
"\n",
"# Now, let's grab the same image as below and shift it down 5 pixels\n",
"shifted_image_down = shift_image(image, \n",
" _______, # dx\n",
" _______) # dy\n",
"\n",
"# and shift the original image left five pixels\n",
"shifted_image_left = shift_image(image, \n",
" _______, # dx \n",
" _______) # dy"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "lgjNMMlebPuY"
},
"outputs": [],
"source": [
"#@title Run this cell to plot the images we made in the cell above. You can always doubleclick it if you want to look at the source code 😀\n",
"# And then create a matplotlib figure where we'll plot the image\n",
"fig, axes = plt.subplots(1, # one row\n",
" 3, # three columns\n",
" figsize=(12,3))\n",
"\n",
"# Plot the original\n",
"axes[0].set_title(\"Original\", fontsize=14)\n",
"axes[0].imshow(image.reshape(28, 28), # turn the flattened data back into a 28x28 image\n",
" interpolation=\"nearest\",\n",
" cmap=\"Greys\");\n",
"\n",
"# Plot the down-shifted\n",
"axes[1].set_title(\"Shifted down\", fontsize=14)\n",
"axes[1].imshow(shifted_image_down.reshape(28, 28), \n",
" interpolation=\"nearest\", \n",
" cmap=\"Greys\");\n",
"\n",
"# Plot the left-shifted\n",
"axes[2].set_title(\"Shifted left\", fontsize=14)\n",
"axes[2].imshow(shifted_image_left.reshape(28, 28), \n",
" interpolation=\"nearest\", \n",
" cmap=\"Greys\");"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BJBzdbGtmn2Y"
},
"source": [
"Now that we've verified that the shift function works, let's go ahead and use it to augment our training dataset.\n",
"\n",
"## Q7) Augment the training dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "neHmqABXoRce"
},
"outputs": [],
"source": [
"# transform the datasets into lists of images. This will make it easy to \n",
"# iterate & append the shifted images\n",
"X_train_augmented = [image for image in _____]\n",
"y_train_augmented = [label for label in _____]\n",
"\n",
"print(f'We have {len(X_train_augmented)} samples originally')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BFd4gYjipUNB"
},
"outputs": [],
"source": [
"# Make a tuple of shifts to apply to the datasets as augmentation.\n",
"# a shift of 1 pixel right and 1 up is recommended for the first shift tuple,\n",
"# and a shift of 1 pixel left, and 1 pixel down are recommended for the second tuple.\n",
"shifts = ((_______,_______),\n",
" (_______,_______))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zq4MxjBMqSGQ"
},
"outputs": [],
"source": [
"# Let's iterate through our shifts. Remember that each shift includes a value\n",
"# for dx and dy\n",
"for _____, _____ in shifts:\n",
"\n",
" # For each shift defined, we'll iterate through the dataset to shift each \n",
" # image\n",
" for image, label in zip(X_train, y_train):\n",
" X_train_augmented.append(shift_image(image, dx, dy))\n",
" y_train_augmented.append(label)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "C4-3qupwsoY8"
},
"outputs": [],
"source": [
"# Transform the lists back into numpy arrays\n",
"X_train_augmented = np._______(_______)\n",
"y_train_augmented = np._______(_______)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-vhErfvAF8TY"
},
"outputs": [],
"source": [
"#Let's print out the number of samples we have after augmenting!\n",
"print(f'We have {len(_______)} samples after augmenting')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ptjjBy3ZGLci"
},
"source": [
"If everything is working as intended, you shoudl have 27 000 samples. Thats *a lot* of handwritten digits.\n",
"✍ 😖"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "Wkm7rz-htL6z"
},
"outputs": [],
"source": [
"#@title Run this cell to shuffle the augmented data...\n",
"\n",
"# Generate scrambled indices list\n",
"aug_shuffler = rnd_gen.permutation(len(X_train_augmented))\n",
"\n",
"# Shuffle the array using the scrambled indices - that way the input data order\n",
"# matches the target data order\n",
"X_train_augmented = X_train_augmented[aug_shuffler]\n",
"y_train_augmented = y_train_augmented[aug_shuffler]\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "906YYXEKYONa"
},
"source": [
"We now have an augmented dataset! Let's train a model using the best hyperparameters we previously found with our grid search.\n",
"\n",
"## **Q8) Load a new K Nearest Neighbors classifier using the `.best_params_` information***\n",
"\n",
"Hint: The documentation of the [`KNeighborsClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html) is [at this link](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4Wj8wnDvYkMF"
},
"outputs": [],
"source": [
"#Complete the code\n",
"knn_clf = _______( # The model class we're instantiating\n",
" _______ = _______['________'], # Load in our best number of neighbors\n",
" _______ = _______['________') # Load in our best weight type"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kgvouf4lY7W3"
},
"source": [
"## **Q9) Fit the new model using the augmented x and y data***\n",
"\n",
"Hint: The `fit` method of the `KNeighborsClassifier` is documented [at this link](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier.fit)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VPdXnVpjY7hk"
},
"outputs": [],
"source": [
"knn_clf.fit(_______, # The input we're fitting \n",
" _______) # The labels we're fitting"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7i9vguyYZM41"
},
"source": [
"## **Q10) Evaluate the accuracy of the model trained with the augmented data***\n",
"\n",
"Hint: You should have already loaded the `accuracy_score` [here](#accuracy)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8mcDqCEPZTMi"
},
"outputs": [],
"source": [
"# Let's get the predictions from our model\n",
"y_pred = _____.predict(______)\n",
"\n",
"# And use them to calculate the accuracy\n",
"accuracy = accuracy_score(_____, _____)\n",
"\n",
"# We'll finish by printing out our accuracy...\n",
"print(f'The accuracy for our model training on augmented data is {accuracy:.1%}!')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84zPQviCSX7H"
},
"source": [
"Congratulations for finishing the first exercise! If everything went well _and_ you used the exact same arguments we did, you should have gotten an accuracy of 97.5% \\\n",
"\n",
"🥳"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "86ThswH3oRcf"
},
"source": [
"## Exercise 2 - Tackling the Titanic dataset"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kaY2vwt2DPlX"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c0wqHh_yDb95"
},
"source": [
"**Caption**: Can you predict which passengers will survive their ride on the Titanic? 🚣 🧊"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_yy2P3nToRcf"
},
"source": [
"In this exercise we will be attempting to predict whether or not a passenger on the titanic survived or not, based on their attributes (e.g., age, sex, passenger class, where they embarked, and so on). For this exercise, we will be relying on [Pandas](https://en.wikipedia.org/wiki/Pandas_(software)) [DataFrames](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html).\n",
"\n",
"\n",
"Let's take a moment to talk about our dataset! The dataset we'll be using today was part of a machine learning challenge - what this means for _you_ is that today we have training data with labels, but ***our test data has no test labels*** (i.e., we have y_train but not y_test).\n",
"\n",
"Exercise Tasks:\n",
"\n",
"9. Calculate the mean of the age the corresponding standard deviation for the female and male populations in the passenger list. \n",
"\n",
"10. Set up a pipeline for the numerical attributes that implements an \"median\" imputer and a scaler \n",
"\n",
"11. Set up a pipeline for the categorical attributes that implements a \"most frequent\" imputer and a one-hot categorical encoder\n",
"\n",
"12. Prepare the training and testing datasets for the Titanic data. What's the difference between `.fit_transform()` and `.transform()` ?\n",
"\n",
"13. Train a random forest classifier with 100 estimators \n",
"\n",
"14. Make a prediction using the trained random forest classifier. How good is the accuracy? \n",
"\n",
"15. Compare this to a 10-fold cross validation mean.\n",
"\n",
"16. Train a support-vector clustering algorithm and compare its 10-fold cross validation mean to that of the random forest classifier."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Mm_4pFmkoRcf"
},
"source": [
"Let's begin by fetching the data from the internet and loading it into memory. To do this, the code cell defines a `fetch_titanic_data` function that downloads the titanic dataset and a `load_titanic_data` function that let's us load the \"train.csv\" data or \"test.csv\" data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "LwggUBydoRcf"
},
"outputs": [],
"source": [
"#@title `fetch_titanic_data` and `load_titanic_data` definition. Double click here to see the source code.\n",
"# Setup - First, We need to fetch the data from the internet...\n",
"import os\n",
"import urllib.request\n",
"import pandas as pd\n",
"\n",
"\n",
"TITANIC_PATH = os.path.join(PROJECT_ROOT_DIR,\"datasets\", \"titanic\")\n",
"DOWNLOAD_URL = \"https://raw.githubusercontent.com/ageron/handson-ml2/master/datasets/titanic/\"\n",
"\n",
"# Define the function that will download the titanic data... \n",
"def fetch_titanic_data(url=DOWNLOAD_URL, path=TITANIC_PATH):\n",
" if not os.path.isdir(path):\n",
" os.makedirs(path)\n",
" for filename in (\"train.csv\", \"test.csv\"):\n",
" filepath = os.path.join(path, filename)\n",
" if not os.path.isfile(filepath):\n",
" print(\"Downloading\", filename)\n",
" urllib.request.urlretrieve(url + filename, filepath)\n",
"\n",
"# And define the function that loads the data into memory...\n",
"def load_titanic_data(filename, titanic_path=TITANIC_PATH):\n",
" csv_path = os.path.join(titanic_path, filename)\n",
" return pd.read_csv(csv_path)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8X9K-2EakFFr"
},
"source": [
"## **Q11) Fetch and load the titanic training and test data. Then, set the passenger ID as the index.**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yAMH0rymoRcg"
},
"outputs": [],
"source": [
"# Let's start by fetching the titanic data\n",
"_______()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TEkE0_Xqkn_T"
},
"outputs": [],
"source": [
"# And load the training data and test data \n",
"train_data = load_titanic_data(\"_______.csv\")\n",
"test_data = load_titanic_data(\"_______.csv\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jJaUX_uwkqJC"
},
"outputs": [],
"source": [
"# We will explicitely set the PassengerId attribute as the DataFrame index \n",
"# since it is unique to each passenger\n",
"train_data = train_data._______(\"PassengerId\")\n",
"test_data = test_data._______(\"PassengerId\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "s8_6YueUoRcg"
},
"source": [
"The data is already split into a training set and a test set. However, please remember that *the test data* does **not** contain the labels: your goal is to train the best model you can using the training data, then make your predictions on the test data.\n",
"\n",
"Let's take a peek at the training set. Use the `.head()` method to return the first 5 rows of the training dataset. \n",
"\n",
"*If you're on colab, you can instead use Google's interactive data table!*\n",
"\n",
"-----------------------\n",
"\n",
"The dataset should include the following attributes:\n",
"* **PassengerId**: a unique identifier for each passenger\n",
"* **Survived**: that's the target, 0 means the passenger did not survive, while 1 means he/she survived.\n",
"* **Pclass**: passenger class.\n",
"* **Name**, **Sex**, **Age**: self-explanatory\n",
"* **SibSp**: how many siblings & spouses of the passenger aboard the Titanic.\n",
"* **Parch**: how many children & parents of the passenger aboard the Titanic.\n",
"* **Ticket**: ticket id\n",
"* **Fare**: price paid (in pounds)\n",
"* **Cabin**: passenger's cabin number\n",
"* **Embarked**: where the passenger embarked the Titanic. C=Cherbourg, Q=Queenstown, S=Southampton."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ijFp0ONWoRcg"
},
"outputs": [],
"source": [
"# Load Google's data table to display the dataframe interactively\n",
"from google.colab import data_table\n",
"data_table.DataTable(train_data, num_rows_per_page=10)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rhbhBw6wJ1u3"
},
"source": [
"Using DataFrames allow us to easily calculate statistics from the data. Here we will use the .median() and .std() methods to get an picture of the passenger population."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KbsuCAnRbwiC"
},
"source": [
"## **Q12) Calculate the median and standard deviation for each listed sex category**\n",
"\n",
"Hint: The \"pre-filled\" code below uses `pandas`' native [`median`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.median.html) and [`std`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.std.html) functions. If you prefer using [`Numpy`](https://numpy.org/doc/stable/index.html), `Numpy` also has functions for the median and the standard deviation. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "V1rpcp1doRci"
},
"outputs": [],
"source": [
"# Calculate the median and standard deviation for the age of passengers listed\n",
"# as having a female sex\n",
"med_age_f = train_data[train_data[\"_______\"]==\"_______\"][\"_______\"]._______()\n",
"stddev_age_f = train_data[train_data[\"_______\"]==\"_______\"][\"_______\"]._______()\n",
"\n",
"# Calculate the median and standard deviation for the age of passengers listed\n",
"# as having a male sex\n",
"med_age_m = train_data[train_data[\"_______\"]==\"_______\"][\"_______\"]._______()\n",
"stddev_age_m = train_data[train_data[\"_______\"]==\"_______\"][\"_______\"]._______()\n",
"\n",
"print_out=(f'The female population had a median age of {med_age_f:.1f}'\n",
" f' with a standard deviation of {stddev_age_f:.2f} \\n'\n",
" f'The male population had a median age of {med_age_m:.1f}'\n",
" f' with a standard deviation of {stddev_age_m:.2f} \\n')\n",
"\n",
"print(print_out)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6h5nn-XhoRch"
},
"source": [
"If you did everything right so far, you'll find a median age of 27/29 with a standard deviation of 14.11/14.68 for female/male sexes. \n",
"\n",
"It's important that we be aware of the gaps in the data available. In order to do this, let's rely on the DataFrames `.info()` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "T2YZzTZaoRch"
},
"outputs": [],
"source": [
"train_data.info()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fg8WxPoBoRci"
},
"source": [
"Okay, the **Age**, **Cabin** and **Embarked** attributes are sometimes null (less than 891 non-null), especially the **Cabin** (77% are null). We will ignore the **Cabin** for now and focus on the rest. The **Age** attribute has about 19% null values, so we will need to decide what to do with them. One reasonable option is to replace null values with the median age. We could be a bit smarter by predicting the age based on the other columns (for example, the median age is 37 in 1st class, 29 in 2nd class and 24 in 3rd class), but we'll keep things simple and just use the overall median age. \n",
"\n",
"(Note: this will be done in a later step, when we develop the data pipeline)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jdLtswehoRci"
},
"source": [
"The **Name** and **Ticket** attributes may have some value, but they will be a bit tricky to convert into useful numbers that a model can consume. So for now, we will ignore them."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d0UO0IrpoRci"
},
"source": [
"Let's take a look at the numerical attributes:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "E2_crvx0oRci"
},
"outputs": [],
"source": [
"train_data.describe()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KgM5Yvp7oRci"
},
"source": [
"* Less than 50% of the passengers survived (About 38%, to be more precise). That's close enough to 40%, so accuracy will be a reasonable metric to evaluate our model.\n",
"* The mean **Fare** was £32.20, which does not seem so expensive, but the [Bank of England estimates](https://www.bankofengland.co.uk/monetary-policy/inflation/inflation-calculator) the value to be around _£3889_ in 2021.\n",
"* The mean **Age** was a little under 30 years old.\n",
"* Over half of the passengers were in 3rd Class "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_Cs27CWRoRci"
},
"source": [
"We will now verify that the `Survived` class is indeed 0 or 1. \n",
"\n",
"We will also look at the types of values we have in our categorical variables, namely: `Passenger Class`, `Sex`, and `Embarqued`.\n",
"\n",
"Reminder: \n",
"The Embarked attribute tells us where the passenger embarked: C=Cherbourg, Q=Queenstown, S=Southampton."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "yQ5WkkNqoRcj"
},
"outputs": [],
"source": [
"#@title Use this cell to print out the values and counts of the catergorical classes... \n",
"# Initialize the string\n",
"print_out = \"\"\n",
"\n",
"# Define the columns to iterate through \n",
"var_list = [\"Survived\", \"Pclass\", \"Sex\", \"Embarked\"]\n",
"\n",
"# Append the information to our print out text by iterating\n",
"for var in var_list: print_out += f'{train_data[var].value_counts()} \\n\\n'\n",
"\n",
"# and print out the final string\n",
"print(print_out)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "T1AvRyGPoRcj"
},
"source": [
"Now let's build our [data preprocessing pipelines](https://en.wikipedia.org/wiki/Data_pre-processing), starting with the pipeline for numerical attributes.\n",
"In this section we will set up a [Pipeline class](https://duckduckgo.com/?q=sklearn+pipeline), which will fill the missing values in the columns using the mean value (see the [SimpleImputer documentation](https://duckduckgo.com/?q=sklearn+simpleimputer)) and scale the data (see the [StandardScaler documentation](https://duckduckgo.com/?q=sklearn+standardscaler))."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EMYdu1iZduv4"
},
"source": [
"## **Q13) Set up the numerical attribute pipeline using a) an \"imputer\" that uses SimpleImputer with \"median\" set as the strategy and b) a \"scaler\" that uses a default StandardScaler**\n",
"\n",
"Hint: The documentation for the `Pipeline` class is [at this link](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "CSiJdS5M0UYJ"
},
"outputs": [],
"source": [
"# Let's import the classes we'll need from scikit-learn in order to set up\n",
"# our data pipeline.\n",
"from sklearn.pipeline import Pipeline\n",
"from sklearn.impute import SimpleImputer\n",
"from sklearn.preprocessing import StandardScaler\n",
"\n",
"# The Pipeline needs to be initiated with a list containing the name of the\n",
"# preprocessor (e.g., \"imputer\" or \"scaler\").\n",
"num_pipeline = _______([ # We'll instantiate the pipeline class\n",
" (\"_______\", _________(_________=\"________\")), # Which will use the imputer\n",
" (\"_______\", _________() ) ]) # as well as the scaler"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Y5K9Y4IzoRcj"
},
"source": [
"Now we can build the pipeline for the categorical attributes. Here, we will use [one-hot encoding](https://en.wikipedia.org/wiki/One-hot#Machine_learning_and_statistics) using the [OneHotEncoder](https://duckduckgo.com/?q=sklearn+onehotencoder) preprocessor in Scikit.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "v7xDH4G8dsIN"
},
"source": [
"## **Q14) Set up the categorical attribute pipeline using: a) an \"imputer\" that uses [`SimpleImputer`](https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html) with \"most_frequent\" set as the strategy and b) a \"cat_encoder\" that uses a [`OneHotEncoder`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) with the sparse parameter set to `False`**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8cxUS4TzoRck"
},
"outputs": [],
"source": [
"# Let's import the One Hot Encoder function from scikit-learn, and set up our\n",
"# categorical data pipeline\n",
"from sklearn.preprocessing import OneHotEncoder\n",
"\n",
"cat_pipeline = _______( [ # Once again we instantiate the pipeline class\n",
" (\"________\", _________(_________=\"________\")), # And set up an imputer with a strategy\n",
" (\"________\", _________(_________=________)) ] ) # and an encoder with a given sparsity parameter"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mtGqgKGzoRck"
},
"source": [
"Finally, let's join the numerical and categorical pipelines using scikit's [ColumnTransformer](https://duckduckgo.com/?q=sklearn+ColumnTransformer):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hP7U2iztoRck"
},
"outputs": [],
"source": [
"from sklearn.compose import ColumnTransformer\n",
"\n",
"num_attribs = [\"Age\", \"SibSp\", \"Parch\", \"Fare\"]\n",
"cat_attribs = [\"Pclass\", \"Sex\", \"Embarked\"]\n",
"\n",
"preprocess_pipeline = ColumnTransformer( [(\"num\", num_pipeline, num_attribs),\n",
" (\"cat\", cat_pipeline, cat_attribs) ] )"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VxCNJ9q4oRck"
},
"source": [
"We now have a preprocessing pipeline that takes the raw data and outputs numerical input features that we can feed to any Machine Learning model we want, and we can also generate the truth variable from the 'Survived' data.\n",
"\n",
"Let's generate our train and test data.\n",
"\n",
"--------------------\n",
"Note! We will continue using X_train, y_train, X_test, and y_test as the variable names for our training and testing datasets. However, the notebook only has one namespace. Thus, runnng the code below will overwrite the X_train/test and y_train/test variables from exercises 1 & 2. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TqQ5pAxidpIZ"
},
"source": [
"## **Q15) Prepare the training and testing datasets. Think about the difference between `.fit_transform()` and `.transform()`!**\n",
"\n",
"Hint 1: Remember to load \"Survived\" into `y_train`!\n",
"\n",
"Hint 2: Here are the documentation links for the [`transform`](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn.pipeline.Pipeline.transform) and [`fit_transform`](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn.pipeline.Pipeline.fit_transform) methods of the [`Pipeline`](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) class.\n",
"\n",
"Hint 3: Because the dataset is from a Kaggle competition, there is no \"Survived\" category in the test set `test_data` to not give away the answer."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2l2zH2_zoRck"
},
"outputs": [],
"source": [
"# Let's preprocess our datasets!\n",
"X_train = preprocess_pipeline._______( # Fit your pipeline using the training input data and transform the training input data in one step!\n",
" _______[_______ + _______] ) # load the categorical and numerical attributes from the training dataframe\n",
"\n",
"X_test = preprocess_pipeline._______( # Transform the testing input data and transform the training data in one step!\n",
" _______[_______ + _______] ) # load the categorical and numerical attributes from the testing dataframe\n",
"\n",
"y_train = ________[\"_______\"] # Load whether or not the passengers in the training survived into y_train\n",
"\n",
"# You might be wondering about y_test. Remember, since the data came from a\n",
"# Kaggle competition, we don't actually have y_test! "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "YYPC6lWe4Ozx"
},
"outputs": [],
"source": [
"#@title Let's print a sample to take a loot at the data being produced\n",
"for idx, attrib in np.ndenumerate(np.hstack((num_attribs,cat_attribs))):\n",
" print(f\"{attrib} has a value of {X_train[0][idx]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PlOJtfUg5Dr2"
},
"source": [
"If you have done everything as we planned out so far, you should see the following:\n",
"\n",
"Age has a value of -0.5657358173332453
\n",
"SibSp has a value of 0.4327933656785018
\n",
"Parch has a value of -0.4736736092984604
\n",
"Fare has a value of -0.5024451714361923
\n",
"Pclass has a value of 0.0
\n",
"Sex has a value of 0.0
\n",
"Embarked has a value of 1.0
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "11NFNA-MoRcl"
},
"source": [
"Now that we've finally prepared the pipeline, we can begin training algorithms!\n",
"Let's start by training a type of classifier called a [`RandomForestClassifier`](https://duckduckgo.com/?q=scikit+randomforestclassifie). This type of a classifier is an _ensemble classifier_, which employs a number of estimators that we have to define. Today, we will use 100 members in our ensemble."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kp6Zc3b2mlJc"
},
"source": [
"## **Q16) Import a RandomForestClassifier from scikit and instantiate it with 100 estimators. Then, fit the model**\n",
"\n",
"Hint: The documentation for the `RandomForestClassifier` is [at this link](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "I2YtFhjJoRcl"
},
"outputs": [],
"source": [
"# Let's import the scikit implementation of random forest classifiers\n",
"from sklearn._______ import _______"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iiZ6J6wE5god"
},
"outputs": [],
"source": [
"# And now let's instantiate the RandomForestClassifier class\n",
"forest_clf = RandomForestClassifier(_______=_______, # Set our number of estimators \n",
" random_state=rnd_seed) # And set our random seed for consistent results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "674EdKAM52-X"
},
"outputs": [],
"source": [
"# We end by fitting the model\n",
"forest_clf._______(_______,_______)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UfCfaWOMoRcl"
},
"source": [
"Great, our model is trained! Let's use it to make predictions on the test set. Let's also check the accuracy of our model on the test dataset.\n",
"\n",
"Wait a second... *we don't have labels for our test dataset!* \n",
"😯\n",
"\n",
"What do we do now? Well, the best we can do is to check how well our model is able to fit our training data. \\\n",
"***Please note that we do this with the very real risk of overfitting our training data!!!***\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iakCUz2s6I6B"
},
"source": [
"## **Q17) Predict a set of answers using the trained model and compare it using the accuracy_score metric. Then, print out a statement reporting the accuracy**\n",
"\n",
"*Hint 1: [Here's the documentation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html) for the `accuracy_score` metric*\n",
"\n",
"*Hint 2: You don't have a set of labels for the test set (i.e., you don't have `y_test`), so you can't measure the accuracy using `X_test`. You do, however, have another dataset with both inputs `X` and labels `y`!*\n",
"\n",
"*Hint 3: Though you won't be able to know if the model is over/underfitting the data if you measure the accuracy on the train set, it **will** give you an idea as to whether the model is able to make accurate predictions from the input data.*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ScTILYjroRcl"
},
"outputs": [],
"source": [
"# Let's import the accuracy score implementation in scikit-learn\n",
"from _______._______ import _______\n",
"\n",
"# Then let's get predictions from our model\n",
"y_pred = _______._______(_______)\n",
"\n",
"# Let's calculate the accuracy...\n",
"accuracy = _______(_______, # We'll need our ground truth\n",
" _______) # and our predictions\n",
"print(f'The accuracy we got on our training dataset is {accuracy:.1%}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SRkCsk6IoRcm"
},
"source": [
"If you did everything like we did during development of the notebook, you should get an accuracy of 98.0%! However, this accuracy is the performance of the model to predict the data that it was trained on. If we make the analogy to a student, *we gave them the list of questions and answers before the test*. Are we surprised that they did well?\n",
"\n",
"While our model gave us a prediction and we were able to compare how accurate our predictions were, we're also interested in knowing how good our model really is. \n",
"\n",
"It's true that now we could just build a CSV file with the predictions based on the test set (respecting the format excepted by Kaggle), then upload it and hope for the best. \n",
"\n",
"***But we can do better than hope.***\n",
"\n",
"Why don't we use cross-validation to have an idea of how good our model is? This means we'll divide the data into several groups of data, and we'll use all but one for training and test on the remaining group. We'll do this as many times as we have groups, which will give use an idea of how well our model is able to predict data it has never seen before.\n",
"\n",
"This schematic from the scikit learn documentation might be useful to help you understand...\n",
"
\n",
"Image Source: Scikit-Learn Cross Validation Documentation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "g2WsNkRqoCBf"
},
"source": [
"## **Q18) Import the cross_val_score function and produce a set of 10-fold cross validation scores using the training data. Then, print the mean score. How does it compare with the answer to Q14?**\n",
"\n",
"Hint: You may consult [the documentation of `cross_val_score`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WMO4Cp0VoRcm"
},
"outputs": [],
"source": [
"# Let's start by importing the cross validation score class from scikit-learn\n",
"from _______._______ import _______\n",
"\n",
"# Let's instantiate the imported class...\n",
"forest_scores = cross_val_score(_______, # Which will need the model to evaluate \n",
" _______, # The input data to use\n",
" _______, # The corresponding target data\n",
" cv=_______) # And the number of folds for evaluation\n",
"\n",
"print(f'The mean score is {_______.mean():.1%}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iRPB-By6oRcm"
},
"source": [
"If you did everything the way we did, you'll get a mean accuracy of 81.4%. \n",
"\n",
"We will now compare to the [leaderboard](https://www.kaggle.com/c/titanic/leaderboard) for the Titanic competition on Kaggle. How does our model compare?\n",
"\n",
"Don't be discouraged by those with 100% accuracy - since you can easily find the [list of victims](https://www.encyclopedia-titanica.org/titanic-victims/) of the Titanic, it seems likely that there was little Machine Learning involved in their performance."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BaM_MPCPoRcm"
},
"source": [
"Let's move on from random forests and try an `SVC`, i.e., a [support-vector clustering algorithm](https://www.jmlr.org/papers/volume2/horn01a/horn01a.pdf). You can find the scikit documentation [here](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GWIdOxoiogP9"
},
"source": [
"## **Q19) Train a support-vector clustering algorithm and generate a set of cross-validated scores**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "A7yzSv1a-4n_"
},
"source": [
"Hint: Don't hesitate to recycle code from before!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "M0OpdmMxoRcm"
},
"outputs": [],
"source": [
"# Let's import the SVC class and instantiate a model\n",
"from sklearn._______ import _______\n",
"svm_clf = _______(gamma=\"auto\")\n",
"\n",
"svm_scores = _______(_______,\n",
" _______,\n",
" _______,\n",
" cv=_______)\n",
"print(f'The mean score is {_______.mean():.1%}')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PSnIqjcJoRcm"
},
"source": [
"If everything went well, this model should look better. Notice though that what we've printed so far is the mean score from the cross validation. While we do appreciate models having higher mean accuracy, consistency is also very important when making predictions! Let's visualize the scores using a box plot (Géron thanking Nevin Yilmaz as the proponents of this idea), where the points will represent the performance on the testing fold in each run. \n",
"\n",
"It's also important to note that the `boxplot()` function detects outliers (called \"fliers\") and does not include them within the whiskers. Specifically, if the lower quartile is Q1 and the upper quartile is Q3 , then the interquartile range IQR=Q3−Q1 (this is the box's height), and any score lower than Q1−1.5×IQR is a flier, and so is any score greater than Q3+1.5×IQR . "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R7fDElKMoRcn"
},
"source": [
"But instead of just looking at the mean accuracy across the 10 cross-validation folds, let's plot all 10 scores for each model, along with a box plot highlighting the lower and upper quartiles, and \"whiskers\" showing the extent of the scores (thanks to Nevin Yilmaz for suggesting this visualization). Note that the `boxplot()` function detects outliers (called \"fliers\") and does not include them within the whiskers. Specifically, if the lower quartile is $Q_1$ and the upper quartile is $Q_3$, then the interquartile range $IQR = Q_3 - Q_1$ (this is the box's height), and any score lower than $Q_1 - 1.5 \\times IQR$ is a flier, and so is any score greater than $Q3 + 1.5 \\times IQR$."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "tdAWlGBIoRcn"
},
"outputs": [],
"source": [
"#@title We'll make a quick boxplot to compare the SVM and Random Forest CV scores\n",
"import matplotlib.pyplot as plt\n",
"\n",
"plt.figure(figsize=(8, 4))\n",
"plt.plot([1]*10, svm_scores, \".\")\n",
"plt.plot([2]*10, forest_scores, \".\")\n",
"plt.boxplot([svm_scores, forest_scores], labels=(\"SVM\",\"Random Forest\"))\n",
"plt.ylabel(\"Accuracy\", fontsize=14)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5n3-x2xFoRcn"
},
"source": [
"If all goes as expected, your plot should look like this:\n",
"
\n",
"\n",
"The random forest classifier should have gotten a very high score on one of the 10 folds, but overall should have a lower mean score and a bigger spread. This suggests that the SVM classifier is more likely to generalize well. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qhhE13nt8E0e"
},
"source": [
"## Bonus Questions\n",
"**Exercise 1 Bonus Objectives:**\n",
"\n",
"1. Train and evaluate the models using precision as the metric\n",
"\n",
"2. Extract & plot instances in which the model was confused \n",
"\n",
"3. Create an imbalanced dataset and study the effect on the model\n",
"\n",
"5. Implement a rotation augmentation strategy\n",
"\n",
"6. Try addressing the imbalance in the dataset using data augmentation\n",
"\n",
"\n",
"**A.)** We trained and tested the model's performance using accuracy as the metric. What happens when you use precision as the scoring metric instead? Note: you'll have to redefine the grid_search scoring parameter\n",
"\n",
"**B.)** Extract some samples that the model made a mistake on and plot them. Can you tell what the digit is?\n",
"\n",
"**C.)** In the setup we reduced the size of the dataset while ensuring that it remained perfectly balanced (♦ ♦ ♦ ♦ ♦♦). What happens to the performance of the model when using an unbalanced dataset? Make one of the digits dominate the dataset (e.g., making over 50% of the dataset one of the digits) and try training the algorithm and test it on the _balanced_ test dataset. \n",
"\n",
"**D.)**We've implemented a shift strategy, but there's a myriad of ways of augmenting your dataset! Try implementing a rotation augmentation strategy using Scipy's ndimage.rotate\n",
"\n",
"**E.)** We've implemented shifts as augmentations in the dataset. In a previous challenge we developed an unbalanced dataset - try addressing the imbalance in the dataset by augmenting the dataset!\n",
"\n",
"**Exercise 2 Bonus Objectives:**\n",
"\n",
"1. Choose two other models and try using them to predict whether or not a passenger would survive. Use grid-search to try out several hyperparameters and determine which model worked best out of the four.\n",
"\n",
"2. Try converting numerical attributes to categorical attributes, such as age group instead of age, or those travelling alone vs with company."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"include_colab_link": true,
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
},
"nav_menu": {},
"toc": {
"navigate_menu": true,
"number_sections": true,
"sideBar": true,
"threshold": 6,
"toc_cell": false,
"toc_section_display": "block",
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 1
}