{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "view-in-github"
},
"source": [
"![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LQmG3MgLh6fC"
},
"source": [
"# Clustering"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ITE51fZYjDwp"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KO7kiUVK5SGi"
},
"source": [
"
\n",
" Photo Credits: Three Assorted-Color Garbage Cans by Hamza Javaid licensed under the Unsplash License\n",
"\n",
"\n",
"*Sorting takes a lot of effort - is there a way that we can get computers to do it automatically for us when we don't even know where to begin?*"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hsXnPPkh5SDp"
},
"source": [
"This notebook will be used in the lab session for week 4 of the course, covers Chapters 9 of Géron, and builds on the [notebooks made available on _Github_](https://github.com/ageron/handson-ml2).\n",
"\n",
"Need a reminder of last week's labs? Click [_here_](https://colab.research.google.com/github/tbeucler/2022_ML_Earth_Env_Sci/blob/main/Lab_Notebooks/Week_3_Decision_Trees_Random_Forests_SVMs.ipynb) to go to notebook for week 3 of the course."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l6_AqYXXCY0c"
},
"source": [
"**Notebook Setup**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SjZzFMhZCYCj"
},
"source": [
"First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FOWGsRI22jOT"
},
"outputs": [],
"source": [
"# Python ≥3.5 is required\n",
"import sys\n",
"assert sys.version_info >= (3, 5)\n",
"\n",
"# Is this notebook running on Colab or Kaggle?\n",
"IS_COLAB = \"google.colab\" in sys.modules\n",
"\n",
"# Scikit-Learn ≥0.20 is required\n",
"import sklearn\n",
"assert sklearn.__version__ >= \"0.20\"\n",
"\n",
"# Common imports\n",
"import numpy as np\n",
"import os\n",
"\n",
"# to make this notebook's output stable across runs\n",
"rnd_seed = 2022\n",
"rnd_gen = np.random.default_rng(rnd_seed)\n",
"\n",
"# To plot pretty figures\n",
"%matplotlib inline\n",
"import matplotlib as mpl\n",
"import matplotlib.pyplot as plt\n",
"mpl.rc('axes', labelsize=14)\n",
"mpl.rc('xtick', labelsize=12)\n",
"mpl.rc('ytick', labelsize=12)\n",
"\n",
"# Where to save the figures\n",
"PROJECT_ROOT_DIR = \".\"\n",
"CHAPTER_ID = \"classification\"\n",
"IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
"os.makedirs(IMAGES_PATH, exist_ok=True)\n",
"\n",
"def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
" path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
" print(\"Saving figure\", fig_id)\n",
" if tight_layout:\n",
" plt.tight_layout()\n",
" plt.savefig(path, format=fig_extension, dpi=resolution)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AVIjgpeDDrJC"
},
"source": [
"**Data Setup**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vJfxDFQBCeVO"
},
"source": [
"We need to load the MNIST dataset from OpenML - we won't be loading it as a Pandas dataframe, but will instead use the Dictionary / ndrray representation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "96ed3Hj8ChTd"
},
"outputs": [],
"source": [
"#Load the mnist dataset\n",
"from sklearn.datasets import fetch_openml\n",
"mnist = fetch_openml('mnist_784', version=1, as_frame=False)\n",
"\n",
"X = mnist['data']\n",
"y = mnist['target'].astype(np.uint8)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tW68lz26CoI6"
},
"source": [
"We're going to subsample the digits in the dataset, choosing a random set of digit classes - we won't even know the number of different digits we will chose; it will be somewhere between 4 and 8!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TxjW4hltD0nL"
},
"outputs": [],
"source": [
"# Generating the random set of digit labels we will use to extract samples from\n",
"# the MNIST handwritten digit dataset.\n",
"\n",
"digits = rnd_gen.choice(np.arange(10), # Digit Possibilities\n",
" int( np.round( rnd_gen.uniform(3.5, 8.5) ) ), # Number of digits to use\n",
" replace = False) # Can't repeat digits"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n4PrwneRD1me"
},
"source": [
"We will learn on a total of 8000 digits, evenly distributed amongst the randomly selected digits in the dataset. Let's store the number of samples to be taken from each class."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TQrrl9j2Iv7t"
},
"outputs": [],
"source": [
"# Let's find a round number of digits to extract for each digit\n",
"num_samples = np.round(8000/len(digits) + 1 ).astype(int)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zbaBrvOIKjyN"
},
"source": [
"With that out of the way, let's generate the dataset!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Wt_XQ9FUCnV7"
},
"outputs": [],
"source": [
"# Placeholder Vars\n",
"sub_X = None\n",
"sub_y = None\n",
"\n",
"# Looping through digit types\n",
"for digit in digits:\n",
" # find indices where target is digit of interest\n",
" y_idxs = y==digit\n",
"\n",
" # rnd_gen.choice chooses n = balanced_size indices from the set of digits\n",
" # available. Since we know the truth is an array with the same number of rows\n",
" # as the subset, full of the current digit\n",
" X_subset = X[y_idxs][rnd_gen.choice(np.arange(y_idxs.sum()),(num_samples,))]\n",
" y_subset = np.full(X_subset.shape[0],digit)\n",
"\n",
" if type(sub_X) == type(None):\n",
" sub_X = X_subset\n",
" sub_y = y_subset\n",
" else:\n",
" sub_X = np.vstack([sub_X, X_subset])\n",
" sub_y = np.hstack((sub_y,y_subset))\n",
"\n",
"# Shuffling the dataset, also limitting the number of digits to 8000 so we can't\n",
"# cheat and tell how many digits there are by looking at the length of the array\n",
"shuffler = rnd_gen.permutation(len(sub_X))\n",
"sub_X = sub_X[shuffler][:8000]\n",
"sub_y = sub_y[shuffler][:8000]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R70HPKpVCgaA"
},
"source": [
"We now have a set of 8000 random samples that we know belong to somewhere between 4 and 8 clusters 😃
Can we divide them into these groups without knowing the labels beforehand?\n",
"\n",
"**Warning**: Don't expect near perfect results this time."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yEwKJDqKmQwW"
},
"source": [
"Clustering with KMeans"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eeR0w4WSmV_K"
},
"source": [
"The first thing we need to do is to import the KMeans model from scikit learn. Let's go ahead and do so.\n",
"\n",
"## Q1) Import the KMeans model from scikit learn.\n",
"\n",
"*Hint: [Here is the documentation](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html) for the Kmeans implementation in sklearn.*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FVRiu373mWYI"
},
"outputs": [],
"source": [
"# Import KMeans class from scikit-learn here\n",
"from _____._____ import _____"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5wuMA-ir7AGG"
},
"source": [
"We don't know how many clusters we should use to split the data. Our first instinct would be to use as many clusters as the number of digits we have, but even that is not necessarily optimal. Why dont we try all K's between 4 and 40?\n",
"\n",
"To do so, we'll need to begin by training a Kmeans algorithm for each value of K we're interested in.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KX2Op57zAZ-P"
},
"source": [
"How long does it take to train a single KMeans model with 10 initial centroid settings? This will give us an idea as to whether it may be a good idea to apply a dimensionality reduction algorithm before fitting our models.\n",
"\n",
"## Q2) Import python's `time` library and measure how long it takes to train a single KMeans model with 3 clusters on the raw data subset.\n",
"\n",
"*Hint 1: [Here is the documetation](https://docs.python.org/3/library/time.html#time.time) for the function used to get timestamps*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GPqc5D1cAY5y"
},
"outputs": [],
"source": [
"# ---------------------------------------------------------------------------------------------------------------\n",
"# Import time to time our training process\n",
"# ---------------------------------------------------------------------------------------------------------------\n",
"import ______\n",
"\n",
"# ---------------------------------------------------------------------------------------------------------------\n",
"# Here we will time the KMeans algorithm very similar to what we did for the first notebook.\n",
"# ---------------------------------------------------------------------------------------------------------------\n",
"t0 = _____._____()\n",
"\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"# We will now train the KMeans model with 3 clusters and rnd_seed for random_state. Time the training process as well.\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"kmeans_test = KMeans(n_clusters=_____, # Number of clusters to split into\n",
" random_state = _____) # Random seed\n",
"kmeans_test.fit(_____) # Fitting to data subset\n",
"t1 = _____._____()\n",
"\n",
"print(f\"Training took {(____ - ____):.2f}s\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7clPTnByCnuD"
},
"source": [
"This should seem like a bit longer than we want. (3.42 seconds during testing of the notebook). Doing this many times means there's a possibility we'll be sitting around doing *nothing*, possiblity for *several minutes*. ***Who has time for this?***\n",
"\n",
"Let's reduce the dataset using PCA, capturing 95% of the variability in the data.\n",
"\n",
"## Q3) Import PCA from scikit and reduce the dimensionality of our input data. 95% of the variance in the data should be captured.\n",
"\n",
"*Hint 1: [Here is the documentation](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) for PCA.*\n",
"\n",
"*Hint 2: `.fit_transform()` will be very useful*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "l7gGFSUiDxGF"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------\n",
"# Import PCA class\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"from sklearn._____ import _____\n",
"\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"# Instantiate PCA, only retain enough components for 95% of variance in data\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"pca = _____(_____)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "75flJdqhBWXE"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------\n",
"# Fit model and transform the sub_X datast\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"reduced_X = pca._____(_____)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "OBUe-CCvGNVN"
},
"source": [
"Let's try training a KMeans model on the reduced dataset and see if our training time improved...\n",
"\n",
"## Q5) Repeat Q2 using the reduced dataset\n",
"*Reminder: We're still splitting the data into 3 clusters*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NIAWAWLwGhdh"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------\n",
"# We will now train the KMeans model with 3 clusters and rnd_seed for random_state. Time the training process as well.\n",
"# The kmeans algorithm should now fit to the reduced_X subset\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"t0 = _____._____()\n",
"kmeans_test = KMeans(n_clusters=_____, # Number of clusters to split into\n",
" random_state = _____) # Random seed\n",
"kmeans_test.fit(_____) # Fitting to reduced data subset\n",
"t1 = _____._____()\n",
"\n",
"print(f\"Training took {(____ - ____):.2f}s\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0H8NeEuXHBNq"
},
"source": [
"## Q6) Train a KMeans model for $\\; 2 \\le k \\le 20$\n",
"\n",
"*Hint 1: Set up a range using python's [`range`](https://docs.python.org/3/library/functions.html#func-range) function or numpy's [`arange`](https://numpy.org/doc/stable/reference/generated/numpy.arange.html)*\n",
"\n",
"*Hint 2: You can store each trained model by appending it to a list as you iterate*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "L10E0hhgBroL"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------\n",
"# Create a list of numbers from 2 to 20 with the range() function\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"k_list = _____\n",
"\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"# Create an empty list to store the fitted models with different k values\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"kmeans_models = _____"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iKajWdb760IG"
},
"outputs": [],
"source": [
"# -----------------------------------------------------------------------------------------------------------------------------------\n",
"# Use a for loop to train and store different kmeans models. What you fill in here should be similar to what you just did from Q1-Q5.\n",
"# ------------------------------------------------------------------------------------------------------------------------------------\n",
"t0 = time.time() # Get a timestamp to keep track of time\n",
"for k in k_list:\n",
" #print out a statement stating which k value you are working on\n",
" t1 = time.time() # Get a current timestamp\n",
" print(f\"\\r Currently working on k={_____}, elapsed time: {(t1 - t0):.2f}s\", end=\"\")\n",
"\n",
" kmeans = ________(_____=k, # Set the number of clusters\n",
" _____= rnd_seed ) # Set the random state\n",
" kmeans._____(_____) # Fit the model to the reduced data subset\n",
"\n",
" kmeans_models._____(_____) # store the model trained to predict k clusters\n",
"\n",
"# -----------------------------------------------------------------------------------------------------------------------------------\n",
"# We spent around 56.5s to train the models\n",
"# ------------------------------------------------------------------------------------------------------------------------------------\n",
"print(f\"\\r Finished training the models! It took: {(time.time() - t0):.2f}s\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7yTraW93IF5z"
},
"source": [
"You should hopefully remember the silhouette score and inertia metrics from the reading. Let's go ahead and make a $k$ vs $silhouette$ plot and a $k$ vs $inertia$ plot."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "I5f4hywgIyFN"
},
"source": [
"## Q7) Import the silhouette score metric and generate a silhouette score value for each model we trained\n",
"\n",
"*Hint 1: [Here is the documentation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html) for the Silhouette score implementation in scikit learn*\n",
"\n",
"*Hint 2: The silhouette score needs the model input data and the model labels as arguments*\n",
"\n",
"*Hint 3: The model labels are stored as an attribute in each model. Check the list of available attributes in the [sklearn KMeans documentation](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html).*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8qmBfe8oIkfF"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------\n",
"# Import silhouette_scores metric\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"from sklearn.________ import ______"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PPKxDVspByPA"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------\n",
"# Use list comprehension to store the silhouette score for all trained models (check the documentaion if you need help\n",
"# setting up in arguments as specified in Hint 2 and 3)\n",
"# --------------------------------------------------------------------------------------------------------------------\n",
"silhouette_scores = [silhouette_score(_______, model._______)\n",
" for model in _____]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pDt_a4PqWQf7"
},
"source": [
"## Q8) Plot comparing $k$ vs Silhouette Score. Highlight the maximum score\n",
"\n",
"*Hint 1: You'll need to find the position of the best score in the silhouette score list. [Here is the documentation ](https://numpy.org/doc/stable/reference/generated/numpy.argmax.html)to a numpy function that would be very useful for this.*\n",
"\n",
"*Hint 2: matplotlib's pyplot has been imported as `plt`. [Here is the documentation](https://matplotlib.org/3.5.0/api/_as_gen/matplotlib.pyplot.subplots.html) to the `subplots()` method.*\n",
"\n",
"*Hint 3: [Here is the documentation](https://matplotlib.org/3.5.0/api/_as_gen/matplotlib.pyplot.plot.html) to the `plot()` method in matplotlib's pyplot. Note that it is also implemented as a method in the `axes` objects created with `plt.subplots()`*\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OkjyAv1jWQme"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Find the index of model with the highest score. We recommend converting the silhouette_scores list to numpy array to use the function in Hint 1\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"best_index = __.______(__._____(______))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "QwYQF4nSB1sY"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Find out which k value the 'best_index' that you just got corresponds to.\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"best_k = _____[best_index] # Get the best K value, per the silhouette score\n",
"\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Store the silhouette score corresponds to 'best_index' here.\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"best_score = silhouette_scores[best_index]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "HXjxgaNGB13n"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Create a k-silhouette score diagram (horizontal axis: K values, vertical axis: silhouette score)\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Make a figure with size (12,4)\n",
"fig, ax = ___.______(____=____)\n",
"\n",
"ax.plot(_______, ______, \"bo-\")\n",
"\n",
"ax.set_xlabel(\"$k$\", fontsize=14)\n",
"ax.set_ylabel(\"Silhouette score\", fontsize=14)\n",
"\n",
"ax.plot(best_k, best_score, \"rs\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "o3TkGqOSjUkd"
},
"source": [
"We got this figure when we ran the code. Does your figure look the same?"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PV5QD2wbjQ3W"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ACdrjMZlz-z-"
},
"source": [
"Now we'll plot the value of $k$ vs inertia\n",
"\n",
"## Q9) Plot comparing $k$ vs Inertia. Highlight the inertia for the model with the highest silhouette score\n",
"\n",
"*Hint 1: If you followed the previous step as it was written, you have the index for the best model stored in*`best_index`.\n",
"\n",
"*Hint 2: [The KMeans documentation](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html#sklearn.cluster.KMeans) details the attribute in which the model's intertia is stored.*\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gEC3lr000RDW"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Iterate through the list of kmeans models and store the model inertias\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"inertias = [model._____ for model in _____]\n",
"\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Get the inertia for the model with the highest silhouette score\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"best_inertia = inertias[____]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LAy0DMLy5KG-"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Create a k-silhouette score diagram (horizontal axis: K values, vertical axis: inertia)\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Make a figure with size (12,4)\n",
"fig, ax = ___.___(____=____)\n",
"\n",
"ax.plot(_______, ______, \"bo-\")\n",
"\n",
"ax.set_xlabel(\"$k$\", fontsize=14)\n",
"ax.set_ylabel(\"Inertia\", fontsize=14)\n",
"\n",
"ax.plot(best_k, best_inertia, \"rs\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hKXBz9qZkVHc"
},
"source": [
"You should see a figure similar to the one below."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "O664MWiAkTke"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p2aDRxJj7Pen"
},
"source": [
"If you ran the notebook with the default random seed, you may be surprised to see that the best performing KMeans model is the one that breaks it off into two clusters! The next best two will be those associated with $k=4$ and $k=5$. Let's get some plots to try to make sense of the results, since we have the actual labels.\n",
"\n",
"There aren't any more questions to answer from here on out, but you *will* have to change the code if you started out from a different random seed! I'll try to be good about pointing out what code you'll need to change.\n",
"\n",
"Let's begin by using [scikit's TSNE implementation](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html) to reduce the dimensionality of the dataset for plotting. This will take a bit of computation time!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Kr24Ye_T8sDZ",
"outputId": "0245f02d-0c02-4334-95f8-e39cfc093063"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.7/dist-packages/sklearn/manifold/_t_sne.py:783: FutureWarning: The default initialization in TSNE will change from 'random' to 'pca' in 1.2.\n",
" FutureWarning,\n"
]
}
],
"source": [
"from sklearn.manifold import TSNE\n",
"\n",
"tsne = TSNE(n_components=2, # We'll project onto 2D plane\n",
" random_state=rnd_seed, # We need a random seed\n",
" learning_rate='auto') #Let the algorithm handle the learning rate\n",
"\n",
"# And now get the input data in 2-component reduced form\n",
"X_plot = tsne.fit_transform(sub_X)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c7BlPG2YdQ1-"
},
"source": [
"Let's continue by making a list of the three best models."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wjo_EZs4-kw6"
},
"outputs": [],
"source": [
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Make an empty list to store the three best models\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"best_models = []\n",
"\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Use the best_index to get the best model\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"best_models.append(kmeans_models[best_index])\n",
"\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------\n",
"# Manually retrieve 2nd and 3rd best models\n",
"# First use np.argsort() to sort all silhouette scores. The second to last and third to last indices correspond to the second and third best models.\n",
"# --------------------------------------------------------------------------------------------------------------------------------------------------second_index,third_index = np.asarray(silhouette_scores).argsort()[-2],np.asarray(silhouette_scores).argsort()[-3]\n",
"second_index,third_index = __.______(np.asarray(___________))[__], __.______(np.asarray(________))[__]\n",
"best_models.append(kmeans_models[second_index])\n",
"best_models.append(kmeans_models[third_index])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "l04hAbNHeF7w"
},
"source": [
"And now, let's get a set of predictions for each model and store it in a list!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7CMme4lpeM3x"
},
"outputs": [],
"source": [
"pred_labels = []\n",
"for model in best_models:\n",
" pred_labels.append(model.predict(reduced_X))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PzRQD2xVeVMC"
},
"source": [
"Pandas will make producing a nice plot a lot simpler. Let's import it and make a dataframe with the reduced input components, the truth labels, and the predicted cluster labels.\n",
"\n",
"Note that the predicted labels don't correspond to the digit labels since this is an unsupervised model!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6vHSkMKSebwF"
},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"plot_data = np.stack([X_plot[:,0], X_plot[:,1], sub_y, *pred_labels],axis=1)\n",
"df = pd.DataFrame(plot_data, columns=['X1','X2','truth','pred_1','pred_2','pred_3'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FW9nDgbxe4ze"
},
"source": [
"And now we'll make a nice, big 4x4 plot that allows us to see the true answers and how our algorithm clustered our data!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "P5gFTg7XfCQM"
},
"outputs": [],
"source": [
"fig, axes = plt.subplots(2, 2, figsize=(16,16))\n",
"\n",
"groups = df.groupby('truth')\n",
"for label, group in groups:\n",
" axes[0,0].plot(group.X1, group.X2, marker='o', linestyle='', markersize=4, label=int(label))\n",
"axes[0,0].legend(fontsize=12)\n",
"axes[0,0].set_title('Truth', fontsize=16)\n",
"axes[0,0].axis('off')\n",
"\n",
"groups = df.groupby('pred_1')\n",
"for label, group in groups:\n",
" axes[0,1].plot(group.X1, group.X2, marker='o', linestyle='', markersize=4)\n",
"axes[0,1].set_title('\"Best\" Clustering', fontsize=16)\n",
"axes[0,1].axis('off')\n",
"\n",
"groups = df.groupby('pred_2')\n",
"for label, group in groups:\n",
" axes[1,0].plot(group.X1, group.X2, marker='o', linestyle='', markersize=4)\n",
"axes[1,0].set_title('$2^{nd}$ Best Clustering', fontsize=16)\n",
"axes[1,0].axis('off')\n",
"\n",
"groups = df.groupby('pred_3')\n",
"for label, group in groups:\n",
" axes[1,1].plot(group.X1, group.X2, marker='o', linestyle='', markersize=4)\n",
"axes[1,1].set_title('$3^{rd}$ Best Clustering', fontsize=16)\n",
"axes[1,1].axis('off')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Fj-JvVk2nO0X"
},
"source": [
"Our figure looks like this."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i7Ag6DgrnJfw"
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kzx17wP7f6i6"
},
"source": [
"Assuming you started out from the intended random seed, you'll be able to see that the \"Best\" model is trying to separate the 0 digits from the non-zero digits. The $2^{nd}$ best model is able to separate the digits pretty well, but lumps 4s and 9s into a single cluster!\n",
"\n",
"The $3^{rd}$ best model begins to group digits into clusters that may not have much significance to us at first glance, but whose metrics seem to indicate worse performance and whose results would need further analysis to try to understand."
]
}
],
"metadata": {
"colab": {
"include_colab_link": true,
"name": "S3_2_Clustering.ipynb",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
}
},
"nbformat": 4,
"nbformat_minor": 1
}