{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "view-in-github", "colab_type": "text" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "IY2vdUC9L0_F" }, "source": [ "# Deep Computer Vision" ] }, { "cell_type": "markdown", "metadata": { "id": "6-Da26nR0EB-" }, "source": [ "
\n", "\n", "\n", "\n", "\n", "Top: Picture of Shinkyo Bridge in Nikko (日光の神橋) (2010), ©Milton Gomez 😀
\n", "Bottom: Style transfers based on the above image, using Google's [Deep Dream Generator](https://deepdreamgenerator.com/)
\n", "\n", "Convolutional networks find spatial relationships in multidimensional data (most often images), which allow computers to detect patterns to make predictions or modify the input images in surprising ways. The following image shows a series of feature map visualizations for a convolutional neural network.\n", "\n", "
\n", "\n", "The images may not seem immediately relevant at first glance without knowing a transfersbit more about the input. For example, with the picture of a cat as an input the CNN could extract features related to the eyes, ears, or details as fine as pupil types!\n", "\n", "
\n", "\n", "For the style transfer examples shown at the beginning of the notebook, the original image is transformed so that it looks similar to a target style image (not shown) until they're virtually indistinguishable using the information from the filters.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "Z0tD9Dpz0C_E" }, "source": [ "Convolution Visualization images from:\n", "Qin, Z., Yu, F., Liu, C., & Chen, X. (2018). How convolutional neural network see the world-A survey of convolutional neural network visualization methods. [*arXiv preprint arXiv:1804.11191*](https://arxiv.org/pdf/1804.11191.pdf?ref=https://githubhelp.com)." ] }, { "cell_type": "markdown", "metadata": { "id": "eJm4QXrxTIyi" }, "source": [ "## Notebook Setup" ] }, { "cell_type": "markdown", "metadata": { "id": "Hkt-bX2feeL6" }, "source": [ "Today we'll be training CNNs, a process which can be very slow when run on a CPU. Instead, we'll be relying on GPU processing! Thankfully, we can easily make the switch on Colab!\n", "\n", "(If you're running the notebooks on your own computer, you *may* have to jump through a few hoops in order to take advantage of your computer's GPU)\n", "\n", "***Do note, however, that access to GPUs on Colab is somewhat limited - running your model's training too many times may limit your access to the GPU resources at Google.***\n", "\n", "###Changing the Runtime to GPU on Colab\n", "
\n", "
Click on the runtime dropdown menu and click on \"change on runtime type\"

\n", "
THen select \"GPU\" on the hardware accelerator dropdown menu

\n", "
\n", "\n", "Once you've changed the runtime type, run the Notebook setup cell. A message confirming that you've succesfully changed runtime type should be printed 😃\n", "\n", "In the setup cell, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We'll also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "XB0vnsGmKtQH" }, "outputs": [], "source": [ "# Python ≥3.5 is required\n", "import sys\n", "assert sys.version_info >= (3, 5)\n", "\n", "# Is this notebook running on Colab or Kaggle?\n", "IS_COLAB = \"google.colab\" in sys.modules\n", "IS_KAGGLE = \"kaggle_secrets\" in sys.modules\n", "\n", "# Scikit-Learn ≥0.20 is required\n", "import sklearn\n", "assert sklearn.__version__ >= \"0.20\"\n", "\n", "# TensorFlow ≥2.0 is required\n", "import tensorflow as tf\n", "import tensorflow_datasets as tfds\n", "from tensorflow import keras\n", "assert tf.__version__ >= \"2.0\"\n", "\n", "if not tf.config.list_physical_devices('GPU'):\n", " print(\"No GPU was detected. CNNs can be very slow without a GPU.\")\n", " if IS_COLAB:\n", " print(\"Go to Runtime > Change runtime and select a GPU hardware accelerator.\")\n", " if IS_KAGGLE:\n", " print(\"Go to Settings > Accelerator and select GPU.\")\n", "else:\n", " print(f\"GPU runtime succesfully selected! We're ready to train our CNNs.\")\n", "\n", "# Common imports\n", "import numpy as np\n", "import os\n", "import pooch\n", "\n", "# to make this notebook's output stable across runs\n", "rnd_seed = 42\n", "rnd_gen = np.random.default_rng(rnd_seed)\n", "\n", "# To plot pretty figures\n", "%matplotlib inline\n", "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "mpl.rc('axes', labelsize=14)\n", "mpl.rc('xtick', labelsize=12)\n", "mpl.rc('ytick', labelsize=12)\n", "\n", "# Where to save the figures\n", "PROJECT_ROOT_DIR = \".\"\n", "CHAPTER_ID = \"cnn\"\n", "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n", "os.makedirs(IMAGES_PATH, exist_ok=True)\n", "\n", "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n", " path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n", " print(\"Saving figure\", fig_id)\n", " if tight_layout:\n", " plt.tight_layout()\n", " plt.savefig(path, format=fig_extension, dpi=resolution)\n", "\n", "# Loading Tensorboard\n", "%load_ext tensorboard" ] }, { "cell_type": "markdown", "metadata": { "id": "cxag7xFHMu9P" }, "source": [ "## Data Setup" ] }, { "cell_type": "markdown", "metadata": { "id": "fng5IvmVwVlj" }, "source": [ "Today, we won't be working on the MNIST dataset! Instead, we'll be working on the [tensorflow flower database](https://knowyourdata-tfds.withgoogle.com/#tab=STATS&dataset=tf_flowers), and we'll be attempting to train a Neural Network to learn to classify the flowers into 1 of 5 flower species: daisies, dandelions, roses, sunflowers, and tulips.\n", "\n", "Let's begin by loading the data into our colab environment. The data is hosted online and loaded directly into Google's servers (if you're running this notebook on Colab, that is!) - which is lucky since its about two-hundred megabytes of data and we can take advantage of Google's servers' download speeds 🤖\n", "\n" ] }, { "cell_type": "code", "source": [ "# Let's clear out the backend and set our random seeds\n", "# Consistency makes things easier for labs!\n", "keras.backend.clear_session()\n", "tf.random.set_seed(rnd_seed)\n", "np.random.seed(rnd_seed)" ], "metadata": { "id": "inzX7in42dtu" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## Q1) Load the `tf_flowers` dataset from Tensorflow. Split it into a training, validation, and test set. Make sure you save the dataset information (it will be useful for sampling the datasets), and shuffle the files for good measure!" ], "metadata": { "id": "o6VIO3l015gk" } }, { "cell_type": "markdown", "source": [ "*Hint 1: Tensorflow Datasets was imported as `tfds` in the notebook setup. Check out the `tfds.load()` method [on the documentation](https://www.tensorflow.org/datasets/api_docs/python/tfds/load).*\n", "\n", "*Hint 2: If you use the `.load()` method with `with_info` set to `True`, the function will return the requested datasplit in a tuple with the dataset information as a separate variable.*\n", "\n", "*Hint 3: The datasets are loaded with `as_supervised` set to `False` by default, which requries that we worry about dictionaries when trying to access the data. We can make our lives easier by setting it to `True`*\n", "\n", "*Hint 4: The dataset we'll be using today includes a single `train` set, but by specifying the `split` list we can tell `tfds` how we want it to split that data. We can also use percentages in the indices to indicate which percent of the dataset to take into the dataset.*" ], "metadata": { "id": "KKigMgtc6oHI" } }, { "cell_type": "code", "source": [ "#@title Hint 5: One example implementation\n", "\n", "'''\n", "(test_set, valid_set, train_set), info = tfds.load(\n", " name=\"tf_flowers\",\n", " split=[\"train[:10%]\", \"train[10%:25%]\", \"train[25%:]\"],\n", " as_supervised=True,\n", " shuffle_files=True,\n", " with_info=True\n", " )\n", "\n", "# Datasets loaded this way don't have a string ID to identify them, so we'll set\n", "# up our own as it will make other code more compact/readable. :)\n", "train_set.name='Training'\n", "valid_set.name='Validation'\n", "test_set.name='Test'\n", "''';" ], "metadata": { "id": "PPCzx8RzLIpU", "cellView": "form" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "bTgW6itAMzJk" }, "outputs": [], "source": [ "(_____, _____, _____), _____ = tfds.load(\n", " name=\"tf_flowers\",\n", " split=[\"train[:___%]\", \"train[___%:___%]\", \"train[___%:]\"],\n", " as_supervised=_____,\n", " shuffle_files=_____,\n", " with_info=_____\n", " )\n", "\n", "# Datasets loaded this way don't have a string ID to identify them, so we'll set\n", "# up our own as it will make other code more compact/readable. :)\n", "_____.name='Training'\n", "_____.name='Validation'\n", "_____.name='Test'" ] }, { "cell_type": "markdown", "metadata": { "id": "VK27vWlQqcyF" }, "source": [ "We now have a set of variables that have the training, validation, and test sets, as well as a variable with the information about the dataset. Let's go ahead and define a function that will let us visualize our data." ] }, { "cell_type": "markdown", "source": [ "## Q2) Define a function that takes in a dataset and the information about the dataset, prints out how many samples are in the dataset, and displays a set of samples from the dataset." ], "metadata": { "id": "cSe6Zo9w6KF0" } }, { "cell_type": "markdown", "source": [ "*Hint 1: Tensorflow datasets include a `.cardinality()` method that counts the number of datapoints in the dataset, and a `.numpy()` method that converts the resulting value to a numpy array for clean print access*\n", "\n", "*Hint 2: We defined the `.name()` attribute for each dataset in the previous cell!*\n", "\n", "*Hint 3: tfds includes a `show_examples` method. [Here is the documentation](https://www.tensorflow.org/datasets/api_docs/python/tfds/visualization/show_examples).*\n", "\n", "*Hint 4: You can shuffle the contents of a dataset by calling its `.shuffle()` method. Try running it with an integer value between 16 and 256 as an argument.*\n", "\n", "*Hint 5: `show_examples` expects a set of (dataset) and (dataset info) objects as argumnets!*" ], "metadata": { "id": "hp_ApU7u6wth" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "_zhp4KrbQabU" }, "outputs": [], "source": [ "def dataset_info(__, _____):\n", " # Extract the number of samples in the dataset in an easily printable format\n", " num_samples = __._____()._____()\n", "\n", " # Print the dataset name we defined in the previous code cell and the number\n", " # of samples\n", " print(f\"\\n{____._____} set contains {num_samples} data samples.\",\n", " \"Let's visualize some of them...\\n\")\n", "\n", " # Show examples from the dataset. Shuffle to make things more interesting!\n", " tfds._____(___._____(__), _____)" ] }, { "cell_type": "markdown", "metadata": { "id": "QdAVRjLpSYFs" }, "source": [ "And now let's run the function on each of our training, validation, and test sets..." ] }, { "cell_type": "markdown", "source": [ "## Q3) Run your defined visualization function on each of the training, validation, and test sets." ], "metadata": { "id": "LhaFPnxj9iaP" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "NADDcqLIH0YW" }, "outputs": [], "source": [ "dataset_info(_____, _____)\n", "dataset_info(_____, _____)\n", "dataset_info(_____, _____)" ] }, { "cell_type": "markdown", "metadata": { "id": "FaibV8Bb3DQ-" }, "source": [ "If everything worked out fine, you'll have something like the following as your output:\n", "
\n", "\n", "The flowers look very nice! (\"*And I'd be pretty bad at classifying them myself...*\" - a botanically challenged TA)\n", "\n", "However, there is one sore point for our purposes - *the images have different resolutions*. Why is this a sore point? Well, in our architecture we'll eventually flatten our convolutions and connect them to a dense layer, and as such we will need for all of the images to have the same dimensions! (There are other ways to address the issue of resolution, but we won't discuss these for now)\n", "\n", "We also note that the images are stored as pixels containing a value between 0 and 255 for each one of three color channels (RGB) - we'd prefer that the values be normalized to fall between 0 and 1.\n", "\n", "Finally, if you paid close attention to the labels on the nice images we displayed you may have noticed that there is a number between 0 and 4 next to the name of each of the flowers - this is the integer value associated with the label. We've previously seen, however, that when addressing classification problems it's often better to use one-hot encoding.\n", "\n", "Let's write a preprocessing function that will help us address all of these issues!" ] }, { "cell_type": "markdown", "source": [ "## Q4) Write a preprocessing function for the images and labels in our dataset. The function should set the image size to 128x128, normalize the pixel values to be between 0 and 1, and one-hot encode the labels" ], "metadata": { "id": "uMmChGZP-xd2" } }, { "cell_type": "markdown", "source": [ "*Hint 1: As we've set the `as_supervised` argument as `True` when loading the dataset, the preprocessing function should take in an image and a label as an argument.* ***Any other parameters taken in by the function should be hardcoded into our function.***\n", "\n", "*Hint 2: In order to modify the image fed into the function, we will have to convert it to a `float32` type using Tensorflow's `.cast()` method ([here is its documentation](https://www.tensorflow.org/api_docs/python/tf/cast)). Similarly, we need to cast the label as an `int32` type object.*\n", "\n", "*Hint 3: Tensorflow has a built-in image resizer, implemented as the `image.resize()` method. [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/image/resize).*\n", "\n", "*Hint 4: Tensorflow has a built-in one-hot encoder, implemented as the `one_hot()` method. [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/one_hot)*\n", "\n", "*Hint 5: After one-hot encoding, the label should be recast to the `float32` datatype*" ], "metadata": { "id": "32NOCtPJ_XI0" } }, { "cell_type": "code", "source": [ "#@title Hint 6: One example function implementation\n", "\n", "'''\n", "def preprocessing_function(image, label):\n", " # We're going to hard code the image size we want to use. We can define this\n", " # with a lambda function, but we won't really need to change this and it's\n", " # more trouble than it's worth for us right now :)\n", " image_size = 128\n", " num_classes = 5\n", "\n", " image = tf.cast(image, tf.float32)\n", " # Normalize the pixel values\n", " image = image / 255.0\n", " # Resize the image\n", " image = tf.image.resize(image, (image_size, image_size))\n", "\n", " # Casts to an Int and performs one-hot ops\n", " label = tf.one_hot(tf.cast(label, tf.int32), num_classes)\n", " # Recasts it to Float32\n", " label = tf.cast(label, tf.float32)\n", " return image, label\n", "''';" ], "metadata": { "cellView": "form", "id": "t3wN17PqL1NB" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "hRlKbZ6C5PA_" }, "outputs": [], "source": [ "def preprocessing_function(image, label):\n", " # We're going to hard code the image size we want to use. We can define this\n", " # with a lambda function, but we won't really need to change this and it's\n", " # more trouble than it's worth for us right now :)\n", " image_size = _____\n", " num_classes = _____\n", "\n", " # Cast the image and label datatypes\n", " image = tf._____(image, tf.float32)\n", " label = __.__(___,____)\n", "\n", " # Normalize the pixel values. Use a float value in the denominator!\n", " image = _____ / _____\n", "\n", " # Resize the image\n", " image = tf._____._____(_____, (_____, _____))\n", "\n", " # Cast the label to int32 and one-hot encode\n", " label = tf._____(_____, _____)\n", " # Recast label to Float32\n", " label = tf.cast(_____, __._____)\n", "\n", " return image, label" ] }, { "cell_type": "markdown", "source": [ "## Q5) Apply the preprocessing function to each of the training, validation, and test sets." ], "metadata": { "id": "CJZCrQJmD8yZ" } }, { "cell_type": "markdown", "source": [ "*Hint 1: the datasets have a `.map()` method that allow applying a function to each image and label combination in the dataset. [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map).*" ], "metadata": { "id": "N0L4JPcqEsft" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "rFbBJtM8mdB_" }, "outputs": [], "source": [ "train = _____.map(_____)\n", "valid = _____.map(_____)\n", "test = _____.map(_____)" ] }, { "cell_type": "markdown", "metadata": { "id": "7qTabL9TlMeK" }, "source": [ "At this point I'd also like to point out that our dataset is not set up to be taken in batches. If you call `train_set.take(1)`, you'll extract a single image! (Let's run some code and verify this)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "leQE-yzhlfhe" }, "outputs": [], "source": [ "for images, labels in train.take(1):\n", " print(f'Images shape: {images.numpy().shape} Labels: {labels.numpy().shape}')" ] }, { "cell_type": "markdown", "metadata": { "id": "ZoUJmcHbmSc3" }, "source": [ "We actually want to work in 32 image batches, so let's go ahead and batch our datasets." ] }, { "cell_type": "markdown", "source": [ "## Q6) Batch each of the training, validation, and test sets" ], "metadata": { "id": "1NRomhbYG5Hc" } }, { "cell_type": "markdown", "source": [ "*Hint 1: You can define a `batch_size` variable to guarantee that the batch size is updated for all three datasets if you change the value and rerun the cell.*\n", "\n", "*Hint 2: Tensorflow datasets include a `.batch()` method that allows you to easily define batch sizes associated with the database instance. [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#batch)*." ], "metadata": { "id": "Pcuw2RF-HDUm" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "8se1R0nBMJsr" }, "outputs": [], "source": [ "# Define the batch size\n", "batch_size = ______\n", "\n", "train = ___.batch(___)\n", "validation = ___.___(___)\n", "test = ___.___(___)" ] }, { "cell_type": "markdown", "metadata": { "id": "kKE4ks4apX20" }, "source": [ "If we now take a sample like we did before, we'll notice that the first dimension in the shape tuple is our batch size!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "5qXkbT4hpWQa" }, "outputs": [], "source": [ "for images, labels in validation.take(1):\n", " print(f'Images shape: {images.numpy().shape} Labels: {labels.numpy().shape}')\n", " print(f'Max pixel value: {images.numpy().max()}, min pixel value: {images.numpy().min()}')\n", " print(\"Here's a sample image:\\n\")\n", " fig,ax = plt.subplots()\n", " ax.axis('off')\n", " ax.imshow(images[0])" ] }, { "cell_type": "markdown", "metadata": { "id": "KfTCA2dZ2UDl" }, "source": [ "\n", "Now that we've verified that our data generators work as intended, let's go ahead and build our model!" ] }, { "cell_type": "markdown", "metadata": { "id": "LEbTdpaTqXQq" }, "source": [ "## Model Setup and Training" ] }, { "cell_type": "markdown", "metadata": { "id": "NdPmqnowJd6W" }, "source": [ "Let's begin by setting up everything we need to define our callbacks! This time, we want to work with multiple runs, and in order to visualize them in Tensorboard we'll want to generate the name automatically using the current date and time!" ] }, { "cell_type": "markdown", "source": [ "## Q7) Define a function that returns a filepath with the format `'./CNN_logs/run_CURRENT-DATE-AND-TIME'`" ], "metadata": { "id": "n8toRxWyPhCD" } }, { "cell_type": "markdown", "source": [ "*Hint 1: Numpy includes a method to return the current date and time as a datetime64 object. Call the `datetime64` method with `'now'` as an argument.*\n", "\n", "*Hint 2: The OS library, imported as `os` in the notebook setup, allows you to join path strings in a manner appropriate to the operating system using the `os.path.join()` method. This helps avoids headaches when running code across windows/linux/macOS! [Here is the documentation](https://docs.python.org/3/library/os.path.html#os.path.join)*.\n", "\n", "*Hint 3: You can use the `.astype()` method to convert a numpy datetime64 object to a string.*\n", "\n", "*Hint 4: OS also includes a `.curdir` attribute that returns the current directory.*\n", "\n", "*Hint 5: If you convert the datetime64 object to a string, it will include the seconds! You can remove these with regular python indexing (i.e., [:-3])*" ], "metadata": { "id": "hqZhNephQHrd" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "iZOUNcnQ_Nc9" }, "outputs": [], "source": [ "def get_CNN_logdir():\n", " time = ___(___).___(___)\n", " run_logdir = __.____.____(__._____, \"CNN_logs\", f\"run_{____}\") # time goes in the fstring\n", " return run_logdir" ] }, { "cell_type": "markdown", "source": [ "Let's try out our function! It should return something like: `./CNN_logs/run_2022-04-10T18:49`" ], "metadata": { "id": "uKgn5djRS97m" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "_Pl2TyQN_wQE" }, "outputs": [], "source": [ "get_CNN_logdir()" ] }, { "cell_type": "markdown", "metadata": { "id": "tRbyw3puKf6U" }, "source": [ "Define your [callbacks](https://keras.io/api/callbacks/) below. For the checkpoint `checkpoint_cb`, we recommend monitoring the validation loss to avoid overfitting. Look for \"monitor\" in the `model_checkpoint`'s documentation [at this link](https://keras.io/api/callbacks/model_checkpoint/).\n" ] }, { "cell_type": "markdown", "source": [ "##Q8) Set up an EarlyStopping Callback, a ModelCheckpoint callback, and a Tensorboard Callback for a CNN model *without* data augmentation.\n", "\n", "*Hint 1: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping) for the EarlyStopping Callback and [here is the one](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint) for the ModelCheckpoint Callback.*" ], "metadata": { "id": "ZM0nDM0NZzS4" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "lk2wNUwIKlUZ" }, "outputs": [], "source": [ "early_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=____)\n", "checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(\"CNN_unaugmented.h5\",\n", " save_best_only=True,\n", " monitor=_____)\n", "tensorboard_cb = tf.keras.callbacks.TensorBoard(get_CNN_logdir())" ] }, { "cell_type": "markdown", "source": [ "We now have a set of callbacks to call during training. Let's go ahead and define the model! Though let's go ahead and clean up our random states first..." ], "metadata": { "id": "9643NwNJaXFm" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "E2Smt-3l2To7" }, "outputs": [], "source": [ "# Let's clear out the backend and set our random seeds.\n", "# Consistency is key :)\n", "keras.backend.clear_session()\n", "tf.random.set_seed(rnd_seed)\n", "np.random.seed(rnd_seed)" ] }, { "cell_type": "markdown", "source": [ "## Q9) Define a convolutional neural network model. Do not use data augmentation techniques! We want to use this same architecture + data augmentation later.\n", "\n", "*Hint 1: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D) for the Conv2D layer in tensorflow. Generally, you want to start out with larger kernels and a smaller number of filters and move on to smaller kernels with a large number of filters as your network becomes deeper.*\n", "\n", "*Hint 2: After the convolutional layers, you can use flatten to change the collection of filtered images into a flat array to feed into a densely connected layer! In essence, the CNN is representing the images in a* ***latent space*** *, and a densely connected ANN is connected on top to classify the images based on their representation in the latent space.*\n", "\n", "*Hint 3: Once your model is defined, we'll be using the `.build()` and `.summary()` methods to check how many parameters our model includes! A TA's model included around 13 million parameters when testing this notebook! This is significantly more parameters than the amount used when we trained our first artificial neural networks.*" ], "metadata": { "id": "vuhrwS7mbBN4" } }, { "cell_type": "code", "source": [ "#@title Hint 4: Example model used during notebook development\n", "\n", "'''\n", "# Note: there is at least one small, immediate change that will make the\n", "# performance of this model more effective :)\n", "\n", "model = keras.models.Sequential([\n", " # Convolution 1\n", " keras.layers.Conv2D(32, kernel_size=7, padding=\"same\", activation=\"relu\"),\n", " keras.layers.MaxPool2D((3,3)),\n", "\n", " # Convolution 2\n", " keras.layers.Conv2D(64, kernel_size=5, padding=\"same\", activation=\"relu\"),\n", " keras.layers.MaxPool2D((2,2)),\n", "\n", " # Convolution 3\n", " keras.layers.Conv2D(96, kernel_size=3, padding=\"same\", activation=\"relu\"),\n", " keras.layers.MaxPool2D((2,2)),\n", "\n", " # Convolution 4\n", " keras.layers.Conv2D(128, kernel_size=3, padding=\"same\", activation=\"relu\"),\n", " keras.layers.MaxPool2D((2,2)),\n", "\n", " keras.layers.Flatten(),\n", " keras.layers.Dense(4096, activation=\"relu\"),\n", " keras.layers.Dropout(0.5),\n", " keras.layers.Dense(5, activation=\"softmax\")\n", "])\n", "''';" ], "metadata": { "cellView": "form", "id": "tcVwvmzxMSt9" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "5zVjB9-TEhvN" }, "outputs": [], "source": [ "model = keras.models.Sequential([\n", " # Convolution 1\n", " keras.layers.Conv2D(___, kernel_size=__, padding=\"same\", activation=____),\n", " keras.layers.MaxPool2D((__,__)),\n", "\n", " # Convolution 2\n", " keras.layers.Conv2D(___, kernel_size=__, padding=\"same\", activation=____),\n", " keras.layers.MaxPool2D((__,__)),\n", "\n", "\n", " # Convolution 3\n", " keras.layers.Conv2D(___, kernel_size=__, padding=\"same\", activation=____),\n", " keras.layers.MaxPool2D((__,__)),\n", "\n", "\n", " # Convolution 4\n", " keras.layers.Conv2D(___, kernel_size=__, padding=\"same\", activation=____),\n", " keras.layers.MaxPool2D((__,__)),\n", "\n", "\n", " keras.layers.Flatten(),\n", " keras.layers.Dense(____, activation=____),\n", " keras.layers.Dropout(___),\n", " keras.layers.Dense(___, activation=\"softmax\")\n", "])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "wKf-oG2E8jrC" }, "outputs": [], "source": [ "# Build the model using our input image resolution to produce the number of\n", "# parameters in our model...\n", "model.build((None, 128 , 128, 3))\n", "\n", "# And visualize the structure of the model\n", "model.summary()" ] }, { "cell_type": "markdown", "source": [ "##Q10) Compile the CNN model. We recommend using `categorical_crossentropy` as the loss function, 'adam' as the optimizer, and 'accuracy' as a metric" ], "metadata": { "id": "9Z-hPa8ulOFz" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "YjxV2z463TGD" }, "outputs": [], "source": [ "model.compile(loss=_____,\n", " optimizer=_____,\n", " metrics=[_____])" ] }, { "cell_type": "markdown", "source": [ "##Q11) Train the CNN model!\n", "\n", "*Hint 1: You can use the training data generator directly as your input and labels; the model class will automatically deal with it!*\n", "\n", "*Hint 2: While it's best to have defined too many epochs instead of too few (considering that the have an early stopping callback), it's not worth training over a large number of epochs for this project. Try setting the epoch limit somewhere around 30-50.*" ], "metadata": { "id": "w__DZuUMl5Z0" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "_7R6jC1sTGO8" }, "outputs": [], "source": [ "history = model.fit(_____, # Training data generator\n", " epochs=____,\n", " validation_data=_____, # Validation data generator\n", " callbacks=[_____,\n", " _____,\n", " _____])" ] }, { "cell_type": "markdown", "source": [ "Well, the performance of our CNN is likely underwhelming. This is what we saw during notebook development:\n", "
\n", "\n", "The model didn't have too hard a time learning on the training set, but the validation loss quickly diverged and we started overfitting our training data. 😯\n", "\n", "A common way to try to address this is by augmenting our training data - we can flip and rotate our images and it shouldn't make too large a difference.\n", "\n", "> \"A rose by any other name would smell as sweet\" - *Shakespeare*\n", "\n", "> \"An upside down rose is still a rose\" - *A significantly less talented poet than Shakespeare*" ], "metadata": { "id": "TBvDCQ3DsG-Z" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "2vNHB28MEfXy" }, "outputs": [], "source": [ "# Let's clear out the backend and set our random seeds\n", "# It's best to start out from a common point, no?\n", "keras.backend.clear_session()\n", "tf.random.set_seed(rnd_seed)\n", "np.random.seed(rnd_seed)" ] }, { "cell_type": "markdown", "source": [ "##Q12) Set up an EarlyStopping Callback, a ModelCheckpoint callback, and a Tensorboard Callback for a CNN model with data augmentation.\n", "\n", "*Hint 1: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping) for the EarlyStopping Callback and [here is the one](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint) for the ModelCheckpoint Callback.*" ], "metadata": { "id": "iGX4qv1juSTc" } }, { "cell_type": "code", "source": [ "early_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=____)\n", "checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(\"CNN_augmented.h5\",\n", " save_best_only=True,\n", " monitor=_____)\n", "tensorboard_cb = tf.keras.callbacks.TensorBoard(get_CNN_logdir())" ], "metadata": { "id": "X5y8o0EwuSmQ" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "## Q13) Train an identical model to that defined in Q9, with the exception of `RandomFlip` and `RandomRotation` augmentation layers added before the convolutional layers.\n", "\n", "*Hint 1: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/RandomFlip) for the `RandomFlip` method.*\n", "\n", "*Hint 1: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/RandomRotation) for the `RandomRotation` method.*" ], "metadata": { "id": "SNJ15Knvtg9s" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "lu9bfEK72fSf" }, "outputs": [], "source": [ "model = keras.models.Sequential([\n", " keras.layers.RandomFlip(), # Flip augmentation\n", " keras.layers.RandomRotation(0.1), # Rotation Aumentation\n", "\n", " # Copy your previous model's layers here\n", "])" ] }, { "cell_type": "code", "source": [ "# Build the model using our input image resolution to produce the number of\n", "# parameters in our model...\n", "model.build((None, 128 , 128, 3))\n", "\n", "# And visualize the structure of the model\n", "model.summary()" ], "metadata": { "id": "mtTE9Si4RCX8" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "source": [ "##Q14) Compile the CNN model. We recommend using `categorical_crossentropy` as the loss function, 'adam' as the optimizer, and 'accuracy' as a metric" ], "metadata": { "id": "ZXiQG_W9uxqa" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "bNiAHpWMEoWm" }, "outputs": [], "source": [ "model.compile(loss=_____,\n", " optimizer=_____,\n", " metrics=[_____])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "rqoc2YtUErRX" }, "outputs": [], "source": [ "history = model.fit(_____, # Training data generator\n", " epochs=____,\n", " validation_data=_____, # Validation data generator\n", " callbacks=[_____,\n", " _____,\n", " _____])" ] }, { "cell_type": "markdown", "source": [ "If everything went according to plan, your model icluding data augmentation should perform a little like this:\n", "
\n", "\n", "Now let's run tensorboard and compare your two runs!" ], "metadata": { "id": "gZ3IAhUcvKmz" } }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "lI622QA5TPAH" }, "outputs": [], "source": [ "%tensorboard --logdir=./CNN_logs --port=6006" ] }, { "cell_type": "markdown", "source": [ "If everything went well, the model trained on the augmented data should have a lower accuracy on the training set compared to the original, but the behavior accross both datasets should be indicative of more meaningful features being extracted and overall better generalization.\n", "\n", "(While the use of augmented data is exciting, this might not be wise to do - e.g., when looking at maps of atmospheric variable data)" ], "metadata": { "id": "uAlGLozrvszF" } }, { "cell_type": "code", "source": [ "# Let's load the models!\n", "non_aug_model = keras.models.load_model('CNN_unaugmented.h5')\n", "aug_model = keras.models.load_model('CNN_augmented.h5')\n", "\n", "# And test them on the testing dataset\n", "non_aug_model.evaluate(test)\n", "aug_model.evaluate(test)" ], "metadata": { "id": "kDCtrdj7fAgY" }, "execution_count": null, "outputs": [] } ], "metadata": { "accelerator": "GPU", "colab": { "toc_visible": true, "provenance": [], "include_colab_link": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }