Image Processing with Keras in Python

Course Description Deep learning methods use data to train neural network algorithms to do a variety of machine learning tasks, such as classification of different classes of objects. Convolutional neural networks are deep learning algorithms that are particularly powerful for analysis of images. This course will teach you how to construct, train and evaluate convolutional neural networks. You will also learn how to improve their ability to learn from data, and how to interpret the results of the training.

1. Image Processing With Neural Networks

Convolutional neural networks use the data that is represented in images to learn. In this chapter, we will probe data in images, and we will learn how to use Keras to train a neural network to classify objects that appear in images.

link chapter

1.1 Introducing convolutional neural networks

1.2 Images as data: visualizations

To display image data, you will rely on Python's Matplotlib library, and specifically use matplotlib's pyplot sub-module, that contains many plotting commands. Some of these commands allow you to display the content of images stored in arrays.

Instructions

100 XP

  • Import the image from the file bricks.png into data.
  • Display the image in data on the screen.
import matplotlib.pyplot as plt

# Load the image
data = plt.imread('bricks.png')

# Display the image
plt.imshow(data)
plt.show()

image.png

Nicely done! Visit Matplotlib's website to learn more about plotting commands you could use.

1.3 Images as data: changing images

To modify an image, you can modify the existing numbers in the array. In a color image, you can change the values in one of the color channels without affecting the other colors, by indexing on the last dimension of the array.

The image you imported in the previous exercise is available in data.

Instructions

100 XP

  • Modify the bricks image to replace the top left corner of the image (10 by 10 pixels) into a red square.
  • Visualize the resulting image.
data[0:10, 0:10, 0] = 1

# Set the green channel in this part of the image to 0
data[0:10, 0:10, 1] = 0

# Set the blue channel in this part of the image to 0
data[0:10, 0:10, 2] = 0

# Visualize the result
plt.imshow(data)
plt.show()

image.png

Great job! You now know how to manipulate images. By the way, if you set both the green and red channels to 1, that part of the image would be yellow.

1.4 Classifying images

1.5 Using one-hot encoding to represent images

Neural networks expect the labels of classes in a dataset to be organized in a one-hot encoded manner: each row in the array contains zeros in all columns, except the column corresponding to a unique label, which is set to 1.

The fashion dataset contains three categories:

  1. Shirts
  2. Dresses
  3. Shoes

In this exercise, you will create a one-hot encoding of a small sample of these labels.

Instructions

100 XP

  • Initialize the ohe_labels variable to hold the one-hot encoded array.
  • Use np.where() to find the location of the category of the item in each iteration in categories.
  • Assign a 1 into the correct row/column combination in every iteration.
n_categories = 3

# The unique values of categories in the data
categories = np.array(["shirt", "dress", "shoe"])

# Initialize ohe_labels as all zeros
ohe_labels = np.zeros((len(labels), n_categories))

# Loop over the labels
for ii in range(len(labels)):
    # Find the location of this label in the categories variable
    jj = np.where(categories == labels[ii])
    # Set the corresponding zero to one
    ohe_labels[ii, jj] = 1

Nice! You can use this array to test classification performance.

1.6 Evaluating a classifier

To evaluate a classifier, we need to test it on images that were not used during training. This is called "cross-validation": a prediction of the class (e.g., t-shirt, dress or shoe) is made from each of the test images, and these predictions are compared with the true labels of these images.

The results of cross-validation are provided as one-hot encoded arrays: test_labels and predictions.

Instructions

100 XP

  • Multiply the arrays with each other and sum the result to find the total number of correct predictions.
  • Divide the number of correct answers (the sum) by the length of predictions array to calculate the proportion of correct predictions.
number_correct = (test_labels * predictions).sum()
print(number_correct)

# Calculate the proportion of correct predictions
proportion_correct = number_correct / len(predictions)
print(proportion_correct)

Great job! Let's talk about fitting classification models using Keras.

1.7 Classification with Keras

1.8 Build a neural network

We will use the Keras library to create neural networks and to train these neural networks to classify images. These models will all be of the Sequential type, meaning that the outputs of one layer are provided as inputs only to the next layer.

In this exercise, you will create a neural network with Dense layers, meaning that each unit in each layer is connected to all of the units in the previous layer. For example, each unit in the first layer is connected to all of the pixels in the input images. The Dense layer object receives as arguments the number of units in that layer, and the activation function for the units. For the first layer in the network, it also receives an input_shape keyword argument.

This course touches on a lot of concepts you may have forgotten, so if you ever need a quick refresher, download the Keras Cheat Sheet and keep it handy!

Instructions

100 XP

  • The first layer receives images as input, has 10 units and 'relu' activation.
  • The second input layer has 10 units and 'relu' activation.
  • The output layer has one unit for each category (3 categories) and 'softmax' activation.
from keras.models import Sequential
from keras.layers import Dense

# Initializes a sequential model
model = Sequential()

# First layer
model.add(Dense(10, activation='relu', input_shape=(784,)))

# Second layer
model.add(Dense(10, activation='relu'))

# Output layer
model.add(Dense(3, activation='softmax'))

Congratulations! You've built a neural network!

1.9 Compile a neural network

Once you have constructed a model in Keras, the model needs to be compiled before you can fit it to data. This means that you need to specify the optimizer that will be used to fit the model and the loss function that will be used in optimization. Optionally, you can also specify a list of metrics that the model will keep track of. For example, if you want to know the classification accuracy, you will provide the list ['accuracy'] to the metrics keyword argument.

Instructions

100 XP

Write code to compile the model with the 'adam' optimizer and 'categorical_crossentropy' as the loss function.

model.compile(optimizer='adam', 
           loss='categorical_crossentropy', 
           metrics=['accuracy'])

This neural network model is now ready to be fit to data.

1.10 Fitting a neural network model to clothing data

In this exercise, you will fit the fully connected neural network that you constructed in the previous exercise to image data. The training data is provided as two variables: train_data that contains the pixel data for 50 images of the three clothing classes and train_labels, which contains one-hot encoded representations of the labels for each one of these 50 images. Transform the data into the network's expected input and then fit the model on training data and training labels.

The model you compiled in the previous exercise, and train_data and train_labels are available in your workspace.

Instructions

100 XP

  • Prepare the data for fitting by reshaping it.
  • Fit the model by passing the input training data and training labels to the model's .fit() method.
train_data = train_data.reshape(50, 28*28)

# Fit the model
model.fit(train_data, train_labels, validation_split=0.2, epochs=3)

This model works well on the training data, but does it work well on test data?

1.11 Cross-validation for neural network evaluation

To evaluate the model, we use a separate test data-set. As in the train data, the images in the test data also need to be reshaped before they can be provided to the fully-connected network because the network expects one column per pixel in the input.

The model you fit in the previous exercise, and test_data and test_labels are available in your workspace.

Instructions

100 XP

  • Reshape the test_data so that it can be used to evaluate the model.
  • Evaluate the model on test_data using test_labels.
test_data = test_data.reshape(10, 28*28)

# Evaluate the model
model.evaluate(test_data, test_labels)

Not too bad! The model cross-validates rather accurately.

2. Using Convolutions

Convolutions are the fundamental building blocks of convolutional neural networks. In this chapter, you will be introducted to convolutions and learn how they operate on image data. You will also see how you incorporate convolutions into Keras neural networks.

2.1 Convolutions

2.2 One dimensional convolutions

A convolution of an one-dimensional array with a kernel comprises of taking the kernel, sliding it along the array, multiplying it with the items in the array that overlap with the kernel in that location and summing this product.

Instructions 100 XP Multiply each window in the input array with the kernel and sum the multiplied result and allocate the result into the correct entry in the output array (conv).

array = np.array([1, 0, 1, 0, 1, 0, 1, 0, 1, 0])
kernel = np.array([1, -1, 0])
conv = np.array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

# Output array
for ii in range(8):
    conv[ii] = (kernel * array[ii:ii+3]).sum()

# Print conv
print(conv)

Nicely done. Notice that we've only multiplied the kernel with eight different positions

2.3 Image convolutions

The convolution of an image with a kernel summarizes a part of the image as the sum of the multiplication of that part of the image with the kernel. In this exercise, you will write the code that executes a convolution of an image with a kernel using Numpy. Given a black and white image that is stored in the variable im, write the operations inside the loop that would execute the convolution with the provided kernel.

Instructions

100 XP

  • Select the right window from the image in each iteration and multiply this part of the image with the kernel.
  • Sum the result and allocate the sum to the correct entry in the output array (results).
kernel = np.array([[0, 1, 0], [1, 1, 1], [0, 1, 0]])
result = np.zeros(im.shape)

# Output array
for ii in range(im.shape[0] - 3):
    for jj in range(im.shape[1] - 3):
        result[ii, jj] = (im[ii:ii+3, jj:jj+3] * kernel).sum()

# Print result
print(result)

In a future exercise, you will see how to use Keras to implement a convolution like this one.

2.4 Defining image convolution kernels

In the previous exercise, you wrote code that performs a convolution given an image and a kernel. This code is now stored in a function called convolution() that takes two inputs: image and kernel and produces the convolved image. In this exercise, you will be asked to define the kernel that finds a particular feature in the image.

For example, the following kernel finds a vertical line in images:

np.array([[-1, 1, -1], 
          [-1, 1, -1], 
          [-1, 1, -1]])
Instructions 1/3

33 XP

Define a kernel that finds horizontal lines in images.

kernel = np.array([[-1, -1, -1], 
                   [1, 1, 1],
                   [-1, -1, -1]])
Instructions 2/3

Define a kernel that finds a light spot surrounded by dark pixels.

kernel = np.array([[-1, -1, -1], 
                   [-1, 1, -1],
                   [-1, -1, -1]])
Instructions 3/3

Define a kernel that finds a dark spot surrounded by bright pixels.

kernel = np.array([[1, 1, 1], 
                   [1, -1, 1],
                   [1, 1, 1]])

2.5 Implementing image convolutions in Keras

2.6 Convolutional network for image classification

Convolutional networks for classification are constructed from a sequence of convolutional layers (for image processing) and fully connected (Dense) layers (for readout). In this exercise, you will construct a small convolutional network for classification of the data from the fashion dataset.

Instructions

100 XP

  • Add a Conv2D layer to construct the input layer of the network. Use a kernel size of 3 by 3. You can use the img_rows and img_cols objects available in your workspace to define the input_shape of this layer.
  • Add a Flatten layer to translate between the image processing and classification part of your network.
  • Add a Dense layer to classify the 3 different categories of clothing in the dataset.
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Flatten

# Initialize the model object
model = Sequential()

# Add a convolutional layer
model.add(Conv2D(10, kernel_size=3, activation='relu', 
               input_shape=(img_rows, img_cols, 1)))

# Flatten the output of the convolutional layer
model.add(Flatten())
# Add an output layer for the 3 categories
model.add(Dense(3, activation='softmax'))

Congratulations! You just built a model with one convolutional layer.

2.7 Training a CNN to classify clothing types

Before training a neural network it needs to be compiled with the right cost function, using the right optimizer. During compilation, you can also define metrics that the network calculates and reports in every epoch. Model fitting requires a training data set, together with the training labels to the network.

The Conv2D model you built in the previous exercise is available in your workspace.

Instructions

100 XP

  • Compile the network using the 'adam' optimizer and the 'categorical_crossentropy' cost function. In the metrics list define that the network to report 'accuracy'.
  • Fit the network on train_data and train_labels. Train for 3 epochs with a batch size of 10 images. In training, set aside 20% of the data as a validation set, using the validation_split keyword argument.
model.compile(optimizer='adam', 
              loss='categorical_crossentropy', 
              metrics=['accuracy'])

# Fit the model on a training set
model.fit(train_data, train_labels, 
          validation_split=0.2, 
          epochs=3, batch_size=10)

Validation accuracy converges to 100%!

2.8 Evaluating a CNN with test data

To evaluate a trained neural network, you should provide a separate testing data set of labeled images. The model you fit in the previous exercise is available in your workspace.

Instructions

100 XP

  • Evaluate the data on a separate test set: test_data and test_labels.
  • Use the same batch size that was used for fitting (10 images per batch).
model.evaluate(test_data, test_labels, batch_size=10)

The first number in the output is the value of the cross-entropy loss, the second is the value of the accuracy. For this model, it's 100%!

2.9 Tweaking your convolutions

Padding allows a convolutional layer to retain the resolution of the input into this layer. This is done by adding zeros around the edges of the input image, so that the convolution kernel can overlap with the pixels on the edge of the image.

Instructions

100 XP

Add a Conv2D layer and choose a padding such that the output has the same size as the input.

model = Sequential()

# Add the convolutional layer
model.add(Conv2D(10, kernel_size=3, activation='relu', 
                 input_shape=(img_rows, img_cols, 1), 
                 padding='same'))

# Feed into output layer
model.add(Flatten())
model.add(Dense(3, activation='softmax'))

Great job! With padding set to 'same', the output layer will have the same size as the input layer!

2.10 Add padding to a CNN

The size of the strides of the convolution kernel determines whether the kernel will skip over some of the pixels as it slides along the image. This affects the size of the output because when strides are larger than one, the kernel will be centered on only some of the pixels.

Instructions

100 XP

Construct a neural network with a Conv2D layer with strided convolutions that skips every other pixel.

model = Sequential()

# Add the convolutional layer
model.add(Conv2D(10, kernel_size=3, activation='relu', 
              input_shape=(img_rows, img_cols, 1), 
              strides=2))

# Feed into output layer
model.add(Flatten())
model.add(Dense(3, activation='softmax'))

With strides set to 2, the network skips every other pixel.

2.11 Add strides to a convolutional network

The size of the strides of the convolution kernel determines whether the kernel will skip over some of the pixels as it slides along the image. This affects the size of the output because when strides are larger than one, the kernel will be centered on only some of the pixels.

Instructions

100 XP

Construct a neural network with a Conv2D layer with strided convolutions that skips every other pixel.

model = Sequential()

# Add the convolutional layer
model.add(Conv2D(10, kernel_size=3, activation='relu', 
              input_shape=(img_rows, img_cols, 1), 
              strides=2))

# Feed into output layer
model.add(Flatten())
model.add(Dense(3, activation='softmax'))

2.12 Calculate the size of convolutional layer output

Zero padding and strides affect the size of the output of a convolution.

What is the size of the output for an input of size 256 by 256, with a kernel of size 4 by 4, padding of 1 and strides of 2?

Instructions

50 XP

Possible Answers

  • 127

  • 255

  • 128

  • 256

image.png

image.png

3. Going Deeper

Convolutional neural networks gain a lot of power when they are constructed with multiple layers (deep networks). In this chapter, you will learn how to stack multiple convolutional layers into a deep network. You will also learn how to keep track of the number of parameters, as the network grows, and how to control this number.

3.1 Going deeper

3.2 Creating a deep learning network

A deep convolutional neural network is a network that has more than one layer. Each layer in a deep network receives its input from the preceding layer, with the very first layer receiving its input from the images used as training or test data.

Here, you will create a network that has two convolutional layers.

Instructions

100 XP

  • The first convolutional layer is the input layer of the network. This should have 15 units with kernels of 2 by 2 pixels. It should have a 'relu' activation function. It can use the variables img_rows and img_cols to define its input_shape.
  • The second convolutional layer receives its inputs from the first layer. It should have 5 units with kernels of 2 by 2 pixels. It should also have a 'relu' activation function.
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Flatten

model = Sequential()

# Add a convolutional layer (15 units)
model.add(Conv2D(15, kernel_size=2, activation='relu', 
              input_shape=(img_rows, img_cols, 1), 
              strides=2))


# Add another convolutional layer (5 units)
model.add(Conv2D(5, kernel_size=2, activation='relu', 
              input_shape=(img_rows, img_cols, 1), 
              strides=2))

# Flatten and feed to output layer
model.add(Flatten())
model.add(Dense(3, activation='softmax'))

Congratulations!! You built a network with multiple convolution layers.

3.3 Train a deep CNN to classify clothing images

Training a deep learning model is very similar to training a single layer network. Once the model is constructed (as you have done in the previous exercise), the model needs to be compiled with the right set of parameters. Then, the model is fit by providing it with training data, as well as training labels. After training is done, the model can be evaluated on test data.

The model you built in the previous exercise is available in your workspace.

Instructions

100 XP

  • Compile the model to use the categorical cross-entropy loss function and the Adam optimizer.
  • Train the network with train_data for 3 epochs with batches of 10 images each.
  • Use randomly selected 20% of the training data as validation data during training.
  • Evaluate the model with test_data, use a batch size of 10.
model.compile(optimizer='adam', 
              loss='categorical_crossentropy', 
              metrics=['accuracy'])

# Fit the model to training data 
model.fit(train_data, train_labels, 
          validation_split=0.2, 
          epochs=3, batch_size=10)

# Evaluate the model on test data
model.evaluate(test_data, test_labels, batch_size=10)

<script.py> output:
    Epoch 1/3

1/4 [======>.......................] - ETA: 3s - loss: 1.1531 - accuracy: 0.3000
4/4 [==============================] - 2s 172ms/step - loss: 1.0844 - accuracy: 0.5000 - val_loss: 0.9340 - val_accuracy: 1.0000
    Epoch 2/3

1/4 [======>.......................] - ETA: 0s - loss: 0.9334 - accuracy: 0.9000
4/4 [==============================] - 0s 10ms/step - loss: 0.8868 - accuracy: 0.9250 - val_loss: 0.7414 - val_accuracy: 1.0000
    Epoch 3/3

1/4 [======>.......................] - ETA: 0s - loss: 0.7673 - accuracy: 0.9000
4/4 [==============================] - 0s 9ms/step - loss: 0.7358 - accuracy: 0.9750 - val_loss: 0.5658 - val_accuracy: 1.0000

1/1 [==============================] - ETA: 0s - loss: 0.8316 - accuracy: 0.7000
1/1 [==============================] - 0s 15ms/step - loss: 0.8316 - accuracy: 0.7000

Accuracy calculated on the test data is not subject to overfitting.

3.4 What is special about a deep network?

Networks with more convolution layers are called "deep" networks, and they may have more power to fit complex data, because of their ability to create hierarchical representations of the data that they fit.

What is a major difference between a deep CNN and a CNN with only one convolutional layer?

Answer the question

50XP

Possible Answers

  • A deep network is inspired by the human visual system.

  • A deep network requires more data and more computation to fit.

  • A deep network has more dense layers.

  • A deep network has larger convolutions.

3.5 How many parameters?

3.6 How many parameters in a CNN?

We need to know how many parameters a CNN has, so we can adjust the model architecture, to reduce this number or shift parameters from one part of the network to another. How many parameters would a network have if its inputs are images with 28-by-28 pixels, there is one convolutional layer with 10 units kernels of 3-by-3 pixels, using zero padding (input has the same size as the output), and one densely connected layer with 2 units?

Instructions

50 XP

Possible Answers

  • 100

  • 1668

  • 15,782

  • 15,682

In [3]:
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Flatten

model = Sequential()

# Add a convolutional layer (10 units)
model.add(Conv2D(10, kernel_size=3, activation='relu', 
              input_shape=(28, 28, 1), 
              strides=1, padding='same'))


# Add another convolutional layer (5 units)
model.add(Flatten())
model.add(Dense(2, activation='softmax'))
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 28, 28, 10)        100       
_________________________________________________________________
flatten (Flatten)            (None, 7840)              0         
_________________________________________________________________
dense (Dense)                (None, 2)                 15682     
=================================================================
Total params: 15,782
Trainable params: 15,782
Non-trainable params: 0
_________________________________________________________________

3.7 How many parameters in a deep CNN?

n this exercise, you will use Keras to calculate the total number of parameters along with the number of parameters in each layer of the network.

We have already provided code that builds a deep CNN for you.

Instructions

100 XP

Summarize the network, providing a count of the number of parameters.

model = Sequential()
model.add(Conv2D(10, kernel_size=2, activation='relu', 
                 input_shape=(28, 28, 1)))
model.add(Conv2D(10, kernel_size=2, activation='relu'))
model.add(Flatten())
model.add(Dense(3, activation='softmax'))

# Summarize the model 
model.summary()

This model has 20,743 parameters!

<script.py> output:
    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    conv2d (Conv2D)              (None, 27, 27, 10)        50        
    _________________________________________________________________
    conv2d_1 (Conv2D)            (None, 26, 26, 10)        410       
    _________________________________________________________________
    flatten (Flatten)            (None, 6760)              0         
    _________________________________________________________________
    dense (Dense)                (None, 3)                 20283     
    =================================================================
    Total params: 20,743
    Trainable params: 20,743
    Non-trainable params: 0
    _________________________________________________________________

3.8 Pooling operations

3.9 Write your own pooling operation

As we have seen before, CNNs can have a lot of parameters. Pooling layers are often added between the convolutional layers of a neural network to summarize their outputs in a condensed manner, and reduce the number of parameters in the next layer in the network. This can help us if we want to train the network more rapidly, or if we don't have enough data to learn a very large number of parameters.

A pooling layer can be described as a particular kind of convolution. For every window in the input it finds the maximal pixel value and passes only this pixel through. In this exercise, you will write your own max pooling operation, based on the code that you previously used to write a two-dimensional convolution operation.

Instructions

100 XP

  • Index into the input array (im) and select the right window.
  • Find the maximum in this window.
  • Allocate this into the right entry in the output array (result).
result = np.zeros((im.shape[0]//2, im.shape[1]//2))

# Pooling operation
for ii in range(result.shape[0]):
    for jj in range(result.shape[1]):
        result[ii, jj] = np.max(im[ii*2:ii*2+2, jj*2:jj*2+2])

The resulting image is smaller, but retains the salient features in every location

3.10 Keras pooling layers

Keras implements a pooling operation as a layer that can be added to CNNs between other layers. In this exercise, you will construct a convolutional neural network similar to the one you have constructed before:

Convolution => Convolution => Flatten => Dense

However, you will also add a pooling layer. The architecture will add a single max-pooling layer between the convolutional layer and the dense layer with a pooling of 2x2:

Convolution => Max pooling => Convolution => Flatten => Dense

A Sequential model along with Dense, Conv2D, Flatten, and MaxPool2D objects are available in your workspace.

Instructions

100 XP

  • Add an input convolutional layer (15 units, kernel size of 2, relu activation).
  • Add a maximum pooling operation (pooling over windows of size 2x2).
  • Add another convolution layer (5 units, kernel size of 2, relu activation).
  • Flatten the output of the second convolution and add a Dense layer for output (3 categories, softmax activation).
model.add(Conv2D(15, kernel_size=2, activation='relu', 
                 input_shape=(img_rows, img_cols, 1)))

# Add a pooling operation
model.add(MaxPool2D(2))

# Add another convolutional layer
model.add(Conv2D(5, kernel_size=2, activation='relu'))

# Flatten and feed to output layer
model.add(Flatten())
model.add(Dense(3, activation='softmax'))
model.summary()

This model is even deeper, but has fewer parameters.

<script.py> output:
    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #   
    =================================================================
    conv2d (Conv2D)              (None, 27, 27, 15)        75        
    _________________________________________________________________
    max_pooling2d (MaxPooling2D) (None, 13, 13, 15)        0         
    _________________________________________________________________
    conv2d_1 (Conv2D)            (None, 12, 12, 5)         305       
    _________________________________________________________________
    flatten (Flatten)            (None, 720)               0         
    _________________________________________________________________
    dense (Dense)                (None, 3)                 2163      
    =================================================================
    Total params: 2,543
    Trainable params: 2,543
    Non-trainable params: 0
    _________________________________________________________________

3.11 Train a deep CNN with pooling to classify images

Training a CNN with pooling layers is very similar to training of the deep networks that y have seen before. Once the network is constructed (as you did in the previous exercise), the model needs to be appropriately compiled, and then training data needs to be provided, together with the other arguments that control the fitting procedure.

The following model from the previous exercise is available in your workspace:

Convolution => Max pooling => Convolution => Flatten => Dense

Instructions

100 XP

  • Compile this model to use the categorical cross-entropy loss function and the Adam optimizer.
  • Train the model for 3 epochs with batches of size 10.
  • Use 20% of the data as validation data.
  • Evaluate the model on test_data with test_labels (also batches of size 10).
model.compile(optimizer='adam', 
              loss='categorical_crossentropy', 
              metrics=['accuracy'])

# Fit to training data
model.fit(train_data, train_labels, 
          validation_split=0.2, 
          epochs=3, batch_size=10)

# Evaluate on test data 
model.evaluate(test_data, test_labels, batch_size=10)

This model is also very accurate!

<script.py> output:
    Epoch 1/3

1/4 [======>.......................] - ETA: 3s - loss: 1.1283 - accuracy: 0.2000
4/4 [==============================] - 2s 223ms/step - loss: 1.0961 - accuracy: 0.3750 - val_loss: 1.0882 - val_accuracy: 0.5000
    Epoch 2/3

1/4 [======>.......................] - ETA: 0s - loss: 1.0671 - accuracy: 0.5000
4/4 [==============================] - 0s 10ms/step - loss: 1.0618 - accuracy: 0.7000 - val_loss: 1.0560 - val_accuracy: 0.6000
    Epoch 3/3

1/4 [======>.......................] - ETA: 0s - loss: 1.0397 - accuracy: 0.9000
4/4 [==============================] - 0s 12ms/step - loss: 1.0248 - accuracy: 0.8500 - val_loss: 1.0239 - val_accuracy: 0.6000

1/1 [==============================] - ETA: 0s - loss: 1.0101 - accuracy: 0.6000
1/1 [==============================] - 0s 19ms/step - loss: 1.0101 - accuracy: 0.6000

4. Understanding and Improving Deep Convolutional Networks

There are many ways to improve training by neural networks. In this chapter, we will focus on our ability to track how well a network is doing, and explore approaches towards improving convolutional neural networks.

4.1 Tracking learning

4.2 Plot the learning curves

During learning, the model will store the loss function evaluated in each epoch. Looking at the learning curves can tell us quite a bit about the learning process. In this exercise, you will plot the learning and validation loss curves for a model that you will train.

Instructions

100 XP

  • Fit the model to the training data (train_data).
  • Use a validation split of 20%, 3 epochs and batch size of 10.
  • Plot the training loss.
  • Plot the validation loss.
import matplotlib.pyplot as plt

# Train the model and store the training object
training = model.fit(train_data, train_labels, validation_split=0.2, 
          epochs=3, batch_size=10 )

# Extract the history from the training object
history = training.history

# Plot the training loss 
plt.plot(history['loss'])
# Plot the validation loss
plt.plot(history['val_loss'])

# Show the figure
plt.show()

That's great! If you continue for many epochs, the validation loss will start going back up.

image.png

4.3 Using stored weights to predict in a test set

Model weights stored in an hdf5 file can be reused to populate an untrained model. Once the weights are loaded into this model, it behaves just like a model that has been trained to reach these weights. For example, you can use this model to make predictions from an unseen data set (e.g. test_data).

Instructions

100 XP

  • Load the weights from a file called 'weights.hdf5'.
  • Predict the classes of the first three images from test_data.
model.load_weights('weights.hdf5')

# Predict from the first three images in the test data
model.predict(test_data[0:3,:])

Nicely done! How would you use these weights to evaluate the model instead?

4.4 Regularization

4.5 Adding dropout to your network

Dropout is a form of regularization that removes a different random subset of the units in a layer in each round of training. In this exercise, we will add dropout to the convolutional neural network that we have used in previous exercises:

  1. Convolution (15 units, kernel size 2, 'relu' activation)
  2. Dropout (20%)
  3. Convolution (5 units, kernel size 2, 'relu' activation)
  4. Flatten
  5. Dense (3 units, 'softmax' activation)

A Sequential model along with Dense, Conv2D, Flatten, and Dropout objects are available in your workspace.

Instructions

100 XP

  • Add dropout applied to the first layer with 20%.
  • Add a flattening layer.
model.add(Conv2D(15, kernel_size=2, activation='relu', 
                 input_shape=(img_rows, img_cols, 1)))

# Add a dropout layer
model.add(Dropout(0.20))

# Add another convolutional layer
model.add(Conv2D(5, kernel_size=2, activation='relu'))

# Flatten and feed to output layer
model.add(Flatten())
model.add(Dense(3, activation='softmax'))

Great! Now the kernels will be more different from each other.

4.6 Add batch normalization to your network

Batch normalization is another form of regularization that rescales the outputs of a layer to make sure that they have mean 0 and standard deviation 1. In this exercise, we will add batch normalization to the convolutional neural network that we have used in previous exercises:

  1. Convolution (15 units, kernel size 2, 'relu' activation)
  2. Batch normalization
  3. Convolution (5 unites, kernel size 2, 'relu' activation)
  4. Flatten
  5. Dense (3 units, 'softmax' activation)

A Sequential model along with Dense, Conv2D, Flatten, and Dropout objects are available in your workspace.

Instructions

100 XP

  • Add the first convolutional layer. You can use the img_rows and img_cols objects available in your workspace to define the input_shape of this layer.
  • Add batch normalization applied to the outputs of the first layer.
model.add(Conv2D(15, kernel_size=2, activation='relu', 
                 input_shape=(img_rows, img_cols, 1)))


# Add batch normalization layer
model.add(BatchNormalization())

# Add another convolutional layer
model.add(Conv2D(5, kernel_size=2, activation='relu'))

# Flatten and feed to output layer
model.add(Flatten())
model.add(Dense(3, activation='softmax'))

That's useful! That should improve training.

4.7 Interpreting the model

4.8 Extracting a kernel from a trained network

One way to interpret models is to examine the properties of the kernels in the convolutional layers. In this exercise, you will extract one of the kernels from a convolutional neural network with weights that you saved in a hdf5 file.

Instructions

100 XP

  • Load the weights into the model from the file weights.hdf5.
  • Get the first convolutional layer in the model from the layers attribute.
  • Use the .get_weights() method to extract the weights from this layer.
model.load_weights('weights.hdf5')

# Get the first convolutional layer from the model
c1 = model.layers[0]

# Get the weights of the first convolutional layer
weights1 = c1.get_weights()

# Pull out the first channel of the first kernel in the first layer
kernel = weights1[0][:, :, 0, 0]
print(kernel)

That's great! You can extract the weights from other layers too.

4.9 Shape of the weights

A Keras neural network stores its layers in a list called model.layers. For the convolutional layers, you can get the weights using the .get_weights() method. This returns a list, and the first item in this list is an array representing the weights of the convolutional kernels. If the shape of this array is (2, 2, 1, 5), what does the first number (2) represent?

Instructions

50 XP

Possible Answers

  • There are 2 channels in black and white images.

  • The kernel size is 2 by 2. bold text

  • The model used a convolutional unit with 2 dimensions.

  • There are 2 convolutional layers in the network.

That's correct, each of the 2s in this shape is one of the dimensions of the kernel.

4.10 Visualizing kernel responses

One of the ways to interpret the weights of a neural network is to see how the kernels stored in these weights "see" the world. That is, what properties of an image are emphasized by this kernel. In this exercise, we will do that by convolving an image with the kernel and visualizing the result. Given images in the test_data variable, a function called extract_kernel() that extracts a kernel from the provided network, and the function called convolution() that we defined in the first chapter, extract the kernel, load the data from a file and visualize it with matplotlib.

A deep CNN model, a function convolution(), along with the kernel you extracted in an earlier exercise is available in your workspace.

Ready to take your deep learning to the next level? Check out Advanced Deep Learning with Keras to see how the Keras functional API lets you build domain knowledge to solve new types of problems.

Instructions

100 XP

  • Use the convolution() function to convolve the extracted kernel with the first channel of the fourth item in the image array.
  • Visualize the resulting convolution with imshow().
import matplotlib.pyplot as plt

# Convolve with the fourth image in test_data
out = convolution(test_data[3, :, :, 0], kernel)

# Visualize the result
plt.imshow(out)
plt.show()

That's nice. You can keep going and visualize the kernel responses for all the kernels in this layer!

image.png

4.11 Next steps