CNN | Haobin Tan

Convolutional Neural Network (CNN)

Wed, 19 Aug 2020 00:00:00 +0000

Time-Delay Neural Network (TDNN)

Sun, 16 Aug 2020 00:00:00 +0000

Motivation

Ensure shift-invariance

The model should produce the same output regardless of the position of the considering object

Overview

Multilayer Neural Network: Nonlinear Classifier
Consider Context (Receptive Field)
Shift-Invariant Learning
- All Units Learn to Detect Patterns Independent of Location in Time
- No Pre-segmentation or Pre-alignment Necessary
- Approach: Weight Sharing
Time-Delay Arrangement
- Networks can represent temporal structure of speech
Translation-Invariant Learning
- Hidden units of the network learn features independent of precise location in time

Structure

Input: spectrum of a speech
- $x$-axis: time
- $y$-axis: frequency

How TDNN works?

Input layer $\to$ Hidden layer

Hidden layer 1 $\to$ Hidden layer 2

As this input flows by, we have these hidden units generated activations over time as activation patterns.
Then we can take a contextual window of activation patterns over time and feed them into neurons in the second hidden layer

Hidden layer $\to$ Output layer

We assemble all the evidence from activations over time and integrate them into one joint output

Shift-Invariance Training

Connections with the same color share the same weight.

Demo

TDNN / Convolutional Nets - Demo

TDNN’s→Convolutional Nets

In Vision the same problem:

Local Contexts – Global Integration – Shared Weights

TDNN is equivalent to 1-dimensional CNN

TDNN Parameters Calculation

Exam WS1819, Task 4.1

Input: matrix of the dimension 16 × 15
- Chunks of a sequence of 15 frames
- Each frame is a feature vector of 16 real numbers
3 subsequent frames are connected to a 1 frame in the first hidden layer in a shift-invariant path, i.e. these connection weights of this shift-invariant matrix are shared.
A similar approach is used for the second hidden layer
No bias for the first and second hidden layer
The output layer is a layer that connects each row of outputs from the previous layer, followed by a soft-max that calculates the probabilities of each letter.
Possible output: 26 letters (a - z) , <blank>, <space>

Question: Number of parameters?

Reference

👍 Convolutional Neural Network (CNN) Basics

Wed, 19 Aug 2020 00:00:00 +0000

Architecture Overview

All CNN models follow a similar architecture

Input
Convolutional layer (Cons-layer) + ReLU
Pooling layer (Pool-layer)
Fully Connected layer (FC-layer)
Output

Input

The input layer represents the input image into the CNN. Essentially, every image can be represented as a matrix of pixel values.

Channel is a conventional term used to refer to a certain component of an image.

Grayscale image: has just one channel
RGB images
- Three channels: Red, Green, Blue
- Imagine those as three 2d-matrices stacked over each other (one for each color), each having pixel values in the range 0 to 255.

We can consider channel as depth of the image.

Convolutional Layer

Convolution operation

Extract features from the input image and produce feature maps
1. Slide the convonlutional filter/kernel over the input image
2. At every location, do element-wise matrix multiplication and sum the result.
This can preserve the spatial relationship between pixels by learning image features using small squares of input data 👍

2D Convolution

Convolution operation in 2D using a $3\times3$ filter

Another example:

3D Convolution

In reality an image is represented as a 3D matrix with dimensions of height, width and depth, where depth corresponds to color channels (RGB). A convolution filter has a specific height and width, like $3 \times 3$ or $5 \times 5$, and by design it covers the entire depth of its input ($\text{depth}\_{\text{filter}} = \text{depth}\_{\text{input}}$).

The convolution filter/kernel has the same depth as the input image

Convolution using a single filter:

Convolution using a single filter

Each filter actually happens to be a collection of kernels, with there being one kernel for every single input channel to the layer, and each kernel being unique. As the input image has 3 channels (RGB), our filter consists of also 3 kernels.

Each of the kernels of the filter “slides” over their respective input channels, producing a processed version of each.

Each of the per-channel processed versions are then summed together to form one channel. The kernels of a filter each produce one version of each channel, and the filter as a whole produces one overall output channel.

We can stack different filters to obtain a multi-channel output “image”.

For example, assuming that

input image has the size $\text{height} \times \text{width} \times \text{depth} = 32 \times 32 \times 3$
filter size is $5 \times 5 \times 3$
and we have 6 different filters

$\to$ we’ll get 6 separate activation maps and stack it together

$\Rightarrow$ The depth of the multi-channel output “image” is 6.

($depth\_\text{activation maps} = \\# filters$)

Convolution Example

A filter (with red outline) slides over the input image (convolution operation) to produce a feature map. The convolution of another filter (with the green outline), over the same image gives a different feature map as shown. It is important to note that the Convolution operation captures the local dependencies in the original image. Also notice how these two different filters generate different feature maps from the same original image.

Non-linearity: ReLU

For any kind of neural network to be powerful, it needs to contain non-linearity. And CNN is no different.

After the convolution operation, we pass the result through non-linear activation function. In CNN we usually use Rectified Linear Units (ReLU), because it has been empirically observed that CNNs using ReLU are faster to train than their counterparts.

$$ \operatorname{ReLU}(x) = \max(0, x) $$

ReLU Example

This example shows the ReLU operation applied to one of the fearure maps obtained in the convolutional example. The output feature map here is also referred to as the ‘Rectified’ feature map.

Stride and Padding

Stride specifies how much we move the convolution filter at each step.

By default the value is 1:

Stride > 1often used to down-sample the image

Stride 2 convolution

What do we do with border pixels?

$\to$ Paddings

Fill up the image borders (zero-padding is most common)
Preserve the size of the feature maps from shrinking
Improves performance and makes sure the kernel and stride size will fit in the input

Height and width of feature map is same as the input image due to padding (the gray area).

Dimension parameters computation

Inpupt size:
$$W\_{1} \times H\_{1} \times D\_{1}$$
(usually $W\_1 = H\_1$)
Hyperparameters:
- Number of filters: $K$
- Filter size: $F \times F \times D\_1$
- Stride: $S$
- Amount of padding: $P$
Output size:

$$ W\_{2} \times H\_{2} \times K $$
with
- $W_{2}=\lfloor \frac{W_{1}-F+2 P}{S}+1 \rfloor$
- $H_{2}=\lfloor \frac{H_{1}-F+2 P}{S}+1 \rfloor$
Number of weights:
$$ \text{#weights} = \underbrace{F \cdot F}\_{\text {Filter size }} \cdot \underbrace{D\_{1}}_{\text {Filter depth }} \cdot \underbrace{K}\_{\text {#Filters }} $$

Connections Calculation

$$ \text{#Connections} = \text{#Neurons of next layer} \times \text{filter size} $$

Nice explanation from cs231n:

Summary of Conv-layer

Convolution operation using filters
Feed into ReLU

Pooling Layer

How Pooling works?

After a convolution operation we usually perform pooling to reduce the dimensionality.

Pooling layers downsample each feature map independently, reducing the height and width, keeping the depth intact. This enables us to reduce the number of parameters, which both shortens the training time and combats overfitting. 👏

The most common type of pooling is max pooling which just takes the max value in the pooling window. Contrary to the convolution operation, pooling has NO parameters. It slides a window over its input, and simply takes the max value in the window. Similar to a convolution, we specify the window size and stride.

Example: max pooling using a $2 \times 2$ window and stride 2

Now let’s work out the feature map dimensions before and after pooling.

If the input to the pooling layer has the dimensionality $32 \times 32 \times 10$, using the same pooling parameters described above, the result will be a $16 \times 16 \times 10$ feature map.

Both the height and width of the feature map are halved. Thus we reduced the number of weights to 1/4 of the input.

The depth doesn’t change because pooling works independently on each depth slice the input.

In CNN architectures, pooling is typically performed with 2x2 windows, stride 2 and NO padding.

Pooling Example

This example shows the effect of Pooling on the Rectified Feature Map we received after the ReLU operation in the above ReLU example. Max refers to Max-Pooling and Sum refers to Sum-Pooling.

Why pooling works?

Because Pooling keeps the maximum value from each window, it preserves the best fits of each feature within the window. This means that it doesn’t care so much exactly where the feature fit as long as it fit somewhere within the window.

The result of this is that CNNs can find whether a feature is in an image without worrying about where it is. This helps solve the problem of computers being hyper-literal.

In particular, Pooling

makes the input representations (feature dimension) smaller and more manageable
reduces the number of parameters and computations in the network, therefore, controlling overfitting
makes the network invariant to small transformations, distortions and translations in the input image (a small distortion in input will not change the output of Pooling – since we take the maximum / average value in a local neighborhood).
helps us arrive at an almost scale invariant representation of our image

Dimension parameters computation

Inpupt size:
$$W\_{1} \times H\_{1} \times D\_{1}$$
(usually $W\_1 = H\_1$)
Hyperparameters:
- Number of filters: $K$
- Filter size: $F \times F \times D\_1$
- Stride: $S$
- Typically no padding
Output size:

$$ W\_{2} \times H\_{2} \times D\_1 $$
with
- $W_{2}=\lfloor \frac{W_{1}-F}{S}\rfloor+1 $
- $H_{2}=\lfloor \frac{H_{1}-F}{S}\rfloor+1 $
Number of weights: 0 (since it computes a fixed function of the input)

Fully Connected Layer

After the convolution + pooling layers we add a couple of fully connected layers to wrap up the CNN architecture.

The Fully Connected layer is a traditional MultiLayer Perceptron (MLP) that uses a softmax activation function in the output layer (other classifiers like SVM can also be used). The term “Fully Connected” implies that every neuron in the previous layer is connected to every neuron on the next layer.

The output from the convolutional and pooling layers represent high-level features of the input image. The purpose of the Fully Connected layer is to use these features for classifying the input image into various classes based on the training dataset.

Remember that the output of both convolution and pooling layers are 3D volumes, but a fully connected layer expects a 1D vector of numbers. So we flatten the output of the final pooling layer to a vector and that becomes the input to the fully connected layer. Flattening is simply arranging the 3D volume of numbers into a 1D vector, nothing fancy happens here.

Apart from classification, adding a fully-connected layer is also a (usually) cheap way of learning non-linear combinations of these features. Most of the features from convolutional and pooling layers may be good for the classification task, but combinations of those features might be even better.

✅ Advantages of CNN (vs. MLP)

CNNs are good for translation invariance
CNN reduces the numbers of parameters
- Locally connected, shared weights, pooling, local feature extractor
- But learning power is still good or even better (generalization)
We can “resize” the next layer to as we want
- By setting kernel size, number of kernel, padding, stride
Design of good architecture based on intuitions (or Neural architecture search)

Reference

👍 CNN Intuition and Visualization

Wed, 19 Aug 2020 00:00:00 +0000

Intuition

A CNN model can be thought as a combination of two components:

feature extraction part

The convolution + pooling layers perform feature extraction. For example given an image, the convolution layer detects features such as two eyes, long ears, four legs, a short tail and so on.
classification part

The fully connected layers then act as a classifier on top of these features, and assign a probability for the input image being a dog.

The convolution layers are the main powerhouse of a CNN model. Automatically detecting meaningful features given only an image and a label is not an easy task. The convolution layers learn such complex features by building on top of each other. The first layers detect edges, the next layers combine them to detect shapes, to following layers merge this information to infer that this is a nose. To be clear, the CNN doesn’t know what a nose is. By seeing a lot of them in images, it learns to detect that as a feature. The fully connected layers learn how to use these features produced by convolutions in order to correctly classify the images.

Visualization

Let’s say this is our originial input image:

And we will use VGG as our CNN architectures.

VGG16 structure

We will visualize 3 components of the VGG model:

Feature maps
Convnet filters
Class output

Feature Maps Visualization

Recap of CONV-layer:

Filter operates on the input performing the convolution operation and as a result we get a feature map.
We use multiple filters and stack the resulting feature maps together to obtain an output volume.

We will visualize the feature maps to see how the input is transformed passing through the convolution layers. The feature maps are also called intermediate activations since the output of a layer is called the activation.

VGG convolutional layers are named as: blockX_convY. For example the second filter in the third convolution block is called block3_conv2.

Now let’s visualize the feature maps corresponding to the first convolution of each block, the red arrows in the figure below.

The following figure displays the first 8 feature maps per layer. Notice that there’re more than 8 feature maps per layer.

Looking at one feature map per layer, we can obtain some interesting observations:

The first layer feature maps (block1_conv1) retain most of the information present in the image. In CNN architectures the first layers usually act as edge detectors.
As we go deeper into the network, the feature maps look less like the original image and more like an abstract representation of it.
- In block3_conv1 the cat is somewhat visible, but after that it becomes unrecognizable.
- The reason is that deeper feature maps encode high level concepts like “cat nose” or “dog ear” while lower level feature maps detect simple edges and shapes. That’s why deeper feature maps contain less information about the image and more about the class of the image. They still encode useful features, but they are less visually interpretable by us.
- The feature maps become sparser as we go deeper, meaning the filters detect less features.
  - It makes sense because the filters in the first layers detect simple shapes, and every image contains those.
  - But as we go deeper we start looking for more complex stuff like “dog tail” and they don’t appear in every image. That’s why in the first figure with 8 filters per layer, we see more of the feature maps as blank as we go deeper (block4_conv1 and block5_conv1).

CONV Filters and Class Output Visualization

Check out this great article: Applied Deep Learning - Part 4: Convolutional Neural Networks

Interactive CNN Visualization

For real-time, dynamic, and interactive CNN visualization, I highly recommend CNN Explainer.

Reference

Applied Deep Learning - Part 4: Convolutional Neural Networks

CNN History

Thu, 20 Aug 2020 00:00:00 +0000

LeNet (1998)

Image followed by multiple convolutional / pooling layers
- Build up hierarchical filter structures
- Subsampling / pooling increases robustness
Fully connected layers towards the end
- Brings all information together, combines it once more
Output layer of 10 units, one for each digit class

ImageNet Dataset (2009)

Standard benchmark for vision:

1.2 M images
1000 classes
> 500 images per class

ImageNet Large Scale Visual Recognition Challenge (ILSVRC)

ILSVRC Classification Task
- 1000 object classes
- 1.2 million training images (732 – 1300 per class)
- 50 thousand validation images (50 per class)
- 100 thousand test images (100 per class)

AlexNet (2012)

Multiple convolutional layers
Couple fully connected (dense) layers
Final classification using a “soft-max” layer
Train end-to-end via back propagation
Details
- first use of ReLU
- used Norm layers (not common anymore)
- heavy data augmentation
- dropout 0.5
- batch size 128
- SGD Momentum 0.9
- Learning rate 1e-2, reduced by a factor of 10 manually when val accuracy plateaus
- L2 weight decay 5e-4

VGG Net (2014)

Small filters, Deeper networks

8 layers (AlexNet) -> 16 - 19 layers (VGG16Net)
Only 3x3 CONV stride 1, pad 1
and 2x2 MAX POOL stride 2

ResNet (2015)

Residual blocks

How can we train such deep networks?

Solution: Use network layers to fit a residual mapping instead of directly trying to fit a

desired underlying mapping

Use layers to fit residual $F(x) = H(x) – x$ instead of $H(x)$ directly
Initially, $F(x)$ is set to 0, so the layer just computes the identity
I.e. adding more layers does not harm 👏

ResNet Architecture

Stack residual blocks
Every residual block has two 3x3 conv layers
Periodically, double # of filters and downsample spatially using stride 2 (/2 in each dimension)
Additional conv layer at the beginning
No FC layers at the end (only FC 1000 to output classes)

Training ResNet in practice

Batch Normalization after every CONV layer (not covered)
Xavier 2/ initialization from He et al.
SGD + Momentum (0.9)
Learning rate: 0.1, divided by 10 when validation error plateaus
Mini-batch size 256
Weight decay of 1e-5
No dropout used

Transfer Learning

ImageNet has 1.2 million images! Typically, we do not have that many! Can we also use these methods with less images?

Yes! With transfer learning!

Features (conv layers) are generic and can be reused!

How?

Train on huge data-set (e.g. Imagenet)
Freeze layers and adapt only last (FC) layers

Pratical Advice

Very little data, very similar dataset:
- Use Linear classifier on top layer
Very little data, very different dataset:
- You’re in trouble… Try linear classifier from different stages and pray 🤪
A lot of data, very similar dataset:
- Finetune a few layers
A lot of data, very different dataset:
- Finetune a larger number of layers

Example: Image Captioning

Example: Face Recognition

Siamese Networks (FaceNet)
Distance
$$ d\left(x\_{1}, x\_{2}\right)=\left\|f\left(x\_{1}\right)-f\left(x\_{2}\right)\right\|\_{2}^{2} $$
- If $d(x\_1, x\_2)$ small: same person
- Otherwise different person
Training: Triplet loss
$$ L(A, P, N)=\max \left(\|f(A)-f(P)\|^{2}-\|f(A)-f(N)\|^{2}+\alpha, 0\right) $$

Computer Vision

Thu, 20 Aug 2020 00:00:00 +0000

Computer Vision (CV) Tasks

Classification
Classification + Localization
Object Detection
Instance Segmentation

Object Localization: Coordinate prediction

Sliding Window

Object Localization

Classification & Localization

Detection

Sliding Window + Classification:

Regioning

Sliding Window Problem: Need to test many positions and scales, and use a computationally demanding classifier
Solution: Only look at a tiny subset of possible positions
- Regioning => propose image regions that are likely to contain objects
- Classify individual regions and correct regions
- R-CNN -> Fast R-CNN -> Faster R-CNN

R-CNN

Propose approx. 2k different regions (bounding boxes) for image classification
For each box, do image classification with CNN
- Discard unlikely boxes
Refine bounding boxes with regression

Fast R-CNN

9x faster training, 213x faster test time
R-CNN is not end to end (first train softmax classifier, use that for training bounding box regressor)
Similar to R-CNN
- Apply Region Proposals on feature map result of applied CNN to input image
- Reshape region proposals on feature map into fixed size
- Feed into FC layer

Faster R-CNN

Both R-CNN and R-CNN rely on Selective Search for region proposals -> most time consuming part 🤪
Use a seperate Network for predicting the regions of interest 💪

YOLO

You Only Look Once: Unified Real-Time Object Detection
„Simple network“, directly from pixels to bounding box / object detection / class prediction

Image Segmentation

Grouping Pixels into regions that belong to same properties
Eg: Segmenting an Image into meaningful objects

Semantic Segmentation

Sliding Window

Label each pixel in image with a category label
Don‘t differentiate instances, only care about pixels
=> just extract small patches from an image and classify center pixel with a normal CNN classifier
Problem: very inefficient

Fully convolutional

Keep the network as an end to end convolutional Neural Network
Predictions are made for all pixels at once
Convolutions at original image resolution are very expensive

Reference

Object Detection for Dummies Part 3: R-CNN Family

CNN Resources

Wed, 19 Aug 2020 00:00:00 +0000

Tutorials

Visualization

Plotting

NN-SVG

Papers Overview

The 9 Deep Learning Papers You Need To Know About (Understanding CNNs Part 3)