Nerual Network Basics | Haobin Tan

Neural Network Basics

Fri, 31 Jul 2020 00:00:00 +0000

Perceptron

Tue, 01 Sep 2020 00:00:00 +0000

Structure

A perceptron is

a single-layer neural network
used for supervised learning of binary classifiers

Perceptron

$$ g(x) = \underbrace{\sum\_{i=0}^n w\_i x\_i}\_{\text{linear separator}} + \underbrace{w\_0}\_{\text{offset/bias}} $$

Decision for classification

$$ \hat{y} = \begin{cases} 1 &\text{if } g(x) > 0 \\\\ -1 &\text{else}\end{cases} $$

Update Rule

$w=w+y x$ if prediction is wrong

If label $y=1$ but predict $\hat{y}=-1$: $w = w + x$
If label $y=-1$ but predict $\hat{y}=1$: $w = w - x$

👍 Activation Functions

Mon, 17 Aug 2020 00:00:00 +0000

Activation functions should be

non-linear
differentiable (since training with Backpropagation)

Q: Why can’t the mapping between layers be linear?

A: Compositions of linear functions is still linear, whole network collapses to regression.

Sigmoid

$$ \sigma(x)=\frac{1}{1+\exp (-x)} $$

Squashes numbers to range $[0,1]$
✅ Historically popular since they have nice interpretation as a saturating “firing rate” of a neuron
⛔️ Problems
- Vanishing gradients: functions gradient at either tail of $1$ or $0$ is almost zero
- Sigmoid outputs are not zero-centered (important for initialization)
- $\exp()$ is a bit compute expensive
Derivative
$$ \frac{d}{dx} \sigma(x) = \sigma(x)(1 - \sigma(x)) $$
(See: Derivative of Sigmoid Function)

Python implementation

def sigmoid(x):
 return 1 / (1 + np.exp(-x))

Derivative
```
def dsigmoid(y):
 return y * (1 - y)
```

Tanh

Squashes numbers to range $[-1,1]$
✅ zero centered (nice) 👏
⛔️ Vanishing gradients: still kills gradients when saturated
Derivative:
$$ \frac{d}{dx}\tanh(x) = 1 - [\tanh(x)]^2 $$
```
def dtanh(y):
 return 1 - y * y
```

Rectified Linear Unit (ReLU)

$$ f(x) = \max(0, x) $$

✅ Advantages
- Does not saturate (in $[0, \infty]$)
- Very computationally efficient
- Converges much faster than sigmoid/tanh in practice
⛔️ Problems
- Not zero-centred output
- No gradient for $x < 0$ (dying ReLU)

Python implementation

import numpy as np

def ReLU(x):
 return np.maximum(0, x)

Leaky ReLU

$$ f(x) = \max(0.1x, x) $$

Parametric Rectifier (PReLu)
$$ f(x) = \max(\alpha x, x) $$
- Also learn $\alpha$
✅ Advantages
- Does not saturate
- Computationally efficient
- Converges much faster than sigmoid/tanh in practice!
- will not “die”

Python implementation

import numpy as np

def ReLU(x):
 return np.maximum(0.1 * x, x)

Exponential Linear Units (ELU)

$$ f(x) = \begin{cases} x &\text{if }x > 0 \\\\ \alpha(\exp (x)-1) & \text {if }x \leq 0\end{cases} $$

✅ Advantages
- All benefits of ReLU
- Closer to zero mean outputs
- Negative saturation regime compared with Leaky ReLU (adds some robustness to noise)
⛔️ Problems
- Computation requires $\exp()$

Maxout

$$ f(x) = \max \left(w\_{1}^{T} x+b\_{1}, w\_{2}^{T} x+b\_{2}\right) $$

Generalizes ReLU and Leaky ReLU
- ReLU is Maxout with $w\_1 =0$ and $b\_1 = 0$
✅ Fixes the dying ReLU problem
⛔️ Doubles the number of parameters

Softmax

Softmax: probability that feature $x$ belongs to class $c\_k$ $$ o\_k = \theta\_k^Tx \qquad \forall k = 1, \dots, j $$

$$ p\left(y=c\_{k} \mid x ; \boldsymbol{\theta}\right)= p\left(c\_{k} = 1 \mid x ; \boldsymbol{\theta}\right) = \frac{e^{o\_k}}{\sum\_{j} e^{o\_j}} $$

Derivative: $$ \frac{\partial p(\hat{\mathbf{y}})}{\partial o\_{j}} =y\_{j}-p\left(\hat{y}\_{j}\right) $$

Advice in Practice

Use ReLU
- Be careful with your learning rates / initialization
Try out Leaky ReLU / ELU / Maxout
Try out tanh but don’t expect much
Don’t use sigmoid

Summary and Overview

See: Wiki-Activation Function

👍 Loss Functions

Mon, 17 Aug 2020 00:00:00 +0000

Quantifies what it means to have a “good” model
Different types of loss functions for different tasks, such as:
- Classification
- Regression
- Metric Learning
- Reinforcement Learning

Classification

Classification: Predicting a discrete class label
Negative log-likelihood loss (per sample $x$) / Cross-Entropy loss
$$ L(\boldsymbol{x}, y)=-\sum\_{j} y_{j} \log p\left(c\_{j} \mid \boldsymbol{x}\right) $$
- Used in various multiclass classification methods for NN training
Hinge Loss: used in Support Vector Machines (SVMs)
$$ L(x, y)=\sum\_{j} \max \left(0,1-x\_{i} y\_{i}\right) $$

Regression

Regression: Predicting a one or multiple continuous quantities $y_1, \dots, y\_n$
Goal: Minimize the distance between the predicted value $\hat{y}\_j$ and true values $y_j$
L1-Loss (Mean Average Error)
$$ L(\hat{y}, y)=\sum\_{j}\left(\hat{y}\_{j}-x\_{j}\right) $$
L2-Loss (Mean Square Error, MSE)
$$ L(\hat{y}, y)=\sum\_{j}\left(\hat{y}\_{j}-x\_{j}\right)^2 $$

Metric Learning / Similarity Learning

A model for measuring the distance (or similarity) between objects
Triplet Loss

$$ \sum_{(a, p, n) \in T} \max \left\\{0, \alpha-\left\|\mathbf{x}\_{a}-\mathbf{x}\_{n}\right\|\_{2}^{2}+\left\|\mathbf{x}\_{a}-\mathbf{x}\_{p}\right\|\_{2}^{2}\right\\} $$

Multilayer Perceptron and Backpropagation

Mon, 17 Aug 2020 00:00:00 +0000

Multi-Layer Perceptron (MLP)

Input layer $I \in R^{D\_{I} \times N}$
- How we initially represent the features
- Mini-batch processing with $N$ inputs
Weight matrices
- Input to Hidden: $W\_{H} \in R^{D\_{I} \times D\_{H}}$
- Hidden to Output: $W\_{O} \in R^{D\_{H} \times D\_{O}}$
Hidden layer(s) $H \in R^{D\_{H} \times N}$
$$ H = W\_{H}I + b\_H $$ $$ \widehat{H}\_j = f(H\_j) $$
- $f$: non-linear activation function
Output layer $O \in R^{D\_{O} \times N}$
- The value of the target function that the network approximates $$ O = W\_O \widehat{H} + b\_O $$

Backpropagation

Loss function
$$ L = \frac{1}{2}(O - Y)^2 $$
- Achieve minimum $L$: Stochastic Gradient Descent
  1. Calculate $\frac{\delta L}{\delta w}$ for each parameter $w$
  2. Update the parameters
    $$ w:=w-\alpha \frac{\delta L}{\delta w} $$
Compute the gradients: Backpropagation (Backprop)
- Output layer
$$ \begin{aligned} \frac{\delta L}{\delta O} &=(O-Y) \\\\ \\\\ \frac{\delta L}{\delta \widehat{H}} &=W\_{O}^{T} \color{red}{\frac{\delta L}{\delta O}} \\\\ \\\\ \frac{\delta L}{\delta W\_{O}} &= {\color{red}{\frac{\delta L}{\delta O} }}\widehat{H}^{T} \\\\ \\\\ \frac{\delta L}{\delta b\_{O}} &= \color{red}{\frac{\delta L}{\delta O}} \end{aligned} $$
(Red: terms previously computed)
- Hidden layer (assuming $\widehat{H}=\operatorname{sigmoid}(H)$ )
  $$ \frac{\delta L}{\delta H}={\color{red}{\frac{\delta L}{\delta \hat{H}}}} \odot \widehat{H} \odot(1-\widehat{H}) $$
  ($\odot$: element-wise multiplication)
- Input layer
$$ \begin{array}{l} \frac{\delta L}{\delta W\_{H}}={\color{red}{\frac{\delta L}{\delta H}}} I^{T} \\\\ \\\\ \frac{\delta L}{\delta b\_{H}}={\color{red}{\frac{\delta L}{\delta H}}} \end{array} $$

Gradients for vectorized operations

When dealing with matrix and vector operations, we must pay closer attention to dimensions and transpose operations.

Matrix-Matrix multiply gradient. Possibly the most tricky operation is the matrix-matrix multiplication (which generalizes all matrix-vector and vector-vector) multiply operations:

(Example from cs231n)

# forward pass
W = np.random.randn(5, 10)
X = np.random.randn(10, 3)
D = W.dot(X)

# now suppose we had the gradient on D from above in the circuit
dD = np.random.randn(*D.shape) # same shape as D
dW = dD.dot(X.T) #.T gives the transpose of the matrix
dX = W.T.dot(dD)

Tip: use dimension analysis!

Note that you do not need to remember the expressions for dW and dX because they are easy to re-derive based on dimensions.

For instance, we know that the gradient on the weights dW must be of the same size as W after it is computed, and that it must depend on matrix multiplication of X and dD (as is the case when both X,W are single numbers and not matrices). There is always exactly one way of achieving this so that the dimensions work out.

For example, X is of size [10 x 3] and dD of size [5 x 3], so if we want dW and W has shape [5 x 10], then the only way of achieving this is with dD.dot(X.T), as shown above.

For discussion of math details see: Not understanding derivative of a matrix-matrix product.

Softmax and Its Derivative

Tue, 08 Sep 2020 00:00:00 +0000

Softmax

We use softmax activation function to predict the probability assigned to $n$ classes. For example, the probability of assigning input sample to $j$-th class is:

$$ p\_j = \operatorname{softmax}(z\_j) = \frac{e^{z\_j}}{\sum\_{k=1}^n e^{z\_k}} $$

Furthermore, we use One-Hot encoding to represent the groundtruth $y$, which means

$$ \sum\_{k=1}^n y\_k = 1 $$

Loss function (Cross-Entropy):

$$ \begin{aligned} L &= -\sum\_{k=1}^n y\_k \log(p\_k) \\\\ &= - \left(y\_j \log(p\_j) + \sum\_{k \neq j}y\_k \log(p\_k)\right) \end{aligned} $$

Gradient w.r.t $z\_j$:

$$ \begin{aligned} \frac{\partial}{\partial z\_j}L &= \frac{\partial}{\partial z\_j} \left(-\sum\_{k=1}^n y\_k \log(p\_k)\right) \\\\ &= -\frac{\partial}{\partial z\_j} \left(y\_j \log(p\_j) + \sum\_{k \neq j}y\_k \log(p\_k)\right) \\\\ &= -\left(\frac{\partial}{\partial z\_j} y\_j \log(p\_j) + \frac{\partial}{\partial z\_j}\sum\_{k \neq j}y\_k \log(p\_k)\right) \end{aligned} $$

$k=j$
$$ \begin{aligned} \frac{\partial}{\partial z\_j} y\_j \log(p\_j) &= \frac{y\_j}{p\_j} \cdot \left(\frac{\partial}{\partial z\_j} p\_j\right) \\\\ &= \frac{y\_j}{p\_j} \cdot \left(\frac{\partial}{\partial z\_j} \frac{e^{z\_j}}{\sum\_{k=1}^n e^{z\_k}}\right) \\\\ &= \frac{y\_j}{p\_j} \cdot \frac{(\frac{\partial}{\partial z\_j}e^{z\_j})\sum\_{k=1}^n e^{z\_k} - e^{z\_j}(\frac{\partial}{\partial z\_j}\sum\_{k=1}^n e^{z\_k}) }{(\sum\_{k=1}^n e^{z\_k})^2} \\\\ &= \frac{y\_j}{p\_j} \cdot \frac{e^{z\_j}\sum\_{k=1}^n e^{z\_k} - e^{z\_j}e^{z\_j}}{(\sum\_{k=1}^n e^{z\_k})^2} \\\\ &= \frac{y\_j}{p\_j} \cdot \frac{e^{z\_j}(\sum\_{k=1}^n e^{z\_k} - e^{z\_j})}{(\sum\_{k=1}^n e^{z\_k})^2} \\\\ &= \frac{y\_j}{p\_j} \cdot \underbrace{\frac{e^{z\_j}}{\sum\_{k=1}^n e^{z\_k}}}\_{=p\_j} \cdot \frac{\sum\_{k=1}^n e^{z\_k} - e^{z\_j}}{\sum\_{k=1}^n e^{z\_k}}\\\\ &= \frac{y\_j}{p\_j} \cdot p\_j \cdot \underbrace{\left( \frac{\sum\_{k=1}^n e^{z\_k} }{\sum\_{k=1}^n e^{z\_k}} - \frac{e^{z\_j} }{\sum\_{k=1}^n e^{z\_k}}\right)}\_{=1 - p\_j} \\\\ &= y\_j(1-p\_j) \end{aligned} $$
$\forall k \neq j$
$$ \begin{aligned} \frac{\partial}{\partial z\_j}\sum\_{k \neq j}y\_k \log(p\_k) &= \sum\_{k \neq j} \frac{\partial}{\partial z\_j}y\_k \log(p\_k) \\\\ &= \sum\_{k \neq j} \frac{y\_k}{p\_k} \cdot \frac{(\overbrace{\frac{\partial}{\partial z\_j} e^{z\_k}}^{=0})(\sum\_i e^{z\_i}) - e^{z\_k}(\overbrace{\frac{\partial}{\partial z\_j}\sum\_i e^{z\_i}}^{=e^{z\_j}})}{(\sum\_{k=1}^n e^{z\_k})^2} \\\\ &= \sum\_{k \neq j} \frac{y\_k}{p\_k} \frac{-e^{z\_k} e^{z\_j}}{(\sum\_{k=1}^n e^{z\_k})^2} \\\\ &= -\sum\_{k \neq j} \frac{y\_k}{p\_k} \underbrace{\frac{e^{z\_k}}{\sum\_{k=1}^n e^{z\_k}}}\_{=p\_k} \underbrace{\frac{e^{z\_j}}{\sum\_{k=1}^n e^{z\_k}}}\_{=p\_j} \\\\ &= -\sum\_{k \neq j}y\_kp\_j \end{aligned} $$

Therefore,

$$ \begin{aligned} \frac{\partial}{\partial z\_j}L &= -\left(\frac{\partial}{\partial z\_j} y\_j \log(p\_j) + \frac{\partial}{\partial z\_j}\sum\_{k \neq j}y\_k \log(p\_k)\right) \\\\ &= -\left(y\_j(1-p\_j) - \sum\_{k \neq j}y\_kp\_j\right) \\\\ &= -\left(y\_j-y\_jp\_j - \sum\_{k \neq j}y\_kp\_j\right) \\\\ &= -\left(y\_j- (y\_jp\_j + \sum\_{k \neq j}y\_kp\_j)\right) \\\\ &= -\left(y\_j- \sum\_{k=1}^ny\_kp\_j\right)\\\\ &\overset{\sum\_{k=1}^{n} y\_k = 1}{=} -\left(y\_j- p\_j\right) \\\\ &= p\_j - y\_j \end{aligned} $$

Useful resources

Generalization

Mon, 17 Aug 2020 00:00:00 +0000

Generalization: Ability to Apply what was learned during Training to new (Test) Data

Reasons for bad generalization

Overfitting/Overtraining (trained too long)
Too little training material
Too many Parameters (weights) or inappropriate network architecture error

Prevent Overfitting

The obviously best approach: Collect More Data! 💪
If Data is Limited
- Simplest Method for Best Generalization: Early Stopping
- Optimize Parameters/Arcitecture
  - Architectural Learning
  - Choose Best Architecture by Repeated Experimentation on Cross Validation Set
  - Reduce Architecture Starting from Large
  - Grow Architecture Starting from Small

Destructive Methods

Reduce Complexity of Network through Regularization

Weight Decay
Weight Elimination
Optimal Brain Damage
Optimal Brain Surgeon

Optimal Brain Damage

💡Idea: Certain connections are removed from the network to reduce complexity and to avoide overfitting
Remove those connections that have the least effect on the Error (MSE, ..), i.e. are the least important.
- But this is time consuming (difficult) 🤪

Constructive Methods

Iteratively Increasing/Growing a Network (construktive) starting from a very small one

Cascade Correlation
Meiosis Netzwerke
ASO (Automativ Structure Optimization)

Cascade Correlation

Adding a hidden unit
- Input connections from all input units and from all already existing hidden units
- First only these connections are adapted
- Maximize the correlation between the activation of the candidate units and the residual error of the net
Not necessary to determine the number of hidden units empirically
Can produce deep networks without dramatic slowdown (bottom up, constructive learning)
At each point only one layer of connections is trained
Learning is fast
Learning is incremental

Dropout

Popular and very effective method for generalization
💡Idea
- Randomly drop out (zero) hidden units and input features during training
- Prevents feature co-adaptation
Illustration
Dropout training & test

Dropout

Sun, 16 Aug 2020 00:00:00 +0000

Model Overfitting

In order to give more “capacity” to capture different features, we give neural nets a lot of neurons. But this can cause overfitting.

Reason: Co-adaptation

Neurons become dependent on others
Imagination: neuron $H\_i$ captures a particular feature $X$ which however, is very frequenly seen with some inputs.
- If $H\_i$ receives bad inputs (partial of the combination), then there is a chance that the feature is ignored 🤪

Solution: Dropout! 💪

Dropout

With dropout the layer inputs become more sparse, forcing the network weights to become more robust.

Dropout a neuron = all the inputs and outputs to this neuron will be disabled at the current iteration.

Training

Given
- input $X \in \mathbb{R}^D$
- weights $W$
- survival rate $p$
  - Usually $p=0.5$
Sample mask $M \in \{0, 1\}^D$ with $M\_i \sim \operatorname{Bernoulli}(p)$
Dropped input:
$$ \hat{X} = X \circ M $$
Perform backward pass and mask the gradients:
$$ \frac{\delta L}{\delta X}=\frac{\delta L}{\delta \hat{X}} \circ M $$

Evaluation/Testing/Inference

ALL input neurons $X$ are presented WITHOUT masking
Because each neuron appears with probability $p$ in training

$\to$ So we have to scale $X$ with $p$ (or scale $\hat{X}$ with $\frac{1}{1-p}$ during training) to match its expectation

Why Dropout works?

Intuition: Dropout prevents the network to be too dependent on a small number of neurons, and forces every neuron to be able to operate independently.
Each of the “dropped” instance is a different network configuration
$2^n$ different networks sharing weights
The inference process can be understood as an ensemble of $2^n$ different configuration
This interpretation is in-line with Bayesian Neural Networks

👍 Data Augmentation

Sun, 16 Aug 2020 00:00:00 +0000

Motivation

Overfitting happens because of having too few examples to train on, resulting in a model that has poor generalization performance 😢. If we had infinite training data, we wouldn’t overfit because we would see every possible instance.

However, in most machine learning applications, especially in image classification tasks, obtaining new training data is not easy. Therefore we need to make do with the training set at hand. 💪

Data augmentation is a way to generate more training data from our current set. It enriches or “augments” the training data by generating new examples via random transformation of existing ones. This way we artificially boost the size of the training set, reducing overfitting. So data augmentation can also be considered as a regularization technique.

Data augmentation is done dynamically during training time. We need to generate realistic images, and the transformations should be learnable, simply adding noise won’t help. Common transformations are

rotation
shifting
resizing
exposure adjustment
contrast change
etc.

This way we can generate a lot of new samples from a single training example.

Notice that data augmentation is ONLY performed on the training data, we don’t touch the validation or test set.

Popular Augmentation Techniques

Flip

Left: original image. Middle: image flipped horizontally. Right: image flipped vertically

Rotation

Example of square images rotated at right angles. From left to right: The images are rotated by 90 degrees clockwise with respect to the previous one.

Note: image dimensions may not be preserved after rotation

If image is a square, rotating it at right angles will preserve the image size.
If image is a rectangle, rotating it by 180 degrees would preserve the size.

Scale

Left: original image. Middle: image scaled outward by 10%. Right: image scaled outward by 20%

The image can be scaled outward or inward. While scaling outward, the final image size will be larger than the original image size. Most image frameworks cut out a section from the new image, with size equal to the original image.

Crop

Left: original image. Middle: a square section cropped from the top-left. Right: a square section cropped from the bottom-right. The cropped sections were resized to the original image size.

Random cropping

Randomly sample a section from the original image
Resize this section to the original image size

Translation

Left: original image. Middle: the image translated to the right. Right: the image translated upwards.

Translation = moving the image along the X or Y direction (or both)

This method of augmentation is very useful as most objects can be located at almost anywhere in the image. This forces your convolutional neural network to look everywhere.

Gaussian Noise

Left: original image. Middle: image with added Gaussian noise. Right: image with added salt and pepper noise.

One reason of overfitting ist that neural network tries to learn high frequency features (patterns that occur a lot) that may not be useful.

Gaussian noise, which has zero mean, essentially has data points in all frequencies, effectively distorting the high frequency features. This also means that lower frequency components (usually, your intended data) are also distorted, but your neural network can learn to look past that. Adding just the right amount of noise can enhance the learning capability.

A toned down version of this is the salt and pepper noise, which presents itself as random black and white pixels spread through the image. This is similar to the effect produced by adding Gaussian noise to an image, but may have a lower information distortion level.

Nerual Network Basics | Haobin Tan

Neural Network Basics

Perceptron

Structure

Update Rule

👍 Activation Functions

Sigmoid

Tanh

Rectified Linear Unit (ReLU)

Leaky ReLU

Exponential Linear Units (ELU)

Maxout

Softmax

Advice in Practice

Summary and Overview

👍 Loss Functions

Classification

Regression

Metric Learning / Similarity Learning

Multilayer Perceptron and Backpropagation

Multi-Layer Perceptron (MLP)

Backpropagation

Gradients for vectorized operations

Softmax and Its Derivative

Useful resources

Generalization

Reasons for bad generalization

Prevent Overfitting

Destructive Methods

Optimal Brain Damage

Constructive Methods

Cascade Correlation

Dropout

Dropout

Model Overfitting

Dropout

Training

Evaluation/Testing/Inference

Why Dropout works?

👍 Data Augmentation

Motivation

Popular Augmentation Techniques

Flip

Rotation

Scale

Crop

Translation

Gaussian Noise

Reference