LSTM | Haobin Tan

RNN Summary

Mon, 03 Aug 2020 00:00:00 +0000

Intuition

Humans don’t start their thinking from scratch every second. As you read this article, you understand each word based on your understanding of previous words. You don’t throw everything away and start thinking from scratch again. Your thoughts have persistence.

Traditional neural networks can’t do this, and it seems like a major shortcoming. For example, imagine you want to classify what kind of event is happening at every point in a movie. It’s unclear how a traditional neural network could use its reasoning about previous events in the film to inform later ones.

Recurrent neural networks (RNNs) address this issue and solve it pretty well.

Sequence Data

Sequence: a particular order in which one thing follows another
Forms of sequence data
- Audio: natural sequence. You can chop up an audio spectrogram into chunks and feed that into RNN’s.
- Text: You can break Text up into a sequence of characters or a sequence of words.

Sequential Memory

RNN’s are good at processing sequence data for predictions by having a concept called sequential memory.

Let’s take a look at an example: the alphabet.

Say the alphabet in your head:

That was pretty easy right. If you were taught this specific sequence, it should come quickly to you.

Now try saying the alphabet backward.

This is much harder. Unless you’ve practiced this specific sequence before, you’ll likely have a hard time.

Now let’s try starting at the letter “F”:

At first, you’ll struggle with the first few letters, but then after your brain picks up the pattern, the rest will come naturally.

So there is a very logical reason why this can be difficult. You learn the alphabet as a sequence. Sequential memory is a mechanism that makes it easier for your brain to recognize sequence patterns.

Recurrent Neural Network (RNN)

How does RNN replicate the abstract concept of sequential memory?

Let’s look at a traditional neural network also known as a feed-forward neural network. It has its input layer, hidden layer, and the output layer.

💡 Get a feed-forward neural network to be able to use previous information to effect later ones: add a loop in the neural network that can pass prior information forward

And that’s essentially what a recurrent neural network does! A RNN has a looping mechanism that acts as a highway to allow information to flow from one step to the next.

Passing Hidden State to next time step

This information is the hidden state, which is a representation of previous inputs.

Unrolled RNN

These loops make recurrent neural networks seem kind of mysterious. However, if you think a bit more, it turns out that they aren’t all that different than a normal neural network.

A recurrent neural network can be thought of as multiple copies of the same network, each passing a message to a successor.

This chain-like nature reveals that recurrent neural networks are intimately related to sequences and lists. They’re the natural architecture of neural network to use for such data.

Chatbot Example

Let’s say we want to build a chatbot, which can classify intentions from the users inputted text. We’re going to tackle this problem as follows:

Encode the sequence of text using a RNN
Feed the RNN output into a feed-forward neural network which will classify the intents.

Now a user types in “What time is it?”

To start, we break up the sentence into individual words. RNNs work sequentially so we feed it one word at a time.

Then we feed each word into the RNN until the final step. In each step, the RNN encodes each input word and produces an output

$$ \vdots $$

we can see by the final step the RNN has encoded information from all the words in previous steps.

Since the final output was created from the rest of the sequence, we should be able to take the final output and pass it to the feed-forward layer to classify an intent.

Python pseudocode for the above workflow:

# initialize network layers
rnn = RNN()
ff = FeedForwardNN()

# initialize hidden state 
# (shape and dimension will be dependent on the RNN)
hidden_state = [0.0, 0.0, 0.0, 0.0]

# Loop through inputs, pass the word and hidden state into the RNN,
# RNN returns the output and a modified hidden state.
# Continue to loop until out of words
for word in input:
 output, hidden_state = rnn(word, hidden_state)

# Pass the output to the feedforward layer, and it returns a prediction
prediction = ff(output)

Problem of RNN

Intuition and Example

Sometimes, we only need to look at recent information to perform the present task. For example, consider a language model trying to predict the next word based on the previous ones.

If we are trying to predict the last word in “the clouds are in the sky,” we don’t need any further context – it’s pretty obvious the next word is going to be sky. In such cases, where the gap between the relevant information and the place that it’s needed is small, RNNs can learn to use the past information.

But there are also cases where we need more context.

Consider trying to predict the last word in the text “I grew up in France… I speak fluent French.”
- Recent information suggests that the next word is probably the name of a language.
- But if we want to narrow down which language, we need the context of France, from further back. It’s entirely possible for the gap between the relevant information and the point where it is needed to become very large.
Unfortunately, as that gap grows, RNNs become unable to learn to connect the information. 😢

Short-term Memory

This issue of RNN is known as short-term memory.

Short-term memory is caused by the infamous vanishing gradient problem, which is also prevalent in other neural network architectures.

As the RNN processes more steps, it has troubles retaining information from previous steps. As you can see, in the above chatbot example, the information from the word “what” and “time” is almost non-existent at the final time step.

Final hidden state of the RNN

Vanishing Gradient

Short-Term memory and the vanishing gradient is due to the nature of back-propagation, an algorithm used to train and optimize neural networks. To understand why this is, let’s take a look at the effects of back propagation on a deep feed-forward neural network.

Training a neural network has three major steps:

It does a forward pass and makes a prediction.
It compares the prediction to the ground truth using a loss function. The loss function outputs an error value which is an estimate of how poorly the network is performing.
It uses that error value to do back propagation which calculates the gradients for each node in the network.

The gradient is the value used to adjust the networks internal weights, allowing the network to learn. The bigger the gradient, the bigger the adjustments and vice versa.

Here is where the problem lies!

When doing back propagation, each node in a layer calculates it’s gradient with respect to the effects of the gradients, in the layer before it. So if the adjustments to the layers before it is small, then adjustments to the current layer will be even smaller. That causes gradients to exponentially shrink as it back propagates down. The earlier layers fail to do any learning as the internal weights are barely being adjusted due to extremely small gradients. And that’s the vanishing gradient problem.

Gradients shrink as it back-propagates down

Let’s see how this applies to RNNs. We can think of each time step in a recurrent neural network as a layer. To train a recurrent neural network, you use an application of back-propagation called Back-Propagation Through Time (BPTT). The gradient values will exponentially shrink as it propagates through each time step. 😢

Gradients shrink as it back-propagates through time

Again, the gradient is used to make adjustments in the neural networks weights thus allowing it to learn. Small gradients mean small adjustments. That causes the early layers NOT to learn. 🤪

Because of vanishing gradients, the RNN doesn’t learn the long-range dependencies across time steps. That means that in our chatbot example there is a possibility that the word “what” and “time” are not considered when trying to predict the user’s intention. The network then has to make the best guess with “is it?”. That’s pretty ambiguous and would be difficult even for a human. So not being able to learn on earlier time steps causes the network to have a short-term memory.

Solution

Long Short-Term Memory (LSTM)
Gated Recurrent Unit (GRU)

Reference

Illustrated Guide to Recurrent Neural Networks 🔥👍
Understanding LSTM Networks 🔥👍

LSTM Summary

Mon, 03 Aug 2020 00:00:00 +0000

Problem of Vanilla RNN

Short-term memory

If a sequence is long enough, they’ll have a hard time carrying information from earlier time steps to later ones. So if you are trying to process a paragraph of text to do predictions, RNN’s may leave out important information from the beginning.
Vanishing gradient problem

The gradient shrinks as it back propagates through time. If a gradient value becomes extremely small, it doesn’t contribute too much learning.

In recurrent neural networks, layers that get a small gradient update stops learning. Those are usually the earlier layers. So because these layers don’t learn, RNN’s can forget what it seen in longer sequences, thus having a short-term memory.

Solution: Long Short Term Memory (LSTM)!

Intuition

Let’s say you’re looking at reviews online to determine if you want to buy Life cereal . You’ll first read the review then determine if someone thought it was good or if it was bad.

When you read the review, your brain subconsciously only remembers important keywords. You pick up words like “amazing” and “perfectly balanced breakfast”. You don’t care much for words like “this”, “gave“, “all”, “should”, etc. If a friend asks you the next day what the review said, you probably wouldn’t remember it word for word. You might remember the main points though like “will definitely be buying again”.

And that is essentially what an LSTM does. It can learn to keep only relevant information to make predictions, and forget non relevant data.

Review of vanilla RNN

LSTM is explicitly designed to avoid the long-term dependency problem. They have internal mechanisms called gates that can regulate the flow of information. These gates can learn which data in a sequence is important to keep or throw away. By doing that, it can pass relevant information down the long chain of sequences to make predictions.

In order to achieve a solid understanding of LSTM, let’s start from the standard (vanilla) RNN.

All recurrent neural networks have the form of a chain of repeating modules of neural network. In standard RNNs, this repeating module will have a very simple structure, such as a single tanh layer.

An RNN works like this

First words get transformed into machine-readable vectors.
Then the RNN processes the sequence of vectors one by one.

While processing, it passes the previous hidden state to the next step of the sequence. The hidden state acts as the neural networks memory. It holds information on previous data the network has seen before.

Calculate the hidden state in each cell:

The input and previous hidden state are combined to form a vector. (That vector now has information on the current input and previous inputs)
The vector goes through the tanh activation, and the output is the new hidden state, or the memory of the network.

Tanh activation

The tanh function squishes values to always be between -1 and 1. Therefore it is used to help regulating the values flowing through the network.

vector transformations with tanh

LSTM

LSTMs also have this chain like structure, but the repeating module has a different structure. Instead of having a single neural network layer, there are four, interacting in a very special way.

Core Idea

Cell state

Cell state is the horizontal line running through the top of the diagram.

Act as a transport highway that transfers relative information all the way down the sequence chain.
Think of it as the “memory” of the network
In theory, it an carry relevant information throughout the processing of the sequence.
Even information from the earlier time steps can thus make it’s way to later time steps, reducing the effects of short-term memory 👏

Gates

The LSTM has the ability to remove or add information to the cell state, carefully regulated by structures called gates.

Gates are

a way to optionally let information through
composed out of a sigmoid neural net layer and a pointwise multiplication operation.

Why sigmoid?

Output of sigmoid layer is between 0 and 1

The sigmoid layer squishes values between 0 and 1, describing how much of each component should be let through.

0: “let nothing through”, “forgotten”
1: “let everything through”, “kept”

Forget gate

The first step in LSTM is to decide what information we’re going to throw away from the cell state. This decision is made by a sigmoid layer called the “forget gate layer.”

It looks at

$h\_{t−1}$: previous hidden state, and
$x\_t$: information from the current input

and outputs a number between $0$ and $1$ for each number in the cell state $C\_{t−1}$.

Value closer to $1$ means to keep
Value closer to $0$ means to forget

Operations of Forget gate

Input gate

To decide what new information we’re going to store in the cell state and update the cell state, we have the input gate.

Pass the previous hidden state and current input
- into a sigmoid function. That decides which values will be updated by transforming the values to be between $0$ and $1$ ($i\_t$)
  - $0$: not important
  - $1$: important
- into the tanh function to squish values between $-1$ and $1$ to create a candidate cell state ($\tilde{C}\_t$) that should be added to the cell state.
Operations of Input gate
Combine these two to create an update to the state

Calculating new cell state

Output gate

The output gate decides what the next hidden state should be. Remember that the hidden state contains information on previous inputs. The hidden state is also output for predictions.

Run a sigmoid layer which decides what parts of the cell state we’re going to output.
Put the newly modified cell state through tanh function (to regulate the values to be between $−1$ and $1$) and multiply it by the output of the sigmoid gate, so that we only output the parts we decided to.

Operations of Output gate

Review

Forget gate: decides what is relevant to keep from prior steps.
Input gate: decides what information is relevant to add from the current step.
Output gate: determines what the next hidden state should be.

Example

Consider a language model trying to predict the next word based on all the previous ones.

In such a problem, the cell state might include the gender of the present subject, so that the correct pronouns can be used.

LSTM gate	In exmple model
Forget gate	When we see a new subject, we want to forget the gender of the old subject.
Input gate	1. We’d want to add the gender of the new subject to the cell state, to replace the old one we’re forgetting. 2. We drop the information about the old subject’s gender and add the new information.
Output gate	Since it just saw a subject, it might want to output information relevant to a verb, in case that’s what is coming next. For example, it might output whether the subject is singular or plural, so that we know what form a verb should be conjugated into if that’s what follows next.

Python Pseudocode

def LSTM_cell(prev_ct, prev_ht, input):

 # Concatenate previous hidden state and current input
 combine = prev_ht + input

 # Forget gate remove non-relevant data
 ft = forget_layer(combine)

 # Candiate holds possible values to add to the cell state
 candidate = candidate_layer(combine)

 # Input layer decides what data from the candidate 
 # should be added to the new cell state
 it = input_layer(combine)

 # Calculate new cell state using forget layer, candidate layer
 # and input layer
 Ct = prev_Ct * ft + candidate * it

 # Output layer decides which part should be output
 ot = output_layer(combine)

 # Pointwise multiplying the output gate and the new cell state 
 # gives us the new hidden state.
 ht = ot * tanh(Ct)
 return ht, Ct


ct = [0, 0, 0]
ht = [0, 0, 0]
for input in iuputs:
 ct, ht = LSTM_cell(ct, ht, input)

Summary

Diagram of Strucutre

{t}=\tanh \left(W{C} \c…

Input gate
decides which values will be updated

Input gate…

$i_{t}=\sigma\left(W_{i} \cdot\left[h_{t-1}, x_{t}\right]+b_{i}\right)$

i_{t}=\sigma\left(W_{i} \cdot…

Calculate new candidate cell state

Calculate new candidate cell state

$=$

$f_t$

f_t

$+$

$c_{t-1}$

c_{t…

$c_{t}$

c_{t}

$c_{t-1}$

c_{t…

$$

$i_t$

i_t

$$

$\tilde{C}t$

\til…

$c{t}$

c_{t}

Output gate
decides what the next hidden state should be

Output gate…

$o_{t}=\sigma\left(W_{o}\left[h_{t-1}, x_{t}\right]+b_{o}\right)$

o_{t}=\sigma\left(W_{o}\left[…

$=$

$\tanh$

\tanh

$($

(

$c_{t}$

c_{t}

$)$

)

$o_t$

o_t

$$

$h_t$

h_t

def LSTM_cell(prev_ct, prev_ht, input):
 combine = concate(prev_ht, input)
 ft = forget_layer(combine)
 candidate = candidate_layer(combine)
 it = input_layer(combine)
 ct = ft * prev_ct + it * candidate
 ot = output_layer(combine)
 ht = ot * tanh(ct)
 return ht, ct

# init
ct = [0, 0, 0]
ht = [0, 0, 0]
for input in inputs:
 ct, ht = LSTM_cell(ct, ht, input)

def LSTM_cell(prev_ct, prev_ht, input):…Viewer does not support full SVG 1.1

Whole Process

Reference

Understanding LSTM Networks 🔥👍
Illustrated Guide to LSTM’s and GRU’s: A step by step explanation 🔥👍

Backpropagation Through Time (BPTT)

Thu, 13 Aug 2020 00:00:00 +0000

Recurrent neural networks (RNNs) have attracted great attention on sequential tasks. However, compared to general feedforward neural networks, it is a little bit harder to train RNNs since RNNs have feedback loops.

In this article, we dive into basics, especially the error backpropagation to compute gradients with respect to model parameters. Furthermore, we go into detail on how error backpropagation algorithm is applied on long short-term memory (LSTM) by unfolding the memory unit.

BPTT in RNN

Left: recursive description. Right: unrolled RNN in a time sequential manner

$\mathbf{x}\_t$: current observation/input
$\mathbf{h}\_t$: hidden state
- dependent on:
  - current observation $\mathbf{x}\_t$
  - previous hidden state $\mathbf{h}\_{t-1}$
- Representation:
  $$ \mathbf{h}\_{t}=f\left(\mathbf{h}\_{t-1}, \mathbf{x}\_{t}\right) $$
  - $f$: nonlinear mapping
$z\_t$: output/prediction at time step $t$

Suppose we have the following RNN:

$$ \begin{array}{l} \mathbf{h}\_{t}=\tanh \left(W\_{h h} \mathbf{h}\_{t-1}+W\_{x h} \mathbf{x}\_{t}+\mathbf{b}\_{\mathbf{h}}\right) \\\\ \alpha\_t = W\_{h z} \mathbf{h}\_{t}+\mathbf{b}\_{z}\\\\ z\_{t}=\operatorname{softmax}\left(\alpha\_t\right) \end{array} $$

Reminder:

$$ \tanh (x)=\frac{\sinh (x)}{\cosh (x)}=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}=\frac{e^{2 x}-1}{e^{2 x}+1} $$

Considering the varying length for each sequential data, we also assume the parameters in each time step are the same across the whole sequential analysis (Otherwise it will be hard to compute the gradients). In addition, sharing the weights for any sequential length can generalize the model well.

As for sequential labeling, we can use the maximum likelihood to estimate model parameters. In other words, we can minimize the negative log likelihood the objective function ($\to$ cross entropy)

$$ \mathcal{L}(\mathbf{x}, \mathbf{y})=-\sum_{t} y_{t} \log z_{t} $$

For simplicity, in the following we will use $\mathcal{L}$ as the objective function
At time step $t+1$: $\mathcal{L}(t+1)=-y\_{t+1}\log z\_{t+1}$

Derivation

$W\_{hz}$ and $b\_z$

Based on the RNN above, by taking the derivative with respect to $\alpha\_t$, we have (refer to derivative of softmax)

$$ \frac{\partial \mathcal{L}}{\partial \alpha\_{t}}=-\left(y\_{t}-z\_{t}\right) $$

Note the weight $W\_{hz}$ is shared across all time sequence, thus we can differentiate to it at each time step and sum all together

$$ \frac{\partial \mathcal{L}}{\partial W_{h z}}=\sum_{t} \frac{\partial \mathcal{L}}{\partial z_{t}} \frac{\partial z_{t}}{\partial W_{h z}} $$

Similarly, we can get the gradient w.r.t. bias $b\_z$

$$ \frac{\partial \mathcal{L}}{\partial b_{z}}=\sum_{t} \frac{\partial \mathcal{L}}{\partial z_{t}} \frac{\partial z_{t}}{\partial b_{z}} $$

$W\_{hh}$

Consider at time step $t \to t + 1$ in the figure above

$$ \frac{\partial \mathcal{L}(t+1)}{\partial W\_{h h}}=\frac{\partial \mathcal{L}(t+1)}{\partial z\_{t+1}} \frac{\partial z\_{t+1}}{\partial \mathbf{h}\_{t+1}} \frac{\partial \mathbf{h}\_{t+1}}{\partial W\_{h h}} $$

Because the hidden state $\mathbf{h}\_{t+1}$ partially depends on $\mathbf{h}\_t$, we can use backpropagation to compute the above partial derivative. Futhermore, $W\_{hh}$ is shared cross the whole time sequence. Therefore, at time step $(t-1) \to t$, we can get the partial derivative w.r.t. $W\_{hh}$:

Thus, at the time step $t + 1$, we can compute gradient w.r.t. $z\_{t+1}$ and further use backpropagation through time (BPTT) from $t$ to $0$ to calculate gradient w.r.t. $W\_{hh}$ (shown as the red chain in figure above). In other words, if we only consider the output $z\_{t+1}$ at time step $t + 1$, we can yield the following gradient w.r.t. $W\_{hh}$:

$$ \frac{\partial \mathcal{L}(t+1)}{\partial W\_{h h}}=\sum_{k=1}^{t+1} \frac{\partial \mathcal{L}(t+1)}{\partial z\_{t+1}} \frac{\partial z\_{t+1}}{\partial \mathbf{h}\_{t+1}} \frac{\partial \mathbf{h}\_{t+1}}{\partial \mathbf{h}\_{k}} \frac{\partial \mathbf{h}\_{k}}{\partial W\_{h h}} $$

Example: $t = 2$

3 = tanh(W{hh}\color{green}{\mathbf{h}2} + W{x…

Computational graph

$\mathcal{L}3$

\math…

$W{hh}$

W{hh}

$\mathbf{h}_1$

\mathbf{…

$\mathbf{h}2$

\mathbf{…

$\mathbf{h}3$

\mathbf{…

$W{hz}$

W{hz}

$\mathbf{z}_3$

\mathbf{…

$\mathcal{L}_3$

\mathcal…

$\frac{\partial \mathcal{L}_3}{\partial \mathbf{z}_3}$

\fra…

$\frac{\partial \mathbf{z}_3}{\partial \mathbf{h}_3}$

\fra…

$\frac{\partial \mathbf{h}3}{\partial W{hh}}$

\fra…

$\frac{\partial \mathbf{h}2}{\partial W{hh}}$

\fra…

$\frac{\partial \mathbf{h}_3}{\partial \mathbf{h}_2}$

\fra…

$\frac{\partial \mathcal{L}3}{W{hh}} = \frac{\partial \mathcal{L}_3}{\partial \mathbf{z}_3} \frac{\partial \mathbf{z}_3}{\partial \mathbf{h}_3} \frac{\partial \mathbf{h}3}{\partial W{hh}} + \frac{\partial \mathcal{L}_3}{\partial \mathbf{z}_3} \frac{\partial \mathbf{z}_3}{\partial \mathbf{h}_3 }\frac{\partial \mathbf{h}_3}{\partial \mathbf{h}_2} \frac{\partial \mathbf{h}2}{\partial W{hh}} + \frac{\partial \mathcal{L}_3}{\partial \mathbf{z}_3} \frac{\partial \mathbf{z}_3}{\partial \mathbf{h}_3 }\frac{\partial \mathbf{h}_3}{\partial \mathbf{h}_2} \frac{\partial \mathbf{h}_2}{\partial \mathbf{h}_1} \frac{\partial \mathbf{h}1}{\partial W{hh}}$

\frac{\partial \mathcal{L}3}{W{hh}} =…

Ground truth

Output/Prediction

Hidden state

Input

$\frac{\partial \mathbf{h}_2}{\partial \mathbf{h}_1}$

\fra…

$\mathbf{h}0$

\mathbf{…

$\frac{\partial \mathbf{h}1}{\partial W{hh}}$

\fra…

$\mathbf{h}0$

\mathbf{…

$W{hh}$

W{hh}Viewer does not support full SVG 1.1

Aggregate the gradients w.r.t. $W\_{hh}$ over the whole time sequence with back propagation, we can finally yield the following gradient w.r.t $W\_{hh}$:

$$ \frac{\partial \mathcal{L}}{\partial W\_{h h}}=\sum\_{t} \sum\_{k=1}^{t+1} \frac{\partial \mathcal{L}(t+1)}{\partial z\_{t+1}} \frac{\partial z\_{t+1}}{\partial \mathbf{h}\_{t+1}} \frac{\partial \mathbf{h}\_{t+1}}{\partial \mathbf{h}\_{k}} \frac{\partial \mathbf{h}\_{k}}{\partial W\_{h h}} $$

$W\_{xh}$

Similar to $W\_{hh}$, we consider the time step $t + 1$ (only contribution from $\mathbf{x}\_{t+1}$) and calculate the gradient w.r.t. $W\_{xh}$:

$$ \frac{\partial \mathcal{L}(t+1)}{\partial W\_{x h}}=\frac{\partial \mathcal{L}(t+1)}{\partial \mathbf{h}\_{t+1}} \frac{\partial \mathbf{h}\_{t+1}}{\partial W\_{x h}} $$

Because $\mathbf{h}\_{t}$ and $\mathbf{x}\_t$ both make contribution to $\mathbf{h}\_{t+1}$, we need to back propagate to $\mathbf{h}\_{t}$ as well. If we consider the contribution from the time step $t$, we can further get

$$ \begin{aligned} & \frac{\partial \mathcal{L}(t+1)}{\partial W\_{x h}}=\frac{\partial \mathcal{L}(t+1)}{\partial \mathbf{h}\_{t+1}} \frac{\partial \mathbf{h}\_{t+1}}{\partial W\_{x h}}+\frac{\partial \mathcal{L}(t+1)}{\partial \mathbf{h}\_{t}} \frac{\partial \mathbf{h}\_{t}}{\partial W\_{x h}} \\\\ =& \frac{\partial \mathcal{L}(t+1)}{\partial \mathbf{h}\_{t+1}} \frac{\partial \mathbf{h}\_{t+1}}{\partial W\_{x h}}+\frac{\partial \mathcal{L}(t+1)}{\partial \mathbf{h}\_{t+1}} \frac{\partial \mathbf{h}\_{t+1}}{\partial \mathbf{h}\_{t}} \frac{\partial \mathbf{h}\_{t}}{\partial W\_{x h}} \end{aligned} $$

Thus, summing up all contributions from $t$ to $0$ via backpropagation, we can yield the gradient at the time step $t + 1$:

$$ \frac{\partial \mathcal{L}(t+1)}{\partial W\_{x h}}=\sum_{k=1}^{t+1} \frac{\partial \mathcal{L}(t+1)}{\partial \mathbf{h}\_{t+1}} \frac{\partial \mathbf{h}\_{t+1}}{\partial \mathbf{h}\_{k}} \frac{\partial \mathbf{h}\_{k}}{\partial W\_{x h}} $$

Example: $t=2$

Computational graph for W_xh

Further, we can take derivative w.r.t. $W\_{xh}$ over the whole sequence as

$$ \frac{\partial \mathcal{L}}{\partial W_{x h}}=\sum\_{t} \sum_{k=1}^{t+1} \frac{\partial \mathcal{L}(t+1)}{\partial z\_{t+1}} \frac{\partial z_{t+1}}{\partial \mathbf{h}\_{t+1}} \frac{\partial \mathbf{h}\_{t+1}}{\partial \mathbf{h}\_{k}} \frac{\partial \mathbf{h}\_{k}}{\partial W\_{x h}} $$

Gradient vanishing or exploding problems

Notice that $\frac{\partial \mathbf{h}\_{t+1}}{\partial \mathbf{h}\_{k}}$ in the equation above indicates matrix multiplication over the sequence. And RNNs need to backpropagate gradients over a long sequence

With small values in the matrix multiplication

Gradient value will shrink layer over layer, and eventually vanish after a few time steps. Thus, the states that are far away from the current time step does not contribute to the parameters’ gradient computing (or parameters that RNNs is learning)!

$\to$ Gradient vanishing
With large values in the matrix multiplication

Gradient value will get larger layer over layer, and eventually become extremly large!

$\to$ Gradient exploding

BPTT in LSTM

The representation of LSTM here follows the one in A Gentle Tutorial of Recurrent Neural Network with Error Backpropagation.

More details about LSTM see: LSTM Summary

Unit structure of LSTM

How LSTM works

Given a sequence data $\left\\{\mathbf{x}\_{1}, \dots,\mathbf{x}\_{T}\right\\}$, we have

Forget gate
$$ \mathbf{f}\_{t}=\sigma\left(W\_{x f} \mathbf{x}\_{t}+W\_{h f} \mathbf{h}\_{t-1}+b\_{f}\right) $$
Input gate
$$ \mathbf{i}\_{t}=\sigma\left(W\_{x i} \mathbf{x}\_{t}+W_{h i} \mathbf{h}\_{t-1}+b\_{i}\right) $$
Candidate of new cell state
$$ \mathbf{g}\_{t}=\tanh \left(W\_{x c} \mathbf{x}\_{t}+W_{h c} \mathbf{h}\_{t-1}+b\_{c}\right) $$
New cell state
$$ \mathbf{c}\_{t}=\mathbf{f}\_{t} \odot \mathbf{c}\_{t-1}+\mathbf{i}\_{t} \odot \mathbf{g}\_{t} $$

Note

$\odot$ is a pointwise/elementwise multiplication.

$\left[\begin{array}{l} > x\_{1} \\\\ > x\_{2} > \end{array}\right] \odot\left[\begin{array}{l} > y\_{1} \\\\ > y\_{2} > \end{array}\right]=\left[\begin{array}{l} > x\_{1} y\_1 \\\\ > x\_{2} y\_{2} > \end{array}\right]$
Output gate
$$ \mathbf{o}\_{t}=\sigma\left(W\_{x o} \mathbf{x}\_{t}+W\_{h o} \mathbf{h}\_{t-1}+b\_{o}\right) $$
New hidden state (and output)
$$ \mathbf{h}\_{t}=\mathbf{o}\_{t} \odot \tanh \left(\mathbf{c}\_{t}\right), \quad z\_{t}=\operatorname{softmax}\left(W\_{h z} \mathbf{h}\_{t}+b\_{z}\right) $$

Derivatives

At the time step $t$:

$\mathbf{h}\_{t}=\mathbf{o}\_{t} \circ \tanh \left(\mathbf{c}\_{t}\right)\Rightarrow $
- $d \mathbf{o}\_{t}=\tanh \left(\mathbf{c}\_{t}\right) d \mathbf{h}\_{t}$
- $d \mathbf{c}\_{t}=\left(1-\tanh \left(\mathbf{c}\_{t}\right)^{2}\right) \mathbf{o}\_{t} d \mathbf{h}\_{t}$ (see: Derivation of $tanh$)
$\mathbf{c}\_{t}=\mathbf{f}\_{t} \circ \mathbf{c}\_{t-1}+\mathbf{i}\_{t} \circ \mathbf{g}\_{t} \Rightarrow$
- $d \mathbf{i}\_{t}=\mathbf{g}\_{t} d \mathbf{c}\_{t}$
- $d \mathbf{g}\_{t}=\mathbf{i}\_{t} d \mathbf{c}\_{t}$
- $d \mathbf{f}\_{t}=\mathbf{c}\_{t-1} d \mathbf{c}\_{t}$
- $d \mathbf{c}\_{t-1}+=\mathbf{f}\_{t} \circ d \mathbf{c}\_{t}$ (derivation see: Error propagation)

What’s more, we backpropagate activation functions over the whole sequence (As the weights $W\_{xo}, W\_{xi}, W\_{xf}, W\_{xc}$ are shared across the whole sequence, we need to take the same summation over $t$ as in RNNs):

$$ \begin{aligned} d W\_{x o} &=\sum\_{t} \mathbf{o}\_{t}\left(1-\mathbf{o}\_{t}\right) \mathbf{x}\_{t} d \mathbf{o}\_{t} \\\\ d W\_{x i} &=\sum_{t} \mathbf{i}\_{t}\left(1-\mathbf{i}\_{t}\right) \mathbf{x}\_{t} d \mathbf{i}\_{t} \\\\ d W\_{x f} &=\sum\_{t} \mathbf{f}\_{t}\left(1-\mathbf{f}\_{t}\right) \mathbf{x}\_{t} d \mathbf{f}\_{t} \\\\ d W\_{x c} &=\sum\left(1-\mathbf{g}\_{t}^{2}\right) \mathbf{x}\_{t} d \mathbf{g}\_{t} \end{aligned} $$

Similarly, we have

$$ \begin{aligned} d W\_{h o} &=\sum_{t} \mathbf{o}\_{t}\left(1-\mathbf{o}\_{t}\right) \mathbf{h}\_{t-1} d \mathbf{o}\_{t} \\\\ d W\_{h i} &=\sum\_{t} \mathbf{i}\_{t}\left(1-\mathbf{i}\_{t}\right) \mathbf{h}\_{t-1} d \mathbf{i}\_{t} \\\\ d W\_{h f} &=\sum\_{t} \mathbf{f}\_{t}\left(1-\mathbf{f}\_{t}\right) \mathbf{h}\_{t-1} d \mathbf{f}\_{t} \\\\ d W\_{h c} &=\sum\_{t}\left(1-\mathbf{g}\_{t}^{2}\right) \mathbf{h}\_{t-1} d \mathbf{g}\_{t} \end{aligned} $$

Since $h\_{t-1}$, hidden state at time step $t-1$, is used in forget gate, input gate, candidate of new cell state, and output gate, therefore:

$$ \begin{aligned} d \mathbf{h}\_{t-1} = &\mathbf{o}\_{t}\left(1-\mathbf{o}\_{t}\right) W_{h o} d \mathbf{o}\_{t}+\mathbf{i}\_{t}\left(1-\mathbf{i}\_{t}\right) W\_{h i} d \mathbf{i}\_{t} \\\\ &+\mathbf{f}\_{t}\left(1-\mathbf{f}\_{t}\right) W_{h f} d \mathbf{f}\_{t}+\left(1-\mathbf{g}\_{t}^{2}\right) W\_{h c} d \mathbf{g}\_{t} \end{aligned} $$

Alternatively, we can derive $d \mathbf{h}\_{t-1}$ from the objective function at time step $t-1$:

$$ d \mathbf{h}\_{t-1}=d \mathbf{h}\_{t-1}+W_{h z} d z\_{t-1} $$

Error backpropagation

Suppose we have the least square objective function

$$ \mathcal{L}(\mathbf{x}, \theta)=\min \sum\_{t} \frac{1}{2}\left(y\_{t}-z\_{t}\right)^{2} $$

where $\boldsymbol{\theta}=\left\\{W\_{h z}, W\_{x o}, W\_{x i}, W\_{x f}, W\_{x c}, W\_{h o}, W\_{h i}, W\_{h f}, W\_{h c}\right\\}$ with bias ignored. For the sake of brevity, we use the following notation

$$ \mathcal{L}(t)=\frac{1}{2}\left(y\_{t}-z\_{t}\right)^{2} $$

At time step $T$, we take derivative w.r.t. $\mathbf{c}\_T$

$$ \frac{\partial \mathcal{L}(T)}{\partial \mathbf{c}\_{T}}=\frac{\partial \mathcal{L}(T)}{\partial \mathbf{h}\_{T}} \frac{\partial \mathbf{h}\_{T}}{\partial \mathbf{c}\_{T}} $$

At time step $T-1$, we take derivative of $\mathcal{L}(t-1)$ w.r.t. $\mathbf{c}\_{T-1}$ as

$$ \frac{\partial \mathcal{L}(T-1)}{\partial \mathbf{c}\_{T-1}}=\frac{\partial \mathcal{L}(T-1)}{\partial \mathbf{h}\_{T-1}} \frac{\partial \mathbf{h}\_{T-1}}{\partial \mathbf{c}\_{T-1}} $$

However, according to the following unfolded unit of structure,

Unfolded unit of LSTM, in order to make it easy to understand error backpropagation

the error is not only backpropagated via $\mathcal{L}(T-1)$, but also from $\mathbf{c}\_T$. Therefore, the gradient w.r.t. $\mathbf{c}\_{T-1}$ should be

$$ \begin{array}{ll} \frac{\partial \mathcal{L}(T-1)}{\partial \mathbf{c}\_{T-1}} &= \frac{\partial \mathcal{L}(T-1)}{\partial \mathbf{c}\_{T-1}}+\frac{\partial \mathcal{L}(T)}{\partial \mathbf{c}\_{T-1}} \\\\ &=\frac{\partial \mathcal{L}(T-1)}{\partial \mathbf{h}\_{T-1}} \frac{\partial \mathbf{h}\_{T-1}}{\partial \mathbf{c}\_{T-1}} + \underbrace{\frac{\partial \mathcal{L}(T)}{\partial \mathbf{h}\_{T}} \frac{\partial \mathbf{h}\_{T}}{\partial \mathbf{c}\_{T}}}\_{=d\mathbf{c}\_T} \underbrace{\frac{\partial \mathbf{c}\_{T}}{\partial \mathbf{c}\_{T-1}}}\_{=\mathbf{f}\_T} \end{array} $$ $$ \Rightarrow \qquad d \mathbf{c}\_{T-1}=d \mathbf{c}\_{T-1}+\mathbf{f}\_{T} \circ d \mathbf{c}\_{T} $$

Parameters learning

Forward

Use the equations in How LSTM works to update states as the feedforward neural network from the time step $1$ to $T$.
Compute loss
Backward

Backpropage the error from $T$ to $1$ using equations in Derivatives. Then use gradient $d\boldsymbol{\theta}$ to update the parameters $\boldsymbol{\theta}=\left\\{W\_{h z}, W\_{x o}, W\_{x i}, W\_{x f}, W\_{x c}, W\_{h o}, W\_{h i}, W\_{h f}, W\_{h c}\right\\}$. For example, if we use SGD, we have:
$$ \boldsymbol{\theta}=\boldsymbol{\theta}-\eta d \boldsymbol{\theta} $$
where $\eta$ is the learning rate.

Derivative of softmax

$z=\operatorname{softmax}\left(W\_{h z} \mathbf{h}+\mathbf{b}\_{z}\right)$: predicts the probability assigned to $K$ classes.

Furthermore, we can use $1$ of $K$ (One-hot) encoding to represent the groundtruth $y$ but with probability vector to represent $z=\left[p\left(\hat{y}\_{1}\right), \ldots, p\left(\hat{y}\_{K}\right)\right]$. Then, we can consider the gradient in each dimension, and then generalize it to the vector case in the objective function (cross-entropy loss):

$$ \mathcal{L}\left(W\_{h z}, \mathbf{b}\_{z}\right)=-y \log z $$

Let

$$ \alpha_{j}(\Theta)=W\_{h z}(:, j) \mathbf{h}\_{t} $$

Then

$$ p\left(\hat{y}\_{j} \mid \mathbf{h}\_{t} ; \Theta\right)=\frac{\exp \left(\alpha\_{j}(\Theta)\right)}{\sum\_{k} \exp \left(\alpha\_{k}(\Theta)\right)} $$

$\forall k = j$:

$$ \begin{array}{ll} &\frac{\partial}{\partial \alpha\_{j}} y\_{j} \log p\left(\hat{y}\_{j} \mid \mathbf{h}\_{t} ; \Theta\right) \\\\ \\\\ =&y\_{j} \left(\frac{\partial}{\partial \alpha\_{j}} \log p\left(\hat{y}\_{j} \mid \mathbf{h}\_{t} ; \Theta\right)\right) \left(\frac{\partial}{\partial \alpha\_{j}} p\left(\hat{y}\_{j} \mid \mathbf{h}\_{t} ; \Theta\right)\right) \\\\ \\\\ =&y\_{j} \cdot \frac{1}{p\left(\hat{y}\_{j} \mid \mathbf{h}\_{t} ; \Theta\right)} \cdot \left(\frac{\partial}{\partial \alpha\_{j}} \frac{\exp \left(\alpha\_{j}(\Theta)\right)}{\sum\_{k} \exp \left(\alpha\_{k}(\Theta)\right)}\right) \\\\ \\\\ =&\frac{y\_{j}}{p\left(\hat{y}\_{j}\right)} \frac{\exp \left(\alpha\_{j}(\Theta)\right) \sum_{k} \exp \left(\alpha\_{k}(\Theta)\right)-\exp \left(\alpha\_{j}(\Theta)\right) \exp \left(\alpha\_{j}(\Theta)\right)}{\left[\sum\_{k} \exp \left(\alpha\_{k}(\Theta)\right)\right]^{2}} \\\\ \\\\ =& \frac{y\_{j}}{p\left(\hat{y}\_{j}\right)} \underbrace{\frac{\exp \left(\alpha\_{j}(\Theta)\right)}{\sum\_k \exp(\alpha\_k(\theta))}}\_{=p\left(\hat{y}\_{j}\right)} \frac{\sum\_k \exp(\alpha\_k(\theta)) - \exp \left(\alpha\_{j}(\Theta)\right)}{\sum\_k \exp(\alpha\_k(\theta))}\\\\ \\\\ =&y\_{j} (\frac{\sum\_k \exp(\alpha\_k(\theta))}{\sum\_k \exp(\alpha\_k(\theta))} - \underbrace{\frac{\exp \left(\alpha\_{j}(\Theta)\right)}{\sum\_k \exp(\alpha\_k(\theta))}}\_{=p\left(\hat{y}\_{j}\right)}) \\\\ \\\\ =&y\_{j}\left(1-p\left(\hat{y}\_{j}\right)\right) \end{array} $$

$\forall k \neq j$:

$$ \begin{array}{ll} &\frac{\partial}{\partial \alpha\_{j}} y\_{j} \log p\left(\hat{y}\_{j} \mid \mathbf{h}\_{t} ; \Theta\right) \\\\ \\\\ =&\frac{y\_{j}}{p\left(\hat{y}\_{j}\right)} \frac{-\exp \left(\alpha\_{k}(\Theta)\right) \exp \left(\alpha\_{j}(\Theta)\right)}{\left[\sum\_{s} \exp \left(\alpha\_{s}(\Theta)\right)\right]^{2}} \\\\ \\\\ =&-y\_{k}p\left(\hat{y}\_{j}\right) \end{array} $$

We can yield the following gradient w.r.t. $\alpha\_j(\Theta)$:

$$ \begin{aligned} \frac{\partial p(\hat{\mathbf{y}})}{\partial \alpha_{j}} &=\sum\_{j} \frac{\partial y\_{j} \log p\left(\hat{y}\_{j} \mid \mathbf{h}\_{i} ; \Theta\right)}{\partial \alpha\_{j}} \\\\ &=\frac{\partial \log p\left(\hat{y}\_{j} \mid \mathbf{h}\_{i} ; \Theta\right)}{\partial \alpha\_{j}}+\sum_{k \neq j} \frac{\partial \log p\left(\hat{y}\_{k} \mid \mathbf{h}\_{i} ; \Theta\right)}{\partial \alpha_{j}} \\\\ &=\left(y\_{j}-y\_{j} p\left(\hat{y}\_{j}\right)\right)-\left(\sum_{k \neq j} y\_{k} p\left(\hat{y}\_{j}\right)\right) \\\\ &=y\_{j}-p\left(\hat{y}\_{j}\right)\left(y_{j}+\sum_{k \neq j} y\_{k}\right)\\\\ &=y\_{j}-p\left(\hat{y}\_{j}\right) \end{aligned} $$ $$ \begin{aligned} \Rightarrow \frac{\partial \mathcal{L}}{\partial \alpha\_j} &= \frac{\partial }{\partial \alpha\_j} \left(-\sum\_{j} \frac{\partial y\_{j} \log p\left(\hat{y}\_{j} \mid \mathbf{h}\_{i} ; \Theta\right)}{\partial \alpha\_{j}}\right) \\\\ &= -\left(y\_{j}-p\left(\hat{y}\_{j}\right)\right) \\\\ &= p\left(\hat{y}\_{j}\right) - y\_{j} \end{aligned} $$

See also:

Derivative of Softmax loss function

The Softmax function and its derivative

Derivative of tanh

$$ \begin{aligned} \frac{\partial \tanh (x)}{\partial(x)}=& \frac{\partial \frac{\sinh (x)}{\cosh (x)}}{\partial x} \\\\ =& \frac{\frac{\partial \sinh (x)}{\partial x} \cosh (x)-\sinh (x) \frac{\partial \cosh (x)}{\partial x}}{(\cosh (x))^{2}} \\\\ =& \frac{[\cosh (x)]^{2}-[\sinh (x)]^{2}}{(\cosh (x))^{2}} \\\\ =& 1-[\tanh (x)]^{2} \end{aligned} $$

Derivative of Sigmoid

Sigmoid:

$$ \sigma(x)=\frac{1}{1+e^{-x}} $$

Derivative:

$$ \begin{aligned} \frac{d}{d x} \sigma(x) &=\frac{d}{d x}\left[\frac{1}{1+e^{-x}}\right] \\\\ &=\frac{d}{d x}\left(1+\mathrm{e}^{-x}\right)^{-1} \\\\ &=-\left(1+e^{-x}\right)^{-2}\left(-e^{-x}\right) \\\\ &=\frac{e^{-x}}{\left(1+e^{-x}\right)^{2}} \\\\ &=\frac{1}{1+e^{-x}} \cdot \frac{e^{-x}}{1+e^{-x}} \\\\ &=\frac{1}{1+e^{-x}} \cdot \frac{\left(1+e^{-x}\right)-1}{1+e^{-x}} \\\\ &=\frac{1}{1+e^{-x}} \cdot\left(\frac{1+e^{-x}}{1+e^{-x}}-\frac{1}{1+e^{-x}}\right) \\\\ &=\frac{1}{1+e^{-x}} \cdot\left(1-\frac{1}{1+e^{-x}}\right) \\\\ &=\sigma(x) \cdot(1-\sigma(x)) \end{aligned} $$