Seq2Seq | Haobin Tan

Encoder-Decoder Models

Sun, 16 Aug 2020 00:00:00 +0000

Sequence to Sequence

Sun, 16 Aug 2020 00:00:00 +0000

Language Modeling

Language model is a particular model calculating the probability of a sequence

$$ \begin{aligned} P(W) &= P(W\_1 W\_2 \dots W\_n) \\\\ &= P\left(W\_{1}\right) P\left(W_{2} \mid W\_{1}\right) P\left(W\_{3} \mid W\_{1} W\_{2}\right) \ldots P\left(W\_{n} \mid W\_{1 \ldots n-1}\right) \end{aligned} $$

Softmax layer
- After linear mapping the hidden layer $H$, a “score” vector $O = [O\_1, O\_2, \dots, O\_{n-1}, O\_n]$ is obtained
- The softmax function normalizes $O$ to get probabilities
  $$ P\_{i}=\frac{\exp \left(O\_{i}\right)}{\sum\_{j} \exp \left(O\_{j}\right)} $$
Cross-Entropy Loss
- Use one-hot-vector $Y = [0,0,0,0,0,\dots,1,0,0]$ as the label to train the model
- Cross-entropy loss: difference between predicted probability and label
  $$ L\_{CE} = \sum\_i Y\_i \log(P\_i) $$
- When 𝑌 is an one-hot vector:
  $$ L\_{C E}=-\log P_{j}(\text{the index of the correct word}) $$

Training

Force the model to “fit” known text sequences (teacher forcing / memorization)

$W=w\_{1} w\_{2} w\_{3} \dots w\_{n-2} w\_{n-1} w\_{N}$
Input: $w\_{1} w\_{2} w\_{3} \dots w\_{n-2} w\_{n-1}$
Output: $w\_{2} w\_{3} \dots w\_{n-2} w\_{n-1} w\_{N}$

In generation: this model uses its own output at time step $t-1$ as input for time step $t$ (Auto regressive, or sampling mode)

RNN Language Modeling

Basic step-wise operation

Input: $x\_t = [0, 0, 0, 1, \dots ]$
Label: $y\_t = [0, 0, 1, 0, \dots]$
Input to word embeddings:
$$ e\_t = x\_t \cdot W\_{emb}^T $$
Hidden layer:
$$ h\_t = \operatorname{LSTM}(x\_t, (t\_{t-1}, c\_{t-1})) $$
Output:
$$ o\_t = h\_t \cdot W\_{out}^T $$
Softmax:
$$ p\_t = \operatorname{softmax}(o\_t) $$
Loss: Cross Entropy
$$ L\_t = \sum\_i y\_{t\_{i}} \log p(y\_{t\_{i}}) $$
($y\_{t\_{i}}$: the $i$-th element of $y\_t$)

$$ > \frac{d L\_t}{d o\_t} = p\_t - y\_t > $$

Output layer

Because of the softmax, the output layer is a distribution over the vocabulary (The probability of each item given the context)

“Teacher-Forcing” method

We do not really tell the model to generate “all”
But Applying the Cross Entropy Loss forces the distribution to favour “all”

Backpropagation in the model

Gradient of the loss function
$$ \frac{d L\_t}{d o\_t} = p\_t - y\_t $$
$y\_t$ in Char-Language-Model is an one-hot-vector

$\to$ The gradient is positive everywhere and negative in the label position
$p\_{t\_i} \in (0, 1)$

Assume the label is in the $i$-th position.
- In non-label position:
  $$ > \forall j \neq i: \qquad y\_{t\_j} = 0 \\\\ > \Rightarrow p\_{t\_i} - y\_{t\_i} = p\_{t\_i} - 0 = p\_{t\_i} > 0 > $$
- In label position:
  $$ > y\_{t\_i} = 1 \Rightarrow p\_{t\_i} - y\_{t\_i} = p\_{t\_i} - 1 < 0 > $$
Gradient at the hidden layer:
- The hidden layer $h\_t$ receives two sources of gradients: - Coming from the loss of the current state - Coming from the gradient carried over from the future - Similary the memory cell $c\_t$: - Receives the gradient from the $h\_t$ - Summing up that gradient with the gradient coming from the future

Generate a New Sequence

Start from a memory state of the RNN (often initialized as zeros)
Run the network through a seed
Generate the probability distribution given the seed and the memory state
“Sample” a new token from the distribution
Use the new token as the seed and carry the new memory over to keep generating

Sequence-to-Sequence Models

💡 Main idea: from $P(W)$ to $P(W \mid C)$ with $C$ being a context

Controlling Generation with RNN

Hints:

The generated sequence depends on
- The first input(s)
- The first recurrent hidden state
The final hidden state of the network after rolling contains the compressed information about the sequence

Sequence-to-Sequence problem

Given: sequence $X$ of variable length (source sequence)
$$ X = (X\_1, X\_2, \dots, X\_m) $$
Task: generate a new sequence $Y$ that has the same content
$$ Y = (Y\_1, Y\_2, \dots, X\_n) $$
Training:
- Given the dataset $𝐷$ containing pairs of parallel sentences $(𝑋, 𝑌)$
- Training objective
  $$ \log P\left(Y^{\*} \mid X^{\*}\right) \qquad \forall \left(\mathrm{Y}^{\*}, \mathrm{X}^{\*}\right) \in D $$

Encoder-Decoder model

Encoder

Transforms the input into neural representation
Input
- Discret variables: require using embedding to be “compatible” with neural networks
- Continuous variables: can be “raw features” of the input
  - Speech signals
  - Image pixels
The encoder represents the $X$ part
The generative chain does not incline any generation from the encoder

Decoder

The encoder gives the decoder a representation of the source input
The decoder would try to “decode” the information from that representation
Key idea: the encoder representation is the $H\_0$ of the decoder network
The operation is identical to the character-based language model
- Back-propagation through time provides the gradient w.r.t any hidden state
  - For the decoder: the BPTT is identical to the language model
  - For the encoder: At each step the encoder hidden state only has 1 source of gradient

Problem with Encoder-Decoder Model

The model worked well to translate short sentences. But long sentences (more than 10 words) suffered greatly 😢

🔴 Main problem:

Long-range dependency
Gradient starving

Funny trick Solution: Reversing the source sentence to make the sentence starting words closer. (pretty “hacky” 😈)

Observation for solving this problem

Each word in the source sentence can be aligned with some words in the target sentence. (“alignment” in Machine Translation)

We can try to establish a connection between the aligned words.

Seq2Seq with Attention

As mentioned, we want to find the alignment between decoder-encoder. However, Our decoder only looks at the final and compressed encoder state to find the information.

Therefore we have to modify the decoder to do better! 💪

We will use superscript $e$ and $d$ to distinguish Encoder and Decoder.

E.g.:

$H\_j^e$:$j$-th Hidden state of Encoder
$H\_j^d$:$j$-th Hidden state of Decoder

Run the encoder LSTM through the input sentence (Read the sentence and encode it into states
The LSTM operation gives us some assumption about $H\_j^e$
The state $H\_j^e$ contains information about
- the word $W\_j$ (because of input gate)
- some information about the surrounding (because of memory))

Now we start generating the translation with the decoder.

The LSTM consumes the EOS token (always) and the hidden states copied over to get the first hidden state $H\_0$
From $H\_0$ we have to generate the first word, and we need to look back to the encoder.
- Here we ask the question:
  
  “Which word is responsible to generate the first word?”
- However, it’s not easy to answer this question
  - First we don’t know 😭
  - Second there might be more than one relevant word (like when we translate phrases or compound words) 🤪

The best way to find out is: to check all of the words! 💪

$H\_0$ has to connect all $H\_i^e$ in the encoder side for querying
Each “connection” will return a score $\alpha\_i$ indicating how relevant $H\_i^e$ is to generate the translation from $H\_0$
- Generate $\alpha_i^0$
  1. A feed forward neural network with nonlinearity
  $$ \alpha\_{i}^{0}=\mathrm{W}\_{2} \cdot \tanh \left(\mathrm{W}\_{1} \cdot\left[\mathrm{H}_{0}, \mathrm{H}\_{1}^{\mathrm{e}}\right]+b\_{1}\right) $$
  1. Use Softmax to get probabilities $$ \alpha \leftarrow \operatorname{softmax}(\alpha) $$
The higher $\alpha\_i$ is, the more relevant the state $H\_i^e$ is

After $H\_0$ asks “Everyone” in the encoder, it needs to sum up the information

$$ C\_0 = \sum\_i \alpha\_i^0 H\_i^e $$

($C\_0$ is the summarization of the information in the encoder that is the most relevant to $H\_0$ to generate the first word in the decoder)

Now we can answer the question “Which word is responsible to generate the first word?”

The answer is: the words with highest $\alpha$ coefficients

Combine $H\_0$ with $C\_0$
Generate the softmax output $P\_0$ from $\hat{H}\_0$
Go to the next step

In general, at time step $t$:

Decoder LSTM generates the hidden state $H\_t$ from the memory $H\_{t-1}$
$H\_t$ “pays attention” to the encoder states to know which source information is relevant
Generate $\alpha\_i^t$ from each $H\_i^e$
Weighted sum for “context vector” $C\_t$
Combine $C\_t$ and $H\_t$ then generates $P\_t$ for output distribution
- Use a feed-forward neural network
  $$ \hat{H}\_t = \operatorname{tanh}(W\cdot [C\_t, H\_t]) $$
- Or use a RNN
  $$ \hat{H}\_t = \operatorname{RNN}(C\_t, H\_t) $$

Training

Very similar to basic Encoder-Decoder
Since the scoring neural network is continuous, we can use backpropagation
No loner gradient starving on the encoder side

Pratical Suggestions for Training

Loss is too high and generation is garbarge
- Try to match your implementation with the LSTM equations
- Check the gradients by gradcheck
- Note that the gradcheck can still return some weights not passing the relative error check, but acceptable if only several (1 or 2% of the weights) cannot pass
Loss looks decreasing and the generated text looks correct but not readable
- In the sampling process, you can encourage the model to generate with higher certainty by using “argmax” (taking the char with highest probability)
- But “always using argmax” will look terrible
- A mixture of argmax and sampling can be used to ensure spelling correctness and exploration
Large network size is normally needed for character based models

👍 Attention

Sun, 16 Aug 2020 00:00:00 +0000

Core Idea

The main assumption in sequence modelling networks such as RNNs, LSTMs and GRUs is that the current state holds information for the whole of input seen so far. Hence the final state of a RNN after reading the whole input sequence should contain complete information about that sequence. But this seems to be too strong a condition and too much to ask.

Attention mechanism relax this assumption and proposes that we should look at the hidden states corresponding to the whole input sequence in order to make any prediction.

Details

The architecture of attention mechanism:

The network is shown in a state:

the encoder (lower part of the figure) has computed the hidden states $h\_j$ corresponding to each input $X\_j$
the decoder (top part of the figure) has run for $t-1$ steps and is now going to produce output for time step $t$.

The whole process can be divided into four steps:

Encoding
Computing Attention Weights/Alignment
Creating context vector
Decoding/Translation

Encoding

$(X\_1, X\_2, \dots, X\_T)$: Input sequence
- $T$: Length of sequence
$(\overrightarrow{h}\_{1}, \overrightarrow{h}\_{2}, \dots, \overrightarrow{h}\_{T})$: Hidden state of the forward RNN
$(\overleftarrow{h}\_{1}, \overleftarrow{h}\_{2}, \ldots \overleftarrow{h}\_{T})$: Hidden state of the backward RNN
The hidden state for the $j$-th input $h\_j$ is the concatenation of $j$-th hidden states of forward and backward RNNs.
$$ h\_{j}=\left[\overrightarrow{h}\_{j} ; \overleftarrow{h}\_{j}\right], \quad \forall j \in[1, T] $$

Computing Attention Weights/Alignment

At each time step $t$ of the decoder, the amount of attention to be paid to the hidden encoder unit $h\_j$ is denoted by $\alpha_{tj}$ and calculated as a function of both $h\_j$ and previous hidden state of decoder $s\_{t-1}$:

$$ \begin{array}{l} e\_{t j}=\boldsymbol{a}\left(h\_{j}, s\_{t-1}\right), \forall j \in[1, T] \\\\ \\\\ \alpha_{t j}=\frac{\displaystyle \exp \left(e\_{t j}\right)}{\displaystyle \sum_{k=1}^{T} \exp \left(e\_{t k}\right)} \end{array} $$

$\boldsymbol{a}(\cdot)$: parametrized as a feedforward neural network that runs for all $j$ at the decoding time step $t$
$\alpha\_{tj} \in [0, 1]$
$\displaystyle \sum\_j \alpha\_{tj} = 1$
$\alpha\_{tj}$ can be visualized as the attention paid by decoder at time step $t$ to the hidden ecncoder unit $h\_j$

Computing Context Vector

Now we compute the context vector. The context vector is simply a linear combination of the hidden weights $h\_j$ weighted by the attention values $\alpha_{tj}$ that we’ve computed in the precdeing step:

$$ c\_t = \sum\_{j=1}^T \alpha\_{tj}h\_j $$

From the equation we can see that $\alpha_{tj}$ determines how much $h\_j$ affects the context $c\_t$. The higher the value, the higher the impact of $h\_j$ on the context for time $t$.

Decoding/Translation

Compute the new hidden state $s\_t$ using

the context vector $c\_t$
the previous hidden state of the decoder $s\_{t-1}$
the previous output $y\_{t-1}$

$$ s\_{t}=f\left(s\_{t-1}, y\_{t-1}, c\_{t}\right) $$

The output at time step $t$ is

$$ p\left(y\_{t} \mid y\_{1}, y\_{2}, \ldots y\_{t-1}, x\right)=g\left(y\_{t-1}, s\_{t}, c\_{i}\right) $$

In the paper, authors have used a GRU cell for $f$ and a similar function for $g$.

Reference

👍 Transformer

Sun, 23 Aug 2020 00:00:00 +0000

TL;DR

Transformer

High-Level Look

Let’s begin by looking at the model as a single black box. In a machine translation application, it would take a sentence in one language, and output its translation in another.

The transformer consists of

an encoding component
a decoding component
connections between them

Let’s take a deeper look:

The encoding component is a stack of encoders (the paper stacks six of them on top of each other – there’s nothing magical about the number six, one can definitely experiment with other arrangements).
The decoding component is a stack of decoders of the same number.

Encoder

The encoders are all identical in structure (yet they do NOT share weights). Each one is composed of two sub-layers:

Self-attention layer: helps the encoder to look at other words in the input sentence as it encodes a specific word.
Feed Forwrd Neural Network (FFNN): The exact same feed-forward network is independently applied to each position.

Decoder

The decoder has both those layers, but between them is an attention layer that helps the decoder focus on relevant parts of the input sentence (similar what attention does in seq2seq models)

Encoding Component

How tensors/vectors flow

As is the case in NLP applications in general, we begin by turning each input word into a vector using an embedding algorithm.

Each word is embedded into a vector of size 512. We’ll represent those vectors with these simple boxes.

Note that the embedding ONLY happens in the bottom-most encoder.

The abstraction that is common to all the encoders is that they receive a list of vectors each of the size 512 (The size of this list is hyperparameter we can set – basically it would be the length of the longest sentence in our training dataset.)

In bottom encoder: word embeddings
In other encoders: output of the encoder that’s directly below

After embedding the words in our input sequence, each of them flows through each of the two layers of the encoder.

The word in each position flows through its own path in the encoder. There are dependencies between these paths in the self-attention layer.
The feed-forward layer does not have those dependencies, thus the various paths can be executed in parallel while flowing through the feed-forward layer. 👏

In summary, An encoder

receives a list of vectors as input
processes this list by passing these vectors into a ‘self-attention’ layer, then into a feed-forward neural network
sends out the output upwards to the next encoder.

Self-Attention

Say the following sentence is an input sentence we want to translate:

”The animal didn't cross the street because it was too tired”

What does “it” in this sentence refer to? Is it referring to the street or to the animal? It’s a simple question to a human, but not as simple to an algorithm.

As the model processes each word (each position in the input sequence), self attention allows it to look at other positions in the input sequence for clues that can help lead to a better encoding for this word. Therefore, when the model is processing the word “it”, self-attention allows it to associate “it” with “animal”.

We can think of how maintaining a hidden state allows an RNN to incorporate its representation of previous words/vectors it has processed with the current one it’s processing. Self-attention is the method the Transformer uses to bake the “understanding” of other relevant words into the one we’re currently processing.

When encoding the word ‘it’ in 5.encoder (the top encoder in the stack), part of the attention mechanism was focusing on ‘The Animal’, and baked a part of its representation into the encoding of ‘it’. (Visualization source: Tensor2Tensor notebook)

Self-Attention in Detail

Calculate self-attention using vectors:

Create three vectors
- Query vector
- Key vector
- Value vector
from each of the encoder’s input vectors by multiplying the embedding by three matrices that we trained during the training process.

These new vectors are smaller in dimension (64) than the embedding vector (512). They don’t have to be smaller, this is an architecture choice to make the computation of multiheaded attention (mostly) constant.

In the following steps we will keep using the first word “Thinking” as example.

calculate a score
- The score determines how much focus to place on other parts of the input sentence as we encode a word at a certain position.
  - Say we’re calculating the self-attention for the first word in this example, “Thinking”. We need to score each word of the input sentence against this word.
- The score is calculated by taking the dot product of the query vector with the key vector of the respective word we’re scoring.
  - So if we’re processing the self-attention for the word in position #1, the first score would be the dot product of q1 and k1. The second score would be the dot product of q1 and k2.
Divide the scores by the square root of the dimension of the key vectors
- In paper the dimension of the key vectors is 64. Therefore devide the scores by 8
  
  (There could be other possible values here, but this is the default)
- This leads to having more stable gradients. 👏
Pass the result through a softmax operation
- Softmax normalizes the scores so they’re all positive and add up to 1.
- The softmax score determines how much each word will be expressed at this position.
  - Clearly the word at this position will have the highest softmax score
  - but sometimes it’s useful to attend to another word that is relevant to the current word.
Multiply each value vector by the softmax score (in preparation to sum them up)
- Keep intact the values of the word(s) we want to focus on
- drown-out irrelevant words (by multiplying them by tiny numbers like 0.001, for example)
Sum up the weighted value vectors
- This produces the output of the self-attention layer at this position (for the first word).
- The resulting vector is one we can send along to the feed-forward neural network.

Matrix Calculation of Self-Attention

In the actual implementation, the above calculation is done in matrix form for faster processing.

Calculate the Query, Key, and Value matrices
- Pack our embeddings into a matrix X
- Multiplying it by the weight matrices we’ve trained ($W^Q$, $W^K$, $W^V$)
Calculate the outputs of the self-attention layer

“Multi-headed” Mechanism

The paper further refined the self-attention layer by adding a mechanism called “multi-headed” attention. This improves the performance of the attention layer in two ways:

Expands the model’s ability to focus on different positions
Gives the attention layer multiple “representation subspaces”
- With multi-headed attention we have not only one, but multiple sets of Query/Key/Value weight matrices
  
  (the Transformer uses eight attention heads, so we end up with eight sets for each encoder/decoder)
- Each of these sets is randomly initialized.
- After training, each set is used to project the input embeddings (or vectors from lower encoders/decoders) into a different representation subspace.
With multi-headed attention, we maintain separate $Q$/$K$/$V$ weight matrices for each head resulting in different $Q$/$K$/$V$ matrices. As we did before, we multiply $X$ by the $W^Q$/$W^K$/$W^V$ matrices to produce $Q$/$K$/$V$ matrices.

If we do the same self-attention calculation as above, just eight different times with different weight matrices, we end up with eight different $Z$ matrices

Since the feed-forward layer is expecting a single matrix (a vector for each word), we concat the matrices then multiple them by an additional weights matrix $W^O$.

Summarize them into a figure

Example

Let’s revisit our example from before to see where the different attention heads are focusing as we encode the word “it” in our example sentence:

As we encode the word ‘it’, one attention head is focusing most on ’the animal’, while another is focusing on ’tired’ – in a sense, the model’s representation of the word ‘it’ bakes in some of the representation of both ‘animal’ and ’tired’. (Visualization source: Tensor2Tensor notebook)

Representing The Order of The Sequence Using Positional Encoding

In order to represent the order of the words in the input sequence, the transformer adds a vector to each input embedding.

These vectors follow a specific pattern that the model learns, which helps it determine the position of each word, or the distance between different words in the sequence.
💡 Intuition: adding these values to the embeddings provides meaningful distances between the embedding vectors once they’re projected into $Q$/$K$/$V$ vectors and during dot-product attention.

To give the model a sense of the order of the words, we add positional encoding vectors – the values of which follow a specific pattern.

For instance, if we assumed the embedding has a dimensionality of 4, the actual positional encodings would look like this:

A real example of positional encoding with a toy embedding size of 4

What might this pattern look like?

In the following figure, each row corresponds the a positional encoding of a vector.

The first row would be the vector we’d add to the embedding of the first word in an input sequence.
Each row contains 512 values – each with a value between 1 and -1.

A real example of positional encoding for 20 words (rows) with an embedding size of 512 (columns). You can see that it appears split in half down the center. That’s because the values of the left half are generated by one function (which uses sine), and the right half is generated by another function (which uses cosine). They’re then concatenated to form each of the positional encoding vectors.

July 2020 Update: The positional encoding shown above is from the Tranformer2Transformer implementation of the Transformer. The method shown in the paper is slightly different in that it doesn’t directly concatenate, but interweaves the two signals. The following figure shows what that looks like.

The Residuals

Each sub-layer (self-attention, FFNN) in each encoder has a residual connection around it, and is followed by a layer-normalization step.

Visualize the vectors and the layer-norm operation associated with self attention

This goes for the sub-layers of the decoder as well.

If we’re to think of a Transformer of 2 stacked encoders and decoders, it would look something like this:

Decoding Component

After finishing the encoding phase, we begin the decoding phase. Each step in the decoding phase outputs an element from the output sequence (the English translation sentence in this case).

The encoder start by processing the input sequence.
The output of the top encoder is then transformed into a set of attention vectors $K$ and $V$.
These are to be used by each decoder in its “encoder-decoder attention” layer which helps the decoder focus on appropriate places in the input sequence

The following steps repeat the process until a special symbol <eos> is reached, indicating the transformer decoder has completed its output.

The output of each step is fed to the bottom decoder in the next time step,
The decoders bubble up their decoding results just like the encoders did.
- Just like we did with the encoder inputs, we embed and add positional encoding to those decoder inputs to indicate the position of each word.

Note that the self attention layers in the decoder operate in a slightly different way than the one in the encoder: In the decoder, the self-attention layer is ONLY allowed to attend to earlier positions in the output sequence.

This can be done by masking future positions (setting them to -inf) before the softmax step in the self-attention calculation.

The attention scores for ‘am’ have values for itself and all other words before, but zero for the word ‘fine’. This essentially tells the model to put NO focus on the word ‘fine’.

The Final Linear and Softmax Layer

The final Linear layer + Softmax layer: Turn a vector of floats (the output of the decoder) into a word

Linear layer: a simple fully connected neural network that projects the vector produced by the stack of decoders, into a much, much larger vector called a logits vector
- Let’s assume that our model knows 10,000 unique English words (our model’s “output vocabulary”) that it’s learned from its training dataset. This would make the logits vector 10,000 cells wide – each cell corresponding to the score of a unique word.
Softmax layer
- Turns those scores into probabilities (all positive, all add up to 1.0).
- The cell with the highest probability is chosen, and the word associated with it is produced as the output for this time step.

Training

Word Representation

During training, an untrained model would go through the exact same forward pass. Since we are training it on a labeled training dataset, we can compare its output with the actual correct output.

To visualize this, let’s assume our output vocabulary only contains six words: “a”, “am”, “i”, “thanks”, “student”, and “<eos>” (short for ‘end of sentence’).

The output vocabulary of our model is created in the preprocessing phase before we even begin training.

After defining the output vocabulary, use One-Hot-encoding to indicate each word in our vocabulary. E.g., we can indicate the word “am” using the following vector:

The Loss Function

Say it’s our first step in the training phase, and we’re training it on a simple example – translating “merci” into “thanks”.

We want the output to be a probability distribution indicating the word “thanks”. But since this model is not yet trained, that’s unlikely to happen just yet.

Since the model’s parameters (weights) are all initialized randomly, the (untrained) model produces a probability distribution with arbitrary values for each cell/word. We can compare it with the actual output, then tweak all the model’s weights using backpropagation to make the output closer to the desired output.

Compare two probability distributions: simply subtract one from the other. (For more details, look at cross-entropy and Kullback–Leibler divergence.)

Note that the example above is an oversimplified example.

In practice, we’ll use a sentence longer than one word. For example

input: “je suis étudiant” and
expected output: “i am a student”.

What this really means, is that we want our model to successively output probability distributions where:

Each probability distribution is represented by a vector of width vocab_size (6 in our toy example, but more realistically a number like 30,000 or 50,000)
The first probability distribution has the highest probability at the cell associated with the word “i”
The second probability distribution has the highest probability at the cell associated with the word “am”
And so on, until the fifth output distribution indicates ‘<end of sentence>’ symbol, which also has a cell associated with it from the 10,000 element vocabulary.

The targeted probability distributions we’ll train our model against in the training example for one sample sentence.

After training the model for enough time on a large enough dataset, we would hope the produced probability distributions would look like this:

Now, because the model produces the outputs one at a time, we can assume that the model is selecting the word with the highest probability from that probability distribution and throwing away the rest (“greedy decoding”).

Reference

The Illustrated Transformer - great explanation with a number of illustrations 👍🔥
Paper: Attention is All You Need
Pytorch implementation: guide annotating the paper with PyTorch implementation