Slide credit from 1

Slide credit from Hung-Yi Lee & Richard Socher 1

Review

Recurrent Neural Network

2

Recurrent Neural Network

Idea: condition the neural network on all previous words and tie the weights at each time step

Assumption: temporal information matters

3

RNN Language Modeling

4

vector of “START”

P(next word is

“wreck”)

vector of “wreck”

P(next word is “a”)

vector of “a”

P(next word is

“nice”)

vector of “nice”

P(next word is

“beach”)

input hidden output

context vector word prob dist

Idea: pass the information from the previous hidden layer to leverage all contexts

RNNLM Formulation

5

At each time step,

…… ……

……

……

vector of the current word probability of the next word

Recurrent Neural Network Definition

6 http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/

: tanh, ReLU

Model Training

All model parameters can be updated by

7 http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/

y

y

y

target

predicted

Outline

Language Modeling

◦N-gram Language Model

◦Feed-Forward Neural Language Model

◦Recurrent Neural Network Language Model (RNNLM) Recurrent Neural Network

◦Definition

◦Training via Backpropagation through Time (BPTT)

◦Training Issue Applications

◦Sequential Input

◦Sequential Output

◦ Aligned Sequential Pairs (Tagging)

◦ Unaligned Sequential Pairs (Seq2Seq/Encoder-Decoder)

8

Backpropagation

9

… …

1 2

j … …

1 2

l

i w

Layer l Layer l  1



 









1

1

1 l x

l a

j l l j



Backward Pass

⋮

⋮

Forward Pass

⋮

⋮

Error

signal

Backpropagation

10 l



Backward Pass

⋮

⋮

Error signal

1

2

n

…

y1

C





 

^



 ^z2^L





 zn^L





y2

C





yn

C





Layer L

2 1

i

…

Layer l

 





 





 

^



δ₁l

δ₂l

l

δi

2

…

 1^L1

 

 z

1

m

Layer L-1

…

…

… _{ }

 

 

C

 L 1

-

 L

 2^{L 1}

 

 z

 ^L1

z_m

δl

Backpropagation through Time (BPTT)

Unfold

◦ Input: init, x

, x

, …, x

◦ Output: o

◦ Target: y

11

init

s_t y_t

x_t o_t

x_t-1 s_t-1

x_t-2

x₁ s₁

s_t-2

o

C





o

C





o

C





  ^y

 C

…

Backpropagation through Time (BPTT)

Unfold

◦ Input: init, x

, x

, …, x

◦ Output: o

◦ Target: y

12

init

s_t y_t

x_t o_t

x_t-1 s_t-1

x_t-2

x₁ s₁

s_t-2

1

2

n

…

1

2

n

…

  ^y

 C

Backpropagation through Time (BPTT)

Unfold

◦ Input: init, x

, x

, …, x

◦ Output: o

◦ Target: y

13

init

s_t y_t

x_t o_t

x_t-1 s_t-1

x_t-2

x₁ s₁

s_t-2

  ^y

 C

Backpropagation through Time (BPTT)

Unfold

◦ Input: init, x

, x

, …, x

◦ Output: o

◦ Target: y

14

init

s_t y_t

x_t o_t

x_t-1 s_t-1

x_t-2

x₁ s₁

s_t-2

j

i

i

j i j

i j

  ^y

 C

Weights are tied together

the same memory

pointer pointer

Backpropagation through Time (BPTT)

Unfold

◦ Input: init, x

, x

, …, x

◦ Output: o

◦ Target: y

15

init

s_t y_t

x_t o_t

x_t-1 s_t-1

x_t-2

x₁ s₁

s_t-2

j

i

k i

k j i j

i

k j

  ^y

 C

Weights are tied together

BPTT

16

For 𝐶

Backward Pass:

For 𝐶

For 𝐶

For 𝐶

Forward Pass: Compute s

, s

, s

, s

……

y

y

y

x

x

x

o

o

o

init

y

x

o

𝐶

𝐶

𝐶

𝐶

s

s

s

s

RNN Training Issue

The gradient is a product of Jacobian matrices, each associated with a step in the forward computation

Multiply the same matrix at each time step during backprop

17

The gradient becomes very small or very large quickly

 vanishing or exploding gradient

Bengio et al., “Learning long-term dependencies with gradient descent is difficult,” IEEE Trans. of Neural Networks, 1994. [link]

Pascanu et al., “On the difficulty of training recurrent neural networks,” in ICML, 2013. [link]

w

w

C ost

Rough Error Surface

Bengio et al., “Learning long-term dependencies with gradient descent is difficult,” IEEE Trans. of Neural Networks, 1994. [link] 18

Pascanu et al., “On the difficulty of training recurrent neural networks,” in ICML, 2013. [link]

The error surface is either very flat or very steep

Possible Solutions

Recurrent Neural Network

19

Exploding Gradient: Clipping

20

w₂

w₁

Cost

clipped gradient Idea: control the gradient value to avoid exploding

Parameter setting: values from half to ten times the average can still yield convergence

Vanishing Gradient: Initialization + ReLU

IRNN

◦ initialize all W as identity matrix I

◦ use ReLU for activation functions

Le et al., “A Simple Way to Initialize Recurrent Networks of Rectified Linear Units,” arXiv, 2016. [link] 21

Vanishing Gradient: Gating Mechanism

RNN models temporal sequence information

◦ can handle “long-term dependencies” in theory

http://colah.github.io/posts/2015-08-Understanding-LSTMs/ 22

Issue: RNN cannot handle such “long-term dependencies” in practice due to vanishing gradient

 apply the gating mechanism to directly encode the long-distance information

“I grew up in France…

I speak fluent French.”

Extension

Recurrent Neural Network

23

Bidirectional RNN

24

ℎ = ℎ; ℎ represents (summarizes) the past and future around a single token

Deep Bidirectional RNN

25

Each memory layer passes an intermediate representation to the next

Concluding Remarks

Recurrent Neural Networks

◦ Definition

◦ Issue: Vanishing/Exploding Gradient

◦ Solution:

• Exploding Gradient: Clipping

• Vanishing Gradient: Initialization, ReLU, Gated RNNs

Extension

◦ Bidirectional

◦ Deep RNN

26