Backpropagation for Optimization

Backpropagation for Optimization

Applied Deep Learning

March 10th, 2020 http://adl.miulab.tw

最佳化參數

Parameter Optimization

2

Notation Summary

3

l

a

a

l

z

z

l

b

b

: output vector of a layer

: input of activation function : input vector of activation function for a layer

: a weight

: a weight matrix : a bias

: a bias vector

Layer Output Relation – from a to z

4

..…

nodes Nl

Layerl

….. …..

Layerl

− 1

−1

Nl

..…

1 2

j

1 2

i

1 1

−

al

1 2

−

al

−1 l

aj

a₁l

a₂l

l

ai

z₁l

z₂l

l

zi

al

zl

−1

al

… …

Layer Output Relation – from z to a

5

…..

nodes Nl

Layerl

….. …..

Layerl

− 1

−1

Nl

…..

1 2

j

1 2

i

1 1

−

al

1 2

−

al

−1 l

aj

a₁l

a₂l

l

ai

z₁l

z₂l

l

zi

al

zl

−1

al

( ) ( )

( )

 

 

 







 

 

 







=

 

 

 







 

 

 















l i l l

l i l l

z z z

a a a







2 1 2

1

( )

i

z

a = 

( )

l

z

a = 

Layer Output Relation

6

…..

nodes Nl

Layerl

….. …..

Layerl

− 1

−1

Nl

…..

1 2

j

1 2

i

1 1

−

al

1 2

−

al

−1 l

aj

a₁l

a₂l

l

ai

z₁l

z₂l

l

zi

al

zl

−1

al

l l

l

l

W a b

z =

+

( )

l

z

a = 

Neural Network Formulation

◉

7

Layer 1 Layer 2 Layer L

Input Output

= = =

M

N

R

R

f : →

x1

x2

….. …..

y1

y2

….. …..

…

… …

…

…

xN

vector x

vector

y

Neural Network Formulation

◉

8

M

N

R

R

f : →

Layer 1 Layer 2 Layer L

Input Output

x1

x2

….. …..

y1

y2

….. …..

…

… …

…

…

xN

vector x

vector

y

Loss Function for Training

9

A “Good” function:

Define an example loss function:

sum over the error of all training samples

f *

“Best” Function

Training Data

Model: Hypothesis Function Set

2

1, f f

Training: Pick the best function f ^*

( ) ( )





“It claims too much.”

-

x

ˆy: function input function output

Gradient Descent for Neural Network

10

Computing the gradient includes millions of parameters.

To compute it efficiently, we use backpropagation.

Algorithm

Initialization: start at 𝜃⁰ while(𝜃^(𝑖+1) ≠ 𝜃^𝑖)

{

compute gradient at 𝜃^𝑖 update parameters }

如何有效率地計算大量參數呢？

Backpropagation

11

Forward v.s. Back Propagation

◉

○ forward propagation

■ from input 𝑥 to output 𝑦 information flows forward through the network

■

○ back-propagation

■ allows the information from the cost to then flow backwards through the network, in order to compute the gradient

■ can be applied to any function 12

x1

x2

… …

y1

y2

… …

…

…

…

xN

Chain Rule

13

z y x w f f

f

forward propagation for cost

back-propagation for gradient

Gradient Descent for Neural Network

14

Computing the gradient includes millions of parameters.

To compute it efficiently, we use backpropagation.

Algorithm

Initialization: start at 𝜃⁰ while(𝜃^(𝑖+1) ≠ 𝜃^𝑖)

{

compute gradient at 𝜃^𝑖 update parameters }

15

… …

Layer^l

1 1

−

al

1 2

−

al

−1 l

aj

… …

Layerl −1

l

ai

… …

1 2

j i

l

bi

1

… …

Layer^l

1 1

−

al

1 2

−

al

−1 l

aj

… …

Layerl −1

l

ai

… …

1 2

j i

l

bi

1

17

Layer^l

… …

l

bi

1

… …

x₁r

x₂r

r

xj

Input

… …

Layer^l

1 1

−

al

1 2

−

al

−1 l

aj

… …

Layerl −1

l

ai

… …

1 2

j i

l

bi

1

19

… …

Layer^l

1 1

−

al

1 2

−

al

−1 l

aj

… …

Layerl −1

l

ai

… …

1 2

j i

l

bi

1

… …

Layer l-1

1 2

j

… …

1 2

i Layer l

… …

1 2

k

Layer l+1

…

… …

…

…

… … …

1 2

n Layer L (output layer)

l

δi

δ₂l

δ₁l

+1 l

δk 1 2

+

δl 1 1

+

δl

L

δn L

δ2

δ₁L

δl δ^l+1 δ^L-1 δ^L

the propagated gradient

corresponding to the l-th layer

Idea: computing 𝛿^𝑙 layer by layer (from 𝛿^𝐿 to 𝛿¹) is more efficient

◉ Idea: from L to 1



Initialization: compute 𝛿

 Compute 𝛿

based on 𝛿

21

◉ Idea: from L to 1



Initialization: compute 𝜹

 Compute 𝛿

based on 𝛿

𝜕𝐶 𝜕𝑦Τ _𝑖 depends on the loss function

◉ Idea: from L to 1



Initialization: compute 𝜹

 Compute 𝛿

based on 𝛿

23 

( )

L

z

z

◉ Idea: from L to 1



Initialization: compute 𝛿

 Compute 𝜹

based on 𝜹

… …

1 2

i Layer l

… …

1 2

k

Layer l+1

l

δi

δ₂l

δ₁l

+1 l

δk 1 2

+

δl 1 1

+

δl

+1

δl

δl l

i l

i

Δa

Δz →

1 1

+

Δz

Δz

+1 l

Δz

……

◉ Idea: from L to 1



Initialization: compute 𝛿

 Compute 𝜹

based on 𝜹

25

… …

1 2

i Layer l

… …

1 2

k

Layer l+1

l

δi

δ₂l

δ₁l

+1 l

δk 1 2

+

δl 1 1

+

δl

+1

δl

δl

…

1 1

−

al

−1 l

aj

… ^lia…

1

j i

l

bi

1

◉

… …

1 2

i

Layer l

… …

1 2

k

Layer l+1

l

δ

δ

δ

+1 l

δ

l+

δ

1 1

+

δ

+1

δ

δ

l

δ

( )

 ^



multiply a constant

+1 l

δ

+

δ

+

δ

… …

output

input

27

2

…

( )

 +

 z^l

( )

 +

 z^l

( )

 z_k^l

1

k 2

1

i

…

Layer l+1 Layer l

( )

^



( )





( )





l

δ

δ

δ

+1 l

δ

+

δ

+

δ

+1

δ

δ

◉ Idea: from L to 1



Initialization: compute 𝛿

 Compute 𝛿

based on 𝛿

1

2

n

…

y1

C





( )





( )





( )

^



y2

C





yn

C



 Layer L

2

…

( )

 +

 z^l

( )

 +

 z^l

( )

z^l_k

1

k 2

1

i

…

Layer l+1 Layer

l

( )

^



( )

^



( )

^



δ₁l

δ₂l

l

δi

2

…

( )

 −

 z

1

m

Layer L-1

…

…

… _{( )}

( )

( )

C

L 1

-

 L

( )

 −

 z

( )

z_m

+1

δl

δl

Backpropagation

29

Forward Pass

⋮

⋮

+1 l

a

+

a

+

a

… …

1 2

i … …

1 2

l

k a

Layer l+1

a

a

a

a

Layer l

Backpropagation

30

Backward Pass

⋮

⋮

2

…

( )

 +

 z^l

( )

 +

 z^l

( )

z_k^l

1

k 2

1

i

…

Layer l+1 Layer l

( )

^



( )

^



( )





l

δi

δ₂l

δ₁l

+1 l

δk 1 2

+

δl 1 1

+

δl

+1

δl

δl

Gradient Descent for Optimization

31

Algorithm

Initialization: start at 𝜃⁰ while(𝜃^(𝑖+1) ≠ 𝜃^𝑖)

{

compute gradient at 𝜃^𝑖 update parameters }

Concluding Remarks

32

… …

1 2

j

… …

1 2

l i wij

Layerl

Layerl

− 1





=

− 

1

1 1 l x

l a

j l l j



Backward Pass

⋮

⋮

Forward Pass

⋮

⋮

Compute the gradient based on two pre-computed terms from backward and forward passes