Machine Learning Soundings ( 機器學習深測)
Lecture 3: Optimization in Deep Learning
Hsuan-Tien Lin (林軒田)Department of Computer Science
& Information Engineering
National Taiwan University ( 國立台灣大學資訊工程系)
Hsuan-Tien Lin (NTU CSIE) Machine Learning Soundings 0/9
Roadmap
1
Deep Learning FoundationsLecture 1: Neural Network
automatic
pattern feature extraction
fromlayers of neurons
withbackprop
for GD/SGDLecture 3: Optimization in Deep Learning Difficulty of Deep Learning Optimization
2 Deep Learning Models
Optimization in Deep Learning Difficulty of Deep Learning Optimization
Difficulty of Deep Learning Optimization
error surface complicated
•
local minima: not as bad as imagined•
saddle points/local maxima: easily escapable (especially with SGD)•
plateau: need larger learning rate η•
ravines: need to avoid oscillationstability <> computation trade-off
slow computation of gradient (backprop)⇒ SGD on minibatch
⇒ ‘instable’ estimate of gradient
getting more stable estimate?
averaging
Hsuan-Tien Lin (NTU CSIE) Machine Learning Soundings 2/9
Running Average Estimate of Gradient
gradient descent:
w t
← wt−1
− η · vt
original minibatch SG
gradient estimate
v t
= ∆t
from one minibatch SGaveraging by multiple SG
if minibatch SG for M times at t-th iteration, each getting ∆
(m) t
, more stable gradient estimate by uniform averagingv t
=M 1
PM
m=1
∆(m) t
—needing M times more computation than original minibatch SGD
speedup by reusing each ∆ t = ∆ (1) t v t
=M 1
PM
m=1
∆t−m+1
—‘moving window’ average of SGissue with ‘moving window’ average:
Optimization in Deep Learning Difficulty of Deep Learning Optimization
Averaging SG Non-uniformly
Running Average
v t
= βvt−1
+ (1 − β)∆t
with 0 ≤ β < 1 to control how much history to take β = 0: original SGD
v t
=t
X
m=1
β
t−m
(1 − β)∆t
—size-t window, exponentially-decreasing aeveraging
SGD
with momentum: optimization direction
= current SG (∆
t
) + historical inertia (vt−1
)Hsuan-Tien Lin (NTU CSIE) Machine Learning Soundings 4/9
Benefits of SGD with Momentum
v t
= βvt−1
+ (1 − β)∆t w t
=w t−1
− ηvt
•
some variance in SG canceled out•
oscilliation across ravine dampened•
shallow local optima/saddle points escapedSGD with momentum: ‘stablize’ SG with running average
Optimization in Deep Learning Difficulty of Deep Learning Optimization
Per-Component Learning Rate
fixed learning rate :
w t
=w t−1
− ηvt
per-component learning rate :w t
=w t−1
− ηt
vt
intuition: scales error surface
want: smaller step for larger gradient component
Hsuan-Tien Lin (NTU CSIE) Machine Learning Soundings 6/9
Running Average of Gradient Magnitude
want: smaller step for larger gradient component, say
η
t
= 1 k∇E(wt
)k•
full gradient ∇E not available, SG only•
using k∆k not very stableidea: running average of ∆
t
∆t
Optimization in Deep Learning Difficulty of Deep Learning Optimization
RMSProp
u t
= βut−1
+ (1 − β)∆t
∆t
ηt
= η./√u t
+ wt
=w t−1
− ηt
∆t
RMSProp: SGD + per-component learng rate using running average of magnitude
Hsuan-Tien Lin (NTU CSIE) Machine Learning Soundings 8/9
Adam: Adaptive Moment Estimation
Adam ≈ momentum + RMSProp + global decay
v t
= β1 v t−1
+ (1 − β1
)∆t u t
= β2 u t−1
+ (1 − β2
)∆t
∆t
ηt
= η./√u t
+ ./p t/Nw t
=w t−1
− ηt
vt
•
momentum inv t
•
RMSProp inu t
•
global decay bypt/N•
(some minor correction of estimation)Adam usually more aggressive than original SGD (but can also overfit faster)
