All Kinds of GAN
Hung-yi Lee
Zoo of GAN: https://github.com/hindupuravinash/the-gan-zoo
General Idea of GAN
Sebastian Nowozin, Botond Cseke, Ryota Tomioka, “f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization”, NIPS, 2016
Basic Idea of GAN
• A generator G is a network. The network defines a probability distribution.
generator G
𝑧 𝑥 = 𝐺 𝑧
Normal Distribution
𝑃𝐺 𝑥 𝑃𝑑𝑎𝑡𝑎 𝑥
As close as possible
https://blog.openai.com/generative-models/
It is difficult to compute 𝑃𝐺 𝑥 We can only sample from the distribution.
Basic Idea of GAN
• Generator G
• G is a function, input z, output x
• Given a prior distribution P
prior(z), a probability distribution P
G(x) is defined by function G
• Discriminator D
• D is a function, input x, output scalar
• Evaluate the “difference” between P
G(x) and P
data(x)
• There is a function V(G,D).
𝐺
∗= 𝑎𝑟𝑔 min
𝐺
max
𝐷
𝑉 𝐺, 𝐷
Hard to learn by maximum likelihood
Basic Idea
𝐺
1𝐺
2𝐺
3𝑉 𝐺
1, 𝐷 𝑉 𝐺
2, 𝐷 𝑉 𝐺
3, 𝐷
𝐷 𝐷 𝐷
𝑉 = 𝐸
𝑥∼𝑃𝑑𝑎𝑡𝑎𝑙𝑜𝑔𝐷 𝑥 + 𝐸
𝑥∼𝑃𝐺𝑙𝑜𝑔 1 − 𝐷 𝑥 Given a generator G, max
𝐷
𝑉 𝐺, 𝐷 evaluate the JS divergence between 𝑃
𝐺and 𝑃
𝑑𝑎𝑡𝑎Pick the G defining 𝑃
𝐺most similar to 𝑃
𝑑𝑎𝑡𝑎𝐺
∗= 𝑎𝑟𝑔 min
𝐺
max
𝐷
𝑉 𝐺, 𝐷
• In each training iteration:
• Sample m examples 𝑥1, 𝑥2, … , 𝑥𝑚 from data distribution 𝑃𝑑𝑎𝑡𝑎 𝑥
• Sample m noise samples 𝑧1, 𝑧2, … , 𝑧𝑚 from the prior 𝑃𝑝𝑟𝑖𝑜𝑟 𝑧
• Obtaining generated data 𝑥1, 𝑥2, … , 𝑥𝑚 , 𝑥𝑖 = 𝐺 𝑧𝑖
• Update discriminator parameters 𝜃𝑑 to maximize
• ෨𝑉 = 1
𝑚 σ𝑖=1𝑚 𝑙𝑜𝑔𝐷 𝑥𝑖 + 1
𝑚 σ𝑖=1𝑚 𝑙𝑜𝑔 1 − 𝐷 𝑥𝑖
• 𝜃𝑑 ← 𝜃𝑑 + 𝜂𝛻 ෨𝑉 𝜃𝑑
• Sample another m noise samples 𝑧1, 𝑧2, … , 𝑧𝑚 from the prior 𝑃𝑝𝑟𝑖𝑜𝑟 𝑧
• Update generator parameters 𝜃𝑔 to minimize
• ෨𝑉 = 1
𝑚 σ𝑖=1𝑚 𝑙𝑜𝑔𝐷 𝑥𝑖 + 1
𝑚 σ𝑖=1𝑚 𝑙𝑜𝑔 1 − 𝐷 𝐺 𝑧𝑖
• 𝜃𝑔 ← 𝜃𝑔 − 𝜂𝛻 ෨𝑉 𝜃𝑔
Algorithm
Repeat k times Learning
D
Learning G
Initialize 𝜃𝑑 for D and 𝜃𝑔 for G
Only Once
Objective Function for Generator in Real Implementation
𝑉 = 𝐸
𝑥∼𝑃𝑑𝑎𝑡𝑎𝑙𝑜𝑔𝐷 𝑥
𝑉 = 𝐸
𝑥∼𝑃𝐺−𝑙𝑜𝑔 𝐷 𝑥
Real implementation:
label x from P
Gas positive
−𝑙𝑜𝑔 𝐷 𝑥
𝑙𝑜𝑔 1 − 𝐷 𝑥
+𝐸
𝑥∼𝑃𝐺𝑙𝑜𝑔 1 − 𝐷 𝑥
𝐷 𝑥 Slow at the beginning
Minimax GAN (MMGAN)
Non-saturating GAN (NSGAN)
𝐷𝑓 𝑃||𝑄 = න
𝑥
𝑞 𝑥 𝑓 𝑝 𝑥
𝑞 𝑥 𝑑𝑥 f is convex f(1) = 0
f-divergence
If 𝑝 𝑥 = 𝑞 𝑥 for all 𝑥
𝐷𝑓 𝑃||𝑄 = න
𝑥
𝑞 𝑥 𝑓 𝑝 𝑥
𝑞 𝑥 𝑑𝑥 = 0 𝐷𝑓 𝑃||𝑄 = න
𝑥
𝑞 𝑥 𝑓 𝑝 𝑥
𝑞 𝑥 𝑑𝑥
≥ 𝑓 න
𝑥
𝑞 𝑥 𝑝 𝑥
𝑞 𝑥 𝑑𝑥
= 𝑓 1 = 0
= 0
Because f is convex
𝑃 and 𝑄 are two distributions. 𝑝 𝑥 and q 𝑥 are the probability of sampling 𝑥.
= 1
If P and Q are the same distributions,
𝐷𝑓 𝑃||𝑄 has the
smallest value, which is 0 𝐷𝑓 𝑃||𝑄 evaluates the difference of P and Q
smallest
𝐷𝑓 𝑃||𝑄 = න
𝑥
𝑞 𝑥 𝑓 𝑝 𝑥
𝑞 𝑥 𝑑𝑥 f is convex f(1) = 0
f-divergence
𝑓 𝑥 = 𝑥 − 1 2 𝐷𝑓 𝑃||𝑄 = න
𝑥
𝑞 𝑥 𝑝 𝑥
𝑞 𝑥 − 1
2
𝑑𝑥 = න
𝑥
𝑝 𝑥 − 𝑞 𝑥 2
𝑞 𝑥 𝑑𝑥
𝐷𝑓 𝑃||𝑄 = න
𝑥
𝑞 𝑥 𝑝 𝑥
𝑞 𝑥 𝑙𝑜𝑔 𝑝 𝑥
𝑞 𝑥 𝑑𝑥 𝑓 𝑥 = 𝑥𝑙𝑜𝑔𝑥
= න
𝑥
𝑝 𝑥 𝑙𝑜𝑔 𝑝 𝑥
𝑞 𝑥 𝑑𝑥
KL
𝑓 𝑥 = −𝑙𝑜𝑔𝑥 𝐷𝑓 𝑃||𝑄 = න
𝑥
𝑞 𝑥 −𝑙𝑜𝑔 𝑝 𝑥
𝑞 𝑥 𝑑𝑥 = න
𝑥
𝑞 𝑥 𝑙𝑜𝑔 𝑞 𝑥
𝑝 𝑥 𝑑𝑥
Reverse KL
Chi Square
Fenchel Conjugate
• Every convex function f has a conjugate function f*
𝑓∗ 𝑡 = max
𝑥∈𝑑𝑜𝑚 𝑓 𝑥𝑡 − 𝑓 𝑥
𝑥2𝑡1 − 𝑓 𝑥2 𝑥3𝑡1 − 𝑓 𝑥3
𝑡1 𝑡2 𝑡
𝑓∗ 𝑡1
𝑓∗ 𝑡2
𝐷𝑓 𝑃||𝑄 = න
𝑥
𝑞 𝑥 𝑓 𝑝 𝑥
𝑞 𝑥 𝑑𝑥
f is convex, f(1) = 0
𝑥1𝑡1 − 𝑓 𝑥1
= max
𝑥∈𝑑𝑜𝑚 𝑓 𝑥𝑡1 − 𝑓 𝑥 𝑓∗ 𝑡1
𝑥1𝑡2 − 𝑓 𝑥1 𝑥2𝑡2 − 𝑓 𝑥2 𝑥3𝑡2 − 𝑓 𝑥3 𝑓∗ 𝑡2 = max
𝑥∈𝑑𝑜𝑚 𝑓 𝑥𝑡2 − 𝑓 𝑥
Fenchel Conjugate
• Every convex function f has a conjugate function f*
𝑓∗ 𝑡 = max
𝑥∈𝑑𝑜𝑚 𝑓 𝑥𝑡 − 𝑓 𝑥 𝑥1𝑡 − 𝑓 𝑥1
𝑥2𝑡 − 𝑓 𝑥2
𝑥3𝑡 − 𝑓 𝑥3
𝑡1 𝑡2 𝑡
𝑓∗ 𝑡1 𝑓∗ 𝑡2
𝐷𝑓 𝑃||𝑄 = න
𝑥
𝑞 𝑥 𝑓 𝑝 𝑥
𝑞 𝑥 𝑑𝑥
f is convex, f(1) = 0
𝑓∗ 𝑡 = max
𝑥∈𝑑𝑜𝑚 𝑓 𝑥𝑡 − 𝑓 𝑥 𝑓 𝑥 = max
𝑡∈𝑑𝑜𝑚 𝑓∗ 𝑥𝑡 − 𝑓∗ 𝑡 𝐷𝑓 𝑃||𝑄 = න
𝑥
𝑞 𝑥 𝑓 𝑝 𝑥
𝑞 𝑥 𝑑𝑥
= න
𝑥
𝑞 𝑥 max
𝑡∈𝑑𝑜𝑚 𝑓∗
𝑝 𝑥
𝑞 𝑥 𝑡 − 𝑓∗ 𝑡 𝑑𝑥 𝑝 𝑥
𝑞 𝑥
𝑝 𝑥 𝑞 𝑥
D is a function whose input is x, and output is t
Connection with GAN
= න
𝑥
𝑝 𝑥 𝐷 𝑥 𝑑𝑥 − න
𝑥
𝑞 𝑥 𝑓∗ 𝐷 𝑥 𝑑𝑥 𝐷𝑓 𝑃||𝑄 ≥ න
𝑥
𝑞 𝑥 𝑝 𝑥
𝑞 𝑥 𝐷 𝑥 − 𝑓∗ 𝐷 𝑥 𝑑𝑥
≈ max
𝐷 න
𝑥
𝑝 𝑥 𝐷 𝑥 𝑑𝑥 − න
𝑥
𝑞 𝑥 𝑓∗ 𝐷 𝑥 𝑑𝑥
Connection with GAN
𝐷𝑓 𝑃||𝑄 ≈ max
D න
𝑥
𝑝 𝑥 𝐷 𝑥 𝑑𝑥 − න
𝑥
𝑞 𝑥 𝑓∗ 𝐷 𝑥 𝑑𝑥
= max
D 𝐸𝑥~𝑃 𝐷 𝑥 − 𝐸𝑥~𝑄 𝑓∗ 𝐷 𝑥
Samples from P Samples from Q 𝐷𝑓 𝑃𝑑𝑎𝑡𝑎||𝑃𝐺 = max
D 𝐸𝑥~𝑃𝑑𝑎𝑡𝑎 𝐷 𝑥 − 𝐸𝑥~𝑃𝐺 𝑓∗ 𝐷 𝑥
𝐺
∗= 𝑎𝑟𝑔 min
𝐺
𝐷
𝑓𝑃
𝑑𝑎𝑡𝑎||𝑃
𝐺familiar? ☺
= 𝑎𝑟𝑔 min
𝐺
max
𝐷
𝐸
𝑥~𝑃𝑑𝑎𝑡𝑎𝐷 𝑥 − 𝐸
𝑥~𝑃𝐺𝑓
∗𝐷 𝑥
= 𝑎𝑟𝑔 min
𝐺
max
𝐷
𝑉 𝐺, 𝐷
Original GAN has
different V(G,D)
Using the f-divergence you like ☺
𝐷𝑓 𝑃𝑑𝑎𝑡𝑎||𝑃𝐺 = max
D 𝐸𝑥~𝑃𝑑𝑎𝑡𝑎 𝐷 𝑥 − 𝐸𝑥~𝑃𝐺 𝑓∗ 𝐷 𝑥
https://arxiv.org/pdf/1606.00709.pdf
Experimental Results
• Approximate a mixture of Gaussians by single
mixture
WGAN
Martin Arjovsky, Soumith Chintala, Léon
Bottou, Wasserstein GAN, arXiv prepring, 2017
Earth Mover’s Distance
• Considering one distribution P as a pile of earth, and another distribution Q as the target
• The average distance the earth mover has to move the earth.
𝑃 𝑄
d
𝑊 𝑃, 𝑄 = 𝑑
Earth Mover’s Distance
Source of image: https://vincentherrmann.github.io/blog/wasserstein/
𝑃
𝑄
Using the “moving plan” with the smallest average distance to define the earth mover’s distance.
There many possible “moving plans”.
Smaller distance?
Larger
distance?
Earth Mover’s Distance
Source of image: https://vincentherrmann.github.io/blog/wasserstein/
𝑃
𝑄
Using the “moving plan” with the smallest average distance to define the earth mover’s distance.
There many possible “moving plans”.
Best “moving plans”
of this example
A “moving plan” is a matrix
The value of the element is the amount of earth from one
position to another.
moving plan 𝛾 All possible plan Π
𝐵 𝛾 =
𝑥𝑝,𝑥𝑞
𝛾 𝑥𝑝, 𝑥𝑞 𝑥𝑝 − 𝑥𝑞 Average distance of a plan 𝛾:
Earth Mover’s Distance:
𝑊 𝑃, 𝑄 = min
𝛾∈Π 𝐵 𝛾
The best plan
𝑃
𝑄
𝑥𝑝
𝑥𝑞
𝑃𝑑𝑎𝑡𝑎
𝑃𝐺0 𝑃𝐺50 𝑃𝑑𝑎𝑡𝑎
𝐽𝑆 𝑃𝐺0, 𝑃𝑑𝑎𝑡𝑎
= 𝑙𝑜𝑔2
𝑃𝑑𝑎𝑡𝑎 𝑃𝐺100
…… ……
𝑑0 𝑑50
𝐽𝑆 𝑃𝐺50, 𝑃𝑑𝑎𝑡𝑎
= 𝑙𝑜𝑔2
𝐽𝑆 𝑃𝐺100, 𝑃𝑑𝑎𝑡𝑎
= 0 𝑊 𝑃𝐺0, 𝑃𝑑𝑎𝑡𝑎
= 𝑑0
𝑊 𝑃𝐺50, 𝑃𝑑𝑎𝑡𝑎
= 𝑑50
𝑊 𝑃𝐺100, 𝑃𝑑𝑎𝑡𝑎
= 0
Why Earth Mover’s Distance?
𝐷
𝑓𝑃
𝑑𝑎𝑡𝑎||𝑃
𝐺𝑊 𝑃
𝑑𝑎𝑡𝑎, 𝑃
𝐺Back to the GAN framework
𝐷
𝑓𝑃
𝑑𝑎𝑡𝑎||𝑃
𝐺= max
𝐷
𝐸
𝑥~𝑃𝑑𝑎𝑡𝑎𝐷 𝑥 − 𝐸
𝑥~𝑃𝐺𝑓
∗𝐷 𝑥 𝑊 𝑃
𝑑𝑎𝑡𝑎, 𝑃
𝐺= max
𝐷∈1−𝐿𝑖𝑝𝑠𝑐ℎ𝑖𝑡𝑧
𝐸
𝑥~𝑃𝑑𝑎𝑡𝑎𝐷 𝑥 − 𝐸
𝑥~𝑃𝐺𝐷 𝑥
𝑓 𝑥1 − 𝑓 𝑥2 ≤ 𝐾 𝑥1 − 𝑥2
Lipschitz Function
K=1 for "1 − 𝐿𝑖𝑝𝑠𝑐ℎ𝑖𝑡𝑧"
𝑊 𝑃
𝑑𝑎𝑡𝑎, 𝑃
𝐺Output change
Input
change 1−Lipschitz?
1−Lipschitz?
Do not change fast
Back to the GAN framework
𝑊 𝑃
𝑑𝑎𝑡𝑎, 𝑃
𝐺= max
𝐷∈1−𝐿𝑖𝑝𝑠𝑐ℎ𝑖𝑡𝑧
𝐸
𝑥~𝑃𝑑𝑎𝑡𝑎𝐷 𝑥 − 𝐸
𝑥~𝑃𝐺𝐷 𝑥
d
𝐷 𝑥1 − 𝐷 𝑥2 ≤ 𝑥1 − 𝑥2 𝐷 𝑥1
= +∞
𝑥1 𝑥2
𝑃𝑑𝑎𝑡𝑎 𝑃𝐺
𝐷 𝑥2
= −∞
𝑘 𝑘 + 𝑑
𝑘 + 𝑑 𝑘
WGAN will provide gradient to push PG towards Pdata
Blue: D(x) for original GAN Green: D(x) for WGAN
𝑊 𝑃𝑑𝑎𝑡𝑎 , 𝑃𝐺 = 𝑑
Back to the GAN framework
𝑊 𝑃
𝑑𝑎𝑡𝑎, 𝑃
𝐺= max
𝐷∈1−𝐿𝑖𝑝𝑠𝑐ℎ𝑖𝑡𝑧
𝐸
𝑥~𝑃𝑑𝑎𝑡𝑎𝐷 𝑥 − 𝐸
𝑥~𝑃𝐺𝐷 𝑥
How to use gradient descent to optimize?
Weight clipping:
Force the weights w between c and -c After parameter update,
if w > c, then w=c; if w<-c, then w=-c 𝐷 𝑥1 − 𝐷 𝑥2 ≤ 𝐾 𝑥1 − 𝑥2 We only ensure that
For some K K
K
No clipping clipping
Do not truly find function D maximizing the function
• In each training iteration:
• Sample m examples 𝑥1, 𝑥2, … , 𝑥𝑚 from data distribution 𝑃𝑑𝑎𝑡𝑎 𝑥
• Sample m noise samples 𝑧1, 𝑧2, … , 𝑧𝑚 from the prior 𝑃𝑝𝑟𝑖𝑜𝑟 𝑧
• Obtaining generated data 𝑥1, 𝑥2, … , 𝑥𝑚 , 𝑥𝑖 = 𝐺 𝑧𝑖
• Update discriminator parameters 𝜃𝑑 to maximize
• ෨𝑉 = 1
𝑚 σ𝑖=1𝑚 𝑙𝑜𝑔𝐷 𝑥𝑖 + 1
𝑚 σ𝑖=1𝑚 𝑙𝑜𝑔 1 − 𝐷 𝑥𝑖
• 𝜃𝑑 ← 𝜃𝑑 + 𝜂𝛻 ෨𝑉 𝜃𝑑
• Sample another m noise samples 𝑧1, 𝑧2, … , 𝑧𝑚 from the prior 𝑃𝑝𝑟𝑖𝑜𝑟 𝑧
• Update generator parameters 𝜃𝑔 to minimize
• ෨𝑉 = 1
𝑚 σ𝑖=1𝑚 𝑙𝑜𝑔𝐷 𝑥𝑖 + 1
𝑚 σ𝑖=1𝑚 𝑙𝑜𝑔 1 − 𝐷 𝐺 𝑧𝑖
• 𝜃𝑔 ← 𝜃𝑔 − 𝜂𝛻 ෨𝑉 𝜃𝑔
Algorithm of Original GAN
Repeat k times Learning
D
Learning G
Only Once
𝐷 𝑥𝑖 − 𝐷 𝑥𝑖
No sigmoid for the output of D
WGAN
Weight clipping
𝐷 𝐺 𝑧𝑖
−
https://arxiv.org/abs/1701.07875
Vertical
Improved WGAN
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent
Dumoulin, Aaron Courville, “Improved Training of Wasserstein GANs”, arXiv prepring, 2017
𝑊 𝑃
𝑑𝑎𝑡𝑎, 𝑃
𝐺= max
𝐷∈1−𝐿𝑖𝑝𝑠𝑐ℎ𝑖𝑡𝑧
𝐸
𝑥~𝑃𝑑𝑎𝑡𝑎𝐷 𝑥 − 𝐸
𝑥~𝑃𝐺𝐷 𝑥
≈ max
𝐷
{𝐸
𝑥~𝑃𝑑𝑎𝑡𝑎𝐷 𝑥 − 𝐸
𝑥~𝑃𝐺𝐷 𝑥
−𝜆
𝑥𝑚𝑎𝑥 0, 𝛻
𝑥𝐷 𝑥 − 1 𝑑𝑥}
−𝜆𝐸
𝑥~𝑃𝑝𝑒𝑛𝑎𝑙𝑡𝑦𝑚𝑎𝑥 0, 𝛻
𝑥𝐷 𝑥 − 1 }
A differentiable function is 1-Lipschitz if and only if it has gradients with norm less than or equal to 1 everywhere.
𝛻
𝑥𝐷 𝑥 ≤ 1 for all x
Improved WGAN
𝑊 𝑃
𝑑𝑎𝑡𝑎, 𝑃
𝐺𝐷 ∈ 1 − 𝐿𝑖𝑝𝑠𝑐ℎ𝑖𝑡𝑧
Prefer 𝛻
𝑥𝐷 𝑥 ≤ 1 for all x
Prefer 𝛻
𝑥𝐷 𝑥 ≤ 1 for x sampling from 𝑥~𝑃
𝑝𝑒𝑛𝑎𝑙𝑡𝑦Improved WGAN
𝑃
𝑑𝑎𝑡𝑎𝑃
𝐺Only give gradient constraint to the region between 𝑃𝑑𝑎𝑡𝑎 and 𝑃𝐺 because they influence how 𝑃𝐺 moves to 𝑃𝑑𝑎𝑡𝑎
−𝜆𝐸
𝑥~𝑃𝑝𝑒𝑛𝑎𝑙𝑡𝑦𝑚𝑎𝑥 0, 𝛻
𝑥𝐷 𝑥 − 1 }
≈ max
𝐷
{𝐸
𝑥~𝑃𝑑𝑎𝑡𝑎𝐷 𝑥 − 𝐸
𝑥~𝑃𝐺𝐷 𝑥 𝑊 𝑃
𝑑𝑎𝑡𝑎, 𝑃
𝐺𝑃𝑝𝑒𝑛𝑎𝑙𝑡𝑦
“Given that enforcing the Lipschitz constraint everywhere is intractable, enforcing it only along these straight lines seems sufficient and experimentally results in good performance.”
“One may wonder why we penalize the norm of the gradient for differing from 1, instead of just penalizing large gradients. The
reason is that the optimal critic … actually has gradients with norm 1 almost everywhere under Pr and Pg”
“Simply penalizing overly large gradients also works in theory, but experimentally we found that this approach converged faster and to better optima.”
Improved WGAN
(check the proof in the appendix)
𝑃
𝑑𝑎𝑡𝑎𝑃
𝐺−𝜆𝐸
𝑥~𝑃𝑝𝑒𝑛𝑎𝑙𝑡𝑦𝑚𝑎𝑥 0, 𝛻
𝑥𝐷 𝑥 − 1 }
≈ max
𝐷
{𝐸
𝑥~𝑃𝑑𝑎𝑡𝑎𝐷 𝑥 − 𝐸
𝑥~𝑃𝐺𝐷 𝑥 𝑊 𝑃
𝑑𝑎𝑡𝑎, 𝑃
𝐺𝑃𝑝𝑒𝑛𝑎𝑙𝑡𝑦
𝛻
𝑥𝐷 𝑥 − 1
2𝐷 𝑥 Largest gradient in 𝐷 𝑥 this region (=1)
Improved WGAN
https://arxiv.org/abs/1704.00028
DCGAN
G: CNN, D: CNN
G: CNN (no normalization), D: CNN (no normalization)
LSGAN Original
WGAN
Improved WGAN
G: CNN (tanh), D: CNN(tanh)
DCGAN LSGAN Original WGAN
Improved G: MLP, D: CNN WGAN
G: CNN (bad structure), D: CNN
G: 101 layer, D: 101 layer
Energy-based GAN
Ref: Junbo Zhao, Michael Mathieu, Yann LeCun, Energy-based Generative Adversarial Network, ICRL 2017
Energy-based GAN (EBGAN)
• Using an autoencoder as discriminator D
Discriminator An image
is good.
It can be reconstructed by autoencoder.
=
0 for best images Generator is
the same.
-
0.1EN DE
Autoencoder
X -1 -0.1
EBGAN
real
gen gen
Hard to reconstruct, easy to destroy
m
0 is for the best.
Do not have to be very negative
Auto-encoder based discriminator only give limited region large value.
Stack and
Progressive GAN
Patch GAN
D
score
D D
score score
https://arxiv.org/pdf/1611.07004.pdf
Stack GAN
Han Zhang, Tao Xu, Hongsheng
Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris Metaxas, “StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks”, ICCV, 2017
Stack GAN ++
• Tree-like structure
Han Zhang, Tao Xu, Hongsheng
Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, Dimitris Metaxas, “StackGAN++:
Realistic Image Synthesis with Stacked
Generative Adversarial Networks”, arXiv 2017
c.f. ACGAN
https://arxiv.org/pdf/1610.09585.pdf
Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen, ” Progressive Growing of GANs for Improved Quality, Stability, and Variation”, arXiv 2017
Progressive Growing of GAN
Ensemble
Observation
陳柏文同學提供 實驗結果
Yaxing Wang, Lichao Zhang, Joost van de Weijer, “Ensembles of
Generative Adversarial Networks”, NIPS workshop, 2016
Ensemble
Generator 1
Generator 2
圖片來源 柯達方 同學
This figure is from the original paper: https://arxiv.org/abs/1612.04021
Generative Adversarial Parallelization
Code: https://github.com/poolio/unrolled_gan/blob/master/Unrolled%20GAN%20demo.ipynb
Unroll GAN – Experimental Results
Style Transfer
Cycle GAN Dual GAN
Disco GAN
Coupled GAN (CoGAN)
Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial Networks”, NIPS, 2016
UNIT: Unsupervised Image-to- image Translation
https://arxiv.org/pdf/1703.00848.pdf https://github.com/mingyuliutw/unit
DTN: Domain Transfer Network
https://arxiv.org/pdf/1611.02200.pdf
XGAN
https://arxiv.org/pdf/1711.05139.pdfStarGAN
https://arxiv.org/pdf/1711.09020.pdf
StarGAN
StarGAN
StarGAN
Feature
Anders Boesen, Lindbo Larsen, Søren Kaae
Sønderby, Hugo Larochelle, Ole Winther, “Autoencoding beyond pixels using a learned similarity metric”, ICML. 2016
InfoGAN
Regular GAN
Modifying a specific dimension, no clear meaning
What we expect Actually …
(The colors represents the characteristics.)
What is InfoGAN?
Discrimi nator
Classifier
scalar Generator
z =
z’
c x
c Predict the code c
that generates x
“Auto- encoder”
Parameter sharing (only the last layer is different) encoder
decoder
What is InfoGAN?
Classifier Generator
z =
z’
c x
c Predict the code c
that generates x
“Auto- encoder”
encoder decoder
c must have clear influence on x
The classifier can
recover c from x.
https://arxiv.org/abs/1606.03657
VAE-GAN
Discrimi nator Generator
(Decoder)
Encoder z x scalar
x
as close as possible
VAE
GAN
➢ Minimize
reconstruction error
➢ z close to normal
➢ Minimize
reconstruction error
➢ Cheat discriminator
➢ Discriminate real, generated and
reconstructed images
Discriminator provides the similarity measure
• Initialize En, De, Dis
• In each iteration:
• Sample M images 𝑥1, 𝑥2, ⋯ , 𝑥𝑀 from database
• Generate M codes ǁ𝑧1, ǁ𝑧2, ⋯ , ǁ𝑧𝑀 from encoder
• ǁ𝑧𝑖 = 𝐸𝑛 𝑥𝑖
• Generate M images 𝑥1, 𝑥2, ⋯ , 𝑥𝑀 from decoder
• 𝑥𝑖 = 𝐸𝑛 ǁ𝑧𝑖
• Sample M codes 𝑧1, 𝑧2, ⋯ , 𝑧𝑀 from prior P(z)
• Generate M images ො𝑥1, ො𝑥2, ⋯ , ො𝑥𝑀 from decoder
• ො𝑥𝑖 = 𝐸𝑛 𝑧𝑖
• Update En to decrease 𝑥𝑖 − 𝑥𝑖 , decrease KL(P( ǁ𝑧𝑖|xi)||P(z))
• Update De to decrease 𝑥𝑖 − 𝑥𝑖 , increase 𝐷𝑖𝑠 𝑥𝑖 and 𝐷𝑖𝑠 ො𝑥𝑖
• Update Dis to increase 𝐷𝑖𝑠 𝑥𝑖 , decrease 𝐷𝑖𝑠 𝑥𝑖 and 𝐷𝑖𝑠 ො𝑥𝑖
Algorithm
Another kind of discriminator:
Discriminator
x
real gen recon
BiGAN
Encoder Decoder Discriminator
Image x code z
Image x code z
Image x code z from encoder
or decoder?
(real) (generated) (from prior distribution)
Algorithm
• Initialize encoder En, decoder De, discriminator Dis
• In each iteration:
• Sample M images 𝑥1, 𝑥2, ⋯ , 𝑥𝑀 from database
• Generate M codes ǁ𝑧1, ǁ𝑧2, ⋯ , ǁ𝑧𝑀 from encoder
• ǁ𝑧𝑖 = 𝐸𝑛 𝑥𝑖
• Sample M codes 𝑧1, 𝑧2, ⋯ , 𝑧𝑀 from prior P(z)
• Generate M codes 𝑥1, 𝑥2, ⋯ , 𝑥𝑀 from decoder
• 𝑥𝑖 = 𝐷𝑒 𝑧𝑖
• Update Dis to increase 𝐷𝑖𝑠 𝑥𝑖, ǁ𝑧𝑖 , decrease 𝐷𝑖𝑠 𝑥𝑖, 𝑧𝑖
• Update En and De to decrease 𝐷𝑖𝑠 𝑥𝑖, ǁ𝑧𝑖 , increase 𝐷𝑖𝑠 𝑥𝑖, 𝑧𝑖
Encoder Decoder Discriminator
Image x code z
Image x code z
Image x code z from encoder
or decoder?
(real) (generated) (from prior distribution)
𝑃 𝑥, 𝑧 𝑄 𝑥, 𝑧 Evaluate the
difference between P and Q To make P and Q the same
Optimal encoder and decoder:
En(x’) = z’ De(z’) = x’
En(x’’) = z’’
De(z’’) = x’’
For all x’
For all z’’
BiGAN
En(x’) = z’ De(z’) = x’
En(x’’) = z’’
De(z’’) = x’’
For all x’
For all z’’
How about?
Encoder Decoder
x z
𝑥
Encoder
Decoder x
z ǁ𝑧
En, De
En, De Optimal encoder
and decoder:
Triple GAN
Chongxuan Li, Kun Xu, Jun Zhu, Bo Zhang, “Triple Generative Adversarial Nets”, arXiv 2017
Domain-adversarial training
• Training and testing data are in different domains
Training data:
Testing data:
Generator
Generator
The same distribution
feature
feature
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, Domain-Adversarial Training of Neural Networks, JMLR, 2016
Domain-adversarial training
Not only cheat the domain classifier, but satisfying label classifier at the same time
This is a big network, but different parts have different goals.
Maximize label classification accuracy
Maximize domain classification accuracy Maximize label classification accuracy +
minimize domain classification accuracy feature extractor
Domain classifier Label predictor
Feature Disentangle
Emily Denton, Vighnesh Birodkar,
“Unsupervised Learning of Disentangled Representations from Video”, NIPS, 2017
Experimental Results
https://arxiv.org/pdf/1705.109 15.pdf
Evaluation
Ref: Lucas Theis, Aäron van den Oord, Matthias Bethge, “A note on the evaluation of generative models”, arXiv preprint, 2015
Likelihood
: real data (not observed during training): generated data
𝑥𝑖
Log Likelihood:
Generator P
G𝐿 = 1
𝑁
𝑖
𝑙𝑜𝑔𝑃
𝐺𝑥
𝑖We cannot compute 𝑃
𝐺𝑥
𝑖. We can only sample from 𝑃
𝐺.
Prior
Distribution
Likelihood
- Kernel Density Estimation
• Estimate the distribution of P
G(x) from sampling
Generator P
GNow we have an approximation of 𝑃
𝐺, so we can compute 𝑃
𝐺𝑥
𝑖for each real data 𝑥
𝑖Then we can compute the likelihood.
Each sample is the mean of a
Gaussian with the same covariance.
Likelihood v.s. Quality
• Low likelihood, high quality?
• High likelihood, low quality?
Considering a model generating good images (small variance)
Generator P
GGenerated data Data for evaluation 𝑥𝑖 𝑃𝐺 𝑥𝑖 = 0
G1
𝐿 = 1
𝑁
𝑖
𝑙𝑜𝑔𝑃𝐺 𝑥𝑖
G2
X 0.99
X 0.01
= −𝑙𝑜𝑔100 + 1
𝑁
𝑖
𝑙𝑜𝑔𝑃𝐺 𝑥𝑖 100 4.6
Evaluate by Other Networks
Well-trained
𝑥 CNN 𝑃 𝑦|𝑥
Lower entropy means higher visual quality
𝑥1 CNN 𝑃 𝑦1|𝑥1
Higher entropy means higher variety
𝑥2 CNN 𝑃 𝑦2|𝑥2
𝑥3 CNN 𝑃 𝑦3|𝑥3
……
1
𝑁
𝑛
𝑃 𝑦𝑛|𝑥𝑛
Evaluate by Other Networks - Inception Score
=
𝑥
𝑦
𝑃 𝑦|𝑥 𝑙𝑜𝑔𝑃 𝑦|𝑥
−
𝑦
𝑃 𝑦 𝑙𝑜𝑔𝑃 𝑦
Negative entropy of P(y|x) Entropy of P(y)
https://arxiv.org/pdf/1606.03498.pdf
Inception Score
Mario Lucic, Karol Kurach, Marcin
Michalski, Sylvain Gelly, Olivier Bousquet, “Are
GANs Created Equal? A Large-Scale Study”, arXiv, 2017 Smaller is better
FIT:
https://arxiv.org/pdf/1706.08500.pdf
Mode Collapse
Missing Mode ?
?
Mode collapse is
easy to detect.
We don’t want memory GAN.
• Using k-nearest neighbor to check whether the generator generates new objects
https://arxiv.org/pdf/1511.01844.pdf
Concluding Remarks
from A to Z
Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, Shakir Mohamed, “Variational Approaches for Auto-Encoding Generative Adversarial Networks”, arXiv, 2017