Slide credit from Hung-Yi Lee
Review
Generative Model
2
Discriminative v.s. Generative Models
Discriminative
◦ learns a function that maps the input data (x) to some desired output class label (y)
• directly learn the conditional distribution P(y|x)
Generative
◦ tries to learn the joint
probability of the input data and labels simultaneously, i.e. P(x,y)
• can be converted to P(y|x) for classification via Bayes rule
3
Advantage: generative models have the potential to understand and explain
the underlying structure of the input data even when there are no labels
Generator
Decoder from autoencoder as generator
4
𝑥
Input layer
𝑊
𝑥′
𝑊′
output layer hidden layer
code 𝑎
The generator is to generate the data from the code
encode decode
Generative Adversarial Network (GAN)
Representati on Learning
5
“There are many interesting recent development in deep learning…The most important one, in my opinion,
is adversarial training (also called GAN for Generative Adversarial Networks). This, and the variations that
are now being proposed is the most interesting idea in the last 10 years in ML, in my opinion.” – Yann LeCun
Generative Adversarial Networks (GAN)
Two competing neural networks: generator & discriminator
Goodfellow, et al., “Generative adversarial networks,” in NIPS, 2014. 6
Training two networks jointly the generator knows how to adapt its parameters in order to produce output data that can fool the discriminator
http://blog.aylien.com/introduction-generative-adversarial-networks-code-tensorflow/
forger trying to produce some counterfeit material
the police trying to detect
the forged items
Generative Adversarial Networks (GAN)
7
the probability that x came from the data rather than generator
Goodfellow, et al., “Generative adversarial networks,” in NIPS, 2014.
GAN Objective
8
D(x): the probability that x came from the data rather than generator
Goodfellow, et al., “Generative adversarial networks,” in NIPS, 2014.
D
G
GAN Training Algorithm
9
Discriminator
Generator
Goodfellow, et al., “Generative adversarial networks,” in NIPS, 2014.
GAN Nash Equilibrium
Global optimality
◦ Discriminator
◦ Generator
Goodfellow, et al., “Generative adversarial networks,” in NIPS, 2014. 10
D G
Two competing networks are trained towards global optimality
s.t.
GAN-Generated Bedrooms
11
Radford et al., “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks,” arXiv:1511.06434.
Applications of Generative Models
Semi-supervised learning
◦ few training samples with annotations
◦ generate more training data using GAN
12
Conditional GAN
Generator Conditioned on Labels
13
Conditional GAN
14
GAN
Conditional GAN
Mirza and Osindero, “Conditional Generative Adversarial Nets,” arXiv:1411.1784.
Generated Images Conditioned on Label
15
Mirza and Osindero, “Conditional Generative Adversarial Nets,” arXiv:1411.1784.
Adversarially Learned Inference (ALI)
Inference with Latent Variables
16
Adversarially Learned Inference (ALI) / Bidirectional GAN (BiGAN)
Idea: incorporate latent variables for inference
Inference: given x, what z is likely to have produced it
17
Inference is important but ignored by GAN
Dumoulin et al., “Adversarially Learned Inference,” arXiv:1606.00704.
Donahue et al., “Adversarial Feature Learning,” arXiv:1605.09782.
ALI / BiGAN
Dumoulin et al., “Adversarially Learned Inference,” arXiv:1606.00704.
18
Donahue et al., “Adversarial Feature Learning,” arXiv:1605.09782.
Conditional ALI
Conditional generation: providing a conditioning variable y for generator, encoder, discriminator
19
Latent variables can represent attributes
Dumoulin et al., “Adversarially Learned Inference,” arXiv:1606.00704.
Improvement of Training GAN
Stableness and Robustness
20
Feature Matching (Generator Objective)
Idea: match the expected values of the features on an intermediate layer of discriminator
Generator’s objective:
21
f (x)
Salimans et al., “Improved Techniques for Training GANs,” arXiv:1606.03498. (Credit from Lu’s slides)
Generators utilize the features used by discriminators as objective
Unrolled GAN (Generator Objective)
Idea: allow generator to consider discriminator’s capability Iterative optimization procedure
Surrogate objective for generator
Metz et al., “Unrolled Generative Adversarial Networks,” arXiv:1611.02163.
22
Unrolled GAN (Generator Objective)
Idea: allow generator to consider discriminator’s capability Surrogate objective for generator
Metz et al., “Unrolled Generative Adversarial Networks,” arXiv:1611.02163.
23
Generators can move towards better directions based on what discriminators tell
Minibatch Discrimination (Discriminator Objective)
Idea: allow the discriminator to see multiple data examples in combination to avoid collapsing to a single mode
Salimans et al., “Improved Techniques for Training GANs,” arXiv:1606.03498.
24
Considering a batch in combination results in better robustness and diversity
Generative Adversarial Parallelization (GAP)
(Discriminator Objective)
Idea: train multiple GANs and allow different discriminators to discriminate difference generators
Im et al., “Generative Adversarial Parallelization,” arXiv:1612.04021.
25
Generative Adversarial Parallelization (GAP)
(Discriminator Objective)
Im et al., “Generative Adversarial Parallelization,” arXiv:1612.04021.
