### Slide credit from Hung-Yi Lee

## Review

### Generative Model

### 2

### Discriminative v.s. Generative Models

### Discriminative

### ◦ learns a function that maps *the input data (x) to some * *desired output class label (y)*

### • directly learn the conditional *distribution P(y|x)*

### Generative

### ◦ tries to learn the joint

### probability of the input data and labels simultaneously, *i.e. P(x,y)*

### • *can be converted to P(y|x) for * classification via Bayes rule

### 3

### Advantage: generative models have the potential to understand and explain

### the underlying structure of the input data even when there are no labels

### Generator

### Decoder from autoencoder as generator

### 4

### 𝑥

### Input layer

### 𝑊

### 𝑥′

### 𝑊′

### output layer hidden layer

### code 𝑎

### The generator is to generate the data from the code

### encode decode

## Generative Adversarial Network (GAN)

### Representati on Learning

### 5

*“There are many interesting recent development in deep learning…The most important one, in my opinion, *

*is adversarial training (also called GAN for Generative Adversarial Networks). This, and the variations that *

**are now being proposed is the most interesting idea in the last 10 years in ML, in my opinion.” – Yann LeCun**

**are now being proposed is the most interesting idea in the last 10 years in ML, in my opinion.” – Yann LeCun**

### Generative Adversarial Networks (GAN)

### Two competing neural networks: generator & discriminator

*Goodfellow, et al., “Generative adversarial networks,” in NIPS, 2014.* 6

### Training two networks jointly the generator knows how to adapt its parameters in order to produce output data that can fool the discriminator

### http://blog.aylien.com/introduction-generative-adversarial-networks-code-tensorflow/

### forger trying to produce some counterfeit material

### the police trying to detect

### the forged items

### Generative Adversarial Networks (GAN)

### 7

*the probability that x came from * the data rather than generator

*Goodfellow, et al., “Generative adversarial networks,” in NIPS, 2014.*

### GAN Objective

### 8

*D(x): the probability that x came from * the data rather than generator

*Goodfellow, et al., “Generative adversarial networks,” in NIPS, 2014.*

*D*

*G*

### GAN Training Algorithm

### 9

### Discriminator

### Generator

*Goodfellow, et al., “Generative adversarial networks,” in NIPS, 2014.*

### GAN Nash Equilibrium

### Global optimality

### ◦ Discriminator

### ◦ Generator

*Goodfellow, et al., “Generative adversarial networks,” in NIPS, 2014.* 10

*D* *G*

### Two competing networks are trained towards global optimality

### s.t.

### GAN-Generated Bedrooms

### 11

Radford et al., “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks,” arXiv:1511.06434.

### Applications of Generative Models

### Semi-supervised learning

### ◦ few training samples with annotations

### ◦ generate more training data using GAN

### 12

## Conditional GAN

### Generator Conditioned on Labels

### 13

### Conditional GAN

### 14

### GAN

### Conditional GAN

Mirza and Osindero, “Conditional Generative Adversarial Nets,” arXiv:1411.1784.

### Generated Images Conditioned on Label

### 15

Mirza and Osindero, “Conditional Generative Adversarial Nets,” arXiv:1411.1784.

## Adversarially Learned Inference (ALI)

### Inference with Latent Variables

### 16

### Adversarially Learned Inference (ALI) / Bidirectional GAN (BiGAN)

### Idea: incorporate latent variables for inference

*Inference: given x, what z is likely to have produced it*

### 17

### Inference is important but ignored by GAN

Dumoulin et al., “Adversarially Learned Inference,” arXiv:1606.00704.

Donahue et al., “Adversarial Feature Learning,” arXiv:1605.09782.

### ALI / BiGAN

Dumoulin et al., “Adversarially Learned Inference,” arXiv:1606.00704.

### 18

Donahue et al., “Adversarial Feature Learning,” arXiv:1605.09782.

### Conditional ALI

*Conditional generation: providing a conditioning variable y* for generator, encoder, discriminator

### 19

### Latent variables can represent attributes

Dumoulin et al., “Adversarially Learned Inference,” arXiv:1606.00704.

## Improvement of Training GAN

### Stableness and Robustness

### 20

### Feature Matching (Generator Objective)

### Idea: match the expected values of the features on an intermediate layer of discriminator

### Generator’s objective:

### 21

**f (x)**

**f (x)**

Salimans et al., “Improved Techniques for Training GANs,” arXiv:1606.03498. (Credit from Lu’s slides)

### Generators utilize the features used by discriminators as objective

### Unrolled GAN (Generator Objective)

### Idea: allow generator to consider discriminator’s capability Iterative optimization procedure

**Surrogate objective for generator**

Metz et al., “Unrolled Generative Adversarial Networks,” arXiv:1611.02163.

### 22

### Unrolled GAN (Generator Objective)

### Idea: allow generator to consider discriminator’s capability Surrogate objective for generator

Metz et al., “Unrolled Generative Adversarial Networks,” arXiv:1611.02163.

### 23

### Generators can move towards better directions based on what discriminators tell

### Minibatch Discrimination (Discriminator Objective)

### Idea: allow the discriminator to see multiple data examples in combination to avoid collapsing to a single mode

Salimans et al., “Improved Techniques for Training GANs,” arXiv:1606.03498.

### 24

### Considering a batch in combination results in better robustness and diversity

### Generative Adversarial Parallelization (GAP)

### (Discriminator Objective)

### Idea: train multiple GANs and allow different discriminators to discriminate difference generators

Im et al., “Generative Adversarial Parallelization,” arXiv:1612.04021.

### 25

### Generative Adversarial Parallelization (GAP)

### (Discriminator Objective)

Im et al., “Generative Adversarial Parallelization,” arXiv:1612.04021.