add realism to the synthetic images preserves the annotation information

.GCTPKPIHTQO5KOWNCVGF

7PUWRGTXKUGF+OCIGU

VJTQWIJ#FXGTUCTKCN6TCKPKPI^



$TKCP*WCPI^

Ruslan Salakhutdinov

@NIPS2016

Outline

– Paper’s contents – Intro

– S+U learning with SimGAN – Experiment

– Conclusion – Discussion

Intro

Intro

– Synthetic images are useful but not good enough – Improve the simulator

– Improve the realism of synthetic images using U

– Labeled synthetic images (generated by computer) – Unlabeled real data

– Train a network to do this

– The network using adversarial loss

SimGAN Architecture

– The algorithm

– The loss functions – Training tricks

– Local adversarial loss

– Update D using the history of refined images

S+U learning with SimGAN

Some notations

Algorithm

___________________________________

___________________________________

____________________

Algorithm

Loss functions - Discriminator

_________ _____________

probability of

being synthetic probability of being real

Cross Entropy

D’s Target: 0 for every y, 1 for every x̃

Loss functions - Refiner network

_______________ ________________

add realism to the synthetic images

preserves the annotation information

D’s Target:

0 for every y 1 for every x̃

Loss functions - Refiner network

need to fake D __________________

_______________________

L1 normalization

_______________ ________________

add realism to the

synthetic images preserves the annotation information

__________

D’s Target:

0 for every y 1 for every x̃

To preserve the annotations

Local adversarial loss

Summing all local adversarial loss as the final D loss

Update D

using a History of refined images

sample b/2 from current refined network and sample b/2 from buffer

then randomly replace b/2 images in buffer and the size of buffer fixed at B

with newly generated refined images B: the size of the buffer

b: the size of the mini-batch

Experiment

– Gaze estimation

– Hand pose estimation

– Analysis of modifications of adversarial training

UnityEyes: Synthetic images MPIIGaze: Real Images

Gaze estimation - dataset (UnityEyes)

Gaze estimation - dataset (MPIIGaze)

UnityEyes: Synthetic images MPIIGaze: Real Images

UnityEyes: Synthetic images MPIIGaze: Real Images

Gaze estimation - Qualitative result

Gaze estimation - visual Turing test

choose correct label 517 times out of 1000 trails, not outperforms than random (p = 0.148)

____ ____

____ ^____

Synthetic images: choose correctly 162 times out of 200 trials (p < 10^(-8))

Gaze estimation - Quantitative result

Train on Refined Images and test on MPIIGaze

Gaze estimation - Quantitative result

Hand pose estimation - dataset

NYU Hand Pose Dataset:

contains real and synthetic

Hand pose estimation - dataset

NYU Hand Pose Dataset:

contains real and synthetic collected by Kinect

Hand pose estimation - Qualitative result

NYU Hand Pose Dataset:

contains real and synthetic

Kinect 拍出來來的照片有深度差距 因此 Real image 會有⽑毛邊

Hand pose estimation - Quantitative result

Analysis of modifications of adversarial training

- local adversarial loss

Analysis of modifications of adversarial training - History of refined images

減少偽造感

Let’s look back to intro

– Synthetic images not good enough – Improve the simulator

– Improve the realism of synthetic images using U – Simulate

– Train a refine network to do this

– The network using adversarial loss – Generative Adversarial Network

Conclusion

– Refine a Simulator's output with Unlabeled data – S+U can add realism into synthetic images

– S+U can also preserve annotations of synthetic images – Refined images really help improving the testing result – Generate > 1 images for each synthetic image

Discussion

– A signal from Apple

– Some future applications of SimGAN

將SimGANs應⽤用於基於無監督學習的⾃自主駕駛

想像你是Comma.AI的⼀一員，有⼤大量量的由Dash收集的真實的未標記的駕駛數據。雖然你當前的標記數據的⽅方法很 棒，但你只有少量量的有標記數據。使⽤用SimGAN，你可以訓練⼀一個改進神經網絡來來改進俠盜⾶飛⾞車車的數據（開發者 使⽤用俠盜⾶飛⾞車車遊戲模擬真實⾞車車輛⾏行行駛），使數據看起來來像是來來⾃自你的真實數據集，同時保留留標註。現在，你可以 在這個幾乎無限精細標記的數據集上訓練你的⽣生產模型，並使⽤用少量量的真實標記數據集作為驗證。

我沒有參參加任何⾃自動駕駛的課程，但我知道他們使⽤用俠盜⾶飛⾞車車和模擬環境來來訓練他們的模型。有了了這樣的技術，

他們的軟件可以更更接近現實世界。SimGAN在現實世界中似乎有許多可能的應⽤用，⾃自主駕駛只是我選擇使⽤用的⼀一 個有趣的例例⼦子。

Reference

– Paper: https://arxiv.org/pdf/1612.07828.pdf

– SimGAN Implemented in tensorflow: ^https://

github.com/carpedm20/simulated-unsupervised- tensorflow

– Comments from experts: http://mp.weixin.qq.com/

s/2Ltb249M71lMWrTbhnYPEQ

– Translation of the paper: http://tech.163.com/16/

1227/07/C99CBP7P00097U80.html

6JCPMUHQTNKUVGPKPI

#P[3WGUVKQPU!