• Texture synthesis

(1)

Textures and Inpainting

Digital Visual Effects g Yung-Yu Chuang

with slides by Alex Efros, Li-Yi Wei, Arno Schedl and Paul Debevec

(2)

Outline

• Texture synthesis

A l i b l i l i d TSVQ

• Acceleration by multi-resolution and TSVQ

• Patch-based texture synthesis

• Image analogies

(3)

Texture synthesis

(4)

Texture synthesis

input image

synthesis

input image

generated image

• Given a finite sample of some texture, the goal is to synthesize other samples from that same is to synthesize other samples from that same texture.

The sample needs to be "large enough"

– The sample needs to be large enough

(5)

The challenge

• How to capture the essence of texture?

• Need to model the whole spectrum: from

repeated to stochastic texture

(6)

Markov property

• P(f

_i

|f

_i-1

,f

_i-2

,f

_i-3

,…f

₀

)= P(f

_i

|f

_i-1

,f

_i-2

,…f

_i-n

)

ff_i Ff_i-1

Ff₀ … Ff_i-n … Ff_i-2

• P(f

_i

|f

_S-{i}

)= P(f

_i

|f

_Ni

)

f_i

N

_i

N

_i

S

(7)

Motivation from language

• [Shannon’48] proposed a way to generate English-looking text using N-grams:

English-looking text using N-grams:

– Assume a generalized Markov model

Use a large text to compute probability – Use a large text to compute probability

distributions of each letter given N-1 previous letters

lette s

• precompute or sample randomly

– Starting from a seed repeatedly sample this g p y p Markov chain to generate new letters

– One can use whole words instead of letters too.

(8)

Mark V. Shaney (Bell Labs)

• Results (using alt.singles corpus):

– "One morning I shot an elephant in my arms and kissed him.”

– "I spent an interesting evening recently with a grain of salt"

• Notice how well local structure is preserved!

– Now let’s try this for video and in 2D...

(9)

Video textures

• SIGGRAPH 2000 paper by Arno Schedl, Riachard Szeliski David Salesin and Irfan Essa

Szeliski, David Salesin and Irfan Essa.

(10)

Still photos

(11)

Video clips

(12)

Video textures

(13)

Problem statement

video clip video texture

(14)

Approach

How do we find good transitions?

(15)

Finding good transitions

Compute L

₂

distance D

_{i, j}

between all frames

frame i vs.

frame j

Si il f k d t iti

Similar frames make good transitions

(16)

Video textures

(17)

Ideally

SAMPLE

p

Infinite sample image

– Assuming Markov property, what is conditional probability Infinite sample image

generated image

g p p y, p y

distribution of p, given the neighbourhood window?

– Instead of constructing a model, let’s directly search the g , y input image for all such neighbourhoods to produce a

histogram for p

– To synthesize p, just pick one match at random

(18)

In reality

SAMPLE

finite sample image

p

image

Generated image – However since our sample image is finite an exact However, since our sample image is finite, an exact

neighbourhood match might not be present

– So we find the best match using SSD error (weighted by a So we find the best match using SSD error (weighted by a

Gaussian to emphasize local structure), and take all samples within some distance from that match

– Using Gaussian-weighted SSD is very important

(19)

Neighborhood size matters

(20)

More results

Increasing window size

(21)

More results

french canvas rafia weave

(22)

More results

wood brick wall

(23)

Failure cases

Growing garbage Verbatim copying

(24)

Summary of the basic algorithm

• Exhaustively search neighborhoods

(25)

Neighborhood

• Neighborhood size determines the quality &

cost cost

33 55 77

423 528 739 s

423 s 528 s 739 s

99 1111 4141

1020 s 1445 s 24350 s

(26)

Summary

• Advantages:

t ll i l – conceptually simple

– models a wide range of real-world textures ll d h l filli

– naturally does hole-filling

• Disadvantages:

– it’s slow

– it’s a heuristic

(27)

Acceleration by Wei & Levoy

• Multi-resolution T

• Tree-structure

(28)

Multi-resolution pyramid

Hi h l i L l i

High resolution Low resolution

(29)

Multi-resolution algorithm

(30)

Benefits

• Better image quality & faster computation (by using smaller windows)

using smaller windows)

1 level 1 level 3 levels

1 level

55 3 levels

1 level 55

1111

(31)

Results

Random Oriented

Regular Semi-regular

(32)

Failures

• Non-planar structures structures

• Global information

(33)

Acceleration

• Computation bottleneck: neighborhood search

(34)

Nearest point search

• Treat neighborhoods as high dimensional points

1 2 3 4 5 6 7 8 9 10 11 12

Neighborhood

11 12

High dimensional point/vector

1 2 3 4 5 6 7 8 9 10 11 12

g p

(35)

Tree-Structured Vector Quantization

(36)

Timing

• Time complexity : O(log N) instead of O(N)

– 2 orders of magnitude speedup for non-trivial images

images

Efros 99 Full searching TSVQ

1941 secs 503 secs 12 secs

(37)

Results

Input put Exhaustive: 360 s TSVQ: 7.5 s ^Q

(38)

Patch-based methods

non-parametric sampling

p p

I t i

B B

• Observation: neighbor pixels are highly correlated

Input image Synthesizing a block

Observation: neighbor pixels are highly correlated

Idea:

Idea: unit of synthesis = block unit of synthesis = block

E tl th b t t P(B|N(B))

•

Exactly the same but now we want P(B|N(B))

•

Much faster: synthesize all pixels in a block at once

(39)

Algorithm

– Pick size of block and size of overlap – Synthesize blocks in raster order

– Search input texture for block that satisfies overlap constraints (above and left)

– Paste new block into resulting texture

• blending

• use dynamic programming to compute minimal error boundary cut

(40)

block

Input texture

B1 B2 B1 B2 B1 B2

Random placement p Neighboring blocks Minimal error of blocks

g g

constrained by overlap boundary cut

(41)

Minimal error boundary

overlapping blocks vertical boundary

2

__ 2 ² ² ₌ ₌

__ ₌ ₌

min. error boundary overlap error

(42)

Results

(43)

Results

(44)

Failure cases

(45)

Texture transfer

• Take the texture from one bj d “ i ” i object and “paint” it onto another object

Then, just add another constraint when sampling:

similarity to underlying image at that spot

(46)

parmesan parmesan

+

+ = =

+ +

rice

+

+ = =

(47)

GraphCut textures

(48)

GraphCut textures

(49)

GraphCut textures

(50)

Inpainting

Growing is in “onion peeling” order

• Growing is in onion peeling order

– within each “layer”, pixels with most neighbors are synthesized first

are synthesized first

(51)

Image extrapolation

(52)

Inpainting

(53)

Inpainting

(54)

Results

(55)

Results

(56)

Results

http://research microsoft com/vision/cambridge/i3l/patchworks htm http://research.microsoft.com/vision/cambridge/i3l/patchworks.htm

(57)

Structure propagation

(58)

Structure propagation

(59)

Image Analogies

?

(60)

Image Analogies Implementation

(61)

Image Analogies Implementation

(62)

Image Analogies Implementation

(63)

Balance between approximate and coherence searches

coherence searches

(64)

Coherence search

(65)

Learn to blur

(66)

Super-resolution

(67)

(68)

Colorization

(69)

Artistic filters

(70)

:

:: :

::

B B’

B B

(71)

(72)

:

:: :

::

B B’

(73)

(74)

Texture by numbers

(75)

Texture by numbers

(76)