### Textures and Inpainting

### Digital Visual Effects g *Yung-Yu Chuang*

*with slides by Alex Efros, Li-Yi Wei, Arno Schedl and Paul Debevec*

**Outline**

### • Texture synthesis

### A l i b l i l i d TSVQ

### • Acceleration by multi-resolution and TSVQ

### • Patch-based texture synthesis

### • Image analogies

**Texture synthesis**

**Texture synthesis**

**Texture synthesis**

input image

### synthesis

input image

generated image

### • Given a finite sample of some texture, the goal is to synthesize other samples from that same is to synthesize other samples from that same texture.

The sample needs to be "large enough"

– The sample needs to be large enough

**The challenge**

### • How to capture the essence of texture?

### • Need to model the whole spectrum: from

### repeated to stochastic texture

**Markov property**

### • P(f

_{i}

### |f

_{i-1}

### ,f

_{i-2}

### ,f

_{i-3}

### ,…f

_{0}

### )= P(f

_{i}

### |f

_{i-1}

### ,f

_{i-2}

### ,…f

_{i-n}

### )

**ff**_{i}
**Ff**_{i-1}

**Ff**_{0} **…** **Ff**_{i-n} **…** **Ff**_{i-2}

### • P(f

_{i}

### |f

_{S-{i}}

### )= P(f

_{i}

### |f

_{Ni}

### )

*f*_{i}

*N*

_{i}*N*

_{i}*S*

**Motivation from language**

### • [Shannon’48] proposed a way to generate English-looking text using N-grams:

### English-looking text using N-grams:

– Assume a generalized Markov model

Use a large text to compute probability – Use a large text to compute probability

distributions of each letter given N-1 previous letters

lette s

• precompute or sample randomly

– Starting from a seed repeatedly sample this g p y p Markov chain to generate new letters

– One can use whole words instead of letters too.

**Mark V. Shaney (Bell Labs)**

### • Results (using alt.singles corpus):

*– "One morning I shot an elephant in my arms and *
*kissed him.”*

*– "I spent an interesting evening recently with a *
*grain of salt"*

### • Notice how well local structure is preserved!

– Now let’s try this for video and in 2D...

**Video textures**

### • SIGGRAPH 2000 paper by Arno Schedl, Riachard Szeliski David Salesin and Irfan Essa

### Szeliski, David Salesin and Irfan Essa.

**Still photos**

**Video clips**

**Video textures**

**Problem statement**

### video clip video texture

### video clip video texture

**Approach**

### How do we find good transitions?

### How do we find good transitions?

**Finding good transitions **

### Compute L

_{2}

*distance D*

_{i, j}### between all frames

*frame i*
vs.

*frame j*

### Si il f k d t iti

### Similar frames make good transitions

**Video textures**

**Ideally**

### SAMPLE

**p**

**Infinite sample image**

– Assuming Markov property, what is conditional probability
**Infinite sample image**

**generated image**

g p p y, p y

distribution of p, given the neighbourhood window?

– Instead of constructing a model, let’s directly search the g , y input image for all such neighbourhoods to produce a

histogram for p

– To synthesize p, just pick one match at random

**In reality**

### SAMPLE

**finite sample ****image**

**p**

**image**

**Generated image**
– However since our sample image is finite an exact However, since our sample image is finite, an exact

neighbourhood match might not be present

– So we find the best match using SSD error (weighted by a So we find the best match using SSD error (weighted by a

Gaussian to emphasize local structure), and take all samples within some distance from that match

*– Using Gaussian-weighted SSD is very important*

**Neighborhood size matters**

**More results**

Increasing window size

**More results**

french canvas rafia weave

**More results**

wood brick wall

**Failure cases**

**Growing garbage** **Verbatim copying**

**Summary of the basic algorithm**

### • Exhaustively search neighborhoods

**Neighborhood**

### • Neighborhood size determines the quality &

### cost cost

### 33 55 77

423 528 739 s

423 s 528 s 739 s

### 99 1111 4141

### 99 1111 4141

1020 s 1445 s 24350 s

**Summary**

### • Advantages:

t ll i l – conceptually simple

– models a wide range of real-world textures ll d h l filli

– naturally does hole-filling

### • Disadvantages:

– it’s slow

– it’s a heuristic

**Acceleration by Wei & Levoy**

### • Multi-resolution T

### • Tree-structure

**Multi-resolution pyramid**

### Hi h l i L l i

### High resolution Low resolution

**Multi-resolution algorithm**

**Benefits**

### • Better image quality & faster computation (by using smaller windows)

### using smaller windows)

### 1 level 1 level 3 levels

### 1 level

### 55 3 levels

### 1 level 55

### 1111

**Results**

Random Oriented

Regular Semi-regular

**Failures**

### • Non-planar structures structures

### • Global information

**Acceleration **

### • Computation bottleneck: neighborhood search

**Nearest point search**

### • Treat neighborhoods as high dimensional points

1 2 3 4 5 6 7 8 9 10 11 12

### Neighborhood

11 12

### High dimensional point/vector

1 2 3 4 5 6 7 8 9 10 11 12

### g p

**Tree-Structured Vector Quantization**

**Timing**

### • Time complexity : O(log N) instead of O(N)

### • Time complexity : O(log N) instead of O(N)

– 2 orders of magnitude speedup for non-trivial images

images

### Efros 99 Full searching TSVQ

### 1941 secs 503 secs 12 secs

**Results**

### Input put Exhaustive: 360 s TSVQ: 7.5 s ^{Q}

**Patch-based methods**

**non-parametric**
**sampling**

**p** **p**

**I** **t i**

**B** **B**

• Observation: neighbor pixels are highly correlated

**Input image**
**Synthesizing a block**

Observation: neighbor pixels are highly correlated

**Idea:**

**Idea: unit of synthesis = block** **unit of synthesis = block**

E tl th b t **t P(B|N(B))**

### •

**Exactly the same but now we want P(B|N(B))**

### •

Much faster: synthesize all pixels in a block at once**Algorithm**

– Pick size of block and size of overlap – Synthesize blocks in raster order

– Search input texture for block that satisfies overlap constraints (above and left)

– Paste new block into resulting texture

• blending

• use dynamic programming to compute minimal error boundary cut

block

**Input texture**

**B1** **B2** **B1** **B2** **B1** **B2**

**Random placement ****p** **Neighboring blocks** **Minimal error**
**of blocks **

**g** **g**

**constrained by overlap** **boundary cut**

**Minimal error boundary**

**overlapping blocks** **vertical boundary**

### 2

## __ 2 ^{2} ^{2} _{=} _{=}

_{=}

_{=}

## __ _{=} _{=}

_{=}

_{=}

**min. error boundary**
**overlap error**

**Results**

**Results**

**Failure cases**

**Texture transfer**

• Take the texture from one bj d “ i ” i object and “paint” it onto another object

**Then, just add another constraint when sampling: **

**Then, just add another constraint when sampling: **

**similarity to underlying image at that spot**

**similarity to underlying image at that spot**

parmesan parmesan

**+**

**+** **=** **=**

**+** **+**

rice

**+**

**+** **=** **=**

**GraphCut textures**

**GraphCut textures**

**GraphCut textures**

**Inpainting**

Growing is in “onion peeling” order

• Growing is in onion peeling order

– within each “layer”, pixels with most neighbors are synthesized first

are synthesized first

**Image extrapolation**

**Inpainting**

**Inpainting**

**Results**

**Results**

**Results**

http://research microsoft com/vision/cambridge/i3l/patchworks htm http://research.microsoft.com/vision/cambridge/i3l/patchworks.htm