## Image-based modeling

### Digital Visual Effects g *Yung-Yu Chuang*

*with slides by Richard Szeliski, Steve Seitz and Alexei Efros*

**Outline**

### • Models from multiple (sparse) images

St t f ti

– Structure from motion – Facade

### d l f l

### • Models from single images

– Tour into pictures – Single view metrology – Other approaches

**Models from multiple images** **(Façade, Debevec ** **et. al.** **1996)**

**Facade**

### • Use a sparse set of images

### C lib d (i i i l )

### • Calibrated camera (intrinsic only)

### • Designed specifically for modeling architecture

### • Use a set of blocks to approximate architecture

### • Three components:

g t t ti

– geometry reconstruction – texture mapping

d l fi t

– model refinement

**Idea** **Idea**

**Geometric modeling**

A block is a geometric primitive
**with a small set of parameters**
**with a small set of parameters**

Hi hi l d li f

Hierarchical modeling for a scene

Rotation and translation could be constrained

**Reasons for block modeling**

### • Architectural scenes are well modeled by geometric primitives

### geometric primitives.

### • Blocks provide a high level abstraction, easier

### t d dd t i t

### to manage and add constraints.

### • No need to infer surfaces from discrete features; blocks essentially provide prior models for architectures.

### • Hierarchical block modeling effectively reduces

### the number of parameters for robustness and p

### efficiency.

**Reconstruction**

### minimize

**Reconstruction**

**Reconstruction**

li

nonlinear w.r.t.

camera and model

**Results**

**3 of 12 photographs**

**Results**

**Texture mapping** **Texture mapping in real world**

### Demo movie

### Michael Naimark,,

### San Francisco Museum

### of Modern Art, 1984 ,

**Texture mapping** **Texture mapping**

**View-dependent texture mapping** **View-dependent texture mapping**

### model VDTM

### model VDTM

### VDTM single

### texture VDTM

### texture

### map

**View-dependent texture mapping** **Model-based stereo**

### • Use stereo to refine the geometry

**known**
**known**
**camera**
**camera**
**viewpoints**
**viewpoints**

**Stereo**

**scene point**
**scene point**

**i** **l**

**i** **l**

**optical center**
**optical center**

**image plane**
**image plane**

**Stereo**

### • Basic Principle: Triangulation

– Gives reconstruction as intersection of two raysy – Requires

• calibration
**i**

**• point correspondence**

**Stereo correspondence**

### • Determine Pixel Correspondence

P i f i t th t d t i t

– Pairs of points that correspond to same scene point

**epipolar plane** **epipolar line****epipolar line**
**epipolar line**

**epipolar line**

**p p** **p**

### • Epipolar Constraint

– Reduces correspondence problem to 1D search along
*conjugate epipolar lines*

**Finding correspondences**

### • apply feature matching criterion (e.g., *correlation or Lucas-Kanade) at all pixels * *correlation or Lucas Kanade) at all pixels * simultaneously

### • search only over epipolar lines (much fewer y p p ( candidate positions)

**Image registration (revisited)**

### • How do we determine correspondences? How do we determine correspondences?

*– block matching or SSD (sum squared differences)*

*d i th di* *it (h i* t l ti )
*d is the disparity (horizontal motion)*

### • How big should the neighborhood be?

**Neighborhood size**

### • Smaller neighborhood: more details

### L i hb h d f i l d i k

### • Larger neighborhood: fewer isolated mistakes

### w = 3 w = 20

**Depth from disparity**

input image (1 of 2)

[Szeliski & Kang ‘95]

depth map 3D rendering

X

[Szeliski & Kang 95]

z

f

x x’

f baseline

C C’

**Stereo reconstruction pipeline**

### • Steps

Calibrate cameras – Calibrate cameras – Rectify images – Compute disparityp p y – Estimate depth

– Camera calibration errors

### • What will cause errors?

– Poor image resolution – Occlusions

– Violations of brightness constancy (specular reflections) – Large motions

Low contrast image regions – Low-contrast image regions

**Model-based stereo**

key image warped offset image

offset image

**Results**

**Comparisons**

single texture, flat VDTM, flat

VDTM, model-, based stereo

**Final results**

**Kite photography**

**Final results**

**Results** **Results**

**Commercial packages**

### • Autodesk REALVIZ ImageModeler

**The Matrix**

Cinefex #79, October 1999.

**The Matrix**

**• Academy Awards for Scientific and Technical ** **achievement for 2000**

**• Academy Awards for Scientific and Technical**

**achievement for 2000**

**achievement for 2000**

**achievement for 2000**

*To George Borshukov, Kim Libreri and Dan * *Pi* *i f th d* *l* *t f * *t* * f * *Piponi for the development of a system for * *image-based rendering allowing choreographed *

* t * *th* *h * *t * *hi *

*camera movements through computer graphic * *reconstructed sets.*

This was used in The Matrix and Mission I ibl II S Th M i Di #2 f Impossible II; See The Matrix Disc #2 for more details

**Models from single images** **Models from single images**

**Vanishing points**

image plane

camera center

vanishing point

center

ground plane

### • Vanishing point

g p

### • Vanishing point

– projection of a point at infinity

**Vanishing points (2D)**

image plane

camera center

vanishing point

center

line on ground planeg p

**Vanishing points**

image plane

camera center

**vanishing point V**

center
**C**

line on ground plane line on ground plane

### • Properties

g p

### p

– Any two parallel lines have the same vanishing point
**v**

**– The ray from C through v is parallel to the lines**
– An image may have more than one vanishing pointg y g p

**Vanishing lines**

**v**_{1}**v**_{2}

### Multiple Vanishing Points

### • Multiple Vanishing Points

– Any set of parallel lines on the plane define a vanishing point

point

*– The union of all of these vanishing points is the horizon line*

*• also called vanishing line*

– Note that different planes define different vanishing lines

**Computing vanishing points**

**V**

**D**
**P**
**P** _{0}*t*
**P**_{0}

**D**

_{}

/ / /

*Y*
*X*
*Y*

*Y*
*X*
*X*
*Y*
*Y*

*X*
*X*

*t*

*D*
*D*
*D* *t*

*t*
*P*

*D*
*t*
*P*
*tD*
*P*

*tD*
*P*

**P**
**P**

### • Properties

^{}

0 /

1 / 1

*Z*
*Z*

*Z*
*Z*
*Z*

*t* *D*

*t*
*D*
*t*
*P*
*tD*
*P*

** ΠP**

### • Properties

**v**

**– P**_{}**is a point at infinity, v is its projection***– They depend only on line direction*

** ΠP**

**v**

*They depend only on line direction*

**– Parallel lines P**_{0}**+ tD, P**_{1}**+ tD intersect at P**_{}

**Tour into pictures**

• Create a 3D “theatre stage” of

• Create a 3D theatre stage of five billboards

• Specify foreground objects p y g j through bounding polygons

• Use camera transformations to navigate through the scene

**Tour into pictures** **The idea**

### • Many scenes (especially paintings), can be represented as an axis aligned box volume represented as an axis-aligned box volume (i.e. a stage)

### K ti

### • Key assumptions:

– All walls of volume are orthogonal

– Camera view plane is parallel to back of volume – Camera up is normal to volume bottom

– Volume bottom is y=0

### • Can use the vanishing point to fit the box to the

### particular Scene!

### particular Scene!

**Fitting the box volume**

### • User controls the inner box and the vanishing point placement (6 DOF)

### point placement (6 DOF)

**Foreground Objects**

### • Use separate billboard for each billboard for each

### • For this to work, three separate images used:

– Original image.

– Mask to isolate desired foreground images.

– Background with bj t d objects removed

**Foreground Objects**

• Add vertical rectangles for rectangles for each foreground object

object

• Can compute 3D Can compute 3D coordinates P0, P1 since they are on known plane.

• P2, P3 can be d computed as before (similar triangles) triangles)

**Example**

**Example** **glTip**

### • **http://www.cs.ust.hk/~cpegnel/glTIP/**

**Criminisi et al. ICCV 1999**

**Criminisi et al. ICCV 1999**

1 Find world coordinates (X Y Z) for a few points 1. Find world coordinates (X,Y,Z) for a few points 2. Connect the points with planes to model geometry

Te t re map the planes – Texture map the planes

**Measurements on planes**

**4**
**3**
**4**

**2**
**1**

**1** **2** **3** **4**

Approach: unwarp then measure What kind of warp is this?

**Image rectification**

**p’**

**p** **p**

To unwarp (rectify) an image

**• solve for homography H given p and p’**

l ti f th f **’** **H**

**• solve equations of the form: wp’ = Hp**
**– linear in unknowns: w and coefficients of H**
– H is defined up to an arbitrary scale factorp y
**– how many points are necessary to solve for H?**

**Solving for homographies**

**Solving for homographies**

**A** **h** **0**

**A** **h** **0**

### • Defines a least squares problem:

**2n × 9** **9** **2n**

### • Defines a least squares problem:

**– Since h is only defined up to scale solve for unit Since h is only defined up to scale, solve for unit **
**vector ĥ**

– Works with 4 or more pointsWorks with 4 or more points

**Finding world coordinates (X,Y,Z)**

1 Define the ground plane (Z 0) 1. Define the ground plane (Z=0)

2. Compute points (X,Y,0) on that plane
*3 Comp te the heights Z of all other points*
*3. Compute the heights Z of all other points*

**Measuring height**

**5** **5.4**
**4**
**5**

**3 3**

Camera height

**2**
**3** **2.8**

**3.3**

**1**
**2**

**Computing vanishing points**

**v**

**q**_{2}**q**_{1}

**p**
**p****2**

### • Intersect p

_{1}

### q

_{1}

### with p

_{2}

### q

_{2}

**p**_{1}

### • Least squares version

– Better to use more than two lines and compute the “closest”

point of intersection

– See notes by Bob Collins for one good way of doing this:See notes by Bob Collins for one good way of doing this:

• http://www-2.cs.cmu.edu/~ph/869/www/notes/vanishing.txt

**Criminisi et al., ICCV 99**

• Load in an image

• Click on lines parallel to X axis

• Click on lines parallel to X axis – repeat for Y, Z axes

Compute vanishing points

• Compute vanishing points

**Criminisi et al., ICCV 99**

**Vertical vanishing**
**point**

**( t i fi it )**
**Vanishing**

**line**

**(at infinity)**
**line**

**Vanishing**
**point**
**Vanishing**

**point** **point**

**p**

**Criminisi et al., ICCV 99**

• Load in an image

• Click on lines parallel to X axis

• Click on lines parallel to X axis – repeat for Y, Z axes

Compute vanishing points

• Compute vanishing points

• Specify 3D and 2D positions of 4 points on reference plane

plane

• Compute homography H

• Specify a reference height

• Specify a reference height

• Compute 3D positions of several points

• Create a 3D model from these points

• Create a 3D model from these points

• Extract texture maps Output a VRML model

• Output a VRML model

**Results**

**Zhang et. al. CVPR 2001** **Zhang et. al. CVPR 2001**

**Zhang et. al. CVPR 2001**

**Zhang et. al. CVPR 2001**

**Oh et. al. SIGGRAPH 2001** **Oh et. al. SIGGRAPH 2001**

**Oh et. al. SIGGRAPH 2001**

**Oh et. al. SIGGRAPH 2001**

### video

### video

**Automatic popup**

Input Geometric Labels Cut’n’Fold 3D Model

Ground Imageg

Vertical

Sky Learned Models

**Geometric cues**

Color Texture

Location Perspective

**Automatic popup** **Results**

Automatic Photo Pop-up Input Images

**Results**

This approach works roughly for 35% of images.

**Failures**

Labeling Errors

**Failures**

Foreground Objects

**References**

• P. Debevec, C. Taylor and J. Malik. Modeling and Rendering Architecture from Photographs: A Hybrid Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach, SIGGRAPH 1996.

• Y. Horry, K. Anjyo and K. Arai. Tour Into the Picture: y, jy Using a Spidery Mesh Interface to Make Animation from a Single Image, SIGGRAPH 1997.

• A. Criminisi, I. Reid and A. Zisserman. Single View Metrology, ICCV 1999.

• L. Zhang, G. Dugas-Phocion, J.-S. Samson and S. Seitz.

Single View Modeling of Free-Form Scenes, CVPR 2001.

B Oh M Ch J D d F D d I B d

• B. Oh, M. Chen, J. Dorsey and F. Durand. Image-Based Modeling and Photo Editing, SIGGRAPH 2001.

D Hoiem A Efros and M Hebert Automatic Photo

• D. Hoiem, A. Efros and M. Hebert. Automatic Photo Pop-up, SIGGRAPH 2005.