• Models from multiple (sparse) images

(1)

Image-based modeling

Digital Visual Effects g Yung-Yu Chuang

with slides by Richard Szeliski, Steve Seitz and Alexei Efros

(2)

Outline

• Models from multiple (sparse) images

St t f ti

– Structure from motion – Facade

d l f l

• Models from single images

– Tour into pictures

– Single view metrology – Other approaches

(3)

Models from multiple images

(Façade, Debevec et. al. 1996)

(4)

Facade

• Use a sparse set of images

C lib d (i i i l )

• Calibrated camera (intrinsic only)

• Designed specifically for modeling architecture

• Use a set of blocks to approximate architecture

• Three components:

g t t ti

– geometry reconstruction – texture mapping

d l fi t

– model refinement

(5)

Idea

(6)

Idea

(7)

Geometric modeling

A block is a geometric primitive with a small set of parameters with a small set of parameters

Hi hi l d li f

Hierarchical modeling for a scene

Rotation and translation could be constrained

(8)

Reasons for block modeling

• Architectural scenes are well modeled by geometric primitives

geometric primitives.

• Blocks provide a high level abstraction, easier

t d dd t i t

to manage and add constraints.

• No need to infer surfaces from discrete features; blocks essentially provide prior models for architectures.

• Hierarchical block modeling effectively reduces

the number of parameters for robustness and p

efficiency.

(9)

Reconstruction

minimize

(10)

Reconstruction

(11)

Reconstruction

li

nonlinear w.r.t.

camera and model

(12)

Results

3 of 12 photographs

(13)

Results

(14)

(15)

Texture mapping

(16)

Texture mapping in real world

Demo movie

Michael Naimark,,

San Francisco Museum

of Modern Art, 1984 ,

(17)

Texture mapping

(18)

Texture mapping

(19)

View-dependent texture mapping

(20)

View-dependent texture mapping

model VDTM

VDTM single

texture VDTM

texture

map

(21)

View-dependent texture mapping

(22)

Model-based stereo

• Use stereo to refine the geometry

known known camera camera viewpoints viewpoints

(23)

Stereo

scene point scene point

i l

optical center optical center

image plane image plane

(24)

Stereo

• Basic Principle: Triangulation

– Gives reconstruction as intersection of two raysy – Requires

• calibration i

• point correspondence

(25)

Stereo correspondence

• Determine Pixel Correspondence

P i f i t th t d t i t

– Pairs of points that correspond to same scene point

epipolar plane epipolar lineepipolar line epipolar line

epipolar line

p p p

• Epipolar Constraint

– Reduces correspondence problem to 1D search along conjugate epipolar lines

(26)

Finding correspondences

• apply feature matching criterion (e.g., correlation or Lucas-Kanade) at all pixels correlation or Lucas Kanade) at all pixels simultaneously

• search only over epipolar lines (much fewer y p p (

candidate positions)

(27)

Image registration (revisited)

• How do we determine correspondences? How do we determine correspondences?

– block matching or SSD (sum squared differences)

d i th di it (h i t l ti ) d is the disparity (horizontal motion)

• How big should the neighborhood be?

(28)

Neighborhood size

• Smaller neighborhood: more details

L i hb h d f i l d i k

• Larger neighborhood: fewer isolated mistakes

w = 3 w = 20

(29)

Depth from disparity

input image (1 of 2)

[Szeliski & Kang ‘95]

depth map 3D rendering

X

[Szeliski & Kang 95]

z

f

x x’

f baseline

C C’

(30)

Stereo reconstruction pipeline

• Steps

Calibrate cameras – Calibrate cameras – Rectify images – Compute disparityp p y – Estimate depth

– Camera calibration errors

• What will cause errors?

– Poor image resolution – Occlusions

– Violations of brightness constancy (specular reflections) – Large motions

Low contrast image regions – Low-contrast image regions

(31)

Model-based stereo

key image warped offset image

offset image

(32)

Results

(33)

Comparisons

single texture, flat VDTM, flat

VDTM, model-, based stereo

(34)

Final results

Kite photography

(35)

Final results

(36)

(37)

Results

(38)

Results

(39)

Commercial packages

• Autodesk REALVIZ ImageModeler

(40)

The Matrix

Cinefex #79, October 1999.

(41)

The Matrix

• Academy Awards for Scientific and Technical achievement for 2000

achievement for 2000

To George Borshukov, Kim Libreri and Dan Pi i f th d l t f t f Piponi for the development of a system for

image-based rendering allowing choreographed

t th h t hi

camera movements through computer graphic reconstructed sets.

This was used in The Matrix and Mission I ibl II S Th M i Di #2 f Impossible II; See The Matrix Disc #2 for more details

(42)

Models from single images

(43)

Vanishing points

image plane

camera center

vanishing point

center

ground plane

• Vanishing point

g p

• Vanishing point

– projection of a point at infinity

(44)

Vanishing points (2D)

image plane

camera center

vanishing point

center

line on ground planeg p

(45)

Vanishing points

image plane

camera center

vanishing point V

center C

line on ground plane line on ground plane

• Properties

g p

p

– Any two parallel lines have the same vanishing point v

– The ray from C through v is parallel to the lines – An image may have more than one vanishing pointg y g p

(46)

Vanishing lines

v₁ v₂

Multiple Vanishing Points

• Multiple Vanishing Points

– Any set of parallel lines on the plane define a vanishing point

point

– The union of all of these vanishing points is the horizon line

• also called vanishing line

– Note that different planes define different vanishing lines

(47)

Computing vanishing points

V

D P

P  ₀ t

P₀

D













 











 











 _

/ / /

Y X Y

Y

X X

Y Y

X X

t

D D D t

t P

D t P tD

P

tD P

P P

• Properties



 



 



 





 



 





 ^

0 /

1 / 1

Z Z

Z

t D

t D t P tD

P

 ΠP

• Properties

v

– P_ is a point at infinity, v is its projection – They depend only on line direction

 ΠP

v

They depend only on line direction

– Parallel lines P₀ + tD, P₁ + tD intersect at P_

(48)

Tour into pictures

• Create a 3D “theatre stage” of

• Create a 3D theatre stage of five billboards

• Specify foreground objects p y g j through bounding polygons

• Use camera transformations to navigate through the scene

(49)

Tour into pictures

(50)

The idea

• Many scenes (especially paintings), can be represented as an axis aligned box volume represented as an axis-aligned box volume (i.e. a stage)

K ti

• Key assumptions:

– All walls of volume are orthogonal

– Camera view plane is parallel to back of volume – Camera up is normal to volume bottom

– Volume bottom is y=0

• Can use the vanishing point to fit the box to the

particular Scene!

(51)

Fitting the box volume

• User controls the inner box and the vanishing point placement (6 DOF)

point placement (6 DOF)

(52)

Foreground Objects

• Use separate

billboard for each billboard for each

• For this to work, three separate images used:

– Original image.

– Mask to isolate

desired foreground images.

– Background with bj t d objects removed

(53)

Foreground Objects

• Add vertical rectangles for rectangles for each foreground object

object

• Can compute 3D Can compute 3D coordinates P0, P1 since they are on known plane.

• P2, P3 can be d computed as before (similar triangles)

triangles)

(54)

Example

(55)

Example

(56)

glTip

• http://www.cs.ust.hk/~cpegnel/glTIP/

(57)

Criminisi et al. ICCV 1999

1 Find world coordinates (X Y Z) for a few points 1. Find world coordinates (X,Y,Z) for a few points

2. Connect the points with planes to model geometry Te t re map the planes

– Texture map the planes

(58)

Measurements on planes

4 3 4

2 1

1 2 3 4

Approach: unwarp then measure What kind of warp is this?

(59)

Image rectification

p p’p

To unwarp (rectify) an image

• solve for homography H given p and p’

l ti f th f ’ H

• solve equations of the form: wp’ = Hp

– linear in unknowns: w and coefficients of H – H is defined up to an arbitrary scale factorp y

– how many points are necessary to solve for H?

(60)

Solving for homographies

(61)

Solving for homographies

A h 0

• Defines a least squares problem:

2n × 9 9 2n

• Defines a least squares problem:

– Since h is only defined up to scale solve for unit Since h is only defined up to scale, solve for unit vector ĥ

– Works with 4 or more pointsWorks with 4 or more points

(62)

Finding world coordinates (X,Y,Z)

1 Define the ground plane (Z 0) 1. Define the ground plane (Z=0)

2. Compute points (X,Y,0) on that plane

3 Comp te the heights Z of all other points 3. Compute the heights Z of all other points

(63)

Measuring height

5 5.4 4

5

3 3

Camera height

2

3 2.8 3.3

1 2

(64)

Computing vanishing points

v

q₂ q₁

p

p₂

• Intersect p

₁

q

₁

with p

₂

q

₂

p₁

• Least squares version

– Better to use more than two lines and compute the “closest”

point of intersection

– See notes by Bob Collins for one good way of doing this:See notes by Bob Collins for one good way of doing this:

• http://www-2.cs.cmu.edu/~ph/869/www/notes/vanishing.txt

(65)

Criminisi et al., ICCV 99

• Load in an image

• Click on lines parallel to X axis

• Click on lines parallel to X axis – repeat for Y, Z axes

Compute vanishing points

• Compute vanishing points

(66)

Criminisi et al., ICCV 99

Vertical vanishing point

( t i fi it ) Vanishing

line

(at infinity) line

Vanishing point Vanishing

point point

p

(67)

Criminisi et al., ICCV 99

• Load in an image

• Click on lines parallel to X axis

• Click on lines parallel to X axis – repeat for Y, Z axes

Compute vanishing points

• Compute vanishing points

• Specify 3D and 2D positions of 4 points on reference plane

plane

• Compute homography H

• Specify a reference height

• Compute 3D positions of several points

• Create a 3D model from these points

• Extract texture maps Output a VRML model

• Output a VRML model

(68)

Results

(69)

Zhang et. al. CVPR 2001

(70)

Zhang et. al. CVPR 2001

(71)

Oh et. al. SIGGRAPH 2001

(72)

Oh et. al. SIGGRAPH 2001

video

(73)

Automatic popup

Input Geometric Labels Cut’n’Fold 3D Model

Ground Imageg

Vertical

Sky Learned Models

(74)

Geometric cues

Color Texture

Location Perspective

(75)

Automatic popup

(76)

Results

Automatic Photo Pop-up Input Images

(77)

Results

This approach works roughly for 35% of images.

(78)

Failures

Labeling Errors

(79)

Failures

Foreground Objects

(80)

References

• P. Debevec, C. Taylor and J. Malik. Modeling and Rendering Architecture from Photographs: A Hybrid Rendering Architecture from Photographs: A Hybrid

Geometry- and Image-Based Approach, SIGGRAPH 1996.

• Y. Horry, K. Anjyo and K. Arai. Tour Into the Picture: y, jy Using a Spidery Mesh Interface to Make Animation from a Single Image, SIGGRAPH 1997.

• A. Criminisi, I. Reid and A. Zisserman. Single View Metrology, ICCV 1999.

• L. Zhang, G. Dugas-Phocion, J.-S. Samson and S. Seitz.

Single View Modeling of Free-Form Scenes, CVPR 2001.

B Oh M Ch J D d F D d I B d

• B. Oh, M. Chen, J. Dorsey and F. Durand. Image-Based Modeling and Photo Editing, SIGGRAPH 2001.

D Hoiem A Efros and M Hebert Automatic Photo

• D. Hoiem, A. Efros and M. Hebert. Automatic Photo Pop-up, SIGGRAPH 2005.