4 Scale and rotation selection

with the Global Similarity Prior

Yu-Sheng Chen and Yung-Yu Chuang^?

Department of Computer Science and Information Engineering, National Taiwan University

{nothinglo,cyy}@cmlab.csie.ntu.edu.tw

Abstract. This paper proposes a method for stitching multiple images together so that the stitched image looks as natural as possible. Our method adopts the local warp model and guides the warping of each im- age with a grid mesh. An objective function is designed for specifying the desired characteristics of the warps. In addition to good alignment and minimal local distortion, we add a global similarity prior in the objective function. This prior constrains the warp of each image so that it resem- bles a similarity transformation as a whole. The selection of the similarity transformation is crucial to the naturalness of the results. We propose methods for selecting the proper scale and rotation for each image. The warps of all images are solved together for minimizing the distortion globally. A comprehensive evaluation shows that the proposed method consistently outperforms several state-of-the-art methods, including Au- toStitch, APAP, SPHP and ANNAP.

Keywords: image stitching, panoramas, image warping

1 Introduction

Image stitching is a process of combining multiple images into a larger image with a wider field of view [17]. Early methods focus on improving alignment accuracy for seamless stitching, such as finding global parametric warps to bring images into alignment. Global warps are robust but often not flexible enough.

For addressing the model inadequacy of global warps and improving alignment quality, several local warp models have been proposed, such as the smoothly vary- ing affine (SVA) warp [12] and the as-projective-as-possible (APAP) warp [20].

These methods adopt multiple local parametric warps for better alignment accu- racy. Projective (affine) regularization is used for smoothly extrapolating warps beyond the image overlap and resembling a global transformation as a whole.

The stitched images are essentially single-perspective. Thus, they suffer from the problem of shape/area distortion and parts of the stitched image could be

?This work was supported by Ministry of Science and Technology (MOST) and Medi- aTek Inc. under grants MOST 104-2622-8-002-002, MOST 104-2628-E-002-003-MY3 and MOST 104-2218-E-002-015.

stretched severely and non-uniformly. The problem is even aggravated when stitching multiple images into a very wide angle of view. In such a case, the distortion accumulates and the images further away from the based image are often significantly stretched. Therefore, the field of view for the stitched image often has a limit. Cylindrical and spherical warps address the problem with a fairly narrow view of the perspective warp by projecting images onto a cylinder or a sphere. Unfortunately, these warps often curve straight lines and are only valid if all images are captured at the same camera center.

Recently, several methods attempt to address the issues with distortion and limited field of view in the stitched image while keeping good alignment quality.

Since a single-perspective image with a wide field of view inevitably introduces severe shape/size distortion, these methods provide a multi-perspective stitched image. Chang et al. proposed the shape-preserving half-projective (SPHP) warp which is a spatial combination of a projective transformation and a similarity transformation [4]. SPHP smoothly extrapolates the projective transformation of the overlapping region into the similarity transformation of the non-overlapping region. The projective transformation maintains good alignment in the overlap- ping region while the similarity transformation of the non-overlapping region keeps the original perspective of the image and reduces distortion. In addition to projective transformations, SPHP can also be combined with APAP for better alignment quality. However, the SPHP warp has several problems. (1) The SPHP warp is formed by analyzing the homography between two images. It inherits the limitations of homography and suffers from the problem of a limited field of view.

Thus, it often fails when stitching many images. (2) SPHP handles distortion better if the spatial relations among images are 1D. When the spatial relations are 2D, SPHP could still suffer from distortions (Figure 5 as an example). (3) As pointed out by Lin et al. [11], SPHP derives the similarity transformation from the homography. If using the global homography, the derived similarity transformation could exhibit unnatural rotation (Figure 4(e) as an example).

They proposed the adaptive as-natural-as-possible (AANAP) warp for address- ing the problem with the unnatural rotation. The AANAP warp linearizes the homography and slowly changes it to the estimated global similarity transfor- mation that represents the camera motion. AANAP still suffers from a couple of problems. First, there are still distortions locally when stitching multiple images (Figure4(f), Figure5 and Figure6). Second, the estimation of the global simi- larity transformation is not robust and there could still exist unnatural rotation and scaling (Figure1(b), Figure3and Figure5).

We propose an image stitching method for addressing these problems and ro- bustly synthesizing natural stitched images. Our method adopts the local warp model. The warping of each image is guided by a grid mesh. An objective func- tion is designed for specifying the desired characteristics of the warps. The warps of all images are solved together for an optimal solution. The optimization leads to a sparse linear system and can be solved efficiently. The key idea is to add a global similarity term for requiring that the warp of each image resembles a similarity transformation as a whole. Previous methods have shown that simi-

(a) APAP+BA

(b) AANAP

(c) Ours (3D method)

(d) Ours with a specified horizon line

Fig. 1. Image stitching of 18 images.

larity transformations are effective for reducing distortions [4,11], but they are often imposed locally. In contrast, we propose a global similarity prior for each image, in which proper selection of the scale and the rotation is crucial to the naturalness of the stitched image. From our observation, rotation selection is essential to the naturalness. Few paid attention to the rotation selection prob- lem for image stitching. AutoStitch assumes that users rarely twist the camera relative to the horizon and can straighten wavy panoramas by computing the up vector [2]. AANAP uses feature matches for determining the best similarity transformation [11]. These heuristics are however not robust enough. We propose robust methods for selecting the proper scale and rotation for each image.

Our method has the following advantages. First, it does not have the prob- lem with a limited field of view, a problem shared by APAP and SPHP. Second, by solving warps for all images together, our approach minimizes the distor- tion globally. Finally, it assigns the proper scale and rotation to each image so

that the stitched image looks more natural than previous methods. In brief, our method achieves the following goals: accurate alignment, reduced shape distor- tion, naturalness and without a limit on the field of view. We evaluated the proposed method on 42 sets of images and the proposed method outperforms AutoStitch, APAP, SPHP and AANAP consistently. Figure 1 showcases com- mon problems of previous methods. In Figure 1(a), APAP+BA (Bundle Ad- justment) [21] overcomes the problem with limited field of view by projecting images onto a cylinder. It however uses the wrong scale and rotation and the result exhibits non-uniform distortions over the image. AANAP does not select the rotations and scales properly. The errors accumulate and curve the stitching result significantly in Figure1(b). Our result (Figure1(c)) looks more natural as it selects the scales and the rotations properly. Our method can also incorporate horizon detection and the result can be further improved (Figure1(d)).

2 Related work

Szeliski has a comprehensive survey on image stitching [17]. Image stitching tech- niques often utilize parametric transformations to align images either globally or locally. Early methods used global parametric warps, such as similarity, affine and projective transformations. Some assumed that camera motion contains only 3D rotations. A projection is performed to map the viewing sphere to an image plane for obtaining a 2D composite image. A noted example is the AutoStitch method proposed by Brown et al. [1]. Gao et al. proposed the dual-homography warping to specifically deal with scenes containing two dominant planes [5]. The warping function is defined by a linear combination of two homographies with spatially varying weights. Since their warp is based on projective transforma- tions, the resulting image suffers from projective distortion (which stretches and enlarges regions).

Local warp models adopt multiple local parametric warps for better align- ment accuracy. Lin et al. prioneered the local warp model for image stitching by using a smoothly varying affine stitching field [12]. Their warp is globally affine while allowing local deformations. Zaragoza et al. proposed the as-projective- as-possible warp which is globally projective while allowing local deviations for better alignment [20].

Instead of focusing on alignment quality, several methods address the prob- lem with distortion in the stitched images. Chang et al. proposed the shape- preserving half-projective warp which is a spatial combination of a projective transformation and a similarity transformation [4]. The projective transforma- tion maintains good alignment in the overlapping region while the similarity transformation of the non-overlapping region keeps the original perspective of the image and reduces distortion. This approach could lead to unnatural rota- tions at times. Lin et al. proposed the adaptive as-natural-as-possible (AANAP) warp for addressing the problem with the unnatural rotation [11].

A few projection models have been proposed for reducing the induced visual distortion due to projection. Zelnik-Manor et al. used a multi-plane projection as

an alternative to the cylindrical projection [22]. Kopf et al. proposed the locally adapted projection which is globally cylindrical while locally perspective [9].

Carroll et al. proposed the content-preserving projection for reducing distortions of wide-angle images [3]. When the underlying assumptions of these models are not met, misalignment occurs and post processing methods (e.g., deghosting and blending) can be used to hide it.

3 Method

Our method adopts the local warp model and consists of the following steps:

1. Feature detection and matching 2. Image match graph verification [2]

3. Matching point generation by APAP [20]

4. Focal length and 3D rotation estimation 5. Scale and rotation selection

6. Mesh optimization

7. Result synthesis by texture mapping

The input is a set of N images, I1, I2, . . . , IN. Without loss of generality, we use I0 as the reference image. We first detect features and their matches in each image by SIFT [13]. Step 2 determines the adjacency between images.

In terms of the quality of pairwise alignment, APAP performs the best. Thus, step 3 applies APAP for each pair of adjacent images and uses the alignment results for generating matching points. Details will be given in Section3.1. Our method stitches images by mesh deformation. Section3.2 describes our design of the energy function. To make the stitching as natural as possible, we add a global similarity term which requires each deformed image undergo a similarity transform. To determine the similarity transform for each image, our method estimates the focal length and 3D rotation for each image (step 4) and then selects the best scale and rotation (step 5). Section 4 describes the details of these two steps. Finally, the result is synthesize by steps 6 and 7.

3.1 Matching point generation by APAP

Let J denote the set of adjacent image pairs detected by step 2. For a pair of adjacent images Iiand Ij in J, we apply APAP to align them using features and matches from step 1. Note that APAP is a mesh-based method and each image has a mesh for alignment. We collect Ii’s mesh vertices in the overlap of Ii and Ijas the set of matching points, M^ij. For each matching point in M^ij, we know its correspondence in Ij since Ii and Ij have been aligned by APAP. Similarly, we have a set of matching points M^jifor Ij.

Figure2gives an example of matching points. Given the features and matches in Figure2(a), we use APAP to align two images. After alignment, for the left image, we have a set of matching points which are simply the grid points in the overlap regions after APAP alignment. For these matching points, we have their

(a) feature points (b) matching points (left to right)

Fig. 2. Feature points versus matching points. (a) feature points and their matches.

(b) matching points and their matches.

correspondences in the right image. In further steps, we use matching points in place of feature points because matching points are distributed more uniformly.

3.2 Stitching by mesh deformation

Our stitching method is based on mesh-based image warping. For each image, we use a grid mesh to guide the image deformation. Let V_i and E_i denote the set of vertices and edges in the grid mesh for the image I_i. V denotes the set of all vertices. Our stitching algorithm attempts to find a set of deformed vertex positions ˜V such that the energy function Ψ (V) is minimized. The criteria for good stitching could be different from applications to applications. In our case, we stitch multiple images onto a global plane and would like to have the stitched image look as natural as the original images. About the definition of natural- ness, we assume that the original images are natural to users. Thus, locally, our method preserves the original perspective of each image as much as possible.

At the same time, globally, it attempts to maintain a good structure by finding proper scales and rotations for images. Both contributes to the naturalness of the stitching. Thus, our energy function consists of three terms: the alignment term Ψa, the local similarity term Ψland the global similarity term Ψg.

Alignment term Ψa. This term ensures the alignment quality after deformation by keeping matching points aligned with their correspondences. It is defined as

Ψ_a(V) =

N

X

i=1

X

(i,j)∈J

X

p^ij_k∈M^ij

k˜v(p^ij_k) − ˜v(Φ(p^ij_k))k², (1)

where Φ(p) returns the correspondence for a given matching point p. The func- tion ˜v(p) expresses p’s position as a linear combination of four vertex positions, P4

i=1α_i˜v_i where ˜v_idenote the four corners of the quad that p sits in and α_i are the corresponding bilinear weights.

Local similarity term Ψ_l. This term serves for regularization and propagates alignment constraints from the overlap regions to the non-overlap ones. Our

choice for this term is to ensure that each quad undergoes a similarity transform so that the shape will not be distorted too much.

Ψl(V) =

N

X

i=1

X

(j,k)∈Ei

k(˜v_kⁱ − ˜v_jⁱ) − Sⁱ_jk(vⁱ_k− vⁱ_j)k², (2)

where v_jⁱ is the position for an original vertex and ˜vⁱ_j represents the position of the vertex after deformation. Sⁱ_jk is a similarity transformation for the edge (j, k) which can be represented as

Sⁱ_jk=

 c(eⁱ_jk) s(eⁱ_jk)

−s(eⁱ_jk) c(eⁱ_jk)



. (3)

The coefficients c(eⁱ_jk) and s(eⁱ_jk) can be expressed as linear combinations of vertex variables. Details can be found in [8].

Global similarity term Ψ_g. This term requires each deformed image undergo a similarity transform as much as possible. It is essential to the naturalness of the stitched image. In brief, without this term, the results could be oblique and non-uniformly deformed as exhibited by AANAP and SPHP (Figure4 and Figure 5). In addition, it eliminates the trivial solution, vⁱ_j = 0. The procedure for determining the proper scale and rotation is described in Section4. Assume that we have determined the desired scale siand rotation angle θi for the image Ii. The global similarity term is defined as

Ψ_g(V) =

N

X

i=1

X

eⁱ_j∈Ei

w(eⁱ_j)²(c(eⁱ_j) − s_icos θ_i)²+ (s(eⁱ_j) − s_isin θ_i)² , (4)

which requires the similarity transform for each edge eⁱ_j in I_iresembles the sim- ilarity transform we have determined for I_i. The functions c(e) and s(e) return the expressions for the coefficients of the input edge e’s similarity transform as described in Equation3. The weight function w(eⁱ_j) assigns more weight to the edges further away from the overlapped region. For quads in the overlap region, alignment plays a more important role. On the other hand, for edges away from the overlap region, the similarity prior is more important as there is no alignment constraint. Specifically, it is defined as

w(eⁱ_j) = β + γ

|Q(eⁱ_j)|

X

q_k∈Q(eⁱ_j)

d(qk, Mⁱ)

pR²_i + C_i², (5)

where β and γ are constants controlling the importance of the term; Q(eⁱ_j) is the set of guads which share the edge eⁱ_j (1 or 2 quads depending on whether the edge is on the border of the mesh); Mⁱ denotes the set of quads in the overlap region of I_i; the function d(q_k, Mⁱ) returns the distance of the quad q_k to the quads in the overlap regions in the grid space; R_i and C_i denote the numbers

of rows and columns in the grid mesh for Ii. At a high level, an edge’s weight is proportional to the normalized distance of the edge to the overlap regions in the grid space.

The optimal deformation of meshes is determined by the following:

V = arg min˜

V˜

Ψa(V) + λlΨl(V) + Ψg(V). (6) Note that there are two parameters, β and γ, in Ψg, controlling the relative importance of the global similarity term. In all of our experiments, we set λ_l= 0.56, β = 6 and γ = 20. Empirically, we found the parameters are quite stable because there is not severe conflict between terms. The optimization can be efficiently solved by a sparse linear solver.

4 Scale and rotation selection

This section describes how to determine the best scale s_iand rotation θ_ifor each image I_i, which is the key to the naturalness of the stitched result.

4.1 Estimation of the focal length and 3D rotation

We estimate the focal length and 3D rotation for each image by improving the bundle adjustment method proposed by AutoStitch [2]. We improve their method in two ways: better initialization and better point matches. Better initialization improves convergence of the method.

From a homography between two images, we can estimate the focal lengths of the two images [17,16,18]. After performing APAP, we have a homography for each quad of a mesh. Thus, each quad gives us an estimation of the focal length of the image. We take the median of these estimations as the initialization of the focal length and form the initial intrinsic matrix K_i for I_i. Once we have K_i, we obtain the initial guess for 3D rotation R_ij between I_i and I_j by minimizing the following projection error:

R_ij = arg min

R

X

p^ij_k∈M^ij

kKjRK⁻¹_i p^ij_k − Φ(p^ij_k)k². (7)

It can be solved by SVD. Note that AutoStich uses features and their matches for estimating the 3D rotation between two images. The problem with features is that they are not uniformly distributed in the image space and it could have adverse influence. We use matching points instead of feature points for estimating 3D rotation.

With the better initialization of K_iand R_ij, bundle adjustment is performed for obtaining the focal length f_i and the 3D rotation R_i for each image I_i. The scale si for Ii in Equation4 can be set as

s_i = f₀/f_i. (8)

(a) AANAP (b) Ours (3D method)

Fig. 3. AANAP does not select the right rotation (a). Our method does a better job and generates a more natural result.

4.2 Rotation selection

As mentioned in Section 1, although the selection of rotation is crucial to the naturalness, few paid attention to it. AutoStitch assumes that users rarely twist the camera relative to the horizon and can straighten wavy panoramas by com- puting the up vector [2]. AANAP uses feature matches for determining the best similarity transformation [11]. The heuristic is not robust enough as illustrated in Figure3.

The goal of rotation selection is to assign a rotation angle θ_i for each image I_i. We propose a couple of methods for determining the rotation, a 2D method and a 3D method. Before describing these methods, we define several terms first.

Relative rotation range. Given a pair of adjacent images I_i and I_j, each pair of their matching points uniquely determines a relative rotation. Assume that the k-th pair of matching points gives us the relative rotation angle θ_k^ij. We define the relative rotation range Θ^ij between Ii and Ij as

Θ^ij= [θ_min^ij , θ^ij_max], (9) where θ_min^ij = minkθ^ij_k and θ^ij_max= maxkθ_k^ij.

Minimum Line Distortion Rotation (MLDR). Human is more sensitive to lines. Thus, we propose a procedure for finding the best relative rotation between two adjacent images with respect to line alignment. We first detect lines using the LSD detector [6]. Through the alignment given by APAP, we can find the correspondences of lines between two adjacent images, Ii and Ij. Each pair of corresponding lines uniquely determines a relative rotation. We use RANSAC as a robust voting mechanism to determine the relative rotation between Iiand Ij. The voting power of each line depends on the product of its length and width.

The final relative rotation is taken as the average of all inliers’ rotation angles.

We denote φ^ij as the relative rotation angle between I_i and I_j determined by MLDR.

Given all relative rotation angles φ^ij estimated by MLDR, we can find a set of rotation angles {θi} to satisfy the MLDR pairwise rotation relationship as much as possible. We represent θ_i as a unit 2D vector (u_i, v_i) and formulate the following energy function:

EM LDR= X

(i,j)∈J

R(φ^ij) ui

vi



− uj

vj



2

, (10)

where R(φ^ij) is the 2D rotation matrix specified by φ^ij. By minimizing E_{M LDR}, we find a set of rotation angles θi to satisfy the MLDR pairwise rotation con- straints as much as possible. To avoid the trivial solution, we need at least one more constraint for solving Equation10. We propose two methods for obtaining the additional constraints.

Rotation selection (2D method) In this method, we make a similar as- sumption with Brown et al. [2] by assuming that users rarely twist the camera relative to the horizon. That is, we prefer that θi= 0^◦if possible. First, we need to determine the rotation angle for one image. Without loss of generality, let the angle of the reference image be 0^◦, i.e., θ0= 0^◦. Once we have the rotation angle θ_i for some image I_i, we can determine the rotation range of the image I_j adjacent to I_i by Θ_j = Θ^ij + θ_i. If 0^◦ is within the range Θ_j, it means that zero rotation is a reasonable one and we should set θ_j = 0. By propagating the rotation ranges using BFS along the adjacency graph, we can find a set of images with 0^◦ rotation. The pseudo code of the detailed process is given in the supplementary material. Let Ω be the set of images whose rotation angles equal 0^◦. We find θi by minimizing

EM LDR+ λzEZERO, where (11)

EZERO=X

i∈Ω

 u_i v_i



− 1 0



2

(12) and λ_z = 1000 so that the images in Ω are likely assigned zero rotation, i.e., keeping their original orientations.

Rotation selection (3D method) In this method, we utilize the 3D rotation matrix Riestimated at the beginning of this section. We first decompose the 3D rotation matrix Ri to obtain the rotation angle αi with respect to the z axis.

The relative rotation between two adjacent images Ii and Ij can be determined as α^ij = αj− αi. If α^ij ∈ Θ^ij, it means the estimation is reasonable and can be used. Otherwise, we should use the relative rotation φ^ij by MLDR. Let Ω be the set of pairs which use φ^ij and ¯Ω = J − Ω for others. The rotation angles are determined by minimizing

X

(i,j)∈Ω

R(φ^ij) u_i v_i



− u_j v_j



2

+ λ_r X

(i,j)∈ ¯Ω

R(α^ij) u_i v_i



− u_j v_j



2

. (13)

We set λ_r= 10 to give 3D rotation more weights.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Fig. 4. An example of stitching two images. (a) AutoStitch, (b) AutoStitch+ours, (c) APAP, (d) ASAP, (e) SPHP+APAP, (f) AANAP, (g) Ours (3D method).

5 Experiments and results

We compare our methods (2D and 3D versions) with four methods, AutoS- titch [2], APAP [20], SPHP [4] and AANAP [11]. The experiments were per- formed on a MacBook Pro with 2.8GHz CPU and 16GB RAM. SIFT features were extracted using VLFeat [19]. The grid size is 40 × 40 for mesh-based meth- ods. We tested the six methods on 42 sets of images (3 from [11], 6 from [4], 4 from [20], 7 from [14], 3 from [5] and 19 collected by ourselves). All comparisons can be found in supplementary material. The numbers of images range from 2 to 35. The test sets collected by us are more challenging than existing ones. We will release all our code and data for facilitating further comparisons.¹ Not account for feature detection and matching, for the resolution of 800 × 600, our method takes 0.1s for stitching two images (Figure4) and 8s for 35 images (Figure6).

Figure 4 compares all methods on stitching two images. Figure4(a) shows the result of AutoStitch. Note that there is obvious misalignment. Our method can be used to empower other methods with APAP’s alignment capability. Fig- ure4(b) shows the result in which the misalignment has been largely removed.

Although with good alignment quality, APAP suffers from the problem with per- spective distortion (Figure4(c)). One could change APAP’s perspective model to similarity model as ASAP which is similar to the method by Schaefer et al. [15].

Figure4(d) shows the result of ASAP. Although similarity performs well on re- ducing distortion, it is not effective for good alignment (closeup). In addition, the results would exhibit artifacts with obliqueness and non-uniform deformation.

SPHP has the problem with unnatural rotation (Figure 4(e)). AANAP gives a reasonable result in this example (Figure 4(f)), but the lines on the floor are slightly distorted as shown more clearly in the closeup. Our method has the best stitching quality in this example (Figure4(g)).

Figure 1 presents an example for obtaining a panorama by stitching 18 im- ages. SPHP failed on this example because of its limited field of view. APAP+BA overcomes the problem by projecting images onto a cylinder [21]. However, due to incorrect scale and rotation estimation, the result exhibits non-uniform dis- tortions over the image (Figure 1(a)). AANAP does not select the rotations and scales properly. The errors accumulate and curve the stitching result sig- nificantly as shown in Figure 1(b). Note that the problem cannot be fixed by the rectangling panorama method [7] because it would maintain the original orientation of the input panorama as much as possible without referring to the original images. The panorama could become rectangular but the scene would remain curved. Our result (Figure 1(c)) looks more natural as it selects the scales and the rotations properly. Our method is flexible and can be extended to comply with some additional constraints. In this example, we use a vanishing point detection method [10] for detecting the horizon for one image. With this additional constraint, the stitched image is better aligned with the horizon for a more natural result (Figure1(d)).

1 The project website :http://www.cmlab.csie.ntu.edu.tw/project/stitching-wGSP/

Fig. 5. An example of stitching six images. (top left) AutoStitch, (bottom left) SPHP+APAP, (top right) AANAP, (bottom right) Ours (2D method).

In the example of stitching six images in Figure 5, AutoStitch introduces obvious distortion because of its spherical projection (top left). SPHP cannot handle 2D topology between images and suffers from distortion (bottom left).

AANAP’s result exhibits unnatural rotation and shape distortion (top right).

Our result looks the most natural among all results (bottom right). The input of Figure6contains 35 images. AutoStitch suffers from the distortion caused by the spherical projection (top left). AANAP has distortions all over the image (top right). Both of our methods give more natural results. The 2D method keeps the perspective of each image better (bottom left) while the 3D method keeps a better 3D perspective of the original scene (bottom right).

In sum, although ASAP, AANAP, SPHP and our method all use similarity, our method gives much better results. The differences come from how similarity is utilized. SPHP attempts to reduce the perspective distortion but it fails when the field of view is wide (Figure1) and the spatial relations among images are 2D (Figure 5). AANAP attempts to address the unnatural rotation but it is not robust enough and fails frequently (Figure1(b), Figure3 and Figure 5). In addition, AANAP does not optimize for shape distortion and it only stitches two images at a time. There could exist distortions locally when stitching multiple images (Figure 4(f), Figure 5 and Figure 6). Our method addresses all these problems better than previous methods.

Fig. 6. An example of stitching 35 images. (top left) AutoStitch, (top right) AANAP, (bottom left) Our 2D method, (bottom right) Our 3D method.

6 Conclusions

This paper proposes an image stitching method for synthesizing natural results.

Our method adopts the local warp model. By adding the global similarity prior, our method can reduce distortion while keeping good alignment. More impor- tantly, with our scale and rotation selection methods, the global similarity prior leads to a more natural stitched image.

This paper presents two main contributions. First, it presents a method for combining APAP’s alignment accuracy and similarity’s less distortion. Although individual components could have been explored, we utilize them in a different way. The method also naturally handles alignment of multiple images. Second, it presents methods for robustly estimating proper similarity transformations for images. They serve as two purposes: further enforcing similarity locally and imposing a good global structure. Experiments confirm the effectiveness and robustness of the proposed method.

References

1. Brown, M., Lowe, D.G.: Recognising panoramas. In: Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2. pp. 1218–. ICCV

’03 (2003)

2. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant fea- tures. Int. J. Comput. Vision 74(1), 59–73 (Aug 2007)

3. Carroll, R., Agrawal, M., Agarwala, A.: Optimizing content-preserving projections for wide-angle images. ACM Transactions on Graphics 28(3), 43 (2009)

4. Chang, C.H., Sato, Y., Chuang, Y.Y.: Shape-preserving half-projective warps for image stitching. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. pp. 3254–3261. CVPR ’14 (2014)

5. Gao, J., Kim, S.J., Brown, M.S.: Constructing image panoramas using dual- homography warping. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. pp. 49–56. CVPR ’11 (2011)

6. Grompone von Gioi, R., Jakubowicz, J., Morel, J.M., Randall, G.: LSD: a Line Segment Detector. Image Processing On Line 2, 35–55 (2012)

7. He, K., Chang, H., Sun, J.: Rectangling panoramic images via warping. ACM Trans. Graph. 32(4), 79:1–79:10 (Jul 2013)

8. Igarashi, T., Igarashi, Y.: Implementing as-rigid-as-possible shape manipulation and surface flattening. J. Graphics, GPU, & Game Tools (1), 17–30

9. Kopf, J., Lischinski, D., Deussen, O., Cohen-Or, D., Cohen, M.: Locally adapted projections to reduce panorama distortions. Computer Graphics Forum 28(4), 1083–1089 (2009)

10. Lezama, J., Grompone von Gioi, R., Randall, G., Morel, J.M.: Finding vanishing points via point alignments in image primal and dual domains. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2014) 11. Lin, C., Pankanti, S., Ramamurthy, K.N., Aravkin, A.Y.: Adaptive as-natural-

as-possible image stitching. In: IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015. pp. 1155–1163 (2015)

12. Lin, W.Y., Liu, S., Matsushita, Y., Ng, T.T., Cheong, L.F.: Smoothly varying affine stitching. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. pp. 345–352. CVPR ’11 (2011)

13. Lowe, D.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 91–110 (2004)

14. Nomura, Y., Zhang, L., Nayar, S.K.: Scene collages and flexible camera arrays. In:

Proceedings of the 18th Eurographics Conference on Rendering Techniques. pp.

127–138. EGSR’07 (2007)

15. Schaefer, S., McPhail, T., Warren, J.: Image deformation using moving least squares. In: ACM SIGGRAPH 2006 Papers. pp. 533–540. SIGGRAPH ’06 (2006) 16. Shum, H.Y., Szeliski, R.: Panoramic image mosaics. Tech. Rep. MSR-TR-97-23,

Microsoft Research (September)

17. Szeliski, R.: Image alignment and stitching: A tutorial. Found. Trends. Comput.

Graph. Vis. 2(1), 1–104 (Jan 2006)

18. Szeliski, R., Shum, H.Y.: Creating full view panoramic image mosaics and environ- ment maps. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques. pp. 251–258. SIGGRAPH ’97 (1997)

19. Vedaldi, A., Fulkerson, B.: Vlfeat: An open and portable library of computer vi- sion algorithms. In: Proceedings of the 18th ACM International Conference on Multimedia. pp. 1469–1472. MM ’10 (2010)

20. Zaragoza, J., Chin, T.J., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving dlt. In: Proceedings of the 2013 IEEE Conference on Com- puter Vision and Pattern Recognition. pp. 2339–2346. CVPR ’13 (2013)

21. Zaragoza, J., Chin, T.J., Tran, Q.H., Brown, M.S., Suter, D.: As-projective-as- possible image stitching with moving dlt. IEEE Trans. Pattern Anal. Mach. Intell.

36(7), 1285–1298 (Jul 2014)

22. Zelnik-Manor, L., Peters, G., Perona, P.: Squaring the circle in panoramas. In:

Proceedings of ICCV 2005. vol. 2, pp. 1292–1299 (2005)