V OLUMETRIC G RAPH C UTS

The volumetric 3D reconstruction problem can be expressed as a labeling problem, which involves deciding whether a given voxel within the volume is inside or outside the surface of the object. The idea of the volumetric graph cuts is as follows. The true surface is assumed to be between a given base surface Sbase and a parallel inner surface Sin. The base surface is an approximation of the true surface, encloses the true surface. In practice, the base surface can be obtained from the visual hull [28]. Each candidate surface under this assumption is then scored mainly according to whether the points on the surface are photo-consistent. The algorithm finds the optimal surface by solving the minimum cut of a corresponding weighted graph. Fig. 24

45

shows the idea of volumetric graph cuts algorithm.

Specifically, for each voxel x∈R³, let ρ(x) be the photo-consistency score of x, where a lower value represents a better photo-consistency. For a candidate surface S, let V (S) be the volume between S and the base surface. Each candidate surface is associated with the energy function consisting of the integral of the photo-consistency score ρ(x) on the surface and the size of the volume V (S). The true surface S is determined by finding the global minimum of the energy function E(S) among all candidate surfaces S,

E(S)

S^∗=arg minS (40)

where

∫∫∫

∫∫ ⁺

=

) (

) ( ) (

S V S

dV dA

x S

E ρ λ (41)

In (41), the first integral tends toward a photo-consistent surface, while the second, called the ballooning term, prefers a fatter reconstructed model. The reason for preferring a fatter model is that finding the global minimum can result in a trend to remove the protrusive parts of the object. The goal of the ballooning term is to counterbalance the protrusion flattening problem. Vogiatzis et al [62] describes the detailed formulation and graph construction.

As is well known, solving the two terminals min-cut problem is equivalent to finding the maximum a posteriori (MAP) estimation of a MRF with two labels. The graph cut energy minimization, such as that used in the volumetric graph cuts, is widely adopted in many computer vision applications. Similar to most of the energy functions that can be minimized by the graph cut, (41) also includes the data and boundary properties.

Let V be the set of voxels within the base surface. Let N be a neighborhood system defined for V, which containing the set of all pairs of neighboring voxels. Let L={l_i |∀x_i∈V} be a family of random variables defined on the set V, in which each variable takes a label li from

46 }

,

{ΙΟ . Given a candidate surface S, a corresponding random field L is uniquely defined such that for any voxel p in V

In the discrete case, it can be easily proven that the energy function E(S) in (41) associated with a candidate surface S can be rewritten as E(L) which corresponds to the joint of data and boundary properties of a random field L

∑ while B can maintain the smoothness prior such that the physical property in the neighborhood of the space offers some coherence and does not change abruptly [30]. In the implementation, The edge weight, as shown in Fig. 24(b), between two neighbor voxels xi and xj is defined as to SOURCE, the terminal node indicates inside object, with the weight w_b =λh³. With the graph G constructed this way, the graph cut algorithm is then applied to find Smin.

47

(a) (b)

Fig. 24. Illustration of volumetric graph cuts algorithm. (a) Graph cuts algorithm is used to find the Smin surface between Sbase and Sin in volumetric graph cuts. (b) xi and xj

are the neighbor voxels. The edge weight between these two voxels is represented as wij

and the edge weight between voxels and source node is represented as wb. h means the length between two voxels.

4.1.1. Problem I: Not Preserving Concavity­Convex Features   

Since the graph cut algorithm usually prefers shorter cuts, concavity-convex features may be lost. This problem was described in [57] in detail. As shown in Fig. 25, the dotted line is the true surface of object, and the solid line is the surface decided by volumetric graph cuts.

Although the voxels on the true surface has high photo-consistency, the total energy is not minimized because the distance of this path is longer.

To counterbalance this problem, a simple constant penalty λ in (44) is chosen to penalize all voxels that are not inside the surface. One problem with the volumetric graph cuts is that the parameter λ has to be chosen through trial and error in order to obtain a satisfactory result.

Furthermore, the ballooning term could lead to a tug-of-war between the original protrusion flattening problem and the following concavity filling problem, where the concavities presented in the object are filled. For some objects, a befitting ballooning term still can not be found out to obtain a correctly reconstructed object even after an exhaustive search of the parameter λ. The phenomenon is also demonstrated in one of our experiments.

48

Fig. 25 Two cases that cause errors may occur in volumetric graph cuts. Because of the shorter cut property of volumetric graph cuts, Concavity-Convex feature will be flattened in volumetric graph cuts.

4.1.2. Problem II: Not Preserving Silhouettes

Because the silhouette information is not considered in [62], the inaccuracy can be observed on the silhouette of volumetric graph cuts result. Fig. 28 shows the reconstructed 3D model of potty owl using volumetric graph cuts algorithm. The ear of the reconstructed 3D is incomplete so that its projected silhouette may not match with the input silhouette. Fig. 26 shows an input silhouette image and the projected silhouette image, and the comparison is shown in Fig. 27. The green and red pixels indicate the differences between the input silhouette and the projected silhouette.

(a) (b)

Fig. 26. Silhouette images. (a) is the input silhouette image for 3D reconstruction. (b) is the silhouette image generated from the reconstruction model using volumetric graph cuts algorithm.

49

Fig. 27. The comparison between silhouette images shown in Fig. 26. The unmatched regions are colored in red and green.

Fig. 28. The broken ears is caused by not considering the silhouette information in volumetric graph cuts.