Segmentation Transfer
5.6 Boundary refinement by snakes
Btcorresponding to Bs. We assume Bshas n vertices and define UBs = {u1, u2, ..., un} as the set of parameterized u coordinate of the vertices. The orientation adjustment process aims to find the best u-offset uo ∈ [0, 1) of the Bsand Btby minimizing the following equation:
argmin where the AV (B, ui) returns the alignment value at the position whose u coordinate is uion the boundary B. If it is on the edge, the alignment value is computed by interpolating the values of the two end points of the edge. The process can be treated as rotating the target boundary along the cylindrical axis so as to align the source boundary.
After computing the best offset uo, we add uo to all the points of the source boundary and transform them to the target mesh to form the initial target boundary.
5.6 Boundary refinement by snakes
In this section, the points of target boundary are refined locally, so that the characteristics of the source and target boundaries can be as similar as possible. We define the characteristics of the boundary as the following two terms:
1. Surface detail: According to the minima rule [HR84], most of the mesh segmentation approaches use curvature, dihedral angle, and normal deviation as the local geometric features, and guide the boundaries to close these features.
2. Shape: Most of the part-type boundaries are expected to have short and smooth shapes.
However, in some scenarios, the user may depict the boundaries to the particular patterns such as jagged or wavy shapes.
The snakes were originally proposed by Kass et al. [KWT88] and were widely applied in the 2D image segmentations and feature detections. Some researches extended them to 3D
5.6 Boundary refinement by snakes 51
mesh surfaces [LL02, JK04, BWK05, LMLR06].The general energy function of the snakes can be represented as:
Esnake(s) = Z
(Eint(s) + Eext(s))ds
where the internal energy term Eint makes the snakes shorten and smooth, and the external energy term Eextdrives the snakes toward the nearby features. The snakes evolve and converge on the final locations by minimizing the energy Esnake.
We use snake to refine target boundary and reformulate the energy function to the equation 5.4:
Esnake(s) = Z
(Ef eature(s) + Eshape(s))ds (5.4)
where Eshape(s) and Ef eature(s) are called the shape term and feature term. In the section 5.5, we have completed the initial boundary mapping. For each point of the target boundary we can find a corresponding point on the source boundary. The ability of the feature term is to evolve the target boundary to increasingly close the geometric features of each pair of corresponding points, and the shape term maintains the similar shape as source boundary during the refinement process.
We detail our snake scheme as follows. We adopt the snake scheme proposed by Bischoff et al. [BWK05]. The snake is composed by a sequence of discrete points called snaxels. The snaxel is stipulated to locate on the vertices or edges of the mesh. The points of the initial target boundary may need a local adjustment in order to satisfy this condition. In the following we call the points on the target boundary as snaxels. When the snake is evolved, for each snaxel several candidates are generated, and the snaxel is moved to the candidate which has smallest snake energy in the next round. If the snaxel locates on the edge of the mesh, two candidates are yielded on both sides of the snaxel on the same edge. If the snaxel locates on the vertex, the candidates are generated to each one-ring edge connected to the vertex. The original location of the snaxel is also as one of the candidates. The interval of the original snaxel and the candidate is set according to the application. In our implementation we set it as the average length of the
5.6 Boundary refinement by snakes 52
edges of the mesh multiplied by a scalar(0.1). Our snake scheme requires the correspondence of the points between the source and target boundary. For each iteration, before the snake evolution we use equation 5.3 to find the corresponding point (may not on the vertex of the mesh) of the source boundary for each snaxel of the target boundary.
For the design of the shape term, we use the offsets of the points of the boundary and its fitting plane as the characteristics of the shape. For a straight and smooth boundary, all the points have small offsets. And for a jagged boundary, the offsets have larger deviations. To compute the offset, we calculate the projection of the point, measure the difference between them, and use the plane normal to judge the sign of the offset. The offset value is normalized by the diagonal of the bounding box of the model. We define the shape term as Eshape = abs(of f set(t) − of f set(s)), where s is the corresponding point of the source boundary for the snaxel t.
For the design of the feature term, we use the dihedral angle as the surface feature on the mesh. First the dihedral angles are calculated on the edges of the mesh, then the feature value is assigned to each vertex as the average dihedral angle of its one-ring edges and normalize it to [0, 1]. Hence for each candidate snaxel of the target boundary and the corresponding point of the source boundary, we can obtain their feature values. If the point is on the edge, the feature value is computed by linear interpolation. Our goal is to make the feature values of the both sides of points as close as possible. An intuitive idea is to design the feature term as Ef eature = abs(f eaV al(t) − f eaV al(s)), where f eaV al(p) is the feature value of the point p. However, after performing experiments, we found that it has two drawbacks. First , when both the source and target boundary locate on the flat regions, the target boundary would be expected not to move. But actually it still moved due to the small differences of the float-point feature values. Second, the snaxel usually falls into the local minimum since the snake scheme only considers a small searching range for each iteration (the candidates are close to the origin location of the snaxel).
So we modified our strategy as follows. For each iteration before the snake movement, we define a searching range, and an adjacent vertex list which collects the vertices whose geodesic
5.6 Boundary refinement by snakes 53
distances to the target boundary are under the searching range. The searching range should be large enough to cover the nearby feature regions, we set it as the longest geodesic distance on the model multiplied by a scalar(0.02). For each snaxel of the target boundary, we get the feature value of its corresponding point of the source boundary, and search the adjacent vertex list to find the closest vertex whose feature value is enough similar to the current snaxel (if the difference of the feature values is smaller than a threshold, they are enough similar. The threshold is set as 0.1). If no vertex is found, the feature energies of all the candidates are set to the same value(1.0). Otherwise, we simply define the feature energy as : Ef eature = geo(v, p), where v is the vertex searched in the adjacent vertex list, and p is the target candidate, geo(v, p) returns the geodesic distance between v and p. Under this design, the candidate which is close to the searched vertex will have small energy. In order to address the small float-point difference problem, we compute the mean(fM)and the standard deviation(fSD) of the of the feature value of the mesh. If both the target snaxel and source point are on the flat region (the feature value is in the range of fM ± fSD), we also set the energies of the candidates to 1.0 to disable the function of the feature term.
We dynamically adjust the weight of the shape and feature term during the snake movement.
If both the source and target boundaries locate around the feature regions, the weight of the feature term should be large in order to make the boundary close to the feature. Otherwise, if both the source and target boundaries locate on the flat regions, then the weight of the shape term should be large to control the shapes of the boundary. For each iteration, we calculate the percentage of the case that a target snaxel successfully find a feature similar vertex in the adjacent vertex list, set the weight of the feature term (wf) as the percentage value and set (1.0−
wf) as the weight of the shape term. The final snake energy for a target snaxel p (candidate) is defined as:
Esnake(p) = wf ∗ Ef eature(p) + (1.0 − wf) ∗ Eshape(p)
Figure 5.13 and 5.14 demonstrate the function of the shape term when both source and target boundaries are on the flat regions. And figure 5.15 shows the function of the feature term which guides the target boundary toward the concave features as similar as the source boundary.
5.6 Boundary refinement by snakes 54
(a) Source. (b) Target(Iteration=0). (c) Iteration=5.
(d) Iteration=12. (e) Iteration=17. (f) Iteration=25.
(g) Iteration=30. (h) Iteration=37. (i) Result (Iteration=45).
Figure 5.13: The evolution of the target boundary from line shape to wave shape.
5.6 Boundary refinement by snakes 55
(a) Source. (b) Target(Iteration=0). (c) Iteration=5.
(d) Iteration=10. (e) Iteration=16. (f) Iteration=24.
(g) Iteration=30. (h) Iteration=35. (i) Result (Iteration=41).
Figure 5.14: The evolution of the target boundary from wave shape to line shape.
5.6 Boundary refinement by snakes 56
(a) Source. (b) Target(Iteration=0). (c) Iteration=6.
(d) Iteration=12. (e) Iteration=18. (f) Iteration=24.
(g) Iteration=35. (h) Iteration=42. (i) Result(Iteration=50).
Figure 5.15: The evolution of the target boundary to the concave features. The color mapping on the model in (a)(b)(i) means the variations of the feature value. Green color represents flat region, red and blue color represent the convex and concave regions.
C H A P T E R 6
Result
In this section, we present the results of the skeleton correspondence, segmentation transfer and the timing information, and then discuss the limitations of our system.
6.1 Skeleton correspondence
Figure 6.1 and 6.2 show the skeleton correspondence results for a variety of models including four-legged animals, human, and some dragons. The dog (figure 6.1(a) and figure 6.2(a)) is set as the source model, then perform the skeleton correspondence to other target models. The matched node pairs are drawn in the same colors, and the unmatched nodes are not drawn.
Our results show good correspondences that most of the nodes belonged to the same semantic parts (e.g. bodies, legs, heads, and ears) are matched correctly without symmetry switching problem. Since the links are grouped with same semantic meanings, we can avoid the wrong correspondences for some parts of a model that do not exist in another models, such as mapping the tail of the dog to the leg of the horse in figure 6.1(b), or mapping the ears of the dog to the jaw of the dinosaur in figure 6.2(e). Our method can handle the models which are largely different in
57
6.1 Skeleton correspondence 58
shapes (e.g. mapping the dog to the human, the armadillo, and the dinosaur in figure 6.1(c-e)), poses (e.g. the cat in figure 6.1(g)), the local different connectivities for the junction nodes (e.g.
the junction nodes of the goat and deer in figure in 6.1(e-f)), and even can treat the skeletons with large topological differences caused by additional junction nodes and small branches (e.g.
the triceratops, elephant, and asian dragon in figure 6.2(f-h)). Due to the topological differences in the skeletons, our method performs the node clustering and link extension. So there may be one-to-many or many-to-many correspondences between nodes and links. That are different from the previous skeleton matching algorithms [HSKK01, BMSF06, ATCO+10] which just establish one-to-one node correspondence.
6.1 Skeleton correspondence 59
(a) Dog. (b) Horse. (c) Camel.
(d) Wolf. (e) Goat. (f) Deer.
(g) Cat. (h) Pig.
Figure 6.1: Skeleton correspondence results (sec. 1).
6.1 Skeleton correspondence 60
(a) Dog. (b) Giraffe. (c) Human.
(d) Armadillo. (e) Dinosaur. (f) Triceratops.
(g) Elephant. (h) Asian dragon.
Figure 6.2: Skeleton correspondence results (sec. 2).
6.1 Skeleton correspondence 61
We also apply our skeleton correspondence method to the various categories of models in the Princeton Segmentation Benchmark dataset [CGF09]. The results are shown as follows.
Note that our method can also handle the skeletons with loops, such as the chairs in figure 6.6.
(a) Source. (b) Target. (c) Source. (d) Target.
Figure 6.3: Skeleton correspondence results of Ant and Teddy models.
(a) Source. (b) Target. (c) Source. (d) Target.
Figure 6.4: Skeleton correspondence results of Bird and Fish models.
6.1 Skeleton correspondence 62
(a) Source. (b) Target 1. (c) Target 2.
Figure 6.5: Skeleton correspondence results of Armadillo models.
(a) Source. (b) Target 1. (c) Target 2.
Figure 6.6: Skeleton correspondence results of Chair models.
6.1 Skeleton correspondence 63
We compare our method with the method of elector voting [ATCO+10]. Since in the paper they demonstrate results with the skeletons proposed in [ATC+08]. For fairness of the compar-ison, we implement the method of elector voting, and use the same MSP skeletons as the input of our and their algorithms.
The matching processes of both our method and the elector voting can be described as the correspondence tree. The method of elector voting uses combinatorial search to establish all possible corresponding node pairs. Our method is based on graph traversing, which generates much less tree nodes of the correspondence tree. The meanings of the tree node are different:
The tree node of the elector voting represents a one-to-one match of a node pair between the source and target skeletons. And the tree node of our method represents a partial correspon-dence between the source and target skeletons(may contains several matches of the node pairs and link pairs). The leaf node of our method represents a complete candidate of the skeleton correspondence. And a candidate correspondence(denote it as a elector) of the elector voting is comprised by all the node pair correspondences along the path from a leaf node to the root.
The pruning tests of our method and elector voting are also different: The method of elector voting use four cascading pruning tests involving geometric features(T1,T2), topology(T3), and spatial configuration(t4) to filter out the bad node correspondences. And the pruning tests of our approach exploit the spatial configuration to exclude the bad link correspondences. Our method does not require the check of topology since the graph traversing method essentially maintains the topological consistent.
After constructing the correspondence tree, their method uses voting scheme to synthesis the final correspondence. Since we have much less leaf nodes, so a traditional approach is used which presents an objective function in order to select the final result(equation 4.7). Currently we are unable to prove that our approach outperforms the voting scheme(on the other hand, the voting scheme also hard to guarantee that it is better than the traditional approach). Table 6.1 and 6.1 list the statistics of the skeleton correspondences for our method and the elector voting.
Comparing to elector voting, our method has less requirements of parameter adjustment, the numbers of the tree node, and computation times.
6.1 Skeleton correspondence 64
Model Wolf Goat Deer Giraffe Cat Pig Human Armadillo Triceratops Asian dragon
tn 0.02 0.02 0.02 0.02 0.02 0.06 0.03 0.03 0.03 0.08
tin 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02
Tree node 17 18 18 14 16 11 12 13 17 18
Time(sec.) 0.35 0.30 0.24 0.32 0.23 0.38 0.23 0.25 0.21 0.44
Table 6.1: The statistics of our skeleton correspondence results.
Model Wolf Goat Deer Giraffe Cat Pig Human Armadillo Triceratops Asian dragon
C 0.8 0.8 0.8 0.8 1.0 1.2 1.0 1.0 1.0 0.8
RD 1.0 1.0 1.0 1.0 1.2 2.5 1.6 1.5 10.0 0.7
Tree node 2880 2385 2937 2335 1817 8523 954 2874 3519 3495
Elector 103 633 324 279 676 196 229 372 902 1128
Time(sec.) 1.21 2.54 1.37 3.13 1.08 1.30 1.31 1.24 1.14 1.36
Table 6.2: The statistics of the results of elector voting.
In figure 6.7(a), we can see that the method of elector voting mismatches the nodes at the head of the horse. The problem is that it applies multidimensional scaling (MDS) [EK03] to the skeletons in order to straighten the skeleton links. However, the directions of the straightened links are easily guided by the beginning directions of the original links. In some case the models with difference poses will cause the directions of the links are not consistent after the MDS process, and disturb their spatial pruning tests (T4). On the other hand, our method determines the link directions in the local range of the original models, which is less sensitive to the poses.
Moreover, we use the volumetric characteristics which can distinguish these small links. The ears and the jaw will be classified into difference groups. The information helps our method to establish correct correspondence as shown in figure 6.7(b).
6.1 Skeleton correspondence 65
(a) Results of elector voting. (b) Our results.
Figure 6.7: The comparison with [ATCO+10] on Dog and Horse models.
In figure 6.8(a), the method of elector voting mismatches the ear of the pig to the upper law of the dragon due to these nodes have similar features (e.g. centricity, path length and radius). On the other hand, our method produces correct result as shown in figure 6.8(b). After performing the link clustering, the two ears of the pig will form a group. And the upper and lower laws of the dragon will be classified into difference groups. Since these groups only have one link, we regards these links as ungrouped links and avoid mapping them to the links which are grouped (the ears of the pig). Using the link clustering, we have more information to establish correct correspondence.
(a) Results of elector voting. (b) Our results.
Figure 6.8: The comparison with [ATCO+10] on Pig and Dragon models.