Disparity Estimation using Multi-Resolution Graph Cuts

In section 3.3, we describe the push-relabel algorithm which can solve the max-flow/min-cut problem. The worst-case running time for this algorithm is , where is the number of nodes and is the number of edges in the graph. Because and increase with the image size, the running time of the graph cuts algorithm greatly increases with the image size. In addition, the - swap method constructs a graph of two terminals (disparity values) at a time. If the disparity range is , we construct graph in total (all combinations).

Therefore, the running time also greatly increases with the disparity range when we

use the - swap method.

We can employ the multi-resolution graph cuts (MRGC) technique for reducing the computation time. Fig. 5-1 shows the flowchart of MRGC. We first use the image down-sampling technique to generate the low-resolution images. The right-side path in the Fig. 5-1 is same as the original GC method except the image size and the disparity range. If the disparity range of the original GC is in the original resolution, the disparity range is in the low-resolution image.

After the low-resolution disparity map is obtained, we come back to the original resolution image size. We up-sample the disparity map. Then, it becomes the initial disparity map for the neighborhood graph cuts.

Fig. 5-1 Flowchart of MRGC

The main components that are different from the original GC method are the image down-sampling, disparity map up-sampling and scaling, and neighborhood graph cuts operations. We describe the detail in the following sub-sections.

5.2.1 Image Down-Sampling

In this section, we describe two down-sampling methods that we use in MRGC.

Fig. 5-2 shows an example of a simple down-sampling method. The sampling factor is 2 for the width and the height. This method simply skips every other pixel in one-dimension. Because of lacking prefitting, the low-resolution images after the down-sampling may suffer the aliasing effect. Therefore, we attempt another method to down-sample the original images. We use a sliding window whose coefficient is

Fig. 5-2 An example of 4 to 1 pixel-skip down-sampling

Because the coefficients of the window are all integer and the sum of the coefficients is 16, we can shift right four bits in calculation instead of using division. This filter does not increase much computing time comparing to the simplest method. We will compare the two methods in section 5.4 by simulation.

5.2.2 Disparity Map Up-sampling and Scaling

In this section, we also describe two methods to up-sample the low-resolution disparity map. Fig. 5-3 shows a simple up-sampling method that duplicates the pixel value to its neighbors directly and multiplies the disparity value by 2. Thus, the derived disparity map becomes the initial disparity map for the neighborhood graph cuts. Because the disparity maps produced by the simple method may produce blocky images, we can reduce artifacts by performing some types of linear or bilinear interpolation in up-sampling. However, the interpolation process increases the computation time and its quality improvement on the disparity map it uncertain.

8 8 10 10 10 10 16 16

Fig. 5-3 An example of disparity map up-sampling and scaling

We attempt to employ the up-sampling method of H.264 [23], which is shown in Fig. 5-4. The pixel can be obtained from pixels , , , , , and by the formula below.

(5.2) We can use pixels , , , , , and , to interpolate the pixel similarly.

(5.3) The coefficients of the interpolation filter are , which mimic the sinc function. After the up-sampling interpolation process, we multiply the disparity values by 2. In section 5.4, we will compare the two methods based on the simulation results.

Fig. 5-4 The up-sampling method of H.264

5.2.3 Neighborhood Graph Cuts

In the original graph cuts, the number of graphs needed for constructing - swap is the total combinations of disparity pairs selected from the disparity range. The neighborhood graph cuts method reduces the number of constructing graph. Unlike the original graph cuts, we use the disparity map obtained from the up-sampling and scaling process as the initial disparity map. We assume that the disparity value of each pixel only differ to its neighborhood disparity values by 1. Therefore, we try to reduce the number of combinations of disparity pairs in - swap to reduce the computing time. Fig. 5-5 shows the disparity pair combination of neighborhood graph cuts. The gray nodes are the disparity values obtained from the scaling. The arrow shows the value that the disparity value can change to. That is, we select two disparity values to do the - swap.

Fig. 5-5 The disparity pair candidates in neighborhood graph cuts (a) (b)

Here, we depict two cases of the neighborhood graph cuts. In Fig. 5-5(a), the search

range of neighborhood graph cuts is . In Fig. 5-5(b), the search range is and their combination is nearly two times more than . Note that we cannot do - swap with the same value. In section 5.4, we will compare the performance of these two methods and show the computing time saved by using MRGC.