CHAPTER 3. PROPOSED DEPTH REFINEMENT ALGORITHM
3.2 D EPTH R EFINEMENT A LGORITHM
3.2.3 Depth refinement
After the previous steps, the unreliable depth value pixels are divided into three categories. The refinement methods for different categories are similar. They are based on the same idea, but there is a little bit difference between the refinement methods for each category.
The detail description of the refinement method is given below.
The main idea for the refinement step is finding the closest reliable depth value pixels and using the depth value of these pixels to interpolate the depth value for the unreliable pixel.
Four-neighbor interpolation of the depth value is proposed here, which means the depth value of the unreliable depth pixel is interpolated from the depth values of its top, bottom, left, and right-nearest reliable depth pixels. If the unreliable depth pixel can not find reliable depth pixel in one direction, the weighting factor for this direction is zero and the interpolated depth value is calculated from the other directions.
Conventionally, the weighting factor for each direction is inversely proportional to the distance from the unreliable depth pixel to the reliable depth pixel when applying
interpolation function. We not only consider the inverse proportion of distance but also consider the reliability of the found reliable depth pixel. The reason is when we use the depth value of the reliable depth pixel to find the corresponding pixel in the other view, the depth value of the corresponding pixel may not be reliable. If the found nearest reliable depth pixel has such property, the reliability of this pixel should be lower than that of the pixel which has reliable corresponding pixel. Thus, we propose a reliable weighted interpolation function. The consideration of reliability can be illustrated as Figure 3-11.
Figure 3-11: Consideration of reliability.
In Figure 3-11, “a” and “c” indicate the reliable depth pixels that map to the unreliable depth pixels in the other view; “b” and “d” indicate the reliable depth pixels that map to the reliable pixels. Thus, reliability of top and left reliable depth pixels should be lower than right and bottom reliable depth pixels.
Let WDTop, WDBottom, WDLeft, and WDRight be the weighting factors calculated by the distance from current unreliable depth pixel to the nearest reliable depth pixel in each direction. The derivation of these distance weighting factors is described as follows.
a b c d
: Unreliable Area
: Reliable Area
: Reliable Area
: Reliable Area
Figure 3-12: Relation of unreliable depth pixel and four-neighbor reliable depth pixels.
In Figure 3-12, dtop, dbottom, dleft, and dright are the distance to the four closest reliable pixels. The weighting factors for each direction is
Right Left
Bottom Top
Top
Top
Quo Quo Quo Quo
WD Quo
Left
Quo Quo Quo Quo
WD Quo
where QuoDirection is represented as
Direction
where “Direction” in lower index can be either Top, Bottom, Left, or Right.
W
HighR and WLowR are the weighting values of high reliability reliable depth pixel and low reliability reliable depth pixel, respectively. The calculation of proposed weighting factor for each direction is described as follows:d
leftd
rightd
topd
bottom: Unreliable Area
: Reliable Area
: Reliable Area
: Reliable Area
if the reliable depth pixel is found to have high reliability
where “Direction” in lower index can be either Top, Bottom, Left, or Right.
Since the weighting factor for each direction is multiplied by WHighR or WLowR, the summation of four weighting factors is not equal to 1. The weighting factor for each direction should be rescaled so that their summation equals 1. The final weighting factors are shown below:
The depth value of unreliable depth pixel is interpolated by the following equation:
"
"
"
" Bottom Bottom Left Left Right Right
Top Top
Unreliable
D WD D WD D WD D WD
D
= ⋅ + ⋅ + ⋅ + ⋅ (10)Here, DUnreliable, DTop, DBottom, DLeft, and DRight are the depth values of unreliable depth pixel, top reliable depth pixel, bottom reliable depth pixel, left reliable depth pixel, and right reliable depth pixel, respectively.
The calculation of the interpolation weighting factors described above serves as the basis of the proposed depth refinement algorithm with some modification for each of the uncovered area and occluded area, respectively. The difference of each category is listed as following:
1. Non-occluded area
The calculation of weighting factor and depth value interpolation can be used directly in non-occluded area.
2. Occluded area
If a pixel is determined as an occluded pixel, its four-neighbor reliable depth pixels are still searched first. Then the reliable depth pixels used to interpolate depth value of occluded pixel are selected as the following.
The depth value is represented from 0 to 255, the smaller depth value means the pixel is in the deeper depth plane. Thus, the object which has smaller depth value will be occluded by the object which has larger depth value. The smallest depth value of the four-neighbor reliable depth pixels is chosen as a reference to filter out the neighbor in the foreground. The difference between this reference depth value and the depth value of reliable depth pixel in the other directions is then calculated. If the difference of any direction is larger than a threshold, the reliable depth pixel in this direction is considered as foreground and discarded. The depth value filtering process is illustrated in Figure 3-13. With depth value filtering we can remove the depth value of the foreground object.
Figure 3-13: Depth value filtering.
After depth value filtering, the depth value of the occluded pixel is interpolated using the same process as non-occluded area.
3. Uncovered area
Theoretically, for the pixels in the uncovered areas, we also have to find four-neighbor reliable depth pixel. However, the uncovered area exists at the left side of the left view image or exists at the right side of the right view image. There is only one side reference for each pixel in uncovered areas. The uncovered pixels in left view only have reliable depth pixel at right side and the uncovered pixels in right view only have reliable depth pixel at left side.
Therefore, the four-neighbor reliable depth pixel search can be reduced and the interpolation can be removed in depth refining process for the uncovered pixels. This process is shown in Figure 3-14. The uncovered pixel in left view copies the depth value of the right reliable depth pixel as its depth value and the uncovered pixel in right view copies the depth value of the left reliable depth pixel as its depth value.
Figure 3-14: Refinement of uncovered area.
Left: depth map of left view, Right: depth map of right view.