D ISPARITY R EFINEMENT P HASE

Although the disparity estimation phase can derive the disparity maps for both of left and right views, however, the disparity maps should be further refined to obtain more accurate disparity estimation results. Therefore, three algorithms called occlusion handling, directional regional voting, and edge-based temporal consistency are proposed in our disparity refinement phase to further punish the disparity maps. In occlusion handling, the occlusion artifacts are treated in order to reduce the artifact effects in the occlusion regions. The directional regional voting process performs the filtering operation for the disparity map in a directional regional manner so that the processed disparity map could be as smoother as possible. Finally, the edge-based temporal consistency operation executes the filter operation for the disparity map in the temporal domain by considering not only the color difference but the edge

41

information so that the filtered disparity map could be very smooth between the consecutive frames. The detailed design principle of each proposed module will be described in the following subsections.

3.4.1 Edge-Based Occlusion Handling

In general, the occlusion effects usually occur at the object edge and boundary.

Therefore, occlusion handing by considering the edge information is a very intuitive and straightforward manner. In our proposal, the idea of strong edge is detected to help the occlusion handing. Fig. 3.10 shows the illustration and flowchart of our proposed occlusion handling algorithm. Our proposed occlusion handling algorithm is composed by three steps called Strong Edge Detection, Reliability Check and Section

Voting and Smoothing.

Strong Edge Detection: In strong edge detection module, the edge information

after edge detection is used to determine the strong edge positions by the following procedure. The variables inside the pseudo code are defined as follows.

PosLeft: Position to indicate the location of left strong edge Pos_Right: Position to indicate the location of right strong edge ImageHeight: Image height

Image_Width: Image width

δOC: Threshold for strong edge detection

42

Set Pos

_Left

and Pos

_Right

to 0 FOR y = 0 to Image

_Height

FOR x = 0 to Image

_Width

IF (Pos

Left

has been found)

IF( |Edge

H

(x,y)| + |Edge

V

(x,y)| ≥ δ

OC

) Pos

Right

= x;

End IF

IF (Both Pos

Left

and Pos

Right

have been found) Reliability Check();

Section Voting and Smoothing ();

Pos

Left

= Pos

Right

; End IF

Else

IF( |Edge

H

(x,y)| + |Edge

V

(x,y)| ≥ δ

OC

) Pos

_Left

= x;

End IF End IF End FOR End FOR

Reliability Check: Once two strong edge positions Pos

Left and PosRight have been successfully detected, the disparities located between Pos_Left and Pos_Right are checked one by one to determine whether the disparity is reliable enough or not. If a disparity has been determined as unreliable, it will be marked as unreliable one and it will not been considered in the following disparity smoothing operation anymore. The idea behinds our reliability checking is to avoid the disparities that have been influenced by the unreliable disparities inside a restricted region. Determining whether a disparity is reliable or not can be achieved by the following operation.

43

𝐻𝑓 �|𝐷(𝑥, 𝑦) − 𝐷(𝑥 − 𝐷(𝑥, 𝑦), 𝑦)| ≤ 𝛿𝑑 𝐷(𝑥, 𝑦) 𝑚𝐻𝑒𝑚𝑏 𝐻𝑏 𝑒𝑒𝑎𝐻𝐻𝑏𝑎𝑒 𝑂𝑏ℎ𝑒𝑒𝑤𝐻𝑏𝑒 𝐷(𝑥, 𝑦)𝑚𝐻𝑒𝑚𝑏 𝐻𝑏 𝑢𝑛𝑒𝑒𝑎𝐻𝐻𝑏𝑎𝑒 𝑤ℎ𝑒𝑒𝑒 𝑥 ∈ {𝑃𝑏𝑏𝐿𝑟𝑟𝑡, 𝑃𝑏𝑏𝐷𝑤𝑎ℎ𝑡} and δ^d

is an empirical defined threshold.

Section Voting and Smoothing: After the unreliable disparities have been marked,

the section voting and smoothing operation is executed for smoothing the disparities located between Pos_Left and Pos_Right. Fig. 3.11 exhibits a step-by-step illustration for explaining our proposed section voting and smoothing algorithm. First, the disparity and reliable/unreliable maps are fed into the section voting operation for counting the occurrences of each disparity with the help of reliable/unreliable map. If a disparity has been marked as unreliable, the counts of corresponding disparity will not been accumulated. Once the section voting operation has been done, the disparity with maximum occurrence will be selected as the dominating disparity and this disparity will be used to replace all disparities located between PosLeft and PosRight.

Strong Edge Detection

Reliability Check

Section Voting and Smoothing Disparity Map Edge Map

PosLeft and PosRight

Disparity Map

Disparity Map Edge Map

Occlusion Handling

Disparity Map

Fig. 3.11. Illustration and flowchart of our proposed occlusion handling algorithm

44

Disparity Map

Reliable/Unreliable Map Reliable UnReliable

Section Voting

1

PosLeft PosRight

2 3 0

Smoothing

Disparity Map Fig. 3.12. Illustration of our proposed Section Voting and Smoothing algorithm

3.4.2 Directional Region Voting

In the traditional region voting approaches, a voting window is used for voting the occurrence of each disparity. However, this voting approach suffers from the inharmonious disparity effect inside the voting window if the covered region of voting window contains two objects. In this situation, the traditional region voting approaches would further damage the estimated disparity results. Therefore, we propose a directional region voting algorithm to avoid the problem caused by the traditional region voting via considering both of the edge and color difference information. Following figure is the diagram of procedure of proposed directional regional voting method.

45

Disparity Histogram Voting Directional Voting Area Determination Edge(x0,y0,x)>δ(x)

Initialization Set left_voting_available = 0 right_voting_available = 0

Fig. 3.13. Flowchart of proposed directional regional voting algorithm According to the algorithm, there are nine types of voting area that will be possible to be determined. Please check them as figure below:

Fig. 4.14. nine types of determined voting area

Figure 3.14 shows nine different types of determined area which might occur in the algorithm. The determined voting area is determined by the edge information on

46

both horizontal and vertical direction and also take the intensity difference between center pixel and other neighbor pixels into account. After the voting area is

determined, the algorithm will start accumulating disparity histogram of the pixels in the determined region. Only the ones which have similar intensity value will be taken into account. After the disparity histogram is complete, the disparity which get the maximum number will be assigned to the center pixel.

3.4.3 Edge-based Temporal Consistency

In traditional temporal consistency treatment, the color difference between successive frames is used for determining whether the disparity of current frame should be replaced by the disparity from the collocated pixel in previous frame to keep the disparity consistence temporally. However, as mentioned before that the occlusion effect usually occurred at the edge region. Therefore, our proposed algorithm will not only consider the color difference but also the edge information for the temporal consistence treatment. In addition, three consecutive frames are considered at the same time for deciding the disparity propagation.

47

Set StateBuffer[h][v][i]=1 where h=0~ImageWidth-1, v=0~ImageHeight-1, i=0~5, x=y=1

f >= 5

Compute edge state EState

Compute color difference state CState and CDiff

Set StateBuffer[x][y][2] = CState and StateBuffer[x][y][5] = EState

Compute color factor Cf =

Set StateBuffer[x][y][0] = StateBuffer[x][y][1] = StateBuffer[x][y][2] = CState and StateBuffer[x][y][5] = EState

y < ImageHeight - 1

Fig. 4.15. Flowchart of proposed temporal consistency algorithm

48

在文檔中 1920x1080@90fps 之深度運算設計與實作 (頁 46-54)