4.3 Hardware-Efficient Disparity Estimation Algorithm
4.3.3 Cost Diffusion Algorithm
In this subsection, we first discuss about the memory requirement of BP-M, and then propose the low memory-cost cost diffusion method to replace the BP-M.
1. Memory Requirement in BP-M
I
H,CtI
H,RtI
H,LtReference Frames Target Frame
D
H,Ct-1Previous Frame
Window-based SSAD (5x5) Temporal Cost Cal.
Cost Diffusion
Low-Resol. Occ. Handling
JBU (5x5) WVote (5x5)
Still-Edge Preservation Low Resolution
D
H,CtDL,Lt
Side-View Process
DL,Rt
DL,Ct
DH,C t
ML,C
EH,C,MH,C
123
The original BP-M updates the messages in four directions as illustrated in Figure IV-31 (a), where the message passing is performed direction by direction. Figure IV-31 (b) shows the data dependency of message passing in the node level. In which, the new message is computed for the
“updated message” using the “used messages”. First, the right message passing updates the left incoming message of each node in the order of left-to-right and row-by-row. Then, the left message passing is performed in inverse direction to update the right incoming message of each node. With the same manner, the down message passing and the up message passing is performed column by column.
Note that the “used messages” in the right message passing and left message passing could be removed, because their values are initially 0 and the horizontal message passing are performed for one time in the single iterative BP-M.
(a)
(b)
Figure IV-31 Concept of BP-M computation
(a) Message passing in four directions, (b) data dependency of messages in four directions
With the data dependency of BP-M, all the messages of whole frame have to be stored in memory until the up message passing and the final disparity selection is performed. Thus, the memory requirement is 4H×W×DR for messages in the HQ-DE algorithm as listed in Table IV-4.
To reduce the memory cost in BP-M, we first propose the horizontal-only BP-M that only performs the left message passing and the right message passing steps. The horizontal-only BP-M
Right Direction Left Direction Down Direction Up Direction
……
……
……
……
……
……
……
……
Right Message Passing Left Message Passing
Down Message Passing Up Message Passing
Updated Message Used Message
Unused Message
124
could reduce the memory cost from frame-scale-magnitude to row-scale-magnitude as shown in Table IV-4. However, its memory cost still has the factor of disparity range DR. Thus, we further propose the cost different method to completely address the high memory cost problem.
Table IV-4 Comparison of memory requirement between BP-M and cost diffusion methods
Method Memory Requirement Operation Times
of Message Passing Single iterative BP-M
(HQ-DE)
4H×W×DR
H×W×DR
(Message)
(Matching Cost) 4H×W Horizontal-Only BP-M
W×DR
W×DR
(Message)
(Matching Cost) 2H×W Cost Diffusion
(HE-DE)
W W
(Matching Cost)
(Disparity Map) 2H×W
2. Proposed Cost Diffusion Method
The main idea of the proposed cost diffusion method is to diffuse the matching cost of current pixel to its neighbor by the message passing mechanism, and immediately determine the best disparity for the current pixel. The cost diffusion method includes the strong horizontal diffusion and the weak vertical diffusion. That is because the human eyes weakly percept the vertical disparity and are sensitive to the horizontal disparity [108], [109]. It implies the demand of vertical disparity is lower than that of horizontal disparity. Therefore, the horizontal diffusion applies a complicated mechanism and the vertical diffusion applies a simple one.
The horizontal diffusion method consists of the right cost diffusion process and the left cost diffusion process. The two processes can generate two disparity rows, which will be merged into one by our specific constraint. In the two processes, the disparities are computed immediately and the diffused costs could be thrown at the same time. In the proposed cost diffusion method, only the best disparity row and the corresponding minimal matching costs need to be stored. Its memory requirement is listed in Table IV-4.
The details of the proposed cost diffusion method are described using the right cost diffusion process as an example. In the right cost diffusion process, the final cost RCfinal is computed from left to
125
right, and it is combined with the original cost Ctotal to determine the disparity row. The final cost
RC
final is computed by𝑅𝐶𝑓𝑖𝑛𝑎𝑙(𝑥, 𝑦, 𝑑) = 𝐶𝑡𝑜𝑡𝑎𝑙(𝑥, 𝑦, 𝑑) + 𝑅𝐷𝐶(𝑥 − 1, 𝑦, 𝑑) . (IV-18)
where the diffused cost RDC(x-1, y, d) at the left neighbor is defined as 𝑅𝐷𝐶(𝑥 − 1, 𝑦, 𝑑) = min
𝑑𝑠 .𝑉(𝑑, 𝑑𝑠) + 𝑅𝐶𝑓𝑖𝑛𝑎𝑙(𝑥 − 1, 𝑦, 𝑑𝑠)/ − 𝜅 , (IV-19) where V is the smoothness term in (IV-7), and κ is the average of RDC for normalization. This equation is similar to the calculation of message passing in (III-7) but all the messages are removed.
Then, the previous diffused cost RDC is combined with the matching cost of current pixel by for the current pixel (x, y). With the final cost RCfinal, the temporary best disparity RDbest could be immediately calculated by the winner-take-all (WTA) manner, and the minimal cost RCmin is also computed for the final disparity decision. They are calculated by
𝑅𝐶𝑚𝑖𝑛(𝑥, 𝑦) = min
𝑑 𝑅𝐶𝑓𝑖𝑛𝑎𝑙(𝑥, 𝑦, 𝑑) (IV-20)
𝑅𝐷𝑏𝑒𝑠𝑡(𝑥, 𝑦) = arg min
𝑑 𝑅𝐶𝑓𝑖𝑛𝑎𝑙(𝑥, 𝑦, 𝑑) . (IV-21)
By the above right cost diffusion process, we could obtain the temporary best disparity RDbest and the minimal cost RCmin of one frame row. We also can acquire the LDbest and LCmin by the left cost diffusion process. Finally, the two disparity row RDbest and LDbest are merged into one by the equation
𝐷𝐿(𝑥, 𝑦) = {𝑅𝐷𝑏𝑒𝑠𝑡(𝑥, 𝑦) 𝑖𝑓 𝑅𝐶𝑚𝑖𝑛(𝑥, 𝑦) < 𝐿𝐶𝑚𝑖𝑛(𝑥, 𝑦)
𝐿𝐷𝑏𝑒𝑠𝑡(𝑥, 𝑦) 𝑒𝑙𝑠𝑒 . (IV-22)
according to the minimal costs RCmin and LCmin. In which, we take the disparity with minimal cost as the final disparity.
On the other hand, the concept of vertical cost diffusion is to propagate the disparities of previous row into the current row. Thus, we define another new vertical cost Cvert as
𝐶𝑣𝑒𝑟𝑡(𝑥, 𝑦, 𝑑) = 𝜆𝑣𝑒𝑟𝑡|𝑑 − 𝐷𝐿𝑡(𝑥, 𝑦 − 1)| (IV-23) where DL
t(x, y-1) is the disparity in previous row, λvert is a scaling term. Note that this cost is constrained by the color consistency between the current pixel IL
t(x, y) and the previous row pixel IL t(x,
126
y-1). If the two pixels are inconsistent, the vertical cost C
vert would be 0. Thus, the total cost cube in the HE-DE algorithm is defined as𝐶𝑡𝑜𝑡𝑎𝑙(𝑥, 𝑦, 𝑑) = 𝐶𝑎𝑔𝑔𝑟(𝑥, 𝑦, 𝑑) + 𝐶𝑡𝑒𝑚𝑝(𝑥, 𝑦, 𝑑) + 𝐶𝑣𝑒𝑟𝑡(𝑥, 𝑦, 𝑑) . (IV-24)
To minimize the memory requirement of cost diffusion method, we could perform the right cost diffusion process first and the left cost diffusion process latter. Thus, only one disparity row and one cost row required to be stored. Compared to the BP-M, the proposed cost diffusion method could reduce the memory cost to 0.00029% for the HD1080p resolution and 128 disparity levels.