由RGB-D影像資訊產生多視角立體電影用之影像

全文

(1)RGB-D. Multiview Stereo Images Generation from RGB-D Images. ○ I.

(2) ! RGB-D. 3D. 3D 3D 3D. RGB+D. 3D. ; rank. 2D 2D. 2D. 3D. 3D. 3D. II. 3D. 3D.

(3) Abstract Multiview Stereo Images Generation from RGB-D Images Chen-Yu Liu. Nowadays, 3D display technology has been well developed and gradually became a matured technology. However, limited 3D contain resources obstruct this technology to be popularized to the market. Even if the customers can afford expensive media equipment, there is still lack of useable resources to function 3D display technology. This research provides the solution of converting RGB+D image to 3D image to partially improve the shortage of 3D resources. In recent decades, many researches are already working on how to create 3D images, which always involved depth measurement and generating image with another perspective. Depth measurement can be done by implementing the solutions such as manual judgments, depths cues, or using depth cameras. The former two solutions are relatively time consuming than the latter one. Especially the depths cues usually cause inaccuracy. Moreover, using depth cameras simplifies the difficulties of getting the depth data and decreases the inaccuracy as well. But there is a problem when using the cameras to collect the depth data, the images may have holes occurs which depends on shooting scenarios. The depth data need to be repaired under a reasonable condition because these two factors impact the 3D images’ qualities. In the past, solution to image inpainting has been proposed from many researches. The main considerations are about the colors and the texture. This research implements two methods to process the missing value of depth images. One is based on images’ low rank feature to use matrix completion technique; the other is based on image segmentation technique to do III.

(4) the depth image repairing. The results of experiment show that our 3D depth quality is obviously higher than the traditional 2D convert to 3D method. Furthermore, depth camera collects the depth data with higher accuracy so we can provide viewers a better experience in 3D display technology.. !. Key word 2D to 3D matrix completion image segmentation 3D image inpainting DIBR. IV.

(5) !. IPCV. 3D. CVIU. KDD. ! V.

(6) ! ! II!. !. III!. ABSTRACT! !. V!. !. VI! !. VIII!. !. X!. !. 1!. !. !. !. 1!. !. !. 2!. !. !. 9!. !. !. 10!. ! ! !. ! !. 11!. !. 11!. !. 12!. !. 17!. !. 17!. !. !. !. 24!. !. !. 28! VI.

(7) !. !. 28!. !. 3-D!IMAGE!WARPING!. 30! 33!. !. !. !. !. 35!. !. !. 35!. !. 36!. !. !. 3D. !. 39!. !. 42!. !. 44!. !. !. !. VII.

(8) !. 1.1. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!3!. 1.2. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!3!. 1.3. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!3!. ! 1.4. 1.5. (STEREOSCOPIC!DISPLAY!TECHNOLOGY). !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!7!. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!9!. DIBR. ! 2.1!:!. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!11!. ! 2.2!:!. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!12! !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!12!. ! 2.3:! ! KINECT! ! 2.4!. (A)!. (C)! ! 2.5!. (B)!. (A). !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!13!. (B). (A)!. (C)!. (B)!. (A). ! 2.6!. (A)!. ! 2.7!. (A)!. REGISTRATION. REGISTRATION. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!14!. (B). !$$$$$$$$$$$$$$$$$$$$$$$$$$$!15!. (B). !$$$$$$$$$$$$$$$$$$$$$$$!15!. (B). ! 2.8. MULTIVIEW. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!16!. ! 2.9. MULTIVIEW. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!16!. 3.1. (A). 3.2. (A). 588 × 453. (B)RFCHANNEL. (C)GFCHANNEL. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!18!. (D)BFCHANNEL. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!19!. 3.3 3.5. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!17!. (B). KFMEANS. SUPERPIXEL. SLIC. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!25! 3.6. SLIC. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!26! VIII.

(9) 3.7. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!27!. SLIC. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!27!. 3.8 4.1. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!29!. 4.2 4.3. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!28!. DIBR. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!29!. 8FBIT. 4.4. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!30!. BILATERAL!FILTER!SMOOTH. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!31!. 4.5 4.6. (A). SMOOTH. 4.7 5.1. (B). (C). BILATERAL!FILTER!. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!32!. WARPING. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!33! 2D. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!35!. 3D. 5.2!. (A). VIA!TNNR!–METHOD!A. (F)[CAI08]SVT. WARPING. (B). (C)!MATRIX!COMPLETION!. (D)!MATRIX!COMPLETION!VIA!TNNR!–METHOD!B. (G)[BER01]. (H)[TEL04]. (E)IMAGE!SEGMENTATION. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!37!. !. IX.

(10) !. 1.1!. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!4!. 1.2!. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!6!. 1.3! ! CSV 5.1! !. 2D+D. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!8! !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!38!. 5.2! ! 3D. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!40!. 5.3! ! 3D. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!40!. 5.4! !. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!41!. 5.5! !. !$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$!41!. X.

(11) !. (High Definition, HD) 3D 70. “ 3D. ;. “2010. 3D. ”. 3D. ”. 3D. 2010 ;. “. ”. 3D 3D. 3D 3D. LG Optimus 3D. HTC EVO 3D. 3D 1. 2. Kinect 3.. 2D 2D. National Geographic. Animal Planet. Discovery Channel. BBC. 3D [. 99]. [Feh04] 2D. 3D. 2D. 2D 1.

(12) 2D 2D. (re-projection). 1. ; 2.. frames. frames. ; 3.. frames. [Ide08] [Bat04] [Com97] [Ang10] [Che10] Depth from cues [Li06] Structure from Motion(SfM) [Har02] Machine Learning Algorithm(MLA). 3D. : 2010. (Microsoft). (RGB) ; 2011. [. (Asus). Kinect. (Depth). WAVI Xtion. RGB-D. (Depth). 10][Rob88]. (Depth cues) Depth from cues (binocular cue. (monocular cue. ). :. 1.1 screen. ). (zero parallax). (Positive parallax). (zero parallax). screen (binocular. parallax) 2.

(13) 1.1. 1.2. 1.3. 3.

(14) 6.5cm 1.2 3D. 1.3. (depth perception); (depth cue). ; ; (vanishing point). ;. ,. ; ;. ; ;. 1.1. 1.1!. !. (Motion Parallax) (Atmospheric Perspective) (Linear Perspective) (Texture Gradient). (Retina Size. Familiar Size). 4.

(15) (Interposition. Occlusion). (Relative Brightness) (Distance to Horizon). (Image Inpainting) (Interpolation.). interpolation. M. Camplani and L. Salgado [Cam12]. joint-bilateral filtering framework ; Xu et al. [Xu12] 4-neighbor-pixels. Structural inpainting Textural inpainting. [Cri04]. Structural and Textural. 5.

(16) 1.2!. !. (Anaglyph Glasses). (Shutter Glasses). (Polarization Glasses). (Pulfrich Glasses). (Head Mounted Display). 6.

(17) 1.4. (Stereoscopic Display Technology). 1.4 (stereoscopic display). (auto-stereoscopic display). display. 2-view auto-stereoscopic display. 3D. (3DTV). [. 99]. multi-view. stereoscopic display. 1. Video, CSV). stereoscopic. 2. 2D. 1.2. (Conventional Stereo (2D+Depth). CSV. 1.3 2D+Depth. 2D. 7.

(18) 1.3! ! CSV. 2D+D. !. CSV. 2D+Depth. Fehn [Feh04]. 2004. (Depth Image based. Rendering, DIBR). (occlusion) C. Fehn DIBR. L. Zhang and W.J. Tam [Zha05] 3D. S. Zinger[Zin10]. Free-view point. Lai et al. [Lai13]. Zhang et al.. 3D. DIBR. 1.5. Fehn. 3D. DIBR. (1). (preprocessing of depth map) (2)3-D image warping DIBR. 8. (3). (hole filling).

(19) 1.5. DIBR. Kinect. (RGB). (Depth). RGB-D. 1.6. Kinect. (diffuser). (laser speckle). Kinect. Kinect. 2D Depth Image Based Rendering (DIBR) Kinect 3D. 3D. Diffuser. Rough surface. IR laser. Depth camera Laser speckle 1.6. Kinect. 9.

(20) ; ; ;. 10.

(21) !. 2.1. 2.2. Kinect Kinect. 3D LED (RGB). (D). RGB-D 3D. 2.1 57. 43. Kinect 1.2m~3.6m. Kinect USB. (RGB) (D). DIBR. 2.1 :. 11. Kinect.

(22) 2.2 : 2.2. Kinect. RGB-D. (RGB). (depth). registration(registration of color and range images) image). (repairing of range. (generation of multiviews). (post. processing of multiviews). (RGB camera) (IR laser). (Depth camera). 2.3: Kinect Kinect. (RGB camera). (depth camera). 2.3 2.4 (a). (b) (c). (a). (b). 2.4(c) 12. registration.

(23) registration 2.5. (a). (b). (c) 2.4. (a). (b) (c). (a). registration (b). 13.

(24) (a). (b). (c) 2.5. (a). (b) (c). (a). registration (b). Kinect. Kinect 0 2.4(b) 2.5(b) 0 0. (matrix completion) 14.

(25) bilateral filter 2.6 2.7. 2.4(b). 2.6. 2.7. (a). (b). (a). (b). (Depth Image Based Rendering). (multi-view). 2.8. 15.

(26) 2.8. multiview (occlusion). 2.8. 2.9. 2.9. multiview. 16.

(27) ! Kinect ; ;. Kinect. 0. (missing value). 1. [Hu13]. (matrix completion). Hu et al. 3.1. ; 2.. Achanta et al. [Ach12] 3.2. 3.1. 3.1(a) (. ?. ). 3.1(b). Hu et al.. Hu et al.. (a). (b) 3.1. (a). (b) image inpainting. [Kom06] [Ras05]. denoising [Ji10]. ill-posed 17.

(28) Lee and Verleysen [Lee10] (low rank) (manifold) Lee and Verleysen 3.2(a). (588 × 453) (R, G, B channels). (Singular Value Decomposition, SVD). 3.2(b), (c), (d) 20. 0 0. x!10000!. 0. 10! 8! 6! 4!. 0!. (b)B-channel. 8!. x!10000!. 6!. 10! 8! 6!. 4!. 4!. 2!. 2! 1! 21! 41! 61! 81! 101! 121! 141! 161! 181! 201! 221! 241! 261! 281! 301! 321! 341! 361! 381! 401! 421! 441!. 0!. (c)G-channel. 3.2 (a). 588 × 453. 1! 21! 41! 61! 81! 101! 121! 141! 161! 181! 201! 221! 241! 261! 281! 301! 321! 341! 361! 381! 401! 421! 441!. x!10000!. (a)an image example. 0!. 1! 21! 41! 61! 81! 101! 121! 141! 161! 181! 201! 221! 241! 261! 281! 301! 321! 341! 361! 381! 401! 421! 441!. 2!. (d)R-channel. (b)R-channel (d)B-channel 18. (c)G-channel.

(29) intensity range. intensity 3.3. Candès and Recht [Can08]. min !!!"#$ ! !. (1). !. !.!!!!!!" = !!" , (!, !) ∈ Ω, ! ∈ ! ℝ!×! (!, !). !. ! ∈ ! ℝ!×! !. Ω. 3.3 (1). NP-hard. rank( ). (non-convex). (convex programming) Fazel [Faz02]. [Rec10]. nuclear norm (NN). Recht et al. rank( ). (lower bound). (1). [Cai10] [Toh10] [Wri09] 19.

(30) min! !. !. ∗. !"#(!,!) !! (!) !!!. =. !!. !. !!" = !!" , !, ! ∈ Ω. ∗. !×!. X. !! (!). nuclear norm. Hu et al. [Hu13]. X. i. r. !"# !, ! − !. !. nuclear norm (TNN). (2). !. Hu et al.. !. !"#(!,!) !! (!) !!!!!. =. truncated nuclear norm. !. (2) !"#! !. (!! ! )!" =. !. !!. !. !! ! = !! ! ,. !!" !"(!, !) ∈ ! 0 !"ℎ!"#$%!. !. (3). Hu et al.. !. (3) X = UΣV !. X ! ∈ !!×!. ! = (!! , ⋯ , !! ) ∈ !!×!. ! ∈ !!×! , ! ∈ !!×!. !. = !. ∗. ! = !! , ⋯ , !!. !. ! = !! , ⋯ , !!. A !! = ! !×! , B !! = ! !×! !. ( !!! = !! , !!! = !! ) !. ! = (!! , ⋯ , !! ) ∈ !!×!. !"#!!! !!,!!! !! !"(!"!! ) =. − !"#!!! !!,!!! !! !"(!"!! ). !. ∗. !. unitary ! ! !! (!). −!"(!"!! ). (3) !"# !. !. ∗. −. !"#. !!! !!,!! ! !!. !"(!"!! ) !!. !. !! ! = !! ! !! = !!. Hu et al. !!. unitary. !!. (4). (l) !!. !!. !!. !!!! !!!! = !"#!!"# ! ! !. ∗. − !"(!! !!!! ) !!. !. !! ! = !! !. Algorithm 1. 2. !!!! (. (5). (5). ) Hu et. al. !"#! !. (5) !(!, !, !, !) = ! !. ∗. ∗. − !"(!! !!!! ) !. !. ! = !, !! ! = !! ! !. − !"(!! !!!! ) + ! ! − !. augmented Lagrange function. L 20. ! !. + !"(! ! (! − !)).

(31) !! = !!. !! = !!. !! (!) = !"# !"#!. Y! = X! ! !. !−!. ! !. !!!! = !!. +! !. !. !. !!!! = !!!! + ! (!!! !! + !! ). ∗. !!!! = !!! (!!!! ) + !! (!) !!!! = !! + !(!!!! + !!!! ). X. Algorithm 2 ADMM Algorithm 1 Input:. Incomplete matrix !! , where ! is the position of the observed entries, tolerance !! .. Initialize:. !! = !! (!). repeat STEP 1. Given !! , !! , ∑! , !! = !"#(!! ), where !! = !! , ⋯ , !! ∈ ℝ!×! , !! = !! , ⋯ , !! ∈ ℝ!×! . Compute !! = !! , ⋯ , !! ! , !! = !! , ⋯ , !! ! . STEP 2. until Return. Solve !!!! = !"# !!!! − !!. !. !"#! ! ∗ − !"(!! !!!! ) !. !. !! (!) = !! (!). ≤ !!. !!!! Algorithm 2 ADMM. Input:. !! , !! , !! and tolerance !.. Initialize:. !! = !! , !! = !! , !! = !! ,!and ! = 1. repeat STEP 1. STEP 2.. !!!! = !!. !! −. !. !!!! = !!!! +. 1 ! ! !. 1 ! (! ! + !! ) ! ! !. Fix values at observed entries !!!! = !!! (!!!! ) + !! (!). STEP 3. !!!! = !! + !(!!!! − !!!! ) until. !!!! − !!. !. !. !! − ! !!. ≤! 21.

(32) ADMM. Hu et al.. (5) !"# ! !. ∗. − !"(!! !!!! ) +. ! ! (!) − !! (!) 2 !. ! !. (6). X !!!! = !"# !"# !. !. ∗. +. 1 ! − (!! − !! !"(!! )) 2!!. ! !. =!!!! !! − !! !"(!! ) =!!! !! − !! !!! !! − ! !! !! − !! ! !! !!!!!!. !!!!. !. ! !!. , !!!! = !!!! + !!. !!!. !!!! − !!. Algorithm 3 Algorithm 3 APGL Input:. !! , !! , !! and tolerance !.. Initialize:. !! = 1, !! = !! , !! = !!. repeat STEP 1. Update !!!! as !!!! = !!! !! + !! !!! !! − ! !! !! − !! ! STEP 2. STEP 3. until. !!!! !!!!. 1 + 1 + 4!!! 2 !! − 1 = !! + !! − !!!! !!!!. !!!! − !!. !. ≤!. 3.4. APGL. 22. ,.

(33) (5). Hu et al.. !!!! = ! ! (!! − !. !!!! =. 1 ! ! !. 1 !! ! ! − !!!! − (!!! !! + !! 2! + !!. !!. !! ). + !!. !! ). + !!!! +. 1 ! (! ! ! ! !. !! ). !!!! = !! + ! !!!! + !!!! !!!! = min !!"# , !!!. β !! ! if!!!!!! !"#. !=. 1. −! 0. !. !! ! =. ! 0. !!!! !!! ! , !!!! !!! ! !. otherwise Algorithm 4 ADMMAP. ℬ ! =. !!"#. 0 0 0 ,! = !! (!) 0 !! (!). !. 0 ,! 0 ℝ!×! →. ℬ. ℝ!!×!!. Algorithm 4 ADMMAP Input:. !! , !! , !! and tolerance !.. Initialize:. !! = !! , !! = !! , ! = 10!! , !! , !! = !! !!"#!!. repeat STEP 1. Set !! and!!! !fixed !!!! = ! ! !! − !!. 1 (! ) . !! ! !!. STEP 2. Set !! and!!!!! !fixed !!!! =. 1 ! ! ! − !!!! − (!!! !! + !! 2!! ! ! + !!. !! ). + !!!! +. 1 ( !!! !! + !! !!. STEP 3. Set !!!! and!!!!! !fixed !!!! = !! + !! ! !!!! + ℬ !!!! − ! STEP 4. Update !! !by !!!! = min !!"# , !!! until. !!!! − !!. !. ≤! 23. !!. !!. ..

(34) Achanta et al. [Ach12] Simple Linear Iterative Clustering(SLIC) (superpixel). superpixel superpixel. superpixel SLIC k !!. cluster 3×3. pixels. lowest gradient. superpixel cluster. edge. SLIC. clustering. [!, !, !, !, !]!. 5-D !. CIE Lab. !. [!, !, !]! [!, !]!. !. !! !! ! = ! !! ! = ! !. ! =. !. !! − !!. !! − !! !! !!. !. cluster. !. !. superpixel. .. !/!. !. superpixels 2!!×!2!. SLIC !-means. pixel. !. !. + !! − !! , !. !! = ! =. 3.5 [Ach12]. !. + !! − !! ,. !! + !!. !! !. + !! − !!. !-means SLIC 24. ! !"# ! !. !.

(35) 3.5. !-means. superpixel. !!. cluster. SLIC. cluster !!. !. !! !! !. !. ! =. !. ! = !!. !!. !. !. !. !! + !. !! + !. .. !. !! .. !. !. superpixel. clustering. ! clustering. superpixel. (edge) !. CIELAB. 1,40 !! ! = !. !! − !!. !. (z). 3D supervoxel. !! ! = !. !! − !!. !. + !! − !!. SLIC 25. !. + !! − !!. !.

(36) /* Initialization */ Initialize cluster centers !! = !! , !! , !! , !! , !! ! by sampling pixels at regular grid steps S. Move cluster centers to the lowest gradient position in a 3×3 neighborhood. Set label l(i)= −1 for each pixel i. Set distance d(i) = for each pixel i. repeat /* Assignment */ for each cluster center !! do for each pixel I in a 2S 2S region around !! do Compute the distance D between !! and i. if D < d(i) then set d(i) = D set l(i) = k end if end for end for /* Update */ Compute new cluster centers. Compute residua error E. until E ≤ threshold. superpixel. SLIC. clustering. cluster connected components cluster. 3.6. SLIC 26. pixel.

(37) 3.7. SLIC SLIC. superpixels. 800. superpixels. 3.6. edge. superpixels. 3.7. superpixels. pixels superpixel. pixel. superpixel. superpixel. superpixel !. !. superpixel superpixel 3.8. threshold. superpixel threshold. ;. 3.8 27.

(38) ! !. Zhang et al.[Zha05] 4.1. (1). (preprocessing of depth map)(4.1 (hole filling) (4.3. DIBR. ) (2)3-D image warping(4.2. ). (3). ). (occlusion). 4.1. DIBR. (a) setting, ZPS). (b). (zero parallax (smoothing). 4.2 (positive parallax). (recessed). (negative parallax). (prominent) ZPS. al.. !! = !. (!!"#$ !!!"# ) !. ZPS. (convergence distance, Zc) Zhang et !!"#$. !"#. 28.

(39) 8-bit. 0. 4.3. !!"#$. 255 (normalized). 0. !"#. 255. 4.2. 0, near. 255, far. 4.3. 8-bit. (bilateral filter, BF) [Tom98] Bilateral Filter BF !. !! = !. !"# !!!. !. !! =. !∈! !!!. ∥ ! − ! ∥ !!! |!! − !! | !!. ∥ ! − ! ∥ !!! |!! − !! |. !!. !!. !. !!. !. !!!. Gaussian filter Gaussian filter ! !, ! =. Gaussian filter. 1 ! ! !! + ! ! / 2! ! 2!! ! 29. !!!.

(40) !. 4.3. Bilateral filter smooth. 4.4. 4.4. DIBR. Bilateral filter smooth. 3-D image warping. (!). !. 4.5. !. (baseline). !. (focal length). !!. !!. !!. !! center) !!. !!. (optical. !!. !. !. !! , !. 4.5. image warping. (re-projecting). !! , !. !! = !! +. virtual images. 30. !! ! ! !. !! , ! !!! = !! −. !! ! ! !.

(41) !. !. !!. !!. !!. ! !!. !!. !! ! !. !!. !! ! !. 4.5 DIBR. image warping. virtual images (holes),. newly exposed (b) newly exposed. 4.6(a)(b)(c) warping. (c). (a). (b) Bilateral filter smooth. warping virtual image. 31.

(42) (a). (b). (c) 4.6. (a). (b) Bilateral filter smooth. warping warping. 32. (c).

(43) image warping. pixel pixels. newly exposed. image warping. image warping. ;. bilateral filter smooth. virtual images. Telea [Tel04]. 4.7. Fast Marching Method(FMM). !:. !":. !". ! !(!). !: !. !(!). !. pixels. known neighborhood !(!) of !. known area. !! !. boundary !". region ! to be inpainted. 4.7 !. !(!). !. !. !(!). ! !! ! = ! ! + ∇!(!)(! − !) !(!) 33. ∇!(!).

(44) ! ! =. !∈!! (!) !(!, !) !!. ! = ! ! + ∇!(!)(! − !) !∈!! (!) !(!, !). !(!, !). !(!) ! !, ! = !"#(!, !) ∙ !"#(!, !) ∙ !"#(!, !). directional component: !"# !, ! FMM. ! = ∇!(. ; geometric distance component: !"# !, !. ). !. ; level set distance component: !"# !, !. !. : !"# !, ! =. !−! ∙! ! !−!. !!! !"# !, ! = !−! !"# !, ! = ! !. ! ! ! =. !. !! 1 + ! ! − !(!). !! !∈!! (!) !(!,!) !!. !. !!. ! !! ! !∇!(!)(!!!). !∈!! (!) !(!,!). !. 34. :1.

(45) ! (5.1 3D. (5.3. ). (5.2. ). ). Microsoft Kinect. RGB-D. LG. Kinect. 3D. 3D. 3. RGB CMOS. 3D. Kinect. (diffuser). (laser speckle). LG. 3D. 3D Polarizing glasses 3D 2D. 3D. 5.1. 5.1. 2D. 35. 3D. 2D.

(46) Matrix Completion via Truncated Nuclear Norm Regularization Method A. Method B. Cai et al. [Cai08]. SVT [Tel04]. [Ber01] 1. Mean squared. !. error(MSE) =!!". !!! !!!. !!! !!!. ! !, ! − ! !, !. !. : MSE ; 2. Peak signal to. noise ratio(PSNR)=10 ∙ log!". !"#!! !"#. = 20 ∙ log!". !"#! !"#. :MSE. db. PSNR ; 3. Structure similarity(SSIM) =! ! !, !. !. ! !, !. !. ! !, !. !. !>0. !>0 !>0. !! ! !!. !! ! !!. ! ! ! ! !, ! = !! !! ! !!. ! !, !. !. !!" !!!. !. !! !! !!!. ! !, !. !!. !. ! !, !. ! ! ! ! !, ! = !! !! ! !!. !. !. !! !! !!. :SSIM. !. !. ! !, ! (c). (l). (s). 100% PSNR Ground Truth 5.2(a). 5.2. Ground Truth 5.2(b) 5.2(c)~(h) 5.1. 36. 5.2. ! !, ! !! !, ! = ! !, !.

(47) (a). (b). (c)Matrix Completion via TNNR –. (d)Matrix Completion via TNNR –. Method A. Method B. (e)Image Segmentation. (f)[Cai08]SVT. (g)[Ber01]. (h)[Tel04]. 5.2 (a) (b) (c) Matrix Completion via TNNR –Method A (d) Matrix Completion via TNNR –Method B (e)Image Segmentation (f)[Cai08]SVT (g)[Ber01] (h)[Tel04]. 37.

(48) MSE. SSIM index. Execution time. 2.40302 44.3232db. 99.02%. Below1min. 1.47114 46.4543db. 99.38%. 21min. 1.69269 45.845db. 99.41%. 17min. SVT. 2.83728 43.6018db. 98.23%. 27min. [Tel04]. 2.70559 43.8082db. 98.51%. Below 1min. [Ber01]. 2.74679 43.7426db. 98.51%. Below 1min. Image Segmentation. PSNR. (Ours proposed method) Matrix Completion via TNNR – ADMM Matrix Completion via TNNR – APGL. 5.1! !. !. Matrix completion. ; SVT TNNR [Tel04]. ;. [Ber01] Matrix. Completion via TNNR - ADMM. APGL. Kinect APGL. APGL Real-time SSIM. Image Segmentation. Matrix Completion via TNNR –APGL Based on Image Segmentation Repairing [Tel04]. [Ber01]. 38. Kinect.

(49) Matrix Completion via TNNR - Method B. Image Segmentation 3D. 2D. 3D. 3D. :. ;. 50. 1. 3D 3D. 3D 5.2. ;. 2. 3D. 5.3 ; 3.. ?(. / ) 5.4. 4.. ( 5.5. 39. / ):. ;.

(50) 3D !. 45!. 42!. 39!. 40! 35!. 32!. 30!. 26!. 25!. 20!. 20!. 20!. 15! 10! 5!. 0!. 0!. 0!. 0!. 0!. Matrix!CompleDon!via! TNNR!-!APGL!. 0!. 4!. 3!. 0!. Image!SementaDon!. 0!. LG. 5.2! ! 3D. !. 3D !. 35! 30!. 29!. 30! 25!. 28!. 21!. 20!. 18!. 18!. 15!. 13!. 10! 5! 0!. 10!. 8!. 0!. 2!. Matrix!CompleDon!via! TNNR!-!Method!B!. 0!. 2!. 0!. Image!SementaDon! 5.3! ! 3D. LG !. 40. 1!.

(51) Matrix Completion- APGL. Image Segmentation. LG. 51. 46. 8. 11. 16. 54. 5.4! !. !. Matrix Completion- APGL. Image Segmentation. LG. 7. 25. 22. 55. 37. 40. 5.5! !. !. Kinect. 3D 3D. Matrix Completion- APGL. 41.

(52) ! 3D 3D. 3D 3D. 3D. 3D. 3D RGB+D. 3D. 3D. ; rank. 2D 42. 3D.

(53) 2D. 3D. Kinect. Matrix completion. Tensor. completion 3D 3D. 3DS 3D. Kinect 3DS. Kinect. 43.

(54) !. [Ide08] I. Ideses, L. Yaroslavsky, and B. Fishbain, “Depth Map Manipulation for 3D Visualization,” 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video, pp.337-340, 2008. [Bat04] S. Battiato, S. Curti, M. La Cascia, E. Scordato, and M. Tortora, “Depth Map Generation By Image Classification,” Proc. of SPIE IS&T/SPIE's 16th Annual Symposium on Electronic Imaging, pp. 95-104, 2004. [Com97] D. Comaniciu, and P. Meer, “Robust Analysis of Feature Spaces: Color Image Segmentation,” Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 750-755, June 1997. [Ang10] L. J. Angot, W. J. Huang and K. C. Liu, “A 2D to 3D video and image conversion technique based on a bilateral filter,” Proc. of SPIE-IS&T Electronic Imaging, Vol. 7526, 2010. [Che10] C. C. Cheng, C. T. Li, and L. G. Chen, “ A 2D-to-3D conversion system using edge information,” Int’l Conf. on Consumer Electronics (ICCE), pp.377-378, 2010. [Li06] P. Li, and R. K. Gunnewiek, “On Creating Depth Maps from Monoscopic Video using Structure from Motion,” Proc. of IEEE Workshop on Content Generation and Coding for 3D-television, pp.508-515, 2006. [Har02] P. Harman, J. Flack, S. Fox and M. Dowley, “Rapid 2D to 3D Conversion,” Proc. SPIE, Vol. 4660, pp.78-86, 2002. [Rob88] L. S. Robert, (1994), “Cognition and the Visual Arts, ” Cambridge, MA, The MIT Pres. [Cam12] M. Camplani and L. Salgado, “Efficient spatio-temporal hole filling strategy for Kinect depth maps,” Proc. SPIE, Vol. 8290, 2012. [Xu12] K. Xu, J. Zhou and Z. Wang, “A method of hole-filling for the depth map generated by Kinect with moving objects detection,” IEEE on Broadband Multimedia Systems and Broadcasting (BMSB), pp.1-5, 2012. [Cri04] A. Criminisi, P. Perez, and K. Toyama, Region filling and object removal by exemplar-based image inpainting,. IEEE Transactions on Image Processing, pp. 1200. - 1212, 2004 . 44.

(55) [Feh04] C. Fehn, “Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV,” Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems, 2004. [Zin10] S. Zinger, L. Do and P. H. N. de With, “Free-viewpoint depth image based rendering,” Journal of Visual Communication and Image Representation, Vol. 21, pp.533-541, 2010 [Lai13] Y. K. Lai, Y. F. Lai, and Y. C. Chen “An Effective Hybrid Depth-Generation Algorithm For 2d-To-3d Conversion In 3d Displays,” Journal of Display Technology, vol.9, pp. 154-161, 2013. [Ach12] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua and S. Süsstrunk, S. “ SLIC Superpixels Compared to State-of-the-Art Superpixel Methods,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, Issue 11, pp.2274-2282, 2012. [Kom06] N. Komodakis and G. Tziritas, “Image Completion Using Global Optimization,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2006. [Ras05] C. Rasmussen and T. Korah, “Spatiotemporal Inpainting for Recovering Texture Maps of Partially Occluded Building Facades,” Proc. IEEE Int’l Conf. Image Processing, 2005. [Ji10] H. Ji, C. Liu, Z. Shen, and Y. Xu, “Robust Video Denoising Using Low Rank Matrix Completion,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010. [Lee10] J. A. Lee and M. Verleysen “Unsupervised dimensionality reduction: Overview and recent advances,” The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2010. [Can08] E. J. Candes and B. Recht, “Exact low-rank matrix completion via convex optimization,” Annual Allerton Conference on Communication, Control, and Computing, pp. 806-812, 2008 [Zha05] L. Zhang and W. J. Tam, “Stereoscopic Image Generation Based on Depth Images for 3DTV,” IEEE Trans. Broadcast, vol. 51, pp. 191–199, 2005. [Tom98] C. Tomasi and R. Manduchi, “Bilateral Filtering for Gray and Color Images,” Proc. IEEE Int’l Conf. on Computer Vision, vol. 51, pp. 191–199, 2005 [Cai10] J. F. Cai, E. J. Cande`s, and Z. Shen, “ (ASi),” SIAM J. Optimization, vol. 20, 45.

(56) pp. 1956-1982, 2010. [Hu13] Y. Hu, D. Zhang, J. Ye, X. Li, and X. He, “Fast and Accurate Matrix Completion via Truncated Nuclear Norm Regularization,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 35, No. 9, Sept. 2013. [Rec10] B. Recht, M. Fazel, and P.A. Parrilo, “Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization,” SIAM Rev., Vol. 52, No. 3, pp. 471-501, 2010. [Toh10] K. C. Toh and S. Yun, “An Accelerated Proximal GradientAlgorithm for Nuclear Norm Regularized Least Squares Problems,” Pacific J. Optimization, pp. 615-640, 2010. [Wri09] J. Wright, A. Ganesh, S. Rao, Y. Peng, and Y. Ma, “RobustPrincipal Component Analysis: Exact Recovery of Corrupted Low-Rank Matrices via Convex Optimization,” Proc. Advances in Neural Information Processing Systems, 2009. [Tel04] A. Telea, “An Image Inpainting Technique Based on the Fast Marching Method,” Graphics, GPU, & Game Tools, 2004. [Ber01] B. Bertlmio, A. L. Bertozzim, and G. Sapiro, “Navier-stokes, fluid dynamics,and image and video inpainting,” Computer Vision and Pattern Recognition, Vol. 1, pp.355-362, 2001. IPPR. IPPR. 46.

(57)