A Total Variation and Group Sparsity Based Tensor Optimization Model for Video Rain Streak Removal

(1)

A Total Variation and Group Sparsity Based Tensor Optimization Model for Video Rain Streak Removal

Ye-Tao Wang, Xi-Le Zhao^∗, Tai-Xiang Jiang^∗, Liang-Jian Deng, Tian-Hui Ma, Yue-Tian Zhang, Ting-Zhu Huang

School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu, Sichuan, 611731, P. R. China

Abstract

Rain streak removal is an important issue of the outdoor vision system and has been investigated extensively. In this paper, we propose a novel tensor optimization model for video rain streak removal by fully considering the discriminatively intrinsic characteristics of rain streaks and clean videos. In specific, rain streaks are group sparse and smooth along the rain streaks’ direction; the clean videos are smooth along the perpendicular direction of rain streaks and the time direction.

For rain streaks, we use thel_2,1 norm to enhance the group sparsity and theUni- directional Total Variation(UTV) to promote the smoothness along rain streaks’

direction. For clean videos, we use two UTV to enhance the smoothness along the perpendicular direction of rain streaks and the time direction. We develop an efficient alternating direction method of multipliers(ADMM) algorithm to solve the proposed model. Experiments on synthetic and real data demonstrate the superiority of the proposed method over state-of-the-art methods in terms of both quantitative and qualitative assessments.

Keywords: video rain streak removal, group sparsity, unidirectional total

variation, tensor optimization model, alternating direction method of multipliers.

1. INTRODUCTION

Bad weather impairs visibility of an image and introduces undesirable inter- ference that can severely hinder the follow-up processing (e.g., object detection,

∗Corresponding author

Email address:xlzhao122003@163.com(Xi-Le Zhao)

(2)

recognition, and tracking [1, 2, 3, 4, 5]). This paper mainly focuses on the rain streak removal problem [6,7,8,9,10,11].

The degradation of rainy images is generally modeled as the sum of the unknown clean images and the rain streaks. A single rainy image is generally modeled asO =B+R[7, 12,13], whereO ∈ R^m×n, B ∈ R^m×n, andR ∈ R^m×n are the observed rainy image, the unknown clean image, and the rain streaks, respectively. This model can be extended to the video case: O =B+R, whereO, B, andR ∈ R^m×n×tare the observed rainy video, the unknown clean video, and the rain streaks, respectively. The goal of rain streak removal is to estimate the clean images from its rainy version. This typical inverse problem is often solved by regularization methods which are based on additional prior knowledge.

Existing rain streak removal algorithms can be categorized into two classes:

the single image rain streak removal algorithms and the video rain streak removal algorithms. For the single image rain streak removal, Kang et al. [7] decomposed a rainy image into low-frequency (LF) and high-frequency (HF) components using a bilateral filter and then performed morphological component analysis(MCA)- based dictionary learning and sparse coding to separate the rain streaks in the HF component. However, learning HF image bases typically results in a loss of de- tailed image information. To alleviate this problem, Sun et al. [14] exploited the structural similarity of the derived HF image bases. Nevertheless, the background- s estimated using their method still tend to be blurry. Chen et al. [12] considered the pattern of the rain streaks and the smoothness of the background, but the con- straints in their objective function were not sufficiently strong. Discriminative sparse coding was adopted by Luo et al.[8]. Their method preserves the clean content well but is not able to remove most of the rain streaks. The recent work by Li et al. [13] was the first to utilize Gaussian mixture model (GMM) patch priors for rain streak removal, with the ability to account for rain streaks of different orientations and scales. Nonetheless, their method tends to yield over-smooth clean images; i.e., the details of the clean image content are not preserved well. To cope with this issue, Zhu et al. [15] proposed a joint bi-layer optimization method progressively separate rain streaks from background details, in which the gradient statistics are analyzed. In [16], the directional property of rain streaks received attentions. The recently developed deep learning technique is also applied to the single image rain streak removal task [17,18].

For the video rain streak removal, Garg et al. [19] firstly raised a video rain streak removal method with comprehensive analysis of the visual effects of rain streaks on an imaging system. Since then, multiple methods have been proposed for the video rain streak removal and attained good rain removing performance in

(3)

videos with different rain circumstances. Tripathi et al. [20] took the spatiotemporal properties into consideration. In [12], the similarity and repeatability of rain streaks were utilized, and a generalized low-rank appearance model was proposed.

Additionally, comprehensive early existing video-based methods are reviewed in [21]. Kim et al. [6] considered the temporal correlation of rain streaks and the low-rank nature of clean videos, but the effectiveness of their method is still low for certain dynamic videos recorded by dynamic cameras. Very recently, the rain streaks were stochastically modeled as a mixture of Gaussians in [22]. In [23], a novel tensor-based video rain streak removal approach was proposed, with considering numerous discriminative prior information.

(a) (b)

Figure 1: (a) The rain streaks, (b) A random sparse image.

In [23], Jiang et at. proposed the model as

minB,R α₁k∇_xRk₁+α₂kRk₁+α₃k∇_yBk₁+α₄k∇_tBk₁+α₅kBk∗, s.t. O =B+R, B,R>0,

(1) where∇_x,∇_y, and∇_tare the derivative operators along rain streaks direction, the perpendicular direction of rain streaks, and time direction, respectively. For simplicity, we assume that the rain streaks direction and the perpendicular direction of rain streaks are the vertical direction and the horizontal direction, respectively.

However, model1 has two drawbacks. First, the rain streaks are not only s- parse and but also group sparse; see Figure 1. Second, the clean video does not exhibit obvious low-rankness; see Figure2. Hence, there is room for improvemen- t. Based on the above observations, we introduce the group sparsity regularizer

(4)

the singular values of the unfolding matrix along vertical

direction the singular values

of the unfolding matrix along horizontal direction the singular values

of the unfolding matrix along time

direction

Figure 2: From left to right: the singular values of unfolding matrices of the rainy video, the clean video, and the rain streaks.

for rain streaks and disuse the low-rankness regularizer for the clean video. The novel tensor optimization model consists of the group sparsity regularizer and the Unidirectional Total Variation(UTV) regularizer along vertical direction for rain streaks and the UTV regularizers along horizontal direction and time direction for clean videos. We build model as

arg min

B,R

α₁kRk_2,1+α₂k∇_xRk₁+α₃k∇_yBk₁+α₄k∇_tBk₁, s.t. O =B+R, B,R ≥ 0.

(2) To solve the proposed model, we develop an efficient ADMM [24, 25, 26, 27]

algorithm. Experimental results demonstrate the superior of the proposed method qualitatively and visually.

(5)

The paper is organized as follows. In Sec. 2, some notations and the basic knowledge are introduced. In Sec.3, the proposed model and proposed algorithm are presented. Experimental results are reported in Sec. 4. Finally, we draw some conclusions in Sec. 5.

2. TENSOR BASICS

Following [23, 28, 29], we use lower case letters (e.g., x) for scalars, bold lower case letters (e.g., x) for vectors, bold upper case letters (e.g., X) for matrixes, and bold upper calligraphic letters (e.g., X) for tensors. An n-mode tensor is denoted as X ∈ R^I¹^×I²^×...×Iⁿ. Its elements are denoted asx_i₁_,...,i_n, where 1 ≤ ik ≤ Ik and 1 ≤ k ≤ n. The inner product of two same-size tensors is defined as

hX,Yi = X

i1,i2...in

x_i₁_,i₂_...i_n×y_i₁_,i₂_...i_n. (3) Based on (3), the Frobenius norm of a tensor is defined as

kX kF :=hX,X i¹² = ( X

i1,i2...in

|xi1,i2...in|²)¹². (4) For an n-mode tensor, we define the derivative along the k-th direction of X as

∇_kX ∈ R^I¹^×I²^×...×Iⁿ in the cyclic boundary condition, where the elements of

∇_kX obey that

(∇_kX)_i₁_,i₂_...i_k_...i_n =x_i₁_,i₂_...i_k_...i_n −x_i₁_,i₂_...(i_k_−1)...i_n.

Wheni_k= 1, thei_k−1will beI_k. The “unfold” operation along thek-th direction on a tensorX is defined as

unfold_k(X) =X_(k) ∈R^I^k^×(I¹^...I^k−1^I^k+1^...Iⁿ⁾. (5) The projection operator “fold” is defined as

foldk(X_(k)) = X. (6)

Based on the unfolding rule (5) and folding rule (6), the tensor and the matrix can be transformed to each other. It is easy to obtain that, for any1≤k ≤n,

kX k_F =kX_(k)k_F, hX,Yi =hX_(k),Y_(k)i,

(6)

and

∇_kX =fold_k(∇₁unfold_k(X)).

Supposex ∈ Rⁿ is a group sparse vector. Let{x_g_i ∈ Rⁿⁱ : i = 1, ..., s}be the grouping of x, where g_i ⊆ {1,2, ..., n} is an index set corresponding to the i-th group, andx_g_i denotes the subvector ofxindexed byg_i [30]. Generally,g_i’s can be any index sets, and they are predefined based on prior knowledge. Thel_2,1 norm is defined as follows:

kxk_2,1 =

s

X

i=1

kx_g_ik₂.

l_2,1 norm is known to facilitate group sparsity [30]. For the matrix, each column is considered as a group. Thusl_2,1 norm for a matrix is usually denoted as

kXk_2,1 =

s

X

i=1

kx_g_ik₂.

Here, g_i’s are the column index set. Since one column is treated as a group, we can extendl_2,1norm from the matrix to the tensor as

kX k_2,1 =kunfold₁(X)k_2,1.

More extensive overview of group sparsity can be found in [30].

3. THE PROPOSED METHOD

This section gives the proposed model and the algorithm for rain streak removal.

3.1. Proposed model

Without loss of generality, we useO, B, andR to represent the rainy video, the target clean video, and the rain streaks, respectively. We recall the proposed model:

arg min

B,R

α₁kRk_2,1+α₂k∇_xRk₁+α₃k∇_yBk₁+α₄k∇_tBk₁, s.t. O =B+R, B,R ≥ 0,

(7) where∇x,∇y, and∇tare the derivative operators along the vertical direction, the horizontal direction, and the time direction, respectively. In what followings, we will explain all components in our model in details.

(7)

Group sparsity of the rain streaks: The rain component is sparser than the clean video, and the rain component exhibits line pattern structure rather than being randomly distributed just like Figure1. Therefore, we use the termkRk2,1

to characterize the group sparse which can simultaneously enhance the sparsity and preserve the line pattern. It is superior over the sparsity itself used in [23].

(a) (b)

Figure 3: (a) The histogram of the absolute values of the derivatives along the vertical direction of the rain streaks. (b) The histogram of the absolute values of the derivatives along the vertical direction of the clean video.

The smoothness along the rain streak direction of the rain streaks: The rain streaks share similar directions. When the angle between the direction of rain streaks and the vertical direction is small, the derivatives of rain streaks and the clean video along the vertical direction are different, i.e., the derivatives along the vertical direction of rain streaks are more sparse as compared with those of the clean video; see Figure3. Therefore, we use thel₁ norm of∇_xR to enhance the smoothness along the vertical direction of the rain streaks.

The smoothness along the horizontal direction of the clean video: Natural images are piecewise smooth, which indicates that the derivatives of frames in a video are not dense along vertical and horizontal directions. The vertical rain streaks destroy the smoothness along the horizontal direction. Compared with the rain streaks, the derivatives of the clean video are sparse along the horizontal direction. As a result, the derivatives along the horizontal direction of rain streaks are dense, which is shown in Figure4. Therefore, we use thel₁ norm of∇_yBto enhance the smoothness along the horizontal direction of the clean video.

The smoothness along the time direction of the clean video: Since that a video maintains at least 25 frames per second, there is a strong smoothness along time direction. The derivatives of the clean video are sparse along the time

(8)

(a) (b)

Figure 4: (a) The histogram of the absolute values of the derivatives along the horizontal direction of the rain streaks. (b) The histogram of the absolute values of the derivatives along the horizontal direction of the clean video.

direction. However, the rain streaks are not smooth. Because of its high velocity, its smoothness is broken. As displayed in Figure5, the derivatives along the time direction of the clean video are sparse while those of the rain streaks Therefore, we use thel₁norm of∇_tBto enhance the smoothness along the time direction of the clean video.

Discussion of low-rankness: Meanwhile, we discard the low-rankness regularizer which is considered in [23]. The clean video is low-rank only when it is static, but not the case even if there is only a light object moving in the clean video. Usually the low-rankness regularizer will be slacked to the singular values of three unfolding matrixes of the video in quantitative analysis. From thesingu- lar value decomposition(SVD) [31] of rain streaks and clean video in Figure2, it can be found the singular value of clean video does not have zero elements in any directions, and the singular values of rain streaks are smaller than those of clean video.

3.2. Proposed algorithm

The proposed model (7) is a convex optimization problem which can be solved by various of convex optimization algorithms. We adopt the ADMM, an effective strategy for solving large scale optimization problems, to solve it. After introduc- ing four auxiliary tensors Y, S, X, and T ∈ R^m×n×t, we rewrite the proposed

(9)

(a) (b)

Figure 5: (a) The histogram of the absolute values of the derivatives along the time direction of the rain streaks. (b) The histogram of the absolute values of the derivatives along the time direction of the clean video.

model (7) as the following equivalent constrained problem:

arg min

R,Y,S,X,T

α1kYk2,1 +α2kSk1+α3kX k1+α4kT k1, s.t. Y =R,

S =∇_xR,

X =∇_y(O − R), T =∇_t(O − R), O >R>0.

(8)

Then the augmented Lagrangian function of (8) is:

L_β(R,Y,S,X,T,Λ) =α₁kYk_2,1+α₂kSk₁+α₃kX k₁+α₄kT k₁ +hΛ₁,Y − Ri+β₁

2 kY − Rk²_F +hΛ₂,S − ∇_xRi+ β2

2 kS − ∇_xRk²_F +hΛ3,X − ∇y(O − R)i+β₃

2 kX − ∇y(O − R)k²_F +hΛ₄,T − ∇_t(O − R)i+ β₄

2 kT − ∇_t(O − R)k²_F, (9)

(10)

where Λ = [Λ₁,Λ₂,Λ₃,Λ₄] are Lagrange multipliers and β = [β₁, β₂, β₃, β₄] are four positive penalty parameters. This joint minimization problem can be decomposed into five subproblems which can be easily solved. By separating the variables of (9) into two groups: Rand (Y, S, X,T), (9) fits the framework of ADMM. It requests us to solve variables of each group by keeping another group fixed. The solution of the five subproblems will be introduced in the following.

Y sub-problem: With other variables fixed, theY sub-problem is arg min

Y

α₁kYk_2,1+ β₁

2kY − R+Λ₁

β₁k²_F, (10) which has a closed-form solution by the soft-shrinkage formula [30], thusYcould be updated as

Y_g^t+1

i = max

kQ_g_ik₂− α₁ β₁,0

Q_g_i

kQ_g_ik₂,Q_g_i =R^t_g

i − (Λ₁^t)_g_i

β₁ , (11) whereQ_g_i denotes thei-th group of the video.

S, X, and T sub-problems: With other variables fixed, S, X, and T subproblems are

arg min

S

α2kSk1+β2

2 kS − ∇xR+ Λ2

β₂ k²_F arg min

X

α3kX k1+β3

2kX − ∇y(O − R) + Λ3

β₃ k²_F arg min

T

α4kT k1+β4

2 kT − ∇t(O − R) + Λ4

β₄ k²_F,

(12)

which have closed-form solutions by soft-thresholding , thusS,X, andT could be updated as

S^(t+1) =Shrink^α2 β2

∇_xR^(t)−Λ^(t)₂ β₂

!

, (13)

X^(t+1) =Shrink^α³

β3

∇_y(O − R^(t))−Λ^(t)₃ β₃

!

, (14)

T^(t+1) =Shrink^α⁴

β4

∇_t(O − R^(t))− Λ^(t)₄ β₄

!

. (15)

(11)

R-subproblem: TheRsub-problem is a least squares problem:

arg min

R

β₁

2 kY − R+ Λ₁

β₁k²_F + β₂

2 kS − ∇_xR+Λ₂ β₂k²_F + β₃

2 kX − ∇_y(O − R) + Λ₃

β₃k²_F +β₄

2 kT − ∇_t(O − R) + Λ₄ β₄k²_F. With the problem is transformed to

(β₁I+β₂∇^T_x∇_x−β₃∇^T_y∇_y−β₄∇^T_t∇_t)R=

β₁Y^(t+1)+Λ^(t)₁ +∇^T_x(β₂S^(t+1)+Λ^(t)₂ ) +∇^T_y(β₃X^(t+1)−β₃∇_xO+Λ^(t)₃ ) +∇^T_t(β₄T^(t+1)−β₄∇_tO^(t+1)+Λ^(t)₄ ).

The solution has the following closed-form solution:

R^(t+1) =F⁻¹

F(K1) F(K₂)

, (16)

where F andF⁻¹ denote the fast Fourier transform (FFT) and its inverse transform, respectively. Here

K₁ =β₁Y^(t+1)+Λ^(t)₁ +∇^T_x(β₂S^(t+1)+Λ^(t)₂ ) +∇^T_y(β₃X^(t+1)

−β₃∇_xO+Λ^(t)₃ ) +∇^T_t(β₄T^(t+1)−β₄∇_tO^(t+1)+Λ^(t)₄ ) and

K2 =β1I+β2∇^T_x∇x−β3∇^T_y∇y −β4∇^T_t∇t.

Multipliers updating: Finally, following the framework of the ADMM, the Lagrange multipliersΛ= [Λ₁,Λ₂,Λ₃,Λ₄]are updated as:











Λ^(t+1)₁ =Λ^(t)₁ +β₁(Y^(t+1)− R^(t+1)), Λ^(t+1)₂ =Λ^(t)₂ +β2(S^(t+1)− ∇xR^(t+1)),

Λ^(t+1)₃ =Λ^(t)₃ +β₃(X^(t+1)− ∇_y(O − R^(t+1))), Λ^(t+1)₄ =Λ^(t)₄ +β₄(T^(t+1)− ∇_t(O − R^(t+1))).

(17)

The proposed algorithm is summarized in Algorithm1. Since the proposed model is convex, the convergence of the proposed algorithm is theoretically guar- anteed under the ADMM framework [32].

(12)

Algorithm 1Algorithm for video rain streak removal Input: The rainy videoO;

1: Initialization:B⁽⁰⁾ =O,R⁽⁰⁾ = zeros(m×n×t);

2: whilenot convergeddo

3: UpdateY via (11);

4: UpdateS via (13),X via (14), andT via (15);

5: UpdateRvia (16);

6: Update the multipliers via (17);

7: end while

Output: The estimation of rain streaksRand the clean videoB=O − R.

4. EXPERIMENTAL RESULTS

Preprocessing: The color video is a four-mode tensor of sizem×n×3×t.

We convert videos from the RGB color space to YUV color space and only con- duct the method on the Y channel. Thus the videos that we process become a three-mode tensor of sizem×n×t. To reduce the boundary effect, we pad the input tensors O ∈R^m×n×tby 5-pixel-width under reflective boundary condition.

Thus the size of the input tensors becomes(m+10)×(n+10)×(t+10). To validate the effectiveness of the proposed method, we compare the proposed method with two state-of-the-art methods: rain streak removal using temporal correlation and low-rank matrix completion (LRMC) [6] and rain streak removal using discriminatively intrinsic priors (DIP) [23]. Readers can find the Matlab code (p-code) to test the performance of our methodthere.

4.1. Synthetic data

For synthetic data, since the clean videos are available, thepeak signal to noise ratio (PSNR) and structure similarity (SSIM) [33] are selected to measure the performance of methods. Six videos named as “carphone”, “container”, “coastguard”, “bridgefar”, “highway” and “foreman”¹are selected as our test datasets.

These videos can be viewed as four-mode tensors of size144×176×3×150.

Rainy videos generation: The rainy videos are generated by the following steps. (1) The salt and pepper noise is added to a zero tensor with the same size as the clean video tensor. (2) The noise tensor is blurred by Gaussian blur. (3) The blurred and noisy tensor is further blurred by motion blur. There exists 5-15

1http://trace.eas.asu.edu/yuv/.

(13)

degrees between motion direction and vertical direction. (4) Finally, the blurred and noisy tensor is directly added to the clean videos, and the intensity values greater than 1 are set as 1.

Parameters setting: The parameters{β₁, β₂, β₃, β₄}are set as 50, and other parameters {α₁, α₂, α₃, α₄} are selected from {0.1, 0.3, 1, 3, 10, 30, 100, 300, 1000}. The stopping criterion is that the relative error of rain streaks is less than 5×10⁻³ or the iteration number is larger than 250.

Performance comparisons: We can observe from Table1, that the proposed method significantly outperforms the companying methods in terms of PSNR values and SSIM values. For light and heavy rain, the proposed method achieves the highest PSNR and SSIM values except the last video for light rain streaks. In average, the PSNR values of the proposed method are 8.016 dB and 2.966 dB higher than those of LRMC and DIP for heavy rain streaks. In average, the PSNR values of the proposed method are 7.292 dB and 0.330 dB higher than those of LRMC and DIP for light rain streaks.

Moreover, the frames of estimated videos are displayed in Figures6and7for visual inspection. As observed, the proposed method achieves significantly better visual quality than the compared methods in rain streak removal, visibility en- hancement, and detail preservation. There are two main reasons. The first reason is that LRMC and DIP both assume the clean video is low-rank, which leads to that some obvious details are lost. However, we disuse the low-rankness of the clean video, which preserves the details in dynamic clean video. For example, DIP and LRMC remove the street lights in “highway” for both heavy rain streaks and light rain streaks . In “bridgefar”, although the clean video is almost static, some small objects such as water pattern destroy the low-rankness. Thus, the details of water pattern are lost in the results of DIP and LRMC. Another reason is that we use the group sparsity to characterize rain streaks, which helps to preserve the line pattern and keep the continuity of the rain streaks, leading to more accu- rate rain streak removal results than other methods. In comparison, DIP does not extract sufficient rain streaks and does not preserve the continuity of rain streaks, e.g., “coastguard” and “foreman” for heavy rain streaks and “carphone” for light rain streaks. Since the continuity is more significant for heavy rain streaks, the proposed method equipped with group sparsity term outperforms the companying methods for heavy rain streaks.

Discussion of each term: We investigate the role of each term in our model (7) by changing one parameter while fixing the others. Figure 8 shows the PSNR curves of the proposed method using different parameter settings, where the testing parameter is chosen from the geometric series {0.1,0.121, ...,0.1×

(14)

Figure 6: Rain streak removal results by different methods. From left to right: the rainy frames, the results by LRMC [6], DIP [23], the proposed method, and the ground truth. From top to bottom:

the “carphone”, “container”, “coastguard”, “highway” “bridgefar” and “foreman” videos with the heavy synthetic rain streaks, respectively.

(15)

the “carphone”, “container”, “coastguard”, “highway” “bridgefar” and “foreman” videos with the light synthetic rain streaks, respectively.

(16)

Figure 8: The PSNR values of the proposed method using different parameter settings.

1.1^k, ...,1000}. It could be found that each parameter has an important contribu- tion to the performance of the proposed method.

Discussion of groups: The group size is an vital important parameter which is set as one column in this paper unless otherwise specified. And it is very in- teresting to investigate the influence on the performance the proposed model with different group sizes. Table2shows the PSNR and SSIM values of the proposed model using different group sizes. From Table 2, we can observe that the group size has an impact on the performance of the proposed model. More specially, heavy videos favor large group sizes while light videos favor small group sizes.

For simplicity, we choose one column as default in all experiments because there is no significant difference between different group sizes.

Discussions of the oblique rain streaks: Generally, the rain drops are falling from top to bottom and the rain streaks are close to being vertical. As we exhibited above, our method is robust to a small range of the angles since the rain streaks in

(17)

the synthetic data are not strictly vertical. However, the assumption is not always established (the angle between the direction of the rain streaks and the vertical direction would be very large). The proposed model consists of 4 regularization terms, which simultaneously contribute to the rain streak removal. When the rain streaks are oblique, the one regularizer corresponding to the directional property and the group sparsity of the rain streaks would not be helpful. Nonetheless, the temporal and the horizontal continuity of the background still exist. Thus, tuning the parameters to enlarge the effects of these two regularizers would help the proposed method to remove the rain streaks. Figures 9 and Table 3 show the results on two synthetic videos the “highway2”(35-55 degrees between the direction of the rain streaks and the vertical direction) and the “waterfall” (15-35 degrees between the direction of the rain streaks and the vertical direction) with oblique rain streaks. It can be found that when the rain streaks are not vertical, our method still works and achieves promising performances.

the “highway2”and “waterfall” videos, respectively.

Discussions of the preprocessing: Before applying our algorithm, there are two preprocessing steps, i.e., (a) the conversion from RGB space to YUV space, and (b) adding reflective boundary condition. We would like to illustrate the influence of the two preprocessing steps using the video, “carphone”, with heavy rain streaks and light rain streaks. Table 4shows the quantitative effects from these two preprocessing steps. It can be found from Table4that our algorithm generated comparative results with and without the conversion from RGB space to YUV space. This conversion would largely reduce the running time and hardly affect the performance. Meanwhile, as we expected, the reflective boundary condition slightly improved the performance. The method in [6] is designed for the RGB

(18)

videos so that we fed the RGB videos to it. The algorithm in [23] is also a tensor based method and involves the fast calculation using Fourier transform. For fair comparison, we did the same preprocessing steps when running the algorithm in [23]. It can be found that without the two preprocessing steps, the proposed method still work best.

4.2. Real data

We test two real rainy videos. One is a clipped part of size260×440×3×128 from the movie “the Matrix”, and the other one is a backyard video of size512× 256×3×128 recorded in a rainy day. It is worth mentioning that the proposed method is not sensitive to parameters. The parameters for real data are the same as those in the first synthetic experiments.

Performance comparisons: For the first real video, we compare all the methods on one extreme cases. The first video is a very challenge video under lightning which enlarges the difference between adjacent frames and breaks the the continuity along time direction. The rain streak removal results are displayed in Figure 10. And we can observe from Figure10that the rain streaks are more effectively removed by the proposed methods as compared with the other methods.

For the second real video, the rain streak removal results are displayed in Figure11. We observe from Figure11that due to the clean video is static, which makes low-rankness a good video description, DIP performs well for this video.

In spite of this, the rain streaks are more effectively removed by the proposed methods as compared with the other methods.

5. CONCLUSIONS

In this paper, we propose a tensor-based rain streak removal model. We use the group sparsity and the smoothness along the vertical direction to characterize rain streaks, and use the smoothness along the horizontal direction of rain streaks and the time direction to characterize the clean video. Meanwhile we discuss low-rankness. We develop an efficient ADMM algorithm to solve the proposed model. The experiments on synthetic and real data demonstrate the superiority of the proposed method over state-of-the-art method in terms of both quantitative and qualitative assessments. We will explore the group sparsity of the derivatives in the vertical direction of the rain streaks in our further work.

(19)

Figure 10: Rain streak removal results by different methods. From left to right: the rainy frames, the results by LRMC[6], DIP [23], and the proposed method. From top to bottom: three frames of the first real video.

(20)

Figure 11: Rain streak removal results by different methods. From left to right: the rainy frames, the results by LRMC[6], DIP [23], and the proposed method.

(21)

ACKNOWLEDGEMENT

The research is supported by NSFC (61876203 61772003, 61702083) and the Fundamental Research Funds for the Central Universities (ZYGX2016J132, ZYGX2016KYQD142, ZYGX2016J129).

[1] S. Maji, A.-C. Berg, and J. Malik. Classification using intersection kernel support vector machines is efficient. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR), pages 1–8, 2008.

[2] O.-L. Junior, D. Delgado, V. Goncalves, and U. Nunes. Trainable classifier- fusion schemes: An application to pedestrian detection. In Proceedings of the IEEE International Conference on Intelligent Transportation Systems (ITSC), pages 1–6, 2009.

[3] M.-S. Shehata, J. Cai, W.-M. Badawy, T.-W. Burr, M.-S. Pervez, R.-J. Johan- nesson, and A. Radmanesh. Video-based automatic incident detection for smart roads: the outdoor environmental challenges regarding false alarms.

IEEE Transactions on Intelligent Transportation Systems, 9:349–360, 2008.

[4] D. Comaniciu, V. Ramesh, and P. Meer. Kernel-based object tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25:564–577, 2003.

[5] K. Garg and S.-K. Nayar. Vision and rain.International Journal of Computer Vision, 75:3–27, 2007.

[6] J.-H. Kim, J.-Y. Sim, and C.-S. Kim. Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Transactions on Image Processing, 24:2658–2670, 2015.

[7] L.-W. Kang, C.-W. Lin, and Y.-H. Fu. Automatic single-image-based rain streaks removal via image decomposition.IEEE Transactions on Image Pro- cessing, 21:1742–1755, 2012.

[8] Y. Luo, Y. Xu, and H. Ji. Removing rain from a single image via discriminative sparse coding. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3397–3405, 2015.

(22)

[9] M. Roser, J. Kurz, and A. Geiger. Realistic modeling of water droplets for monocular adherent raindrop recognition using bezier curves. In Proceed- ings of the Asian Conference on Computer Vision (ACCV), pages 235–244, 2011.

[10] S.-D. You, R.-T. Tan, R. Kawakami, Y. Mukaigawa, and K. Ikeuchi. Adher- ent raindrop modeling, detectionand removal in video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38:1721–1733, 2016.

[11] L.-J. Deng, T.-Z. Huang, X.-L. Zhao, and T.-X. Jiang. Rain removal from a single image via a unidirectionally global sparse model. Applied Mathemat- ical Modeling, 2017.

[12] Y.-L. Chen and C.-T. Hsu. A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In Proceedings of the IEEE Inter- national Conference on Computer Vision (ICCV), pages 1968–1975, 2013.

[13] Y. Li, R.-T. Tan, X.-J. Guo, J.-B. Lu, and M.-S. Brown. Rain streak removal using layer priors. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR), pages 2736–2744, 2016.

[14] S.-H. Sun, S.-P. Fan, and Y.-C.-F. Wang. Exploiting image structural similarity for single image rain removal. In Proceedings of the IEEE International Conference on Image Processing (ICIP), pages 4482–4486, 2014.

[15] L. Zhu, C.-W. Fu, D. Lischinski, and P.-A. Heng. Joint bi-layer optimization for single-image rain streak removal. In Proceedings of the IEEE Interna- tional Conference on Computer Vision (ICCV), pages 2545–2553, 2017.

[16] Y. Chang, L.-X. Yan, and S. Zhong. Transformed low-rank model for line pattern noise removal.In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1726–1734, 2017.

[17] W.-H. Yang, R.-T. Tan, J.-S. Feng, J.-Y. Liu, Z.-M. Guo, and S.-C. Yan.

Deep joint rain detection and removal from a single image.In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1685–1694, 2017.

[18] X.-Y. Fu, J.-B. Huang, X.-H. Ding, Y.-H. Liao, and J. Paisley. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Processing, 26:2944–2956, 2017.

(23)

[19] K. Garg and S.-K. Nayar. Detection and removal of rain from videos. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR), pages 528–535, 2004.

[20] A.-K. Tripathi and S Mukhopadhyay. Video post processing: low-latency spatiotemporal approach for detection and removal of rain. IET image processing, 6:181–196, 2012.

[21] A.-K. Tripathi and S. Mukhopadhyay. Removal of rain from videos: a re- view. Signal, Image and Video Processing, 8:1421–1430, 2014.

[22] W. Wei, L.-X. Yi, Q. Xie, Q. Zhao, D.-Y. Meng, and Z.-B. Xu. Should we encode rain streaks in video as deterministic or stochastic? In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2535–2544, 2017.

[23] T.-X. Jiang, T.-Z. Huang, X.-L. Zhao, L.-J. Deng, and Y. Wang. A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR), pages 4057–4066, 2017.

[24] X.-L. Zhao, W. Wang, T.-Y. Zeng, T.-Z. Huang, and M.-K. Ng. Total variation structured total least squares method for image restoration. SIAM Jour- nal on Scientific Computing, 35:1304–1320, 2013.

[25] X.-L. Zhao, F. Wang, and M.-K. Ng. A new convex optimization model for multiplicative noise and blur removal. SIAM Journal on Imaging Sciences, 7:456–475, 2014.

[26] L.-J. Deng, W.-H. Guo, and T.-Z. Huang. Single image super-resolution by approximated heaviside functions. Information Sciences, 348:107–123, 2016.

[27] L.-J. Deng, W.-H. Guo, and T.-Z. Huang. Single-image super-resolution via an iterative reproducing kernel hilbert space method. IEEE Transactions on Circuits and Systems for Video Technology, 26:2001–2014, 2016.

[28] T.-X. Jiang, T.-Z. Huang, X.-L. Zhao, T.-Y. Ji, and L.-J. Deng. Matrix fac- torization for low-rank tensor completion using framelet prior. Information Sciences, 2018. Accepted.

(24)

[29] T.-X. Jiang, T.-Z. Huang, X.-L. Zhao, L.-J. Deng, and Y. Wang. Fastderain:

A novel video rain streak removal method using directional gradient priors.

arXiv preprint arXiv:1803.07487, 2018.

[30] W. Deng, W.-T. Yin, and Y. Zhang. Group sparse optimization by alternating direction method. In Proceedings of the The International Society of Optics and Photonics (SPIE), 8858, 2013.

[31] K. Dan. A singularly valuable decomposition: The svd of a matrix. College Mathematics Journal, 27:2–23, 1996.

[32] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations & Trends in Machine Learning, 3:1–122, 2011.

[33] Z. Wang, A.-C. Bovik, H.-R. Sheikh, and E.-P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13:600–612, 2004.

(25)

Table 1: Quantitative comparisons of rain streak removal results by LRMC [6], DIP [23], and the proposed method, on the selected 6 synthetic videos, respectively.

Rain type Heavy Light

Video Method PSNR SSIM TIME(S) PSNR SSIM TIME(S)

carphone

Rainy 28.151 0.751 - 36.641 0.926 -

LRMC 30.496 0.848 2230.193 36.490 0.978 1381.876 DIP 35.196 0.955 190.997 42.742 0.987 280.895 Proposed 38.486 0.971 230.311 43.021 0.991 343.444

container

Rainy 28.551 0.758 - 37.162 0.929 -

coastguard

Rainy 28.128 0.833 - 36.579 0.956 -

highway

Rainy 29.056 0.744 - 37.524 0.925 -

bridgefar

Rainy 28.945 0.713 - 37.264 0.910 -

foreman

Rainy 28.341 0.808 - 36.954 0.947 -

(26)

Table 2: Quantitative comparisons of rain streak removal results by the proposed method with one column, half of one column, quarter of one column, eighth of one column.

Video Method PSNR SSIM TIME(S) PSNR SSIM TIME(S)

carphone

Rainy 28.151 0.751 - 36.641 0.926 -

one column 38.486 0.971 230.311 43.021 0.991 343.444 half of one column 38.138 0.973 224.136 41.372 0.990 330.496 quarter of one column 37.486 0.973 234.334 42.248 0.991 344.667 eighth of one column 35.166 0.956 242.899 43.081 0.991 339.799

container

Rainy 28.551 0.758 - 37.162 0.929 -

coastguard

Rainy 28.128 0.833 - 36.579 0.956 -

highway

Rainy 29.056 0.744 - 37.524 0.925 -

bridgefar

Rainy 28.945 0.713 - 37.264 0.910 -

foreman

Rainy 28.341 0.808 - 36.954 0.947 -

(27)

Table 3: Quantitative comparisons of rain streak removal results by LRMC [6], DIP [23], and the proposed method, on the selected 2 synthetic videos, respectively.

Rain video Quantitative comparisons

Video Method PSNR SSIM TIME(S)

highway2

Rainy 27.170 0.803 -

LRMC 27.640 0.878 2530.393 DIP 33.406 0.929 258.067 Proposed 36.783 0.953 343.453

waterfall

Rainy 28.551 0.758 -

LRMC 31.338 0.877 1850.684 DIP 35.593 0.939 184.324 Proposed 37.782 0.960 293.509

Table 4: Quantitative comparisons of rain streak removal results by LRMC [6], DIP [23], and the proposed method on the “carphone”synthetic videos, respectively.

Method PSNR SSIM TIME(s) PSNR SSIM TIME(s)

Rainy 28.151 0.751 - 36.641 0.926 -

LRMC 30.496 0.848 2230.193 36.490 0.978 1381.876 DIP 35.196 0.955 190.997 42.742 0.987 280.895 Proposed 38.486 0.971 230.311 43.021 0.991 343.444 Proposed without (a) 38.406 0.969 763.256 43.005 0.990 1027.011 Proposed without (b) 37.856 0.962 221.054 42.958 0.989 310.520