Chapter 5 Proposed Method of Rain/Snow Removal in Single Image
5.2 Proposed Method
5.2.5 Color Transfer Post-processing
With the snow-free image can be obtained by section 5.2.3 but has higher dynamic than the non-dehazed one. Color mapping should be done as essential step to allow the visual comparison of the resulting images obtained by our proposed method as well as for comparison with the non-pre-processing image. Balancing the two comparisons, we adopt the snow-free image and the snow-free image without fog removal pre-processing image to obtain the composite information and use color transfer model [33] to obtain a linear color mapping on the resulting images. After this paragraph, we will introduce the basic concept of color transfer model. The multiscale retinex with color restoration (MSRCR) [36] is a visibility restoration algorithm which makes the recovered image have wealthy color information and color distortion. MSRCR is applied to the original image I(x) to obtain R(x) and the bottom third part of I(x) is regarded as less fog influenced areas. According to the concepts, we combine R(x) with the bottom third part of I(x) by a weight to obtain the source image (non fog removal pre-processing image). The rain/snow-free image J(x) is the target image. Color transfer model is a
linear transformation from the source image to the target image based on the simplest statistics of two images’ global color distributions:
1
R I
1
R I
J deviation of the source and target images in the l color space, respectively. The weight balances the effects of two images and is set as 0.5. The final obtained image T(x) will be with suitable visual appearance.A fast and effective technique for color transfer between images has been proposed by Reinhard at al.[33]. This method changes the color characteristics of the source image I from the target image It by statistic correction. The basic concept of the proposed method is to decorrelate color space so that an uncorrelated orthogonal color space can be identified between the axes. First, we process a conversion from RGB to XYZ tristimulus values and convert XYZ space to LMS space. Further, the resulting of these two converted matrices between RGB and LMS cone space can be combined as
L 0.3811 0.5783 0.0402
Apparently, the data in this color space shows a great deal of skews and the skews must be eliminated. We convert the data to logarithmic space for eliminating these skews by
log L
By avoiding unwanted cross effect and treating the three color channels separately, Reinhard at al. [33] uses the l color space for ensuring each channel maximal
uncorrelated and represents as the following transform
After transferring the color space from RGB to l, each channel of the source image I subtracts the mean of itself for every pixel as
*
where l represents an achromatic channel, α and β are chromatic yellow-blue and red-green opponent channels, and 〈 〉 is the calculation of mean. Then, the difference of each channel in (5.11) is scaled by the standard deviations of target image (t) and source image (s), respectively. Besides, the scaled data must add the mean of the target image (It) [33]:
After this, the source image I has the average color tone and luminance from the target image, and the result is converted back to R’G’B’ color space for acquiring the output image I’ by
' 0.5773 0.2621 11.3918 '
In Fig. 5.17, it is shown that the transfer process of snow images by using the proposed method.
(a) (b)
(c) (d) Fig. 5.17 Transfer process of snow images by using the proposed method. (a) The
original snow image. (b) The result of snow removal. (c) The result of snow removal is further used to haze remove by our proposed method. (d) The result after the color transfer mapping.
5.3 Experimental Results
The experiment is separated into 4 cases: Case 1: the input rainy images. Case 2:
snow in video sequences with dynamic cases. Case 5: the rain presents on the window surface.
In Case 1, the input image is used the proposed method to remove rain. The rain removal results are shown in Fig. 5.18. Compare with our method, Kang et al. [5] and Kim et al. [6] in Fig. 5.18, and it shows that the background textures is be distorted because the most vertical patterns is be regarded as rain streaks in the result by Kang et al. Note that the background texture patterns in Fig. 5.18(b) are not contained in the original image in Fig. 5.18(a). In the image, rain streaks are removed and original textures are preserved by Kim et al. approach. But the defocused streaks and streaks on bright backgrounds show very small intensity changes so that they are hard to detect as shown in Fig. 5.18(c). In contrast, we see that most rain streaks in all test images are successfully removed, and original textures are faithfully preserved at the same time by the proposed algorithm. The reason why the result is caused is that the proposed algorithm selectively applies more properties in spatial and frequency domain.
In Fig. 5.19, the proposed, Xu et al. [4], and Kang et al. [5] approaches are compared, and it shows that the result of the Xu et al. approach removes more rain, but it still simultaneously removes other image detail and blurs the image as shown in Fig 5.19(b). The background textures is be distorted because the most vertical patterns is be regarded as rain streaks in the result by Kang et al. Note that the background texture patterns in Fig. 5.19(c) are not contained in the original image in Fig. 5.19(a). In contrast, we see that most rain streaks in all test images are successfully removed, and original textures are faithfully preserved at the same time by the proposed algorithm.
The reason why the result is caused is that the proposed algorithm selectively applies more properties in spatial and frequency domain.
In Case 2, the input image is snow “streak” removed by applying the proposed
method. Compare with the result of the Xu et al.’s [4] in Fig 5.20(b), their approach removes more snow, but it still simultaneously removes other image detail and blurs the image. In Fig 5.20(c), the background textures is be distorted because the most vertical patterns is be regarded as rain streaks in the result by Kang et al. Note that the background texture patterns. In contrast, we see that most snow streaks in all test images are successfully removed, and original textures are faithfully preserved at the same time by the proposed algorithm. Furthermore, clearer and brighter defogged images can be acquired, and the fine transitions can be preserved by the proposed method in the fog like appearance without introducing unpleasing artifacts.
In Case 3, the input image is used the proposed method to remove the snow
“flake”. In Fig 5.21(b) and Fig 5.21(c), the refined guidance images have similar contour with the un-degraded image and also maintain the detailed information which may be lost by using the guided filter [55]. Although most snow can be removed by these methods, other image detail will be removed and image will be blurred in the meantime. We see that most snowflakes in all test images are successfully removed, and original textures are faithfully preserved at the same time by the proposed algorithm.
Fig 5.22 shows other snowflakes removal results.
In Case 4, the frequency-based analysis method [7] focuses on the removal of rain and snow streaks in video sequences captured by static or dynamic cameras. Compare with the results of Barnum et al. [7] shown in Fig 5.23-26, we just consider frame by frame in our proposed method. The evaluation method as [69] is used in their approach, and it is used to track points while each sequence is played forward then backwards.
Because each sequence starts and ends on the same frame, each point should be in the same location at the beginning and the end. Tracking accuracy is defined as the distance
textures around the rain removal regions, but there are not producing unpleasing artifacts as shown in Fig 5.23 in our method. In Fig 5.24, when raindrops are broken with splashes falling on the subjects, the orientation of rain-streak rotation angle may be horizontal. In this case, the rain streaks cannot be removed well by the frequency-based [7], but our proposed method can be used to deal with this problem. In Fig 5.25, we see that most rain streaks in the video sequence are successfully removed, and original textures are faithfully preserved at the same time by the proposed method. This is a very difficult sequence with a lot of high frequency textures, very heavy snow, and multiple moving objects as shown in Fig 5.26. In Fig 5.26(b), much of the snow is removed, but the edge of the umbrella is misclassified. In contrast, we see that most snow streaks in this video sequence are successfully removed, and original textures are faithfully preserved at the same time by the proposed method as shown in Fig 5.26(c).
In Case 5, we focus that rain presents on the window surface. In Fig 5.27(b), the scene and reflections are preserved; raindrops on the window are removed but a few small artifacts do remain by Eigen et al. [61] approach. There are no artifacts and most of water drops are removed while retaining image detail as shown in Fig 5.27(c) in our method. Fig 5.28 shows another focused rain removal results. We see that most raindrops in single image are successfully removed, and original textures are faithfully preserved at the same time by the proposed method as shown in Fig 5.28(f). Compare with the result of the Cord et al.’s [63] in Fig 5.29(h), their approach has much false and missed detection. Fig 5.30 shows another unfocused rain removal results. More experimental results can be found [70]:
https://sites.google.com/site/yutaitsaithesis/rain_removal https://sites.google.com/site/yutaitsaithesis/snow_removal
(a) (b)
(c) (d)
(b) (b)
(c) (d) Fig. 5.18 Comparison of rain removal results: (a) the original rain image; the
rain-removed version via: (b) Kang et al.’s method with the single-frame-based image decomposition [5], (c) Kim et al.’s method with the adaptive nonlocal mean filter [6];
(a) (b)
(c) (d)
(a) (b)
(c) (d)
(a) (b)
(c) (d)
(a) (b)
(a) (b)
(c) (d) Fig. 5.19 Comparison of rain removal results: (a) the original rain image; the
rain-removed version via: (b) Xu et al.’s method with the guided filter [4]; (c) Kang et al.’s method with the single-frame-based image decomposition [5]; and (d) the proposed rain removal scheme.
(a) (b)
(c) (d)
(a) (b)
(a) (b)
(c) (d) Fig. 5.20 Comparison of snow “streak” removal results: (a) the original snow image;
the snow-removed version via: (b) Xu et al.’s method with the guided filter [4]; (c) Kang et al.’s method with the single-frame-based image decomposition [5]; and (d) the proposed snow removal scheme.
(a) (b)
(c) (d) Fig. 5.21 Comparison of snow “streak” removal results: (a) the original snow image;
the snow-removed version via: (b) Xu et al.’s method with the guided filter [4]; (c) Xu et al.’s method with the refined guidance image [4]; and (d) the proposed snow removal scheme.
(a) (b)
(a) (b)
Fig. 5.22 Snowflakes removal results: (a) Input snow images. (b) Our results.
(a) (b) (c)
Fig. 5.23 Comparison of rain streaks removal results: (a) the sitting man sequence. (b) rain removal by the frequency-based analysis [7]. (c) the proposed rain removal scheme.
(a) (b) (c)
Fig. 5.24 Comparison of rain streaks removal results: (a) the window building sequence.
(b) rain removal by the frequency-based analysis [7]. (c) the proposed rain removal scheme.
(a) (b) (c)
Fig. 5.25 Comparison of snow removal results: (a) the mailbox sequence. (b) snow removal by the frequency-based analysis [7]. (c) the proposed snow removal scheme.
(a) (b) (c)
Fig. 5.26 Comparison of snow removal results: (a) the walker in the snow sequence. (b) snow removal by the frequency-based analysis [7]. (c) the proposed snow removal scheme.
(a) (b) (c)
(a) (b) (c)
Fig. 5.27 Comparison of rain drops removal results: (a) smartphone shot through a rainy window on a train. (b) rain removal by [61]. (c) the proposed rain removal scheme.
(a) (b)
(c) (d)
(a) (b)
(c) (d) Fig. 5.28 Comparison of focused raindrops detection results: (a) original image
acquired by camera. (b) the dark region. (c) raindrops detected by our proposed scheme.
(d) the result after the proposed rain removal scheme.
(a) (b)
(c) (d)
(e) (f)
(g) (h) Fig. 5.29 Comparison of unfocused raindrops detection results: (a) original image
acquired by camera. (b) the dark region (raindrops are visible in this region). (c) the background image. (d) the difference between the original image and background image.
(e) raindrops detected by our proposed scheme. (f) the result after the proposed rain removal scheme. (g) ground truth: red stars correspond to the mark made by operator. (h) raindrops detected using the background subtraction method [63]. Green: correctly detected raindrops. Blue: missed raindrops. Red: false detection.
(a) (b)
(c) (d) Fig. 5.30 Comparison of unfocused raindrops detection results: (a) original image
acquired by camera. (b) the dark region (raindrops are visible in this region). (c) raindrops detected by our proposed scheme. (d) the result after the proposed rain removal scheme.
5.4 Conclusions
According to the natural property of rain and snow, we divide the rain or snow removal scheme into two parts: the first part is detection of rain or snow and the second
the column extended into both side columns to produce a block-matrix for rain removal so that it may get more benefits for single image consideration when streak or flake is not obvious. The fuzzy random impulse reduction method [61] is also used for noise removal. Specifically, to obtain a quality vision of a resulting image, color transfer is utilized to protect the final snow-free image’s color from high dynamic. The results show that our method is attractive and effective for the rain and snow removing quality.
As follows, rain and snow removal methods with low complexity, low computational time, and high rain removing quality are proposed.
Our proposed method is computationally effective taking approximately 2 seconds (Matlab code) for a 400×400 image. Kang et al.’s [5] method taking approximately 66 seconds and Xu et al.’s [4] method taking approximately 0.3 seconds. Although Xu et al.’s [7] method use the shortest time, this method simultaneously removes other image detail and blurs the image. The performance of Kang et al.’s [5] method depends on the clustering of dictionary basis vectors. When the clustering is not effective, their method may erase textures, as well as rain streaks, and yield visual artifacts in a restored image.
Furthermore, the procession of dictionary learning needs much more processing time.
Our proposed method can also deal with the case that rain presents on the window surface. Although Eigen et al.’s [61] method can effective remove dirt and rain in outdoor test conditions; this method simultaneously removes other image detail and blurs the image. Furthermore, the quality of the results does depend on the statistics of test cases being similar to those of the training set. In the cases, where this does not hold, and we see significant artifacts in their output. The corruption cannot be much larger than the training patches. It means that their input image may need to be down-sampled, e.g. as in the rain application, leading to a loss of resolution relative to the original. The performance of Cord et al.’s [63] method depends on the watershed segmentation and
background subtraction. Their method may have much false and missed detection.
In the future work, the rain and snow removal techniques will be expected to be used for many applications, such as traffic safety, film processing, and computer vision, etc..