Chapter 5 Proposed Method of Rain/Snow Removal in Single Image
5.2 Proposed Method
5.2.2 Rain/Snow Detection and Selection Process
In the case of the rain presents on the window surface, raindrops located against a clear area (generally the sky) are not perceptible even by a human expert as shown in Fig 5.3. Therefore, the first step is separated into dark and clear regions, but we only apply the detection methods in the darker region. Dark areas are extracted by a combination of morphological operations as following process:
Step 1: Erosion of the original image by a disk of a diameter of 20 pixels as a structuring element.
Step 2: Morphological reconstruction [59] of the original image with the eroded image as a mask.
Step 3: Application of step1 and 2 on the complement of the reconstructed image.
Step 4: Segmentation of the result using Ostu’s method [60].
(a) (b) (c)
(a) (b) (c)
Fig. 5.3 Segmentation of dark versus clear region. (a) Original image. (a) The sky region. (b) The dark region (raindrops are visible in this region).
In natural property, the rain and snow are brighter than background. Thus, we exploit three properties of rain and snow. First, a rain or snow pixel tends to be brighter than a rain-free or snow-free pixel. Second, pixels in rain and snow have similar color.
Third, the power spectral distribution of rain and snow “streaks” in the 2D frequency space is elliptical and oriented with certain direction, this ellipse is the sum of all the rain or snow streaks, but snow ”flakes” is random and not orientated to any directions, as show in Fig.5.4. Hence, we need different model to process.
(a) (b)
(a) (b)
(a) (b)
Fig. 5.4 Power spectral distributions of rain and snow images in frequency domain. (a) The original images. (b) Its two dimensional Fourier transform.
The first property is related to the intensity levels of pixels, As described in [49], when a pixel has ever been covered by rain, the average of luminance in a rain or snow region A is higher than the average of luminance for those in rain-free or snow-free region B in general. Thus, we design the following process to detect rain and snow effectively:
Step 1: Calculate the average intensity of the pixel p in a single image, where N is the number of pixels.
Step 2: Determine the threshold TH1. X is a parameter and may be various in different cases of rain.
1 1 ˆ %
TH A X (5.2)
Step 3: Determine whether the pixel p is larger than the threshold (TH1). If the pixel p is larger than the threshold (TH1), it is regarded as the rain or snow pixel; otherwise, it is regarded as the rain-free or snow-free region. According to the experiments, we get the following conclusion: if X is smaller we can get more candidate rain or snow pixel, and vice versa.
Fig 5.5 shows the different threshold with the candidate rain pixels. We can simply and readily get a preliminary rain detection results through the rain detection process (see Fig. 5.6) and then save them as a binary field (1 & 0) to represent the rain (snow) and non-rain (non-snow) pixel.
(a) (b) (c) (d) Fig. 5.5 The passible rain candidates with different threshold (initial binary field). (a) Input rain image. (b) The passible rain candidates (TH1is set to A ). (c) The passible rain candidates ( TH1is set to smaller than A ). (d) The passible rain candidates (TH1is set to larger than A ).
Calculate the average intensity of the pixel p in single image .
Determine the threshold .
1 1 ˆ %
TH A X
1 ? pi TH
is rain pixel piis backround pixel pi
Fig. 5.6 Flow chart of the rain or snow detection process with first property.
The second property is related to the pixels in rain or snow which have similar color. According to the chromatic property, the amounts of change ΔR, ΔG, and ΔB for pixels covered by rain and snow are approximately the same (actual rain and snow pixel). When a pixel has ever been covered with the rain or snow, the peak values can be obtained in its RGB component histogram. Although there can be more than one local maximum, there is only one global maximum (peak). The global maximum can be obtained in its histogram distribution of RGB components. A pixel has ever been covered by rain or snow, and the RGB components in a rain or snow region A are close to its global maximum. Thus, we design the following process to detect rain effectively
Method 1:
Step 1: Draw the histogram distribution of the RGB components, as show in Fig 5.7.
(a) (b)
Fig. 5.7 The histogram distribution of the RGB components. (a) The original snow image. (b) Its histogram distribution of RGB components.
Step 2: Calculate the average pixel index x from the global maximum in the RGB components, where x is the pixel index and H is the histogram operator.
, ,
Step 3: Determine whether the pixel p is larger than the threshold (TH2). If the pixel p is larger than the threshold (TH2), it is regarded as the rain or snow pixel; otherwise, it is regarded as the rain-free or snow-free region.
2 ,rain or snow pixel
Method 2: It is similar to the process of first property.
Step 1: Calculate the mean R, G, and B values of the pixels p in the image, where N is the number of pixels.
Step 3: Determine whether the pixel p in the RGB component are larger than the threshold (TH3). If the pixel p is larger than the threshold (TH3), it is regarded as the rain or snow pixel; otherwise, it is regarded as the rain-free or snow-free region. According to the experiments, we get the following conclusion: if X is smaller we can get more candidate rain or snow pixel, and vice versa.
3 , rain or snow pixel
We can simply and readily get a preliminary rain detection results through the rain detection process (see Fig. 5.8) and then save them as a binary field (1 & 0) to represent
5.9 (a) highlights regions in the video in which pixel colors are examined. Fig. 5.9 (b) shows the mean R, G, and B values of the pixels in the selected regions, and Fig. 5.9 (c) shows the corresponding means and standard deviations of ΔR, ΔG, and ΔB. These examples show that the mean ΔR, ΔG, and ΔB are indeed different and related to
Fig. 5.8 Flow chart of the rain or snow detection process with second property.
Rain pixels
Rain pixels
Rain pixels
Fig. 5.9 (a) The regions selected for investigation. (b) The mean R, G, and B values of the pixels in the selected regions. (c) The corresponding means the standard deviation of ΔR, ΔG, andΔB.
The third property is related to the power spectral distribution of rain and snow
“streaks” in the 2D frequency space is elliptical and oriented with certain direction, this ellipse is the sum of all the rain or snow streaks, but snow ”flakes” is random and not orientated to any directions. We design the following process to detect rain or snow effectively
exploit these unique properties of rain and snow to detect the candidates of rain and snow pixel. For the purpose of getting the rain or snow initial candidate (Cinitial) pixel of the snow or rain, we merge the saturation layer (S) in HSI color model and Intensity layer (I) in color model as
1
initial
C S I (5.9)
Step 2: Use a high pass filter to enhance the rain or snow candidates by multiplying the high pass filter and the rain or snow candidates in the frequency domain. The High pass filter can be expressed as
1 rHF e (5.10)
where r is the distance between the core of the image and the pixel p , is the coefficient which decides the stop band of the filter.
Step 3: Use the orientation filter to separate the rain (snow) streak or background object.
Rain or snow streaks generally fall in almost the same direction, so only one p is needed. However, snowflake is not.
Step 4: Determine whether the rotation angle in pixel p is in the constrained threshold
. If the pixel p is smaller than the constrained threshold ( ), it is regarded as the rain or snow “streak” pixel; otherwise, it is regarded as the background region.
p 2
(5.11)
We can simply and readily get a preliminary rain detection results through the rain detection process (see Fig. 5.10) and then save them as a binary field (1 & 0) to represent the rain (snow) and non-rain (non-snow) pixel.
1
initial
C S I
2 ?
p
pi
pi
1 rHF e
p
Fig. 5.10 Flow chart of the rain or snow detection process with third property.
Above all, there may be a problem caused by other brighter subjects in the scene (e.g.
white car, accumulated snow) so that the rain or snow detecting result is not quite perfect. Hence, we add the following restrict to exclude the above situation: the actual number of pixels in the region
Rpof rain or snow will smaller than a threshold (TH4) as
p 4
R TH (5.12)
We can find that other brighter subjects have been removed as illustrated in Fig 5.11.
(a) (b) (c) Fig. 5.11 Result of the identification of snow streaks by our proposed method. (a) All potential snow streaks (initial snow binary map). (a) All potential snow streaks (initial snow jet colormap). (b) Initial snow mask refined by area restrict.
Besides, some surrounding rain pixels have a chance to be undetected because of their low intensity or unfocused, when we process the single rain or snow (e.g. [61]-[64]). In order to avoid this, we extend the outer contour of the detection regions. We assume that a rain or snow streak column (i.e., (4, 5), (5, 5), (6, 5)) is detected, after that we extend this column into both side columns to produce a block-matrix as in Fig. 5.12. Finally we combine the binary field (1 & 0) from the properties to get the precise rain or snow pixel. We assume that two binary fields are detected from the snow image, and then we combine them to get the precise one as in Fig. 5.13. Further, we apply the proposed rain removal process to remove the rain from the image.
(4, 4) (4,5) (4,6)
Fig. 5.12 Schematic diagram of rain or snow extending.
(a) (b)
(c) (d) Fig. 5.13 The result of combination process. (a) The original snow image. (b) The first
binary field. (c) The second binary field. (d) The combinated binary field.