Chapter 3 Protection of Selected Private Regions in Surveillance
3.5 Summary
In this chapter, we have proposed a new method for protecting selected private region in a surveillance video using the prediction-based mapping. Based on the technique, we can convert a privacy-sensitive image to a camouflage image which looks similar to the background image. We have also proposed a method for improving the quality of the generated camouflage image. The privacy-sensitive image concealed in the camouflage image can be retrieved losslessly.
Sometimes the color difference between the background image and the privacy-sensitive image results in visible stitching seams. To solve this problem, we may adjust the color of the pre-selected background image to suit the privacy-sensitive image. The details of the techniques will be described in Chapter 4.
With this proposed privacy protection system, we can conceal the privacy parts of surveillance video contents to avoid legal disputes and to protect the personal
50
privacy of non-suspicious people. Moreover, the proposed region-based privacy protection method is different from some traditional object-based privacy protection techniques [3] that need to recognize protected persons in advance.
(a) (b)
(c) (d)
(e) (f)
Figure 3.10 Six representative frames of a surveillance video. (a) The first frame with the protected region enclosed by a rectangle. (b) The second frame. (c) The third frame. (d) The 4th frame. (e) The 5th frame. (f) The 6th frame.
51
(a) (b)
(c) (d)
(e) (f)
Figure 3.11 Six representative frames of a privacy protected video. (a) The first frame. (b) The second frame. (c) The third frame. (d) The 4th frame. (e) The 5th frame. (f) The 6th frame.
52
(a) (b)
(c) (d)
(e) (f)
Figure 3.12 Six representative frames of a recovered video. (a) The first frame. (b) The second frame.
(c) The third frame. (d) The 4th frame. (e) The 5th frame. (f) The 6th frame.
53
Chapter 4
Protection of Private Motion
Activities in Surveillance Videos Using a Reversible
Prediction-based Mapping
4.1 Introduction
The proposed method for region-based privacy protection, which can be utilized for protecting selected private regions, is described in the previous chapter.
Specifically, we use a portion of a pre-selected background image to cover a selected private region in an image frame of a surveillance video by a reversible prediction-based mapping scheme. Furthermore, it is desirable that the proposed technique can be applied to conceal private motion activities. In this view, a method for object-based privacy protection which can be used for protecting private motion activities in surveillance videos is proposed in this study as mentioned previously, which will be described in this chapter. In Sections 4.1.1 and 4.1.2, the related problem definitions and the major idea of the proposed method are given. The detailed algorithms of motion-activity concealment and recovery are presented in Section 4.2. Experiment results showing the feasibility of the method are given in Section 4.3. Finally, a brief summary is given in the last section of this chapter.
54
4.1.1 Problem definition
In the problem of protecting private motion activities in videos dealt with in this study, we first select a region for detecting motion events in each image frame of the surveillance video automatically. Then, we try to apply the previously-proposed process of the private region concealment (described in Chapter 3) to the selected region in each video frame. However, the color difference between the pre-selected background image which is the first frame of the video and the currently-processed surveillance image frame results in an unnatural camouflage image with visible stitching seams. Two major issues are involved in this problem. The first is how to reduce the color difference between the pre-selected background image and the currently-processed surveillance image. The second is how to stitch the pre-selected background image part to the currently-processed surveillance image seamlessly. It goes without saying that the proposed scheme to solve this problem must allow the concealed privacy-sensitive image part to be retrieved losslessly, as mentioned before.
4.1.2 Major idea of proposed method
In order to overcome the above-mentioned problem of color difference between the pre-selected background image part and the currently-processed surveillance image, which is encountered when the former is superimposed on the latter, we adjust the color characteristics of the pre-selected background image part to suit the currently-processed privacy-sensitive image by a color conversion. Moreover, a seamless image stitching technique which modifies only the appearance of the background image part is used in this application to eliminate the visible stitching seams. After these two steps are conducted, we take the resulting background image part and the privacy-sensitive image part as inputs to create the camouflage image by
55
the proposed method for privacy protection as described in Chapter 3. Later, the parameters used in the color conversion and the position of the detected motion activity region in each surveillance image frame are embedded into the resulting camouflage image. Then, in the later source image recovery process, we can first recover the modified background image by the extracted parameters and the seamless image stitching technique. And then we can retrieve the privacy-sensitive image part from the camouflage image and the modified background image part. More details are described subsequently.
4.2 Proposed method for Protecting Private Motion Activities in
Surveillance Videos
In this section, we describe in detail the proposed method for protecting private motion activities in surveillance videos. In Section 4.2.1, the proposed method for adjusting the color difference is described. In Section 4.2.2, the proposed method a modified version of the previously-described privacy concealment process for use in protecting private motion activities is described. In Section 4.2.3, the process of motion-activity recovery is presented.
4.2.1 Adjustment of color differences
The above-mentioned problems appearing in the camouflage image are discussed in this section, and the solutions to them are also presented. The proposed method includes two main phases: color modification of the pre-selected background image part and seamless image stitching. Both schemes are described as follows. And the method used to detect moving object and decide the protected region is stated as well.
56
A. Color modification of the pre-selected background image part
In the proposed method for protecting private motion activities, at the beginning we search motion activities in videos, and then decide a corresponding protected region R for detecting motion event automatically. Also, the image content in R is defined as a privacy-sensitive image part and will be disguised as a pre-selected background image part B which corresponds to the privacy-sensitive image part in position in the first image frame of the video. Then, we apply the previously-proposed concealment process (described in Chapter 3) with the background image part to the privacy-sensitive image part to produce a camouflage image looking close to the background image part B.
Since the color characteristics of B and the currently-processed surveillance image frame T may be different from each other, it leads to an inconsistent color appearance in the resulting camouflage image, as shown by the example in Figure 4.1(c). How to modify the color appearance of B to match T is the major issue here.
Reinhard et al [29] proposed a work about color transfer in this aspect, which uses a statistical technique to convert the color characteristic of a source image to be that of another target image. The idea is an answer to our problem here and is adopted to solve it in this study. But instead of conducting color conversion in the lαβ color space as done in [29], we do it in the RGB color space. More specifically, let a source image IP and a target image IQ be described as two pixel sets the source pixel set CS = {P1, P2, …, Pn} and the target pixel set CT = {Q1, Q2, …, Qn}, respectively, assuming that both images are of the same size n. The two sets then are used to compute their respective means, denoted as µp and µq, and standard deviations, denoted as σp and σq, for each color channel (R, G, or B) by the following equations:
57
This results in a new image I′p with the color characteristic close to that of IQ.
(a) (b)
(c)
Figure 4.1 An example of results of protecting private motion activities. (a) The privacy-sensitive image part enclosed by a rectangle. (b) The pre-selected background image part enclosed by a rectangle. (c) The camouflage image generated from (a) and (b) using the proposed private region concealment process.
58
In the proposed scheme, it is desirable that the color characteristic of the background image part B can be adjusted to be close to that of the corresponding part in the currently-processed surveillance image frame T. Therefore, we do not use the color information of the objects in T but take that of the background part in region R of T to derive the parameters for color conversion. Specifically, we first take the background image part and the privacy-sensitive image part as the above-described target pixel set CT and the source pixel set CS, respectively. Then, for each pixel P in region R of B, we compute the value Dc of the color difference between P and the corresponding pixel Q in T by the following equation:
Dc = |rp − rq| + |gp − gq| + |bp − bq|, (4.4) where (rp, gp, bp) and (rq, gq, bq) are the color values of P and Q in the RGB color space, respectively. Then, we use a pre-selected threshold Tc for Dc to decide whether or not P and Q should be abandoned from the CT and the CS for color conversion if Dc > Tc, then P and Q are abandoned, otherwise, not. For example, as illustrated in Figure 4.2, the pixels with yellow color in R denote those which are selected in this process as the target pixel set, which together with the corresponding pixels in B as the source pixel set, is used to derive the respective means and standard deviations by Equations (4.1) and (4.2) for each color channel. After this process, we compute the value of each pixel in B by Equation (4.3) with the derived means and standard deviations to yield a new background image part B′. Then, we use B′ and T to generate a camouflage image in which the inconsistent color appearance is improved.
Furthermore, we have to embed into the generated camouflage image necessary information about the modified background image part B′ for use in later recovery of T. For this, we limit the number of bits used to represent the mean and standard deviation values. Specifically, for each color channel we allow the means of T and B
59
to have 8 bits with values 0 ~ 255 and the standard deviation quotients qi = σq / σp to have 7bits with value 0.1 ~ 12.8. Moreover, we do not allow qi or σp to be 0.
(a) (b)
Figure 4.2 An example of the proposed method for selecting the target pixel set used to derive the mean and standard deviations. (a) The pixels with yellow color are selected as the target pixel set. (b) The pixels with blue color are selected as the source pixel set.
B. Seamless image stitching
Image stitching is a common practice in generation of panoramic images and applications such as object insertion. In this study, we use a portion of the background image like a sticker to cover a private region in a surveillance image frame. Such an idea is an analogy to the image stitching technique, which can be used to compose a privacy-free surveillance image frame. After color modification of the background image part is conducted as described before, the stitching seam should be dealt with.
It seems that for the purpose of seamless image stitching, we may blend the seam in the resulting composite image which is the camouflage image. However, we should not do so because the data of the camouflage image should be kept intact for later privacy-sensitive image part recovery. For this reason, instead we manipulate only the pixel values pi of the background image part within the stitching border before the values pi are mapped as the target values to compose the camouflage image. This idea is illustrated in Figure 4.3. How to manipulate pi to compose the camouflage image
60
seamlessly is the major issue here.
Figure 4.3 Illustration of proposed idea for eliminating the visible stitching seams in the camouflage image.
Patrick et al. [30] introduced a method, called Poisson image editing, for seamless editing of image regions, which combines suitably the gradient of a source image with that of a destination image to superimpose a source image on the destination image seamlessly, as shown by the example in Figure 4.4.
First, it is well known that slow gradients of intensity, which are suppressed by the Laplacian operator, can be superimposed on an image with a barely noticeable effect. Conversely, the second-order variations extracted by the Laplacian operator are the most significant perceptually. Therefore, the method first supposes that the destination image part replaced with the source image part is missing, and then completes the missing area as smoothly as possible, as shown in Figure 4.4(d). More specifically, the method supposes that for each pixel D at coordinates (x, y) with the unknown value d, denoted as d(x, y), in the destination image, the unknown value d is expected to exhibit the following principle of smoothness in a 4-neighborhood H of D:
d(x, y − 1) + d(x − 1, y) − 4d(x, y) + d(x + 1, y)+ d(x, y + 1) = 0 where d(x ± 1, y ± 1) are the values of the four pixels in H.
61
It is pointed out that the pixels at the stitching boundary have some neighborhood with the given pixel values bi of the destination image. Therefore, for such pixels we should update the equation above using the given pixel value bi. Specifically, as an example, if d(x, y − 1) has the pixel value b1 which is already known in the destination image, in order to follow the principle of smoothness the equality d(x, y − 1) − d(x, y) = b1 may be assumed, so that the unknown value d is now dependent only on the other three neighboring pixel values, meaning that the above equation should be modified to be:
d(x − 1, y)− 3d(x, y) + d(x + 1, y)+ d(x, y + 1) = −b1.
(a) (b) (c) (d) (e)
Figure 4.4 An example of seamless image stitching result. (a) A source image part enclosed by a yellow outline. (b) The destination image part enclosed by a yellow outline. (c) A result generated from (a) and (b) by superimposing (b) on (a) directly. (d) A result generated from (b) by smooth completion. (e) A result generated from (a) and (b) by using the algorithm of Poisson image editing [30].
Let each d(x, y) be rewritten as an unknown variable xi, where i is an index of the involved pixel, and suppose that there are n unknown values in the region. Therefore, we can reduce the problem to an algebraic system to form a classical, sparse (banded) symmetric, positive-definite matrix to solve these unknown variables, as shown in the following:
62
And then we can use a numerical method iterative solver to solve the sparse matrix. The simple method produces a blurred interpolation. Furthermore, the method can introduce a gradient field V which is a vector field taken directly from the source image to guide the completion as described above to fill the missing area with the gradient field V of the source image, and the unknown value d is now expected to exhibit the following principle:
d(x, y − 1) + d(x − 1, y) − 4d(x, y) + d(x + 1, y)+ d(x, y + 1)
= s(x, y − 1) + s(x − 1, y) − 4s(x, y) + s(x + 1, y)+ s(x, y + 1).
where s is the pixel values of the source image. In the same way, we can solve the spare matrix to interpolate the unknown value d. The seamless stitching method thus obtained ensures the compliance of source and destination boundaries. As a result, the source image part is combined with the destination image seamlessly. The method has been used extensively in computer vision and is adopted in this study. Moreover, it has a good property for our applications here, i.e., it revises only the pixel values of the source image part to suit the destination image part. With this useful property, we can ensure that the information outside the protected region R is kept intact for later privacy-sensitive image recovery.
More specifically, in the proposed method for seamless image stitching, we manipulate only the pixel values in the stitching border by the above-mentioned method to achieve the aim of composing camouflage image seamlessly. After the
63
process of adjusting the color appearance of the background image part B, we first decide the width of the stitching border by the width or height of the resulting B, and then combine the internal portion of B (excluding the stitching border) and the currently-processed surveillance image frame T in which the privacy-sensitive image part has been removed into a new image B′. And then, we take B′ as the new destination image and the remaining portion of B (the stitching border) as the source image as inputs into the algorithm of Poisson image editing [30] to revise the pixel values in the stitching border in order to generate a new background image part B′′
which does not cause the visible stitching seams in the camouflage image later. After these steps, a modified background image part B′′ is obtained. The above-described steps are illustrated in Figure 4.5, and included in an algorithm given below.
Figure 4.5 Illustration of proposed method for generating a new background image.
64
Algorithm 4.1: editing the background image part which does not cause the visible stitching seams in the camouflage image later.
Input: a background image part B, a surveillance image T, a protected region R.
Output: a modified background image part B′ which does not cause the visible stitching seams in the camouflage image.
Steps.
Step 1. For each pixel P of T in R, set the color value of P to be (0, 0, 0).
Step 2. Perform the following steps to compute the width Ws of the stitching border SB.
2.1 Compare the width and height of R and denote the smaller one by d.
2.2 Compute Ws by Ws = 6 d
; and if Ws = 0, assign Ws to be d.
Step 3. Compute an internal region R′of T by subtracting SB form R, which will be filled with the corresponding part in B later.
Step 4. Set the value of each pixel of T in R′ to be the value of the corresponding pixel in B to compose an image as the destination image ID for the seamless image editing.
Step 7. Compose the modified IS and the pixels of B in R′ into the new background image part B′ and output B′.
65
C. Detection of moving objects and decision of protected region in a surveillance image.
Before detecting the moving objects in the surveillance image, a surveillance image without moving objects should be captured, called a background image, which is the first frame of the video, and each of the currently-processed surveillance images is called a foreground image. An example of background and foreground images is shown in Figures 4.6(a) and 4.6(b).
First, we transform the background and foreground images, which are color ones, into two grayscale images. Second, by subtracting the background image from the foreground one, we obtain a difference image between the two images. And then we set an appropriate threshold parameter Td to threshold the difference image to detect the regions of the moving objects in the foreground image; if the difference value of a pixel is larger than the parameter Td, it is recorded as “1”; else, as “0”. After this step, a binary image IB is obtained. Third, because there is a lot of noise in the surveillance area, we apply consecutively two operations of morphology to the resulting binary image IB to eliminate noise. One is erosion and the other is dilation. An erosion operation can eliminate the noise in the shape, and a dilation operation can mend unconnected area to make the shape smooth. Then, we use a region growing method to connect related regions in IB to form several candidate motion regions. After these steps, we will obtain a binary image IB with detected moving objects labeled by “1.”
To decide the protected region R, we first need to find out the evident motion activities from the resulting binary image IB so that we compute the area Ai of each candidate motion region in IB and select a threshold parameter Ta to decide whether the candidate motion region should be protected or not; if Ai is larger than Ta, then the candidate motion region is protected; otherwise, not. If a certain candidate motion region is not decided to be protected, we will revise the pixel values of the region in IB
66
to be “0.” After this step, we will obtain a binary image IB′ with protected motion regions labeled as “1.” Consequently, we scan IB′ to find out the smallest start point and the maximum end point to decide the protected region R automatically. Finally we will circumscribe R with a rectangle and preserve the stitching border by magnifying the region of R.
A sequence of images yielded as intermediate results of the above-mentioned
A sequence of images yielded as intermediate results of the above-mentioned