Image Retrieval and Copyright Notification Based on Multipurpose Watermarking Scheme

Watermarking Scheme

Zhe-Ming Lu 1,* _{and He Wang} 2

1 _{Visual Information Analysis and Processing Research Center}

Harbin Institute of Technology Shenzhen Graduate School Shenzhen, 518055, P. R. China

[email protected]

2 _{Department of Automatic Test and Control}

Harbin Institute of Technology Harbin, 150001, P. R. China [email protected]

Received 19 March 2006; Revised 21 April 2006 ; Accepted 15 June 2006

Abstract. The rapid development of Internet and multimedia technologies has made image retrieval and copyright problem be the two most important issues in the digital world. To solve these problems simultane-ously, this paper presents a multipurpose watermarking scheme: to notify the copyright owner with a visible watermark, and to retrieve the image with an invisible watermark. The proposed scheme consists of two main phases, offline process and online retrieval process. A copyright symbol is used as the visible watermark and the feature vector is extracted from each image as the invisible watermark to be embedded into the image, which is the preprocessing operation called offline process. The online retrieval process consists of three processes, i.e., query feature computation, watermark extraction and feature vector matching. Since the fea-tures are embedded in the image data, it is unnecessary to compute the feafea-tures but only to extract it from the watermarked image. We carry out a series of experiments on a watermarked image database, and simulation results indicate the advantage of the proposed watermarking scheme.

Keywords: image retrieval, copyright notification, watermark

1 Introduction

With the development of computer, multimedia and network technologies, the amount of visual information available in digital format has grown exponentially recently, which has resulted in information explosion and has exceeded the limit of human acceptability. Therefore, two important issues have arisen. First, the introduction of the World Wide Web and the increased memory capacity allow the storage of large amounts of digital data and the need to handle queries and browse in large image databases has become a hotspot. Since the beginning of the 1990’s, there has been an increased research activity in the area of content based image retrieval (CBIR). Both large research teams, for instance, the QBIC project at IBM, the ADVENT project at Columbia University and smaller project groups in the academic world have devoted themselves to this task. The second problem is that there is almost no limit for anyone to make lossless and unlimited copies of digital contents distributed over Internet and via CD-ROM, which is a major obstacle from the owner’s viewpoint for entering the digital world. Copyright has therefore been one of the most important issues in the digital world. Now we concern with the problems in image retrieval and copyright respectively.

In typical Content-Based Image Retrieval (CBIR) systems, the information of the images in the database are extracted and described by multi-dimensional feature vectors. The feature vectors of the images in the database form a feature database. To retrieve images, users provide the retrieval system with example images or sketched figures. The system then changes these examples into its internal representation of feature vectors. The similari-ties between the feature vectors of the query example or sketch and those of the images in the database are then calculated and retrieval is performed with the aid of an indexing scheme. Most existing CBIR systems focus primarily on the feature analysis [1][2][3][4][5], the similarity measure[6][7] and the feedback learning algo-rithm[8][9].

Over the last decade, digital watermarking has become one of the most common ways to deter people copying your images without your permission. A watermark can be classified into two sub-types: visible and invisible. Invisible watermarks operate by embedding information within the image itself. As a rule, watermarks that are less visible are weaker and easier to remove. When choosing a variant it is important to consider the interaction between watermark invisibility and resilience. Invisible watermarks can be broadly classified into two types, robust and fragile (or semi-fragile) watermarks. Robust watermarks [10] are generally used for copyright protec-tion and fragile or semi-fragile watermarks [11] are mainly applied to content authenticaprotec-tion. In comparison, visible watermark is more resilient and may be used to immediately identify copyright without significant effort by the user. A balance should be reached between the need to make the watermark difficult to remove and its use to the user. Although the need for visible watermarking for copyright notification is apparent, visible watermark-ing has received much less attention than invisible watermarkwatermark-ing.

In general, the above two issues are taken into account separately. This paper presents a simple multipurpose watermarking scheme to solve these two problems simultaneously. Copyright symbol and feature vectors of im-age are embedded offline in watermark format. During online retrieval, we can query based on the feature water-mark. The paper is organized as follows: Section 2 provides a brief introduction to the feature computation .The proposed offline multipurpose watermarking scheme is described in Section 3. In Section 4, some experiments are performed based on a test platform for watermarked image retrieval. And finally, we draw a conclusion in Section 5.

2 Feature Computation

Generally, we should extract the available features as many as possible. However, considering the embedding of watermarks will affect the image quality, a few best representative features are chosen here. Generally speaking, similarity between images is measured by computing the difference between their features such as color, shape, texture and spatial properties. This paper concentrates on three kinds of features in RGB space, i.e. the optimum threshold value from color histogram developed by Ridler and Calvard [12], texture feature based on grey level co-occurrence matrix (GLCM) [13] and Hu moments [14], which can be described in detail as follows.

2.1 Color Feature

T. W. Ridler and E. S. Calvard presented a method of image threshold value in previous work which was further mathematically developed by H. J. Trussel. The principle of this method is to evaluate the unique threshold

T

for any image with a bimodal histogram, by assuming the threshold to be: ( _{0 1}) / 2

T = µ +µ .

Whereµ₀andµ₁are the means of each of the two components of the histogram separated by the threshold. Calculation of the threshold value analytically from the histogram is not possible, because the means of the two parts can be evaluated only after the threshold is determined, while the threshold needs to be computed from the two means. Therefore, an iterative algorithm was suggested: first, an initial threshold is selected (the mean of the entire histogram seems to be sufficient as a starting point), then the two means for the two distributions on either side of the threshold are calculated; a new threshold is obtained by averaging these means. The process continues until the value of the threshold converges. The algorithm is described as follows:

1. Select an initial threshold T (e.g. the mean intensity). ₀ 2. Partition the image into two groups (R and₀ R ) using the₁ T . ₀

3. Calculate the mean intensity values µ₀and µ₁ of the partitions R and₀ R : ₁

( )

( )

0 0 0 T i h i di T h i di µ = ∫ ⋅ ∫

( )

( )

1 N i h i di T N h i di T µ = ∫ ⋅ ∫

Where

i

: the gray level of the pixels (varying from 0 to

N

). h i : the histogram weighting for every gray level.

( )

T: the current threshold value.

4. Select a new threshold _Ti =(µ₀+µ₁) / 2.

With the threshold value

T

, the histogram is separated into two sections. We can acquire from literature [12] that the final threshold value are the best possible solution for dividing the histogram while preserving the image average luminance. In the paper, we use the normalized optimum threshold value: T N/ and the value

( )

( )

0 0 T h i di N h i di ∫

∫ as the color feature (i.e. double-typed value).

2.2 Texture Feature

Image texture, defined as a function of the spatial variation in pixel intensities (grey values), is useful in a variety of applications and has been a subject of intense study by many researchers. One immediate application of image texture is the recognition of image regions using texture properties. Statistics-based method, structure-based method and spectrum-based method are put forward. Statistic method refers to carrying out texture analysis in the condition of unknown the basic cell of texture, and it mainly describes the basic cell of texture or random and spatial statistic character in local pattern, such as GLCM (Grey Level Co-occurrence Matrices), wave transform, fractal representation, “visual” properties random field models and other representation.

Haralick [13] suggested the use of grey level co-occurrence matrices (GLCM) to extract second order statistics from an image. GLCM has been used very successfully for texture classification in evaluations. Haralick defined the GLCM as a matrix of frequencies at which two pixels, separated by a certain vector, occur in the image. The distribution in the matrix will depend on the angular and distance relationship between pixels. Varying the vector used allows the capturing of different texture characteristics. Once the GLCM has been created, various features can be computed from it. 14 statistical measures of texture can be extracted from the matrix into a feature vector, i.e. Inverse Difference Moment, Energy (Angular Second Moment), Contrast, Correlation, Entropy. Here we choose three most commonly used features, listed in Table 1, for our evaluation.

Table.1 Features calculated from the normalized co-occurrence matrix P(i,j)

Feature Formula

Contrast

∑ ∑

_{i j}

(

i

−

j

) ( )

2

P i j

,

Energy

∑ ∑

_{i j}

P

2

( )

i j

,

Entropy

∑ ∑

_{i j}

P i j

( )

,

log

P i j

( )

,

2.3 Shape Feature

Hu moments are a set of algebraic invariants that combine regular moments [14]. They are invariant under change of size, translation, and rotation. Hu moments have been widely used in pattern recognition and proved successful in various applications. These moments can be used to describe the shape information of the image. If the object R is represented as an image, then the central moment of order

p

+

q

for the shape of the object R is defined as, ( ) ( ) , ( , ) p q x x y y p q c c x y R µ = ∑ − − ∈ . (1)

Where

(

x_c,y_c

)

is the center of the image. This central moment can be normalized to be scaling invariant as,

00 pq pq r µ η µ = 2 2 p q r= + + . (2)

Based on these moments, a set of moments invariant to translation, rotation and scaling can be derived, we only use first 2 moments for luminance component (i.e. two double-typed values) as follows:

1 20 02

2 2

( ) 4

2 20 02 11

φ = η − η + η . (4)

3 The proposed offline multipurpose watermarking

In our system, before we perform the online retrieval, we first embed two kinds of watermarks into each image in the database. These two watermarks possess different purposes, which is why our watermarking is called a multi-purpose watermark scheme. One watermark is a visible copyright symbol watermark, which is used for copyright notification. The other is the invisible feature watermark, which is composed of the extracted features.

These two watermarks are embedded in different blocks with different methods [15]. For convenience, let the original image X and the visible watermark V be 256 gray-level image of size512 512× and128 128× , respec-tively.

3.1. Visible Watermark Algorithm.

1. Divide the original image and the visible watermark into 4096 and 256 blocks of size8 8× , respectively.

Cal-culate the variance

V

_klfor all original image blocks. Find the maximal variance

V

_max, and the minimal vari-ance

V

_min, and calculate the normalized variance by:

min max min V_kl V kl _{V V} α = − − . (5)

2. From the 4096 blocks, we select 256 blocks for visible watermarking. Perform the visible watermarking process in the spatial domain based on the following equation:

(

)

' 1 255 Xij X_ij X_ij V_ij kl kl α α = ⋅ + − ⋅ ⋅ . (6)

Where_{Xij , Xij and Vij denote the pixels at position} '

( )

i j in the block ,

( )

k l of the visibly watermarked image, , the original image and the visible watermark, respectively. α_klis the normalized variance of pixels in the origi-nal image block

( )

k l . ,

3.2. Invisible Watermarking Algorithm

The other blocks are used for invisible watermarking in DCT domain. Here, we normalized the feature value to be in the interval [0, 1]. After changing the feature vector (i.e. watermark) to binary sequences (e.g., we just use 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 0001 0010 to denote a double-typed value 0.012345678912), each bit is inserted by modifying the DC coefficient of each 8 8× block. Here the quantization index modulation (QIM) [16] is adopted, which can solve the distortion problem if an appropriate quantization step is selected. Assume

d

is the quantization step, in this paper we use

d

=24, and

f

is the coefficient to be modified,

w

is the watermark bit. Calculate m=mod[f d/ ] and r= f −m× , and then the modification can d be illustrated as follows:

(a) If m =0 andw = , then1 f =d/ 2.

(b) Else if m =0 andw =0, then f = ×3 d/ 2. (c) Else if m ≠0 and w = , then 1

2 / 2 2 / 2 2 2 / 2 kd d f kd d kd d d + = + + +









f if if

i

2 2 1 2 1 m k m k m k = = + = +

and

and

/ 2 / 2 r d r d ≤ >

(

2 1

)

/ 2 2 / 2 2 / 2 k d d f kd d kd d d + + = − + +











f if if

i

2 1 2 2 m k m k m k = + = = and and / 2 / 2 r d r d ≤ >

The extracting principle is very simple: ifm%2=0 , thenw =0, otherwisew =1. When extracting the water-mark, DC coefficient is extracted in the same order as embedding.

4 Experimental Results

The proposed system has been implemented using Visual C++ 6.0 software. In our experiment, we use a standard image database including 1000 miscellaneous images [17] of size384×256or256 384× , which are classified into ten classes, each class including 100 images. During the offline process, we embed a 128 128× sized binary copyright watermark and 21 double-typed feature values, gaining the dual watermarked image, shown in Fig.1. In the experiment, we combine the feature watermark and the copyright watermark to construct the watermark to be embedded for each database image. After we obtain a watermarked image database, we can perform the online retrieval with various queries. Euclidean distance is employed as the similarity measure.

For the query based on features, we show an example of retrieval results in Fig.2. Most common evaluation measures used in image retrieval are precision and recall [18]. Precision is the number of the retrieved relevant images over the total number of retrieved images, and recall is the number of the retrieved relevant images over the total number of relevant images in the database. The average precision and recall for each class is shown in Table 2. The average precision and recall for each class can be obtained as follows: First, we randomly select ten images from the class. Then, we use each image to be the query image. For each query image, we get the preci-sion by obtaining the ratio of returned relevant images in this class in the first 64 returned images, and find the number of the returned relevant images and divide it by 100 to obtain the recall. After getting ten recalls and ten precisions, we average them to get the average recall and precision.

For our image retrieval system, when carrying out offline operation, a problem arising is the error in bits, even only one bit, may make the extracted value very different from the embedded one. So we must guarantee the extracted feature watermark should be no bit loss. With regard to this, the experiment results show that we can extract the feature watermark 100%similar to the original embedded information without any attacks.

Table.2 The average recall and precision for each class

Class NO. Semantic Average Recall Average Precision 1 People 0.235 0.367 2 Beach 0.220 0.343 3 Building 0.244 0.381 4 Bus 0.351 0.548 5 Dinosaur 0.618 0.965 6 Elephant 0.172 0.268 7 Flower 0.534 0.834 8 Horse 0.203 0.317 9 Mountain 0.152 0.237 10 Food 0.218 0.340

5. Conclusions

This paper proposed a content-based image retrieval system based on a multipurpose watermarking scheme. It can be used for image retrieval and copyright notification simultaneously. This kind of technology is particularly useful if you intend placing image database on your website where you may be required to protect your copyright. The visible watermark obviously notifies your copyright. In addition, the system embeds the features in the im-ages, and we need no extra space to save the feature data. Therefore, the storage space is saved.

6. Acknowledgement

The authors gratefully thank Prof. Jia Li for providing the image database for research.

Fig.1.Visible copyright watermark and dual watermarked image

Fig.2. The retrieval system of our experiment

References

[1] Smith, J. R., “Color for Image Retrieval”, in Castelli V, Beagman L D, eds. Image Databases-Search and Retrieval of Digital Imagery, John Wiley & Sons, Inc. Ch. 11, 2002, pp. 285-311.

[2] Sebe, N., Lew, M. S, “Texture Features for Content Based Retrieval”, in Principles of Visual Information Retrieval, Lew M S, ed. Springer, Ch. 3, 2001, pp. 51-85.

[3] Lee, K. M., Street, W. N., “Incremental Feature Weight Learning and Its Application to A Shape Based Query System”, Pattern Recognition Letters, Vol. 23, No.7, 2002, pp. 265-274.

[4] Sciascio, E. D., Donini, F. M., Mongiello, M., “Spatial Layout Representation for Query by Sketch Content-based Image Retrieval”, Pattern Recognition Letters, Vol. 23, No. 13, 2002, pp. 1599-1612.

[5] Lew, M. S., “Features Selection and Visual Learning”, In Principles of Visual Information Retrieval. Lew, M. S., ed. Springer, Ch. 12, 2001, pp. 297-318.

[6] Qin, X. and YH Yang, “Similarity Measure and. Learning with Gray Level Aura Matrices (GLAM) for Texture Image Retrieval. ICVPR, 2004, pp. 326-333.

[7] Guo-Dong Guo, Jain, AK, Wei-Ying Ma and Hong-Jiang Zhang, “Learning Similarity Measure for Natural. Image Rtrieval with Relevance Feedback”, IEEE Transactions on Neural Networks, Vol. 13, No. 4, July 2002, pp. 811 – 820. [8] Lee, and L. Guan, “Semi-Automated Relevance Feedback for Distributed Content Based Image Retrieval”, Proc. of IEEE

International Conference on Multimedia and Expo, Taipei, Taiwan, pp. 1871-1874, Jun 2004.

[9] Rui Y, Huang TS, Mehrotra S., “Content-Based image retrieval with relevance feedback in MARS”. In Proc. of the IEEE Int'l. Conf. on Image Processing, New York: IEEE Press, 1997, pp. 815-818.

[10] Wang, Y., Doherty, J. F., Van Dyck, R. E., “A Wavelet-based Watermarking Algorithm for Ownership Verification of Digital Images”, IEEE Trans. Image Processing, Vol. 11, No.2, 2002, pp. 77-88.

[11] Jaejin, L., Chee, S. W., “A Watermarking Sequence Using Parities of Error Control Coding for Image Authentication and Correction”, IEEE Trans. Consumer Electronics, Vol.46, No. 2, 2000, pp. 313-317.

[12] T. W. Ridler and E. S. Calvard, “Picture threshold using an iterative selection method”, IEEE Trans. Syst. Man Cybern. Vol. SMC-8, 630-632, Aug. 1978.

[13] Haralick, R., “Statistical and structural approaches to texture”, In Proc. of IEEE. Vol. 67, 1979, pp. 786-804.

[14] S.O. Belkasim, M. Shridhar, and M. Ahmadi, “Pattern recognition with moment invariants: a comparative study”, Pat-tern Recognition, Vol. 24, No. 12, 1991, pp. 1117-1138.

[15] Zhe-Ming LU, Hao-Tian WU, Dian-Guo XU and Sheng-He SUN, “A Multipurpose Image Watermarking Method for Copyright Notification and Protection”, IEICE Transactions on Information and Systems. Vol. E86-D, No. 9, 2003, pp. 1931-1933.

[16] Chen, B., Wornell, G. W., “Digital Watermarking and Information Embedding Using Dither Modulation”, IEEE Sec-ond Workshop on Multimedia Signal Processing, 1998, pp. 273–278.

[17] LI,J.Photography image database. http://www.stat.psu.edu/~jiali/index.download.html.

[18] Müller, H., Müller, W., Squire, D., Marchand-Maillet S., Pun, T., “Performance Evaluation in Content-based Image Retrieval: Overview and Proposals”, Pattern Recognition Letters, Vol. 22, No. 5, 2001, pp. 593–601.