使用相似照片進行照片品質評估

全文

(1)國立台灣師範大學資訊工程研究所碩士論文. 指導教授：葉梅珍. 博士. 使用相似照片進行照片品質評估 Photo Quality Assessment with Similar Images. 研究生：中華民國. 莊淳卉. 撰. 年. 7. 102. 月.

(2) 摘要使用相似照片進行照片品質評估莊淳卉. 照片品質評估問題是對於照片的美感來給照片打分數，在之前的研究中，對於照片品質評估都是從單張照片擷取特徵值。有別以往的實驗，在本論文中我們提出使用谷歌圖片搜索 (Google Image Search) 以圖搜圖找出相似照片來提升照片排名準確度。我們提出了從一組類似的照片中提取新的特徵值：相對 (relative) 特徵值和標準差 (standard deviation) 特徵值來輔助照片評分。我們使用兩個回歸模型：支持向量回歸（SVR）和排序支持向量機（Ranking SVM）從視覺特徵學習出評分器去預測照片的分數。在實驗中，我們從 Photo.net 收集 9,000 照片，每張照片使用 Google Image Search 搜索前 10 張相似的圖像，在我們的資料庫中總共有 99,000 張圖像。我們研究哪些種類的照片用類似的圖像有利於照片評分。. 關鍵字：支持向量回歸、排序支持向量機、照片品質評估. ii.

(3) ABSTRACT Photo Quality Assessment with Similar Images by Chuang Chun-Hui. In this paper, we explore using Google Image Search that finds multiple similar images to facilitate the photo quality assessment problem. In particular, we present relative and standard deviation features that are extracted from a group of similar images. We further evaluate two regression models: support vector regression (SVR) and ranking support vector machine (RankSVM) to predict aesthetic scores from visual features. In the experiment, we collected 9,000 images from Photo.net, and for each image we collected 10 similar images using Google Image Search. We examine and identify cases where similar images would benefit the photo quality assessment task.. Keywords: support vector regression; ranking support vector machine; photo quality assessment.. iii.

(4) 誌. 謝. 首先能完成此碩士論文研究最要感謝我的指導教授葉梅珍老師，在我讀研究所的這段時間不管是在專業知識上帶領我如何做研究並指引我正確的研究方向，或是在其他方面給予我很多的機會與幫助，像是每星期實驗室的 Meeting 和參加過內或國外的研討會，讓我在這期間成長很多，老師並且在我們身上花了相當多的時間與精力，從中讓我也向老師學習到非常多，希望之後能跟老師一樣做每件事都完美有效率。並感謝 MM Lab 的學長們: 銘琦、宇辰、志傑和文博教會我許多研究相關的知識，讓我學習成長的更快速，感謝 Lab 的同學: 柏逸、百琮、浩禎、媖詞陪我一起學習成長完成研究，大家一起討論並完成事情還蠻有趣的，也感謝 Lab 的學弟妹: 晏瑋、尹廷、曉薇、少廷帶來很多歡樂並幫我們做很多雜事，省去我們許多的麻煩與時間。還有感謝口試委員花凱龍老師與陳昇瑋老師給的有用意見，讓我的論文更臻於完整。因為有你們讓我順利過完充實的碩士生活。很幸運能在碩士期間到瑞典烏普薩拉大學當交換學生一年，這之中讓我變得更加獨立，也認識了許多的人和得到許多人的幫助。最後也最重要的是感謝我的家人讓我完成碩士學位，無論是精神或物質上的支持，能有任何的小成果，都要歸功於你們。接下來要往人生的下一個階段邁進，謝謝你們陪伴我一路向前。. 淳卉 2013/08/17. iv.

(5) TABLE OF CONTENTS LIST OF TABLES ................................................................................... vi LIST OF FIGUERS ................................................................................. vii Chapter 1 Introduction ...................................................................... …. 9 Chapter 2 Related Work .................................................................... ….13 Chapter 3 Aesthetic Features............................................................. ….15 3.1 Texture ................................................................................................... 15 3.2 Blur Metric............................................................................................. 16 3.3 Relative Feature ..................................................................................... 18 3.4 Standard Deviation Feature ................................................................... 20. Chapter 4 Regression Models ................................................................. 23 4.1 Support Vector Regression .................................................................... 23 4.2 Ranking Support Vector Machine ......................................................... 25. Chapter 5 Experiment ............................................................................. 27 Chapter 6 Conclusion .............................................................................. 36 References ................................................................................................. 41. v.

(6) LIST OF TABLES Table 3-1 Examples of the mean brightness of one image in two groups (a) and (b), then calculated that. in the group (a) and. in the. group (b)............................................................................................................... 20 Table 3-2 Examples of the mean brightness of one image in two groups (a) and (b), then calculated that. in the group (a) and. in the. group (b), although group (a) and group (b) their relative feature are the same, however we can use standard deviation feature to get the difference of these two groups, and then calculated that. in the group (a) and. in the group (b) ....................................................................................... 22 Table 5-1 Score prediction result with single feature and state-of-the-art methods in the Regression ............................................................................................................ 30 Table 5-2 Score prediction result with state-of-the-art methods and multiple features in the Regression ...................................................................................................... 30 Table 5-3 Score prediction result with single feature and state-of-the-art methods in the Ranking SVM ...................................................................................................... 31 Table 5-4 Score prediction result with state-of-the-art methods and multiple features in the Ranking SVM ................................................................................................ 31. vi.

(7) LIST OF FIGURES Figure 1-1 Examples of searching for similar images using Google Image Search. The left images are query images and the right images are those returned by the search engine. The results can be categorized into two cases: (a) returned images are similar to the query, and (b) returned images are not similar to the query .......... 11 Figure 1-2 The system flowchart ................................................................................. 12 Figure 2-1 The interface of ACQUINE [6].................................................................. 14 Figure 3-1 Frequency region division with HSV filter [11] ........................................ 16 Figure 3-2. The blur value of two images. The left image is blurry and the blur value is 0.2606. The right image is sharp and the blur value is 0.8692 ............................ 17 Figure 3-3 X axle is the blur metric value of each image in our dataset and Y axle is the image score of ground truth ........................................................................... 17 Figure 3-4 Examples of the mean brightness of one image in two groups (a) and (b), the left most image of group (a), its. = 0.6, and the mean brightness. value is lower (the darkest) than other images in the same group, but the left most image of group (b), its. = 0.6, and the mean brightness value is higher. (the brightest) than other images in the same group ............................................ 19 Figure 3-5 Examples of the mean brightness of one image in two groups (a) and (b), the left most image of group (a), its. = 0.8, and the mean brightness. value is closer other images in the same group, but the left most image of group (b), its. = 0.8, and the mean brightness value is away from other images. in the same group ................................................................................................. 21 Figure 5-1 The score of images in our dataset ............................................................. 27 Figure 5-2 The number of image score in our dataset ................................................. 28 Figure 5-3 Some examples in our dataset. We collected 9,000 images from Photo.net ............................................................................................................................ 29 vii.

(8) Figure 5-4 The comparison of the rating performance of SVR and RankSVM in terms of the Spearman coefficient ................................................................................. 32 Figure 5-5 The comparison of the rating performance of SVR and RankSVM in terms of the Pearson coefficient..................................................................................... 32 Figure 5-6 The comparison of the rating performance of SVR and RankSVM in terms of the Kendall’s Taub coefficient ........................................................................ 33 Figure 6-1 Some examples of landscape photos. The left images are query images and the right images are those returned by the search engine .................................... 37 Figure 6-2 Some examples of photos with clear topics. The left images are query images and the right images are those returned by the search engine ................. 37 Figure 6-3 Some examples of landmark photos. The left images are query images and the right images are those returned by the search engine .................................. 38 Figure 6-4 Some examples of blurred photos. The left images are query images and the right images are those returned by the search engine .......................................... 39 Figure 6-5 Some examples of dark photos. The left images are query images and the right images are those returned by the search engine .......................................... 39 Figure 6-6 Some examples of photos with many small colorful blocks. The left images are query images and the right images are those returned by the search engine . 40. viii.

(9) CHAPTER 1 INTRODUCTION. People nowadays have one or more, electronic devices (i.e. camera, smart phone) with the capability of taking photographs. In occasion, where we go traveling, have delicious food, or in special events such as a graduation ceremony, we take pictures and record the daily life with photos. Photos can also be uploaded to social websites to share with friends, seamlessly. It would be beneficial to know if the taken photographs are beautiful or not, in other words, is a photo is taken professionally in terms of the shooting technique. For this reason, there exist a number of websites where a user can communicate with other people and receive comments or feedbacks on a photo. Given the case where the number of photographs under evaluation is large, an automatic approach for effectively rating a photograph based on its aesthetic quality is important. In this paper, we propose a new method for such a task. It is different from existing approaches in which features are extracted from the photo under evaluation alone. Instead, we propose to extract the features from a number of similar photos, and argue that evaluation on a group basis could help the prediction of a photo’s aesthetic quality. In the implementation, we use Google Image Search to find similar images where in some cases the returned images are not relevant to the query image, which creates new and additional challenges for the task. For example, Figure 1-1 shows that using Google Image Search to find similar images and Figure 1-2 shows our system flowchart. The remainder of this paper is organized as follows. Chapter 2 contains a review of related works. In Chapter 3, we present our implementation of state-of-the-art 9.

(10) aesthetic features for photo quality assessment and describe our new feature design based on a group of similar images. We then describe the regression model in Chapter 4. Finally, Chapter 5 provides experimental results and performance comparisons, followed by conclusions that summarize our findings in Chapter 6.. 10.

(11) (a). (b) Figure 1-1. Examples of searching for similar images using Google Image Search. The left images are query images and the right images are those returned by the search engine. The results can be categorized into two cases: (a) returned images are similar to the query, and (b) returned images are not similar to the query. 11.

(12) Figure 1-2. The system flowchart.. 12.

(13) CHAPTER 2 RELATED WORK. Recently there are various approaches proposed for solving the photo quality assessment problem [12][9][4][6][13][10]. In [12], Tong et al. proposed the usage of low-level features in images, including blur, intensity contrast, and colorfulness features, which are combined with a classifier to classify photos into two classesprofessional and amateur photos. Ke et al. [9] proposed high-level features, such as color distribution, hue, blur, contrast and brightness, and trained a module for classification. Datta et al. [4] proposed features based on composition rules and image retrieval, such as rule of thirds and aspect ratio. ACQINE [6] was the first automatic online photo rating system, to do real-time image quality assessment. In addition to using the global characteristics of the image, Figure 2-1 shows the interface of ACQUINE. Wong et al. [13] also used the exposure value from the subject area, sharpness and texture. Luo et al. [11] designed and compared the global features and regional features in different types of photos, such as animals, plants, buildings and human.. 13.

(14) Figure 2-1. The interface of ACQUINE [6].. In summary, most of the existing approaches capture image features, train a module, and sort the photos according to the score obtained from the regression. A study in [15] shows that using similar pictures helps the photo rating process. In this paper, we propose new features based on group comparison, and investigate cases where similar photos are beneficial for photo quality assessment.. 14.

(15) CHAPTER 3 AESTHETIC FEATURES. In this chapter, we present two types of features implemented in our system—texture and blur metric, which have been shown to perform well for photo quality assessment in previous works. We then describe the proposed relative and standard deviation features extracted from a group of similar images. All features are normalized before being used as input to a regression model.. 3.1. Texture. According to previous studies in [14], the texture feature performs the best among a set of features under testing for the photo quality assessment problem. In the implementation, we developed the MPEG-7 Homogeneous Texture [11] as our texture features. This feature vector has 32-dimensions. The first dimension represents the mean brightness and the second dimension represents the standard deviation of the brightness values of an image. The remaining 30 dimensions represent the energy and the energy deviation values computed by the Fourier Transform of the image. These features are extracted from partitioned frequency channels based on the human visual system (HSV) see Figure 3-1.. 15.

(16) Figure 3-1. Frequency region division with HSV filter [11].. 3.2. Blur Metric. Whenever a photo is taken out of focus, it usually becomes blurred. The degree of blur has an impact on how we perceive a photo. We use the method presented in [3] to measure the clarity of a photo. We assign a greater score to a sharper photo, and vice versa. See Figure 3-2 the blur value ranges from zero to one. The main idea of this blur estimation approach is to use low-pass filter on the input photo and to analyze the neighborhood variations of each pixel. If we blur a sharp image, gray levels of neighboring pixels will significantly be changed. Otherwise, if we do the same procedure on a blurry image, the pixel variations still exist, but the changes should be smaller than a sharp image. See Figure 3-3 shows X axle is the blur metric value of each image in our dataset and Y axle is the image score of ground truth.. 16.

(17) Figure 3-2. The blur value of two images. The left image is blurry and the blur value is 0.2606. The right image is sharp and the blur value is 0.8692.. Figure 3-3. X axle is the blur metric value of each image in our dataset and Y axle is the image score of ground truth.. 17.

(18) 3.3. Relative Feature. In previous works on photo quality assessment, the evaluation is mostly conducted on a single image directly. In this study, we instead use multiple images for the task. We collected image from photo.net and Google Image Search for creating the dataset. We argue that the same type of images could be used to facilitate the photo quality assessment problem. Our previous study in [15] shows that when we score an image, most people agree that the scoring is more effective if there exist similar images for reference and comparison. Therefore, in [15] we proposed the relative feature that is one additional one–dimensional feature given a normalized aesthetic feature (e.g. the blur metric). In this paper we extend the idea to multiple dimensions and exploit the idea of comparisons among a number of similar photos. We compute the feature differences among every image in the same group. The purpose of the relative feature is to locally normalize the original feature. The equation is defined as follow:. ∑. where. (. ). is one particular feature extract from image i, and N is the number of images. in the same group. In our experiment, we used the top 10 returned images from the Google Image Search engine. See Figure 3-4 and Table 3-1 show examples of relative feature. Figure 3-4 Examples of the mean brightness of one image in two groups (a) and (b), the left most image of group (a), its. = 0.6, and the mean brightness. value is lower (the darkest) than other images in the same group, but the left most 18.

(19) image of group (b), its. = 0.6, and the mean brightness value is higher (the. brightest) than other images in the same group. Table 3-1 Examples of the mean brightness of one image in two groups (a) and (b), then calculated that in the group (a) and. in the group (b).. (a). (b) Figure 3-4. Examples of the mean brightness of one image in two groups (a) and (b), the left most image of group (a), its. = 0.6, and the mean brightness value is. lower (the darkest) than other images in the same group, but the left most image of group (b), its. = 0.6, and the mean brightness value is higher (the brightest). than other images in the same group.. 19.

(20) Table 3-1. Examples of the mean brightness of one image in two groups (a) and (b), then calculated that. in the group (a) and. in the. group (b). Group (a). 1. 2. 3. 4. 5. Sabsolute. 0.6. 0.8. 0.8. 1.0. 1.0. Group (b). 1. 2. 3. 4. 5. Sabsolute. 0.6. 0.4. 0.4. 0.2. 0.2. 3.4. Standard Deviation Feature. The standard deviation (STD) feature shows how much variation or dispersion a feature value exists from the mean value of a group images. A low standard deviation value indicates that the data points tend to be close to the mean; on the other hand, a high standard deviation value indicates that the data points spread out over a large range of values. The STD feature is defined as follow:. ∑ √. (. ̅). (. ). where Xi is one particular feature extract from image i, and N is the number of images in the same group. Figure 3-5 and Table 3-2 show examples of standard deviation feature. Figure 3-5 Examples of the mean brightness of one image in two groups (a). 20.

(21) and (b), the left most image of group (a), its. = 0.8, and the mean brightness. value is closer other images in the same group, but the left most image of group (b), its = 0.8, and the mean brightness value is away from other images in the same group. Table 3-2 Examples of the mean brightness of one image in two groups (a) and (b), then calculated that. in the group (a) and. in the. group (b), although group (a) and group (b) their relative feature are the same, however we can use standard deviation feature to get the difference of these two groups, and then calculated that. in the group (a) and. in the group (b).. (a). (b) Figure 3-5. Examples of the mean brightness of one image in two groups (a) and (b), the left most image of group (a), its. = 0.8, and the mean brightness value is 21.

(22) closer other images in the same group, but the left most image of group (b), its = 0.8, and the mean brightness value is away from other images in the same group.. Table 3-2. Examples of the mean brightness of one image in two groups (a) and (b), then calculated that. in the group (a) and. in the group. (b), although group (a) and group (b) their relative feature are the same, however we can use standard deviation feature to get the difference of these two groups, and then calculated that. in the group (a) and. in the group (b).. Group (a). 1. 2. 3. 4. 5. Sabsolute. 0.8. 0.5. 0.5. 0.5. 0.5. Group (b). 1. 2. 3. 4. 5. Sabsolute. 0.8. 0.9. 0.5. 0.3. 0.3. 22.

(23) CHAPTER 4 REGRESSION MODELS. As described in Chapter 1, we formulate the photo quality assessment problem as a machine learning problem. In our implementation we have tested two regression models: Support Vector Regression (SVR) [7] and Ranking Support Vector Machine (RankSVM) [8].. 4.1. Support Vector Regression. We used support vector regression (SVR) [7] to learn the mapping of a feature vector and the aesthetic score. Then, we use the learnt function to predict the score for a new photo. The idea of support vectors can be used not only in the classification problems, but can also be used to deal with the regression issue. The term “regression” refers to the case where each instance corresponds to a label which is a continuous real number rather than a discrete category. We used the linear regression model developed in LIBSVM [14] to construct the model. Assuming training data is expressed as (. )(. ). (. ). ,. where xi represents the input image features, yi represent the score of each image. We would like to learn a linear regression model: ( ) For each instance xi if the difference between f(xi) and yi is small, then we know that f(xi) can predict yi from xi accurately. The problem is thus formulated as follows:. 23.

(24) (. ). (. ). (. ). where ε ≥ 0 indicates the largest gap of the predicted and the actual values, and therefore the algorithm is called ε-SVR. SVR considers that the difference between the predicted value and the actual value should be less than ε. The goal is to find the optimum space plane (hyperplane) where SVR is able to accurately predict the distribution of data points. In circumstances where the solution can be obtained by solving the formula (4-2), these conditions are known as feasible. However, in some situations, the data may be noisy and contain some errors, and the situation is not feasible. Therefore we have to add additional items to allow instances that fall outside ε.. ∑(. {. ). (. ). (. ). In formula (4-4), each training instance has its corresponding. and. , that are. slack variables used to determine the training instance can fall outside the range of ε. The constant C > 0 is used to adjust the training model to prevent the overfitting or underfitting problem.. 24.

(25) 4.2. Ranking Support Vector Machine. In the second approach, we used ranking support vector machine (RankSVM) [2] to learn the ranking function. RankSVM is one of the pair-wise ranking methods, which learns a wide-margin ranking function that enforces the desired ordering on training points. Given a set of photo, and let X = {x1, x2,…, xn} be the features that are extracted from an image in G, where xk is normalized and represents the kth type of feature and Y= {y1, y2,…,yn} be the score of each image. We give two sets of images: one is ordered pairs Om ={(i, j)} and another is unordered pairs Um ={(i, j)} such that (. ). , i.e. the score rank of image i is higher than image j, and. (. ). i.e. the score rank of image i is equal to image j. We seek to. automatically learn the ranking function:. ( ). (. ). ( ). (. ). ( ). (. ). with the following constraints being satisfied:. The optimization formulation of Ranking SVM is similar to SVM and we can solve such a problem with the regular SVM solver as follows:. 25.

(26) (∑. ). ( { |. ( ). |. (. ). (. ). ). where C determines the trade-off between maximizing the margin and satisfying the pairwise relative constraints. In the implementation, we used the absolute difference value of the ground-truth scores for a pair of training images as the penalization for the training error. That is, if two images are mapped to the output space (the predicted score) where the order is not consistent with the original score, the training error should be proportional to the original score difference.. 26.

(27) CHAPTER 5 EXPERIMENT. The dataset used in our experiments is collected from Photo.net [5] that contains 9,000 photos. Each photo is associated with a quality score obtained from the web users. We only used the photos that have received at least 10 ratings in order to ensure that the photograph had a greater stability and representativeness in the average aesthetics scores. Figure 5-1 shows the score of images in the photo.net and Figure 5-2 shows the number of image score in our dataset and Figure 5-3 shows some examples in our dataset. Most photos in the Photo.net dataset scores most fall in the middle of the score values (5 points). The highest score is 7 points, and the lowest score is 0. For each image we used the Google Image Search engine to collect 10 top similar images. These images do not have an aesthetic score. We used them to calculate the relative and the STD features. In summary, we have in total 99,000 images in our dataset.. Figure 5-1. The score of images in our dataset.. 27.

(28) Figure 5-2. The number of image score in our dataset.. We used five-fold cross validation for model training and testing in our experiment. We have partitioned dataset into five subsets with equal sizes (1,800 images in each subset), denoted as F1, F2, F3, F4, and F5, respectively. We used three subsets for training, from which we learned the regression/ranking models. One subset is used for parameter tuning, and the remaining subset is used to evaluate the rating performance.. 28.

(29) Figure 5-3. Some examples in our dataset. We collected 9,000 images from Photo.net.. 29.

(30) Table 5-1. Score prediction result with single feature and state-of-the-art methods in the Regression. Feature. Spearman. Pearson. Kendall’s Tau-b. Texture (32). 0.2943. 0.2921. 0.2001. Blur (1). 0.2038. 0.2141. 0.1366. Texture Relative (32). 0.2772. 0.2748. 0.1862. Texture STD (32). 0.2506. 0.2486. 0.1689. Texture + Blur (33). 0.3232. 0.3215. 0.2180. Table 5-2. Score prediction result with state-of-the-art methods and multiple features in the Regression.. Feature. Spearman. Pearson. Kendall’s Tau-b. Texture + Blur (33). 0.3232. 0.3215. 0.2180. Texture + Blur + Texture Relative (65). 0.3345. 0.3390. 0.2265. Texture + Blur + Texture STD (65). 0.3362. 0.3425. 0.2276. Texture + Blur + Texture Relative + Texture STD (97). 0.3368. 0.3429. 0.2284. 30.

(31) Table 5-3. Score prediction result with single feature and state-of-the-art methods in the Ranking SVM. Feature. Spearman. Pearson. Kendall’s Tau-b. Texture (32). 0.2936. 0.2904. 0.1995. Blur (1). 0.2038. 0.2142. 0.1366. Texture Relative (32). 0.2797. 0.2776. 0.1880. Texture STD (32). 0.2414. 0.2334. 0.1624. Texture + Blur (33). 0.3234. 0.3216. 0.2183. Table 5-4. Score prediction result with state-of-the-art methods and multiple features in the Ranking SVM.. Feature. Spearman. Pearson. Kendall’s Tau-b. Texture + Blur (33). 0.3234. 0.3216. 0.2183. Texture + Blur + Texture Relative (65). 0.3580. 0.3591. 0.2427. Texture + Blur + Texture STD (65). 0.3540. 0.3551. 0.2395. Texture + Blur + Texture Relative + Texture STD (97). 0.3622. 0.3631. 0.2458. Yeh et al., IEEE ICIP 2012. 0.3258. 0.3238. 0.2201. 31.

(32) Spearman. Spearman 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0. Regression RankSVM. Features. Figure 5-4. The comparison of the rating performance of SVR and RankSVM in terms of the Spearman coefficient.. Pearson. Pearson 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0. Regression RankSVM. Features. 32.

(33) Figure 5-5. The comparison of the rating performance of SVR and RankSVM in terms of the Pearson coefficient.. Kendall's tau-b 0.3. Kendall's tau-b. 0.25 0.2 0.15 0.1 Regression. 0.05. RankSVM. 0. Features. Figure 5-6. The comparison of the rating performance of SVR and RankSVM in terms of the Kendall’s Taub coefficient.. In the evaluation, we used the Spearman, the Pearson and the Kendall’s Tau-b coefficients to measure the rating results. The Spearman coefficient is a measure of statistical dependence of two variables (in our case, the predicted score and the ground-truth score), while the Pearson coefficient measures the correlation between two linearly dependent rankings. The Kendall’s tau-b coefficient is a statistic used to measure the association between two measured quantities. These three coefficients range from -1 to 1.When the coefficient is -1, it means that a 100% negative. 33.

(34) correlation; when the coefficient is 1, it means a 100% positive correlation. When two ranked lists are more consistent, the coefficient value should be closer to 1 (positive correlation). If the coefficient value is 0, it indicates no correlation. Therefore, if the coefficient value is farther from 0, the result is better. Table 5-1 and Table 5-2 show the rating and the ranking results with single feature and multiple features in the Regression. Table 5-3 and Table 5-4 show the rating and the ranking results with single feature and multiple features in the Ranking SVM. The baseline method (texture + blur) is 0.3234 (spearman), 0.3216 (pearson) and 0.2183 (kendall’s tau-b), which is better than using any single feature. The combination of the developed four features gets a performance of 0.3622 (spearman), 0.3631 (pearson) and 0.2458 (kendall’s tau-b), which is the best among any other combinations. In both of the SVR and RankSVM cases, relative features and STD features improve the ranking performances. Figure 5-4 shows the coefficient of the Spearman both in SVR and RankSVM, Figure 5-5 shows the coefficient of the Pearson both in SVR and RankSVM and Figure 5-6 shows the coefficient of the Kendall’s Tau-b both in SVR and RankSVM. The result performance of single or two features in SVR and RankSVM almost the same, but the result of multiple features, RankSVM is worked well than SVR. It shows that no matter which regression models we applied, the proposed relative and the STD features are effective as they provide complementary information for the task. To understand how state-of-the-art methods perform on our dataset, we reported the ranking performance of [15] using our dataset. We used the features based on the work in [15] and obtained a ranking list of all images in our dataset. The results show that their methods have an unsatisfactory performance, 0.3258 (spearman), 0.3238 (pearson) and 0.2201 (kendall’s tau-b). To compare with our method, our performance. 34.

(35) is good than [15]. However, we should notice that our systems were trained with a different dataset (mostly professional photographs), and it explains the significant disparity between the results reported in the papers and those of real-world field tests.. 35.

(36) CHAPTER 6 CONCLUSION. In the paper we study the photo quality assessment problem with similar photos. The experimental results show that the use of multiple photos for photo rating is effective; however, the impact of group comparison using similar images is content dependent. In general, the use of grouped photos can better predict the score in the following cases: landscape photos (see Figure 6-1), photos with clear topics (see Figure 6-2), and landmark photos (see Figure 6-3). However, we also observe that the proposed features are less effective in blurred photos (see Figure 6-4), dark photos (see Figure 6-5), and photos with many small colorful blocks (see Figure 6-6). We will investigate a hybrid approach as a future work.. 36.

(37) Figure 6-1. Some examples of landscape photos. The left images are query images and the right images are those returned by the search engine.. 37.

(38) Figure 6-2. Some examples of photos with clear topics. The left images are query images and the right images are those returned by the search engine.. Figure 6-3. Some examples of landmark photos. The left images are query images and the right images are those returned by the search engine.. 38.

(39) Figure 6-4. Some examples of blurred photos. The left images are query images and the right images are those returned by the search engine.. 39.

(40) Figure 6-5. Some examples of dark photos. The left images are query images and the right images are those returned by the search engine.. Figure 6-6. Some examples of photos with many small colorful blocks. The left images are query images and the right images are those returned by the search engine.. 40.

(41) REFERENCES. [1] Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27, 2011. [2] Olivier Chapelle. Training a support vector machine in the primal. Neural Computation, 19(5):1155-1178, 2007. [3] F. Crete, T. Dolmiere, P. Ladret, and M. Nicolas. The blur effect: perception and estimation with a new no-reference perceptual blur metric. Human Vision and Electronic Imaging XII, 6492:11. [4] R. Datta, D. Joshi, J. Li, and J.Wang. Studying aesthetics in photographic images using a computational approach. Computer Vision ECCV 2006, pages 288-301, 2006. [5] Ritendra Datta, Jia Li, and James Ze Wang. Algorithmic inferencing of aesthetics and emotion in natural images: An exposition. In Proc. IEEE ICIP, Special Session on Image Aesthetics, Mood and Emotion, San Diego, CA, 2008. [6] Ritendra Datta and James Z Wang. Acquine: aesthetic quality inference engine-real-time automatic rating of photo aesthetics. In Proceedings of the international. conference. on. Multimedia. information. retrieval,. pages. 421-424.ACM, 2010. [7] H. Drucker, C.J.C. Burges, L. Kaufman, A. Smola, and V. Vapnik. Support vector regression machines. Advances in neural information processing systems, pages 155-161, 1997.. 41.

(42) [8] Thorsten Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133-142. ACM, 2002. [9] Y. Ke, X. Tang, and F. Jing. The design of high-level features for photo quality assessment. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, volume 1, pages 419-426. IEEE, 2006. [10] Wei Luo, Xiaogang Wang, and Xiaoou Tang. Content-based photo quality assessment. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 2206-2213. [11] Y.M. Ro, M. Kim, H.K. Kang, BS Manjunath, and J. Kim. Mpeg-7 homogeneous texture descriptor. ETRI journal, 23(2):41-51, 2001. [12] H. Tong, M. Li, H.J. Zhang, J. He, and C. Zhang. Classi_cation of digital photos taken by photographers or home users. Advances in Multimedia Information Processing-PCM 2004, pages 198-205, 2005. [13] Lai-Kuan Wong and Kok-Lim Low. Saliency-enhanced image aesthetics class prediction. In Image Processing (ICIP), 2009 16th IEEE International Conference on, pages 997-1000. [14] C.H. Yeh, Y.C. Ho, B.A. Barsky, and M. Ouhyoung. Personalized photograph ranking and selection system. In Proceedings of the international conference on Multimedia, pages 211-220. ACM, 2010. [15] Mei-Chen Yeh and Yu-Chen Cheng. Relative features for photo quality assessment. In Image Processing (ICIP), 2012 19th IEEE International Conference on, pages 2861-2864.. 42.

(43)