Chapter 6 Determination of Impact Research Topics via the Bayesian Estimation of
6.2 Experiment to Validate the Impact Power of Publications
6.2.3 Comparing the impact power of publications with the experts survey
Recall rate of the model
Recall=A/B 73.19% 71.34% 71.32%
Precision rate of the model
Precision=A/C 17.41% 20.17% 16.72%
Table 6-12 Precision of Each Model While Considering the Lists of Suggested Publications.
Rosen-Zvi,
6.2.3 Comparing the impact power of publications with the experts survey
Comparing the precision or recall rate with the list of authors’ publications as recommended by different models is not the only approach to validate the impact publications. Surveying experts in the same domain is also a good approach. Since the results from the three different models are not fully accurate, the comparison only applies to an acceptable solution. To validate the quality of the proposed publications, we also survey experts who work in the same topic area. The five experts are also interviewed for the impact author part. We give the publication lists of journals and conferences proposed by our Bayesian estimation model to these experts and ask them to determine whether the publication has an impact or not. The scoring method is similar to that used to evaluate the impact authors, where yes is 1, no is -1, and no opinion is represented by 0. The survey results are shown in Table 6-13, Table 6-14 and Table 6-15. The former discusses the impact journals and the latter two are about
Model
Data Description
Model
Data Description
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
impact conferences.
Table 6-13 Comparison of Journals Ranks Between the Experts and Our Model.
ID Title of the publication Scores Experts’
Rank Our Rank
J3 Data Mining and Knowledge Discovery 5 1 1
J6 IEEE Transactions on Knowledge and Data Engineering 5 1 2
J8 Knowledge and Information Systems 4 3 3
J10 SIGKDD Explorations 3 4 4
J4 IEEE Bulletin of the Technical Committee on Data
Engineering(IEEE Data(base) Engineering Bulletin) 1 8 5
J5 IEEE Computers 2 5 6
J11 SIGMOD Record 2 5 7
J2 Communications of the ACM (CACM) 1 8 8
J1 Bioinformatics 0 10 9
J9 Machine Learning 2 5 10
J7 IEEE Transactions on Visualization and Computer Graphics -1 11 11 The comparison results can be separated into 3 parts. The first part contains journals ID J3, J6, J8 and J10. The ranking of the top 4 journals given by the experts are almost the same as those ranked with our model. There are 2 journals, Data Mining and Knowledge Discovery and IEEE Transactions on Knowledge and Data Engineering assigned the same value by experts as first place. Our model gives almost the same results. The second part contains journals ID J4, J5, J11 and J2. The ranks between the experts and our model are also similar. Our rank is 5-8 but the experts’
are 5 or 8 because of they give several journals the same scores and there is opposition. The situation is similar in the third part which contains ID J1, J9 and J7.
The rank 9-11 obtained with our model is similar to the ranking of experts. The exception is the J9, the journal, Machine Learning. The experts give it the higher score a rank of 5. A possible reason for this may be that although the journal is well-known in machine learning and artificial intelligence, and there are some data mining papers discussed in the journal, however the volume published and citation frequency from 1990-2002 does not exceed that of other journals ranked higher rank on the list.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Table 6-14 Comparison of Conference Ratings Between the Experts and Our Model.
No. ID Title of the publication Scores Experts’
Rank
Our Rank 1 C3 ACM SIGKDD Conference on Knowledge Discovery
and Data Mining (KDD) 5 1 1
2 C19 Pacific-Asia Conference on Knowledge Discovery and
Data Mining (PAKDD) 5 1 7
3 C16 International Conference on Information and Knowledge
Management (CIKM) 4 3 9
4 C20 Principles of Data Mining and Knowledge Discovery
(PKDD) 4 3 5
5 C21 SIAM International Conference on Data Mining (SDM) 4 3 10 6 C23 SIGMOD Workshop on Research Issues in Data Mining
and Knowledge Discovery (DMKD) 4 3 14
7 C24 Very Large DataBase (VLDB) 4 3 3
8 C11 IEEE International Conference on Tools with Artificial
Intelligence (ICTAI) 3 8 22
9 C12 Industrial Conference on Data Mining (ICDM) 3 8 4 10 C13 International Conference on Data Engineering (ICDE) 3 8 6 11 C14 International Conference on Data Warehousing and
Knowledge Discovery (DaWaK) 3 8 12
12 C4 ACM Symposium on Principles of Database Systems 2 12 11 13 C6 Advances in Distributed and Parallel Knowledge
Discovery 2 12 21
14 C8 Advances in Neural Information Processing Systems
(NIPS) 2 12 24
15 C17 Lecture Notes in Computer Sciences (LNCS) 2 12 8 16 C18 Lecture Notes in Artificial Intelligence (LNAI) 2 12 13
17 C22 SIGMOD 2 12 2
18 C9 European Conference on Machine Learning (ECML) 1 18 17 19 C15 International Conference on Database Theory (ICDT) 0 19 16 20 C2 ACM International Conference on Digital Libraries -1 20 19 21 C7 Advances in Large Margin Classifiers -1 20 20 22 C10 Genetic and Evolutionary Computation Conference
(GECCO) -1 20 18
23 C1 ACM Conference on Computers and Security -2 23 15 24 C5 Advances in Digital Libraries Conference (ADL) -3 24 23
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Table 6-15 Comparison of Conferences (Higher Than the Average Score).
No. ID Title of the publication Scores Experts’
Rank
Our Rank 1 C3 ACM SIGKDD Conference on Knowledge Discovery
and Data Mining (KDD) 5 1 1
2 C19 Pacific-Asia Conference on Knowledge Discovery and
Data Mining (PAKDD) 5 1 7
3 C16 International Conference on Information and Knowledge
Management (CIKM) 4 3 9
4 C20 Principles of Data Mining and Knowledge Discovery
(PKDD) 4 3 5
5 C21 SIAM International Conference on Data Mining (SDM) 4 3 10 6 C23 SIGMOD Workshop on Research Issues in Data Mining
and Knowledge Discovery (DMKD) 4 3 14
7 C24 Very Large DataBase (VLDB) 4 3 3
8 C11 IEEE International Conference on Tools with Artificial
Intelligence (ICTAI) 3 8 22
9 C12 Industrial Conference on Data Mining (ICDM) 3 8 4 10 C13 International Conference on Data Engineering (ICDE) 3 8 6 11 C14 International Conference on Data Warehousing and
Knowledge Discovery (DaWaK) 3 8 12
12 C4 ACM Symposium on Principles of Database Systems 2 12 11 13 C6 Advances in Distributed and Parallel Knowledge
Discovery 2 12 21
14 C8 Advances in Neural Information Processing Systems
(NIPS) 2 12 24
15 C17 Lecture Notes in Computer Sciences (LNCS) 2 12 8 16 C18 Lecture Notes in Artificial Intelligence (LNAI) 2 12 13
17 C22 SIGMOD 2 12 2
There are too many publications in Table 6-14 so we miss the focus of the researchers. We only include the publications which receive higher than the average score of 1.958 although the expert value is 0. There are 17 conferences that pass this threshold. The top 17 publications according to our model and the experts’ model can be the observations of the assessment. The recall rate of our model in comparison to the expert’s ranking is 82.35% while the precision rate is 70.83%. The effect of our Bayesian estimation model is acceptable and significant during 452 conferences. The associate professors and professors among our experts claim that the lists we propose
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
include almost all the conferences with impact in the topic of data mining.