The study of sequential minimal transductive support vector machine Le Hoang Hiep、陳木松
E-mail: 322113@mail.dyu.edu.tw
ABSTRACT
Support Vector Machine (SVM) is a novel machine learning algorithm based on the statistical learning theory. SVMs obtain high performance in real-world applications and are used as one of the standard tools for machine learning and data mining. There are, however, still problems whenever data is rare or difficult to collect. Moreover, the data is likely to change over time. Currently, there are many variants of SVMs were purposed. The Sequential Minimal Transductive Support Vector Machine (SMTSVM) is one of them. Two key points of SMTSVM are updating Lagrange coefficients of two inconsistent patterns and estimating empirical risks value for the whole data. If the empirical risk decreases, we change the label of the inconsistent pattern with the greatest slack value.
Otherwise, we increase regularization parameters and retrain the complete data set. In order to investigate the efficiency and effectiveness of the SMTSVM, simulations of linear, non-linear data, and USPS data set are carried out to compare with Transductive SVM (TSVM), Progressive TSVM (PTSVM), and Online TSVM (OTSVM).
Keywords : Support Vector Machine、Transductive Support Vector Machine、Sequential Minimal Transductive Support Vector Machine
Table of Contents
CREDENTIAL AUTHORIZATION LETTER...iii ABSTRACT (English)...iv ABSTRACT (Chinese)...v ACKNOWLEDGMENT...vi TABLE OF CONTENTS...vii TABLE OF FIGURES...ix LIST OF TABLE...x Chapter 1. INTRODUCTION...1 1.1.Machine Learning...1 1.2.Supervised Learning, Unsupervised Learning and Semi-supervised Learning...1 1.2.1.Supervised Learning...1 1.2.2.Unsupervised Learning...2 1.2.3.Semi-supervised Learning...2 1.2.4.SVM...3 1.2.5.TSVM...3 1.2.6.Sequential Minimal Transductive Support Vector Machine (SMTSVM)...4 Chapter 2. SUPPORT VECTOR MACHINE...5 2.1.Linear Hard-margin SVM...6 2.2.Linear Soft-margin SVM...8 2.3.Non-linear Separable SVM...10 Chapter 3. TRANSDUCTIVE SUPPORT VECTOR MACHINE...13 3.1.TSVM...13 3.2.Progressive TSVM (PTSVM)...18 3.3.Online TSVM...21 Chapter 4.
SEQUENTIAL MINIMAL TRANSDUCTIVE SUPPORT VECTOR MACHINE...24 4.1.Dual Form in Sequential Minimal Transductive SVM (SMTSVM)...24 4.2.Updating Lagrange Parameters for Inconsistent Patterns...26 4.3.Sequential Minimal Transductive SVM Algorithm...27 4.4.SMTSVM Flowchart...28 4.5.Comparisons...31 Chapter 5.
SIMULATIONS...33 5.1.Linearly Separable Data...33 5.2.Linearly Non-separable Data...35 5.3.Benchmark Data...38 Chapter 6. Conclusions...42 APPENDIX A: Compute Objective Function (Dual form)...44 REFERENCES...45
REFERENCES
[1]Kumar S. Neural Networks: A Classroom. McGraw-Hill Publishing Company Limited, International Edition, ISBN 007-048292-6, 2005.
[2]Richard O. Duda, Peter E. Hart, David G. Stork. Pattern Classification. John Wiley & Sons, Inc. 2nd Edition, 2001.
[3]Yisong Chen, Guoping Wang and Shihai Dong. Learning with progressive transductive support vector machine. Pattern Recognition Letters 24, 2003, pp.1845–1855.
[4]Ye Wang, Shang-Teng Huang. Training TSVM with the proper number of positive samples. Pattern Recognition Letters 26, 2005, pp.2187 –2194.
[5]Gert Cauwenberghs, Tomaso Poggio. Incremental and Decremental Support Vector Machine Learning. In T. K. Leen, T. G. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems, volume 13, pages 409-415. MIT Press, 2001.
[6]T. Joachims. SVM-Light: An implementation of Support Vector Machines. Department of Computer Science, Cornell University.
http://svmlight.joachims.org/.
[7]Chih-Chung Chang and Chih-Jen Lin. LIBSVM – A Library for Support Vector Machines. Department of Computer Science and and Information Engineering, National Taiwan University. http://www.csie.ntu.edu.tw/~cjlin/libsvm/.
[8]Dayan P. Unsupervised learning. The MIT Encyclopedia of the Cognitive Sciences. Wilson, RA & Keil, F, editors.
[9]T. T. Nguyen, M.S. Chen. Online Transductive Support Vector Machine. Conference on Engineering Science/Technology, Da-Yeh University, Changhua, Taiwan, June 2008.
[10] http://en.wikipedia.org/wiki/Machine_learning.
[11] http://en.wikipedia.org/wiki/Semi-supervised_learning.
[12]N. Kasabov and S.N Pang. Transductive Support Vector Machines and Applications in Bioinformatics for Promoter Recognition. Neural Information Processing. Vol. 3, No. 2, May 2004.
[13]Christopher J.C. Burges. A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and Knowledge Discovery, volume 2, number 2, page 121–167, June, 1998.
[14]T. Joachims, “Transductive inference for text classification using support vector machines” in Proc. ICML, pp. 200-209, 1999.
[15]V. N. Vapnik, Statistical Learning Theory. New York: Wiley, 1998.
[16]Q.M. Ha, N. Partha and Y. Yuan. Mercer’s Theorem, Feature Maps, and Smoothing. Lecture Notes in Computer Science, Springer Berlin / Heidelberg Publisher, Volume 4005/2006, page 154–168, 2006.
[17]John C. Platt. Fast training of support vector machines using sequential minimal optimization. In B. Scholkopf, C. J. C. Burges, and A. J.
Smola, editors, Advances in Kernel Methods — Support Vector Learning, pages 185–208, Cambridge, MA, 1999. MIT Press.
[18]G. Teng, Y.H. Liu, J.B. Ma, F. Wang, H.T Yao. Improved Algorithm for Text Classification Based on TSVM. First International Conference on Innovative Computing, Information and Control (ICICIC'06), IEEE, 2006.
[19]Xinjun Peng and Yifei Wang. Learning with Sequential Minimal Transductive Support Vector Machine. FAW 2009, LNCS 5598, pp. 216 –227, Springer – Verlag Berlin Heidelberg, 2009.
[20]Fernando Pe'rez-Cruz, Angel Navia-Va'zquez, Ani'bal R. Figueiras-Vidal, Antonio Arte's-Rodri'guez. Empirical Risk Minimization for Support Vector Classifiers. IEEE Transactions on Neural Networks, Vol. 14, No 2, March, 2003.
[21] http://en.wikipedia.org/wiki/Empirical_risk_minimization.
[22]USPS Data, http://www.ee.columbia.edu/~xlx/ee4830/hws/hw3.html.
[23] http://crsouza.blogspot.com/2010/03/kernel-functions-for-machine-learning.html.
[24]Vladimir Vovk, Alex Gammerman and GlennShafer. Algorithmic Learning in a Random World. pp. 291, Springer, March 2005.
[25]Fan Sun, Maosong Sun. Transductive Support Vector Machines Using Simulated Annealing. CIS 2005, Part I, LNAI 3801, pp. 536 – 543, Springer – Verlag Berlin Heidelberg, 2005.
