Theoretical Analysis on the Feasibility of Using the Direct Method to Build Sparse Kernel Principal Component Analysis

Neural Information Processing – Letters and Reviews Vol. 11, No. 1, January 2007

1

Theoretical Analysis on the Feasibility of Using the Direct Method to Build

Sparse Kernel Principal Component Analysis

Department of Automatic Test and Control, Harbin Institute of Technology, Harbin, 150001,P.R. China, Phone: +86-451-6413531-8603, Fax: +86-451-6418083, E-mail: [email protected]

Jeng-Shyang Pan

Department of Electronic Engineering, National Kaohsiung University of Applied Sciences, D415 Chien-Kung Road, Kaohsiung 807, Taiwan，

Phone: +886-7-3814526 Ext. 5636，Fax:+886-7-3811182， Email: [email protected]

Shu-Chuan Chu

Department of Information Management, Cheng Shiu University, Taiwan. (Submitted on October 10, 2006)

Abstract — The previous work in [1] uses a direct method to build sparse kernel learning algorithms. In this paper, our goal is to prove the feasibility of using the direct method for building sparse kernel principal component analysis from the theoretical derivation. Firstly we present a least square support vector machine formulation for kernel principal component analysis algorithm, and then we build sparse kernel principal component analysis using the direct method same to the method mentioned in [4], and finally we prove the feasibility of the algorithm from the mathematical derivation. The computation complexity and memory capacity of the algorithm is analyzed.

Keywords — Kernel method, kernel principal component analysis, sparse KPCA

1. Introduction

In KPCA, this nonlinearity is firstly mapping the data into another space F using a nonlinear map Φ:RN →F, and then PCA is implemented in F using the mapped examples Φ(xk) [4]. The map Φ and the space F are determined implicitly by the choice of a kernel function k(x,y) which computes the dot product between two input examples x and y mapped into F via k(x,y)=Φ(x)⋅Φ(y). If k(x,y) is a positive definite kernel, then there exists a map Φ into a dot product space F. The space F has the structure of a so-called

Reproducing Kernel Hilbert Space (RKHS) [2]. First, inner products in F can be evaluated without computing )

(x

Φ explicitly. This allows us to work with a very high-dimensional, possibly infinite-dimensional RKHS F. Secondly if a positive definite kernel is specified, we need to know neither Φ(x) and F explicitly to perform KPCA since only inner products are used in the computations. Commonly used examples of such positive definite kernel functions are the polynomial kernel and Gaussian kernel, each of them implying a different map and RKHS. PCA based feature extraction needs to store the r× coefficient matrix m W , where r is the number of principal components, and m is the number of training samples. While KPCA based feature extraction need to store the original sample information owing to computing the kernel matrix, which leads to a huge store and a high computing consuming. In order to solve the problem, we apply the method mentioned in [1] to build the sparse KPCA. And from the theoretical view we prove the feasibility of the algorithm.

Neural Information Processing – Letters and Reviews Vol. 11, No. 1, January 2007 5 Let

[

]

β = 1 2 L . For we choose m eigenvector α corresponding to m largest eigenvalue. Let

[

]

β = _L , the feature can be obtained as follows.

zx

BK x

z( )= (35)

3. Conclusion

High time consuming is needed during training KPCA, but in the practical application, processing speed is a crucial problem such as face recognition, so Direct Sparse KPCA (KPCA) is very meaningful. So DS-KPCA can not only accelerate the evaluation of the test data, but also save the memory of storing the trained data. In this paper, we build the sparse kernel component analysis and prove the feasibility of using the direct method for building sparse kernel principal component analysis from the theoretical derivation. From the analysis on the computation complexity and memory capacity of the algorithm, Sparse KPCA can save the store space and reduce the time consuming.

References

[1] Mingrui Wu, Bernhard Scholkopf and Gokhan Bakır, “A Direct Method for Building Sparse Kernel Learning Algorithms”, Journal of Machine Learning Research, Vol. 7, pp. 603–624, 2006.

[2] B. Scholkopf and A. Smola, Learning with Kernels. Cambridge, Mass.: MIT Press, 2002.

[3] J. A. K. Suykens, T. Van Gestel, J. Vandewalle, and B. De Moor,” A Support Vector Machine Formulation to PCA Analysis and Its Kernel Version”, IEEE Trans. Neural Network, vol.14, no.2, pp. 447-450, Mar. 2003.

[4] B. Scholkopf, A. Smola, and K. Muller, “Nonlinear Component Analysis as a Kernel Eigenvalue Problem,” Neural Computation, vol. 10, no. 5, pp. 1299-1319, 1998.

[5] Zhen Kun Gon; JunKang Feng; Fyfe, C.,” A comparison of sparse kernel principal component analysis methods”, Knowledge-Based Intelligent Engineering Systems and Allied Technologies, 2000. Proceedings. Fourth International Conference on. pp.309–312, Sept. 2000

Jun-Bao Li received the B.Sc. and M.Sc. degree from Harbin Institute of Technology

(HIT), Harbin, P. R. China in 2002 and 2004, respectively. He is currently working toward the Ph.D. degree in the Measurement Technology and Instrument, in HIT, Harbin, P. R. China. His research interests are mainly in pattern recognition and image processing.

Jeng-Shyang Pan received the B. S. degree in Electronic Engineering from the

National Taiwan University of Science and Technology, Taiwan in 1986, the M. S. degree in Communication Engineering from the National Chiao Tung University, Taiwan in 1988, and the Ph.D. degree in Electrical Engineering from the University of Edinburgh, U.K. in 1996. Currently, he is a Professor in the Department of Electronic Engineering, National Kaohsiung University of Applied Sciences, Taiwan. Professor Pan has published more than 50 journal papers and 120 conference papers. He joints the editorial board for LNCS Transactions on Data Hiding and Multimedia Security, Springer, International Journal of Knowledge-Based Intelligent Engineering Systems, IOS Press, and International Journal of Hybrid Intelligent System, Advanced Knowledge International. He is the Co-Editors-in-Chief for International Journal of Innovative Computing, Information and Control. His current research interests include data mining, information security and image processing.

Theoretical Analysis on the Direct Method to Sparse Kernel PCA Jun-Bao Li, Jeng-Shyang Pan, and Shu-Chuan Chu

6

Shu-Chuan Chu received the Ph.D. degree in School of Informatics and Engineering, Flinders University of South

Australia. She is an assistant Professor of Dept. of Information Management, Cheng Shiu University, Taiwan. His research interests are mainly in Data Mining, Computational Intelligence, Information Hiding, and Signal Processing.