The proof for the theorem on page 5 is given below. We prove the theorem in two steps.
1. For t = 1. By SVD, we have
Since B(1) is a constant, minimizing
B(1) − A(1)X(1) is equivalent to minimiz-ingB(1)− Σ(1)VT(1)X(1). U(1)Σ(1)VT(1). Therefore,
J (X(2))
Clearly, minimizing J (X(2)) is equiva-lent to minimizing the following expres-sion:
Similarly, minimizing M (X(2)) is equiv-alent to minimizing
J (X(2))ˆ
=B(2)− Σ(2)VT(2)X(2). (139) Therefore, minimizing Eq.(14) is equiva-lent to minimizing Eq.(15) for t = 2.
(b) Assume that minimizing Eq.(14) is equivalent to minimizing Eq.(15) for 3≤ t≤ k.
(c) Now we want to prove that minimiz-ing Eq.(14) is equivalent to minimizminimiz-ing Eq.(15) for t = k + 1. For t = k + 1, we
By following the same reasoning path of t = 2, we conclude that minimizing J (X(k + 1)) is equivalent to minimizing the following expression:
Similarly, minimizing Eq.(141) is equiv-alent to minimizing
B(k) qk+1
−
Σ(k)VT(k)
ak+1T
X(k + 1). (142)
Moreover, minimizing Eq.(142) is equiv-alent to minimizing
J (X(k + 1)) =ˆ B(k + 1)
−Σ(k + 1)VT(k + 1)X(k + 1).(143) Therefore, we conclude that minimizing Eq.(14) is equivalent to minimizing Eq.(15) for t≥ 2.
By combining the cases of t = 1 and t ≥ 2, we complete the proof.
ü låA‹AÇ
The goal of our project has been achieved. In particular, (1) a static neuro-fuzzy modeling tech-nique was developed in the first year; (2) a dy-namic neuro-fuzzy modeling technique was devel-oped in the second year; and (3) the recurrent fuzzy network obtained in the second year was applied successfully to the recognition of spoken words. Besides, research results of this project have been published in well-known international journals and conferences. In total, we have pub-lished 5 journal papers and 12 conference papers related to this project, as listed below.
Published Journal Papers
[1] J.-W. Lin, S.-J. Lee, and H.-T. Yang, “A stroke-based neuro-fuzzy system for hand-written chinese character recognition,” Ap-plied Artificial Intelligence, vol. 15, no. 6, pp. 561–586, 2001.
[2] S.-J. Lee and C.-L. Hou, “An ART-based con-struction of RBF networks,” IEEE Trans-actions on Neural Networks, vol. 13, no. 6, pp. 1308–1321, 2002.
[3] S.-J. Lee, C.-S. Ouyang, and S.-H. Du, “A neuro-fuzzy approach for extracting human objects in image sequences,” IEEE Transac-tions on Systems, Man, and Cybernetics – Part B: Cybernetics, vol. 33, no. 3, pp. 420–
437, 2003.
[4] S.-J. Lee and C.-S. Ouyang, “A neuro-fuzzy system modeling with self-constructing rule generation and hybrid SVD-based learning,”
IEEE Transactions on Fuzzy Systems, vol. 11, no. 3, pp. 341–353, 2003.
[5] H.-L. Tsai and S.-J. Lee, “Entropy-based gen-eration of supervised neural networks for classification of structured patterns,” IEEE Transactions on Neural Networks, September 2003. Accepted.
Published Conference Papers
[1] C.-S. Ouyang and S.-J. Lee, “A hybrid al-gorithm for structure identification of neuro-fuzzy modeling,” in Proceedings of IEEE In-ternational Conference on Systems, Man, and Cybernetics, vol. 5, (Mashville, TN, USA), pp. 3611–3616, October 2000.
[2] S.-J. Lee and C.-L. Hou, “A self-constructed radial basis function neural network and its applications,” in Proceedings of IEEE Inter-national Conference on Systems, Man, and Cybernetics, (Mashville, TN, USA), pp. 3623–
3628, October 2000.
[3] S.-H. Du, C.-S. Ouyang, and S.-J. Lee, “A neuro-fuzzy approach to detection of human objects for MPEG video compression,” in Proceedings of Multimedia Technologies and Applications Conference (MTAC), (Irvine, CA, USA), November 2001.
[4] C.-S. Ouyang and S.-J. Lee, “A hybrid learn-ing algorithm for fuzzy neural networks,”
in Proceedings of 8th International Con-ference on Neural Information Processing (ICONIP’01), (Shanghai, China), pp. 311–
316, November 2001.
[5] W.-G. Chen and S.-J. Lee, “Fuzzy classifica-tion using hierarchical genetic algorithm with multiple rule gene tables,” in Proceedings of National Computer Symposium, (Taipei, Tai-wan), pp. B001–B010, December 2001.
[6] W.-J. Lee, C.-S. Ouyang, and S.-J. Lee,
“Constructing neuro-fuzzy systems with TSK fuzzy rules and hybrid SVD-based learning,”
in Proceedings of IEEE International Con-ference on Fuzzy Systems, vol. 2, (Honolulu, Hawaii, USA), pp. 1174–1179, May 2002.
[7] W.-J. Lee, C.-S. Ouyang, and S.-J. Lee, “A novel approach for neuro-fuzzy modeling,” in Proceedings of 78th Anniversary Conference of the Military Academy, ROC, (Kaohsiung, Taiwan), May 2002.
[8] C.-S. Ouyang, W.-J. Lee, and S.-J. Lee, “An adaptive neuro-fuzzy approach for system modeling,” in Proceedings of 1st Interna-tional Conference on Machine Learning and Cybernetics (ICMLC02), (Beijing, China), pp. 1875–1880, IEEE, November 2002.
[9] L.-M. Huang, C.-S. Ouyang, W.-J. Lee, and S.-J. Lee, “A hybrid clustering and SVD-based approach for fuzzy-neural system mod-eling,” in Proceedings of 7th Conference on Artificial Intelligence and Applications, (Taichung, Taiwan), pp. 6–11, November 2002.
[10] H.-L. Tsai and S.-J. Lee, “Construction of neural networks on structured domains,”
in Proceedings of 9th International Con-ference on Neural Information Processing (ICONIP’02), vol. 1, (Singapore), pp. 50–54, November 2002.
[11] J.-S. Chung, C.-S. Ouyang, and S.-J. Lee,
“Vector quantization using a self-constructing fuzzy neural network,” in Proceedings of 10th National Conference on Fuzzy Theory and Its Applications, (Hsinchu, Taiwan), pp. E3:14–
19, December 2002.
[12] C.-S. Ouyang and S.-J. Lee, “An improved tsk-type recurrent fuzzy network for dy-namic system identification,” in Proceedings of IEEE International Conference on Sys-tems, Man, and Cybernetics, (Hyatt Regency, Washington, D.C., USA), pp. 3342–3347, Oc-tober 2003.
References
[1] L. B. Almeida. A learning rule for asyn-chronous perceptrons with feedback in a com-binatorial environment. In Proceedings of ICNN, vol. 2, pp. 609–618, 1987.
[2] R. Bellman. Dynamic programming. Princeton Univ. Press, New Jersey, 1957.
[3] B. S. Atal and M. R. Schroeder. Predictive coding of speech signals. in Proceeding of the 6th International Congress on Acoustics., vol.
5, no. 4, 1968.
[4] J. J. Buckley, “Sugeno type controllers are uni-versal controllers,” Fuzzy Sets and Systems, vol. 53, no. 3, pp. 299–303, 1993.
[5] ˚A. Bj¨orck, Numerical Methods for Least Squares Problems. Philadelphia, PA, USA: So-ciety for Industrial and Applied Mathematics, 1996.
[6] C. L. Blake and C. J. Merz, “UCI repos-itory of machine learning database,”
http://www.ics.uci.edu/∼mlearn/ MLReposi-tory.html, 1998.
[7] S.-Y. Cho and T. W. S. Chow, “Training multi-layer neural networks using fast global learning algorithm – least-squares and penalized opti-mization methods,” Neurocomputing, vol. 25, no. 1-3, pp. 115–131, 1999.
[8] M. Delgado, A. F. Gomez-Skarmeta and F. Martin, “A fuzzy clustering-based rapid pro-totyping for fuzzy rule-based modeling,” IEEE Transactions on Fuzzy Systems, vol. 5, no. 2, pp. 223–233, 1997.
[9] K. Funahashi, “On the approximate realization of continuous mappings by neural networks,”
Neural Networks, vol. 2, no. 3, pp. 183–192, 1989.
[10] W. A. Farag, V. H. Quintana, and G. Lambert-Torres, “A genetic-based neuro-fuzzy approach for modeling and control of dy-namical systems,” IEEE Transactions on Neu-ral Networks, vol. 9, no. 5, pp. 756–767, 1998.
[11] G. H. Golub and C. F. Van Loan, Matrix Computations. The Johns Hopkins University Press, 1991.
[12] J. B. Gomm and D. L. Yu, “Selecting ra-dial basis function network centers with recur-sive orthogonal least squares training,” IEEE Transactions on Neural Networks, vol. 11, no. 2, pp. 306–314, 2000.
[13] S. Haykin, Neural networks: a comprehensive foundation. NJ, USA: Prentice Hall, 1999.
[14] M. Hornik, K. Stinchcombe and H. White,
“Multilayer feedforward networks are univer-sal approximators,” Neural Networks, vol. 2, pp. 359–366, 1989.
[15] J. B. Hampshire II and A. H. Waibel. A novel objective function for improved phoneme recognition using time-delay neural networks.
IEEE Transactions on Neural Networks, vol. 1, pp. 216–228, 1990.
[16] F. Jelinek. Continuous speech recognition by statistical methods. Proceeding of the IEEE, vol. 64, no. 4, pp. 532–556, 1976.
[17] J.-S. R. Jang, “ANFIS: adaptive-network-based fuzzy inference system,” IEEE Trans-actions on Systems, Man and Cybernetics, vol. 23, no. 3, pp. 665–685, 1993.
[18] C.-F. Juang, “A TSK-type recurrent fuzzy network for dynamic systems processing by neural network and genetic algorithms”, I EEE Transactions on Fuzzy Systems, vol. 10, no. 2, pp. 155–170, 2002.
[19] C.-F. Juang and C.-T. Lin, “An on-line self-constructing neural fuzzy inference network and its applications,” IEEE Transactions on Fuzzy Systems, vol. 6, no. 1, pp. 12–32, 1998.
[20] C.-F. Juang and C.-T. Lin, “A recurrent self-organizing neural fuzzy inference net-work,” IEEE Transactions on Neural Net-works, vol. 10, no. 4, pp. 828–845, 1999.
[21] J.-S. R. Jang and C.-T. Sun, “Neuro-fuzzy modeling and control,” Proceedings of the IEEE, vol. 83, no. 3, pp. 378–406, 1995.
[22] A. Kandel, Fuzzy Expert Systems. Boca Ra-ton, FL, USA: CRC Press, 1992.
[23] B. Kosko, “Fuzzy systems as universal ap-proximators,” IEEE Transactions on Comput-ers, vol. 43, no. 11, pp. 1329–1333, 1994.
[24] E. Kim, M. Park, S. Ji, and M. Park, “A new approach to fuzzy modeling,” IEEE Transac-tions on Fuzzy Systems, vol. 5, no. 3, pp. 328–
337, 1997.
[25] Y. Lin and G. A. Cunningham III, “A new approach to fuzzy-neural system modeling,”
IEEE Transactions on Fuzzy Systems, vol. 3, no. 2, pp. 190–198, 1995.
[26] Y. Lin, G. A. Cunningham III, and S. V.
Coggeshall, “Using fuzzy partitions to create fuzzy systems from input-output data and set
the initial weights in a fuzzy neural network,”
IEEE Transactions on Fuzzy Systems, vol. 5, no. 4, pp. 614–621, 1997.
[27] C.-J. Lin and C.-T. Lin, “An ART-based fuzzy adaptive learning control network,”
IEEE Transactions on Fuzzy Systems, vol. 5, no. 4, pp. 477–496, 1997.
[28] S.-J. Lee and J.-S. Ouyang, “A neuro-fuzzy system modeling with self-constructing rule generation and hybrid SVD-based learning,”
IEEE Trans. Fuzzy Syst., vol. 11, no. 3, pp.
341-353, 2003.
[29] S. E. Levinson L. R. Rabiner and M. M.
Sondhi. On the application of vector quanti-zation and hidden markov models to speaker-independent, isolated word recognition. Bell Systems Technique journal, vol. 62, no. 4, pp.
1075–1105, 1983.
[30] R. Meir and V. E. Maiorov, “On the optimal-ity of neural-network approximation using in-cremental algorithms,” IEEE Transactions on Neural Networks, vol. 11, no. 2, pp. 323–337, 2000.
[31] J. Meng and Z. Sun, “Application of com-bined neural networks in nonlinear function ap-proximation,” in Proceedings of the 3rd World Congress on Intelligent Control and Automa-tion, (Hefei, China), pp. 839–841, June 2000.
[32] P. A. Mastorocostas and J. B. Theocharis,
“A recurrent fuzzy-neural model for dynamic system identification”, I EEE Transactions on Systems, Man, and Cybernetics-Part B, vol.
32, no. 2, pp. 176–190, 2002.
[33] A. M. Noll. Cepstrum pitch determination.
Journal of the Acoustic Society of America, vol. 41, pp. 293–309, 1967.
[34] W. Pedrycz, Fuzzy Control and Fuzzy Sys-tems. NY, USA: John Wiley, 1989.
[35] T. Poggio and F. Girosi, “Networks for ap-proximation and learning,” Proceedings of the IEEE, vol. 78, no. 9, pp. 1481–1497, 1990.
[36] N. S. Rubanov, “The layer-wise method and the backpropagation hybrid approach to learning a feedforward neural network,” IEEE Transactions on Neural Networks, vol. 11, no. 2, pp. 295–305, 2000.
[37] L. R. Rabiner and S. E. Levinson. A speaker-independent syntax-directed, connected word recognition system based on hidden markov models and level building. IEEE Transaction on Acoustics, Speech, Signal processing, vol.
33, no. 3, 1985.
[38] I. Rojas, H. Pomares, J. Ortega, and A. Pri-eto, “Self-organized fuzzy system generation from training examples,” IEEE Transactions on Fuzzy Systems, vol. 8, no. 1, pp. 23–36, 2000.
[39] H. Sakoe and S. Chiba. Recognition of contiunously spoken words based on
time-normalization by dynamic programming. Jour-nal of the Acoustic Society of Japan, vol. 27, no. 9, pp. 483–500, 1971.
[40] H. Sakoe and S. Chiba. Dynamic program-ming algorithm optimization for spoken word recognition. IEEE Transaction on Acoustic, Speech, Signal processing, vo. 26, no. 1, pp. 43–
49, 1978.
[41] M. Sugeno and G. T. Kang, “Structure iden-tification of fuzzy model,” Fuzzy Sets and Sys-tems, vol. 28, no. 1, pp. 15–33, 1988.
[42] H. Sakoe and M. Watari. Clockwise propa-gating dp-matching algorithm for word recog-nition. Transaction Committee on Speech Re-search, Acoustic Society of Japan, pp. S81–65, 1981.
[43] M. Sugeno and T. Yasukawa, “A fuzzy-logic-based approach to qualitative modeling,”
IEEE Transactions on Fuzzy Systems, vol. 1, no. 1, pp. 7–31, 1993.
[44] R. Thawonmas and S. Abe, “A novel ap-proach to feature selection based on analysis of class regions,” IEEE Transactions on Sys-tems, Man and Cybernetics, Part B, vol. 27, no. 2, pp. 196–207, 1997.
[45] R. Thawonmas and S. Abe, “Function ap-proximation based on fuzzy rules extracted from partitioned numerical data,” IEEE Transactions on Systems, Man and Cybernet-ics, Part B, vol. 29, no. 4, pp. 525–534, 1999.
[46] T. Takagi and M. Sugeno, “Fuzzy identifica-tion of systems and its applicaidentifica-tion to modeling and control,” IEEE Transactions on Systems, Man and Cybernetics, vol. 15, no. 1, pp. 116–
132, 1985.
[47] L.-X. Wang, “Fuzzy systems are universal ap-proximations,” in Proceedings of the First In-ternational Conference on Fuzzy Systems, (San Diego, CA, USA), pp. 1163–1170, March 1992.
[48] C.-C. Wong and C.-C. Chen, “A hybrid clus-tering and gradient descent approach for fuzzy modeling,” IEEE Transactions on Systems, Man, and Cybernetics, Part B, vol. 29, no. 6, pp. 686–693, 1999.
[49] S. Wu and M. J. Er, “Dynamic fuzzy neu-ral networks-a novel approach to function ap-proximation,” IEEE Transactions on Systems, Man and Cybernetics, Part B, vol. 30, no. 2, pp. 358–364, 2000.
[50] S. Wu, M. J. Er, and Y. Gao, “A fast approach for automatic generation of fuzzy rules by generalized dynamic fuzzy neural net-works,” IEEE Transactions on Fuzzy Systems, vol. 9, no. 4, pp. 578–594, 2001.
[51] J. Yen, L. Wang, and C. W. Gillepie, “Im-proving the interpretability of TSK fuzzy mod-els by combining global learning and local learning,” IEEE Transactions on Fuzzy Sys-tems, vol. 6, no. 4, pp. 530–537, 1998.
[52] L. A. Zadeh, “Fuzzy sets,” Information and Control, vol. 8, pp. 338–353, 1965.
[53] Y.-Q. Zhang and A. Kandel, “Compensatory neurofuzzy systems with fast learning algo-rithms,” IEEE Transactions on Neural Net-works, vol. 9, no. 1, pp. 83–105, 1998.
[54] J. Zhang and A. J. Morris, “Recurrent neuro-fuzzy networks for nonlinear process model-ing,” IEEE Transactions on Neural Networks, vol. 10, no. 2, pp. 313–326, 1999.