行政院國家科學委員會專題研究計畫 成果報告
混合型時間序列模型之分析
計畫類別： 個別型計畫 計畫編號： NSC91-2118-M-004-003-執行期間： 91 年 08 月 01 日至 92 年 07 月 31 日 執行單位： 國立政治大學統計學系 計畫主持人： 翁久幸 報告類型： 精簡報告 處理方式： 本計畫可公開查詢中
華
民
國 92 年 9 月 10 日
行政院國家科學委員會專題研究計畫成果報告
混合型時間序列模型之分析
Analysis of mixtures of time series models
計畫編號：NSC 91-2118-M-004-003
執行期限：91 年 8 月 1 日至 92 年 7 月 31 日
主持人：翁久幸 政治大學統計系
e-mail: chweng@nccu.edu.tw
http://stat.nccu.edu.tw/~chweng
一、 中文摘要 時間序列分析所討論的課題很多,這裡我們考慮關於時間序列的分段與確認 (segmentation and identification) 之問題。也就是說,若一時間序列由若干個未知 模型分別在不同的時間區間內生成,我們要找出其分段點及其生成之模型。我們 的解決辦法是結合支撐向量法(support vector machines)與統計的叢聚分析(clustering analysis)。該方法可以應用於許多複雜的時間序列,例如
Mackey-Glass,EEG。本論文創新處包括提出一個支撐向量法的新形式,與一個調 整控制模型間競爭程度之參數的新方法。前者主要是對支撐向量法模型的誤差項 給予不同的權重,以配合該時間序列是由若干個未知模型分別生成的特質;後者 則是利用最大概似估計法調整參數。
此研究成果已發表於研討會(Chang, Lin, and Weng [2]),而此研討會論文經過重新 整理後,已經投稿於 IEEE Transactions on Neural Networks,目前已被接受,即將刊 登[3]。上述之方法也被應用在 traveling salesman problems, 並且發表於研討會 (Chang, Lin, and Weng [3]) 。
關鍵詞：時間序列, 混合模型, 支撐向量法, 期望值-最大化. Abstr act
segmentation of switching dynamics using support vector machines. Following the architecture by Pawelzik et al. [8] where annealed competing neural networks were used to segment a non-stationary time series, in this article we exploit the use of support vector machines, a well-known learning technique. First, a new formulation of support vector regression is proposed. Second, an expectation-maximization (EM) step is suggested to adaptively adjust the annealing
parameter. Experimental results using chaotic time series indicate that the proposed approach is promising.
Keywor ds: time series, mixture models, support vector machines, expectation-maximization.
二、Introduction
Recently support vector machines (SVMs) have been a promising method for data classification and regression. For an account of the method, see [10]. However, its application to unsupervised learning problems has not been exploited much. In this paper we aim to apply it to unsupervised segmentation of time series. Practical applications of unsupervised segmentation of time series include, for example, speech recognition, signal classification , and brain data.
The topic of unsupervised segmentation has been investigated by researchers in various fields. See for example, an early survey on various approaches in [7] and a combination of supervised and unsupervised learning using hidden Markov models based on neural networks [4].
In [6,8], annealed competing neural networks were used to segment a non-stationary time series, where non-stationarities are caused by switching dynamics. This method is called ``Annealed Competition of Experts" (ACE).
Unlike the mixtures of experts architecture [5] which used an input-dependent gating network, the ACE method drives the competition of experts by an evaluation of prediction performance so that the underlying dynamics can have overlapping input domains. Two main features of this approach are memory derived from a slow
switching rate and deterministically annealed competition of the experts during
training. The assumption of slow switching rate is imposed to resolve problems caused by overlapping input-output relations. The idea of annealing is to avoid getting stuck in local minima and resolve the underlying dynamics in a hierarchical manner. (The deterministic annealing method was described in the context of clustering [9].)
The present paper aims to solve the same unsupervised segmentation problem as in [6,8]. We propose a framework using competing support vector machines (SVMs).
三、Main results
The standard SVMs assume equal weights on all error terms, which mean that each data point is equally important. However, due to the switching nature of the problem, the data points actually come from different sources and therefore, the contribution of each data point to each predictor would not be the same.
In order to solve this problem, we propose a modified formulation of SVMs which allows different weights on the error terms. The dual problem and the implementation for this formulation are also new. Here the weights are adjusted by relative prediction performance, as in [6,8]. In addition to the new formulation, this paper is novel in presenting an adaptive annealing method.
A key observation is that the annealing parameter characterizes some statistical property of the error terms. Therefore, at each iteration during training, we treat the annealing parameter as an unknown parameter and estimate it based on the current weighting coefficients and error terms. This estimate is essentially the maximum likelihood estimate, which can be obtained by an expectation-maximization (EM) step. We call this annealing methodadaptive deterministic annealing.
四、Conclusions
In this paper we present a framework for the unsupervised segmentation of non-stationary time series using SVMs. The problems we consider here are the same as those in [8]. The method used in this paper is novel in two aspects. First, a new formulation of SVM and its implementation are proposed for the switching problems. Second, we use statistical reasoning to adjust the annealing parameter. The annealing property of the proposed method can be explained. The figures we obtained using chaotic time series, our results are comparable to those in [8]. Moreover, as we adjust the annealing parameter adaptively, our algorithm requires much fewer iterations than theirs.
五、Bibliogr aphy
[1] M.-W. Chang, C.-J. Lin, and Ruby C. Weng, “Analysis of Switching Dynamics with Competing Support Vector Machines.” International Joint Conference on
Neural networks, Honolulu, Hawaii, May 12-17, 2002. Proceedings of IJCNN,
p2387-2392.
[2] M.-W. Chang, C.-J. Lin, and Ruby C. Weng, “Adaptive deterministic annealing for two applications: competing SVR of switching dynamics and travelling salesman problems.” The 9th International Conference on Neural Information Processing,
Singapore, November 18-22, 2002.
[3] M.-W. Chang, C.-J. Lin, and R. C. Weng. "Analysis of switching dynamics with competing support vector machines," to appear IEEE Transactions on Neural Networks, 2003.
[4] L. A. Feldkamp, T. M. Feldkamp, and Danil V. Prokhorov, "An approach to adaptive classification," Intelligent signal processing,
editors Simon Haykin and Bart Kosko, IEEE Press, 2001.
[5] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. "Adaptive mixtures of local experts," Neural Computation, 3(1):79-87, 1991.
[6] K.-R. Muller, J. Kohlmorgen, and K. Pawelzik, "Analysis of Switching Dynamics with Competing Neural Networks," IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, E78-A(10): 1306-1315, 1995.
[7] R. Murray-Smith and Tor Arne Johansen (Eds.), "Multiple Model Approaches to Modelling and Control," Taylor and Francis, London, 1997.
[8] K. Pawelzik, J. Kohlmorgen and K. Muller. A. P. Dawid. “Annealed Competition of Experts for a Segmentation and Classification of Switching Dynamics,” Neural Computation, 8(2): 340-356, 1996.
[9] K. Rose, F. Gurewitz, and G. Fox, "Statistical mechanics and phase transitions in clustering," Physical Rev. Letters, 65(8): 945-948, 1990.
[10] V. Vapnik, "Statistical Learning Theory", Springer Verlag, New York, NY, 1998.
