國立臺灣大學生物資源暨農學院生物環境系統工程學研究所 博士論文
Graduate Institute of Bioenvironmental Systems Engineering College of Bioresources and Agriculture
National Taiwan University Doctoral Dissertation
創建新穎動態類神經網路於水文環境系統
Development of Novel Dynamic Artificial Neural Networks for Hydro-Environmental Systems
陳品安 Pin-An Chen
指導教授﹕ 張斐章 博士 Advisor: Fi-John Chang, Ph.D.
中華民國 103 年 7 月
July 2014
誌謝
2009 年七月學生很榮幸於臺灣大學就讀研究所，進入水資源資訊系統研究室 並順利完成博士論文。因為有各位的幫助，同時在研究與計畫案例中找尋研究方 向，訓練邏輯思考和組織架構的能力，增進熟練文章報告的撰寫，學生才能夠在 這五年的研究之路讓本篇論文能夠順利完成。最由衷感謝的是恩師 張斐章老師，
有您的悉心指導與栽培，無論是研究上適切的提點或是生涯的規劃與建議，都使 我獲益匪淺，讓我們擁有一個充滿自由地研究風氣、完善資源與良好讀書環境的 研究室。在這樣的環境下，從學長姐身上學習，得到了研究方面的協助與鼓勵，
與學弟妹互相切磋、解決問題，這段過程中學習到很多待人處世的道理，我也與 研究室成員們建立深厚的感情，這些都是在人生的道路上，很難得的緣分與機會。
在此也由衷的感謝學位論文口試委員成功大學工學院游保杉院長、臺灣大學 生物環境系統工程學系劉振宇教授、海洋大學河海工程學系黃文政教授、交通大 學土木工程學系張良正教授與淡江大學水資源及環境工程學系張麗秋教授針對論 文內容的不吝指正，並於學位考口試給予諸多寶貴建議與指教，使得本篇研究更 臻完整。在研究所的這段時間，也感謝臺大與系上各位老師於修課期間的幫助以 及教導，在此也特別感謝劉振宇老師研究室的學姊高雨瑄博士後研究員，提供本 篇論文於水質議題中許多化學領域的專業知識與建議，給予我相當大的協助。
感謝水資源資訊系統研究室各位夥伴的照顧與陪伴，熱心指導學弟妹的學長 姐－衍銘、國威、孟蓉、文柄、昌翰，論文研究以及生活上的經驗分享都受到你 們很大的照顧與關心；感謝辛苦的助理們－郁垂在生活上的照顧與叮嚀、恒玥對 於論文的修改與建議，以及惠茵在許多細節及報告上的協助；也感謝眾多學弟妹 政華、承賢、桂宏、英秦、宇軒、琬渝、昱中、英睿、俊霖的陪伴與幫忙；感謝 一起為論文奮鬥的文柄學長與碩士班同學孫維與逸鴻，這段時間的共事，有你們
的幫助與照應，讓我順利完成博士學位。感謝研究室的夥伴們營造這麼美好的環 境，總是充滿著活力與朝氣。
最後要感謝我的家人，感謝爸爸、媽媽無私的支持與照顧，讓我在求學路上 無後顧之憂，我才能順利完成博士學位；感謝和我一同聽音樂、運動、攝影、出 遊的好朋友們，紓解了許多研究時累積的壓力；感謝女友劍謙的陪伴，妳的支持 讓我更有力量繼續學習且對自己有信心。在此謹與你們分享研究成果與畢業的喜 悅心情，由衷的感謝你們一路的鼓勵與相伴。
摘要
水為人類賴以為生的重要資源，水與環境在複雜的交互作用下形成的水文環境系 統孕育了各種生命，使得人類社會得以繁榮發展。然而近年來由於氣象、水文、
地文及人為開發等各種錯綜複雜之因素影響，水文環境系統逐漸面臨失衡，加上 全球氣候變遷現象日趨顯著，導致突發性極端水文事件發生之頻率提高，都市下 游河川水質劣化。為能進一步分析掌握其變化，本論文發展新穎系統化動態類神 經網路與相關之分析方法應用於水文環境中兩大重要議題：水文量與水質之推估。
動態回饋式類神經網路(Recurrent neural networks, RNNs)具內部回饋連結，對於複 雜且具有回饋特性的水文環境系統之模擬有較高之精確性，因此近年來受到相當 地重視與大量地應用。本論文第一部分發展並推導多時刻強化型即時回饋線上學 習演算法於回饋式類神經網路 (Reinforced real-time recurrent learning algorithm for RNNs, R-RTRL NN)。此演算法可充分利用最新的觀測值與網路過去之預測值，不 斷地遞迴修正網路權重參數，改善多時刻預報之可靠度與精確度。而為了驗證此 演算法之可靠性與效能，本研究針對著名的混沌時間序列與石門水庫颱洪時期之 入流量進行二、四至六時刻之多階段預報。此外亦選用了三個常用之類神經網路 模式(兩個動態與一個靜態類神經網路)進行效能比較。結果顯示發展之 R-RTRL 類 神經網路較其他類神經網路於多時刻混沌時間序列與水庫入流量預報整體表現優 異，且可有效減緩線上學習法於多時刻預報網路權重參數調整延遲之問題。本論 文 第 二 部 分 則 發 展 一 系 統 化 動 態 類 神 經 網 路 模 擬 架 構 (Systematical dynamic-neural modeling, SDM)。SDM 主要包含 Gamma test 與非線性自回歸與外 部輸入類神經網路(Nonlinear autoregressive with exogenous input, NARX)。Gamma test 可有效且快速地篩選影響目標變數最為顯著之輸入因子組合；NARX 類神經網 路則具輸出層至輸入層之回饋連結，有優異的時間與空間推估能力。本論文運用 發展之 SDM 於都市防洪抽水站多時刻前池水位預報，並探討回饋連結於不同模式
情境下之貢獻。而建構之水位預報模式之準確度與穩定性皆相當高，可有效率且 準確地預測洪汛時期臺北市玉成抽水站之前池水位。此外，SDM 亦可應用於區域 地下水砷濃度與河川總磷濃度之時空推估，精確度與可靠度皆較傳統倒傳遞類神 經網路(BPNN)高，並有效解決進行區域水質推估常面臨之問題，如：影響因子組 合之選取、資料稀少、過度描述與推估效果不佳等。而藉由 NARX 類神經網路之 回饋連結與輸入因子資訊，可進一步將標的水質序列間隔較長之推估時間尺度轉 換為與輸入因子相同且較短之觀測頻率，提供額外資訊進行水質狀況評估。整體 而言，本論文創建之新穎動態類神經網路與分析方法：R-RTRL NN 與 SDM 皆具 廣泛之應用性，十分適合分析處理或推估水文環境中水文量與水質變化等重要議 題，提供政府有關單位於水庫、都市河川與地下水等經營管理之參考資訊。
關鍵詞: 回饋式類神經網路、強化型即時回饋線上學習演算法(R-RTRL)、Gamma test、非線性自回歸與外部輸入類神經網路(NARX-NN)、多時刻時間序列預測、都 市防洪、砷、總磷。
Abstract
Water is a precious and scarce resource on the Earth and can be utilized by human beings. Due to the complex interaction between hydrological, meteorological, geographical factors and human activities with climate change effects, the hydro-environment that we live by is facing imbalanced conditions, such as intensive storms and typhoons with short durations and the degradation of the water quality in groundwater and urban rivers. Therefore, this dissertation is dedicated to the two main problems encountered in hydro-environmental systems: water quantity and water quality, and endeavors to develop novel dynamic artificial neural networks and modeling schemes to overcome problems for analyzing and estimating the dynamic variability of water quantity and water quality. Recurrent neural networks (RNNs) are computationally powerful nonlinear models that are capable of extracting dynamic behaviors from complex systems through internal recurrence and have attracted much attention for years. In the first part of this dissertation, a multi-step-ahead (MSA) reinforced real-time recurrent learning algorithm for RNNs (R-RTRL NN) is developed for adjusting connection weights by incorporating the latest observed values and model outputs into the online training process, and the sequential formulation of the R-RTRL NN is derived. To demonstrate its reliability and effectiveness, the proposed R-RTRL
NN is implemented to make 2-, 4- and 6-step-ahead forecasts through a famous benchmark chaotic time series and a reservoir inflow series during typhoon events in North Taiwan. Numerical and experimental results indicate that the R-RTRL NN not only achieves superior performance than the comparative networks but also significantly improves the precision of MSA forecasts with effective mitigation in time-lag problems for both chaotic time series and reservoir inflow case during typhoon events. In the second part of the dissertation, the systematical dynamic-neural modeling (SDM) scheme that consists of the Gamma test for input factor selection and the nonlinear autoregressive with exogenous input (NARX) network for spatio-temporal estimation is proposed. The SDM is then applied to urban flood control to explore the contribution of recurrent connections and provide reliable results for forecasting the floodwater storage pond (FSP) water level in the Yu-Cheng pumping station. And the SDM is further utilized to estimate the regional arsenic (As) and total phosphate (TP) concentrations in groundwater and river systems, respectively. Results demonstrate that the SDM satisfactorily overcomes the difficulty raised by traditional methods in estimating the temporal and spatial variability of water quality parameters, such as identification of key input factors, data scarcity issue, model over-fitting problem and poor estimation performance. In addition, the SDM bears the ability to reconstruct the time series of the estimated water quality parameter from the original monitoring scale
to a shorter monitoring scale through the recurrent connections of the NARX network.
In summary, the two developed novel techniques in learning algorithm and modeling scheme, the R-RTRL NN and SDM, have broad applicability and are suitable to deal with water quantity and water quality issues in hydro-environmental systems, which beneficially provides useful information to water authorities for the management of reservoir operation, river basin, urban flood control and groundwater contamination.
Keywords : Recurrent neural network (RNN); Reinforced real-time recurrent learning
(R-RTRL) algorithm; Gamma test; Nonlinear autoregressive with eXogenous input (NARX) neural network; Multi-step-ahead forecast; Urban flood control; Water quality;
Arsenic (As); Total phosphate (TP).
Contents
誌謝 ... I 摘要 ... III Abstract ... V Contents ... VIII Figure contents ... IX Table contents... XII
1. Introduction ... 1
1.1 Motivation ... 1
1.2 Research objectives ... 3
1.3 Dissertation layout ... 15
2. Methodology ... 18
2.1 MSA R-RTRL algorithm for RNNs ... 18
2.2 Systematical Dynamic-neural Modeling (SDM) ... 25
2.3 Comparative neural network models ... 38
3. Case studies ... 40
3.1 Reinforced recurrent neural networks for multi-step-ahead flood forecasting ... 40
3.2 Real-time multi-step-ahead water level forecasting by recurrent neural networks for urban flood control ... 54
3.3 Regional estimation of groundwater Arsenic concentration through Systematical Dynamic-neural Modeling ... 78
3.4 Modeling spatio-temporal total phosphate (TP) concentration through Systematical Dynamic-neural Modeling ... 95
4. Conclusion and suggestion ... 110
4.1 Conclusion ... 110
4.2 Suggestion ... 116
5. Reference ... 119 Appendix ... A-1 Acronym list ... A-1 Publications ... A-2 Research projects involved ... A-4 Awards and scholarship ... A-4 Appendix A ... A-5 Appendix B ... A-9
Figure contents
Fig. 1.1 Framework of the dissertation... 17
Fig. 2.1 Weight adjustment procedure for the n-step-ahead R-RTRL NN. ... 20
Fig. 2.2 Architecture of the multi-step-ahead RNN with the R-RTRL online learning algorithm. ... 22
Fig. 2.3 Implementation procedure of the proposed SDM ... 26
Fig. 2.4 Architectures of the NARX network ... 30
Fig. 2.5 Architectures of (a) the BPNN; and (b) the Elman NN. ... 39
Fig. 3.1 Locations of the Shihmen Reservoir and rainfall gauging stations. ... 45
Fig. 3.2 2SA inflow forecast residuals (of testing data sets) based on (a) R-RTRL NN, (b) RTRL NN, (c) Elman NN and (d) BPNN, respectively. ... 50
Fig. 3.3 4SA inflow forecast residuals (of testing data sets) based on (a) R-RTRL NN, (b) RTRL NN, (c) Elman NN and (d) BPNN, respectively. ... 50
Fig. 3.4 6SA inflow forecast residuals (of testing data sets) based on (a) R-RTRL NN, (b) RTRL NN, (c) Elman NN and (d) BPNN, respectively. ... 51
Fig. 3.5 4SA inflow forecasting (Typhoons Krosa, Sinlaku and Jangmi in testing data sets) based on (a) R-RTRL NN, (b) RTRL NN, (c) Elman NN and (d) BPNN, respectively. ... 51
Fig. 3.6 Relationship between inflow forecast errors (RMSE) and forecasting steps of four neural network models. ... 53
Fig. 3.7 Study flow of real-time MSA water level forecasting. ... 55
Fig. 3.8 Locations of the Yu-Cheng catchment and rainfall gauging stations. ... 57
Fig. 3.9 Correlation analysis between FSP water levels and rainfall gauging stations in different time steps. ... 61
Fig. 3.10 Determination of effective rainfall stations by the GT results. ... 64 Fig. 3.11 (a) 20, (b) 50 and (c) 60-min-ahead forecasting of the 612 heavy rainfall event for scenario I with respect to the BPNN, the Elman NN and the NARX network... 67 Fig. 3.12 (a) CE of 10- to 60-min-ahead forecasting and (b) relationship between FSP water level forecast errors (RMSE) and forecasting steps with respect to three forecasting models in the testing stages for scenario I. ... 69 Fig. 3.13 Rainfall input datasets from three gauging stations and 50-min-ahead forecasting of the 612 heavy rainfall event for scenario II with respect to the BPNN, the Elman NN and the NARX network. ... 73 Fig. 3.14 (a) CE of 10- to 60-min-ahead forecasting and (b) relationship between FSP water level forecast errors (RMSE) and forecasting steps with respect to three forecasting models in the testing stages for scenario II. ... 75 Fig. 3.15 Implementation procedure of the proposed SDM for regional estimation of As concentrations in groundwater ... 79 Fig. 3.16 Locations of twenty-six groundwater wells at the Yun-Lin coastal area, Taiwan. ... 81 Fig. 3.17 Determination of non-trivial factors by the GT results. ... 86 Fig. 3.18 Scatter plots of observed and estimated As concentration (conc.) derived from the NARX network and the BPNN at twenty ungauged sites (1995-1999). .. 91 Fig. 3.19 Estimation results of As concentrations at ungauged well #14 during 1995 and 1999 in the testing phases of the NARX network and the BPNN. ... 92 Fig. 3.20 Exceeding probability maps of As concentration under the threshold of WHO drinking water standard (10ugl-1) between 1995 and 1999. ... 94 Fig. 3.21 Research flowchart of the proposed SDM for the estimation of TP
concentration. ... 97 Fig. 3.22 Locations of the Dahan River basin and water quality monitoring stations S1-S7. ... 98 Fig. 3.23 Determination of key factors by the GT results. ... 102 Fig. 3.24 Estimation results of TP concentrations at water quality monitoring stations (a)
S4, (b) S5, (c) S6 and (d) S7 during 2010 and 2012 in the testing phases of the NARX network and the BPNN. ... 107 Fig. 3.25 Reconstructed monthly TP time series based on quarterly TP data estimated at water quality monitoring stations S1-S7 (2010-2012). ... 108 Fig. 3.26 Colormap of the reconstructed monthly TP concentrations at water quality monitoring stations S1-S7 (2010-2012). ... 109 Fig. 3.27 Monthly WQI values (including TP) at water quality monitoring stations S1-S7 by incorporating the reconstructed monthly TP concentrations (2010-2012). ... 109
Table contents
Table 3.1 Model performance of two- to six-step-ahead forecasting for Mackey-Glass
time series ... 44
Table 3.2 Summary statistics of reservoir inflow and average hourly rainfall in training and testing datasets... 45
Table 3.3 Model performance of two-step-ahead forecasting for reservoir inflow ... 49
Table 3.4 Model performance of four-step-ahead forecasting for reservoir inflow ... 49
Table 3.5 Model performance of six-step-ahead forecasting for reservoir inflow ... 49
Table 3.6 Summary statistics for FSP water levels (m) and the peaks of average rainfall (mm/10 min) ... 60
Table 3.7 Model performance of one- to six-step-ahead forecasting for FSP water levels ... 66
Table 3.8 Model performance of one- to six-step-ahead forecasting for FSP water levels ... 72
Table 3.9 Statistics of groundwater quality parameters at six gauging stations (wells) during 1992 and 2005. ... 83
Table 3.10 Correlation matrix of As concentration and water quality parameters collected at six gauging stations (wells) during 1992 and 2005. ... 84
Table 3.11 Estimation performance of the NARX network and the BPNN for As concentration at 20 ungauged sites between 1995 and 1999 in the testing phase... 90
Table 3.12 Test methods and preliminary statistics of water quality parameters collected at water quality monitoring stations S1-S7 in the Dahan River basin during the model calibration period (2002-2009). ... 99 Table 3.13 Performance of the NARX network and the BPNN in the testing phases
(2010-2012) for TP concentration estimation at water quality monitoring stations S1-S7. ... 105
1. Introduction
1.1 Motivation
Water resources are precious and can be utilized by human beings. The scarce sources of water on the Earth have gained more and more attention for decades. Water resources support the activities of agricultural, industrial, household, recreational, environmental and various sectors. Therefore, we all live in hydro-environmental systems and have complex interactions with these systems. In recent years, due to the comprehensive interactions between hydrological, meteorological, geographical factors and human activities with climate change effects, the hydro-environment that we live by is facing imbalanced conditions. The intensive storms and typhoons with short durations and the degradation of the water quality in groundwater and urban rivers are becoming serious but common disasters. To tackle such challenges, this dissertation focuses on two main problems that have occurred in hydro-environmental systems: water quantity and water quality issues, and endeavors to develop novel dynamic artificial neural networks and modeling schemes to solve these water-related problems.
Artificial neural networks (ANNs) are biologically motivated methods in which large numbers of neurons communicate with each other through weighted connections, and have the ability to approximate nonlinear functions for modeling time series. ANNs
are also considered as an alternative computational approach to modeling physical-based problems. In the last decades, ANNs have been widely applied with success to various water resources problems, such as rainfall-runoff modeling (Antar et al., 2006; Chang et al., 2007; Chen and Yu, 2007; Chang et al., 2013; Chang et al., 2014;
Yang et al., 1999), flood control (Chang et al., 2008), evaporation estimation (Chang and Sun, 2013; Chang et al., 2013), reservoir operation (Chang et al., 2010; Chang and Wang, 2013; Wang et al., 2010), groundwater level prediction (Krishna, et al., 2008 ; Nikolos, et al., 2008), and water quality estimation (Khalil et al., 2011; McNamara et al., 2008; Sahoo et al., 2006). However, static neural networks may not be able to establish reliable nonlinear models for predicting dynamical systems, especially for many time-step-ahead forecasting or regional estimation. Recurrent neural networks (RNNs), which belong to a class of dynamic ANNs, are powerful nonlinear tools capable of extracting dynamic behaviors from complex systems through internal recurrence and have attracted much attention for years (Assaad et al., 2005; Chang et al., 2012; Chiang et al., 2010; Ma et al., 2008; Serpen and Xu, 2003). Nevertheless the batch training of an RNN could be time consuming (Ahmad and Jie, 2002; Xie et al., 2006), and the behaviors of recurrent connections in spatio-temporal estimation has not been fully explored yet. Therefore, this dissertation develops a reinforced online-learning algorithm for RNNs and explores the practical meaning as well as the importance of
recurrent connections of the NARX network through topic 1: water quantity issues involving reservoir inflow forecasting and urban flood control; and topic 2: water
quality issues involving the spatio-temporal estimation with respect to the arsenic (As)
concentration in groundwater and the total phosphate (TP) concentration in a river
basin.
1.2 Research objectives
A. Topic 1: Water quantity issues for reservoir inflow forecasting and urban flood
control
Accurate multi-step-ahead (MSA) forecasting is valuable and desired in many engineering problems, however it is a challenging task that is difficult to achieve. A common approach for improving the accuracy of MSA forecasting is to update network parameters through online learning techniques. Online learning is a supervised machine-learning framework, which adopts the latest information to adjust model parameters for a better mapping between instances and true values in an arbitrary system. Because most observational disciplines tend to infer the properties of an uncertain system from the analysis of time-dependent data, the analytical techniques for extracting the meaningful characteristics of time series data have certain inherent limitations, which have been widely discussed (Brockwell and Davis, 1987; Jaeger and Haas, 2004). Owing to the continual receipt of true values for adjusting model
parameters, online learning algorithms have several practical and theoretical advantages such as memory-efficient implementation, runtime-efficient implementation and strong guarantees on performance, even in a highly variable data structure of time series (Shalev-Shwartz et al., 2004). Nevertheless, the main defect of online learning can be attributed to the requirement for continual true values. Engineering problems often require models to predict many time steps into the future without the availability of measurements in the horizon of interest. The lack of true values makes it difficult to make MSA forecasts. In addition, many studies indicated that it is not an adequate strategy to recursively adopt single-step-ahead predictions for many time steps into the future because the errors of MSA predictors will be accumulated based on the single-step-ahead predictor (Parlos et al., 2000; Yong et al., 2010). Such time-lag problems may cause significant degradation in performance when dealing with MSA forecasting for real-world applications. For the MSA streamflow forecasting during typhoon events, models with time-lag problems (i.e., the latest observed values are unavailable) cannot keep flow trails, especially in peak flows, as the forecasting step increases. To mitigate time-lag phenomena that occur in online learning algorithms, it is argued that whether iterative adjustments of model parameters in consideration of additional information, such as the latest true values and/or antecedent model outputs, would be beneficial to MSA forecasting.
The real-time recurrent learning (RTRL) algorithm, proposed by Williams and Zipser (1989), is an effective and efficient online learning algorithm for training recurrent networks, in which real-time adjustments are made to the synaptic weights of recurrent networks. Several studies demonstrated that the RTRL algorithm for RNNs is very effective in modeling the dynamics of complex processes for providing accurate predictions (Chang et al., 2002; Chang et al., 2012; Hirasawa et al., 2000; Li et al., 2002).
The first main goal of topic 1 is to develop a reinforced RTRL algorithm for RNNs (R-RTRL NN) to mitigate time-lag effects for increasing the accuracy of MSA forecasting. The sequential formulation of the R-RTRL NN is derived, and its reliability and applicability are further demonstrated through two-step-ahead (2SA), four-step-ahead (4SA) and six-step-ahead (6SA) forecasting made for a famous benchmark chaotic time series and a reservoir inflow case in Taiwan. Comparative models consist of the original RTRL algorithm for RNNs (RTRL NN), the Elman neural network (Elman NN) (Elman, 1990; Liu and Wang, 2008; Liu et al., 2012) and the backpropagation neural network (BPNN, the most popular static ANN).
Urban flood control is a crucial and challenging task, particularly in developed cities. Urban floods are flashy in nature mainly due to severe thunderstorms and occur both on urbanized surfaces and in small urban creeks, which deliver mass water to cities.
On account of more impervious areas resulting from the rapid urbanization in metropolitan areas, less water infiltration has resulted in an increase in the flow rate and the amount of surface runoff over the last decades. Taiwan is located in the northwestern Pacific Ocean where subtropical air currents frequently introduce typhoons and convective rains. The urban flood hydrographs in Taiwan typically have large peak flows and fast-rising limbs in a matter of minutes, which could cause serious disasters.
For example, Typhoon Nari brought massive rainfalls at an astonishing level of 500 mm/day on September 17th in 2001, which resulted in 27 deaths, inundations at some stations of the Taipei Metro System, and countless economic losses. The heavy rainfall event on June 12th in 2012 brought astonishing rainfalls with a cumulative amount of 54.1 mm/hr, which directly resulted in quick and wide surface flooding such that the transportation system collapsed in most of the southern Taipei City. It appears that floods cannot be prevented, but planning emergency measures through flood management might mitigate disastrous consequences.
In response to the flood threats to residents and property, the Taipei City Government has long-term endeavored to develop flood control-related infrastructures, such as increasing levee heights and enhancing sewerage systems, and therefore urban inundations have been significantly mitigated and controlled in recent years. As a result, the main threat to the city turns out to be the floodwater inside the levee system. A
surface inundation will inevitably take place if surface runoff exceeds the capacity of a storm drainage system. To tackle this problem, pumping stations play an important role in flood mitigation at metropolitan areas and are principal hydraulic facilities built to manage internal stormwater flows at places under the condition that gravity drainage cannot be achieved. The operation of a pumping station highly depends on the water level information of its floodwater storage pond (FSP). Within the catchment of a pumping station, surface runoff will drain to its FSP for storage and subsequent disposal through gravity drainage. When the water level of the FSP reaches the start level of duty pumps, the pumps will be activated according to operation rules for discharging the stored floodwater into the nearby river of the pumping station. For floodwater control management during heavy rainfall or typhoon events, it is imperative to construct an efficient and accurate model to forecast many step-ahead FSP water levels by utilizing the information of the current FSP water level and the rainfall measured at the neighboring rainfall gauging stations of the pumping station. The proposed model is expected to provide sufficient response time for warming up the pumps in advance for enhancing secure pumping operations and urban flood control management.
The greatest success in flood forecasting is commonly achieved on large rivers.
Nevertheless, flash urban floods associated with heavy thunderstorms in cities are often very uncertain and are more difficult to predict due to complex dynamic phenomena
involved. Many studies demonstrated the predictability of streamflow through soft computation methods (Maity and Kumar, 2008) while only few papers investigated the prediction performance of inundation and/or sewerage systems in urban areas (Chiang et al., 2010). The second main goal of topic 1 intends to investigate the reliability and accuracy of short-term (10- to 60-minute) forecasting models for the FSP of a sewer-pumping system in Taipei City. Multi-step-ahead FSP water level forecasting models for flood pumping control during heavy rainfall and/or typhoon events are tailored made through a static ANN (the BPNN) and two dynamic ANNs (the Elman NN; the NARX network). Consequently, the comparison results of these three ANN models are evaluated to identify the effectiveness of recurrent connections. The forecasting system is designed to anticipate the occurrence of flooding and to take measures necessary to reduce flood-induced losses. The study will give a boost to the efforts for urban flood disaster management and will strengthen the Taipei City Government with more proactive disaster preparedness.
B. Topic 2: Water quality issues for the spatio-temporal estimation with respect to
the As concentration in groundwater and the TP concentration in a river basin
The second topic of this dissertation focuses on water quality issues for which the stabilization and variation of concentrations are important tasks for preserving healthy human and hydro-environmental systems.
As contamination in groundwater has been reported and resulted in a massive epidemic of As toxication in several countries such as Bangladesh, Vietnam, Cambodia, China and Taiwan. It is estimated that approximately 57 million people have drunk As-contaminated groundwater with concentrations exceeding the drinking water standard recommended by the WHO (World Health Organization) (BGS-DPHE, 2001;
Chakraborti et al., 2010). As pollution affects not only crop productivity and water quality but also the quality of water bodies, which threatens the health of animals and human beings by way of food chains. Long-term exposure to As through drinking water has been implicated in a variety of health concerns including cancers, cardiovascular diseases, diabetes and neurological effects (National Research Council, 1999).
Blackfoot disease and cancers of the skin, bladder, lung and liver have been associated with drinking As-contaminated groundwater (Chiou et al., 1997; Rahman, 1999).
As-contaminated groundwater is derived naturally from As-rich aquifer sediments, and the geochemistry of As can be rather complex (Stollenwerk, 2003). Various hydrogeological and biogeochemical factors affecting As concentration in groundwater have been detected, such as sediment mineralogy, microbial oxidation or reduction of As, groundwater recharge, groundwater flow paths (Ford et al., 2006; Wang et al., 2007
& 2011; Xie et al., 2012), and the presence of fractures in bedrock formations (Ayotte et al., 2003; Liao et al., 2011). Even though the processes controlling the release of As into
groundwater systems have been extensively discussed over the past decades, exact chemical conditions and reactions leading to As mobilization still remain a subject of intense debate (Goovaertset al., 2005; Polizzotto et al., 2006; Winkel et al., 2008).
Moreover, the high variability of As concentration can occur within a short distance and/or in different depths of groundwater wells due to the diversity in geology and geomorphology (Serre et al., 2003; Yu et al., 2003). Besides, the detection of As contamination in groundwater by using graphite atomic absorption spectrophotometry or inductively coupled plasma mass spectroscopy can be laborious and cost intensive.
Consequently, how to adequately estimate As concentrations in complex hydro-geological systems is a crucial and challenging task.
The hyper-endemic blackfoot disease in the Yun-Lin County of Taiwan has been verified to be associated with high As concentrations in groundwater (Chen et al., 1995;
Chiou et al., 1997). The residents have long-term exposed themselves to As through various paths such as the ingestion of aquacultural and agricultural products, and thus have dangerously posed carcinogenic risks to their health (Liu et al., 2008). Due to great concern for the potential effects of As on human health, there is a growing need for efficiently modeling the spatial distribution of As contamination in groundwater. One of the popular modeling approaches in use is the multiple linear regression (MLR), this approach, however, may fail to estimate the spatial distribution of As contamination due
to the great variability of As concentration and complex nonlinear processes involved in geology and geomorphology. Lately, using ANNs for the estimation of heavy metal concentration in groundwater has been attempted and gained a reasonably good degree of success (Chang et al., 2010; Cho, et al., 2011; Giri et al., 2011; Mondal et al., 2012;
Purkait et al., 2008). The modeling results indicated that ANN techniques can produce higher estimation accuracy than conventional methods such as MLR. These studies were mostly dedicated to exploring the applicability of static ANNs, such as the BPNN, for building the relationship between As concentration in groundwater and hydro-geological parameters in As-affected areas. Nevertheless, the natural characteristics of hydrogeological processes are not only complex but also dynamic.
The static neural networks might fail to establish reliable models for predicting the dynamical features, such that the delivered relationship might be simply the possible impacts of factors on temporal characteristics of local environments. Consequently, the comprehensive analysis of dynamic hydrogeological features and the estimation of As concentration variability over As-affected regions remains a great challenge that needs to be overcome.
The seasonal variation of steamflow in Taiwan is very high, where long-lasting low flows in drought seasons could dramatically increase the pollution levels in rivers.
Pollution in the downstreams of rivers raises a major environmental issue because many
industrial facilities and large populated cities are located along rivers. The water quality of the Dahan River in northern Taiwan has deteriorated rapidly due to heavy pollutant loads from surrounding urban areas. Considering the scattered watersheds over Taiwan and the high cost of field sampling, it is unlikely to obtain continuous water-quality time series data with complete properties at all sampling locations. Alternatively, the Water Quality Index (WQI) has been designed to assess the general conditions of water bodies in rivers, lakes or reservoirs. The WQI is sensitive to light pollution, and therefore it is a more suitable index adopted for water quality management. The WQI numerically summarizes the information of multiple water quality parameters into a single value, including dissolved oxygen (DO), coliform group, power of hydrogen (pH), biochemical oxygen demand (BOD), ammonia nitrogen (NH3-N), suspended solid (SS) and total phosphate (TP). Except for TP (measured quarterly), the other water quality parameters adopted in the WQI are measured monthly in Taiwan. Therefore, a monthly WQI incorporated with TP would be more comprehensive and more beneficial to short-term (monthly) water quality management.
TP, a combination of orthophosphate, polyphosphate and organic phosphate, is regarded as an index used in representation of the phosphorus quantity in river water.
Phosphorus is an essential element for all life forms (Correll, 1998). When phosphorus enters into a river, it is usually in the form of phosphate and can be transported from
upstream to downstream by flowing water. Excessive phosphorus is the most common cause of eutrophication in freshwater lakes, reservoirs, streams, and headwaters of estuarine systems. Orthophosphate chemicals are commonly used in agricultural fertilizers, and thus enter surface water easily during rainfall periods. Many studies reported that the phosphorus fertilizer form affects phosphorus loss to waterways (Azevedo et al., 2013; Davis and Koop, 2006). Polyphosphate is a primary chemical element added with considerable amount into detergents. Organic phosphates are basically formed by biochemical procedures associated with excrement, kitchen waste, water plants, etc. Phosphorus is one of the key elements essential for the growth of plants and animals. Nevertheless, the anthropogenic nutrient enrichment of natural water is of environmental importance as it can evoke declines in water quality, changes in biotic population structures, and low dissolved oxygen concentrations in rivers (Dodds et al., 2009; Austin et al., 1996). Excessive phosphorus has been shown to be a main cause of eutrophication, for example, naturally-occurring nutrients in large concentrations can often cause algae blooms (McDowell et al., 2010; Carpenter et al., 1998).
Water quality models are useful tools for estimating the levels and risks of chemical pollutants in a given water body (Duda, 1993). When building (or just applying) a water quality model, it is necessary to have long and sufficient field data to
validate model applicability and reliability. Water quality monitoring programs, however, are expensive and time-consuming. Modeling practices commonly face limited budgets and time, and thus suffer a deficiency of field data. Under this condition, the implemented water quality models might fail to fit known hypotheses and/or assumptions or cause difficulties in making estimations within an acceptable range of errors or uncertainty. With the development of model theory and the fast-updating computer techniques, many artificial intelligent techniques have been developed with various analytical algorithms to overcome data scarcity issues and simultaneously increase model reliability.
The NARX network (Lin et al., 1996), a sub-class of RNNs, is suitable to build long-term temporal input-output patterns (Menezes Jr. and Barreto, 2008). The NARX network has been demonstrated to perform well in several nonlinear systems, such as waste water treatment plants (Su and McAvoy, 1991; Su et al., 1992) and time series forecasting (Shen and Chang, 2013). However, the dynamic feature and feasibility of the recurrent connections in the NARX network as a nonlinear tool for water quality time series modeling under limited data sets has not been fully explored yet. Therefore, topic 2 will explore the practical meaning and importance of recurrent connections in
the NARX network when dealing with spatio-temporal water quality estimation problems.
In topic 2, a systematical dynamic-neural modeling (SDM) scheme incorporated with a dynamic neural network and advanced statistical methods is developed for building spatio-temporal estimation models for (1) As concentration at decommissioned wells based on the easily-measured water quality parameters at nearby functioning wells to offer an applicable and useful reference to decision makers for dealing with groundwater management and preventing residents from drinking or using toxic groundwater; and (2) TP concentrations at seven sites along the Dahan River in a quarterly scale based on easily-measured water quality parameters. In addition, TP concentration data are reconstructed in a monthly scale through a process that adopts the dynamical neural architecture of the constructed NARX network, and thus the reconstructed monthly data can be used to produce the monthly WQI for short-term hydro-environmental management.
1.3 Dissertation layout
The framework of this dissertation is shown in Fig. 1.1. The novel learning algorithm (R-RTRL) and the modeling procedure (SDM) are developed to deal with two main topics in hydro-environmental systems: water quantity and water quality. First, The R-RTRL NN can repeatedly adjust model parameters through the reinforced process with the current information including the latest observed values and model outputs to enhance the reliability and the forecast accuracy of the proposed method. To
demonstrate its reliability and effectiveness, the proposed R-RTRL NN is implemented to make 2-, 4- and 6-step-ahead forecasts in a famous benchmark chaotic time series and a reservoir flood inflow series in North Taiwan. Second, the SDM which consists of the Gamma test (GT) for input factor selection and the NARX network for spatio-temporal estimation is proposed. The SDM is then applied to 1) urban flood control problems to explore the contribution of recurrent connections and 2) the spatio-temporal estimation of As and TP concentrations to provide useful information to water authorities for dealing with groundwater and river basin management.
Fig. 1.1 Framework of the dissertation
2. Methodology
2.1 MSA R-RTRL algorithm for RNNs
A. Rationale of MSA online learning algorithm
Two common strategies for MSA forecasting are the iterated prediction and the direct prediction. For n-step-ahead (nSA) prediction, the iterated method tackles the issue by iterating n times a one-step-ahead prediction whereas the direct method trains the model by conducting a direct forecast at time t+n. The debate on the superiority between these two methods still remains open; nevertheless both methods possess a common feature: the visibility of stochastic dependencies between future values becomes relatively vague as the time of prediction horizon increases, consequently the reliability and accuracy of predictions decreases. A possible way to remedy this shortcoming is to implement online learning techniques for repeatedly adjusting model parameters with the most current information including the latest true (observed) values and model outputs. An online learning algorithm proceeds in a sequence of trials through receiving an instance and making a prediction in each online-learning round to improve model performance.
The original RTRL algorithm, an online learning algorithm, was derived for one-step-ahead forecasting from the fact that real-time adjustments are made to the synaptic weights of a fully connected recurrent neural network (Williams and Zipser,
1989). For nSA forecasting, the weight adjustment of the RTRL algorithm cannot be conducted until obtaining the observed value at time t+n, in which the observed values and model outputs during time t+1 and t+n-1 are worthless and totally ignored.
Therefore, the effectiveness of the original RTRL algorithm decreases considerably when time step n increases, which implies time lags occur in the weight adjustment process.
This dissertation proposes a novel reinforced RTRL algorithm based on RNN infrastructures (R-RTRL NN) for MSA forecasting through incorporating the latest antecedent forecasted and observed values into consecutive temporary networks for weight adjustments in the learning process. In other words, the R-RTRL algorithm repeatedly updates the synaptic weights by utilizing the most current obtainable information. The applicability and effectiveness of the R-RTRL NN is further investigated in Section 3.
The upper diagram of Fig. 2.1 shows the weight adjustment procedure of the R-RTRL algorithm for 2SA forecasting (Chang et al., 2012). At time t+2, the weights are adjusted by the differences between observed and forecasted values. A reinforced process is introduced: the RNNtemp with adjusted weights can be used to produce a temp output zˆ(t3, 1)at time t+1, and the error between the temp output and forecasted output at time t+1 can then be utilized to reinforce the weight adjustments, ˆ ()
1 t
W and
n-Step-Ahead Two-Step-Ahead
) 1 (t X RNNtemp
X
RNN
) (t X X(t1)
) 2 (t Z
RNN
) 1 (t X
) 3 (t Z
RNNtemp
X
) 2 (t X
) 4 ˆ (t Z
) 2 (t X
) 4 (t Z
X
) 3 (t Z
) 3 ˆ (t Z
X
V W
,
V Wˆ, ˆ
V
W
,
)
D(t 2 D(t 3 )
...
EXPAND V
W
, Weight adjustment
V Wˆ,ˆ
Reinforced weight adjustment RNNtemp Temporary RNN RNN Recurrent neural
network
X Reinforced error
X Error
) ˆ t(
Z Repredicted output vector at time t
) (t
Z Predicted output vector at time t
) (t
D Target output vector at time t
) (t
X Input vector at time t
) 1 (t X RNNtemp
X
RNN
) (t
X X(t1) ) ( nt
Z Z(tn1)
) 1 ˆ (tn Z
X ) 1 ˆ(tn e
V W
,
...
...
) 1 (tn X
) 1 2 (t n Z
) 2 (t X RNNtemp
) 2 ˆ (tn Z
X
...
) 1 (tn X
RNNtemp
) 1 2 ˆ (tn Z
X
...
V Wˆ, ˆ
) 2 (t X
) 2 (tn Z
n) D(t
) 2 ˆ(tn e
) 1 2 ˆ(t n e )
ˆ t(
e Reinforced error at time t
Fig. 2.1 Weight adjustment procedure for the n-step-ahead R-RTRL NN.
) 1 ˆ(
1
V t . As this reinforced process repeats n-1 times, the weight adjustment procedure
can be extended to a general procedure for nSA (n2, nN) forecasting, shown in the lower diagram of Fig. 2.1. In summary, the proposed R-RTRL algorithm not only adequately utilizes the up-to-date information of the observed values and their corresponding model outputs but also strengthens the usefulness of the latest observed values by the reinforced process to mitigate the time-lag phenomenon for MSA forecasting. The detailed sequential formulation of the R-RTRL algorithm is described as follows.
B. Deriving the MSA R-RTRL algorithm
Fig. 2.2 shows the MSA RNN architecture incorporated with the R-RTRL algorithm, in which there are M external inputs and one output. Let X(t) denote the M × 1 input vector at discrete time t, Y(t+1) denote the corresponding N × 1 vector at time t+1 in the processing layer, and Z(t + n) denote the corresponding output value for nSA
(n2, nN) forecasting.
The X(t) and Y(t) are concatenated to form the ( M + N ) × 1 vector U(t), whose ith element is denoted by μi(t). Let A denote the set of indices i for which xi(t) is an external input, and B denote the set of indices i for which yi(t) is the output of the processing layer. Thus, vector μi(t) can be represented as Eq. (1).
1
1
N 1 M
1
N
Output Layer
Processing Layer
Concatenated Input Layer Output
Z(t+n)
…
…
…
VNx1
WNx(M+N)
Y(t+1)
Input vector
X(t) Y(t)
unit time delay
Fig. 2.2 Architecture of the multi-step-ahead RNN with the R-RTRL online learning algorithm.
𝜇𝑖(𝑡) = {𝑥𝑖(𝑡) 𝑖𝑓 𝑖 ∈ 𝐴
𝑦𝑖(𝑡) 𝑖𝑓 𝑖 ∈ 𝐵 (1)
W and V denote the weight matrices in the processing layer and the output layer, respectively. 𝑊 ↔ 𝑤𝑗𝑖 and 𝑉 ↔ 𝜈𝑗 are of matrix forms. The output of neuron 𝑗 in the
processing layer that presents the transformation of information from the concatenated layer through nonlinear system 𝑓 is given by:
𝑦𝑗(𝑡 + 1) = 𝑓 (𝑛𝑒𝑡𝑗(𝑡 + 1)) = 𝑓 ( ∑ 𝑤𝑗𝑖(𝑡)𝜇𝑖(𝑡)
𝑖∈𝐴∪𝐵
) (2)
The net output of the output layer at time t + n through nonlinear system 𝑓 is
computed by:
𝑧(𝑡 + 𝑛) = 𝑓(𝑛𝑒𝑡(𝑡 + 𝑛)) = 𝑓 (∑ 𝜈𝑗(𝑡 + 1)𝑦𝑗(𝑡 + 1)
𝑗
) (3)
Let d(t + n) denote the target value at time t+n. The time-varying error e and instantaneous error E is defined by:
𝐸(𝑡 + 𝑛) =1
2𝑒2(𝑡 + 𝑛) =1
2[𝑑(𝑡 + 𝑛) − 𝑧(𝑡 + 𝑛)]2 (4)
Then the weight adjustments can be computed by minimizing the instantaneous error at time t + n.
∆𝜈𝑗(𝑡 + 1) = −𝜂1𝜕𝐸(𝑡 + 𝑛)
𝜕𝜈𝑗(𝑡 + 1) (5)
∆𝑤𝑗𝑖(𝑡) = −𝜂2𝜕𝐸(𝑡 + 𝑛)
𝜕𝑤𝑗𝑖(𝑡) (6)
where η1 and η2 are learning-rate parameters. The detailed recurrent learning algorithm for one-step-ahead weight adjustment can be found in Williams and Zipser (1989), and that for two-step-ahead weight adjustment can be found in Chang et al.
(2012). The entire antecedent information is considered crucial and could diminish time-lag effects. Consequently, the reinforced two-step weight adjustments (Chang et al.
2012) can be extended to n-step weight adjustments, and the information obtained from time t+1 to t+n-1 can contribute to weight adjustments. The adjusted weights are used to re-calculate the forecasted values at time from t+1 to t+n-1, and then the adjusted weights are further modified by minimizing the total error between original forecasted
values (𝑧(𝑡 + 𝑛 + 1) to 𝑧(𝑡 + 2𝑛 − 1)) and the re-forecasted values (𝑧̂(𝑡 + 𝑛 + 1) to 𝑧̂(𝑡 + 2𝑛 − 1)).
The re-forecasted value 𝑧̂(𝑡 + 𝑛 + 𝑝) is calculated by the following equations:
𝑦̂𝑗(𝑡 + 𝑝 + 1) = 𝑓 (𝑛𝑒𝑡̂𝑗(𝑡 + 𝑝 + 1))
= 𝑓 ( ∑ (𝑤𝑗𝑖(𝑡) + ∆𝑤𝑗𝑖(𝑡)) 𝜇𝑖(𝑡 + 𝑝)
𝑖∈𝐴∪𝐵
)
(7)
𝑧̂(𝑡 + 𝑛 + 𝑝) = 𝑓(𝑛𝑒𝑡̂ (𝑡 + 𝑛 + 𝑝))
= 𝑓 (∑ (𝜈𝑗(𝑡 + 1) + ∆𝜈𝑗(𝑡 + 1)) 𝑦̂𝑗(𝑡 + 𝑝 + 1)
𝑗
) (8)
where p denotes the time step (p = 1, 2, …, n-1). Therefore, the total reinforced error is defined by:
𝐸̂ =1
2∑ 𝑒̂2(𝑡 + 𝑛 + 𝑝)
𝑛−1
𝑝=1
=1
2[𝑧̂(𝑡 + 𝑛 + 𝑝) − 𝑧(𝑡 + 𝑛 + 𝑝)]2 (9)
The reinforced weight adjustments can be expressed as:
∆𝜈̂𝑗(𝑡 + 1) = −𝜂3 𝜕𝐸̂
𝜕𝜈𝑗(𝑡 + 1) (10)
∆𝑤̂𝑗𝑖(𝑡) = −𝜂4 𝜕𝐸̂
𝜕𝑤𝑗𝑖(𝑡) (11)
where η3 and η4 are learning-rate parameters.
The weight adjustments of the R-RTRL algorithm for n-step-ahead RNNs are then shown as follows:
𝑤𝑗𝑖(𝑡 + 1) = 𝑤𝑗𝑖(𝑡) + ∆𝑤𝑗𝑖(𝑡) + ∆𝑤̂𝑗𝑖(𝑡) (12)
νj(t+2)=νj(t+1)+∆νj(t+1)+∆ν̂j(t+1) (13)
In sum, the reinforced process is implemented for nSA forecasting so that the adjusted weights are further modified through the comparison between the original forecasted values and the re-forecasted values.
2.2 Systematical Dynamic-neural Modeling (SDM)
The proposed SDM incorporates two core methods, the GT and the NARX network, with three optional statistical techniques to tackle rainfall-runoff modeling and spatio-temporal estimation problems, and its implementation procedure is shown in Fig.
2.3. The SDM first adopts the GT to effectively extract the non-trivial factors that significantly affect the fluctuations of the target variable (e.g., FSP water level, As or TP concentrations). The NARX network is then utilized to obtain forecasted or estimated values of the target variable with inputs consisting of the extracted non-trivial factors and the previous output values derived from recurrent connections. In addition, the time series of the target variable can be reconstructed in a minor scale (e.g., from a seasonal scale to a monthly scale) through the constructed NARX network for further evaluation of water quality. The scarcity of field data is commonly encountered in water quality modeling, and the following three optional methods can be incorporated into the
Fig. 2.3 Implementation procedure of the proposed SDM
proposed SDM to tackle data scarcity problems: the Bayesian regularization method;
the cross validation technique; and the indicator kriging (IK). The Bayesian regularization method is configured to control the network complexity for preventing over-fitting. The cross validation technique is used to produce a low-bias estimator of the generalizability, and thus provides a sensible criterion for model selection in the calibration stage. Finally, the IK can be implemented to derive the probability map of the target variable in unsampled areas. The methods for use in the SDM are introduced
as follows:
A. Core techniques in the SDM
1) Gamma test (GT)
The GT, presented by Koncar (1997) and Agalbjorn et al. (1997), is a data analysis technique for assessing the extent to which a given set of M data points can be modeled by an unknown smooth non-linear function.
Suppose a set of input-output observation data is given in the form of:
(xi ,yi),1iM
(14)where vectors xi are d dimensional vectors (with a record length of M) and the corresponding outputs yi are scalars. The underlying relationship of the system is expressed as:
r x x f
y ( 1.... d) (15)
where f is an unknown smooth function, and r denotes a random variable that represents noise. The Gamma statistic () is an estimate of the variance of model outputs, which cannot be accounted for through a smooth data model. The GT is assessed based on the kth
1k p
nearest neighborXN( ki, )for each vector Xi, and then the GT can bederived from the Delta function of input vectors:
1 k p
) 1
(
2
1 ,
M
i ik i
M X X
k M
(16)
where is the Euclidean distance, and the corresponding Gamma function of the
output values is given in Eq. (15). The number of p depends on the density of sampling data (Koncar, 1997). In this dissertation, the number of p is determined as the value that
produces the minimum value through trial and error (p ranges from 10 to 50).
1 k p
2
) 1 (
2
1 (, )
M
i N ik i
M y y
k M
(17)
where yN( ki, ) is the corresponding y-value for the kth nearest neighbor of Xi, in Eq. (14).
For computing , a least squares regression line is constructed for p points ))
( ), (
(M k M k as Eq. (16):
Γ
γAδ (18)
where A is the gradient.
Performing a single GT is a fast procedure, which can provide the noise estimate
( value) for each subset of input variables. When the subset for which its associated
value is the closest to zero, it can be considered as “the best combination” of input
variables. As the result, the GT is different from other input selection methods or preprocessing methods, such as the correlation coefficient analysis or the principle component analysis (PCA). The correlation coefficient analysis can only provide linear estimation for the input-output datasets, and the principle components determined by the PCA are calculated from all input variables and thus can neither extract the most important factors nor reduce the dimension of input datasets.
Several studies discussed about the GT theory and its applications in time series
forecasting (Durrant, 2001; Tsui et al., 2002). Lately, research findings indicated that it is suitable and effective to combine ANNs with the GT for reducing the input dimension through identifying non-trivial input variables and thus produces precise outputs of ANNs (Moghaddamnia et al., 2009; Noori et al., 2010; Noori et al., 2011).
Therefore, the NARX network combined with the GT is used to estimate water quality and forecast water level in this dissertation.
2) Nonlinear Autoregressive with eXogenous input (NARX) network
The NARX network is a recurrent network, which is suitable for time series prediction (Chang et al., 2013; Jiang and Song, 2011; Menezes Jr. and Barreto, 2008, Shen and Chang, 2013). Figure 2.4 shows the architecture of the NARX network used in this dissertation. The NARX network consists of three layers and produces recurrent connections from the outputs, which may delay several unit times to form new inputs.
Therefore, this nonlinear system for estimation (N 0) and N-step-ahead forecasting
(N 1) can be mathematically represented by the following equation:
( 1), , ( ); ( )
)
(t N f z t N z t N q U t
z (19)
where U(t) and z(t+N) denote the input vector and output value at the time step t, respectively. f(‧) is the nonlinear function, and q is the output-memory order. There are
1
) (t U
) (t N z
) 1 (t N ) z
(t N q
z z(t N 2) Observed data
∑
) (t N d
-
+
) (t N e
V
W
1
1
Fig. 2.4 Architectures of the NARX network
two input regressors: the regressor z(t+N-i) (i ranges from 1 to q) plays the role of an autoregressive model while U(t) plays the role of an implicit exogenous variable in time series.
There are two ways to train the NARX network. The first mode is the Series-parallel (SP) mode, where the output’s regressor in the input layer is formedonly
by the target (actual) values of the system, d(t).
( 1), , ( ); ( )
)
(t N f d t N d t N q U t
z (20)
The other alternative is the Parallel (P) mode, where estimated outputs are fed back into the output’s regressor in the input layer, which can also be mathematically
represented as Eq. (19). In practice, when forecasting is conducted for more than two-step-ahead (N 1), the q antecedent actual values (d(t+N−1), d(t+N−2),. . . ,
‧‧‧
‧‧‧
‧‧‧
