MSA R-RTRL algorithm for RNNs

A. Rationale of MSA online learning algorithm

Two common strategies for MSA forecasting are the iterated prediction and the direct prediction. For n-step-ahead (nSA) prediction, the iterated method tackles the issue by iterating n times a one-step-ahead prediction whereas the direct method trains the model by conducting a direct forecast at time t+n. The debate on the superiority between these two methods still remains open; nevertheless both methods possess a common feature: the visibility of stochastic dependencies between future values becomes relatively vague as the time of prediction horizon increases, consequently the reliability and accuracy of predictions decreases. A possible way to remedy this shortcoming is to implement online learning techniques for repeatedly adjusting model parameters with the most current information including the latest true (observed) values and model outputs. An online learning algorithm proceeds in a sequence of trials through receiving an instance and making a prediction in each online-learning round to improve model performance.

The original RTRL algorithm, an online learning algorithm, was derived for one-step-ahead forecasting from the fact that real-time adjustments are made to the synaptic weights of a fully connected recurrent neural network (Williams and Zipser,

1989). For nSA forecasting, the weight adjustment of the RTRL algorithm cannot be conducted until obtaining the observed value at time t+n, in which the observed values and model outputs during time t+1 and t+n-1 are worthless and totally ignored.

Therefore, the effectiveness of the original RTRL algorithm decreases considerably when time step n increases, which implies time lags occur in the weight adjustment process.

This dissertation proposes a novel reinforced RTRL algorithm based on RNN infrastructures (R-RTRL NN) for MSA forecasting through incorporating the latest antecedent forecasted and observed values into consecutive temporary networks for weight adjustments in the learning process. In other words, the R-RTRL algorithm repeatedly updates the synaptic weights by utilizing the most current obtainable information. The applicability and effectiveness of the R-RTRL NN is further investigated in Section 3.

The upper diagram of Fig. 2.1 shows the weight adjustment procedure of the R-RTRL algorithm for 2SA forecasting (Chang et al., 2012). At time t+2, the weights are adjusted by the differences between observed and forecasted values. A reinforced process is introduced: the RNNtemp with adjusted weights can be used to produce a temp output zˆ(t3, 1)at time t+1, and the error between the temp output and forecasted output at time t+1 can then be utilized to reinforce the weight adjustments, ˆ ()

1 t

W and

n-Step-Ahead

 Reinforced weight adjustment RNNtemp Temporary RNN RNN Recurrent neural

Z Repredicted output vector at time t

) (t

Z Predicted output vector at time t

) (t

D Target output vector at time t

)

e Reinforced error at time t

Fig. 2.1 Weight adjustment procedure for the n-step-ahead R-RTRL NN.

) 1 ˆ(

1 

V t . As this reinforced process repeats n-1 times, the weight adjustment procedure

can be extended to a general procedure for nSA (n2, nN) forecasting, shown in the lower diagram of Fig. 2.1. In summary, the proposed R-RTRL algorithm not only adequately utilizes the up-to-date information of the observed values and their corresponding model outputs but also strengthens the usefulness of the latest observed values by the reinforced process to mitigate the time-lag phenomenon for MSA forecasting. The detailed sequential formulation of the R-RTRL algorithm is described as follows.

B. Deriving the MSA R-RTRL algorithm

Fig. 2.2 shows the MSA RNN architecture incorporated with the R-RTRL algorithm, in which there are M external inputs and one output. Let X(t) denote the M × 1 input vector at discrete time t, Y(t+1) denote the corresponding N × 1 vector at time t+1 in the processing layer, and Z(t + n) denote the corresponding output value for nSA

(n2, nN) forecasting.

The X(t) and Y(t) are concatenated to form the ( M + N ) × 1 vector U(t), whose i^th element is denoted by μi(t). Let A denote the set of indices i for which xi(t) is an external input, and B denote the set of indices i for which yi(t) is the output of the processing layer. Thus, vector μi(t) can be represented as Eq. (1).

1

1

N 1 M

1

N

Output Layer

Processing Layer

Concatenated Input Layer Output

Z(t+n)

…

…

…

VNx1

WNx(M+N)

Y(t+1)

Input vector

X(t) Y(t)

unit time delay

Fig. 2.2 Architecture of the multi-step-ahead RNN with the R-RTRL online learning algorithm.

𝜇_𝑖(𝑡) = {𝑥_𝑖(𝑡) 𝑖𝑓 𝑖 ∈ 𝐴

𝑦_𝑖(𝑡) 𝑖𝑓 𝑖 ∈ 𝐵 (1)

W and V denote the weight matrices in the processing layer and the output layer, respectively. 𝑊 ↔ 𝑤_𝑗𝑖 and 𝑉 ↔ 𝜈_𝑗 are of matrix forms. The output of neuron 𝑗 in the

processing layer that presents the transformation of information from the concatenated layer through nonlinear system 𝑓 is given by:

𝑦_𝑗(𝑡 + 1) = 𝑓 (𝑛𝑒𝑡_𝑗(𝑡 + 1)) = 𝑓 ( ∑ 𝑤_𝑗𝑖(𝑡)𝜇_𝑖(𝑡)

𝑖∈𝐴∪𝐵

) (2)

The net output of the output layer at time t + n through nonlinear system 𝑓 is

computed by:

𝑧(𝑡 + 𝑛) = 𝑓(𝑛𝑒𝑡(𝑡 + 𝑛)) = 𝑓 (∑ 𝜈_𝑗(𝑡 + 1)𝑦_𝑗(𝑡 + 1)

𝑗

) (3)

Let d(t + n) denote the target value at time t+n. The time-varying error e and instantaneous error E is defined by:

𝐸(𝑡 + 𝑛) =1

2𝑒²(𝑡 + 𝑛) =1

2[𝑑(𝑡 + 𝑛) − 𝑧(𝑡 + 𝑛)]² (4)

Then the weight adjustments can be computed by minimizing the instantaneous error at time t + n.

∆𝜈_𝑗(𝑡 + 1) = −𝜂₁𝜕𝐸(𝑡 + 𝑛)

𝜕𝜈_𝑗(𝑡 + 1) (5)

∆𝑤_𝑗𝑖(𝑡) = −𝜂₂𝜕𝐸(𝑡 + 𝑛)

𝜕𝑤_𝑗𝑖(𝑡) (6)

where η₁ and η₂ are learning-rate parameters. The detailed recurrent learning algorithm for one-step-ahead weight adjustment can be found in Williams and Zipser (1989), and that for two-step-ahead weight adjustment can be found in Chang et al.

(2012). The entire antecedent information is considered crucial and could diminish time-lag effects. Consequently, the reinforced two-step weight adjustments (Chang et al.

2012) can be extended to n-step weight adjustments, and the information obtained from time t+1 to t+n-1 can contribute to weight adjustments. The adjusted weights are used to re-calculate the forecasted values at time from t+1 to t+n-1, and then the adjusted weights are further modified by minimizing the total error between original forecasted

values (𝑧(𝑡 + 𝑛 + 1) to 𝑧(𝑡 + 2𝑛 − 1)) and the re-forecasted values (𝑧̂(𝑡 + 𝑛 + 1) to 𝑧̂(𝑡 + 2𝑛 − 1)).

The re-forecasted value 𝑧̂(𝑡 + 𝑛 + 𝑝) is calculated by the following equations:

𝑦̂_𝑗(𝑡 + 𝑝 + 1) = 𝑓 (𝑛𝑒𝑡̂_𝑗(𝑡 + 𝑝 + 1))

= 𝑓 ( ∑ (𝑤_𝑗𝑖(𝑡) + ∆𝑤_𝑗𝑖(𝑡)) 𝜇_𝑖(𝑡 + 𝑝)

𝑖∈𝐴∪𝐵

)

(7)

𝑧̂(𝑡 + 𝑛 + 𝑝) = 𝑓(𝑛𝑒𝑡̂ (𝑡 + 𝑛 + 𝑝))

= 𝑓 (∑ (𝜈_𝑗(𝑡 + 1) + ∆𝜈_𝑗(𝑡 + 1)) 𝑦̂_𝑗(𝑡 + 𝑝 + 1)

𝑗

) (8)

where p denotes the time step (p = 1, 2, …, n-1). Therefore, the total reinforced error is defined by:

𝐸̂ =1

2∑ 𝑒̂²(𝑡 + 𝑛 + 𝑝)

𝑛−1

𝑝=1

=1

2[𝑧̂(𝑡 + 𝑛 + 𝑝) − 𝑧(𝑡 + 𝑛 + 𝑝)]² (9)

The reinforced weight adjustments can be expressed as:

∆𝜈̂_𝑗(𝑡 + 1) = −𝜂₃ 𝜕𝐸̂

𝜕𝜈_𝑗(𝑡 + 1) (10)

∆𝑤̂_𝑗𝑖(𝑡) = −𝜂₄ 𝜕𝐸̂

𝜕𝑤_𝑗𝑖(𝑡) (11)

where η₃ and η₄ are learning-rate parameters.

The weight adjustments of the R-RTRL algorithm for n-step-ahead RNNs are then shown as follows:

𝑤_𝑗𝑖(𝑡 + 1) = 𝑤_𝑗𝑖(𝑡) + ∆𝑤_𝑗𝑖(𝑡) + ∆𝑤̂_𝑗𝑖(𝑡) (12)

ν_j(t+2)=ν_j(t+1)+∆ν_j(t+1)+∆ν̂_j(t+1) (13)

In sum, the reinforced process is implemented for nSA forecasting so that the adjusted weights are further modified through the comparison between the original forecasted values and the re-forecasted values.