國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
3.6 Model Accuracy and Performance
To evaluate the accuracy of our proposed model, we adopt two methods to indicate our RNN model performance. One is to count whether our prediction is equal to ground true throughout all layer. The other is to calculate the traversal distance, ladder distance, between the predicted cluster and correct cluster. Here the ladder distance is denoted as total steps moving from one cluster to the other. For instance, the distance between [2, 3, 1] and [2, 3, 5] is two steps since it takes one step from [2, 3, 1] toward its parent node [2, 3] and one step from [2, 3] to its child [2, 3, 5]. The distance between [2, 3, 1] and [2, 1, 6] is 4, shown in fig. 7. [2, 3, 1] has to move two steps to parent node [2] and then move two steps back to its child [2, 1, 6]. To estimate the delay correctness, we use d, which is denoted as the absolute mean distance between ground true and predicting delay, to represent our model accuracy on predicting delay.
In the case study, we show that using wMSE and lMSE as our loss function can help the RNN to apprehend the layer di↵erences and significantly improve the prediction accuracy.
‧
國立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 7: Cluster distance
‧
In this section, we conduct our case study with a real-world electronics distributor, Com-pany W, and report our outcome and findings while applying the presented cluster tra-jectory and prediction approach to track process variability. Based in Taipei, Taiwan, Company W is one of the world’s largest electronics component distributors, with more than 30 branches worldwide. Company W acts as a franchise partner to more than 60 international electronics component suppliers, including Intel, Texas Instruments, Philips, Hynix, Vishay, and Omni Vision. This company plays a bu↵er role for the entire sup-ply chain by coordinating order quantities and production schedules with its downstream companies (OEM, ODM) and upstream manufacturing partners. For instance, as small-medium manufacturers generally order small quantities, it is hard for them to negotiate a good price with vendors. Company W, therefore, assists by aggregating orders from small companies to obtain a better quantity price. Company W can also help small compa-nies to control their inventory pooling. Currently, product managers in the company W depend solely on their experience and intuition to make ordering decisions since the com-pany lacks clear rules or methods for educating its product managers, and the comcom-pany faces tremendous challenges to recognize patterns of upstream and downstream demand fluctuations.
4.1 System Settings
All experiments were carried out on the data collected from real transactions of the company W, covering two-years historical data with more than eight thousand transaction records of three thousand items. Each item has samples up to 92 weeks. We conduct three experiment settings in this case study as shown in Tab. 6.
To evaluate the model accuracy, we slice the cluster sequences into training and testing sequences. The training sequences include two portions that exist one step di↵erence from each other. The first portion is item sequence [c0, d0] to [cn 2, dn 2] and the second portion is [c1, d1] to [cn 1, dn 1]. The first portion sequences are used to train the RNN model,
‧
We train one RNN model for all the items. We consider all the 3000+ items can be predicted with one RNN model in this setting.
RN Ncluster
We cluster items based on their trajectory features and train one RNN model for each cluster. In this setting, we employ GHSOM to cluse 3000+ items into 104 clusters, and train 104 RNN models with item trajectories of each cluster.
RN Nitem
We train one RNN model for each item. That is to say, we generate 3000+ RNN model for each item trajectory in this setting.
and the second portion sequences are used to generate the prediction on ˆxn and check whether it is accurate against [cn, dn] in the real data. The measurement for our model accuracy is to calculate the ratio that the prediction is exactly the same as the ground truth by layers and the delay. We also evaluate our model performance by calculating the traversal distances and the mean no-transaction weeks di↵erence t which point out the di↵erence when predicting no-transaction weeks before next transaction. We implement BasicRNN in tensorflow as our model, setting our training stop condition to 15000 iterations or accepted loss lower than 0.0001, employing AdamOptimizer [33]
as the optimization approach. We compare using the proposed weighted mean square error (wMSE) and ladder mean square error (lMSE) as loss function against MSE. The experiment results shown in Tab. 7. We also test if our model overfit the training data and Tab. 8 shows the training result.
In all the settings, we are able to achieve better accuracy ratios using wMSE and lMSE as the loss function compared to using MSE in training the RNN models to predict the trajectory movement. However, there is no significantly di↵erence between adopting wMSE and lMSE in predicting the ground true. We then calculate the mean ladder distance between these models. Tab. 9 shows the mean ladder distance when adopting di↵erent loss function. As we can see, lMSE is better to predict the similar cluster with shorter ladder distance.
‧
Table 7: Accuracy comparison on wMSE, lMSE and MSE in all settings
Sequence Loss function Layer accuracy l1 l1, l2 l1, l2, l3 d
RN Nall
M SE 0.431 0.233 0.108 19.86
wM SE 0.752 0.49 0.248 13.82
lM SE 0.762 0.399 0.151 10.11
RN Ncluster
M SE 0.481 0.262 0.136 13.85
wM SE 0.718 0.532 0.37 4.88
lM SE 0.817 0.688 0.469 2.34
RN Nitem
M SE 0.554 0.498 0.49 14.23
wM SE 0.948 0.773 0.553 5.52
lM SE 0.972 0.78 0.57 2.84
Table 8: Training data accuracy on wMSE, lMSE and MSE in all settings
Sequence Loss function Layer accuracy
‧
Table 9: Traversal distance comparison on MSE, wMSE and lMSE in all settings Sequence Loss function Mean ladder distance
RN Nall