4. EXPERIMENT RESULTS
4.1 Results and discussion
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
4. EXPERIMENT RESULTS
4.1 Results and discussion
During the experiment, we will automatically increase or decrease the number of hidden nodes in the hidden layer of the network. Therefore, the number of hidden nodes may be different for different numbers of samples. Stage 1 is Step 1 in our proposed mechanism, which can also be represented by 1st RL with envelope width = 2. Stage 2 is Step 2, and can be represented by 2nd RL with envelope width = 2.
In addition, for the testing progress of the experiment, we divide credit score into three rating groups: Excellent (100~70) is a group of high credit score which we suggest you can lend them money, Moderate (69~31) is a group we should consider about lending them money, Poor (30~0) is a group we should not lend them.
Figure 5: The distribution of credit score rating groups
Due to the high complexity of the algorithm, the training time is often a big issue in the past. After using the GPU and TensorFlow in this study, the training time of ANN can be completed within a few days in both stage 1 and stage 2. Table 10 and Table 11 shows the total number of adopted hidden nodes and training time spent in proposed
40%
45%
15%
Credit Score Rating Groups
Excellent group(78) Moderate group(87) Poor group(30)
‧
ANN mechanism of each stage.
Table 10: The training data and number of hidden nodes in different experiment.
Experiment 1 Experiment 2 Experiment 3 Average
StageTable 11: The training time spent in different experiment
Experiment Stage
‧
In Figure 6~17, the gray dots represent the envelope module wrapped the response instances seen as inlier instances in the envelope, the blue dots represent the actual credit score and the orange dots represent the predicted credit score we got from ANN.
Figure 6 and Figure 7 shows the envelope module of the 1st RL (with envelope width = 2) to identify potential outliers. After the Stage 1, No. 40, 42, 48, 67, 84, 108 and 125 instances in the training data is identified as outlier candidates.
In the proposed ANN mechanism Stage2, the 2nd RL (with envelope width = 2), we remove the outlier candidates. Figure 8 and Figure 9 shows the actual return of credit score in = /10 (4.5). After Stage2, all inlier instances are wrapped in the envelope of the 2nd RL.
Figure 6: The envelope module of the 1st RL to identify outliers in 1st experiment
Figure 7: The envelope module of the 1st RL to identify outliers in 1st experiment outlier candidates : No. 40, 42, 48, 67, 84, 108, 125
Predicted Credit Score Actual Credit Score Envelope outlier candidates : No. 40, 42, 48, 67, 84, 108, 125
Predicted Credit Score Actual Credit Score Envelope
‧
Figure 10 and Figure 11 shows the envelope module of the 1st RL (with envelope width = 2) to identify potential outliers. After the Stage 1, No.72, 88, 89, 97, 102, 125 and 132 instances in the training data is identified as outlier candidates.
In the proposed ANN mechanism Stage2, the 2nd RL (with envelope width = 2), we remove the outlier candidates. Figure 12 and Figure 13 shows the actual return of credit score in = /10 (4.46). After Stage2, all inlier instances are wrapped in the envelope of the 2nd RL.
Predicted Credit Score Actual Credit Score Envelope remove outlier candidates : No. 40, 42, 48, 67, 84, 108, 125
0
Predicted Credit Score Actual Credit Score Envelope remove outlier candidates : No. 40, 42, 48, 67, 84, 108, 125
‧
Figure 11: The envelope module of the 1st RL to identify outliers in 2nd experiment
Figure 12: Desired and predicted credit score in 2nd RL in 2nd experiment -150
Predicted Credit Score Actual Credit Score Envelope outlier candidates : No.72, 88, 89, 97, 102, 125, 132
0
Predicted Credit Score Actual Credit Score Envelope remove outlier candidates : No.72, 88, 89, 97, 102, 125, 132 -20
Predicted Credit Score Actual Credit Score Envelope outlier candidates : No.72, 88, 89, 97, 102, 125, 132
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 13: Desired and predicted credit score in 2nd RL in 2nd experiment
Figure 14 and Figure 15 shows the envelope module of the 1st RL (with envelope width = 2) to identify potential outliers. After the Stage 1, No. 23, 61, 70, 88, 89, 105 and 127 instances in the training data is identified as outlier candidates.
In the proposed ANN mechanism Stage2, the 2nd RL (with envelope width = 2), we remove the outlier candidates. Figure 16 and Figure 17 shows the actual return of credit score in = /10 (4.34). After Stage2, all inlier instances are wrapped in the envelope of the 2nd RL.
0 10 20 30 40 50 60 70 80 90 100
71 81 91 101 111 121 131
Stage2 , envelope width = 2 (4.46)
Predicted Credit Score Actual Credit Score Envelope remove outlier candidates : No.72, 88, 89, 97, 102, 125, 132
‧
Figure 14: The envelope module of the 1st RL to identify outliers in 3rd experiment
Figure 15: The envelope module of the 1st RL to identify outliers in 3rd experiment
Predicted Credit Score Actual Credit Score Envelope outlier candidates : No. 23, 61, 70, 88, 89, 105, 127
-150
Predicted Credit Score Actual Credit Score Envelope outlier candidates : No. 23, 61, 70, 88, 89, 105, 127
‧
instances separately based on the ANN after training. In the Table 12, we calculate mean and standard deviation of deviation between desired and predicted credit score in different experiment. After we remove the outlier candidates, there are significant declines on both mean of deviation and SD of deviation. For the 2nd RL with smaller envelope width, the mean of deviation and SD of deviation are also less than 1st RL.Table 12: Deviation between desired and predicted credit score in different experiment 0
Predicted Credit Score Actual Credit Score Envelope remove outlier candidates : No. 23, 61, 70, 88, 89, 105, 127 0
Predicted Credit Score Actual Credit Score Envelope remove outlier candidates : No. 23, 61, 70, 88, 89, 105, 127
‧
training instances, our ANN can make sure all deviation between desired and predicted credit score is within one standard deviation, expect for the outliers instances. In testing instances, the average percentage of deviation between desired and predicted credit score which is within one standard deviation is about 63%. And only about 10%instances will beyond two standard deviations.
Table 13: Accuracy of credit score prediction in different experiment
deviation
-ge Number Percenta
-ge Number Percenta -ge
Deviation 1.8501 28.6476 0.5347 20.7167
2
ndMean of
Deviation 0.9882 8.3558 0.2842 25.1536
SD of
Deviation 1.6601 28.4262 0.4829 28.9855
rage Ave
Mean of
Deviation 1.0918 7.8589 0.2549 24.7889
SD of
Deviation 1.7067 29.7587 0.5398 23.7098
‧
Table 14 shows the accuracy of credit score prediction in different experiment. A represents a group with a credit score of Excellent we have mentioned in the beginning of Chapter 4. B represents a group with a credit score of Moderate and C represents a group with a credit score of Poor. After the Step2, except for some prediction errors at the edge of the group, it gave each P2P loan information into the right rating group. For the testing instance, it can also predict the rating group within a certain accuracy rate.
Table 14: Accuracy of credit score prediction in different experiment
Group
All (140) training instances
Inlier (133) training instances after 2
ndRL
‧
Table 15 presents the predict accuracy in training instances and testing instances.
After Stage2, the ANN model can predict the inlier training instances well with an accuracy rate of 95% or more. The results shows that it is better to predict under the assumption that there is a possibility of outliers in the process of the ANN training. For the training instances, it can predict both Excellent and Moderate group with an accuracy rate of 65% or more. However, it can only predict the low credit score rating group with an accuracy rate of 54%.
Table 15: Accuracy of credit score prediction in different experiment