CHAPTER 3: CLUSTER BASED DYNAMIC SUBSPACE METHOD
3.5 Experimental Results and Findings
3.5.3 Classification result
Tables 3.7 (a) to (i) show the mean accuracies of the IPS dataset in nine experiments, respectively, and the comparisons of mean accuracies between all algorithms using five different classifiers are displayed in Figures 3.13 (a) to (i).
Note that shadow part indicates the best accuracy of each experiment and the best accuracy of each applied classifier among all algorithms is written in bold type. The classification times of all combinations are listed in Tables 3.8 (a) to (i), and the classification results of the area of Figure 3.5 (a) are displayed in Figures 3.14 to 3.22. The following are some findings based on these results.
Cluster 1 Cluster 2 Cluster 3
Cluster 4 Cluster 5 Cluster 6
1. The best accuracies of each experiment (the shadow parts) all occurs in the proposed algorithms (CDSMkm or CDSMfcm) with SVM classifier. The highest accuracies are 92.6 % in experiment 9, and it occurs in CDSMkm with SVM classifier. It achieves 3.5 %, 2.1 %, and 1~1.5 % better than single, RSM, and WRSMs, respectively.
2. The best accuracies with different base classifiers, GC, kNN, Parzen, CART, and SVM, are 91.3 % (CDSMkm), 85.9 % (CDSMfcm), 85.5 % (CDSMfcm), 86.3 % (CDSMfcm), and 92.6 % (CDSMkm), respectively.
3. No matter what learning algorithms we applied, all MCS perform better than single classifiers; and furthermore, the best accuracies occur mostly when applying CDSMfcm. In terms of kNN classifier, all ensemble methods are not obviously superior to single classifier.
4. GC still does not work (noted as N/A) in Ni = 20 < N < n and Ni = 40 < n <
N under the algorithms of single and RSM due to the singularity problem. In the other hand, WRSM and CDSM perform well in these two conditions.
5. Comparing the results of Tables 3.8 (a), (d), and (g), as the training sample size increases, the accuracies also represent ascending tendencies in all combinations. The same results are also occurred in Tables 3.8 (b), (e), and (h), and Tables 3.8 (c), (f), and (i).
6. In most cases, the classification performances and classification times would be increasing when the ensemble size was increasing.
7. From Figure. 3.12, we find that CDSMkm and CDSMfcm obtain sound results than other algorithms especially when the number of classifiers is small.
Single (N/A) RSM (N/A) WRSM WRSM1 WRSM2 CDSMkm CDSMfcm
1 10 50 100
0.50 0.55 0.60 0.65 0.70 0.75 0.80
The Number of Classifiers (B)
Classification Accuracy Rate (100%)
Figure 3.12. The accuracy on the IPS dataset using GC in experiment 1.
8. From Figure. 3.12, we find that CDSMkm and CDSMfcm obtain sound results than other algorithms especially when the ensemble size is small.
9. Compared to the ground truth in Figure 3.5 (b), we can find that the classification results of three kinds of MCS are better than those of single classifier especially in Soybeans-min, Soybeans-notill, and Corn-notill which are the most difficult parts to accurately classify.
10. The best classification result occurs in Figure 3.22 (t) by using CDSMfcm with SVM classifier.
Table 3.7 (a). The mean accuracies of single classifier and six MCSs with five classifiers in experiment 1 (Ni = 20 < N < n, B = 10).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 66.7 65.6 39.0 74.5
RSM N/A 67.0 66.9 58.6 76.5
wRSM 68.8 66.6 69.0 59.8 78.4
wRSM1 64.0 66.9 65.8 58.4 75.9
wRSM2 66.3 67.1 66.0 59.6 75.3
CDSMkm 68.5 67.3 71.0 60.5 77.0
CDSMfcm
10
74.5 66.6 73.3 60.8 81.0
Accuracy: %
0 20 40 60 80 100
GC kNN Parzen CART SVM
Accuracy (%)
Single RSM wRSM wRSM1 wRSM2 CDSMkm CDSMfcm
Figure 3.13 (a). The comparison of mean accuracies between all algorithms using five classifiers in experiment 1 (Ni = 20 < N < n, B = 10).
Table 3.8 (a). The classification times of single classifier and six MCSs with five classifiers in experiment 1 (Ni = 20 < N < n, B = 10).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 0.41 0.42 0.18 0.14
RSM N/A 1.90 18.29 4.94 18.00
wRSM 2.36 4.83 20.71 8.79 25.77
wRSM1 10.69 21.98 106.29 67.73 26.25
wRSM2 2.55 5.63 20.03 11.37 25.80
CDSMkm 2.23 6.00 21.37 12.81 23.41
CDSMfcm
10
2.39 12.22 24.96 16.68 30.44 Time: sec.
(a) Single GC (b) RSM + GC (c) WRSM + GC (d) CDSMfcm + GC
(e) Single kNN (f) RSM + kNN (g) WRSM + kNN (h) CDSMfcm + kNN
(i) Single Parzen (j) RSM + Parzen (k) WRSM + Parzen (l) CDSMfcm + Parzen
(m) Single CART (n) RSM + CART (o) WRSM + CART (p) CDSMfcm +CART
(q) Single SVM (r) RSM + SVM (s) WRSM + SVM (t) CDSMfcm + SVM Figure 3.14. The classification results of Figure 3.5 (a) in experiment 1.
Table 3.7 (b). The mean accuracies of single classifier and six MCSs with five classifiers in experiment 2 (Ni = 20 < N < n, B = 50).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 66.7 65.6 39.0 74.5
RSM N/A 67.6 66.0 63.4 78.3
wRSM 73.9 66.9 72.4 63.8 77.9
wRSM1 74.8 67.1 70.9 64.8 75.9
wRSM2 73.6 67.3 70.4 66.5 76.3
CDSMkm 75.0 67.4 72.8 64.0 77.5
CDSMfcm
50
78.9 66.8 74.4 65.5 81.3 Accuracy: %
0 20 40 60 80 100
GC kNN Parzen CART SVM
Accuracy (%)
Single RSM wRSM wRSM1 wRSM2 CDSMkm CDSMfcm
Figure 3.13 (b). The comparison of mean accuracies between all algorithms in experiment 2 (Ni = 20 < N < n, B = 50).
Table 3.8 (b). The classification times of single classifier and six MCSs with five classifiers in experiment 2 (Ni = 20 < N < n, B = 50).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 0.41 0.42 0.18 0.14
RSM N/A 11.06 39.28 20.38 83.55
wRSM 11.38 54.89 79.79 64.95 134.23
wRSM1 19.06 79.18 127.59 136.27 134.88
wRSM2 11.02 59.05 74.53 68.55 145.89
CDSMkm 11.20 60.83 79.70 73.25 144.98
CDSMfcm
50
13.19 84.31 88.79 91.85 168.11 Time: sec.
(a) Single GC (b) RSM + GC (c) WRSM + GC (d) CDSMfcm + GC
(e) Single kNN (f) RSM + kNN (g) WRSM + kNN (h) CDSMfcm + kNN
(i) Single Parzen (j) RSM + Parzen (k) WRSM + Parzen (l) CDSMfcm +Parzen
(m) Single CART (n) RSM + CART (o) WRSM + CART (p) CDSMfcm +CART
(q) Single SVM (r) RSM + SVM (s) WRSM + SVM (t) CDSMfcm + SVM Figure 3.15. Theclassification results of Figure 3.5 (a) in experiment 2.
Table 3.7. (c) The mean accuracies of single classifier and six MCSs with five classifiers in experiment 3 (Ni = 20 < N < n, B = 100).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 66.7 65.6 39.0 74.5
RSM N/A 67.5 66.1 65.3 78.9
wRSM 74.8 67.1 73.7 64.4 77.5
wRSM1 74.8 67.3 70.9 66.0 76.0
wRSM2 73.9 67.0 71.6 66.6 76.0
CDSMkm 76.8 67.1 73.3 64.8 78.1
CDSMfcm
100
79.1 66.8 75.0 67.0 81.8
Accuracy: %
0 20 40 60 80 100
GC kNN Parzen CART SVM
Accuracy (%)
Single RSM wRSM wRSM1 wRSM2 CDSMkm CDSMfcm
Figure 3.13 (c). The comparison of mean accuracies between all algorithms in experiment 3 (Ni = 20 < N < n, B = 100).
Table 3.8 (c). The classification times of single classifier and six MCSs with five classifiers in experiment 3 (Ni = 20 < N < n, B = 100).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 0.41 0.42 0.18 0.14
RSM N/A 20.77 45.53 40.52 164.19
wRSM 27.17 177.45 188.98 191.53 349.66
wRSM1 37.33 199.08 253.48 259.97 399.90 wRSM2 28.92 183.17 193.25 194.34 400.83
CDSMkm 30.14 190.17 202.43 209.66 387.61
CDSMfcm
100
33.70 211.16 223.47 245.06 411.20 Time: sec.
(a) Single GC (b) RSM + GC (c) WRSM + GC (d) CDSMfcm + GC
(e) Single kNN (f) RSM + kNN (g) WRSM + kNN (h) CDSMfcm + kNN
(i) Single Parzen (j) RSM + (k) WRSM + (l) CDSMfcm + Parzen
(m) Single CART (n) RSM + CART (o) WRSM + CART (p) CDSMfcm +CART
(q) Single SVM (r) RSM + SVM (s) WRSM + SVM (t) CDSMfcm + SVM Figure 3.16. The classification results of Figure 3.5 (a) in experiment 3.
Table 3.7 (d). The mean accuracies of single classifier and six MCSs with five classifiers in experiment 4 (Ni = 40 < n < N, B = 10).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 71.9 73.0 46.8 77.0
RSM N/A 70.4 72.5 64.3 82.5
wRSM 78.8 71.4 78.1 64.6 83.9
wRSM1 77.3 72.4 80.5 65.6 83.6
wRSM2 75.4 69.1 75.8 67.0 83.6
CDSMkm 78.6 72.7 81.3 67.5 83.9
CDSMfcm
10
79.8 72.9 79.8 66.1 84.3 Accuracy: %
0 20 40 60 80 100
GC kNN Parzen CART SVM
Accuracy (%)
Single RSM wRSM wRSM1 wRSM2 CDSMkm CDSMfcm
Figure 3.13 (d). The comparison of mean accuracies between all algorithms in experiment 4 (Ni = 40 < n < N, B = 10).
Table 3.8 (d). The classification times of single classifier and six MCSs with five classifiers in experiment 4 (Ni = 40 < n < N, B = 10).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 0.19 1.11 0.31 0.35
RSM N/A 4.76 45.20 7.51 5.26
wRSM 9.31 7.37 48.60 9.64 7.98
wRSM1 20.25 53.84 199.03 133.57 17.45
wRSM2 9.87 8.06 48.01 10.67 8.04
CDSMkm 10.64 7.62 49.75 11.09 8.81
CDSMfcm
10
13.37 10.96 51.87 15.18 12.95 Time: sec.
(a) Single GC (b) RSM + GC (c) WRSM + GC (d) CDSMfcm + GC
(e) Single kNN (f) RSM + kNN (g) WRSM + kNN (h) CDSMfcm + kNN
(i) Single Parzen (j) RSM + (k) WRSM + (l) CDSMfcm + Parzen
(m) Single CART (n) RSM + CART (o) WRSM + CART (p) CDSMfcm +CART
(q) Single SVM (r) RSM + SVM (s) WRSM + SVM (t) CDSMfcm + SVM Figure 3.17. The classification results of Figure 3.5 (a) in experiment 4.
Table 3.7 (e). The mean accuracies of single classifier and six MCSs with five classifiers in experiment 5 (Ni = 40 < n < N, B = 50).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 71.9 73.0 46.8 77.0
RSM N/A 71.4 73.6 71.5 81.4
wRSM 82.3 70.6 79.5 71.1 84.0
wRSM1 82.6 73.0 79.9 70.1 84.3
wRSM2 80.1 71.3 75.4 71.5 84.0
CDSMkm 83.6 72.4 80.8 70.8 85.8
CDSMfcm
50
83.3 72.6 80.1 72.4 84.4 Accuracy: %
0 20 40 60 80 100
GC kNN Parzen CART SVM
Accuracy (%)
Single RSM wRSM wRSM1 wRSM2 CDSMkm CDSMfcm
Figure 3.13 (e). The comparison of mean accuracies between all algorithms in experiment 5 (Ni = 40 < n < N, B = 50).
Table 3.8 (e). The classification times of single classifier and six MCSs with five classifiers in experiment 5 (Ni = 40 < n < N, B = 50).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 0.19 1.11 0.31 0.35
RSM N/A 18.68 68.51 30.10 19.79
wRSM 54.40 59.70 110.26 65.68 58.84
wRSM1 69.54 107.15 263.60 190.40 70.87
wRSM2 57.79 63.01 109.60 72.26 63.82
CDSMkm 62.01 60.43 111.96 71.18 63.90
CDSMfcm
50
72.89 71.14 125.35 89.25 81.18 Time: sec.
(a) Single GC (b) RSM + GC (c) WRSM + GC (d) CDSMfcm + GC
(e) Single kNN (f) RSM + kNN (g) WRSM + kNN (h) CDSMfcm + kNN
(i) Single Parzen (j) RSM + Parzen (k) WRSM + Parzen (l) CDSMfcm + Parzen
(m) Single CART (n) RSM + CART (o) WRSM + CART (p) CDSMfcm +CART
(q) Single SVM (r) RSM + SVM (s) WRSM + SVM (t) CDSMfcm + SVM Figure 3.18. The classification results of Figure 3.5 (a) in experiment 5.
Table 3.7 (f). The mean accuracies of single classifier and six MCSs with five classifiers in experiment 6 (Ni = 40 < n < N, B = 100).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 71.9 73.0 46.8 77.0
RSM N/A 70.4 73.3 71.9 81.1
wRSM 82.5 70.5 79.1 72.0 84.3
wRSM1 84.3 73.4 80.0 70.9 84.5
wRSM2 82.1 71.0 75.9 72.1 83.8
CDSMkm 83.8 72.4 80.3 73.4 85.9
CDSMfcm
100
83.9 73.1 80.1 73.0 85.0
Accuracy: %
0 20 40 60 80 100
GC kNN Parzen CART SVM
Accuracy (%)
Single RSM wRSM wRSM1 wRSM2 CDSMkm CDSMfcm
Figure 3.13 (f). The comparison of mean accuracies between all algorithms in experiment 6 (Ni = 40 < n < N, B = 100).
Table 3.8 (f). The classification times of single classifier and six MCSs with five classifiers in experiment 6 (Ni = 40 < n < N, B = 100).
Algorithm B GC kNN Parzen CART SVM
Single 1 N/A 0.19 1.11 0.31 0.35
RSM N/A 35.50 97.87 58.48 37.56
wRSM 175.21 186.79 248.84 197.79 185.17 wRSM1 192.21 235.46 406.71 322.09 200.25 wRSM2 181.84 191.48 249.42 212.70 195.75
CDSMkm 188.56 189.02 251.75 209.67 194.57
CDSMfcm
100
201.62 211.34 279.93 246.39 226.42 Time: sec.
(a) Single GC (b) RSM + GC (c) WRSM + GC (d) CDSMfcm + GC
(e) Single kNN (f) RSM + kNN (g) WRSM + kNN (h) CDSMfcm + kNN
(i) Single Parzen (j) RSM + Parzen (k) WRSM + Parzen (l) CDSMfcm + Parzen
(m) Single CART (n) RSM + CART (o) WRSM + CART (p) CDSMfcm +CART
(q) Single SVM (r) RSM + SVM (s) WRSM + SVM (t) CDSMfcm + SVM Figure 3.19. The classification results of Figure 3.5 (a) in experiment 6.
Table 3.7 (g). The mean accuracies of single classifier and six MCSs with five classifiers in experiment 7 (n < Ni = 300 < N, B = 10).
Algorithm B GC kNN Parzen CART SVM
Single 1 67.8 83.9 83.6 67.5 89.1
RSM 86.5 83.0 83.8 80.3 89.8
wRSM 87.4 83.6 84.7 78.1 91.6
wRSM1 89.3 82.9 83.1 81.4 91.5
wRSM2 85.6 84.0 83.4 79.6 89.9
CDSMkm 89.4 85.3 84.3 83.3 91.6 CDSMfcm
10
88.1 82.9 85.0 80.9 91.8
Accuracy: %
0 20 40 60 80 100
GC kNN Parzen CART SVM
Accuracy (%)
Single RSM wRSM wRSM1 wRSM2 CDSMkm CDSMfcm
Figure 3.13 (g). The comparison of mean accuracies between all algorithms in experiment 7 (n < Ni = 300 < N, B = 10).
Table 3.8 (g). The classification times of single classifier and six MCSs with five classifiers in experiment 7 (n < Ni = 300 < N, B = 10).
Algorithm B GC kNN Parzen CART SVM
Single 1 2.87 5.38 9.73 5.24 7.63
RSM 9.79 112.14 97.78 45.17 244.85
wRSM 12.78 111.82 94.98 53.85 186.92
wRSM1 18.70 1678.20 7484.43 834.42 671.15
wRSM2 11.64 112.35 99.84 50.18 229.42
CDSMkm 9.10 114.57 99.65 52.40 147.10
CDSMfcm
10
20.218 118.64 97.75 58.60 131.28 Time: sec.
(a) Single GC (b) RSM + GC (c) WRSM + GC (d) CDSMfcm + GC
(e) Single kNN (f) RSM + kNN (g) WRSM + kNN (h) CDSMfcm + kNN
(i) Single Parzen (j) RSM + Parzen (k) WRSM + Parzen (l) CDSMfcm + Parzen
(m) Single CART (n) RSM + CART (o) WRSM + CART (p) CDSMfcm +CART
(q) Single SVM (r) RSM + SVM (s) WRSM + SVM (t) CDSMfcm + SVM Figure 3.20. The classification results of Figure 3.5 (a) in experiment 7.
Table 3.7 (h). The mean accuracies of single classifier and six MCSs with five classifiers in experiment 8 (n < Ni = 300 < N, B = 50).
Algorithm B GC kNN Parzen CART SVM
Single 1 67.8 83.9 83.6 67.5 89.1
RSM 88.6 82.5 83.6 84.8 90.1
wRSM 86.0 83.8 84.9 84.5 92.3
wRSM1 89.0 83.6 84.0 84.4 91.4
wRSM2 87.5 84.6 83.5 85.3 91.5
CDSMkm 90.9 85.6 85.5 86.4 92.6
CDSMfcm
50
89.5 84.9 85.3 86.1 92.4
Accuracy: %
0 20 40 60 80 100
GC kNN Parzen CART SVM
Accuracy (%)
Single RSM wRSM wRSM1 wRSM2 CDSMkm CDSMfcm
Figure 3.13 (h). The comparison of mean accuracies between all algorithms in experiment 8 (n < Ni = 300 < N, B = 50).
Table 3.8 (h). The classification times of single classifier and six MCSs with five classifiers in experiment 8 (n < Ni = 300 < N, B = 50).
Algorithm B GC kNN Parzen CART SVM
Single 1 2.87 5.38 9.73 5.24 7.63
RSM 38.51 442.39 375.70 152.26 1015.51
wRSM 57.51 457.25 385.60 225.92 821.62
wRSM1 65.12 2036.41 7795.35 999.01 1306.25 wRSM2 61.43 473.18 409.609 206.31 888.46
CDSMkm 58.59 478.56 398.73 214.81 686.01
CDSMfcm
50
104.31 493.07 408.25 232.32 521.39 Time: sec.
(a) Single GC (b) RSM + GC (c) WRSM + GC (d) CDSMfcm + GC
(e) Single kNN (f) RSM + kNN (g) WRSM + kNN (h) CDSMfcm + kNN
(i) Single Parzen (j) RSM + Parzen (k) WRSM + Parzen (l) CDSMfcm + Parzen
(m) Single CART (n) RSM + CART (o) WRSM + CART (p) CDSMfcm +CART
(q) Single SVM (r) RSM + SVM (s) WRSM + SVM (t) CDSMfcm + SVM Figure 3.21. The classification results of Figure 3.5 (a) in experiment 8.
Table 3.7 (i). The mean accuracies of single classifier and six MCSs with five classifiers in experiment 9 (n < Ni = 300 < N, B = 100).
Algorithm B GC kNN Parzen CART SVM
Single 1 67.8 83.9 83.6 67.5 89.1
RSM 87.9 82.4 84.5 85.4 90.4
wRSM 89.0 83.6 85.3 86.0 91.6
wRSM1 89.5 83.6 84.3 85.8 91.5
wRSM2 88.6 84.8 83.4 85.1 91.1
CDSMkm 91.3 85.5 85.0 86.0 92.6
CDSMfcm
100
90.5 85.9 85.5 86.3 92.5
Accuracy: %
0 20 40 60 80 100
GC kNN Parzen CART SVM
Accuracy (%)
Single RSM wRSM wRSM1 wRSM2 CDSMkm CDSMfcm
Figure 3.13 (i). The comparison of mean accuracies between all algorithms in experiment 9 (n < Ni = 300 < N, B = 100).
Table 3.8 (i). The classification times of single classifier and six MCSs with five classifiers in experiment 9 (n < Ni = 300 < N, B = 100).
Algorithm B GC kNN Parzen CART SVM
Single 1 2.87 5.38 9.73 5.24 7.63
RSM 75.95 869.17 725.34 289.21 2100.29
wRSM 164.65 945.98 812.06 504.34 1636.43 wRSM1 174.73 2540.78 8245.27 1276.88 2268.19 wRSM2 173.84 983.82 850.68 471.03 1946.21
CDSMkm 167.62 987.07 833.45 482.71 1349.68
CDSMfcm
100
260.79 1033.45 860.82 506.96 1124.24 Time: sec.
(a) Single GC (b) RSM + GC (c) WRSM + GC (d) CDSMfcm + GC
(e) Single kNN (f) RSM + kNN (g) WRSM + kNN (h) CDSMfcm + kNN
(i) Single Parzen (j) RSM + Parzen (k) WRSM + Parzen (l) CDSMfcm + Parzen
(m) Single CART (n) RSM + CART (o) WRSM + CART (p) CDSMfcm +CART
(q) Single SVM (r) RSM + SVM (s) WRSM + SVM (t) CDSMfcm + SVM Figure 3.22. The classification results of Figure 3.5 (a) in experiment 9.