Figure 3.5: Comparison between hard-/soft-input geometric decoders in the ML-ECC with the BCH code using BR learners
3.4 Experimental Results of Soft-input Geometric Decoder
Now, we experimentally compare the soft-input geometric decoder with the hard-input one. The settings of these experiments are the same as in Section 2.4 and 3.2.
3.4.1 Soft-input Decoding for Binary Relevance Learners
All of the base learners we used, Random Forests from WEKA, Gaussian SVM from LIBSVM, and logistic regression from LIBLINEAR, support predicting class probability distribution. To take the class probability distribution as soft inputs, we first try on Binary Relevance approaches. In the Binary Relevance approach, each base learner learns a single bit, so its probability output is indeed the soft signal we want.
The results on the BCH code using the Gaussian SVM base learner is shown in Fig-ure 3.5. Since the value of ∆0/1 and ∆HL varies greatly from dataset to dataset but little from decoder to decoder, in the figures, we present the ∆0/1and ∆HL changes based on the results of the algebraic decoder. We denote the result of hard-input geometric de-coder as hard, and that of soft-input geometric dede-coder as soft. The value lower than 0 means that the geometric decoder performs better than the algebraic one. We can see from Figure 3.5(a) that the soft-input geometric decoder is similar to or slightly better than the hard-input one in terms of ∆0/1, and both geometric decoders are significantly
(a) 0/1 loss change (b) Hamming loss change
Figure 3.6: Comparison between hard-/soft-input geometric decoders in the ML-ECC with the HAMR code using BR learners
better than the algebraic one. From Figure 3.5(b), we can see that soft-input geometric decoder is much better than the hard-input one in terms of ∆HL, and it is even better than the algebraic one on the scene dataset. The result suggests that soft inputs are helpful on ∆HLand also ∆0/1for geometric decoder and for BCH code.
In contrast to the BCH code, the results on the HAMR code is a little different, as shown in Figure 3.6. First, we look at the 0/1 loss shown in Figure 3.6(a). The result is dataset dependent. Soft-input geometric decoder performs better than both hard-input one and algebraic one on three datasets, and worse than those two decoders on other two datasets. In terms of Hamming loss in Figure 3.6(b), soft-input geometric decoder is significantly better than both hard-input one and algebraic one on four datasets, and performs similar on the other datasets. The result indicates that soft inputs are helpful on
∆HL, but not on ∆0/1, when applying to HAMR code.
Similar results show up when using other base learners, as shown in Table 3.3 and 3.4.
The bold-face entries are the best entries on each dataset given the ECC and base learner.
We also report the micro and macro F1 scores, and the pairwise label ranking loss in Tables A.15, A.16, and A.17, respectively. In the tables, we can see that the soft-input geometric decoder is usually better than the hard-input ones in these measures. This again shows that the soft inputs, i.e. confidence information from base learners, are useful on
decoding.
Table 3.3: 0/1 loss of ML-ECC with hard-/soft-input geometric decoders and BR
base learner ECC decoder scene (M =127) emotions (M =127) yeast (M =255) tmc2007 (M =511) Random Forest HAMR alg-hard .3212 ± .0021 .6574 ± .0050 .7910 ± .0020 .7578 ± .0025 Random Forest HAMR geo-hard .3111 ± .0021 .6480 ± .0051 .7833 ± .0022 .7566 ± .0026 Random Forest HAMR geo-soft .3294 ± .0021 .6584 ± .0053 .7976 ± .0020 .7588 ± .0025 Gaussian SVM HAMR alg-hard .2876 ± .0018 .8073 ± .0043 .7681 ± .0022 .7215 ± .0025 Gaussian SVM HAMR geo-hard .2829 ± .0017 .7927 ± .0051 .7650 ± .0023 .7205 ± .0024 Gaussian SVM HAMR geo-soft .2782 ± .0016 .8155 ± .0048 .7705 ± .0021 .7176 ± .0022 Logistic Regression HAMR alg-hard .4050 ± .0020 .7175 ± .0054 .8282 ± .0015 .7405 ± .0024 Logistic Regression HAMR geo-hard .3969 ± .0022 .7097 ± .0059 .8270 ± .0016 .7399 ± .0024 Logistic Regression HAMR geo-soft .3875 ± .0024 .7177 ± .0069 .8284 ± .0016 .7379 ± .0024 Random Forest BCH alg-hard .2562 ± .0020 .6404 ± .0060 .7792 ± .0019 .7149 ± .0022 Random Forest BCH geo-hard .2462 ± .0019 .6304 ± .0049 .7217 ± .0022 .6917 ± .0020 Random Forest BCH geo-soft .2526 ± .0020 .6264 ± .0049 .7287 ± .0022 .6949 ± .0024 Gaussian SVM BCH alg-hard .2552 ± .0018 .7809 ± .0050 .7515 ± .0015 .7053 ± .0024 Gaussian SVM BCH geo-hard .2503 ± .0018 .7371 ± .0051 .7203 ± .0018 .6975 ± .0025 Gaussian SVM BCH geo-soft .2505 ± .0019 .7350 ± .0050 .7212 ± .0021 .6966 ± .0030 Logistic Regression BCH alg-hard .3291 ± .0020 .6982 ± .0048 .8094 ± .0020 .7205 ± .0022 Logistic Regression BCH geo-hard .3164 ± .0017 .6886 ± .0046 .7698 ± .0019 .7102 ± .0021 Logistic Regression BCH geo-soft .3142 ± .0016 .6787 ± .0046 .7713 ± .0023 .7112 ± .0025 base learner ECC decoder genbase (M =511) medical (M =1023) enron (M =1023)
Random Forest HAMR alg-hard .0288 ± .0019 .6387 ± .0025 .8851 ± .0036 Random Forest HAMR geo-hard .0280 ± .0019 .6377 ± .0027 .8845 ± .0036 Random Forest HAMR geo-soft .0271 ± .0017 .6373 ± .0027 .8848 ± .0036 Gaussian SVM HAMR alg-hard .0243 ± .0022 .3675 ± .0036 .8718 ± .0042 Gaussian SVM HAMR geo-hard .0231 ± .0021 .3671 ± .0036 .8720 ± .0042 Gaussian SVM HAMR geo-soft .0233 ± .0022 .3627 ± .0035 .8716 ± .0043 Logistic Regression HAMR alg-hard .3509 ± .0089 .5499 ± .0247 .8740 ± .0036 Logistic Regression HAMR geo-hard .3541 ± .0099 .5514 ± .0249 .8744 ± .0036 Logistic Regression HAMR geo-soft .2777 ± .0107 .5301 ± .0229 .8723 ± .0036 Random Forest BCH alg-hard .0250 ± .0019 .4567 ± .0034 .8737 ± .0038 Random Forest BCH geo-hard .0255 ± .0018 .4130 ± .0037 .8429 ± .0038 Random Forest BCH geo-soft .0250 ± .0018 .4157 ± .0037 .8371 ± .0043 Gaussian SVM BCH alg-hard .0255 ± .0019 .3499 ± .0030 .8561 ± .0043 Gaussian SVM BCH geo-hard .0255 ± .0019 .3431 ± .0034 .8428 ± .0045 Gaussian SVM BCH geo-soft .0253 ± .0020 .3376 ± .0035 .8376 ± .0045 Logistic Regression BCH alg-hard .0295 ± .0018 .4022 ± .0076 .8579 ± .0038 Logistic Regression BCH geo-hard .0422 ± .0026 .3704 ± .0048 .8362 ± .0040 Logistic Regression BCH geo-soft .0395 ± .0026 .3670 ± .0046 .8475 ± .0040
3.4.2 Soft-input Decoding for k-powerset Learners
We have shown that soft inputs are beneficial for geometric decoder when using Binary Relevance base learners. Now, we would like to see if we can apply this to the k-powerset learners. As mentioned in Section 3.3, it is non-trivial to estimate the confidence per bit from the confidence information on k-powerests, i.e., the probability distribution over 2K
Table 3.4: Hamming loss of ML-ECC with hard-/soft-input geometric decoders and BR
base learner ECC decoder scene (M =127) emotions (M =127) yeast (M =255) tmc2007 (M =511) Random Forest HAMR alg-hard .0726 ± .0005 .1781 ± .0017 .1878 ± .0007 .0652 ± .0002 Random Forest HAMR geo-hard .0718 ± .0006 .1768 ± .0017 .1872 ± .0007 .0651 ± .0003 Random Forest HAMR geo-soft .0726 ± .0005 .1763 ± .0018 .1873 ± .0007 .0651 ± .0002 Gaussian SVM HAMR alg-hard .0717 ± .0004 .2472 ± .0023 .1861 ± .0007 .0616 ± .0003 Gaussian SVM HAMR geo-hard .0716 ± .0005 .2508 ± .0023 .1858 ± .0007 .0615 ± .0003 Gaussian SVM HAMR geo-soft .0701 ± .0004 .2441 ± .0020 .1855 ± .0007 .0611 ± .0003 Logistic Regression HAMR alg-hard .0959 ± .0006 .2047 ± .0022 .2003 ± .0007 .0635 ± .0003 Logistic Regression HAMR geo-hard .0956 ± .0006 .2065 ± .0023 .2002 ± .0007 .0634 ± .0003 Logistic Regression HAMR geo-soft .0920 ± .0006 .2032 ± .0024 .1990 ± .0007 .0631 ± .0003 Random Forest BCH alg-hard .0717 ± .0006 .1826 ± .0018 .1898 ± .0008 .0638 ± .0003 Random Forest BCH geo-hard .0728 ± .0006 .1895 ± .0017 .1968 ± .0010 .0643 ± .0003 Random Forest BCH geo-soft .0703 ± .0006 .1822 ± .0016 .1910 ± .0008 .0622 ± .0003 Gaussian SVM BCH alg-hard .0739 ± .0006 .2569 ± .0030 .1880 ± .0007 .0619 ± .0003 Gaussian SVM BCH geo-hard .0735 ± .0005 .2761 ± .0031 .1952 ± .0007 .0649 ± .0003 Gaussian SVM BCH geo-soft .0721 ± .0006 .2614 ± .0028 .1911 ± .0007 .0624 ± .0003 Logistic Regression BCH alg-hard .0955 ± .0006 .2172 ± .0023 .2037 ± .0008 .0638 ± .0003 Logistic Regression BCH geo-hard .0957 ± .0006 .2312 ± .0027 .2116 ± .0007 .0673 ± .0003 Logistic Regression BCH geo-soft .0913 ± .0006 .2170 ± .0026 .2067 ± .0008 .0640 ± .0003 base learner ECC decoder genbase (M =511) medical (M =1023) enron (M =1023)
Random Forest HAMR alg-hard .0012 ± .0001 .0179 ± .0001 .0474 ± .0003 Random Forest HAMR geo-hard .0012 ± .0001 .0179 ± .0001 .0474 ± .0003 Random Forest HAMR geo-soft .0011 ± .0001 .0179 ± .0001 .0474 ± .0003 Gaussian SVM HAMR alg-hard .0010 ± .0001 .0112 ± .0001 .0451 ± .0004 Gaussian SVM HAMR geo-hard .0009 ± .0001 .0112 ± .0001 .0450 ± .0004 Gaussian SVM HAMR geo-soft .0010 ± .0001 .0111 ± .0001 .0450 ± .0004 Logistic Regression HAMR alg-hard .0186 ± .0005 .0191 ± .0011 .0452 ± .0003 Logistic Regression HAMR geo-hard .0187 ± .0006 .0192 ± .0011 .0453 ± .0003 Logistic Regression HAMR geo-soft .0134 ± .0005 .0180 ± .0010 .0453 ± .0003 Random Forest BCH alg-hard .0010 ± .0001 .0152 ± .0001 .0494 ± .0004 Random Forest BCH geo-hard .0010 ± .0001 .0154 ± .0001 .0566 ± .0005 Random Forest BCH geo-soft .0010 ± .0001 .0150 ± .0002 .0535 ± .0004 Gaussian SVM BCH alg-hard .0010 ± .0001 .0117 ± .0001 .0487 ± .0005 Gaussian SVM BCH geo-hard .0010 ± .0001 .0126 ± .0001 .0577 ± .0006 Gaussian SVM BCH geo-soft .0011 ± .0001 .0121 ± .0001 .0534 ± .0005 Logistic Regression BCH alg-hard .0024 ± .0002 .0161 ± .0006 .0472 ± .0004 Logistic Regression BCH geo-hard .0054 ± .0004 .0154 ± .0004 .0558 ± .0004 Logistic Regression BCH geo-soft .0046 ± .0003 .0141 ± .0003 .0526 ± .0004
combinations of labels. Here, we experimentally examine the methods for such estimation described in Section 3.3.
The results on the BCH code are shown in Figure 3.7. Similar to the above exper-iments, we also plot the changes on ∆0/1 and ∆HL based on the result of the algebraic decoder. In the figures, maximum, margin, diff, s-margin, and s-diff stand for the soft-input geometric decoders using the corresponding confidence estimation methods described in Section 3.3. Note that maximum is exactly the same as supplying hard-input
(a) 0/1 loss change (b) Hamming loss change
Figure 3.7: Comparison between hard-/soft-input geometric decoders in the ML-ECC with the BCH code using 3-powerset learners
to the geometric decoder.
From Figure 3.7(a), we can see that most of the soft-input geometric decoders have similar performance to the hard-input geometric decoder on ∆0/1, except diff, whose result is worse than the hard-input one. The result implies that such bitwise soft inputs calculated from confidence score on k-powersets may not be useful to improve geometric decoding result in terms of ∆0/1.
Next, we look at the Hamming loss shown in Figure 3.7(b). On the contrary, soft-input geometric decoders using any confidence estimation method are better than hard-input geometric decoder on ∆HL. Among the confidence estimation methods, diff is the best and margin is the second best. These two soft-input geometric decoders are even better than the algebraic decoder in terms of ∆HLon some datasets.
Putting the results on ∆0/1and ∆HL together, we can see that for the BCH code, the soft-input geometric decoder using the “marginal probability” method is a good choice since it has better ∆HLand similar ∆0/1comparing to the hard-input geometric decoder.
However, things are different for the HAMR code, as shown in Figure 3.8. From Figure 3.8(a), the hard-input geometric decoder beats all soft-input ones in terms of ∆0/1. In the soft-input geometric decoders, s-diff has the best performance. This one is also the only soft-input geometric decoder, which is better than the algebraic decoder.
(a) 0/1 loss change (b) Hamming loss change
Figure 3.8: Comparison between hard-/soft-input geometric decoders in the ML-ECC with the HAMR code using 3-powerset learners
Table 3.5: Comparison of soft-input geometric decoders using different confidence esti-mation methods on k-powerset learners