Threshold Definition and Drowsiness Classification

In drowsiness classification, we use the true-false table to define sensitivity and specificity. Sensitivity and specificity are statistical measures of the performance of a binary classification test. The sensitivity measures the proportion of actual positives which are correctly identified as such (e.g. the percentage of drowsy people who are identified as having the condition); and the specificity measures the proportion of negatives which are correctly identified (e.g. the percentage of alert people who are identified as not having the condition). The relationship between sensitivity and specificity is shows in Fig. 5-6 and the description of binary classification test was in

Fig. 5-6: The relationship between sensitivity and specificity

Table 5-2: The description of binary classification test Type Description

True positive Drowsy people correctly diagnosed as drowsy False positive Alert people wrongly identified as drowsy True negative Alert people correctly identified as alert

False negative Drowsy Sick people wrongly identified as alert

To define the drowsy state in driving performance and MD*, we need to collect the true positive, false positive, and false negative parameters, hence to analyze the sensitivity and positive predictive value.

A. Positive Predictive Value:

number of True Positives

PPV =number of True Positives number of False Positives

+ (5-1)

The positive predictive value, or precision rate, or post-test probability of disease, is the proportion of patients with positive test results who are correctly diagnosed. It is the most important measure of a diagnostic method as it reflects the probability that a

positive test reflects the underlying condition being tested for. Its value does however depend on the prevalence of the disease, which may vary.

B. Sensitivity:

number of True Positives Sensitivity

number of True Positives number of False Negatives

= + (5-2)

A sensitivity of 100% means that the test recognizes all drowsy people as drowsy.

Thus in a high sensitivity test, a negative result is used to rule out the disease.

Sensitivity alone does not tell us how well the test predicts other classes (that is, about the negative cases). In the binary classification, as illustrated above, this is the corresponding specificity test, or equivalently, the sensitivity for the other classes.

However, sensitivity is not the same as the positive predictive value (ratio of true positives to combined true and false positives), which is as much a statement about the proportion of actual positives in the population being tested as it is about the test.

The calculation of sensitivity does not take into account indeterminate test results.

If a test cannot be repeated, the options are to exclude indeterminate samples from analysis (but the number of exclusions should be stated when quoting sensitivity), or, alternatively, indeterminate samples can be treated as false negatives (which gives the worst-case value for sensitivity and may therefore underestimate it).

After explaining the definitions of sensitivity and positive predictive value, the next step is to define the threshold of driving performance and MD*(MDT, MDA, and MDC). The threshold of driving performance can follow above conclusion of sorting analysis which separated into 4 parts: alertness (0.2 ~ 1s), slight drowsiness (1

deviation time is smaller than 1 second to be alert, and others are drowsiness. On the other hand, we need to define the threshold of MD*. Because the results of MD* had been normalized, so we are beneficial to collect all 15 subjects’ MD* data and analyze them. In Fig. 5-7 and Fig. 5-8, we set the threshold of MD* from 1 ~ 13 respectively and analyzed the sensitivity and positive predictive value in different threshold. In linear combination, we also tried to separate into 9 conditions: a = 0.1, 0.2 … 0.8, 0.9. Following the different conditions to find the sensitivity and positive predict value in different threshold of MD*.

0 5 10 15

Fig. 5-7: Positive predictive value vs. threshold of MD* (MDT, MTA, and MDC)

0 5 10 15

Fig. 5-8: Sensitivity vs. threshold of MD* (MDT, MTA, and MDC)

When finished calculating positive predictive value and sensitivity in different conditions of linear combination, then we needed to choose the suitable threshold of MD*. According to equation 5-3, the F-measure can be used as a single measure of performance of the test. In information retrieval positive predictive value is called

precision, and sensitivity is called recall. The F-measure is the harmonic mean of The results of passing through F-measure were shown as Fig. 5-9. The percent of F-measure mean the ratio of drowsy accuracy actually. Both parameters are associated with drowsiness. In different linear combinational conditions, we could find out the highest result of F-measure in condition a = 0.9. According to this conclusion, this condition composed of the best linear combination of the MDC. Hence, the maximum value of F-measure, 77.59%, happened in the most suitable threshold of MDC, 7.5. So that the corresponding sensitivity was 88.28% and positive predictive value was 69.21%. Those results classified in Table 5-3.

The reason of which F-measure was not high enough was described into 3 critical points:

1. The trials of driving trajectories and corresponding MD* which we picked out didn’t use moving average to smooth, because of those sectional EEG information were too short. So that the MD* were not good enough in performance sorting analysis.

2. We found out the relation between driving performance and MD*, hence driving performance and MD* were a sufficient condition but not a necessary condition. When MD* value was high, the corresponding driving performance wasn’t high too. There were other variables appending to user’s EEG waves.

3. When subjects became drowsy, the MD* will increase, but will not happen

step. So we used trials of driving trajectories to analyze drowsiness was not sufficient to know the exact information of the EEG.

0 5 10 15

Fig. 5-9: F-measure vs. threshold of MD* (MDT, MTA, and MDC)

Table 5-3: The results of binary classification test

Types Max F-measure (%) Corresponding threshold PPV (%) Sensitivity (%)

MDT 73.92 6.5 60.15 95.86

MDA 77.34 7.5 69.68 86.90

MDC (a = 0.1) 73.46 6.5 59.29 96.55

MDC (a = 0.2) 73.76 6.5 58.91 98.62

MDC (a = 0.3) 74.02 7 63.73 88.28

MDC (a = 0.4) 75.12 7 63.94 91.03

MDC (a = 0.5) 76.00 7 63.88 93.79

MDC (a = 0.6) 76.28 7 64.28 93.79

MDC (a = 0.7) 76.65 7.5 70.87 83.45

MDC (a = 0.8) 77.40 7.5 70.69 85.52

MDC (a = 0.9) 77.59 7.5 69.21 88.28