Proposed P300-based BCI System
3.4 Performance evaluation
3.4.2 Bit rate
An information transfer rate, a bit rate, can be used in order to take into account both accuracy and speed of a BCI. The bit rate is a standard measure of any communication
system(which a BCI basically is). It tells the amount of information communicated per time unit. The bit rate R measures the achievable information rate per unit time, given the decision accuracy and duration. The number of achievable bits per decision is given by following equation [22]
C = log2N + P log2P + (1 − P )log2((1 − P )
(N − 1)) (3.3)
where N is the number of possible selections and P is the correct classification rate. The bitrate R in bits/minute is given by R = CM where C is the number of bits per decision (bits/trial), and M is the average number of decisions per minute. The bit rate as a function
Furthermore, if their efforts are to be productive and their results credible, BCI research programs must adhere to certain principles in experimental design, data evaluation, and documentation and dissemination of results.
4.9.1. Assessment of inter- and intra-user variations Users are likely to differ greatly in the prominence and stability of specific signal features, and in their capacities for initially demonstrating or acquiring and subsequently maintaining control over these features. Users with disabil-ities are likely to display even more variation. Thus, BCI methods should be tested in more than one or a few users, and the test populations should include users with relevant disabilities. Intra-individual variation is an equally impor-tant issue. Those few research programs that have acquired long-term data have found that marked variations in perfor-mance typically occur over minutes, hours, days, weeks, and months. Thus, data should be gathered from each user many times over substantial periods. Furthermore, appropriate and comprehensive statistical tests should be applied. Simply describing the single best result or the performance for a few sessions is not enough.
4.9.2. Online validation of offline analyses
Offline analyses of data stored during BCI operation are not by themselves sufficient for assessing and comparing alternative signal features, feature extraction routines, trans-lation algorithms, etc. While they can suggest which meth-ods are likely to work best online, they cannot predict the short- or long-term effects of differences among methods in the user feedback. Methods that appear promising in offline analyses must ultimately be validated by extensive online testing over prolonged periods in adequate numbers of users, and this testing should incorporate to the greatest extent possible appropriate internal (i.e. intra-individual) and/or external (i.e. inter-individual) controls.
4.9.3. Assessment of both user performance and system performance
Effective assessment of BCI performance requires two levels of evaluation: the user and the system. The user must control the signal features, and the system must recognize that control and translate it into device control effectively and consistently. User performance can be defined as the level of correlation between user intent and the signal feature(s) the BCI employs to recognize that intent. One useful measure of this correlation is r2 (Section 4.5). Perfect correlation produces an r2value of 1.00. As illustrated in Fig. 3, this measure proved very useful in choosing the best spatial filter method for extracting mu- or beta-rhythm signal features (McFarland et al., 1997b). It is also valuable for selecting the electrode location and frequency band used to determine mu- or beta-rhythm amplitude (e.g. Wolpaw et al., 2000b).
Evaluation of system performance has two parts: perfor-mance in a specific application, assessed as speed and/or accuracy, and theoretical performance, measured as
infor-mation transfer rate. Up to now, most studies have simply reported the accuracy and/or speed for specific applications.
These data are certainly important. At the same time, they are affected by the characteristics of the application and the success with which the system interfaces the user’s control of the signal features with that application. Thus, they make comparisons between different studies difficult, and they do not reveal what might theoretically be done with the degree of control that the user has.
The standard method for measuring communication and control systems is information transfer rate, or bit rate. It is the amount of information communicated per unit time.
Derived from Shannon and Weaver (1964) (summarized in Pierce, 1980), this measure incorporates both speed and accuracy in a single value. Fig. 4 shows the relationship between accuracy and bit rate for different numbers of choices. Bit rate is shown both as bits/trial (i.e. bits/selec-tion), and as bits/min when 12 selections are made per min (a rate comparable to that of several current BCIs (e.g.
Birbaumer et al., 2000; Donchin et al., 2000; Pfurtscheller et al., 2000a; Wolpaw et al., 2000b)). For example, the bit rate of a BCI that selects between two choices with 90%
accuracy is equal to that of a BCI that selects among 4 choices with 65% accuracy. The great importance of accu-racy, shown in Fig. 4, has often not received proper recogni-tion in BCI research. With two choices, 90% accuracy is
J.R. Wolpaw et al. / Clinical Neurophysiology 113 (2002) 767–791 785
Fig. 4. Information transfer rate in bits/trial (i.e. bits/selection) and in bits/
min (for 12 trials/min) when the number of possible choices (i.e. N) is 2, 4, 8, 16, or 32. As derived from Pierce (1980) (and originally from Shannon and Weaver, 1964), if a trial has N possible choices, if each choice has the same probability of being the one that the user desires, if the probability (P) that the desired choice will actually be selected is always the same, and if each of the other (i.e. undesired) choices has the same probability of selec-tion (i.e. ð1 2 PÞ=ðN 2 1Þ), then bit rate, or bits/trial (B), is:
B ¼ log2N 1 P log2P 1 ð1 2 PÞlog2½ð1 2 PÞ=ðN 2 1Þ. For each N, bit rate is shown only for accuracy $ 100=N (i.e. $chance) (from Wolpaw et al., 2000a).
Figure 3.6: Bit rate in bit/trial and bits/min(assuming 12 trials/min), when the number of possible selections N is 2, 4, 8, 16 and 32 [22].
of accuracy for a different number of choice can be examined in Figure 3.6.