II. Material
2.1. Dynamic driving environment
2.1.1. VR scene
Our VR scene was developed by using the World Tool Kit (WTK) 3D engine. The 3D view was composed of seven identical PCs running the same VR program. Seven PCs were synchronized by LAN so all scenes were going at exactly same pace. The VR scenes of different viewpoints were projected on corresponding locations. Figure 2-3 shows the layout of our simulator. The front screen marked 1 and 2 was overlapped by two polarized frames to
frontal screen with two projectors, respectively. By wearing special glasses with a polarized filter, the configuration provides a stereoscopic VR scene for a 3D visualization.
Literatures showed that the horizontal field of view (FOV) of 120° was needed for correct speed perception [31]. In our VR scene, the surrounded screens covered 206° frontal FOV and 40° back FOV, as shown in Figure 2-4. Frames projected from 7 projectors are connected side by side to construct a surrounded VR scene. The size of each screen has diagonal measuring 2.6-3.75 meters. The vehicle was placed at the center of the surrounded screens. Detailed information is shown in Table 2-1.
Figure 2-3: The configuration of the 3D surrounded scene. The 3D VR scene consists of 7 projectors, creating a surrounded view. Frontal screen is overlapped by 2 projector frames in different polarizations, providing a stereoscopic VR scene for 3D visualization.
2.1.2. Hydraulic Hexapod Motion Platform
Several studies showed that vestibular cues have a role in speed control and steering [32][33]. The vestibular cues or the motion cues could be provided by a motion platform controlled by six hydraulic linear actuators. This hexapod configuration was also called Stewart Platform [36] (as shown in Figure 2-5). The platform generated accelerations in vertical, lateral and longitudinal direction of vehicle as well as pitch, roll and yaw angular
Table 2-1: The Specification of driving simulator
Screen Number or Location Dimension
Screen Number 1, 2, 3, 4 (FOV 42°) (W)×(H) = (300 cm)×(225 cm) Screen Number 5, 6 (FOV 40°) (W)×(H) = (270 cm)×(202 cm) Screen Number 7 (FOV 40°) (W)×(H) = (210 cm)×(157 cm)
Vehicle Dimension (L)×(W)x(H) =
(430 cm)×(155 cm)×(140 cm) Driver to Front Screen (1, 2) 370 cm
Driver to Left and Right Screen (5, 6) 220 cm (Left) and 300 cm (Right) Driver Head Height Relate to Screen 1 120 cm
Figure 2-4: The overview of surrounded VR scene. The VR-based four-lane highway scenes are projected into surround screen with seven projectors.
accelerations. Figure 2-6 shows a basic idea how the motion platform simulates driving
(a) (b)
Figure 2-5: The Stewart platform. (a) The sketch map for the Stewart platform. (b) The actual Stewart platform. A driving cabin is mounted on this platform in our Lab.
Figure 2-6: How motion platform works. http://www.force-dynamics.com/
motion. When in deceleration, the driver feels a force pushes him/her against the belts, the platform tilts forward simultaneously to change the gravity direction sensed by the driver, and thus simulates the deceleration force. Similarly, the platform tilts backward to simulate acceleration force. This (or comparable) technique had been used widely in driving simulation studies [37].
The Hexapod Stewart Platform has superior performance in position control compared to traditional series manipulator. The parallel manipulator provides high-precision platform manipulations. Six extensible actuators equally share the loading of the platform, which provide high capability for realistic applications. Inverse kinematics analysis is used to solve the problem of converting the position and orientation of the payload platform with respect to the base platform. A singular solution of the inverse kinematics can be evaluated by simple formulae [38], which provides a high-speed platform and creats many possibilities for applications.
2.2. Introduction to motion tracking device
An accelerometer (inertia sensing, InertiaCube2 [39], as shown in Figure 2-7) was placed in the vehicle, at the center of movement. The InertiaCube2 is an inertial 3-DOF (Degree of Freedom) orientation tracking system. It obtains its motion sensing using a miniature solid-state inertial measurement unit, which senses angular rate of rotation, gravity and earth magnetic field along three perpendicular axes. The angular rates are integrated to obtain the orientations (yaw, pitch, and roll) of the sensors. Gravitometer and compass measurements are used to prevent the accumulation of gyroscopic drift. The InertiaCube2 is a monolithic part
based on micro-electro-mechanical systems (MEMS) technology involving no pinning wheels Figure 2-7: The accelerometer or platform motion tracking, Inertia Sensing, InertiaCube 300
(a) (b) Figure 2-8: The recording of orientation of InertiaCube2. (a) The demo program that shows
Pitch, Yaw and Roll recording. (b) The Roll, Pitch and Yaw axes.
that might generate noise, inertial forces and mechanical failures [40]. InertiaCube2 transfers digital data using RS232 protocol and converts it to USB with a small converter box. This accelerometer records orientations of the vehicle in pitch, roll and yaw during driving simulation, as shown in Figure 2-8. We will analyze physiological data and the orientation recording to investigate the relationship between human cognition and kinesthetic stimulus.
2.3. EEG and EMG acquisition
Subjects wore a movement-proof electrode cap with 36 sintered Ag/AgCl electrodes to measure the electrical activities of brain, i.e., EEG. The EEG electrodes were placed according to the international 10-20 system (as shown in Figure 2-9) with a unipolar reference at the right earlobe.
(a) (b) Figure 2-9: The International 10-20 system of electrode placement [43]. (a) lateral view.
(b) top view.
The impedance between EEG electrodes and skin was kept to less than 5kΩ by injecting NaCl based conductive gel. Data were amplified and recorded by the Scan NuAmps Express system (Compumedics Ltd., VIC, Australia) shown in Figure 2-10, a high-quality 40-channel digital EEG amplifier capable of 32-bit precision sampled at 1000 Hz. Table 2-2 shows the specifications of the NuAmps amplifier. The EEG data were recorded with 16-bit quantization levels at a sampling rate of 500 Hz in this study.
Data were preprocessed using a low-pass filter with a cut-off frequency of 50 Hz in order to remove the power line noise and other high-frequency noise. Similarly, a high-pass filter with a cut-off frequency at 0.5 Hz was applied to remove baseline drifts.
Figure 2-10: The NuAmps EEG Amplifier and the Electrode Cap
2.4. Subjects
The purpose of this study is to investigate the subject brain responses to kinesthetic stimulus. The subjects were instructed to perform the driving task consciously. Statistical results showed that the drowsiest period occurs from late night to early morning, and during the early afternoon hours [42]. According to these results, the driving experiments were conducted in middle morning or late afternoon to avoid the drowsiest time.
Ten healthy subjects participated in this research (one female and eleven males, aged between 20 and 28). Subjects were instructed to keep the car at the center of the lane by controlling the steering wheel, and to perform the driving task consciously. Each subject completed four 25-minute sessions in each driving experiment. To prevent subjects from feeling drowsy during experiments, they rested for few minutes after each session until they were ready for the next one. The whole driving experiment lasted about 2 hours. Subjects performed at least 2 driving experiments on different days for verifying the cross-session consistency.
Table 2-2: NuAmps Specifications
Analog inputs 40 unipolar (bipolar derivations can be computed) Sampling frequencies 125, 250, 500, 1000 Hz per channel
Input Range ±130mV
Input Impedance Not less than 80 MOhm
Input noise 1 µV RMS (6 µV peak-to-peak)
Bandwidth 3dB down from DC to 262.5 Hz, dependent upon sampling frequency selected
III. Experimental Setup
To investigate the influence of driving kinesthetic stimulus on cognitive states, we designed three simple driving events: deceleration, acceleration and deviation. The 6-DOF motion platform provided corresponding movements for different driving events. By switching the platform between “motion mode” and “motionless mode”, we produced two identical driving conditions with the only difference of the presence of platform motion. The EEG signals were recorded, analyzed and compared under these two conditions.
3.1. Driving Experiment Event
Figure 3-1: The simulated high way scene. The visual information is reduced to minimum to avoid unnecessary stimuli.
We developed a VR highway environment with a monotonic scene as shown in Figure 3-1 and eliminated all unnecessary visual stimuli. In the VR scene, the simulated driving speed was controlled by a scheduled program, thus subjects needed not to step on paddles, to prevent large muscle activity on the throttle or brake. We designed three driving events: stop,
go and deviation event. The stop and go events are paired, which means go event always follow the stop event, so we defined stop and go events as Stop-Go event. The deceleration and acceleration in Stop-Go event was controlled by a program. Figure 3-2 shows the time course of a Stop-Go event. If we define beginning of stop event as 0 second (bold arrow in Figure 3-2), a yellow light cue was shown on screen 1 second prior to the stop event. When a stop event began, the yellow light was replaced with a red light in same position, and the deceleration began. The car then slowed down and completely stopped in 4 seconds, the red light out. Stop lasted 7 seconds, and then the go event began. A green light was shown on screen, and start accelerating for 3 second. Then the green light was out, the vehicle was moving at constant speed, and a Stop-Go event ended.
During the experiment, subjects did not need to do anything in the stop and go events, but subjects were asked to handle the wheel to keep the car position at the center of cruising lane. During the experiment, the vehicle were randomly drifted away the crusing position, and the subjects were instructed to steer the vehicle back to the center of the crusing lane as
Figure 3-2: Illustration of the design for stop-go event.
moment the subject first steered the wheel to compensate the drift, were recorded for further analysis.
Figure 3-3 illustrates a deviation event. In phase 1 (Figure 3-3a), vehicle was moving forward in a straight line. Deviation event first occurred at the beginning of Phase 2 (Figure 3-3b), in which the vehicle deviated from the original cruising position. The vehicle deviated either to left or right. The deviation event enters Phase 3 when the subject started steering the vehicle back to the cruising position (Figure 3-3c). The subject would continue to steer the car until s/he thinks the car has returned to the center of the cruising lane. The moment of subject stopped the steering effort marked the beginning of the Phase 4 of the drift event (Figure 3-3d). The inter-event interval is 9 seconds. The Stop-Go event and deviation event were randomly occurred with the probability of 50%.
(a) (b) (c) (d) Figure 3-3: Illustration of the deviation event. (a) Vehicle moving in straight line. (b)
Deviation event occurred. (c) Subject’s reaction. (d) Vehicle back to middle lane.
Response time was recorded
3.2. Motion Profile
The experiment includes two conditions, the “motion mode” and “motionless mode”.
This was achieved by enabling or disabling the motion platform action. The platform motion in “motion session” was recorded through accelerometer as shown in Figure 3-2. The platform performed a pitching forward action in a stop-event, the angle was nearly 5 degrees, and a pitching backward action was performed in a go-event, the angle was nearly 4 degrees.
With these movements, subjects felt forward force while decelerating, backward force while accelerating and a sudden shaking while deviating due to the platform position change. In the
“motionless session” the platform was static and not response to the speed change or deviation of vehicle, only visually event was presented to driver. Each subject completed 4 sessions in a driving experiment, sessions were motion and motionless counterbalanced and each lasted 25 minutes. After each session subjects took a 5-10 min rest to prevent the drowsiness in driving. In order to confirm the consistency of subject’s brain activities from different experiments, each subjects completed 2~4 driving experiments for within-subject analysis.
IV. Data Analysis Procedure
In this study, the mutli-channel EEG signals were first separated into independent brain sources using Independent Component Analysis (ICA) [71]. Then we ploted the separated signals using Event Related Spectral Perturbation (ERSP) plot developed by Makeig, 1993 [61] and Event Related Potential (ERP). We then investigated the stability of component activations and scalp topographies of meaningful components across sessions within each subject. To test the reproducibility of component maps and activations across subjects, we performed component clustering analysis (detailed below).
4.1. Independent Component Analysis
The joint problems of electroencephalographic (EEG) source segregation, identification, and localization are very difficult since the EEG data collected from any point on the human scalp includes activity generated within a large brain area. The problem of determining brain electrical sources from potential patterns recorded on the scalp surface is mathematically underdetermined. Although the conductivity between the skull and brain is different, the spatial smearing of EEG data by volume conduction does not cause significant time delay and it suggests that the ICA algorithm is suitable for performing blind source separation on EEG data. The ICA methods were extensively applied to blind source separation problem since 1990s [45]-[52]. In recent years, subsequent technical reports [53]-[60] demonstrated that ICA was a suitable solution to the problem of EEG source segregation, identification, and localization based on the following assumptions: (1) The conduction of the EEG sensors is instantaneous and linear such that the measured mixing signals are linear and the propagation delays are negligible. (2) The signal source of muscle activity, eye, and, cardiac signals are
not time locked to the sources of EEG activity which is regarded as reflecting synaptic activity of cortical neurons [53][54].
In this study, we attempt to completely separate the twin problems of source identification and source localization by using a generally applicable ICA. Thus, the artifacts including the eye-movement (EOG), eye-blinking, heart-beating (EKG), muscle-movement (EMG), and line noises can be successfully separated from EEG activities. The ICA is a statistical “latent variables” model with generative form:
)
where A is a linear transform called a mixing matrix and the s are statistically mutually i independent. The ICA model describes how the observed data are generated by a process of mixing the components s . The independent components i s (often abbreviated as ICs) are i latent variables, meaning that they cannot be directly observed. Also the mixing matrix A is assumed to be unknown. All we observed are the random variables x , and we must estimate i both the mixing matrix and the IC’s s using the i x . i
Therefore, given time series of the observed data x(t)=
[
x1(t) x2(t) L xN(t)]
T in N-dimension, ICA will find a linear mapping W such that the unmixed signals u(t) arestatically independent.
Supposed the probability density function of the observations x can be expressed as:
)
the learning algorithm can be derived using the maximum likelihood formulation with the log-likelihood function derived as:
∑
Thus, an effective learning algorithm using natural gradient to maximize the log-likelihood with respect to W gives:
[
I u u]
Wand WTW rescales the gradient, simplifies the learning rule and speeds the convergence considerably. It is difficult to know a priori the parametric density function p u , which ( ) plays an essential role in the learning process. If we choose to approximate the estimated probability density function with an Edgeworth expansion or Gram-Charlier expansion for generalizing the learning rule to sources with either sub- or super-Gaussian distributions, the nonlinearity ϕ( u) can be derived as:
Since there is no general definition for sub- and super-Gaussian sources, we choose
(
(1,1) (-1,1))
before the tanh function and can be determined using a switching criterion as:[ ]
where
{ } { } { }
(
sec 2( i) i2 tanh( i) i)
,i =signE h u Eu −E u u
κ (10)
represents the elements of N-dimensional diagonal matrix K. After ICA training, we can obtain N ICA components u(t) decomposed from the measured N-channel EEG data x(t). In this study, N=30, thus we obtain 30 components from 30 channel signals.
).
Figure 4-1 shows a result of the scalp topographies of ICA weighting matrix W corresponding to each ICA component by projecting each component onto the surface of the scalp, which provide evidence for the components' physiological origins, e.g., eye activity
Figure 4-1: Scalp topography of ICA decomposition.
was projected mainly to frontal sites, and the drowsiness-related potential is on the parietal lobe and occipital lobe [68], motor related potential will locate at left and right side of front parietal lobe, etc. We can see that most of artifacts and channel noises are effectively separated into independent components 1 and 3.
4.2. ERP Analysis
Dawson first recorded the evoked potentials (EP) from cerebral cortex by taking pictures and accumulation skill in 1947 [73]. Dawson initiated the new field of neuro-physiology by introducing the technology of averaging evoked potentials (AEP) in 1951. The AEP technology is extensively applied to many experiments which relate to specific stimulus event, and is named event-related potentials (ERP) in recent years. The narrow definition of ERP is to present a specific region of perceptual systems and elicit potential changes on the cerebral cortex when the stimulus appears or disappears. The board definition of ERP suggests the responses come from all parts of neural system.
Generally, the ERP induced by the stimulus is 2 ~ 10 μV, much smaller than the amplitudes of ongoing EEG, and it is thus often buried in the EEG recordings. EEG signals are composed of small signals and big noise. In order to extract the ERP from EEG signal, we need to increase the signal to noise ratio by presenting the same type of stimuli to the subject repeatedly. ERP is often obtained by averaging EEG signals of accumulated single trials of the same condition. Ongoing EEG signals across single trials are considered random and independent of the stimulus. However, it is assumed that the waveform and latency of ERP pattern are invariant to the same stimulus. Through phase cancellation, time- and phase locked EEG signals will be more prominent. For example, if the number of trials for condition is n, the ERP will be n times the amplitude of original wave pattern and the EEG amplitude will only be n times of the initial signal. Therefore, the signal to noise ratio (SNR) will be
improved by n times. Therefore, ERP sometimes can be named Averaged Evoked Potentials and this is the basic theorem of extracting the ERP [44].
Figure 4-2: An ERP image includes the averaged ERP, response time and inter-trial information.
4.3. ERSP Analysis
The Event Related Spectral Perturbation, or ERSP, was first proposed by Makeig [61].
The ERSP reveals aspects of event-related brain dynamics not contained in the ERP average of the same response epochs. The limitation of ERP is that it must be coherent time-and-phase-locked activities. Averaging same response epochs would involve phase cancellation, brain activities not exactly synchronized in both time and phase are averaged out.
The ERSP measures average dynamic changes in amplitudes of the broad band EEG spectrum as a function of time following cognitive events. Through ERSP, we are able to observe time-locked but not necessary phase-lock activities.
The processing flow is shown in Figure 4-2. The time sequence of EEG channel data or ICA activations are subject to Fast Fourier Transform (FFT) with overlapped moving
windows. Spectrums prior to event onsets are considered as baseline spectra. The mean baseline spectra were converted into dB power and subtracted from spectral power after stimulus onsets so that we can visualize spectral ‘perturbation’ from the baseline. To reduce random error, spectrums in each epoch were smoothed by 3-windows moving-average. This procedure is applied to all the epochs, the results are then averaged to yield ERSP image.
The ERSP image mainly shows spectral differences after event, since the baseline spectra prior to event onsets have been removed. For instance, in the bottom of Figure 4-2 we can see very clearly that only little or no changes in high frequency band (the lower position the higher frequency) but very significant changes in low frequency band after event. This allows us to visualize spectral power change related to event.
After performing bootstrap analysis (usually 0.01 or 0.03, here we use 0.01) on ERSP, only statistically significant (p<0.01) spectral changes will be shown in the ERSP images.
Non-significant time/frequency points are masked (replaced with zero). Any perturbations in frequency domain become relatively prominent.
While the ERSP reveals new and potentially important information about event-related
While the ERSP reveals new and potentially important information about event-related