Organization of the Thesis

The thesis is organized as follows. The experimental setup including data collection, the setup of the virtual environments and the design of the virtual reality scene is given in Chapter

2. The analysis procedures of the multi-stream physiological signals related to motion sickness are given in Chapter 3. Chapter 4 describes the analysis EEG change corresponding to motion sickness. The discussions of experimental results are summarized in Chapter 5.

Finally, the conclusions and future research directions are given in the closing chapter.

Chapter 2

Design of Experiments

The induce factors of motion sickness are going to be introduced in this chapter, by means of a simple introduction to the vestibular system in human barin and inner ear. The experimental setup and environment are also given in this chapter, including design of the experiment scheme, Stewart platform, VR technology and physiological signal collection.

2.1 Introduction of Motion Sickness

Motion sickness begins when the brain receives visual and sensory clues that contradict each other. The introduction of the vestibular system in the inner ears are going to be introduced in the first subsection. A short description of motion sickness symptoms and the motion sickness experiments design in previous studies are summarized at follow subsections.

2.1.1 The Vestibular System

The vestibular system, a sensory apparatus localized bilaterally in the inner ears, provides information about movement and orientation of the head and body in space. The vestibular system is comprised of the otoliths and the non-acoustic portion of the inner ear which consists of three semicircular canals. The otoliths are the blue and green colored areas in Fig. 2-1, which detect linear acceleration. The three semicircular canals are the red, orange and pink areas in Fig. 2-1, which detect angular acceleration [23].

Fig. 2-1. The vestibular system and its measurement principles [23].

The otoliths provide the inertia required to drag the hair cells from side to side to provide the perception of motion. Once a constant speed is achieved, the otoliths stabilize and perceive motion disappears with respect to the vestibular system. An example of this phenomenon is sitting in a car when it first accelerates and then stabilizes. Normal function of this system is essential in many types of sensory-motor processes (e.g. compensatory eye movements, postural control, etc.).

The three semicircular canals correspond to the three dimensions of human movement can take place, respectively. Therefore, each canal detects motions in a single plane. Each canal is filled with a fluid called endolymph, which flows through the canal as the head experiences angular acceleration. As the fluid flows through the canal, it deflects small hair-like cells, called cupula, which send signals to the vestibular receiving areas of the brain, as shown in Fig. 2-2. It is noted that there are two vestibular components, located on two sides of the head that mirror each other and act in a push-pull manner. Since each group of hair cells is polarized, they can be either excited (pushed) or inhibited (pulled) based on which direction the cupula move. It is important for both vestibular apparatuses to agree with each other.

Under normal operation, one side of the head should push and the other should pull. If both sides are pushed, for example, vertigo will result [24].

Fig. 2-2. A cross section of a semicircular canal [52].

Furthermore, vestibular information plays important roles in perceptual tasks such as ego motion estimation. More recently, vestibular information was shown to disambiguate the interpretation of dynamic visual information experienced simultaneously during observer’s movement. As an example, considering the constant small wavers and rocking back and forth when someone is trying to stand still. This is a direct reflection of the vestibular system at work.

Under normal circumstances, the brain gathers information using the eyes, the inner ear, expectations and previous experiences. The problem begins when the brain receives visual and sensory clues that contradict each other. For example, if someone is inside a ship, with no view of the horizon, the room appears to be still but the balance mechanism in his inner ear detects that he is rocking back and forth when motion sickness begins.

2.1.2 The Symptoms of Motion Sickness

The malaise of motion sickness can be detected in several ways. Paleness is often the first symptom, then cold sweat on the forehead when people feel sick. There are a number of symptoms that can occur due to motion sickness including eye strain, headache, pallor, sweating, dryness of mouth, fullness of stomach, disorientation, vertigo, ataxia, nausea and vomiting [27]. There is an important concept which should be declared, that the symptoms of motion sickness are not identical to everyone.

Symptoms are greatly depending on different individuals. It is possible that there could be some symptoms which are not listed here. According to the study of Kolasinski [9], some people get motion sickness easier, while the others not. People who suffer from migraines are five times more likely to be affected. Women are more susceptible than men, and children are also more susceptible than adults. Some researches [28] have indicated that children age from 2 to 12 is easiest to get motion sickness, and the Asian suffer more than the others. Some abilities of individuals are also highly related to susceptivity of motion sickness. People with low ability of mental rotation or low concentration ability get the symptoms of nausea easier than others. These researches reveal that the individual difference between different subjects is large even in the same environment or stimuli condition.

2.1.3 Findings from Previous Studies

Study of motion sickness by meaning and analysis the physiological signals started in 1970s. Heart rate, and body temperature were first used at the initial stage [3]. Then, some researches involved more other physiological signals such as breathe, electrogastrography (EGG), galvanic skin response (GSR) signals to this study [4-7]. Analysis of electroencephalogram (EEG) related to motion sickness was approached in recent years [8].

The first famous research of motion sickness was done by Graybiel et al. in 1980 [3].

The subjects were asked to sit on a rotation chair in a metal cylinder as shown in Fig. 2-3. The physiological signals including heart rate, blood pressure ,and body temperature were recorded and then analyzed after the 30-second rotating duration. They concluded that the symptoms of motion sickness are not related to those recorded signals according to their results.

Fig. 2-3. A rotating chair in a metal cylinder [3].

However, the symptoms of motion sickness are reaction results between sympathetic nervous system and the parasympathetic nervous system, which are very dynamic. Graybiel et al. may wrong when they said that the symptoms of motion sickness are not related to the heart rate, blood pressure ,and body temperature, because they first analysis the signal with a 30-second averaging procedure, the operation has eliminated the dynamic property of the signals.

Stern et al. [4] first used EGG to study the onset, time course and symptoms of motion sickness induced by circularvection around the subject’s spinal axis, as shown in Fig. 2-4. A metal cylinder 91.5 cm in height and 76 cm in diameter was designed to provide visual stimulus. The interior of the cylinder was covered with alternating 3.8 cm-wide black and 6.2 cm-wide white vertical stripes. The subject’s head was maintained in the center of the drum by means of a chin rest. Based on visual inspection of the EGG waveform with time, Stern et al. [4] reported a shift in the dominant frequency of myoelectrical activity from 3 cycles per minute (cpm) to 5-8 cpm in all symptomatic subjects. 14 symptomatic subjects were labeled as “tachygastria” among a total of 21 subjects.

Fig. 2-4. A rotating drum in a metal cylinder [4].

A later definition of by the same authors specifically indicated that tachygastria is the absence of the 3 cpm and an increased tachygastria activity in the 4-9 cpm frequency range.

Sickness was defined as when subjects reported one or more of following symptoms: nausea, salivation, dizziness, and/or warmth. In the 14 symptomatic subjects, tachygastria appeared on the average 5 min after the start of drum rotation. In addition, tachygastria preceded the symptoms of motion sickness by 1 min in 9 of 14 symptomatic subjects.

Cowings et al. induced the symptoms of motion sickness of 58 subjects during two chair-rotating tests [5]. They studied the self-regulation of autonomic nervous system (ANS) activity. The measured physiological signals included heart rate, finger pulse volume, respiration rate, and skin conductance. According to the scores of the subjective questionnaire, they examined the stability of specific magnitude responses across both tests. Correlation analysis revealed marked, but quite stable, individual differences in ANS responses to both mild and severe motion sickness. These findings confirmed their prior observation that people are sufficiently unique in their ANS responses to motion sickness provocation such that it necessary to tailor self-regulation training individually. They also observed that individual ANS patterns are sufficiently consistent from test to test to serve as an objective indicator of individual motion sickness malaise levels.

As shown in Fig. 2-5 Miller et al. used a rotorcraft simulator to study the simulator-induced sickness [6]. Their objective was to investigate the sensitivity of physiological measures relative to self-reports of simulator sickness severity. The data suggested that heart period, tachygastria, and skin conductance level were more sensitive to simulator sickness than vagal tone and normal myoelectrical gastric activity. The effect of the motion sickness to the heart rate variability (HRV) and electrogastrogram (EGG) has been examined by Hirohisa et al. in 1996 [7]. The experiments were done on a boat in the Tokyo Bay to record the physiological signals in the motion sickness. It is seen EGG 0.05 Hz rhythm slightly enhanced during the motion sickness whereas HRV spectra show no significant difference.

Fig. 2-5. A rotorcraft simulator [6].

Min et al. project the virtual reality (VR) scene on a LCD monitor to induce motion sickness [8] as shown if Fig. 2-6 to induce motion sickness. Both subjective evaluation and EEG signals of each 5-min interval were analyzed to evaluate motion sickness. The results indicated that the frequency band 4 ~ 8 Hz of Fz EEG channel and the scores of subjective questionnaire are highly correlated.

Fig. 2-6. The virtual reality scene of Min’s experiments [8].

In this research, a consecutive curve-road driving experiment on a virtual-reality dynamic motion platform is designed to induce motion sickness that can be experienced in the real world. Various physiological signals of the subject including EEG, EKG, EGG, and GSR are recorded simultaneously when he/she is driving. More details are presented in the following sections.

2.2 Experimental protocol

For the reason of safety and reality, the virtual reality (VR) technology is used in our experiments to induce motion sickness. It takes the advantages of low cost, reality and time saving. The VR scene combined with the Stewart dynamic platform can provide the visual and kinesthetic stimuli to subjects. The subjects can interact directly with the environment and perceive more realistic driving conditions during the experiments.

A three-stage driving task is designed for this study. A ten minutes practice session was hold before each experiment which allowed the subjects to get used to the situation and the control methods. It consists of a 5-minute session of straight road driving at the beginning of the experiment, a 20-minute consecutive-curve road driving to induce motion sickness, and a 10-minute straight road for rest. The experimental scheme is shown in Fig. 2-7.

Fig. 2-7. The experimental scheme.

Start

The first 5-min session of each experiment, was regarded as the baseline. It is assumed that the subjects will fall into motion sickness after the 20-min curve road driving session.

The physiological signals collected during the “Motion-Sickness” session are then compared with those in the “Baseline” session. We assume that the differences of signals between the two sessions may be induced by motion sickness. The assumption is more impregnable if the differences of signals are vanished in the 10-min “Rest session”.

Different from a four-lane street scene used in the study of Min et al. [8], a VR tunnel scene was designed for our experiments. The VR tunnel scene provides shallow depth of the field that is suitable for the purpose of the study since fast movement of objects can induce the motion sickness effectively.

In addition, the subjects are asked to answer a motion sickness questionnaire (MSQ) after the navigation of “Motion-Sickness” session as shown in Fig. 2-7 for subjective evaluation of motion sickness. The designed MSQ will be introduced in section 3.1.

2.3 Virtual-Reality-based Dynamic Driving Environment

Some previous works indicated that the malaise of motion sickness can sometimes cause self-control ability decline and lead to serious traffic accident fatalities[27]. For the purpose of safety, a VR-based dynamic driving environment was developed for the motion sickness experiments to mimic realistic stimuli.

Fig. 2-8. A 360-degree 3D VR dynamic driving environment.

The developed VR dynamical simulation system mainly consists of three elements: (1) a six-degree-of-freedom motion platform, (2) a real car, and (3) an interactive VR scene. The subjects are asked to sit inside the car on the platform with their hands holding the steering wheel to control the car in the VR scene. Seven projectors are used in the experiment to construct a 360-degree 3D scene as shown in Fig. 2-8. The movements of the platform are according to the operation of the subject and the condition of the road surface.

2.3.1 Stewart Platform

A typical Stewart platform has a lower base platform and an upper payload platform connected by six extensible legs with ball joints at both ends as shown in Fig. 2-9. A Stewart platform is also called a six-degree-of-freedom motion platform, which means the payload platform of the system has 6-dimension of freedom, including the X, Y, and Z dimensions in space, and the Roll, Pitch and Yaw directions of motion. Positions of the payload platform in the applications can be decided according to the six parameters.

Fig. 2-9. The Stewart platform in NCTU Brain Research Center.

The Stewart platform has excellent performance in position control while comparing with the traditional series manipulator. Non-accumulation of position errors provides high-precision in the operation of platform control. The six extensible legs equally share the loading on the platform, which provide high loading capability in realistic applications.

Inverse kinematics analysis is used to treat the problem of converting the position and orientation of the payload platform with respect to the base platform. Excellent possibility of high-speed platform application can be provided because a singular solution of the inverse kinematics can be evaluated by simple formulae [32].

2.3.2 Apparatus in the Real Car

A real car is placed on the Stewart platform as shown in Fig. 2-10. Many high-sensitivity sensors and equipments are set up inside the car to accomplish the experiment.

Fig. 2-10. A real car fixed on the Stewart platform.

A video camera was placed beside the steering wheel as shown in Fig. 2-11 to capture the movements and facial expression of subjects during the experiments. The operating staff can remotely monitoring the subject in the operation room. The experiments will be terminated by the operating staff immediately when the subject seems unable to continue.

Fig. 2-11. A video camera placed beside the steering wheel.

Two physiological signal amplifiers, NuAmps [47] and GSR100C [48] are also fixed on the car for the purpose of data acquisitions as shown in Fig. 2-12. An EEG electrode cap was mounted the on the subject’s head to record the EEG signals. Four Ag-AgCl electrodes were placed on the chest of subjects to record EKG and EGG signals respectively. The 32-channel EEG, 2-channel EKG and 2-channel EGG signals were collected and enhanced by NuAmps amplifier. The galvanic skin response (GSR) signal was collected at the back of the neck of the subject with two Ag-AgCl electrodes and the recordings were enhanced with the GSR amplifier, GSR100C.

Fig. 2-12. The NuAmps and the GSR100C for physiological signal acquisition.

2.3.3 Design of the VR Scene

For the development of the VR driving scene, the 3DS-max software is used to create the three-dimensional models and the WorldToolKit (WTK) library is used to program the VR scenes. The 3DS-man software is a popular graphic software to create a three-dimensional model. The WTK library is an advanced cross-platform development environment for high-performance, real-time and three-dimensional graphics applications.

The development flow of the VR sense is shown in Fig. 2-13. Firstly, the 3DS-max is used to build three-dimensional models accurately for a true system (such as the road) and to define the parameters of each model (such as the width of the road). Then, the C program including the WTK library is used and its library function is called up to move the three-dimensional models.

Fig. 2-13. Development flow of the VR scene.

2.4 Data Acquisition

2.4.1 Subjects

Ten healthy volunteers (including five males and five females) with no history of gastrointestinal, cardiovascular, or vestibular disorders participated in the experiments of the motion sickness study. The ages of these subjects are from 18 to 26 years with a average of 22 years. They were requested not to smoke, drink caffeine, use drugs, or drink alcohol for a week prior to the main experiment to avoid influencing the central and autonomic nervous system.

3DS-Max

Model Parameter

C Program WTK

VR Scene

2.4.2 Data Collection Sensors

According to the symptoms of motion sickness discussed in subsection 2.1.2, some physiological signals can be considered as important indices related to the motion sickness.

For instance, the galvanic skin response (GSR or skin conductance response) and the Electrogastrography (EGG) are highly related to the symptoms of sweating and vomiting, respectively. The EGG and EKG are the most popular ones used in the studies of motion sickness. Some researches indicated that EEG and EKG are related to the onset of motion sickness directly [7,8]. In this study four different physiological signals including EEG, EGG, EKG and GSR were recorded simultaneously for further analysis.

2.4.2.1 Electroencephalogram (EEG)

An electrode cap is mounted on the subject’s head for signal acquisitions on the scalp.

The 10-20 International System of Electrode Placement standard to place the EEG electrodes proposed by Jasper in 1958 [10], was used in this study. An illustration of the 10-20 system is shown in Fig. 2-14.

Fig. 2-14. The 10-20 international electrode placement system [10].

The letters F, C, T, P, and O represent the frontal, central, temporal, parietal, and occipital cortical regions on the scalp, respectively. The term “10-20” means 10% and 20% of the total distance between specified skull locations. The percentage-based system allows differences in skull locations. A total of 32 electrodes are used in this system and the top view of the scalp is shown in Fig. 2-15.

Fig. 2-15. The 32 EEG electrodes locations.

The NuAmps, manufactured by NeuroScan Company, is a high-quality 40-channel digital EEG amplifier that is capable of 22 bit sampling at 1000 Hz, measuring signals from

DC to 260Hz. Table 2-1 shows that the specifications of the NuAmps. The 500 Hz sampling frequency was uesd in this study. Photos of the NuAmps amplifier and the electrode cap are shown in Fig. 2-16.

Table 2-1: NuAmps Specifications

NuAmps Specifications

Analog inputs 40 unipolar (bipolar derivations can be computed) Sampling frequencies 125, 250, 500, 1000 Hz per channel

Input Range ±130mV

Input Impedance Not less than 80 MOhm

Input noise 1 µV RMS (6 µV peak-to-peak)

Bandwidth 3dB down from DC to 262.5 Hz, dependent upon sampling frequency selected

Fig. 2-16. The electrode cap and the EEG signal amplifier.

2.4.2.2 Electrocardiogram (EKG)

An electrocardiogram (EKG) is one of the simplest and fastest procedures used to assess

An electrocardiogram (EKG) is one of the simplest and fastest procedures used to assess