• 沒有找到結果。

利用腦波之獨立成份分析結合虛擬實境動態模擬系統開發駕駛員瞌睡偵測技術

N/A
N/A
Protected

Academic year: 2021

Share "利用腦波之獨立成份分析結合虛擬實境動態模擬系統開發駕駛員瞌睡偵測技術"

Copied!
107
0
0

加載中.... (立即查看全文)

全文

(1)國 立 交 通 大 學 電機與控制工程學系 碩 士 論 文 利用腦波之獨立成份分析結合虛擬實境動態 模擬系統開發駕駛員瞌睡偵測技術 EEG-Based Drowsiness Estimation Using Independent Component Analysis in Virtual-Reality Dynamic Driving Simulator. 研 究 生: 指導教授:. 陳 林. 俞 進. 傑 燈. 教授. 中 華 民 國 九 十 四 年 七 月.

(2) 利用腦波之獨立成分分析結合虛擬實境動態 模擬系統開發駕駛員瞌睡偵測技術 EEG-Based Drowsiness Estimation Using Independent Component Analysis in Virtual-Reality Dynamic Driving Simulator. 研 究 生:陳 俞 傑 指導教授:林 進 燈 教授. Student:Yu-Jie, Chen Advisor:Prof. Chin-Teng, Lin. 國立交通大學 電機與控制工程學系 碩 士 論 文. A Thesis Submitted to Institute of Electrical and Control Engineering College of Electrical Engineering and Computer Science National Chiao Tung University in partial Fulfillment of the Requirements for the Degree of Master in Electrical and Control Engineering July 2005 Hsinchu, Taiwan, Republic of China. 中華民國九十四年七月.

(3) 授 權 書 (碩士論文) 本授權書所授權之論文為本人在 國立交通 大學 電資 學院 電機與控制工程 系所 電機控制 論文名稱:. 組. 九十三. 學年度第. 二 學期取得. 碩. 士學位之論文。. 利用腦波之獨立成分分析結合虛擬實境動態模擬系統 開發駕駛員瞌睡偵測技術 EEG-Based Drowsiness Estimation Using Independent Component Analysis in Virtual-Reality Dynamic Driving Simulator. 1. 5同意. 不同意. 本人具有著作財產權之論文全文資料,授予行政院國家科學委員會科學技術資料中心、 國家圖書館及本人畢業學校圖書館,得不限地域、時間與次數以微縮、光碟或數位元化 等各種方式重製後散佈發行或上載網路。本論文為本人向經濟部智慧財產局申請專利的 附件之一,請將全文資料延後兩年後再公開。 (請註明文號: ) 2. 5同意. 不同意. 本人具有著作財產權之論文全文資料,授予教育部指定送繳之圖書館及本人畢業學 校圖書館,為學術研究之目的以各種方法重製,或為上述目的再授權他人以各種方 法重製,不限地域與時間,惟每人以一份為限。 上述授權內容均無須訂立讓與及授權契約書。依本授權之發行權為非專屬性發行權利。 依本授權所為之收錄、重製、發行及學術研發利用均為無償。上述同意與不同意之欄位 若未鉤選,本人同意視同授權。 指導教授姓名:. 林. 研 究 生簽名: (親筆正楷) 日 期: 民國. 進. 燈. 九十四. 教授. 年. 七. 學號: 9212556 (務必填寫) 月 十七 日. 1. 本授權書請以黑筆撰寫並影印裝訂於書名頁之次頁。 2. 授權第一項者,所繳的論文本將由註冊組彙總寄交國科會科學技術資料中心。 3. 本授權書已於民國 85 年 4 月 10 日送請內政部著作權委員會(現為經濟部智慧財產 局)修正定稿。 4. 本案依據教育部國家圖書館 85.4.19 台(85)圖編字第 712 號函辦理。.

(4) EEG-Based Drowsiness Estimation Using Independent Component Analysis in Virtual-Reality Dynamic Driving Simulator Student:. Yu-Jie, Chen. Advisor:. Prof. Chin-Teng, Lin. Institute of Electrical and Control Engineering National Chiao Tung University. ABSTRACT Preventing accidents caused by drowsiness has become a major focus of active safety driving in recent years. It requires an optimal estimation system to online continuously detect drivers’ cognitive state related to abilities in perception, recognition and vehicle control. The propose of this thesis is to develop an adaptive drowsiness estimation system based on electroencephalogram (EEG) by combining with independent component analysis (ICA), time-frequency spectral analysis, correlation analysis and fuzzy neural network model to estimate a driver’s cognitive state in Virtual-Reality (VR) dynamic driving simulator. Moreover, the VR-based motion platform with EEG measured system is the innovation of brain and cognitive engineering researches. Firstly, there is good evidence to show that the necessary of VR-based motion platform for brain research in driving simulation. This is an important fact to stress that the kinesthetic stimuli obviously influence the cognitive states and the phenomenon can be indicated by the EEG signals. Secondly, a single-trial event-related potential (ERP) is applied to recognize different brain potentials by the five degrees of drowsiness in driving. And we demonstrate a close relationship between the fluctuations in driving performance and the EEG signal log bandpower spectrum. Our Experimental results show that it is feasible to accurately estimate the driving performance. Then we observe that the brain source related to drowsiness is on cerebral cortex. Finally, the spiked dry electrodes and the corresponding movement artifact removal technology were designed to replace the regular wet electrode for the purpose of applications in the realistic driving or working environments. Keyword : Drowsiness, Electroencephalogram, Virtual Reality, Dynamic Platform, Cognitive State, Event-Related Potential, Kinesthetic Stimulus, Independent Component Analysis, Dry Electrode. i.

(5) 利用腦波之獨立成分分析結合虛擬實境動態 模擬系統開發駕駛員瞌睡偵測技術 研究生:陳. 俞 傑. 指導教授:林. 進. 燈. 教授. 國立交通大學電機與控制工程學系碩士班. 中文摘要 摘 要 近年來,預防瞌睡所導致的交通意外,已經成為交通安全研究的重要課題,我們需 要一個最理想的估測系統,可以即時連續的偵測駕駛員的精神認知狀態、知覺以及控制 車輛的能力。本論文的目的在發展一套有效的駕駛精神認知狀態估測系統,利用腦電波 訊號結合頻譜分析、獨立成分分析演算法、相關係數分析以及類神經網路模型,結合虛 擬實境動態模擬駕駛系統,開發駕駛員的瞌睡偵測技術;此外在虛擬實境的環境中,結 合腦波量測系統與動感平台,進行神經認知系統研究,在腦科學與認知工程領域上都是 一項創新。 首先我們證明利用虛擬實境結合動感平台,以進行實用之認知工程研究是必要的, 動態刺激會明顯的影響腦波訊號認知狀態。我們亦利用單一試驗的事件相關腦電位分 析,去識別開車時不同瞌睡程度的腦電位變化,並且證明人類腦波特定的頻帶活動與開 車行為表現之間的關係非常密切,並經由實驗結果顯示,利用腦波訊號分析以估測駕駛 員行為表現是可行的。我們亦研究在大腦皮層上與發生瞌睡相關的區域,最後為了實際 應用的可行性,我們利用乾式電極結合獨特之雜訊消除技術取代傳統電極,以期將本論 文所開發的技術,未來應用於實際駕駛與工作環境中。. 關鍵字:瞌睡偵測,腦電波,虛擬實境,動態平台,認知狀態,事件相關電位,動覺刺 激,獨立成分分析,乾式電極。. ii.

(6) 誌. 謝. 本論文的完成,首先要感謝我的指導教授 林進燈博士在過去兩年研究期間,提供 豐富的研究資源和實驗環境,並從旁指導協助,使得本文得以順利完成。 其次,我要感謝我的父母對我的照顧與栽培,教導我做人品德為最,強調人格健全 之發展與學習生活之態度,由於他們辛勞的付出和細心的照顧,才有今天的我;亦感謝 我哥哥在我研究的路上隨時給予指導與協助,豐富的經驗減少我許多錯誤的發生。 特別感謝美國加州聖地牙哥大學的 鐘子平教授、 段正仁教授及 黃瑞松學長,給 予我研究上最大的協助,從實驗設計、實驗分析、實驗結果討論到論文撰寫,給我最專 業的意見跟看法。 另外,我要感謝腦科學研究實驗室的全體成員,沒有他們也就沒有我個人的成就。 特別感謝 梁勝富教授給予我在各方面的指導,無論是研究上疑難的解答、研究方法、 寫作方式、經驗分享以及生活上壓力調適等惠我良多。另外要感謝士政、行偉以及欣泓 同學,在過去兩年研究生活中同甘共苦,相互扶持。此外,我也要感謝陳玉潔學姊與黃 騰毅學長在研究上的幫助,還有感謝力碩、宗哲以及弘義學弟,在過去這一年中的相伴。 同樣地也感謝實驗室助理在許多事務上的幫忙。 最後,我要感謝我的女朋友黃莉萍小姐,替我分擔許多研究上的壓力與挫折,也讓 我在研究所的生活當中,增添更多色彩。 謹以本文獻給我親愛的家人與親友們,以及關心我的師長,願你們共享這份榮耀與 喜悅。. iii.

(7) Content Abstract in English ....................................................................................................................i Abstract in Chinese ..................................................................................................................ii Acknowledgement....................................................................................................................iii Content .....................................................................................................................................iv List of Tables............................................................................................................................vi List of Figures .........................................................................................................................vii Abbreviation .............................................................................................................................x Ⅰ.Introduction .........................................................................................................................1 1.1. Current Researches of Drowsiness Estimation.......................................................2 1.1.1 Detecting Physical Changes .......................................................................3 1.1.2 Measuring Physiological Changes .............................................................5 1.2 Virtual Reality Dynamic Simulator........................................................................7 1.3 Organization of This Thesis .................................................................................10 Ⅱ. System Architecture .........................................................................................................12 2.1 3D Virtual Reality Environment...........................................................................13 2.2 Stewart Motion Platform ......................................................................................15 2.3 EEG Data Acquisition ..........................................................................................16 2.4 Subject ..................................................................................................................17 2.5 Spiked Dry Electrode ...........................................................................................18 Ⅲ.Experimental Design .........................................................................................................20 3.1 The Influence of Kinesthetic Stimulus on Cognitive State ..................................21 3.2 Investigation of Drowsiness Event-Related Potentials.........................................23 3.3 Adaptive Estimation of Continuous Driving Performance...................................24 3.4 Search for Brain Source of Drowsiness on Cerebral Cortex ................................25 3.5 Application of Dry Electrodes in the Drowsiness Experiment.............................28 Ⅳ.Data Analysis .....................................................................................................................29 4.1 4.2 4.3 4.4 4.5 4.6 4.7. Event-Related Potential (ERP) Analysis ..............................................................29 Analysis of Continuous EEG Data .......................................................................32 Independent Component Analysis (ICA) .............................................................36 Time-Frequency Spectral Analysis ......................................................................40 Correlation Analysis .............................................................................................42 Adaptive Feature Selection Mechanism ...............................................................44 Self-cOnstructing Neuro-Fuzzy Inference Network (SONFIN) ..........................45. iv.

(8) Ⅴ.Results and Discussions.....................................................................................................49 5.1. 5.2. 5.3. 5.4. 5.5. The Influence of the VR-based Motion Platform on Cognitive States.................49 5.1.1 The Brain Source of Kinesthetic Stimulus on the Cerebral Cortex..........49 5.1.2 Necessity of VR-based Motion Platform .................................................53 The Brain Activity of Drowsiness in Different Cognitive States.........................59 5.2.1 The Degree of Cognitive States in Drowsiness ........................................59 5.2.2 The Dynamic Platform Influences Drowsiness ERP................................63 The Performance of Adaptive Drowsiness Estimation.........................................64 5.3.1 Relationship between the ICA/EEG Power Spectrum and Drowsiness ...65 5.3.2 The Dominant ICA Components and EEG Channels for Drowsiness .....67 5.3.3 Selection of Frequency Bands Based on Spectral Correlation and AFSM 69 5.3.4 Drowsiness Estimation based on ICA Components or EEG Channels ....72 5.3.5 Driving Performance Estimation based on AFSM and SONFIN.............74 5.3.6 Performance Comparison Using Different Moving-Average Window Length 76 The Brain Source of Drowsiness on the Cerebral Cortex ....................................77 5.4.1 Comparison with Using Different Number of EEG Channels .................77 5.4.2 Comparison of Using Different Region of EEG Channels.......................80 Actual Application of the Spiked Dry Electrodes ................................................83. Reference .................................................................................................................................90. v.

(9) List of Tables Table 1-1 Techniques for Detecting Drowsiness.......................................................................3 Table 5-1 The scalp topographies of two ICA components have different response in the two conditions .....................................................................................................................52 Table 5-2 Two ICA components and two EEG channels of the five participants with the highest correlation coefficients with the driving deviation are selected for adaptive drowsiness estimation...................................................................................................68 Table 5-3 The correlation coefficients between the log subband power spectra and the driving performance of Subject 3 corresponding to the different frequency bands from 8 to 15 Hz of the ICA component 11 and 13 in the training and testing sessions that uses the same ICA weighting matrix obtained from the training session ..................................69 Table 5-4 The correlation coefficients between log subband power spectra and the driving performance of subject 3 using the optimal frequency bands (from 10 to 14 Hz) corresponding to single component..............................................................................70 Table 5-5 The frequency bands for the two ICA components in Table 5-2 selected by manual method and the AFSM technology corresponding to different subjects ......................71 Table 5-6 Driving performance estimation using the optimal frequency bands and linear regression model of the five participants by two ICA components or two EEG channels ........................................................................................................................73 Table 5-7 Driving performance estimation using the frequency bands selected by manual method and the AFSM technology based on two dominant ICA components as input features of the linear regression model and SONFIN models for five subjects ...........75 Table 5-8 Comparison of driving performance estimation obtained from different number EEG channels by using the optimal frequency bands of two EEG channels or ICA components as the features of linear regression model for the five participant ...........77 Table 5-9 Comparison of driving performance estimation using two EEG channels of the four different regions............................................................................................................80 Table 5-10 Driving performance estimation of subject 2 by using two spiked dry electrodes87. vi.

(10) List of Figures Fig. 1-1: The vestibular system and its measurement principles. ..............................................8 Fig. 2-1: The block diagram of the dynamic VR-based driving simulation environment with the EEG-based physiological measurement system. .......................................................12 Fig. 2-2: Flowchart of the VR-based highway scene development. The dynamic models and shapes of the 3D objects in the VR scene are created and linked to the WTK library to form a complete interactive VR simulated scene. ...........................................................13 Fig. 2-3: The VR-based four-lane highway scenes are projected into 360° surround screen with seven projectors. Several photos captured from different view angle at a fixed point are connected to form this wide figure. ...........................................................................14 Fig. 2-4: The Stewart platform. (a) The sketch map for the Stewart platform. (b) The actual Stewart platform. A driving cabin is mounted on this platform in our laboratory. .........15 Fig. 2-5: The International 10-20 system of electrode placement. (a) A lateral view, (b) A top view. ................................................................................................................................16 Fig 2-6: Corresponding equivalent circuit illustrated below shows that spiked dry electrodes can perform a low-impedance interface better than the standard electrodes. (a) Standard wet electrode, (b) Spiked dry electrode. ..........................................................................18 Fig. 2-7: Photographing of fabrication result of spiked dry electrodes busing optics microscope.......................................................................................................................19 Fig. 3-1: The flowchart of designs and goals of all experiments. ............................................20 Fig. 3-2: The view of the driving cabin forward at rear in VR-based highway scene. ............21 Fig. 3-3: Illustration of the design for stop and start experiments. ..........................................22 Fig. 3-4: The width of highway is equally divided into 256 units and the width of the car is 32 units. ................................................................................................................................23 Fig. 3-5: The continuous driving performance of long-term recordings in the driving simulation. (a) The distribution of driving performance, (b) Moving averaged driving error in a 60-minute experiment with at least 2 drowsy periods. ....................................25 Fig. 3-6: Five conditions for different number of EEG channels. (a) 30 channels, (b) 20 channels, (c) 15 channels, (d) 10 channels, (e) 6 channels..............................................26 Fig. 3-7: Four clusters of electrodes on the scalp. (a) Frontal location, (b) Left temporal location, (c) Right temporal location, (d) Parietal and occipital location........................27 Fig. 4-1: The flowchart of EEG data analysis in the first experiment......................................31 Fig. 4-5: Moving-averaged log power spectral analysis for ith ICA component. .....................40 Fig. 4-6: Canonical correlation spectral matrix of subject 3. Note that the higher correlation coefficients appear at 9 ~ 25 Hz in ICA components 11 and 13, respectively................43. vii.

(11) Fig. 4-7: Example of the adaptive feature selection mechanism for subject 3. Note that the band power of ICA components 11 and 13 at frequency bands 10 ~ 14 Hz are selected as input feature of the estimators.....................................................................................45 Fig. 4-8: The network structure of SONFIN. ...........................................................................46 Fig. 5-1: The scalp topographies of all ICA components trained by EEG data from Subject 1. .........................................................................................................................................50 Fig. 5-2: Two ICA components have different responses between the two conditions of all events. (a) The source near FC3 location, (b) The source near FC4 location. ................50 Fig. 5-3: The ERP and ERSP analyses of component 20 for stop event. (a) The ERP of motion condition, (b) The ERP of motionless condition, (c) Overplot power spectrum of two conditions, (d) The ERSP of motion condition, (e) The ERSP of motionless condition. .........................................................................................................................54 Fig. 5-4: The ERP and ERSP analyses of component 20 for start event. (a) The ERP of motion condition, (b) The ERP of motionless condition, (c) Overplot power spectrum of two conditions, (d) The ERSP of motion condition, (e) The ERSP of motionless condition. .........................................................................................................................55 Fig. 5-5: The ERP and ERSP analyses of component 20 for deviation event. (a) The ERP of motion condition, (b) The ERP of motionless condition, (c) Overplot power spectrum of two conditions, (d) The ERSP of motion condition, (e) The ERSP of motionless condition. .........................................................................................................................56 Fig. 5-6: The ERP analysis of component 20 for deviation event with reaction time. (a) The ERP with reaction time of motion condition, (b) The ERP with reaction time of motionless condition, (c) Aligning onset by reaction time from (a), (d) Aligning onset by reaction time from (b)......................................................................................................57 Fig. 5-7: The trials are sorted according to reaction time and equally divided into five groups of subject 4. .....................................................................................................................59 Fig. 5-8: One component is related to drowsiness and the ERP analysis with all single-trials. .........................................................................................................................................60 Fig. 5-9: The results of ERSP analysis in five different cognitive states. (a) Drowsiness level from 1 ~ 20 %, (b) Drowsiness level from 21 ~ 40 %, (c) Drowsiness level from 41 ~ 60 %, (d) Drowsiness level from 61 ~ 80 %, (e) Drowsiness level from 81 ~ 100 %. ........61 Fig. 5-10: The results of ERSP analysis in five different cognitive states if the dynamic platform is motionless. (a) Drowsiness level from 1 ~ 20 %, (b) Drowsiness level from 21 ~ 40 %, (c) Drowsiness level from 41 ~ 60 %, (d) Drowsiness level from 61 ~ 80 %, (e) Drowsiness level from 81 ~ 100 %. ...........................................................................63 Fig. 5-11: The results of correlation coefficient analysis for Subject 3. (a) The correlation coefficient spectra of ICA components, (b) The correlation coefficient spectra of EEG channels, (c) The ICA component with highest correlation with the driving performance, (d) The EEG channel with highest correlation with the driving performance. ...............65 viii.

(12) Fig. 5-12: Two ICA components and two EEG channels with the highest correlation coefficient with the driving performance index. (a) ICA Component 11, (b) ICA Component 13, (c) EEG Pz channel, (d) EEG P4 channel..............................................66 Fig. 5-13: Driving performance estimation of Subject 3 using linear regression model with the optimal frequency bands selected manually. (a) Result of training session by using ICA components, (b) Result of testing session by using ICA components, (c) Result of training session by using EEG channels, (d) Result of testing session by using EEG channels. ..........................................................................................................................72 Fig. 5-14: Correlation coefficients between the driving performance and EEG log power spectrum from 9 ~ 15 Hz in Pz channel of subject 3 by using different moving averaged windows lengths. .............................................................................................................76 Fig. 5-15: Comparison of estimating results by using ICA components and EEG channels...79 Fig. 5-16: The ESEI comparison between dry electrodes and wet electrode with/without skin preparation. (a) Without skin preparation, (b) With/without skin preparation................83 Fig. 5-17: The scalp topographies of all ICA components trained by EEG data of Subject 2 using 2 spiked dry electrodes and 30 wet electrodes.......................................................84 Fig. 5-18: The raw EEG data are measured by placing the spiked dry electrodes at FP1 and FP2 channels using the standard electrodes for the others channels of Subject 2. (a) The EEG signals of FP1 and FP2 channels with movement artifacts, (b) The EEG signals after ICA-based artifact removal. ....................................................................................85 Fig. 5-19: The EEG signals measured by using the spiked dry electrodes before/after artifacts removal using ICA decomposition technology. (a) and (b) EEG power spectra signals of FP1 and FP2 channels before removing all noise components, (c) and (d) EEG power spectra of FP1 and FP2 channels after removing all noise components. ........................86 Fig. 5-20: Correlation coefficients spectra of all EEG channels after removing all noise components......................................................................................................................87. ix.

(13) Abbreviation Subject. Be an Abbreviation for. AFSM. Adaptive Feature Selection Mechanism. API. Application Programmer’s Interface. DOF. Degree Of Freedom. ECG / EKG. Electrocardiogram. EEG. Electroencephalogram. EMG. Electro Muscle-movement Graph. EOG. Electrooculogram. ERP. Event-Related Potential. ERSP. Event-Related Spectral Perturbation. FNN. Fuzzy Neural Network. GSR. Galvanic Skin Response. HMD. Head Mounted Display. HRV. Heart Rate Variability. ICA. Independent Component Analysis. MEMS. Micro Electro Mechanical Systems. NREM. Non-Rapid Eye Movement. REM. Rapid Eye Movement. SONFIN. Self-cOnstructing Neuro-Fuzzy Inference Network. SVM. Support Vector Machine. VR. Virtual Reality. WTK. WorldToolKit. x.

(14) Ⅰ.Introduction During the past few years, driving safely has received extensive attention from the publics due to the growing number of traffic accidents. Drivers’ fatigue has been a causal factor in many accidents because the marked decline in the drivers’ abilities of perception, recognition and vehicle control abilities while sleepy. In the United States, according to the National Highway Traffic Safety Administration’s (NHTSA) conservative estimation, 100,000 police-reported crashes are direct results of driver’s fatigue in each year [1], which results in about 1,550 deaths, 71,000 injuries and $12.5 billion in monetary losses. The National Science Foundation (NSF) also reported that 51% of adult drivers felt drowsy while driving vehicles and 17% actually fall asleep in 2002 [2]. Although many governments and vehicle manufacturers try to make policies, including strategies to address rates of speed, alcohol consumption, promotion of using helmets and seat belts, and enhancements of vehicle structures, etc [3-4], to prevent accidents, it is difficult to avoid disasters resulted from drivers’ loss of alertness and lack of attentions. Driving under drowsiness will cause: (a) longer reaction time, which increases the risk of crash, particularly at high speeds; (b) vigilance reduction, including no or delaying response to emergency; (c) deficits in information processing, which will reduce accuracy in decision-making tasks [5-7]. Many factors, including lack of sleep, long driving hours, use of sedating medications, consumption of alcohol and some driving patterns such as driving at midnight, early morning, or mid-afternoon hours, will cause drowsiness or fatigue in driving. In addition, the nature of the task, such as driving in a monotonous environment, may also cause fatigue. The improvement of vehicles has made drivers more and more effortless to operate their vehicles on the road. An examination of the situations when drowsiness occurred shows that most of the accidents occur on freeways [8]. Hence, accurate and non-intrusive 1.

(15) real-time monitoring of driver's drowsiness would be highly desirable, particularly if this measure could be further used to predict changes in driver's performance capacity. The purpose of this thesis is to develop an adaptive drowsiness estimation system based on electroencephalogram by using independent component analysis. In the following session, we first survey current researches of drowsiness estimation. Then we emphasize the importance of the Virtual-Reality-based dynamic motion platform to brain research in driving experiments. Finally, the organization of this thesis is summarized in the last section.. 1.1 Current Researches of Drowsiness Estimation Table 1-1 summarizes a number of methods that have been proposed to detect drowsiness [8]. For the sensing approaches of human physiological phenomena, these methods can be categorized into two main fields. For drowsiness estimation, these methods can be further classified in two categories, non-contact and direct-contact. Direct-contact methods require sensors attached to the driver’s body. Non-contact methods use optical sensors or video cameras to detect vigilance changes and achieve a satisfactory recognition rate. However, these parameters vary in different environmental situations and driving conditions. It is necessary to devise different detection logic for different types of vehicles.. 2.

(16) Table 1-1 Techniques for Detecting Drowsiness Detection Techniques. Sensing of Human. Physiological Signals. Physiological Phenomena. Physical Reactions. Sensing of Driving Operation. Sensing of Vehicle Behavior Response of Driver Traveling Conditions. Description. Detection Accuracy. Practicality Extendibility. Detection by Changes in Brain Waves, Blinking, Heart Rate, Pulse Rate, Skin Electric Potential,. ◎. ╳. △. ◎. ○. ╳. ○. ◎. ╳. ○. ◎. ╳. △. ╳. ◎. ╳. ○. ◎. etc. Detection by Changes in Inclination Driver’s Head, Sagging posture, Frequency at Which Eyes Close, Gripping force on Steering Wheel, etc. Detection by Changes in Driving Operations (Steering, Accelerator, Braking, Shift Lever, etc.) Detection by Changes in Driving Behavior (Speed, Lateral G, Yaw Rate, Lateral Position, etc.) Detection by Periodic Request for Response Detection by Measurement of Traveling Time and Conditions (Daytime or Nighttime, Speed, etc.). ◎ : Very Good ○ : Good. △ : Average. ╳ : Poor. Reference: Hiroshi Ueno and Masayuki Kaneda and Masataka Tsukino (1994) [8]. 1.1.1 Detecting Physical Changes The physical change during approaches detect of eye-closure over time, eye tracking as quantization of drowsiness level, driver’s head movements, and steering wheel angle [8]. In Hamouda’s study, all available information was collected by the police and recorded in the accident report. It showed that the presence of driver fatigue relating drowsiness is an important cause of truck accidents. This study also proposed to classify truck accident relating fatigue and non-fatigue by neural network model [9]. Richard Grace, a researcher at the Carnegie Mellon Driving Research Center, developed a driver monitoring system [11]. The. 3.

(17) vehicle performance and physiological data were measured when drivers were driving trucks. This research proposed two drowsiness detection methods, including a video-based system that measures drowsiness associated with slow eye closure and the other based on vehicle performance data. The video-based system measured eye closure to obtain the percentage of eye-closure over time (PERCLOS) for detecting driver drowsiness in real time under nighttime driving conditions. A video-based system (CCD), the PERCLOS Camera, successfully measured eye closure and detect drowsiness in heavy vehicle truck operators [12]. Perez developed a non-invasive interface that tracks eye positions using digital image processing techniques. This approach detected eyes positions using image processing algorithms and a non-invasive interface and labeled eye tracking into five stages as a quantization index of driver’s drowsiness. Only gray-level images were processed in this research: 102 images from the Purdue University’s database and 897 images from a video sequence were pre-processed using face detection algorithm and the results of correct detection rates were very high. [13]. Pilutti proposed an identification approach to assess driver’s state in lane-keeping tasks [14]. This approach used a driver model to simulate a real highway driving situation, including the perfectly smooth, asphalt, and concrete road surface. They obtained lateral positions of vehicles to assess drivers’ performance, using driver steering wheel as the input of the driver model, and extracted the parameters from the chosen candidate model (ARX model). The model parameters are estimated for driving task and the results are good for model fit with the ARX model to represent the relationship between vehicle lateral position and the driver steering wheel angular position for detecting driving patterns, assessing driver performance, and improving vehicle active safety [14]. Popieul proposed a set of drowsiness indicators using evolution of driver’s head movements for monitoring drivers’ drowsiness efficiently. They considered variables related to the head position and the driving performance.. 4.

(18) The approach developed a driving simulator with basic highway section such as straight line, right line, and left line. The driving performance (steering wheel angle) and physiological signals (driver’s head movements) of the subjects participated in a long-term simulated highway trip were measured. [15]. Ji et al. predicted driver’s fatigue by a real-time noninvasive monitoring system. They remotely acquired video images of the driver by using charge-coupled-device (CCD) cameras which are equipped with active infrared illuminators. Ji’s research team used the Support Vector Machine (SVM) to catch eyes in the facial images and the Kalman filter to track eyes. The approach indicated that the fatigue relating to eyelids’ movement of a person can be used as driver’s drowsiness index. They quantized the fatigue of the eyelids’ movement with two methods percentage of eye-closure over time (PERCLOS) and the average of eye-closure speed (AECS). They used the Bayesian network (BN) to model fatigue index, extract the level of alertness of a person, infer the driver’s fatigue level, and systematically display the fatigue level on fatigue evidence window in real-time [16].. 1.1.2 Measuring Physiological Changes The other field focused on measuring drivers’ physiological changes such as the heart rate variability (HRV), the galvanic skin response (GSR), and especially the electroencephalogram (EEG), as a means of detecting the human cognitive states [17-21]. Studies show that the human EEG generated by synchronous post-synaptic currents in large populations of neurons in the cortex can reflect brain activities. It has been known that abundant information in EEG recording can be related to drowsiness, arousal, sleep, and attention [22]. Previous psychophysiological studies show that typical sleep rhythm regulated. 5.

(19) by the circadian process can be divided into non-rapid-eye-movement (NREM) sleep and rapid-eye-movement (REM) sleep [23-24]. NREM sleep is further subdivided into 4 stages. In the first part of falling into sleep (micro-sleep at NREM), increasing amplitudes of slow alpha waves of the EEG signals are observed with positive correlation at occipital sites (O1 and O2) and negative correlation at central sites (C3 or C4) [25-26]. While the approaches based on EEG signals have the advantages for making accurate and quantitative judgments of alertness levels, relatively little information has been captured in real time until signal processing methods and computer power are fast enough to extract the relevant information from the EEG. Thus, it is practicable and appealing to know what information about human cognitive state and behavior are available through analyzing complex EEG signals. Roberts developed a tool to characterize the level of the vigilance of vehicle drivers by recording the physiological signals in real-time [19]. This approach builds up a portable device for the alertness detection of vehicle drivers by recording the EEG signals, then, studying the implementation of a decision algorithm based on Kohonen artificial neural networks by the variations of alpha, beta, theta and delta waves of the EEG signals according to a data base of 12 files of 24-hour EEG registered in volunteers. They observed a negative correlation between the score of vigilance and the percentage of the beta band and a positive correlation between the score of vigilance and the percentage of the other EEG (theta, alpha, and beta) spectral bands. Wilson detected the instance at which a person had lost the level of alertness necessary to assure safe operation of a vehicle or display vigilance. They proposed a neural network to detect the driver’s alertness state. The input of the neural network system is a feature vector composed of the Wavelet transforms representations of EEG signals at different scales, and the output of the system is a binary decision to decide the EEG represents either an alert state or a drowsy state [21]. In Parikh’s study, the subjects EEG data were recorded while driving a 6.

(20) vehicle simulator and the EEG data was analyzed using the four-ordered Wavelet Transform as an indicator. The subjects were asked to observe their driving in the same position without any movement. In this study, increasing amplitudes of slow alpha waves of the EEG signals were observed during the monotony of the long distance driving because of repeating driving, viewing of the same track, less tension, or the tendency to drowsiness [25]. Some issues remain in practical applications using EEG signals such as the handling of artifacts. While driving, subjects move their hands, torso, head, and eyes, which create huge muscle movements, eye movements, and blink artifacts. Low pass filtering cannot resolve this problem. Another issue is individual difference in EEG dynamics accompanying loss of alertness. It is not easy to accurately estimate or predict individual changes in alertness and performance [27-31].. 1.2 Virtual Reality Dynamic Simulator Virtual reality (VR) technology is gradually being recognized as a useful tool for the study and assessment of normal and abnormal brain function, as well as for cognitive rehabilitation. Virtual Environments (VE) are created by powerful computers that generate realistic animated graphics in three dimensions. The computers are configured with peripheral devices, such as immersible head-mounted displays (HMDs) that allow complex interactions within the VE with a sense of presence. Creating carefully controlled, dynamic, 3D stimulus environments combined with physiological and behavioral response recording can be offer more assessment options that are not available by traditional neuropsychological methods. The VR technique allows subjects to interact directly with a virtual environment rather than monotonic auditory and visual stimuli. It is an excellent strategy for brain research on 7.

(21) interactive and realistic tasks due to low cost and avoiding risk of operating on the actual machines. In recent years, some researchers designed the VR senses to provide the appropriate environments for brain activity study. In this study, a VR-based dynamic motion platform combined with EEG measured system is an innovation in brain and cognitive engineering researches. Without combining with dynamic motion platform, it is unable to study the influence of kinesthetic stimulus on cognitive state. Human brain can deal with complicated information. An example is the balance between optic scenes and kinesthetic perception. If the simulator environments cannot produce visual and kinesthetic stimuli simultaneously, the subjects may not correctly response in the real world.. Fig. 1-1: The vestibular system and its measurement principles. Reference: Andras Kemeny and Francesco Panerai (2003) [32]. The relevant organ system of human body to kinesthetic perception is the vestibular system [32]. The vestibular system is a sensory apparatus located bilaterally in the inner ears. Its function is to detect the motion of the head and body in space [33]. A vestibular system is composed of two functional parts shown as Fig. 1-1: (1) the otolith organs (Fig. 1-1, blue and 8.

(22) green colored areas), and (2) the semicircular canals (Fig. 1-1, red, pink and orange areas), which are selectively sensitive to linear and angular accelerations respectively. There are three semicircular canals filled with a viscous liquid, the endolymph. The pressure on the cupula, a specialized structure at the end of each canal, is increased by the liquid when the head moves. Pressure stimuli are transformed into nerve discharge, encoding the angular acceleration of the head. In the some way, the otolith receptors, composed of a mass of crystals floating in the endolymph, encode both linear acceleration and tilt of the head. [34]. Moreover, the otoliths signal the rotation of the head relative to gravity, that is, head tilt [35], which the nervous system resolves from linear acceleration by means of internal models [36]. In many types of sensori-motor processes such as the postural control, normal functioning of this system is essential. Additionally, vestibular information plays an important role in perceptual tasks such as egomotion estimation [37]. Vestibular information was shown to disambiguate the interpretation of dynamic visual information experienced simultaneously during observer’s movement recently [38]. During the simulation process of driving, the absence of vestibular information increases steering reaction times to external movement perturbations [39], and also decreases safety margins in the control of lateral acceleration in curve driving [40]. In real driving, improper signals from disordered vestibular organs are reported to determine inappropriate steering adjustment [41]. Furthermore, the presence of vestibular information in driving simulators seems important because it influences the perception of illusory self-tilt and illusory self-motion [42].. 9.

(23) 1.3 Organization of This Thesis The purpose of this study is to develop methods of using EEG signals to accurately and non-intrusively monitor the continuous fluctuations of driver's global level of drowsiness accompanying changes in driver's performance near real-time in a realistic driving task. We first construct a Virtual-Reality interactive driving environment consisting of a highway scene and a six degree-of-freedom (6-DOF) motion platform. By several simple driving actions such as deceleration, acceleration, and deviation, we demonstrate that distinct cognitive state responses are discernible between the dynamic platform which is motion and motionless. This is a good evidence to show that the dynamic motion platform is required for the study of human cognitive state estimation. Secondly, we design a lane-keeping driving experiment to indirectly quantify driver’s drowsiness level [43]. It helps to illustrate the changes of drowsy event-related-potential (ERP) between different drowsiness states. After we recognize the feature of brain activities in drowsiness, we develop a novel adaptive feature selection mechanism (AFSM) for EEG spectra. And then we build an individualized fuzzy neural network models to assess the EEG dynamics accompanying loss of alertness for each subject. Finally we consider the feasibility of the proposed method for practical applications. The main issue is to use less EEG channels to perform satisfactory results. The main purpose of the experiment is to investigate the cortical sources of drowsiness. The driving performance can be estimated according to the analysis of the number of dominated EEG channels and the source regions on the scalp. Finally, we try to use spiked dry electrodes to replace the standard wet electrodes on the prior experiment. The reason is that driver may be difficult to use electrodes cap and electrolytic gel in a realistic driving situation. This thesis is organized as follows. Section II describes the details the EEG-based drowsiness experimental setup, VR-based dynamic driving environment, EEG data collection, 10.

(24) instructions, and spiked dry electrode. In Section III, we design a series of experiments for drowsiness estimation for EEG processing. We explore the innovative methods by applying ICA, time-frequency spectral analysis, correlation analysis, and fuzzy neural network in Section IV. Detailed discussions of our experimental results are given in section V. Finally, we conclude our findings in SectionⅥ.. 11.

(25) Ⅱ. System Architecture In this chapter, a VR-based dynamic driving environment is designed and built up for interactive driving experiments. It includes four major parts as shown in Fig. 2-1: (1) the 3D highway driving scene based on the virtual reality technology, (2) the driving cabin simulator mounted on a 6-DOF dynamic Stewart motion platform, (3) the EEG physiological signal measurement system with 36-channel EEG/EOG/ECG sensors, and (4) the proposed signal processing modules including ICA decomposition, power spectral analysis, and fuzzy neural work model. This environment will be presented in details as follows. In addition, the novel spiked dry electrodes used in our experiments for EEG acquisition are also being introduced.. Fig. 2-1: The block diagram of the dynamic VR-based driving simulation environment with the EEG-based physiological measurement system. 12.

(26) 2.1 3D Virtual Reality Environment In this thesis, a VR-based high-fidelity 3D interactive highway scene and its emulation software, WorldToolKit (WTK) library and application programmer’s interface (API) are developed [60]. The detailed development diagram of the VR-based scene is shown in Fig. 2-2. Firstly, we create the models of various objects (such as cars, roads, and trees, etc.) for the scene and setup the corresponding positions, attitudes, and other relative parameters. Then we develop the dynamic models among these virtual objects and build a complete highway simulated scene of full functionality with the aid of the high-level C-based API program.. Fig. 2-2: Flowchart of the VR-based highway scene development. The dynamic models and shapes of the 3D objects in the VR scene are created and linked to the WTK library to form a complete interactive VR simulated scene. 13.

(27) Generally, the VR scenes are projected onto a curved screen or one or more flat screens, and some simulators use head-mounted displays (HMDs) to provide stereoscopic viewing. In our laboratory, the VR-based four-lane highway scenes are projected into the 360° surround screen with seven projectors at different positions as shown in Fig. 2-3.. Fig. 2-3: The VR-based four-lane highway scenes are projected into 360° surround screen with seven projectors. Several photos captured from different view angle at a fixed point are connected to form this wide figure.. In order to increase stereoscopic perception and avoid the questions caused by using HMDs such as uncomfortableness, a little oppression, and the overheated instrument, we use two projectors to reach the binocular parallax. The VR scenes for the left and right eyes are projected onto the frontal screen with two projectors, respectively. By wearing the light 3D glasses, such configuration provides more stereoscopic VR scene than using HMDs.. 14.

(28) 2.2 Stewart Motion Platform Since Stewart developed a prototype of a six-degree-of-freedom (6-DOF) parallel manipulator in 1953 [61]. It has attracted tremendous attention from researchers for high-precision robotic tasks where the requirements of accuracy and sturdiness are more essential than those of a large workspace and manoeuvrability [62-64]. A typical Stewart platform has a lower base platform and an upper payload platform connected by six extensible legs with ball joints at both ends, as shown in Fig. 2-4. The parallel manipulator has 6-DOF including coordinates of X, Y, Z for position and roll, pitch, yaw for direction in space. In the following, an inverse kinematics analysis of the Stewart platform will first be made. Then a fuzzy control algorithm will be designed for the position control. Lastly, a washout filter is designed for the angular velocity/linear acceleration control of the Stewart platform [65].. (a). (b). Fig. 2-4: The Stewart platform. (a) The sketch map for the Stewart platform. (b) The actual Stewart platform. A driving cabin is mounted on this platform in our laboratory.. 15.

(29) 2.3 EEG Data Acquisition 34 EEG/EOG channels (using sintered Ag/AgCl electrodes with an unipolar reference at right earlobe), 2 ECG channels (bipolar connections between the right clavicle and left rib), and one 8-bit digital signal produced form VR scene are simultaneously recorded by the Scan NuAmps Express system (Compumedics Ltd., VIC, Australia). All EEG/EOG channels were located based on a modified International 10-20 system as shown in Fig. 2-5 [66]. The 10-20 system is based on the relationship between the location of an electrode and the underlying area of cerebral cortex. Before acquiring EEG data, the contact impedance between EEG electrodes and skin was calibrated to be less than 5kΩ by injecting NaCl based conductive gel. The EEG data were recorded with 16-bit quantization levels at a sampling rate of 500 Hz and were down sampled to 250 Hz for the simplicity of data processing. All EEG data were preprocessed using a simple low-pass filter with a cut-off frequency at 60 Hz in order to remove the line noise and other high-frequency noise. Similarly, a high-pass filter with a cut-off frequency at 0.5 Hz was applied to remove baseline drifts for further analysis.. (a). (b). Fig. 2-5: The International 10-20 system of electrode placement. (a) A lateral view, (b) A top view [66]. 16.

(30) 2.4 Subject It is known that the drowsiness often occurs during late nights, early morning and mid-afternoon. During these periods, alertness may easily diminish within one-hour monotonous working [7-8]. In drowsiness experiments, the subjects participated in the highway-driving simulation after lunch in the early afternoon. All the subjects were instructed to keep the car at the center of the cruising lane by controlling a steering wheel. In all sessions, the subjects drive the car continuously for 60 minutes and were asked to try their best to stay alert. Participants then returned on different days to complete a second 60-minute driving session or more sessions if necessary. In opposition to the drowsiness experiments, for the kinaesthetic stimulus experiments, we arrange the experiment time in the morning or in the afternoon to keep the best condition for subjects. Each subject has to participate in two 30-minute sessions which replace the order of dynamic platform is motion and motionless of once experiment. In the same way, participants must return on different days to accumulate enough data to analyze. We collected EEG data from 16 subjects (ages from 20 to 35 year old) participating in the VR-based driving task. In drowsiness estimation experiment, we select participants who had two or more micro-sleeps checked by video recordings in both driving sessions for further analysis. Based on these criteria, five subjects were selected for further modeling and cross-session testing.. 17.

(31) 2.5 Spiked Dry Electrode In recent years, the fabrication and characterization of Micro-Electro-Mechanical -Systems (MEMS) based silicon micro probe arrays, namely spiked dry electrodes, were explored for EEG measurement applications. A series of practical in-vivo tests had showed that the MEMS based spiked dry electrodes have more advantages and conveniences than the conventional standard electrodes. Comparing to the standard wet electrodes, the spiked dry electrodes can collect stronger signal intensity with a smaller device area, which means the design of related amplifier circuit can be simpler and easier. In addition, the spiked dry electrodes can be used without electrolytic gel, and they will not cause an uncomfortable feeling for the tested subject [67].. (a). (b). Fig 2-6: Corresponding equivalent circuit illustrated below shows that spiked dry electrodes can perform a low-impedance interface better than the standard electrodes. (a) Standard wet electrode, (b) Spiked dry electrode. Reference: P. Griss, P. Enoksson, H. K.Tolvanen-Laakso, P. Merilainen, S. Ollmar (2001) 18.

(32) Biopotential electrode for EEG transforms the bio-signals from skin tissue to the amplifier circuit. Therefore, the most important characteristic of a biopotential electrode is low electrode-skin interface impedance to propagate signals without attenuation or production of noise. As the Fig. 2-6 indicates, the spiked dry electrode is designed to pierce the stratum corneum (SC) into the electrically conducting tissue layer of stratum germinativum (SG) in order to circumvent the high impedance characteristics of the SC. In the Brain Research Center of the University System of Taiwan, the μ System & Control Lab led by Prof. J.C. Chiou had already developed the spiked dry electrodes. Three types of spiked dry electrodes varied in dimension including 4×4 mm², 3×3 mm² and 2×2 mm² are successfully fabricated using MEMS technology. Etch spiked dry electrode consists of 20×20 micro probes with 35 μm in diameter and 300 μm in height as shown in Fig. 2-7.. μSystem & Control Lab.. Fig. 2-7: Photographing of fabrication result of spiked dry electrodes busing optics microscope.. 19.

(33) Ⅲ.Experimental Design This study investigates the feasibility of using multi-channel EEG data to estimate and predict non-invasively the continuous fluctuations in human global level alertness in a realistic driving task. For this purpose, our concern is to carefully design a series of experiments for the scientific discovery and practical applications. Experimental designs are important because correct designs of experiments will distinctly acquire the expectable and incontrovertible results. Therefore, this chapter describes the design of each experiment in details and the flowchart is shown in Fig. 3-1.. Fig. 3-1: The flowchart of designs and goals of all experiments. 20.

(34) 3.1 The Influence of Kinesthetic Stimulus on Cognitive State This topic intends to investigate the influence of kinesthetic stimulus on cognitive state and the purpose here is to justify the necessity of using VR-based motion platform. Through the movements of 6-DOF motion platform, this configuration provides drivers dynamic feeling with such as deceleration, acceleration, and deviation. We can investigate the cognitive states of the same driving actions with or without platform motion. For this fundamental research, we must simplify our concerned topic and reduce the other variations between the experiment and control. We develop a VR-based highway environment with a monotonic scene as shown in Fig. 3-2, because a complicated scene may bring unexpected visual stimulus. We keep the driving speed of simulation at 100 km/hr in order to avoid the stepping, that will cause large muscle activity on the throttle or brake. Similarly, the driving speed of simulation will automatically increase or decrease with the movements of motion platform if the traffic light is displayed on the screen.. Fig. 3-2: The view of the driving cabin forward at rear in VR-based highway scene.. 21.

(35) In this research, each subject participated in two 30-minute sessions in one single day of experiment until enough EEG data for the ERP analysis were accumulated. The procedure of this experiment which we must comply with is to alternate two conditions, with and without platform movement. The motion and motionless will appear randomly to avoid expecting effect with a fixed order of two conditions. During the session, the VR-based scene and the car will be stopped, started and deviated according to the traffic lights in order to simulate the driving situations in the real world.. Fig. 3-3: Illustration of the design for stop and start experiments.. One trial in this experiment is explained as a combination of a stop and a start event with a 5 ~ 10 seconds time interval between two events. The stop and start events are maintained for 3 seconds with the displayed traffic light in red and green, respectively. Simultaneously, the movements of the platform, such as deceleration and acceleration, will depend on the corresponding events. In addition, the yellow light is displayed for 1 second before each trial so that the subject will not be shocked by the sudden deceleration of motion platform. The time interval between the trial and deviation event is 10 ~ 15 seconds. The time course of experiments is shown in Fig. 3-3.. 22.

(36) 3.2 Investigation of Drowsiness Event-Related Potentials First of all, we have to find the relationship between the measured EEG signals and the subject’s behavioral performance. One point should be taken as a quantified level of the subject’s alertness while driving. Hence, we define the subject’s driving performance index as the deviation between the center of the vehicle and the center of the cruising lane [43]. By examining the video recordings, the pilot experimental studies show that when the subject is drowsy, the driving performance will decrease and vice versa. The four lanes from left to right are separated by a median stripe in the VR-based scene. The distance from the left side to the right side of the road is equally divided into 256 points for outputting digital signal from WTK program, and the width of each lane and the car is 60 units and 32 units, respectively. All the descriptions about the width are depicted in Fig. 3-4.. 0. 60 63. 123. 132. 192 195. 0. 255. 32. Fig. 3-4: The width of highway is equally divided into 256 units and the width of the car is 32 units. 23.

(37) The refresh rate of highway scene was set properly to emulate a car cruising at a fixed speed of 100 km/hr. The subject’s performance is defined as the deviations between the center of the vehicle and the center of the cruising lane. The car is randomly drifted away from the center of the cruising lane to mimic the consequences of a non-ideal road surface. So the driver must maintain high attention to immediately correct the direction of vehicle in the cruising lane. When the driver is drowsy, the reaction time between the onset of deviation and steering wheel is increased. This event can be used for ERP analysis of different drowsiness states using 30-channel EEG signals. The reaction time is continuously and simultaneously measured by the WTK program and recorded in the physiological measurement system accompanying with EEG/EOG/ECG physiological signals. In this design, the subjects are asked to participate in the 60 minutes experiment twice for data accumulation. Although we fix the experiment time in the early afternoon hours such that drowsiness time often occurs, the drivers must try to stay alert and not to fall asleep. Otherwise the wrong cognitive state will be erroneous judged due to intentionally sleeping in driving.. 3.3 Adaptive Estimation of Continuous Driving Performance In addition to recognize the feature of brain activity in drowsiness, we also want to develop a drowsiness estimation system for driving. In differentiation to Experiment 2 of single-trial analysis, we deal with the continuous 30-channel EEG signals of long-term recordings. This design is similar to Experiment 2 because we use the same VR-based highway scene and the same length of experimental time. Therefore, the subject’s performance is also defined as the deviations between the center of the vehicle and the center of the cruising lane. We select the participants who have two or more micro-sleeps checked by video recordings in both driving sessions for further analysis. The individual model which 24.

(38) estimates driving performance using the features will be established by the two sessions for training and testing respectively. Fig. 3-5 shows driving performance recorded in a 60-minute session of one subject.. (a). (b). Fig. 3-5: The continuous driving performance of long-term recordings in the driving simulation. (a) The distribution of driving performance, (b) Moving averaged driving error in a 60-minute experiment with at least 2 drowsy periods.. 3.4 Search for Brain Source of Drowsiness on Cerebral Cortex After establishing the individual model to estimate driving performance, we will assess the feasibility of proposed method for practical applications. The main purpose of this experiment is to investigate the brain source of drowsiness. Hence we can use less EEG channels on relative region to perform satisfactory result for estimating driving performance. In this research, the estimation of driving performance will be evaluated to analyze the. 25.

(39) number of EEG channels and the regions on the scalp. We expect to find out the universal brain source of drowsiness on cerebral cortex among our participators. First we compare five results based on different number of EEG channels. These five conditions include 30, 20, 15, 10 and 6 EEG channels proportionally distributed on scalp by the International 10-20 system. We arbitrarily decide the locations of 6-channel EEG electrodes because they are unable to proportionally distribute in the International 10-20 system. Therefore, six most frequently used channels for common experiment are selected in this design. The detailed channel locations on scalp map we consider are shown as Fig. 3-6.. (a). (b). (d). (c). (e). Fig. 3-6: Five conditions for different number of EEG channels. (a) 30 channels, (b) 20 channels, (c) 15 channels, (d) 10 channels, (e) 6 channels.. 26.

(40) The Cz channel is the center of the International 10-20 system and the scalp is divided into four regions according to the position of Cz channel in this experiment. The frontal location is defined as the region from Cz to forehead as shown in Fig. 3-7 (a). The left and right temporal locations are defined as the regions from Cz to temples respectively as shown in Fig. 3-7 (b) (c). Finally, the parietal and occipital location is defined as the region including parietal and occipital bone as shown in Fig. 3-7 (d). Each region contains 7 electrodes for analysis.. (a). (b). (c). (d). Fig. 3-7: Four clusters of electrodes on the scalp. (a) Frontal location, (b) Left temporal location, (c) Right temporal location, (d) Parietal and occipital location.. 27.

(41) 3.5 Application of Dry Electrodes in the Drowsiness Experiment So far, we utilize a few channels from a region on the scalp to achieve a satisfactory result of estimating driving performance through a series of experiments. Although the estimation system has excellent performance in our experiments, it is difficult to apply the electrode cap with electrolytic gel in the realistic driving situations. The spiked dry electrode was designed in this experiment to replace the standard electrode to avoid using electrolytic gel. However, it still has difficulty in using the spiked dry electrodes at present. The first question is that the height of probes on the spiked dry electrodes, which are limited to the MEMS technology, is too short. The probes are difficult to contact stratum germinativum even stratum corneum because the thickness of human hair is usually about 80 μm. The hair elasticity also makes it difficult to fix the spiked dry electrode on the scalp. Therefore we try to fix the spiked dry electrodes in the places without hair in this experiment, such as the forehead. In order to test the feasibility of using the spiked dry electrodes, we replace the standard electrodes on FP1 and FP2 channels with dry electrodes. We repeat the same experiment of drowsiness estimation in this design, but the only difference is that it includes two spiked dry electrodes as well as all EEG channels. The two EEG signals measured by the spiked dry electrodes will be used in our drowsiness estimation system in this experiment. We have adequate reason to believe that the cognitive state of drowsiness can be recognized in frontal region of the cerebral cortex. The result of estimation performance will verify the feasibility of practical application in the future.. 28.

(42) Ⅳ.Data Analysis Our study includes five topics of experiments as described in Chapter Ⅲ. Two methodologies are used for data analysis. The first one is to deal with the single-trial EEG signals for ERP analysis in Experiment 1 and Experiment 2. The second one is to analyze the continuous EEG data of long-term recordings for the last three experiments. This chapter describes the data analysis procedure of the five experiments in terms of these two methodologies in details. The technology and algorithms applied in our experiments will also be presented in this chapter, including Independent Component Analysis (ICA), time-frequency spectral analysis, correlation analysis, adaptive feature selection mechanism (AFSM) and Self-cOnstructing Neuro-Fuzzy Inference Network (SONFIN).. 4.1 Event-Related Potential (ERP) Analysis Dawson first reported to record the evoked potentials (EP) from cerebral cortex by taking pictures and accumulation skill in 1947 [66]. Dawson initiated the new field of neuro-physiology by introducing the technology of averaging evoked potentials (AEP) in 1951. The AEP technology is extensively applied to many experiments due to the relative stimulus, so the AEP is gradually named event-related potentials (ERP) in recent years. The narrow definition of ERP is to present a specific region of perceptual systems and induce potential changes on the cerebral cortex when the stimulus appears or disappears. The board definition of ERP suggests the responses come from all neural system. Generally, the ERP induced by the stimulus is 2 ~ 10 μV, much less than ongoing potential of EEG amplitude, and it is hidden among the EEG signals. EEG signals are 29.

(43) composed of small signals and big noise so that the ERP is cannot to be directly measured and analyzed from EEG signal. In order to extract the ERP from EEG signal, the stimuli must be presented to the subject repeatedly. ERP is obtained by averaging EEG signals of accumulated single trials of the same condition. EEG signals across single trials are considered random and independent of the stimulus. However, it is assumed that the waveform and latency of ERP pattern are invariant to the same stimulus. After accumulating all ERP, the ERP increases proportionally to the number of trials and the EEG amplitude is the sum of adding according to random noise theorem. For example, if the number of trials for condition is n, the ERP will be n times the amplitude of original wave pattern and the EEG amplitude will only be n times of the initial signal. Therefore, the signal to noise ratio (SNR) will be improved to. n multiples of the original ratio. ERP is the average of n trials. of EEG epochs. Therefore, ERP sometimes can be named AEP and this is the basic theorem of extracting the ERP [68]. The ERP techniques are applied to Experiment 1 and Experiment 2 for analyzing events. We also use event-related spectral perturbation (ERSP) analysis in these experiments. In first experiment, we demonstrate the three events including stop, start and deviation events of VR-based driving simulation. The dynamic platform is either in moving or motionless conditions. For the stop and start events, the continuous EEG signals are extracted into several epochs, each of which contains the sampled EEG data from -1500 ms to 4000 ms with a light onset at 0 ms and the length of baseline is 500 ms foremost in each epoch. Similarly, the duration of the deviation event is 3000ms, ranging from -1000 ms to 2000 ms, with deviation onset at 0 ms. The baseline is computed from -1000 ms to 0 ms. Then we combine with the four events including the stop and start event in two conditions and use ICA algorithm to decompose 30-channel EEG signals into the 30 independent components. Simultaneously, we apply the ICA mixing matrix from above result to the deviation event and indicate reaction. 30.

(44) time of each deviation event. Therefore we can compare with each component by ERP and ERSP analysis of these three events in the two conditions to justify the necessity of VR-based motion platform in driving simulation. The detailed flowchart of EEG data analysis is shown as Fig. 4-1.. Fig. 4-1: The flowchart of EEG data analysis in the first experiment.. For Experiment 2, we study the ERP and ERSP of drowsiness single-trial in different cognitive states. From design point of view, the drowsiness event is similar to the deviation event of above experiment because the stimulus of these two events is equal. The continuous EEG signals are separated into several epochs where an epoch contains the sampled EEG data from -500 ms to 3500 ms with deviation onset at 0 ms and the baseline region of each epoch is before the onset. The duration of drowsiness event is longer than the deviation event because the driver may need more reaction time while he/she is drowsy. Then we combine with all drowsiness events of recordings from different day and use ICA algorithm to decompose 30-channel EEG signals into the 30 independent components. The reaction time of 31.

(45) each event is recorded for the analysis of drowsiness in different cognitive states. The reaction time of each event is sorted in ascending order and the sorted trials are equally divided into five groups, where each group has 20 percentages in order. Obviously, the first one group indicates that the driver is more alert than other groups while driving. Therefore we can compare with the five conditions of different cognitive states corresponding to the ERSP of drowsiness related component. The detailed flowchart of analysis is shown as Fig. 4-2.. Fig. 4-2: The flowchart of analysis in Experiment 2.. 4.2 Analysis of Continuous EEG Data We attempt to apply the analysis from single-trial into continuous EEG signals for drowsiness for the last three experiments. By averaging accumulated single trials, the ERP analysis reduces noise and makes characteristic more visible in EEG signals. When dealing with the continuous EEG data, we must try to remove high-frequency noise by some. 32.

(46) technology and the simplest way we used is moving average filter. In Experiment 3 we propose an adaptive alertness estimation methodology based on EEG, time-frequency spectral analysis, Independent Component Analysis and FNN models for continuously monitoring driver’s drowsiness level with concurrent changes in the driving performance. Fig. 4-3 shows the flowchart of the proposed signal processing procedure.. Fig. 4-3: The flowchart of data processing procedure for the drowsy estimation system.. In this experiment, participants who demonstrated waves of drowsiness containing two or more micro-sleep in both sessions were selected for training and testing, respectively. In the training process, the 33-channel EEG signals are first applied to train the ICA model. By applying ICA algorithm to the EEG recordings from the scalp, we attempt to achieve the twin goals: removing artifacts and possible source separation based on stabilities of ICA spatial weighting matrix and temporal independence between artifacts and EEG signals. The effectiveness for removing eye blinking and other artifacts by using ICA had been 33.

(47) demonstrated in many studies [52-59]. Secondly, we use time-frequency spectral analysis to transfer all 33 ICA components into log subband power spectrum with time. Since the fluctuates of drowsiness level have cycle lengths longer than 4 minutes [27-28, 30], the spectral signals of 33 components and driving performance are smoothed by a causal 90-second square moving average filter advancing at 2-second steps to eliminate variance at cycle lengths shorter than 1~2 minutes. The correlation coefficients between the smoothed driving error and the subband power spectra of all ICA components at each frequency band form a correlation spectrum. The log subband power spectra of two ICA components with the highest correlation coefficient are further selected as features. Then we use the AFSM technology to select the log bandpower spectra of these two ICA components in some critical bands as the normalized input features to the linear regression or SONFIN model. Therefore the training data will establish the model to estimate the individual subject’s driving performance. The ICA weighting matrix, the EEG critical bands of the drowsy related source and the parameters of model in the training session were applied to estimate the individual subject’s driving performance in the testing session. Finally, we use correlation analysis between the estimated and actual driving performance to evaluate the performance of model. For comparing with the result by ICA algorithm, the 33-channel EEG signals are directly used for our procedure without using ICA decomposition. We also repeatedly test that finding the most appropriate frequency bands for the best estimating result to prove the dependability of the AFSM technology. Then the performance of estimating results will be discussed by using linear regression or SONFIN to establish model. Detailed analyses are described in the following sub sections. After we developed an adaptive drowsiness estimation system for driving, we find that this estimation system can get excellent results with only 2-channel EEG signals even with 34.

參考文獻

相關文件

 Promote project learning, mathematical modeling, and problem-based learning to strengthen the ability to integrate and apply knowledge and skills, and make. calculated

Step 4: : :模擬結果分析 : 模擬結果分析 模擬結果分析(脈寬為 模擬結果分析 脈寬為 脈寬為90%) 脈寬為.

This kind of algorithm has also been a powerful tool for solving many other optimization problems, including symmetric cone complementarity problems [15, 16, 20–22], symmetric

Step 3: : : :模擬環境設定 模擬環境設定 模擬環境設定 模擬環境設定、 、 、 、存檔與執行模擬 存檔與執行模擬

Our current solution, called “Iterative Earliest Deadline First+Best Fit” (IEDF+BF), includes three steps: First, estimate the number of high frequency cores; second, schedule

Each course at the Institute is assigned a number of units corresponding to the total number of hours per week devoted to that subject, including classwork, laboratory, and the

The closing inventory value calculated under the Absorption Costing method is higher than Marginal Costing, as fixed production costs are treated as product and costs will be carried

Microphone and 600 ohm line conduits shall be mechanically and electrically connected to receptacle boxes and electrically grounded to the audio system ground point.. Lines in