Future Work

Review the initial motivation of this study. Our goal is to construct a systematic model which links music perception and relating physiological responses. The model can be shown in Fig. 6.5. There are many features in music. Each physiological or psychological response (Result1, Result2, etc…) detected by all kinds of biosensors may be resulted from one main music feature or the combination of them (Feature1, Feature2, etc…). The final descriptive emotional or physical state may be identified by integrating these physiological or psychological responses (Result1, Result2, etc…).

Either music perception or physiological modulation is not straightforward. So this work starts from a simplified problem. The musical feature choused is rhythm. The physiological and psychological responses are observed by HRV. For completing the physiology based intelligent music playing system proposed in Fig. 1.2 further, some future works are suggested as follows:

Music Perception Analysis

Two musical rhythmic characteristics, tempo and complexity, are proposed to be two

main features in modulating the autonomic nervous system. The complexity is judged by each subject in this study. Some measures of complexity that corresponds to a high degree with a human’s subjective notion of complexity have been discussed [38-39]. The automatic algorithm for extracting the complexity of simple rhythm pattern (drum loops) should be developed in the future.

Drum loops are widely used in computer music composition and production as a means to generate high-quality music tracks in a quick and easy manner. Most pop music use the drum loop music as the background rhythm base. So the drum loop extraction algorithm is helpful for the automatic music analysis system [43-44]. These algorithms will be integrated to the system for fully automatic musical rhythm analysis in the future.

Bio-signals analysis

Besides the time and frequency domain analysis method, the nonlinear method is also important in biomedical signal processing. Nonlinear phenomena are certainly involved in the genesis of HRV. They are determined by complex interactions of haemodynamic, electrophysiological and humoral variables, as well as by autonomic and central nervous regulations. Therefore, the nonlinear method may be another suitable observation window to explore the physiological modulation induced by music perception [45-46].

For more comprehensive physiological signal analysis, more biomedical signal (pulse, photoplethysmograph abbreviated as PPG, etc…) will be captured for cross analysis in the future. On the other hand, a non-contact optical measurement system for acquiring the HRV signal is under developing [47-48]. The HRV signal will be easier to be captured and the physiology based intelligent music playing system is more portable in the future.

Emotion recognized by physiological responses

As describing in the section 1.2, the physiology based intelligent music playing system chooses suitable music for user to make them achieve desired physiological or emotional state (ex. powerful, active, calm, etc…). I think emotion recognition by physiological responses is important and interesting [49-50]. If the music induced emotion can be measured and recognized correctly by the physiological signals, the emotional responses for various music

can be recorded continuously when user is listening. It is helpful for the system to understand the individual preference or emotion response for the specific music type by long term machine learning. The experience of music listening will be improved through the interactive system. So the emotion recognition should be integrated into the system in the future.

Bibliography

[1] Hon EH; Lee ST, “Electronic Evaluations of the Fetal Heart Rate Patterns Preceding Fetal Death: Further Observations,” Am J Obstet Gynecol, vol. 87, no. 6, pp. 814-826, 1965.

[2] Akselrod S, Gordon D, Ubel FA, Shannon DC, Barger CA, Cohen RJ, “Power Spectrum Analysis of Heart Rate Fluctuation: A Quantitative Probe of Beat-to-Beat Cardiovascular Control,” Science, vol. 213, no. 4504, pp. 220-222, 1981.

[3] Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology, “Heart Rate Variability: Standards of Measurement, Physiological Interpretation, and Clinical Use,” Circulation, vol. 93, no. 5, pp. 1043-1065, 1996.

[4] P. T. BASON; B. G. CELLER, “Control of the Heart Rate by External Stimuli,” Nature, vol. 238, pp. 279-280, Aug. 1972.

[5] Levy; Matthew N., “Sympathetic-Parasympathetic Interactions in the Heart,” Circ. Res., vol. 29, no. 437, 1971.

[6] Leon Glass, “Synchronization and Rhythmic Processes in Physiology,” Nature, vol. 410, pp. 277-284, March 2001.

[7] M Clayton; R Sager; U Will, “In Time with the Music: The Concept of Entrainment and its Significance for Ethnomusicology,” ESEM CounterPoint, vol. 1, 2004.

[8] Cook Perry, Music, Cognition, and Computerized Sound, Cambridge: The MIT Press, 2001.

[9] L Bernardi; C Porta1; P Sleight, “Cardiovascular, Cerebrovascular, and Respiratory Changes Induced by Different Types of Music in Musicians and Non-Musicians: the Importance of Silence,” Heart, vol. 92, pp. 445-452, 2006.

[10] Johnson JE, “The Use of Music to Promote Sleep in Older Women,” Journal of Community Health Nursing, vol. 20, no. 1, pp. 27-35, 2003.

[11] Cooke, Marie; Chaboyer, Wendy; Schluter, Philip; Hiratos, Maryanne, “The Effect of Music on Preoperative Anxiety in Day Surgery,” Journal of Advanced Nursing, vol. 52, no. 1, pp. 47-55(9), October 2005.

[12] Gertjan Wijnalda; Steffen Pauws; Fabio Vignoli; Heiner Stuckenschmidt, “A Personalized Music System for Motivation in Sport Performance,” IEEE Pervasive Computing, vol. 4, no. 3, pp. 26-32, July-September 2005.

[13] Christopher F. Chabris, “Prelude or Requiem for the 'Mozart Effect'?,” Nature, vol. 400, pp. 826-827, August 1999.

[14] N., Matthew. Vagal Control of the Heart: Experimental Basis and Clinical Implications.

City: Futura Publishing Company, 1994.

[15] G. Berntson; J. T. Bigger; Jr.; D. Eckberg; P. Grossman; P. Kaufmann; M.Malik; H.

Nagaraja; S. Porges; J. Saul; P. Stone; W. V. D.Molen, “Heart Rate Variability: Origins, Methods and Interpretive Caveats,” Psychophysiology, vol. 34, pp. 623–648, 1997.

[16] I A O'Brien; P O'Hare; R J Corrall, “Heart Rate Variability in Healthy Subjects: Effect of age and the derivation of normal ranges for tests of autonomic function,” British Heart Journal, vol. 55, pp. 348-354, 1986.

[17] Carter, James B.; Banister, Eric W.; Blaber, Andrew P, “Effect of Endurance Exercise on Autonomic Control of Heart Rate,” Sports Medicine, vol. 33, no.1, pp. 33-46, 2003.

[18] P Ekman; RW Levenson; WV Friesen, “Autonomic Nervous System Activity Distinguishes among Emotions,” Science, vol. 221. no. 4616, pp. 1208-1210, Sep. 1983.

[19] Pagani M; Mazzuero G; Ferrari A; Liberati D; Cerutti S; Vaitl D; Tavazzi L; Malliani A,

“Sympathovagal Interaction during Mental Stress: A Study using Spectral Analysis of Heart Rate Variability in Healthy Control Subjects and Patients with a Prior Myocardial Infarction,”

Circulation, vol.83, pp. 43-51, 1991.

[20] Rollin McCraty, “Influence of Cardiac Afferent Input on Heart-Brain Synchronization and Cognitive Performance,” Journal of Psychophysiology, vol. 45(1-2), pp. 72-73, 2002.

[21] Pinna GD; Maestri R; Di Cesare A; Colombo R; Minuco G, “The accuracy of power spectrum analysis of heart-rate variability from annotated RR list generated by Holter systems,” Physiol. Meas., vol. 15, pp. 163-179, 1994.

[22] Kohler, B.-U.; Hennig, C.; Orglmeister, R., “The Principles of Software QRS Detection,”

Engineering in Medicine and Biology Magazine, IEEE, vol. 21, no. 1, 2002.

[23] Arzeno, N.M.; Chi-Sang Poon; Zhi-De Deng, “Quantitative Analysis of QRS Detection Algorithms Based on the First Derivative of the ECG,” Engineering in Medicine and Biology Society 28th Annual International Conference of the IEEE, pp. 1788-1791, Aug. 2006.

[24] Jiapu Pan; Willis J. Tompkins, “A Real-Time QRS Detection Algorithm,” Biomedical Engineering, IEEE Transactions on , vol. BME-32, no. 3, pp. 230-236, 1985.

[25] Thakor, Nitish V.; Webster, John G.; Tompkins, Willis J., “Estimation of QRS Complex Power Spectra for Design of a QRS Filter,” Biomedical Engineering, IEEE Transactions on, vol. BME-31, no. 11, pp. 702-706, Nov. 1984.

[26] http://www.physionet.org/

[27] Mateo, J.; Laguna, P., “Analysis of Heart Rate Variability in the Presence of Ectopic Beats using the Heart Timing Signal,” Biomedical Engineering, IEEE Transactions on, vol.50, no.3, pp. 334-343, Mar 2003.

[28] Carvalho, J.L.H.; Rocha, A.F.; dos Santos, I.; Itiki, C.; Junqueira, L.F., Jr.; Nascimento, F.A.O., "Study on the Optimal Order for the Auto-Regressive Time-Frequency Analysis of Heart Rate Variability," Engineering in Medicine and Biology Society, 2003. Proceedings of the 25th Annual International Conference of the IEEE, vol. 3, pp. 2621- 2624, Sept. 2003.

[29] Tarvainen, M.P.; Ranta-aho, P.O.; Karjalainen, P.A., "An Advanced Detrending Method with Application to HRV Analysis," Biomedical Engineering, IEEE Transactions on, vol. 49, iss. 2, pp. 172-175, Feb 2002.

[30] McNames, J.; Thong, T.; Goldstein, B, "Reliability and Accuracy of Heart Rate Variability Metrics versus ECG Segment Duration," Proceedings of the 25th Annual International Conference of the IEEE, vol. 1, pp. 212-215. Sept. 2003.

[31] Kay SM; Marple; SL., “Spectrum Analysis: A Modern Perspective,” Proc IEEE, vol. 69, pp. 1380-1419, 1981.

[32] Welch, P., “The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method based on Time Averaging over Short, Modified Periodograms,” Audio and Electroacoustics, IEEE Transactions on, vol.15, no.2, pp. 70-73, Jun 1967.

[33] Pomeranz M, Macaulay RJB, Caudill MA., “Assessment of Autonomic Function in Humans by Heart Rate Spectral Analysis,” Am J Physiol, vol.248, pp. H151-153, 1985.

[34] Pagani M; Lombardi F, Guzzetti S et al, “Power Spectral Analysis of Heart Rate and Arterial Pressure Variabilities as a Marker of Sympatho-Vagal Interaction in Man and Conscious Dog,” Circ Res, vol. 59, pp. 178-193, 1986.

[35] Fabien Gouyon; Simon Dixon, “A Review of Automatic Rhythm Description Systems,”

Computer Music Journal, vol. 29, iss. 1, pp. 34-54, February 2005.

[36] Bello, J.P.; Daudet, L.; Abdallah, S.; Duxbury, C.; Davies, M.; Sandler, M.B., “A Tutorial on Onset Detection in Music Signals,” Speech and Audio Processing, IEEE Transactions on , vol.13, no.5, pp. 1035-1047, Sept. 2005.

[37] Simon Dixon, “Automatic Extraction of Tempo and Beat from Expressive Performances,” Journal of New Music Research, vol. 30, no. 1, pp. 39-58, March 2001.

[38] Ilya Shmulevich; Olli Yli-Harja; Edward Coyle; Dirk-Jan Povel; Kjell Lemstrom,

“Perceptual Issues in Music Pattern Recognition: Complexity of Rhythm and Key Finding,”

Computer and Humanities, vol. 35, no. 1, pp. 23-35, Feb. 2001.

[39] Streich Sebastian, “Music Complexity: A Multi-Faceted Description of Audio Content,”

Pompeu Fabra, Barcelona, Dissertation, 2006.

[40] W. Massagram; O. Boric-Lubecke; L. Macchiarulo; Mingqi Chen, “Heart Rate Variability Monitoring and Assessment System on Chip,” Engineering in Medicine and Biology 27th Annual International Conference, 2005.

[41] Meng-chou Chang; Zong-xin Lin; Che-wei Chang; Hsiao-lung Chan; Wu-shiung Feng,

“Design of a system-on-chip for ECG signal processing,” Circuits and Systems, The 2004 IEEE Asia-Pacific Conference on, vol.1, no., pp. 441-444, Dec. 2004.

[42] Luk, W.; Jones, G., “Systolic Arrays for Recursive Digital Filtering,” Electronics Letters, vol. 23, no. 22, pp.1174-1175, October 22 1987.

[43] Gillet, O.; Richard, G., “Extraction and Remixing of Drum Tracks from Polyphonic Music Signals,” Applications of Signal Processing to Audio and Acoustics, 2005. IEEE Workshop on, pp. 315-318, Oct. 2005.

[44] Kazuyoshi Yoshii; Masataka Goto; Hiroshi G. Okuno, “Drum Sound Recognition for Polyphonic Audio Signals by Adaptation and Matching of Spectrogram Templates With Harmonic Structure Suppression," Audio, Speech, and Language Processing, IEEE Transactions on [see also Speech and Audio Processing, IEEE Transactions on], vol. 15, no.

1, pp. 333-345, Jan. 2007.

[45] Plamen Ch. Ivanov; Luís A. Nunes Amaral; Ary L. Goldberger; Shlomo Havlin; Michael G. Rosenblum; Zbigniew R. Struzik; H. Eugene Stanley, “Multifractality in Human Heartbeat Dynamics,” Nature, vol. 399, pp. 461-465, Jun. 1999.

[46] Maestri, Roberto; Gian Domenico Pinna,; Porta, Alberto; Balocchi, Rita; Sassi, Roberto; Maria Gabriella Signorini; Dudziak, Maria; Raczak, Grzegorz, “Assessing Nonlinear Properties of Heart Rate Variability from Short-Term Recordings: Are these Measurements Reliable?,” Physiological Measurement, vol. 28, no. 9, pp. 1067-1077(11), Sep. 2007.

[47] John Allen, “Photoplethysmography and its Application in Clinical Physiological Measurement.” Physiological Measurement, vol. 28, no. 3, pp. R1-R39 (1), Mar. 2007.

[48] M.J. Drinnan; J. Allen; A. Murray, “Relation between Heart Rate and Pulse Transit Time during Paced Respiration,” Physiological Measurement, vol. 22, no. 3, pp. 425-432(8), 2001.

[49] K. H. Kim; S. W. Bang; S. R. Kim, “Emotion Recognition System using Short-Term Monitoring of Physiological Signals,” Medical and Biological Engineering and Computing, vol. 43, no. 3, pp. 419-427, May 2004.

[50] Christine Latitia Lisetti; Fatma Nasoz, “Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals,” EURASIP Journal on Applied Signal Processing, vol. 2004, no. 11, pp. 1672-1687, April 2004.

Appendix A

Most PhysioBank databases include one or more sets of annotations for each recording.

Annotations are labels that point to specific locations within a recording and describe events at those locations. For example, many of the recordings that contain ECG signals have annotations that indicate the times of occurrence and types of each individual heart beat ("beat-by-beat annotations"). The standard set of annotation codes was originally defined for ECGs, and includes both beat annotations and non-beat annotations. Most PhysioBank databases use these codes as described in Table A.1.

Table A.1: PhysioBank Annotations

Beat annotations Non-beat annotations

N : Normal beat [CS3- : Isolated QRS-like artifact L : Left bundle branch block beat ! : Ventricular flutter wave

R : Right bundle branch block beat ] : End of ventricular flutter/fibrillation B : Bundle branch block beat x : Non-conducted P-wave

A : Atrial premature beat ( : Waveform onset a : Aberrated atrial premature beat ) : Waveform end J : Nodal (junctional) premature beat p : Peak of P-wave S : Supraventricular premature or ectopic beat t : Peak of T-wave V : Premature ventricular beat u : Peak of U-wave r : R-on-T premature ventricular contraction ` : PQ junction F : Fusion of ventricular and normal beat ' : J-point

e : Atrial escape beat ^ : (Non-captured) pacemaker artifact j : Nodal (junctional) escape beat | : Isolated QRS-like artifact

n : Supraventricular escape beat (atrial or nodal) ~ : Change in signal quality E : Ventricular escape beat +: Rhythm change

/ : Paced beat s : ST segment change

f : Fusion of paced and normal beat T : T-wave change

Q : Unclassifiable beat * : Systole

? : Beat not classified during learning D : Diastole

In this study, all the forty-eight recordings in the MIT-BIH Arrhythmia Database are used to evaluate the QRS detector algorithm. Each recording records half-hour annotated ECG, but just first ten minutes data are used to evaluate the QRS detector performance for simplicity.

The evaluation result of each recording is listed in Table A.2. The column named Record lists all of the recording names in the MIT-BIH Arrhythmia Database. The column named Total annotated lists the number of annotated beats in each recording. The column named Total beat-annotated lists the number of beats coded by beat-annotations in each recording. The column TP, FP and FN mean true positive, false positive and false negative.

Table A.2: The evaluation results of the simplified QRS detector

Record Total Peaks Total Normal TP FP FN

116 796 796 792 0 4

234 924 920 910 0 10

Sum 33339 216 1517

Appendix B

A simple method for removing or compensating these abnormal beats is utilized and the detailed algorithm is formulated as follows:

( ) ( )( )( )

At some peak time instant t criterion is not met

t t t t is a false positive peak remove it elseif f t t t

if f t t t

t is a false positive peak remove it

elseif f t t t and f t t t

is a false positive peak remove it end

t t is a false negative peak interpolate it t t

elseif f t t

t t is a false negative peak interpolate it end

( )

Appendix C

The deviation of each processing stage between the software QRS detector and the hardware QRS detector is simulated through all the MIT-BIH Arrhythmia Database, but just first ten minutes data are used to evaluate the QRS detector performance for simplicity. The deviation is represented by averaging the differences of each stage outcomes between the hardware and software methods.

Table C.1: The detailed deviation between the hardware and software QRS detector of each record

119 0 0.015119 0.002898 0.0374 0.006469 0.00304