A vision-based analysis system for gait recognition in patients with Parkinson's disease

(1)

A vision-based analysis system for gait recognition in patients

with Parkinson’s disease

Chien-Wen Cho

a

, Wen-Hung Chao

a,b

, Sheng-Huang Lin

c

, You-Yin Chen

a,*

a

Department of Electrical and Control Engineering, National Chiao Tung University, No. 1001, Ta-Hsueh Road, Hsinchu City 300, Taiwan, ROC b_{Department of Biomedical Engineering, Yuanpei University, Taiwan, ROC}

c_{Department of Neurology, Tzu Chi General Hospital, Tzu Chi University, Taiwan}

a r t i c l e

i n f o

Keywords: Parkinson’s disease Gait analysis

Linear discriminant analysis (LDA) Principal component analysis (PCA) Vision-based

a b s t r a c t

Recognition of specific Parkinsonian gait patterns is helpful in the diagnosis of Parkinson’s disease (PD). However, there are few computer-aided methods to identify the specific gait patterns of PD. We propose a vision-based diagnostic system to aid in recognition of the gait patterns of Parkinson’s disease. The pro-posed system utilizes an algorithm combining principal component analysis (PCA) with linear discrimi-nant analysis (LDA). This scheme not only addresses the high data dimensionality problem during image processing but also distinguishes different gait categories simultaneously. The feasibility of the proposed system for the recognition of PD gait was tested by using gait videos of PD and normal subjects. The effi-ciency of feature extraction using PCA and LDA coefficients are also compared. Experimental results showed that LDA had a recognition rate for Parkinsonian gait of 95.49%, which is higher than the conven-tional PCA feature extraction method. The proposed system is a promising aid in identifying the gait of Parkinson’s disease patients and can discriminate the gait patterns of PD patients and normal people with a very high classification rate.

1. Introduction

Most of the current methods used for evaluating Parkinson’s disease (PD) rely heavily on human expertise, e.g., the use of the uniﬁed Parkinson disease rating scale (UPDRS) (_{Martı´n et al.,} 2004). UPDRS is a rating tool that follows the longitudinal course of PD. It is composed of 5 separate categories including mentation, behavior, mood, activities of daily living and motor examinations, all evaluated by interview. Some sections require multiple grades assigned to each extremity.

The analysis of gait characteristics, as documented by Knutsson (1972), shows that PD patients exhibit large gait variability. Compared with normal people, PD patients’ walking speed is slower, duration of gait cycle is longer, stride length is shorter, and amplitude of range of movement of joints is decreased. These speciﬁc gait patterns (e.g. festinating gait, freezing gait) are widely accepted as a prominent feature of PD (McDowell, 1971). However, since posture and gait movement can vary from person to person, the evaluation of Parkinsonian gait tends to be subjective and de-pends greatly on the experience and judgment of the clinician (Blin, Ferrandez, & Serratrice, 1990; Lubik et al., 2006; Melnick,

Radtka, & Piper, 2002; Shan et al., 2001; Salarian et al., 2004; Sof-uwa et al., 2005; Stern et al., 1983; Vokaer, Azar, & Beyl, 2003).

In addition to evidence-based practice, therapists also use objective, quantitative methods to improve diagnosis of Parkinso-nian gait. As a result, engineering-oriented machine learning-based methods have attracted more and more attention in this ﬁeld (Engin et al., 2007; Fahrenberg et al., 1997; Makikawa & Iizumi, 1995; Sekine et al., 2002; Veltink et al., 1996). Many previ-ous studies have used dc and ac accelerometers to assess gait pat-terns. They classiﬁed the accelerometer signals into different types of walking and correlated them with energy consumption. Never-theless, those methods often used a number of sensors, causing patient discomfort.

Vision-based gait analysis systems avoid this problem. Since these systems require no physical contact, they are more comfort-able and acceptcomfort-able to the patients. Vision-based gait analysis is di-vided into two main categories, model-based and holistic. Model based approaches ﬁt their model to the image data (Cunado, Nixon, & Carter, 1999; Yam, Nixon, & Carter, 2002). These image process-ing systems use markers on the body and record several steps of the patient. An average of three or more walks is then computed. The temporal characteristics of gait, e.g., stride length, width, cadence and velocity are measured (Melnick et al., 2002). In one study (Cunado et al., 1999) the gait signature was extracted using a Fourier series to describe the motion of the leg and temporally

* Corresponding author. Tel.: +886 3 571 2121x54427; fax: +886 3 612 5059. E-mail address:[email protected](Y.-Y. Chen).

Contents lists available atScienceDirect

Expert Systems with Applications

(2)

Many types of optimization criteria can be used to determine an appropriate W, such as maximizing the variance, non-Gaussianity for independancy, negentropy, or the ratio of between- and with-in-class variations (Hyvarinen et al., 2001;Liu & Wechsler, 1998; Murase & Sakai, 1996). Among them, principal component analysis (PCA) is well-known and widely used (Polat & Günesß, 2007). PCA focuses on computing eigenvectors that account for the largest var-iance of the data selected, but these directions do not necessarily provide the best separation of gait classes. On the other hand, the ratio of between- and within-class variations (Fisher’s linear discriminant criterion) appears to be an especially valid index since it allows simultaneous balancing between the maximization and minimization of the between- and within-class variations. Based on Fisher’s linear discriminant criterion, linear discriminant analy-sis (LDA) then produces a linear projection matrix, which greatly enhances classiﬁcation.

The aim of this paper is to discriminate PD patients from normal subjects using a vision-based gait analysis approach. The scheme utilizes the holistic image of subjects, and extracts and reduces the feature space by using PCA and LDA. The meaning of the ob-tained LDA transformation matrix (reduced to a vector in our case) is not only treated as a black box but is also used to describe the posture information of PD patients in a numerical way.

2. Mathematical background used for signal processing 2.1. Principal component analysis (PCA)

PCA is a classic technique used in statistical data analysis, featuring extraction and data compression (Jolliffe, 2002). It is useful in reducing the dimensionality of an input data space by transforming the data from a correlated high-dimensional space to an uncorrelated low-dimensional space. We brieﬂy describe PCA as follows. Suppose that there are NTvectors being grouped

into c classes. We can express these vectors as x11; . . . ;x1N1; . . . ; xi1; . . . ;xiNi; . . . ;xc1; . . . ;xcNc, where xij is the jth vector of the ith class and Niis the number of vectors in the ith class. To proceed,

the mean mxof the entire set of vectors is given by

mx¼ 1 NT Xc i¼1 XNi j¼1 xij: ð1Þ

To compute the covariance matrix C, we have

C ¼ 1 NT Xc i¼1 XNi j¼1 ðxij mxÞðxij mxÞT: ð2Þ

If the rank of C is K, the eigenvalues of C,k1,k2,. . .,kK, and the

asso-ciated eigenvectors p1,. . .,pKcan be computed accordingly. Suppose,

without loss of generality, jk1j P jk2j P P jkKj. A partial

eigen-Hence, the mean vector of the entire setU= {U1,U2,. . .,Uc} is given

by my¼ 1 NT Xc i¼1 Nimyi: ð5Þ

The within-class matrix Swand the between-class matrix Sbof the

entire set of vectors can be calculated by

Sw¼ 1 NT Xc i¼1 XNi j¼1 ðxij myiÞðxij myiÞ T ð6Þ and Sb¼ 1 NT Xc i¼1 Niðmi myÞðmi myÞ T : ð7Þ

Maximizing the between-class variance and minimizing the within-class variance simultaneously is equivalent to maximizing

JðWÞ ¼ TracefðWT_S

wWÞ1ðWTSbWÞg: ð8Þ

By solving the eigenvectors of the matrix S1wSb, we can obtain c 1

eigenvectors w1, . . ., wc 1to span a canonical space. Thus, we can

further project a vector yijon the partial eigenspace to a vector zij

on the canonical space by zij¼½w1; . . . ;wc1Tyij

¼Wyij; ð9Þ

where zijis also named the LDA coefﬁcients in this paper.

Consequently, the mean vector of the ith class on the canonical space can be computed by

mzi¼ 1 Ni XNi j¼1 zij: ð10Þ

2.3. Minimum distance classiﬁer (MDC) (Duda, Hart, & Stork, 2000) In this study, MDC is used to classify an input vector u to the ith class whose centroid miminimizes the Euclidian distance from u

(using either PCA or LDA coefﬁcients). The minimum distance clas-siﬁer can be expressed as

i ¼ arg miniðu miÞTðu miÞ: ð11Þ

3. System used for detection of PD gait patterns

As shown inFig. 1, we propose a gait analysis system which can detect the gait pattern of Parkinson’s disease using computer vision. This system comprises three main parts: (1) preprocessing,

(3)

(2) training and (3) recognition. In this study, we first captured sev-eral videos of both normal subjects and patients with PD. We then processed the images from the videos to characterize the subjects. All subjects were encoded as vectors such that we could use PCA and LDA to extract features. An MDC was then used as the classi-fier. The flow chart of the proposed system is shown inFig. 1. 3.1. Environment set-up, image acquisition and subject detection

A structured environment—a corridor with a deep blue cur-tain—was utilized and the room was well illuminated as shown inFig. 2. Image acquisition equipment, including a CCD camera connected to a PC or a handy DV, was also used.

There are many background models proposed (Haritaoglu, Har-wood, & Davis, 2000), to help extract clear foreground objects. For simplicity, the background model was constructed by taking a pho-tograph of the environment beforehand. Both normal subjects and

PD patients were asked to wear light-colored clothing to achieve high color contrast between the background and their proﬁles. Each subject was instructed to walk from the left to the right end (and then walk back if needed). Afterwards, the subjects’ image se-quences were captured, as illustrated inFig. 3.

The difference between the background and each of the image frames was then computed. The absolute value of the difference was then calculated such that every pixel of the input image was judged to belong to the foreground object pixel if the correspond-ing absolute value exceeded a threshold. This threshold depended on the color contrast of the curtain and the clothing of the subjects and the illumination condition. As a result, we binarized each of the images during walking (seeFig. 4a).

To obtain a more compact silhouette size, we projected the bin-ary image to the vertical axis. The histogram of the projection is shown inFig. 4(b). From this plot, it is clear that the upper and low-er bounds of the silhouette can be computed by using a threshold

Fig. 1. Flow chart of the proposed system.

(4)

of 2% of the maximum projection amount. Similarly, the left and right boundaries of the silhouette were also obtained by projecting the binarized image to the horizontal axis. Finally, we show two examples of the sequences of the binarized and truncated images of the normal subjects and PD patients inFig. 5a and b, respec-tively. It is noted that the resultant images were normalized to the size of 64 64 pixels to further reduce the computational costs.

3.2. Training and testing

According to the view point of Machine Learning (Alpaydin,

2004), when designing a classifier, we should use training data, which are independent of test data used in the test phase. Thus, we divided the videos of the subjects into two groups: the first group was used to train the system and the remainder was used to evaluate the classification performance of the system.

3.3. Feature extraction

For comparison purposes, two kinds of feature extraction meth-ods, PCA and LDA, were used in this study, as described below. 3.3.1. PCA coefﬁcient extraction

The silhouette of a subject in an image frame was originally rep-resented as a binary matrix. Since this matrix, in general, contains

Although PCA can reduce the dimensionality of silhouette vec-tors, it is not used aiming for classiﬁcation. As a result, further pro-cessing is needed to improve the recognition capability of the proposed system. In this study, we focus on two categories of sub-jects: normal people and patients with Parkinson’s disease. The means of the silhouette vectors of each category and of the entire set of vectors were computed by(4) and (5), respectively. After evaluating the LDA transformation matrix, each silhouette vector in the partial eigenspace was mapped to a new vector on the canonical space by(9).

4. Experimental results and discussion

Seven PD patients and seven normal people from Buddhist Tzu Chi General Hospital in Taiwan were enrolled in this study. All the experiments were conducted in the laboratory of the neurosurgery department of the hospital. Under supervision, the subjects were asked to walk from left to right and then to walk back. A SONY HDR-HC3 camcorder was utilized to capture the motion video of the subjects. All video recordings were then extracted to image clips with a sampling rate of 15 frames/s. Because the subjects walked at different speeds, the lengths of the video sequences var-ied from person to person (seeTable 1).

Pixel candidates of the silhouettes of the subjects were labeled to construct binary images in order to ensure that the absolute val-ues of the difference valval-ues were larger than an intensity threshold (10, in the experiments). The binarized silhouettes were then ob-tained by truncating the input image frames and the shapes were normalized to the size of 64 64 matrix. After encoding the de-tected silhouette images to image vectors, we obtained 3551 vectors.

The mean of the silhouette vectors was computed and the covariance matrix was then calculated. Accordingly, the

eigen-Fig. 4. Subject detection. (a) The detection of a subjection using image differences. The histogram of the detected subject on the (b) vertical and (c) horizontal axes.

Fig. 5. Two examples of detected subjects: (a) a normal person and (b) a patient of Parkinson’s disease.

Table 1

The recorded video sequences

Subject number Status Length (min) Sampling rate (frame/s)

1 Normal 3 15 2 Normal 2 15 3 Normal 2 15 4 Normal 3 15 5 Normal 3 15 6 Normal 3 15 7 Normal 3 15 8 PD 3 15 9 PD 2 15 10 PD 2 15 11 PD 3 15 12 PD 3 15 13 PD 2 15 14 PD 2 15

(5)

values and associated eigenvectors of the covariance matrix were

computed.Figs. 6 and 7show the sorted magnitudes and

accumu-lated variance of sorted eigenvalues, respectively. Among the eigenvectors calculated, the ﬁrst 280 eigenvectors corresponding to the largest 280 eigenvalues (accumulating 90% of the total var-iance) were selected as the bases of the partial eigenspace. The im-age vectors were then projected onto the obtained partial eigenspace to extract the PCA coefﬁcients.

To further discriminate among each class of silhouette vector, the obtained PCA coefﬁcients of the silhouette vectors were pro-cessed using LDA. First, the mean vectors of each class and of the entire set of the vectors on partial eigenspace were computed. Then, maximization of the ratio of between-class variance and within-class variance was carried out. Accordingly, the ratios be-fore and after LDA were 0.0446 and 16.0000, respectively. It is clear that LDA raised the ratio. Finally, the LDA coefﬁcients of the silhou-ette vectors were calculated.

To visualize the different features obtained by PCA and LDA, the distributions of the first two components (for illustration pur-poses) of PCA and only one LDA coefficient of silhouette vectors of normal and PD subjects are plotted inFigs. 8 and 9, respectively. In the figures, the red ‘‘O” stands for the normal subjects and the green ‘‘M” for PD patients.1

We observed that the PCA coefficients of different groups of sil-houette vectors had large overlaps for both vertical and horizontal axes. On the other hand, the LDA coefficients of different groups of silhouette classes clearly separated from one another. Note that, although we illustrated the scatter plots by using only two and one coefficients for PCA and LDA, respectively, we adopted 280 and one coefficients for PCA and LDA, respectively, during the clas-sification of normal people and PD patients. Since the LDA coeffi-cients’ dimensionality is only one, the counting index is then used as the horizontal axis ofFig. 9to show the scatter plot.

For further insight, we investigated the LDA projection matrix carefully (in this study, it reduced to a vector). We computed the absolute values of this vector and shifted and scaled the obtained projection vector elements to fall in the range from 0 to 255. We reshaped this vector to match the shape of captured images. There-fore, we could visualize the projection vector and correlate it to the silhouettes of subjects. We show the image form of the projection vector inFig. 10. The image pixels with higher intensity inFig. 10

had more discriminating ability according to LDA projection. The head and neck (a) were especially emphasized by LDA. The system_{was able to discern PD patients by ‘‘carefully observing” the head} and neck parts of the silhouettes of the subjects.

MDC was adopted in this study to evaluate the classiﬁcation performance of the proposed scheme. During the training phase,

Fig. 6. The sorted eigenvalue diagram. Fig. 7. The accumulated variance diagram.

Fig. 8. The distribution of the ﬁrst two components of PCA coefﬁcients of the training silhouette vectors.

Fig. 9. The distribution of the LDA coefﬁcients of the training vectors.

1

For interpretation of color in Figs. 8 and 9, the reader is referred to the web version of this article.

(6)

both PCA and LDA had classiﬁcation rates of 100%.Table 2 com-pares the test performance of PCA–MDC with that of LDA–MDC for 1529 test cases. It is obvious that LDA–MDC outperformed

PCA–MDC by 18.31%.Tables 3 and 4further demonstrate the

con-fusion matrix: LDA–MDC improved the detection rate for both nor-mal and PD subjects. PCA, functioning mainly as a preprocessing of LDA–MDC for data dimensionality reduction, did not separate dif-ferent groups of silhouette vectors well. As illustrated inFig. 8, the silhouette vectors of PCA coefﬁcients of the two groups were still close to each another. Although these components corresponded to the eigenvectors with maximum variations of the original vec-tors, the directions of these eigenvectors were not consistent with the directions that best discriminate between normal subjects and PD patients.

5. Conclusions

The diagnosis of PD is an important issue in the neuroscience ﬁeld. Although gait analysis is important in the diagnosis of PD,

in Taiwan. References

Alpaydin, E. (2004). Introduction to machine learning. Cambridge, MA: MIT Press. Blin, O., Ferrandez, A. M., & Serratrice, G. (1990). Quantitative analysis of gait in

Parkinson patients: Increased variability of stride length. Journal of the Neurological Sciences, 98(1), 91–97.

Cunado, D., Nixon, M. S., & Carter, J. N. (1999). Automatic gait recognition via model-based evidence gathering. In Proceedings of IEEE workshop on automated identiﬁcation advanced technologies (pp. 27–30).

Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern classiﬁcation (2nd ed.). New York: John Wiley & Sons.

Engin, M. et al. (2007). The classiﬁcation of human tremor signals using artiﬁcial neural network. Expert Systems with Applications, 33(3), 754–761.

Fahrenberg, J. et al. (1997). Assessment of posture and motion by multichannel piezoresistive accelerometer recordings. Psychophysiology, 34(5), 607–612. Haritaoglu, I., Harwood, D., & Davis, L. S. (2000). W4: Real-time surveillance of

people and their activities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), 809–830.

Huang, P. S., Harris, C. J., & Nixon, M. S. (1999a). Recognizing humans by gait via parametric canonical space. Artiﬁcial Intelligence in Engineering, 13(4), 359–366. Huang, P. S., Harris, C. J., & Nixon, M. S. (1999b). Human gait recognition in canonical space using temporal templates. Proceedings of IEEE Vision, Image, and Signal Processing, 146(2), 93–100.

Hyvarinen, A. et al. (2001). Independent component analysis. New York: Wiley. Jolliffe, I. T. (2002). Principal component analysis (2nd ed.). New York: Springer. Knutsson, E. (1972). An analysis of Parkinsonian gait. Brain, 95(3), 475–486. Little, J. J., & Boyd, J. E. (1998). Recognizing people by their gait: The shape of

motion. Videre: International Journal of Computer Vision, 1(2), 83–105. Liu, C., & Wechsler, H. (1998). Enhanced Fisher linear discriminant model for face

recognition. In Proceedings of the 14th international conference on pattern recognition (pp. 1368–1372).

Lubik, S. et al. (2006). Gait analysis in patients with advanced Parkinson disease: Different or additive effects on gait induced by levodopa and chronic STN stimulation. Journal of Neural Transmission, 113(2), 163–173.

Makikawa, M. & Iizumi, H. (1995). Development of an ambulatory physical activity memory device and its application for the categorization of actions in daily life. In Proceedings of 8th conference on medical informatics (pp. 747–750). Martı´n, P. M. et al. (2004). Uniﬁed Parkinson’s disease rating scale characteristics

and structure. Movement Disorders, 9(1), 76–83.

McDowell, F. H. (1971). The diagnosis of Parkinsonism or Parkinson syndrome. Contemporary Neurology Series, 8, 163–174.

Melnick, M. E., Radtka, S., & Piper, M. (2002). Gait analysis and Parkinson’s disease. Rehab Management, 15(6), 46–48.

Murase, H., & Sakai, R. (1996). Moving object recognition in eigenspace representation: Gait analysis and lip reading. Pattern Recognition Letter, 17(2), 155–162.

Polat, K., & Günesß, S. (2007). Automatic determination of diseases related to lymph system from lymphography data using principles component analysis (PCA), fuzzy weighting pre-processing and ANFIS. Expert Systems with Applications, 33(3), 636–641.

Shan, D. E. et al. (2001). Gait analysis in advanced Parkinson’s disease – effect of levodopa and tolcapone. Journal of the Neurological Sciences, 28(1), 70–75. Salarian, A. et al. (2004). Gait assessment in Parkinson’s disease: Toward an

ambulatory system for long-term monitoring. IEEE Transactions on Bio-medical Engineering, 51(8), 1434–1443.

Sekine, M. et al. (2002). Discrimination walking of patterns using wavelet-based fractal analysis. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 10(3), 188–196.

Sofuwa, O. et al. (2005). Quantitative gait analysis in Parkinson’s disease: Comparison with a healthy control group. Archives of Physical Medicine and Rehabilitation, 86(5), 1007–1013.

Table 2

Classiﬁcation accuracy comparison between PCA–MDC and LDA–MDC

Algorithm Accuracy (%)

PCA–MDC 77.1746

LDA–MDC 95.4872

Table 3

Confusion matrix of PCA-MDC

Normal PD

Normal 518 109

PD 240 662

Table 4

Confusion matrix of LDA-MDC

Normal PD

Normal 599 28

PD 41 861

(7)

Stern, G. M. et al. (1983). Quantitative assessments of gait and mobility in Parkinson’s disease. Journal of Neural Transmission Supplementum, 19, 201–214.

Veltink, P. H. et al. (1996). Detection of static and dynamic activities using uniaxial accelerometers. IEEE Transactions on Rehabilitation Engineering, 4(4), 375–385.

Vokaer, M., Azar, N. A., & Beyl, D. Z. (2003). Effects of levodopa on upper limb mobility and gait in Parkinson’s disease. Journal of Neurology, Neurosurgery, and Psychiatry, 74(9), 1304–1307.

Yam, C. Y., Nixon, M. S., & Carter, J. N. (2002). Gait recognition by walking and running: A model-based approach. In Proceedings of 5th Asian conference on computer vision (pp. 1–6).