Visualization design for location-aware services

2006IEEEInternational Conferenceon Systems, Man, andCybernetics

October8-11, 2006, Taipei, Taiwan

Visualization

Design for Location-Aware Services

Chia-HowLin, Kai-Tai Song, Sheng-Po Kuo,Yu-Chee Tseng,and Yau-Jen Kuo

Abstract-This paper presents a design and implementation

of a location-aware service system, which combines a

radio-frequency-based positioning engine and a multimedia human-machine interface (HMI). In this system, a number of Zigbeesensors aredeployed for the localizationpurpose.Users' locations can be estimated according to their received signal strength samples from theZigbeesensors. In ordertoimprove

the positioning efficiency, a hierarchical positioning architecture is proposed.Themultimedia HMIistohelpusers to accesstheprovided services. A speech recognition system is

designed asthe input ofthe system and avisualization system has beendevelopedasthe outputinterfaceof the location-aware system. Moreover, the visualization service is integrated with

the Open Service Access (OSA) platform via mobile network systems. Thewhole system can be directedto future homecare

applications.

Keywords: Homecare, Location-aware service, Wireless networks, Locationsensing.

I. INTRODUCTION

As aresult of the progress of medicaltechnology andthe reduction in birthrate in recent years, the population

ratio of the elderly has been increasing in many countries.

Elderly-care services will soon become essential and cause social problems inthe near future. Technology can play an important role by providing a more intelligent home environment to reduce the man power needed for such labor-intensive work and increase the quality of life of the

elderly. One interesting area is the assistance provided by

various context-aware systems. For example, an image surveillance system can transmit videos from

home,

so one

can monitor the elderly throughthe Internet. However, one will still needtocontrol thecamera to find the

elderly

since the system is not able to find the location of the elderly automatically. An enhance visualization

design

using locationinformationprovides such services deserves urgent attention.

This paper emphasizes a human-machine interface for location-aware services. Fig.1 illustrates the architecture of the system. A family member (user B) can request or

This workwassupportedby the Institute for Information Industry, Taiwan R.O.C. undergrant(94)0357.

Chia-How Lin and Kai-TaiSongarewith theDepartment ofElectrical and Control Engineering, National Chiao Tung University, Hsinchu, Taiwan R.O.C.(e-mail:jotarun.ece87gnctu.edu.tw,ktsonggmail.nctu.edu.tw).

Sheng-PoKuoand Yu-CheeTsengarewith theDepartment ofComputer Science,National ChiaoTungUniversity, Hsinchu,Taiwan R.O.C.(e-mail:

spkuogcs.nctu.edu.tw, yctsenggcs.nctu.edu.tw).

Yau-Jen Kuo waswith Institute for InformationIndustry, Taipei,Taiwan R.O.C.(e-mail:yzgaogiii.org.tw).

subscribetheimages and locationinformation of the elderly (userA) through e-mail or cell phone. The location is then estimated by thelocation server, and sent to the video server. The video server controls all the cameras in the house to acquire images. Theselected camera will take pictures from theelderlyaccordingtothe location information provided by thelocationserver.Finally,theinformation will be sent to the user Bthroughan Open Service Access (OSA) Gateway[l]. Inadditiontopassive monitoring, the elderly can also ask for help via the speechrecognition system. His/her location can be found andimage can be sent to the userBinrealtime.

Intheproposed system, the location is estimated by using radio signals transmitted or received by the active beacons such as WiFi or Zigbee modules. Location estimation is obtainedbyapatternmatching approach

[2-4].

Thisapproach is more proper to handle uncertainty and errors in signal strengthmeasurement. However,suchapproach often incurs large computation overheads when the location database is

large. In orderto reduce the computation load for matching the signal strength patterns, a hierarchical positioning architecture is proposed. Under such architecture, the

samples with similar characteristics will be classified into a cluster.Inthepositioning phase,the observedsignal-strength

sample will becomparedwith thepattern of each cluster and classified into the most likely one first. Then, the pattern matching process is launched again with the samples only

belongingtothiscluster. Becausethesamplesinaclusterare muchless than the whole ones inthe location database, the

positioningefficiencycanbeimproved substantially.

Onthe otherhand, considering that most elderly are not used to input devices such as keyboards, mouse or complicated touch panels, the system is also expected to provideanintelligent human-machine interface for easy and efficient access of the location results. A user can thus manipulatethesystemby speechandobservethe location on apanelscreentheimage acquired byacamera.Flexibility and connectivity is also important for location-aware services. Many works have paid attention to creating an extendable framework with generalized application programming interface (API),inorder to share thelocation information and integrate the location services with other useful applications. Inthiswork,the location query service adopts the open and platform-independent Mobile Location Protocol (MLP) to communicate location information. Further, the video/audio information is integrated with an OSA platform, which provides an industrial standard of a common platform for value-add servicesvia mobile network system such as GPRS.

Camera

0

Zigbee Camera .. Sensor

4.

i

J

t

1 Speech *- PtcniztrW?A N, OSA 5. User B

0

Zigbee Sensor

0

Zigbee Sensor GCarmera

Fig. 1. Systemarchitecture of the location-aware service system The rest ofthe paper isorganizedasfollows: the methods

and functions of eachsubsystemaredescribed in Section2. In Sections 3practical experimentsoftheintegrated system are

presented. Theconclusions aresummarized inSection 4. It. PROPOSEDSYSTEMFRAMEWORK

The developed location aware system consists of three mainsub-systems: positioningengine, multimedia HMI, and OSAplatform.

A. Positioning Engine

The positioning engine in this paper adopted the pattern

matching algorithms presented in [6], [7]. In such design, a set of access points which can transmit radio signal for localization purposeareplaced in the environment.Anumber of signal strength measurements are performed at some training locations, and then the measured signal samples and their corresponding training locations will be recorded in a location database. Afterward, given a signal sample with unknown location, the localization algorithm is able to estimate its location by classifying the signal sampleto the

training location which has the most likely signal characteristics in the location database. The positioning enginedevelopedinthis papercanbedivided into threemain parts: location estimation module, database enhancement method,and API.

I) Locationestimationmodule

The locationestimation modulebasedonpatternmatching

methodusually contains twophases. In thetrainingphase, a

setofZigbeesensorsZ={zZ,Z2.. Zn} areinstalled inafield. Within thisfield,a setoftraininglocationsL=

{11,

12, ..., /m}

is defined and at each of which the signal strength is

measured from each access point for a period of time to build upalocation database.

In the positioning phase, given a set signal strength samples S r[sI, S2, ...,

sJ,

theobjective is to match the set to

an training location in the database. The approach in this work is adopted from the nearest neighbor in signal space (NNSS) algorithm mentioned in [6] to match S to the most likely location. In NNSS, for each training location

l1,

the location database will maintain a characteristic vector

C,

=

[C'1,

C'2,

..., c' ] such that c' is the averaged signal

strength

received from

zj.

Givenasignal strength sample S, the NNSS

algorithm computes the Euclidean distance in signal space between S and the characteristic vector of each training location

li

inthelocation database, for 1< i<m, i.e.,

| 2

dist(S,1I)= S,

1i

E (Si-Cj)

jl

According to the Euclidean distance of each training

location, the one that has the minimum Euclidean distance will bedetermined as the estimated location.

2) Database enhancement method

Scalability isoneof the critical issues for the localization

algorithms based on

pattern

matching. When the number of

traininglocations is large,thecomputation costwill behigh.

Hence, anovel clustering algorithm is proposed to improve theefficiencyof thepatternmatching process.

Thebasic idea ofclusteringis to format thesignal strength measurements in the location database into a hierarchical structure. Similar samples are clustered beforehand during

database buildup. Thus, the system is able to compare only

thesamplesin aclusterrather than allthetrainingdataduring

thematchingphase.

..Ql.

t., ,q:.:....

The clustering is implemented with K-Means algorithm. Given an integer k, the clustering algorithm can be summarizedasfollows:

1. Partition characteristic vectors into k nonempty subsets;

2. Compute mean as the centroids of the clusters of the currentpartition;

3. Relocate each characteristic vector to the nearest centroids of thecluster;

4. GobacktoStep 2, stopwhenno more newrelocation. In the estimating-locations process, the observed

signal-strength sample is classified into the cluster whose centroid is closesttothesample insignalspace. The similar process mentioned in theprevious section is thenperformed by comparing the sample with the characteristic vectors in this cluster. The one with the minimum Euclidean distance canthus be found.

Fig. 2 shows the simulation result inanenvironment with 600traininglocations. It canbe observed that thepositioning

accuracyof theproposedclustering algorithmdoes nothave apparentnegative effect when the number of clusters is less than 100. However, theclustering techniquecansubstantially

decrease thecomputationcostinthepositioning phase.

3) Location QueryAPI

Thepositioningengine providesaweb containertoaccept queries from clients. The query format follows the Mobile Location Protocol 3.1 defined by Open Mobile Alliance

(OMA) [9]. Theprotocol stack is shown in

Fig.

3. A client can query the location ofa target recorded in the location server, via the LBSAPI on aPCorthe JSR-179locationAPI on acellphone. TheseAPIswillconstructthe

corresponding

XMLmessageaccordingtothe query request and transmit it tothe locationserverthroughtheHTTP

protocol.

B. Visualization System

The coordinates estimated and provided by the Positioning Engine are in format of

numbers,

which is difficultfor humanuserstounderstand. Therefore in

practice

a usable system needs to represent the results in a more comprehensibleway. Themost commonmethodnowadaysis to usefigureorimagessuchasfloorplanormap.However, in order to monitor the current state of the elders, the located

elderlywill beactuallyshownonthephysicalimageobtained fromapan-tiltcamera. To achieve thisfeature,oneneedsto applycameracalibrationtechnique. Usingcameracalibration, theprojection between world and image coordinates can be obtained.Inorder toincreasethe coverageareaofthecamera, the pant-tilt ability of the camera is also necessary. The systemfirstcalibratethecameraandbuild the camera model, whichrepresents the mapping from the world coordinateto theimage planeand thepan-tilt movement ofthe camera.

The calibrationprocess starts from the geometric camera calibration procedure, which aims to estimate the intrinsic andextrinsic parameters of astatic pin-hole camera model. Theimplemented algorithm is the linearestimation method given in [10]. Assumedthat thetarget environment observed

AveragedPositioningError vs.Cluster Number

4.8 4.7 a) 0 454 aU) I

I3

1 2 3 4 5 10 100 300 600 ClusterNumber

Fig. 2. The simulation result of the averagedpositioning

errorwhen thenumber of clustersincreases. J2ME J2SE LocationServer Applications Applications

JSR-179 LBS API Coordinate_System Positioning_Engine Privac_Rules

Mobile LocationProtocol MobileLocationProtocol

Servlet Container ServletContainer

Transport Layer (HTTP, SOAP,WebServices)

Fig.3. Protocol stack of the location interface

by a cameraand theimage positionp, (u1,v1)T, ofn feature

pointswith known world coordinatevectorsPi-(xi,y1,zi)Tare

matched in theimage,i=1...n.Theprojectionbetweenworld

andimagecoordinatescanthen be modeledby

~~~~~~~~~~i(2)

pi= MP1 ,where M =m -<j) P=IJ

z

MisdecomposedtoM=K(R t),where(R t)representsthe extrinsic parameters, including rotation matrix R and

translation vector t. K represents intrinsic parameters, including skew and aspect ratio. Collecting the constraints from(2)associated withnpairs points yieldsasystemof 2n

homogeneouslinearequations in 12coefficientsofthe matrix

M-namely,Pm=0 ,where

piT

oT

oT

PT ... ... P7 07 0'

PT

-uP -n Pn -nPn ,andm = [m7

mTm

m ] (3)

As long as

n.6

(since P is a 2n by 12 matrix), the homogeneous linear least-squares can be exploited to compute the unit vector m that minimize

LpmI2

and thus estimate M. In this system, the n correlation points are

markedmanually fromthe captured image scenewith a GUI interface. To increase the accuracy and avoid outliers, the Random Sample Consensus (RANSAC) algorithm

[10]

is alsoimplemented. This algorithm picksnpoints from alarge set of sample points randomly and the estimate the matrix iteratively until the fitting error is below the expected

criterion. In the experiment, the average error is around 2 pixels.

Thesecond step istoestimatethe influencebrought bythe

pan-tiltmovement. Intheproposed system apan-tiltcamera model, as shown in Fig.4, is used. A rotation around the x-axis, Rx, and the y-axis,

Ry,

correspond to tilt and pan respectively. With this model, the projection between world and imagecoordinatescanthen be modifiedas:

pi

=-MR,RxPi

(4)

The choice ofthissimplifiedcameramodel isatradeoff for

straightforwardness overaccuracy. The model assumes that the center of rotation of the camera is fixed and coincides with the lens center ofprojection during operation, which many cameras mayviolate suchanassumption. However, in thisapplication, the deviation of thecenterisnegligible.

Another problem is that many low-cost cameras cannot provide the actual pan-tiltangle. Toovercomethis problem,

the kinematics model of the camera has to be calibrated beforehand. The kinematics calibration of this system is realized by randomly choosing several poses of the camera, and performing the camera calibration procedure. The kinematics model of pan and tiltcanthen be estimated from

alternativeresults of calibrations iteratively. Furthermore, in ordertoguaranteethat the targetcab be showncompletelyin the image, the system will first query theheightand width of the target, calculate itsprojectedarea ontheimage plane,and thenmovethecamerato ensurethatall thepointsareshown, if possible. A target can then be marked on the image

accordingtoitslocation. C. SpeechRecognitionsystem

Considering that elderly members are generally few in a

family, say around 3 persons, the mode of the speech recognition system is set as speaker-dependent. The command anduserindexarepre-defined integerstorepresent

y

..x

Fig.4. ASimplifiedmodel of thepan-tiltcamera

the recognition results of the input speech signals. The recognition algorithm is based on the dynamic time warping (DTW)method[11]. The procedure ofthe speech recognition isdivided into three parts.

The first part isthe process of speech information sampling, which is the setting of gain control and sampling frequency when analog signals transforms to digital signals. The second part is the extraction of speech features. The endpoint detection process can determine the location of real speech signals by the short time energy detection and zero-crossing rate detection. The started 128 samples are used to determine thethreshold value of the energy detection and zero-crossing rate detection. When the real speech signals are determined, theyaredivided intoa 1 6mslength frame. Each frame has an overlap region. Inorder to do the encroachment of the high frequency spectrum, the pre-emphasis work is adopted. Considering thecontinuity of signals at two sides of a frame; Hamming window is used on every frame. It can hold the

signalsatmid parts and press down the signals at two sides. The Mel Frequency Cestral Coefficients (MFCCs) is extracted for every frame. A feature vector represents each frame. The DTW recognizer compares the input feature vectors with the reference speech and gets a minimum matchingerrorastherecognition result. The third part is the process offeature matching by the dynamic time warping method. The speech recognition system is implemented in a TITMS320VC5402 DSK board.

D. OSAplatform

The OSA API is an open, standardized interface for

applications to use the capabilities of a network without

owning it or knowing its technology. It consists of a framework, which is in charge of access control and service

discovery;andsomeServiceCapability Features, which map tonetworkcapabilities. Itisspecified and standardized in the Joint API Group, with participation of 3GPP, ETSI and the Parlay Group: a single API for the whole developer

community.

Parlay/OSA [1] enables operator and 3rd party applications to make use of network functionality through a setof open, standardized interfaces. This leadstoseveraladvantagessuch as shorter time to market (TTM) for applications, network

independency, etc. Parlay/OSA Gateway consists of several Service Capability Servers (SCS): functional entities that provide Parlay/OSA interfaces towards applications. Each SCSisseenbyapplicationsasone ormoreService Capability Features(SCFs): abstractions ofthe functionality offered by thenetwork, accessible via the Parlay/OSAAPI. Sometimes they are also called services. The Parlay/OSA SCFs are

specifiedin termsof interface classes and theirmethods. In this system, the OSA gateway is connected with the location and visualization system in order to extend the serviceto mobile network. Fig.5 shows components applied

in this system. The OSA application server receives the request from users using mobileterminals, andrequest data

from the OSGi gateway at home. The data can then be sent to the user through two different paths: by e-mail via the internet, orby MMS via cellular network.

III. EXPERIMENTAL RESULTS

Inorder to evaluate the capability andpracticability of the system,the system is tested in asample room. Fig. 6 shows the layoutand the physical scene of the room. The pan-tilt cameraDCS-5300Gfrom D-Linkwasinstalled in the middle of the room. Nine Zigbee modules were placed around the room,while three users carrying theZigbee beacons. A. Examiningthe Positioning Engine

The goal of this experiment is to testthe accuracy of the estimated locations. 9 training locations is collected in the location database. Users wearing Zigbee beacons move around the room, while the system triesto keep positioning

their locations. Fig. 7 shows the experimental result. The

averaged location error of 1.86 meters wasrecorded in this experiment.

B. Examiningthe visualization system

Thisexperiment aimstoinvestigate thecapacityofcamera calibration when usedtomark apersonintheimage. Eight

positionswereassignedonthefloor,to emulate the result of the positioning engine. As shown in Fig. 8, giventhe exact coordinate, thecamerasuccessfulturnsand marks the person (in the bathroom)ontheimage. Thus the localized personcan be visualized and thepicture can be transferred through the OSAplatform.

C. Examining thespeechrecognition system

Asdescribe in the firstsection,it is desirable thatanelderly

canask forhelp via thespeechrecognitionsystem.Therefore,

the system hasto atleastdistinguish whotheelderly is, and what heorshe needs. Thespeechrecognitionsystemis tested with 3 different names and 6 phrases, for demonstration. Table I shows the result of the recognizing names ofthree persons. Table II shows the result of recognizing some

daily-life Chinese phrases. The recognition rates are satisfactory forpractical applications.

D. Integrationtest

Theintegrate systemhas been testedto examthe scenarios Applicaien

OA

-dass

OoNoil dt d

Fig.5. OSAnetwork servicesappliedintheproposedsystem

_

_-I

LI

Fig. 6. Thescenarioof experimental setup CumulativeDistributionFunction

8 7 6 23 w 2 0 0 10 20 30 40 50 60 70 80 90 100 Percentage(%)

Fig. 7. Theexperimentalresult ofthe positioningengine

described in the first section: location request and

"asking-for-help". However,the marker function isneglected due to the accuracy ofthe Positioning Engine. In the first scenario, several people with Zigbee beacons move around some room simultaneously. A user then requests to watch these people from the cell phone. The camera successfully moves to the target person and the user retrieves the target image for each requestsuccessfully. Inthe secondscenario,

several people wearing a headset microphone, as shown in Fig. 9. They took turns saying their name, to pretend they were in a situationto ask for help. The speech recognition

system successfully recognized these people, translated the messagestothe location server, and launched thecamera to rotate and capture image of the person. The images successfully transmitted to other users' cell phones and PC via e-mails.

IV. CONCLUSIONSANDFUTUREWORK

This paper proposesa location-aware system combininga RF-based positioning engine and a multimedia HMI. A hierarchical positioning architecture is applied in order to improve the positioning efficiency. The multimedia HMI facilitatesauser tovisualizethelocated target, and command the system by speech. A home-care application has been demonstrated to evaluate the performance and possible applications of the system. The developed RF-based positioning engine has been demonstrated its advantages in practical applications, but also reveals its problem on not accurateenough in position estimation. It would be beneficial to increase the accuracy of the visualization system by marking the target using image recognition and feature

TABLEI

EXPERIMENTAL RESULTSOFRECOGNIZINGTHENAME ChiaHow ShengPo Jin Huai Recognition Rate

Chia How 25 0 0 100%

Shenh Po 0 23 2 92%

Jin Huai 0 1 24 96%

TABLEII

EXPERIMENTALRESULTSOFRECOGNIZING COMMON PHRASES*

Phrase Recognition Rate a b c d e f a 17 0 3 0 0 0 85% b 0 16 0 0 4 0 80% c 0 1 16 0 1 0 80% d 0 0 0 12 8 0 60% e 0 0 1 2 14 3 70% f 0 1 1 0 2 16 80%

*Phrases a-f represent: help, blistered, ambulance, thief, fire, and hungry.ThesephrasesarespokeninChineseintheexperiment extracting techniques. Furthermore, the system can be

appliedto a"never-lost" system, whilenot only human, but also anyobjectwhich isRF-taggedin thehouse.Theycanbe foundthroughlocation-aware system andvisualized on aTV set, forinstance.

ACKNOWLEDGMENT

The authors would liketo thank Mr. Kun-MingYan and Mr. Jin-Huai Hsufor their support in the speech recognition

systemandexperiments.

REFERENCES

[I] "OSA/Parlay Overview," [Onlinedocument],2002Sep 7, [cited 2006

February 20], Available HTTP:

http://www.openapisolutions.com/brochures/OSAParlayOverview.pdf [2] P. Bahl and V. Padmanabhan, "RADAR: AnIn-Building RF-Based

UserLocation andTracking System," in Proc. IEEEInfocom 2000,

IEEECSPress,LosAlamitos,Calif,2000, pp.775-784.

[3] P. Castro, P. Chiu, T. Kremenek, and R. Muntz. "A Probabilistic Location Service for Wireless NetworkEnvironments," International JournalofWirelessInformationNetworks,Vol.9,No.3,pp. 155-164, 2002.

Fig. 8.Anexampleof the"targetmarking" of the visualization

system

Fig.9. Anexample of the "asking-for-help" scene

[4] S. Banerjee, S. Agarwal, K. Kamel, A. Kochut, C. Kommareddy, T. Nadeem, P. Thakkar, B. Trinh, A. M. Youssef, M. Youssef, R. L. Larsen, A. U. Shankar, and A. K. Agrawala, "Rover: Scalable Location-Aware Computing," IEEE Computer 35, No. 10, pp. 46-53, 2002.

[5] T. Gu, H.K. Pung, and D.Q. Zhang, "Toward an OSGi-Based Infrastructure for Context-Aware Applications," IEEE Pervasive Computing, vol. 03, no.4, pp. 66-74, 2004.

[6] Y.Chen,X. Y.Chen,F.Y.Rao, X. L. Yu, Y. Li, and D.Liu,"LORE: aninfrastructuretosupportlocation-awareservices," IBM Journal of ResearchandDevelopment,v.48n.5/6, p.601-615, 2004.

[7] X.Chen,Y.Chen,and F.Rao,"AnEfficientSpatialPublishSubscribe System for Intelligent Location-Based Services," in Proc. the 2nd International Workshop on Distributed Event-Based Systems (DEBS

'03),June2003,pp. 1-6.

[8] A. K. Narayanan, "Realms and States: A Framework for Location-awareMobileComputing," in Proc. of the Ist International Workshop on Mobile Commerce, July 2001, pp. 48-54.

[9] "Open Mobile Alliance (OMA):Mobile Location Protocol (MLP). CandiateVersion 3.1- 16 Marz2004," [Onlinedocument], 2004 March 16,[cited 2006 February 20], Available HTTP:

http://www.openmobilealliance.org/releaseprogram/docs/MLP/OMA

-LIF-MLP-V3 1-20040316-C.pdf

[10] D.A.Forsythand J.Ponce, ComputerVisionAModern Approach, New Jersey: PrenticeHall,2003.

[11] C. S. Myers and L. R. Rabiner, "A comparative study of several dynamic time-warping algorithms for connected word recognition," TheBellSystem TechnicalJournal,vol.60,no.7, pp.1389-1409, 1981.