2006 IEEEInternationalConferenceon Systems, Man,andCybernetics
October8-11, 2006,
Taipei,
TaiwanSmart
Pantries
for Homes
C. F. Hsu, H. Y. M. Liao,P. C. Hsiu, Y. S. Lin, C. S. Shih, Member,IEEE T.W. Kuo, SeniorMember, IEEEand Jane W. S.Liu,Fellow, IEEE
Abstract-A smart pantry holds non-perishable household
supplies and automates the purchasing and delivery of their replenishments. Byrelieving itsuserfromthe choreofkeeping
the home stocked of essentials, it provides convenience and peace of mind not only to elderly individuals but also busy
people ofall ages. This paper describes twoalternative smart pantry designs and tradeoffs. Underlying methods and
technologiesusedfor theirimplementationarealsodescribed.
I. INTRODUCTION
In recent years, population of developed and developing countries is aging at a rapidrate [1, 2]. There isa growing need for low-cost, easy-to-use, and dependable devices and services designed to help the elderly live independently, improve their quality of life and reduce thecostof theircare. Examples of these devices includeobjectlocators forfinding misplaced household andpersonalobjectsand automatic and robotic helpers for enhancing physical dexterity and accessibility [3, 4]. Given the fact that the average percentage ofpopulation 65 orolder will soonexceed the percentage of population under 15 inthesecountries,oneexpectsthat such devices may someday be as demanded as iPod, game consoles and robotic toys.
Anotherexample ofconsumerelectronics fortheelderlyis smartpantriesfor storage ofnonperishable supplies. Such a
pantry monitors its contents and automates thejust-in-time replenishment ofobjectsinit.Thus,it relieves ofitsuserfrom the chore ofkeeping objectssuch asshampoo anddetergent on hand. Smart pantries are for convenience of elderly individuals, aswell as busy people of all ages who haveno time or interest to shop (or place order) for boring but essential householdsupplies.
This paper describestwo alternative smartpantrydesigns andimplementations: thepicture-idversionandthebar-code version. The difference between the versions arises from differences inthetechnologies used for content capture and object identification. The picture-id version, called PID pantry, uses an overhead camera to capture its contents. In purchase orders sentbya PID pantryto the suppliers, each
C. F. Hsu is with Department of Computer Science and Information
Engineering, National Chiao-Tung University, Hsinchu, Taiwan (e-mail:
aldarishsu(Zyahoo.com.tw)
H. Y. M. LiaoandJ.W. S. Liu are withInstituteof InformationScience,
Academia Sinica, Nankang, Taipei, Taiwan (e-mail: {liao,
janeliu}@iis.sinica.edu.tw)
P.C.Hsiu,C. S. Shihand T. W. Kuo are with Department of Computer
Science andInformation Engineering,National Taiwan University, Taiwan
(e-mail: {r91004, cshih, ktw} @csie.ntu.edu.tw)
Y.S.Lin is with Department of Computer Science, National Tsing Hua
University,Taiwan(e-mail:[email protected])
objectto be delivered isspecified byapicturecaptured bythe camerapriortothe removal oftheobject from the pantry.The supplier mustprocess the photo image, either manually or automatically, in order to identifythebrand and size ofthe object.Forthisreason,theusability ofPIDpantry isnotideal from the supplier's point of view. It is easy to use from owner'spointofview,however. Other thanmakingsurethat nothing blocks the view of the camera, the user can treat a PID pantry just like a dumb pantry. The bar-code version, called BACpantry, identifiesobjectsin itbytheir bar codes. Because every object in every purchase order is unambiguously identified by a bar code, BAC pantries are easy to use from the supplier's point of view. On the other hand, theuser must scanthe bar code of every kind ofobjects inthe pantry. Unlessgivenabarcode before thesupply ofthe kind runs out, the pantry will not be able to order replenishmentautomatically.
Anatural question iswhynot RF identifiers (RFID). If every household object were to come with a smart tag, a pantry equipped with a RFID reader can easily maintain inventoryas the user movesobjectsinandoutof thepantry. This version would have the advantages of both PID and BACversionsandnoneof theirdisadvantages.Infact, this is howsmartcabinets used inhospitals for storage of medical supplies work. Unfortunately, the RFID-version of smart pantries for homeuseis stillnoteconomicallyfeasible and is likelytoremain soforsometimeto come[5]. Acostof tens ofcentsper tagis lowenoughfortaggingexpensivemedical supplies but is orders of magnitude too high for tagging ordinary household items individually.
Smartpantry isone ofafamilyofappliances thatarethe research focus of the SISARLproject [6, 7]. SISARL devices and appliances are consumer electronics of
convenience,
personal safety and health maintenance. Targeted users are elderly individuals who may have some functional limitations, but are still in relative good health, live independently and, most likely, in homes of their younger years. Like assistive devices and home care equipments, SISARLdevices must beeasyto useandhighly dependable. However, assistivedevices (e.g., [8-15])typically assume a
smart operating environment equipped with computer(s), Internet and, often, a variety of smart sensors; their targeted users, being in need ofhelp in daily living, subscribe to assistive services, and so on. These assumptions are unrealistic forSISARLdevices. To keep the costs of SISARL devices to a small fraction ofthe costs of typical assistive devices is anotherchallenge.
Following this introduction, Section II discusses assumptions and constraints common to both versions of smart pantry. Section III describes of their architectures and implementations. Section IV describes thetechniques used by PID pantries for object identification purpose and summarizes preliminary experimental results. Section V describeswireless sensingschemesfor BACpantry. Section VI summarizesthe paper and discusses future work.
II. COMMONALITIES AND DIFFERENCES As stated earlier, a smart pantry is used to hold non-perishable household supplies. The pantry knows its contents andcan automatetheirreplenishments.Forexample, each pantryinFig. 1 knows that abag of paper towelsis on the top shelf. When the last roll is removed, the pantry contacts a supplier ofuser's choice and requests the store to deliverareplacement bag.
(a) (b)
Fig.1 SmartPantries
Anunderlying assumption is that one ormore grocery and discountstores have agreedtoreceive and processpurchase orderssentby the pantryonthe user'sbehalf and delivereach order by a specified date. (Alternatively, the panty may be provided totheuserbya supplier.) The information required for contacting suppliers, placing orders and arranging payments and deliveries were entered into the pantry at initialization time, together with information on user preferences. By default, purchase orders are sent via messages over a dial-up connection, but a user with broadband internet access can configure thepantryto place orders via Internet.
Both versions are designed to accommodate concurrent insertions and retrievals ofobjects by multiple users. They use a keypad, a microphone, a speaker and a recorder to support user-pantry interaction. The microphone,
speaker,
and recorder form an audio interface that records voice segments of the user and plays back user voice interleaved with pre-recorded pantry voice. By allowing the pantry to
interact with theuser, the audio interface makes the pantry friendlierand more toleranttomisuse.
Like ordinary dumbpantries, the storage space ina smart pantry is divided into compartments. Figure 1 shows two
configurations. The picture-idversion is constrained to use
theonein Fig. 1(a). The fact that eachcompartmentis clearly defined byarectangular boundary simplifies the extractionof pictures of individual compartments from a picture of the entirepantry. Thebar-code version is constrained to use the configuration in Fig.1(b). In this configuration, shelves are notnecessarily divided vertically. Rather, each compartment on a shelf corresponds to a switch that is in the front of the shelf and a spring-loaded plate that moves perpendicular to the shelf. (This construction is similar to the ones used in many drug and grocery store shelves.) When aplate is at the front of the shelf, as illustrated by the plate on the bottom shelfinthe figure, it presses the switchclosed, indicating that the corresponding compartment is empty. The compartment isnonempty when the plate is pushed towards the back of the shelf byan object, leaving the switch open. By sensing the states of the switches, a BAC pantry can distinguish empty compartments from non-empty ones.
Bothversions require that objects in each compartment are identical. As we will see in the next section, a PID pantry cannottell whether objects in two or more compartments are identical. By default, it will order replenishment when the last ofthe objects in any compartment is removed, even when some other compartments may contain more of the same objects. Moreover, it is constrained to order all objects from the same default supplier with the same replenishment time (i.e. the length of time-to-delivery interval). The bar-code version does not have these limitations.
III. ARCHITECTURES AND IMPLEMENTATIONS Inour discussions, we refer to compartments by 2-tuples of rowsandcolumns. As an example, (3, 4) refers to the fourth compartment from the left on the third shelf from the top. We use rowsand columnsto meannumbers ofrowsandcolumns of compartments in the pantry.
A. PIDPantry Design
Fig.2and Fig. 3(a) show the physicalcomponents of a PID pantry. Itconsists ofabaseunit anda remoteunit. The base unit contains a digital camera, together with the processing andstoragemodules that domostof thework. The base unit is mounted overhead so that the camera can capture a front viewof the pantry shelves. Theremoteunit contains all the I/O devices and is within an easy reach ofthe user. It also provides access to the supplier(s). The units are connected wirelessly.
Fig.4describes theoperations ofthepantry. The pantryis empty during initialization. After capturing currentpicture of the pantry, the pantry controller processes itto determine the boundaries of thecompartments andvalues ofrows and columns and captures and stores EMPTY, a picture of an empty compartment (For the sake of simplicity, our
discussion assumes that all compartments look alike when empty. This restriction can be easily removed.) It then allocates an array, calledpicture[rows, columns], to store pictures of individual compartments and initializes every elementtoEMPTY
cordlessphone,respectively.) On the other hand, becausethe
pantry cannot distinguish objects from each other, it has the limitations mentioned in SectionIL.
i11E1137 o Purchase .
lEIEllSO r ~~~order - 040044000040
*n-o
X 8~~
generator
~~~~~~~~
o
o
00X0~~~a=>to
u*-1EO:C^
~~~~~~~~~~~~printer
o Keypad& 1|111--1E1 O X ~~~~~~~~~~~
Picture & E E Audio*0111|11|1~uppie _DBO Interface
Compartments Base unit Remote unit
Fig.2 ComponentsofaPIDpantry
rows
(a)
Fig. 3 Arrangement and Contents
Paper Towels;-... 2 columns I picture[rows, columns] (b)
During normal operation, the controller examines the
contents of the pantry periodically (e.g., every 200
milliseconds) by having the camera take a snapshot of the
pantryatthestartof theperiod.When the controller finds that
thenewsnapshot
(current_picture)
is essentiallyidenticaltothe previous one (previouspicture), it does nothing.
Otherwise, it extracts the picture current content ofevery
compartment(i, k)fromcurrentpictureandcomparesitwith
theonestoredatpicture[i, k]. Thus it determines whether the compartment has changed from being empty to non-empty
and viceversa. Inthe formercase,it stores thenewpictureof
the compartment inpicture[i, k] for use later. In the latter case, the compartment has just been emptied. The pantry inserts thepicture storedatpicture[i, k]into the list ofobjects
to be ordered and then sets the elementtoEMPTY. The list
thusgeneratedcanbeprintedand usedas ashoppinglistifthe
user chooses to shop personally, rather than relying on the
pantry,orsentbythe pantrytoasupplieras apurchaseorder.
Fig. 3(b) shows what picturemaycontainatsometime. The
solid colored boxes indicate EMPTY. To ease the task of
automatic objectidentification,the pantry also sends EMPTY ineverypurchase order.
Wenotethatabasic PID pantry suchas onedescribed here
merely extracts pictures ofthe objects in compartments. It
cannotidentifytheobjects. AswewillseeinSectionIV, the
objectidentification function for determining the brands and
sizesofobjectsfromtheirpictures requiressignificantlymore
processing power than what is needed to segment a big
picture into smaller pictures. The function also requires a
database ofhigh quality pictures of objects to be identified. Byoff-loading this functiontoserversatthesupplierside, the
pantry is kept as simple aspossible. (The base and remote
units can be built from commodity digital camera and
Fig. 4 Operations of PIDpantry
B. BACPantry Design
Again, objectsinaBACpantryare identifiedby their bar
codes. With user'shelp,the pantryacquiresincrementally the
bar codes and voicedescriptionsofallobjects ever storedin
the pantry 2 rows 1 2 columns
IL
state=EMPTY; object id=NULL;Fig. 5 Compartment and objectdescriptions
Some of the date structuresmaintained by thepantryare
shown inFig. 5. Each element of thecompartmentarrayon
the left containsthe state ofacompartment and apointerto
thedescriptionoftheobjectinit. Theright half of the figure depictstheinformationoneachobject, which includes the bar
code andarecordedvoicedescription of the object. Theuser may choose to keep some kind of objects in multiple
compartmentsand havethepantryorderreplenishment only when allthe compartmentsholding them become empty. In
thatcase,thecompartmentsfield gives pointersto structures
ofcompartmentsholding thesamekind of object.Theuserof
aBACpantryhastheflexibility of ordering differentkinds of
objects from different suppliers and specifying different replenishment times. This is why the pantry maintains
Initialization:
Initializesupplier information and user preferences; Command the camera to take current picture of the pantry;
previouspicture=current picture;
Processcurrent picture to identify compartment boundaries, Determine numbersof rows(rows) and columns (columns); Allocate array picture[rows, columns];
Initializeelements of picture to EMPTY; while pantry runs, periodically do {
Command the camera to take currentpicture of the pantry; if(currentpicture!= previous picture)
for every compartment(i,k), do{
Get current content of(i, k) from current picture; if(current content != picture[i, k])
if(current_content ==EMPTY)
Insert picture[i, k] to items to orderlist; picture[i,k]=EMPTY;
else{
picture[i,k]=current content;
Generateapurchase order for objects in items to-order;
supplier and user preference information on each kind of objectindividually.
The architecture of BAC version is similar tothat ofPIC version. Adifference is, obviously, that the remote unit of a BAC pantry includes a bar code scanner. Another difference is that BAC versionuses anarray of switches to monitor the states (i.e., empty or non-empty) of the compartments. Whenever the state of a compartment (i, k) changes from empty to non-empty, the pantry controller acquires, with the user's help, the bar code andavoice description of the object just placed in the compartment. When the state of (i, k) changes from non-empty to empty, the controller puts the bar code of the object that was in the compartment in the items_to_orderlist, sendsapurchase order to the supplier of the user's choice and then marks the compartment empty. Detailsonthe architecture, operations and implementation of aninterrupt-driven BAC pantry controller can be foundin[4]. That implementation assumes that switches used to monitor compartment states are wired to the pantry controller interface. We will return in Section V todescribe a ways to configure an existing dumb pantry into a smart one using wirelesssensors.
Likethe PIC version, multipleusers ofaBAC pantrycan place objects and remove them in any order. Scenarios of user-pantry interactions putting objects in pantry and removing them can also be found in[4].
Errors during placements and removals are inevitable. When theuser does not followthe normal sequence ofscan andplacement, someobjects inthe pantry may have nobar codes and some objects may have wrong bar codes. These errors arerecoverable. Anerror of the formertype isknown to the pantry. Ithandles the error by asking theuser to scan the object atthe time of the removal. Whenan objecthasa wrongbar, the voice confirmation from the pantryduringthe removal process provides the user with an opportunity to discover theerror and initiatea corrective action.An erroris unrecoverable when it causes the pantry to fail in ordering correct replenishment intime. Unfortunately, unrecoverable errors can occur when the user ignores the voice from the pantry verifying with the help of the user the accuracy its knowledge aboutonthecontentsof the compartments.
IV. OBJECTIDENTIFICATION MODULE To make PID pantries easy to use, a supplier add to its order processing server an object identification module (OIM) that processes pictures contained in purchase orders from smartpantries and returns as results the brands, sizes and locations of the objects. Roughly speaking, themodule identifies the object in each picture by determining which image amongallimagesofknownobjectsinitsdatabase best matchestheimageitextractsfrom the
input picture.
The OIM works faster and more accurately when the number ofcandidate images to match is small. Hence, the module maintainsfor eachuser asmall
repository
ofimages
ofobjects that have identifiers andare known tohave been purchased bytheuser.When theuserordersnonew
products,
the OIM only needs to search the user's repository when it tries to identify objects in the input pictures. It searches the imagedatabase of all objects in the supplier's inventory only on rareoccasions when the user orders something new.
A. ApproachandAlgorithms
The flow chart in Fig. 6 gives an overview of OIM operations. The major steps are background subtraction, low-level image processing, color matching and shape context-based search. Query Input (picture) Query Shape Result <: Context-Based
(object id) Search
Fig. 6 Object identification operations
Background Subtraction (BS) The first step is to extract the foreground image of the object from the input picture of a non-empty compartment. For this work, the OIM uses as background the picture, EMPTY, of empty compartments, whichthepantry sends inthe purchase order. Let B(x, y) and I(x, y) be the values of the background image and the input image at pixel (x, y), respectively. A way to determine the valueJ(x,y) of theforeground image at (x, y) is to let it be I(x, y) ifthe absolute difference between I(x, y) and B(x, y) is larger thanapresetthreshold T; otherwise,J(x, y) is equal to 0. Themodule usesthis simple way to determine the values of most foreground pixels. It chooses the threshold Tbasedon thestatistics on past pictures sent by the pantry.
Low-Level Image Processing (LLIP) Because of shadow points and noises, the foreground image produced after the BS step may have fragments that originally belong to the same component. A goal of the LLIP step is to fix this problem. In this step, OIM carries out morphological operations (i.e., dilation followed by erosion) several times [16]. It then executes a connected-component-labeling processtolabel distinctobjects in the foreground.Itconsiders objects that are smaller than a threshold size as noises and removesthem from theforegroundimage.
Shadowis anotherproblemthat must be dealtwith. Some shadows may be oflarge enoughsizestoberetainedafter the connected-component-labeling process. The object identification module applies a shadow removal algorithm described in [17] to remove them. The algorithm compares the hue, saturation and intensity values between the foreground and the background images and distinguish shadowpixelsfrom the foreground pixelsusingtheequation given below: In the equation, I
(x, y)
, I(x,
y)
and IH
(x,
y)
represent, respectively, theintensity,
saturation,and hue values ofa
foreground
pixel
at(x, y);
BkV
respectively; and a, j, us,and TH are thresholds chosen onthe basis oflightingcondition andpre-determinedstatistics.
1 if a<
Bv~(x,y)
I, ( )<A Ik(x,y)-Bk (x,y))<T
A|k (,y-k (,)|<'rH
0 otherwise
SPk(x, y) =1 representsthat thepixelunder consideration isa
shadow pixel and is to be removed from the foreground image.Otherwise, thepixel remainsin theimage.
ColorMatching(CM) Theforegroundobjectobtained after theLLIPstepisacandidate forobjectrecognition.TheOIM uses acoarse-to-fine matching mechanism. For coarselevel search, it works with two standard descriptors in MPEG-7 [18], dominant color and color layout, to perform object recognition. Inthisprocess, itcalculates the dominantcolors fromR,GandBchannels,quantizes256colors into 32bins, and then distributes all pixels belonging to an object into these bins. The top 3 bins of each channel are picked to
representthechannel forcomparison.Thecomparisonmetric is Euclidean distance, and the weights assigned to the channelsareidentical.
Since the dominant color is a global feature, it does not
carry any relational information. Therefore, the OIM also includes colorlayout anduses aquad-tree toexpressit. This way, a different object that has the same set of dominant colors or an identical object that is placed in different orientations will not be mistakenly chosen in the coarse
search stage.
Shape Context-Based Search (SCBS) In addition to the dominant color and colorlayout, adetaileddescriptionof the foreground object to be recognized is required for the fine searchprocess. For thispurpose,thecurrentversion of OIM usesashapecontext-baseddescriptor [19, 20] tocharacterize theshapeofanobject.
Tocomputethedescriptor, the OIM firstappliestheCanny edgedetector [21]to extract asilhouette ofthetargetobject. (The Canny type detector is one ofthe best existing edge detectors butcannotguaranteethefullextraction ofcomplete silhouette.) The OIM then selects rcontrol points from the detected silhouette. The distance between everyconsecutive controlpoint pair is almost equalexceptfor thebrokenparts due to incompleteness of the silhouette. Fig. 7 shows an example. The top row shows the picture of an object. The picture in the left of the bottom row shows the almost complete silhouette produced by Canny edge detector. The picture in the right shows the r selected control points generated from the silhouette.
Next, theobject identification module computes log-polar histograms corresponding to the r control points for each object that is to be compared. Each histogram characterizes the relationship among the r chosen control points of the
object [19, 20]. To derive the histograms, the OIM uses a
circle mask to coverevery control point ofthe target object. It first divides the circle into 12 30-degree bins along the circular direction and then divides each bin in the radius direction into 5 binsequally according to the log ofthe radius. The circle mask after partition is shown in Fig. 8(a). The log-polar histogram of a controlpoint can then be found by putting the center of a circle mask on the control point and calculatingitslog-polar histogram. Calculation of alog-polar histogram when a circle mask is placedat a control pointis illustrated in Fig. 8(b).
Fig. 7 Example on context-based descriptor
(a)
4b *
10
-(b)*
Fig. 8 Acircle mask and a log-polar histogram
After computing sets of r log-polar histograms of the objects in pictures contained in a purchase order, the OIM puts them and pre-computed sets ofr histograms of known objects contained in the user's repository into a bipartite graph.Asthe finalstep, the module calculatesthedegrees of matchesaccordingtoHungarian algorithm[22].
B. Preliminary Performance Results
Inorder to testthe efficiency and effectiveness oftheOIM, we constructed a database of the forty objects and a small pantry.Theobjectsareshown in Fig. 9. In an experiment, we processed a purchase order containing pictures of five objects inways described above. The picture on the left in Fig. 10 shows how the pantry looks when empty, and the picture on the right shows how the five objects looked in the pantry beforetheywereremoved.
ri-u
1aN1r1~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~..~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~I
Fig. 9 Objectsindatabase
N..NSE_
I_
43WeltBljl5R*} 13 A.4171,M S 134 3.3i4s
Fig. (a) Resultsof coarsesearch
Qw-yllbjet Ttqe-.aes[rbied
t....
s
-~~~~~~~~~~~~~~~~~~~~~~~~~~
F
G,-en e e 1G- t.4 FU OFV15 f1 0.ie ikS Z5 la'Fig. 10 Empty and loaded pantry
The images extracted by the OIM are in the leftmost column inFig. 11(a). The remaining six columnsinthefigure givethe top six(orfewer) candidate objects retrieved bythe module for eachinputobject after thecoarsesearch (i.e., the CM step). Obviously, a coarse search does not eliminate enough candidates. However, the coarse search process is veryfast, allowingthemoduletoquicklyscreenthedatabase and obtainasignificantlysmallersetofcandidates.Inthe fine searchprocess, themoduleappliedtheshapecontextfeatures
on candidate objects and obtained much more accurate results.Fig. 11(b) showstheresultsof finesearch. Thespeed ofthe fine search process is slower than the coarse search because ofthe computation ofshape context, but the time consuming process is applied to only a small number of candidates.
Fig. 11(b) Results of finesearch
V. WIRELESSSENSOR ARRAY
IntheBACversiondescribed in Section3and [4], switches formonitoringcompartment states are connectedbywires to the sensor interface on the pantry controller
11O
bus. This construction is suitable forpantriesspecially builttobesmart. However,wiringup anexisting dumbpantry tomakeitsmart would beanunattractiveoption. Abetteralternativeistouse wireless sensors, allowing the user to configure a dumb pantry (or a part of the pantry) into a smartBAC pantry as easilyas aPIDpantry.A wireless binary sensing scheme for this purpose is describedin[4]. That schemeusespassiveRFIDtags,oneper compartment. The controller contains areader. The shelves and spring-loaded boards are constructed so that the tag corresponding to acompartment isshielded from thereader when the compartment is emptybut is visibleto the reader when the compartment is non-empty. The controller determinesthestatesof thecompartmentsbyreadingthe tags periodically. While the scheme works in principle, itis not
ideal. Metal objects inthe pantry can shieldthereader from
somevisibletags.Solutions(e.g.,useofmultipleantennae)to
thisproblemleadtoaddedcostandinstallationdifficulty.
I'''''''.
f6-SISARL project is developing an ultra-low-cost wireless sensorarray (WSA) foruseinBACpantries,aswell as other SISARL devices (e.g. medication dispensers [7].) A WSA contains acoordinator andanumber ofsensornodes. When used in a BAC pantry, the coordinator is a part of the controller, which isACpowered.
Foreach compartment, there isa sensornode(SN), which is batteryassisted. Let Nbe the numberof sensornodes in the WSA,and the id's of the nodesare 1, 2, ...,N.
TheWSA resembles a wirelesspersonal-area network(see http:
//ww.ieee802.org/
15) inphysical size, but because of itsapplications,aWSAhasmanydifferent characteristics:1.Nissmall (say less than265)
2. The distances between nodes and the coordinator are small (say less than5meters).
3. Each SN contains a sensor thathas a small number of states. (Sensors in BACpantrieshaveonly2 states.) 4.A SN sends to the coordinator a fixed size frame
containing its id and one byte ofdata on thenew state whenever thestateofitssensorchanges.
5. The chance thatmorethan a few(e.g., 3)nodes having frames to sendwithinonesecondisnegligiblysmall. 6.The chance that a SN hastwo frames to send within a
second isnegligibly small.
7.The response time ofsendingaframe should be in order oftensof milliseconds.
The combination of low data rate and small physical size means that the end-to-end delay between nodes and the coordinator in a WSA is negligibly small compared to the timerequiredtosendabit.This fact makes theWSA medium accesscontrol(MAC) scheme illustratedbyFig. 12possible.
Coordinator SNI SN2 SN3 .., , .... i..f.i.fg.I..S....
N clockpulses ,iiLizI
"sSF~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I
_
III
...t7 .
,,
I'flifitLiTlUllt
.. s60iog".j 7~~~L~~~~~.LLLLLLL....LL.L... r I Wait~~~~~~~~lng:fbi.l si~e4ing data orlbea lramlE
A B, B2 C D E F
Fig.12 MAC of WSA
Accordingto thescheme, when no sensor node has frames to send, the coordinator continuously polls the nodes by sending a beacon frame followed by a sequence of clock pulses. (Beacon frames are depicted in Fig. 12 by light-colored square boxes on the top time line, and the clock pulses are depicted by narrow dark boxes.) After sending each clock pulse, the coordinator pauses to listen briefly. Hearing no data frame, it continues to send clock pulses.
After ithas sent the N-th pulse inthe sequence, it repeats a beacon frame followedby clock pulses.
Ifthe coordinator hears a data frame immediately after it sendsaclockpulse, it switches toreceive the data frame. At the end of the data frame, it sends clock pulses, listens in between pulses, switches toreceive if ithears a dataframe; otherwise, it sends a beacon frame after it completes the sequence ofNclockpulses.
Medium access by the sensor nodes are prioritized according to their ids: the smaller the id, the higher the priority. Specifically, when anode with id k hasadataframe to send, it waits until it hears a beacon frame or data frame and then counts theclockpulsesafter the end of the frame. If it hearsadata framebefore the kthclockpulse, it waitsuntil thedata frame ends and then counts clockpulses again. The node sends its data frame, whenever it hears the kth clock pulsein apulse sequence afterabeaconordata frame.
Fig. 12 illustratestheoperations of sensornodes with ids 1, 2, and3. Atthe start, bothSNI and SN2 have frames ready tosend.They wait untilthey hear the beacon frame from the coordinator.At timeA,SNI sendsits dataframe after ithears the first pulse following the beacon frame. Its transmission causes SN2 to wait. Since SNI no longerhas data frame to send, SN2 gets to send at time B2 after hearing two clock pulsesfollowing the data frame from SNi.AttimeC,nonode has data to send, and the coordinator begins to send a sequence ofNclock pulses. Suppose that SN3 awakes at time D inthe midst ofthe sequence. Itmustwait until it hears the next beacon and then starts to count. In this example, SN3 getstosend attimeFafter thethird clock pulse.
To estimate the worst-case response time in a WSA, suppose that the time to send a data or clock pulse is 10 microseconds. The coordinator takes 20 microseconds to send a clockpulse. The array has 250 sensors. Beacon and data frames are 100 bits and 50bits long, respectively. The times the coordinator takes to sendabeacon frame and 250 pulses are 2 milliseconds and 5 milliseconds, respectively. Suppose that threelowest-prioritynodes, SN248, SN249 and SN250, have data frames to send immediately after the coordinatorstarts tosend clockpulses. The SN250mustwait for the timerequired to send 4 sequences of clockpulses, a beacon frame and two data frames. So, the worst case response time of SN250 is approximately equal to 25 milliseconds.
VI. SUMMARY
This paper describes the PID and BAC versions of smart pantry. One canget aPIDpantry by adding a digital camera and pantry electronics to any dumb pantry that has compartments. The pantryowner can use it much like a dumb pantry when placing and removing objects. On the supplier's side,pictures ineachpurchase order sent by the pantry must beprocessedtoidentifytheobjects in the order and find their inventory control codes for locating the objects to be delivered. Theprocesscanbeautomated by the suppliers. The
jl..., ~:B[1,:E:,,-:t7 ---
paperdescribes anobjectidentification module designedfor this purpose.
From both thetechnical and usability points of view, the BAC version represents a reasonable compromise. The supplier can rely on the bar codes in purchase orders to identify and locate the objectstobe delivered. However, the usersmust scanthecontentof eachcompartmentatleast once before the last object in it is removed. For users who are willingto follow this rule, a bar-codeversion is sufficiently user friendly and reliable.
Much workremainstobe doneto access the merits of the OIM and WSA described here. We are refining the algorithms used in the OIMso that itcanaccurately identify objects in low resolution pictures. The WSA is being prototyped. An assumption is that the transmission is sufficient error-freeas to make error detection and recovery unnecessary. Some form of ARQ will be added if this assumption is foundinvalid in ourevaluation.
ACKNOWLEDGEMENT
This work is partially supported by the Taiwan Academia Sinica thematicprojectSISARL:SensorInformationSystems for Active Retirees and AssistedLiving. Theauthors wish to thank ProfessorA. C.Pang for herhelpinthedesign ofMAC forWSA.
REFERENCE
[1] Japan Assistive Products Association, http://www.iaspa.gJ.p/, April
2003.
[2] "Global Aging,"BusinessWeek,January31, 2005.
[3] Jane W. S.Liu,etal,"ReferenceArchitecture ofIntelligent Appliances for theElderly," Proceedings ofInternational ConferenceonSystem Engineering,LasVegas,NV, August 2005.
[4] Jane W. S.Liu,etal., "User Scenarios andDesignsof Smart Pantry, ObjectLocator and Walker's Buddy: Consumer Electronics for the Elderly," Technical Report No. TR-IIS-05-007, Institute of Information Science, AcademiaSinica,Taiwan, July2005.
[5] R. Glidden, et al. "Design of Ultra-low-cost UHF RFID Tags for SupplyChain Applications,"IEEECommunications, August 2004.
[6] http sisarl.org. SISARL: SensorInformationSystems(and Services)
forActiveRetireesand AssistedLiving,.
[7] P. H. Tsai, et.al,"Compliance Enforcement ofTemporal and Dosage
Constraints," Technical Report No. TR-IIS-06-006, Institute of
Information Science, Academia Sinica,Taiwan,October2005.
[8] S. Helal W. Mann, H. El-Zabadani, J. King, Y. Kaddoura, and E. Jansen, "The Gator Tech Smart House: A Programmable Pervasive Space,"IEEEComputer, March 2005.
[9] E.Dishman, "Inventing WellnessSystems for Aging inPlace," IEEE
Computer, May 2004.
[10] A. Pentland, "Healthwear: Medical TechnologyBecome Wearable,"
IEEEComputer,May2004.
[11] D. A.Ross, "Cyber Crumbs for Successful Aging with VisionLoss," IEEEPervasive Computing,April2004.
[12] ltX/arcitectuir.mitcdiihouse, Changing Places Consortium,MIT.
[13] http://www.cc.c,itech.edu/fce/seminar/fa98-iiifo/smart homes.html, FCE Smart HouseResearchSurvey,Georgia Tech.
[14] htt,:/w-\v,.tuturelialth.rochestcr.edu/, Center for Future Health, Universityof Rochester.
[15] [ttp./inar..n.d.vii giti.edu projects sathonemonitor.htn Marc smarthome,University of Virginia.
[16] J. C.Russ, TheImageProcessing Handbook, Fourth Edition, CRC
Press,July 2002.
[17] A. Prati, I. Mikic, C. Crana, M. M. Trivedi, "Shadow Detection Algorithms for Traffic Flow Analysis: AComparative Study."IEEE
ConferenceonIntelligent Transportation System, Oakland, California, pp.340-345, 2001.
[18] Y.Deng, B. S. Manjunath, C. Kenney, M. S. Moore, H. Shin,"An
Efficient Color Representation for Image Retrieval." IEEE Transactions on Image Processing,Vol.10,No.1,pp.140-147,January 2001.
[19] S. Belongie, J. Malik, J. Puzicha, "Shape Matching and Object Recognition Using Shape Contexts." IEEE Transactions on Pattern
Analysis and Machine Intelligence, Vol.24,No.4,April 2002.
[20] G. Mori, S. Belongie,J.Malik,"Efficient Shape Matching Using Shape Contexts." IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27, No.1,pp. 1832-1837 November,2005.
[21] J. Canny, "A Computational Approach Edge Detection." IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.8,
No.6,pp.679-698, November 1986.
[22] R. C. Gonzales and R. W. Woods. Digital Image Processing, Addison-Wesley Publishing Company, 1992.
