© Springer International Publishing Switzerland 2014
L.T. De Paolis and A. Mongelli (Eds.): AVR 2014, LNCS 8853, pp. 3–18, 2014.
DOI: 10.1007/978-3-319-13969-2_1
Chi-Fu Lin1( ), Pai-Shan Pa2, and Chiou-Shann Fuh1
1 Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan
[email protected], [email protected]
2 Graduate School of Toy and Game Design, National Taipei University of Education, Taipei, Taiwan
Abstract. This paper intends to propose an interactive system that allows users to remotely control device and combine augmented reality content with the applica- tion. It is possible to generate games or education applications. Using augmented Reality and remote-control module as starting point, we design a MAR (Mobile Augmented Reality) game that contains shooting game, roadblock hindrance, and traffic signal game. Then use the mobile remote-control module to control the device. Finally, use augmented reality technology to allow real objects to interact directly with the virtual objects. As a platform to integrate mobile phone, we complete the development of augmented reality remote-control module applica- tion. The difference from the general AR applications is that our proposed method combines the remote-control objects to interact with virtual objects.
Keywords: MAR · Game design · Remote control
1 Introduction
1.1 Background and Motivation
With the development of mobile Augmented Reality (AR) enabling technology, there have been many mobile AR applications, services, and contents. Due to the demand for software resources in recent years, more and more large, relatively rising smart- phone hardware, augmented reality technology in the past can only be used on the computer, because the pixels of the camera to enhance graphics processing progress, can now also be applied in the mobile phone software. Because the phone is easy to carry, augmented reality is able to generate more interactive applications, such as interactive advertising, navigation, books, and games.
Nowadays augmented reality still needs to use the mark to reach a recognition tar- get. Mark setup as well as mark portability still has much room for improvement.
Thus many studies aim to improve recognition technology. For example, use the mark to replace a specific image augmented reality, or to replace the electronic information other than the image tag identification, such as geographic information to achieve the effect of augmented reality and Global Positioning System (GPS).
Lack of dynamic expansions of the new features or styles is disadvantage of physic toys. If we want to add new features or a new style must be additional to buy, but it
will waste space and increase costs. Thus, the concept is to make augmented reality mobile platform. Use digital content within the practical function of the mobile device to match hardware and software to achieve the effect of human-computer interaction.
This paper consists of five sections. In Section 2, we will summarize related works.
The system overview is in Section 3. Then, we describe implementation of the system in Section 4. Finally, we will conclude in Section 5.
2 Related Work
2.1 Development and Application of Augmented Reality Definition of the Augmented Reality
Augmented reality is a form of virtual reality. Virtual reality allows users to fully integrate into the computer-generated virtual environment. We cannot see the reality of its surrounding environment. Augmented Reality allows the user to see the reality of the environment, as well as the synthesis of virtual objects in the real environment superimposed or contrasted with virtual reality. Therefore, Augmented Reality aug- ments real environment rather than completely replaces real environment. Augmented reality has the following three characteristics: combination of real and virtual, real- time interactivity, and three-dimensions [1].
Principle of the Augmented Reality
Real and virtual images are combined and displayed to the user on the small screen inside the helmet [1, 2]. The equipment and technology, coupled with immaturity, high cost, inconvenience to carry, cannot be accepted for general users. But the popularity of PC and network cameras makes a new way to achieve augmented reality. Use a webcam to identify the real environment. The video is output to PC monitors. Hardware price in this mode of operation is quite cheap. Therefore the popularity of augmented reality increases [3]. Another paper describes an RF-based approach to mobile augmented reality [4].
Because of the rapid development of mobile devices, augmented reality has ex- tended from personal computers to mobile devices. The highly interactive nature of augmented reality with its user has given rise to various augmented reality applica- tions for mobile devices [5]. AR can be used in handheld games, it has more novel play experience for people [6]. After integrating the virtual and real world, it can im- prove the convenience for people. AR can be used in classroom as teaching tool. A study analysis by the use of the AR book in classroom indicates that the interaction can strengthen the study result for all ages [7]. The authors research how augmented reality can increase the selective and sustained attention of children with autism dur- ing object discrimination therapies and elicit more positive emotions [8].
There are several SDKs and platforms to realize augmented reality. Qualcomm Vuforia [9] is a software platform for Android, iOS, and package for Unity3d [23]
that enables app to recognize mark from a Vuforia target database or in the Vuforia Cloud Recognition Service. It allows users to track objects such as two-dimensional planar mark, three-dimensional cube, and cylinder-shape object. It also supports real- time text recognition. ARtag and ARtoolkit are C/C++ open source platforms which support tracking 2D mark and displaying openGL [10]. BazAR is a computer vision library based on matching and feature point detection. In particular, it is able to quick- ly detect and register known planar objects in images [11].
2.2 Analyses and Discussions
The human-computer interaction toys can be classified into two types. The first type is mobile device combined with Augmented Reality. Another one is physical toy communication with mobile device. We discuss and analyze two types below.
Interactive Mobile Device and Augmented Reality
Use the camera on mobile device to recognize target and generate 3D models on the screen. Augmented reality has for years been one of the focuses in mobile phone ap- plication development, ranging from mere interaction to marketing, games, naviga- tion, and so on. The highly interactive nature of augmented reality with its user has given rise to various augmented reality applications for mobile phones.
With Hoops AR [12], user simply views the ticket using a mobile device, and then the basketball ticket turns into an interactive basketball game. By using user’s finger to shoot the basketball to control the speed of the shot. An iPad app, Barbie Digital Mirror [13] uses augmented reality to let kids try on makeup. When user looks in the interactive mirror, user dips a make-up tool into pretend makeup, and it appears on her face. The iButterfly [14] app generates a butterfly on the screen combining AR with Global Positioning System (GPS) information, and then user can catch butterfly which becomes a coupon. Popar [15] Toys uses Augmented Reality technology to create an immersive reading experience. It is designed to change the way users inter- act and experience stories, adventures, and learning. Table 1 shows a comparisons of various AR software.
Interactive Toys and Mobile Device
It generates different experiences by interactive toys and mobile device using sensor on mobile device connected to toys and interacting with each other.
The first type is mobile device connected with toy via Bluetooth, Infrared Module, Wireless, and so on by using mobile device for remote control. User can control toys through mobile device remotely when the connection is successful. Helicopter and remote-control car use this type. Table 2 shows comparisons of various toys com- bined with mobile software.
The second type uses toys to control game. Apptivity [16] action games include toys and an iPad app. The iPad recognizes the toy thanks to Mattel’s patented “Active Touch” technology when kids use the physical toys to play the app game. The Apptivity action games will include several popular themes such as Fruit Ninja, An- gry Birds, and Batman: The Dark Knight Rises. Adding different shapes of touch points to toy’s bottom to achieve recognition. It recognizes shapes and triggers the corresponding event by touching the toy on iPad.
Augmented reality interactions have been classified into three categories. The first category uses computer and webcam to identify the specified tag. The disadvantage is lack of mobility in this method. The second category is a popular method in recent years. This method uses mobile device and camera to identify the specified tag, then generates 3D models on the screen. A user clicks on the touch screen to interact with digital content on the mobile device. The advantage is more mobility in this method.
However the real interaction experiences are not enough. The third category is similar
to the second category, but it provides more realistic experiences. Through augmented reality detection effect, user directly uses fingers to interact with digital content in a real environment. Table 3 shows a comparison of augmented reality interactions.
Table 1. Comparisons of Various AR Software
Advantage Disadvantage
AR Drone [17] The effect of the AR and play against other players.
Need to buy expensive AR Drone aircraft.
Word Lens [18]
Simple and practical transla- tion software can be used without Internet.
Four translation languages (English, Italian, French, and Spanish) are too few.
iButterfly [14]
Butterfly shape combined with advertising, marketing effect by capturing various butterflies.
Single marketing function, low user viscosity.
Hoops AR [12]A basketball court for the
mark, to enhance game play. Need to prepare a mark for the game.
Layar [19] Intuitive navigation screen.
Need to turn on the camera and In- ternet connection to use navigation, consume much electricity.
Table 2. Comparisons of Various Devices Combined with Mobile Software
Mobile Device Devices
AR Drone [17] Bluetooth: Mobile device connects with toy via Blue- tooth.
Bluetooth: Receive mo- bile device’s signal to control toy.
RoboMe [20] Microphone: voice command recognition, and remote vid- eo control.
Camera: Face detection and tracking
User will be able to change the face that ap- pears on the screen and the voice to different accents.
Apptivity [16] The iPad recognizes the toy thanks to Mattel’s patented
“Active Touch” technology.
Touch point with differ- ent shapes at toy bottom.
Table 2. (Continued) Silverlit R/C Car
[21] Bluetooth: Mobile device connects with toy via Blue- tooth.
Bluetooth: Receive mo- bile device’s signal to control toy.
Silverlit helicopter [21]
Table 3. Comparison of Augmented Reality Interactions
Illustration pictures Description
Virtual button of AR [9]
Through setting the virtual buttons on the marker, when virtual but- ton occlusion is detected, our system triggers corre- sponding event.
Popular aug- mented reality on the mobile device.
A user clicks on the touch screen to interact with digital content on the mobile device.
Augmented reality on an early computer
Webcam identification of the main tag. Users also need to use other specified tag to interact with digital content.
3 System Overview
Our proposed system enables the user to remotely control device by mobile device and let physical remote control device or toy to interact with virtual object. We analyze operational processes summarized into two modes in Figures 1 and 2. There are five major roles in the system consisting of user, mobile device, remote-control device, tag, and virtual object. In Mode 1, user can use mobile device to remotely control device. Use the mobile device’s camera to recognize the predefined tag and render virtual object. The remote-control device via virtual button interacts with virtual object.
The difference between Modes 2 and 1 is that Mode 2 adds a tag in the remote- control device. When mobile device identifies the tag, it will produce a corresponding virtual object, allowing original device to have more diversification. The concept is somewhat similar to the movie “Transformers [22]” and each device can be trans- formed into any types. Mode 2 allows more interesting interaction.
Fig. 1. System process diagram - Mode 1
Fig. 2. System process diagram - Mode 2 User
Mobile Device Play
Remote-Control Device
Control
Virtual Object Interactive Recognition
Tag Rendering
Interactive
User
Mobile Device Play
Remote-Control Device and Tag
Control
Virtual Object Interactive Recognition
Tag Rendering
Interactive
Virtual Object
Recognition
Rendering
4 Results and Disc
4.1 Structure and Proc In this study, we use augm remote-control module. We Traffic signal theme featur car. The system’s hardware application can be divided Android. We use Unity 3D
4.2 Development Tools The design tools of the gam 1. 3D computer graphics so 2. Augmented Reality: Uni 3. Arduino software: Ardui 4. Eclipse: Eclipse 3.6.2 wi Table 4 shows a classi tools.
Fig.
cussions
cess
mented reality features and interactive games to constru e choose a remote-control car in our design MAR ga res the design of the interactivity of the users and rem e and software architecture is shown in Fig. 3. The mob
into two platforms generally, Apple’s iOS and Goog [23] to develop a mobile AR game.
s
me are categorized into the following four items:
oftware: Autodesk Maya
ity 3D, Augmented Reality (Vuforia™) ino 1.0.5
ith Android Development Tools (ADT)
ification of the various augmented reality game des
. 3. Hardware and software architecture
ct a me.
mote bile le’s
sign
Tabl Name
Hardware
Arduino
Arduino Sh
Car
Bluetooth m
Software
Autodesk M
Vufori
Unity 3
Eclipse (A SDK)
4.3 The Development F Remote-Control Car Asse Remote-control car compa ment Kit (SDK). The toy co order to have much flexib
le 4. Analysis of Hardware and Software
Diagram Description
Uno
Arduino is an open source electronic prototyping platform based on flexible easy-to-use hardwar and software.
hield
Shields are board that can be plugge on top of the Arduin PCB extending i capabilities.
It consists of tw servomotors and power device.
module
Receiving an transmitting signal via Bluetooth.
Maya
Make a traffic signa model and anima tion.
ia Development of AR
main suite.
3D
Integrate Vufori suite, and output th APP on mobil phone.
Android Use Bluetooth func
tion to contro Arduino car.
Flow embled
nies in the market do not release any Software Devel ompany only releases a control app to allow user to use bility and programmability, we choose Arduino fina n- cs m e, re
ds ed no ts
wo a
nd ls
al a-
R
ia he le c- ol
lop- e. In ally.
Arduino is a single-board microcontroller designed to make the process of using elec- tronics in multidisciplinary projects more accessible. User can connect the board with many different devices, such as Light-Emitting Diode (LED), temperature sensor, speaker, servo motor, infrared sensor, Bluetooth chip, Ethernet, XBee, Radio Fre- quency Identification (RFID), Global Positioning System (GPS), and so on. Arduino is great development tool intended for everybody who wants to easily and quickly create his own application. For example it could be a blinking LED or a system to control home appliances via Wireless network.
The following are brief assembly steps:
1. Assemble Arduino Uno board and Arduino Shield board.
2. The servomotors connect with corresponding pins on Arduino Shield board.
3. Assemble car and Arduino boards.
4. Add battery power supply.
5. Connect Bluetooth module with corresponding pins on Arduino Shield board.
After assembling the car, then write an Arduino blink program. We write a control servo code in setup function and loop function. After finishing remote-control car assem- bly in Fig. 4, we start to implement connection function in mobile device and remote- control car. After remote-control car modification in Fig. 5, the "Cars" outfit is virtual.
Fig. 4. Remote-control car
Fig. 5. After remote-control car modification Mobile Device and Remote-Control Car
There are many different ways to implement connection function, such as Bluetooth, Wi-Fi, infrared, RFID, and so on. Because the Bluetooth has been widely utilized in mobile device, we choose it as our connection module.
Bluetooth connections comprise mobile device and Arduino car.
Implement connection function in mobile device.
1. Mobile device scans the Bluetooth device and connects with it.
2. Send control signals to Arduino car.
Implement connection function in Arduino car.
1. Add a function control to the Arduino car consisting of forward, backward, left turn, right turn, and stop.
2. Receive signals from mobile device and trigger the corresponding event.
Augmented Reality Interaction Design
Use Unity 3D and Qualcomm Vuforia SDK to develop the application. The game aims to construct traffic signals for education. We design some tags to be recognized.
With Image Recognition AR, user holds camera over tags, and event happens in a virtual environment. The following are brief development steps:
1. Launch the Unity and load Vuforia kit.
2. Use the material dragged into the Unity project library.
3. Identify tag loading Unity.
4. Set up a virtual button on the scene.
5. Combine the connection function in Unity.
The Car Tracked via a Marker
After remote-control car modification in Fig. 5, we use the car’s pattern as marker by Vuforia kit. When mobile device’s camera recognizes target, we can get the car’s position. Via a car’s position, we can calculate distance between a car in real world and the object in virtual world or collision detection.
Game Mechanism and Interface
The game mechanism created from Augmented Reality integrates real and virtual features. There are three scenarios in our game mechanism that contains shooting game, roadblock hindrance, and traffic signal game.
Shooting game
Shooting game scenario uses a tag on remote-control car and sets a corresponding 3D model. This scenario’s system process diagram uses Mode 2 in Fig. 2. When mobile device’s camera recognizes target, the weapon will appear at the top of the car in Fig. 6. It is similar to the movie “Transformers” that cars can transform into the other type. We design some virtual enemies in the game and player can control the car and shoot to be a winner.
Roadblock hindrance
This scenario’s system process diagram uses Mode 1 in Fig. 1. The game scenario to identify tags is roadblock hindrance in Fig. 7. When a user drives and crashes into roadblocks, a car will stop forward in Fig. 8. In this scenario, we realize the virtual objects to interact with physical objects.
Traffic signal game
This scenario’s system process diagram uses Mode 1 in Fig. 1. The screen of game playing is shown in Fig. 9. There are five buttons consisting of “forward”, “left”,
“right”, “back”, and “stop” on the screen. User can control car via those buttons and the virtual objects shown on the screen. The use of a set of virtual buttons in the pro- gram allows the game to generate feedback on the car's behavior. We design a game situation when the user drives through a green light then the pass has been shown on the screen in Fig. 10. On the other hand, when the user drives through a red light, then the warning has been shown on the screen in Fig. 11. By this way, user will learn the traffic rules.
Fig. 6. The weapon will appear at the top of the car
Fig. 7. A roadblock hindrance on the screen
Fig. 8. Drive and crash into roadblocks
Fig. 9. The green traffic light is safe to pass
Fig. 10. The green traffic light is safe to pass
Fig. 11. Passing through the red traffic light causes warning
5 Conclusions
We construct virtual object settings and create the appearance by traffic signal as well as various animation of virtual traffic light, for the reality and remote toys as starting point. We uses the mobile device to control the car. Finally, it allows the car to inter- act directly with the virtual traffic light, using augmented reality technology. We complete the development of augmented reality car application with a platform to integrate mobile phone. It combines the real toys with virtual objects.
Our proposed remote-control module can be a car, robot, boat, helicopters, and so on. AR interaction with different devices produces more interesting applications. In this paper, we propose two system process modes and design three play scenarios.
Combining AR and remote-control car gets more different experience and pleasure.
Future studies should further investigate user friendliness and usability with respect to remote-control toys and AR system. The development of additional interaction tech- nology to create Augmented Reality content allows users to understand and use easily.
Acknowledgment. This research was supported by the National Science Council of Taiwan, R.O.C., under Grants NSC 98-2221-E-002 -150 -MY3 and NSC 101-2221-E-002 -194, by Winstar Technology, Test Research, and Lite-on.
References
1. Azuma, R.: A Survey of Augmented Reality. Presence: Teleoperators and Virtual Envi- ronments 6(4), 355–385 (1997)
2. Milgram, P., Kishino, F.: A Taxonomy of Mixed Reality Visual Displays. IEICE Transac- tions on Information System 77, 1321–1329 (1994)
3. Roland, J.P., Holloway, R.L., Fuchs, H.: A Comparison of Optical and Video See-through Head-Mounted Displays. Proceedings of Society of Photographic Instrumentation Engi- neers: Telemanipulator and Telepresence Technologies 2351, 293–307 (1994)
4. Regen, H.T., Specht, R.: A Mobile Passive Augmented Reality Device-mPARD. In: Pro- ceedings of International Symposium on Augmented Reality, pp. 81-84. Munich (2000) 5. Wagner, D., Schmalstieg, D.: First Steps towards Handheld Augmented Reality. In:
Proceedings of IEEE International Symposium on Wearable Computers, White Plains, NY, pp. 127–135 (2003)
6. Chang, Y.N., Koh, R.K.C., Duh, H.B.L.: Handheld AR Games—a Triarchic Conceptual Design Framework. In: Proceedings of ISMAR, Basel, Switzerland, pp. 29–36 (2011) 7. Billinghurst, M., Duenser, A.: Augmented Reality in the Classroom. Computer 45(7),
56–63 (2012)
8. Escobedo, L., Tentori, M., Quintana, E., Favela, J., Garcia-Rosas, D.: Using augmented reality to help children with autism stay focused. In: Proceedings of Pervasive Computing, Budapest, Hungary, vol. 13, pp. 38–46 (2014)
9. Qualcomm: Vuforia (2014). http://www.vuforia.com
10. Abawi, D.F., Bienwald, J., Dorner, R.: Accuracy in Optical Tracking with Fiducial Mark- ers: An Accuracy Function for ARToolKit. In: Proceedings of the IEEE/ACM Internation- al Symposium on Mixed and Augmented Reality, Washington DC, pp. 260-261 (2004) 11. EPFL, Bazar (2014). http://cvlab.epfl.ch/software/bazar/
12. Bigplayar, Hoops AR (2014). http://www.bigplayar.com/hoopsar.html 13. Mattel, Barbie® Digital Makeover (2014).
https://itunes.apple.com/us/app/barbie-digital-makeover/id549872444?mt=8 14. Dentsu, iButterfly (2014). http://ibutterfly.hk/chi/
15. Digital Tech Frontier, Popar Toys (2014). https://popartoys.com/
16. Mattel, Inc. Apptivity (2014). http://www.mattelapptivity.com/
17. Parrot SA, AR Drone (2014). http://cdn.ardrone2.parrot.com/
18. Quest Visual, Word Lens (2014).
http://itunes.apple.com/tw/app/word-lens/id383463868?mt=8 19. Layar, Layar (2014). https://www.layar.com/
20. WowWee, Robome (2014). http://www.wowwee.com/robome/
21. Silverlit, Interactive Bluetooth R/C (2014).
http://silverlit.com/brand/interactive-bluetooth-rc
22. IMBD, Transformers (2014). http://www.imdb.com/title/tt0418279/
23. Unity, Unity3D (2014). http://unity3d.com/
