Mobile Application of Interactive Remote Toys with Augmented Reality
Chi-Fu Lin*, Pai-Shan Pa†, and Chiou-Shann Fuh*
* Deaprtment of Computer Science and Information Engineering, National Taiwan University
† Graduate School of Toy and Game Design, National Taipei University of Education
Abstract— Recently, because the rapid development of mobile devices, augmented reality has extended from personal computers to mobile devices. The highly interactive nature of augmented reality with its user has given rise to various augmented reality applications for mobile devices, ranging from mere interaction to marketing, games, navigation, and so on. As such, augmented reality has for years been one of the focuses in mobile devices application development. There are many physical toys with integrated electronic sensor chips or wireless communication mechanism, so that the toys are no longer boring and it produced new ways to interact. In this study, a human- computer interactive interface between physical toys and virtual objects was designed with the aim of incorporating augmented reality.
I. INTRODUCTION A. Background and Motivation
With the demand for software resources in recent years, more and more large, relatively rising smart phone hardware, augmented reality technology in the past can only be used on the computer, because the pixels of the camera to enhance graphics processing progress, and now also to be applied in the mobile phone software. Because the phone is easy to carry, augmented reality is able to generate more interactive applications, such as interactive advertising, navigation, and games.
Nowadays augmented reality still needs to use the mark to reach a recognition target. Mark setup as well as mark portability still have much room for improvement. Thus many studies aim to improve recognition technology. For example, use the mark to replace a specific image augmented reality, or to replace the electronic information other than the image tag identification, such as geographic information to achieve the effect of augmented reality and Global Positioning System (GPS).
Unable dynamic expansions of the new features or styles are disadvantage of physic toys. If you want to add new features or a new style must be additional to buy, but it will waste space and increased costs. Based on the above, the concept is to make augmented reality mobile platform. Use digital content within the practical function of the mobile device to match hardware and software to achieve the effect of human-computer interaction.
II. RELATEDWORK
A. Development and Application of Augmented Reality
1. Definition of the Augmented Reality
Augmented reality is a form of virtual reality. Virtual reality allows users to fully integrate into the computer- generated virtual environment. We cannot see the reality of its surrounding environment. Augmented Reality allows the user to see the reality of the environment, as well as the synthesis of virtual objects in the real environment superimposed or contrasted with virtual reality. Therefore, Augmented Reality augments real environment rather than completely replace real environment. Augmented reality has the following three characteristics: combination of real and virtual, real-time interactivity, and three-dimensions [1].
2. Principle of the Augmented Reality
In early years it can be done by two Head-Mounted Displays (HMD): optical HMD and image-based HMD that users can integrate into augmented reality combined with either real or virtual images. Optical HMD is shown in “Fig.
1”. Use camera on the helmet to identify tag. Directly project virtual object onto the transparent lens to achieve the effect of augmented reality. Image type HMD in “Fig. 2,” uses a helmet camera identification mark and reality. Real and virtual images are combined and displayed to the user on the small screen inside the helmet [1, 2]. The equipment and technology, coupled with immaturity, high cost, inconvenience to carry, cannot be accepted for general users.
But the popularity of PC and network cameras makes a new way to achieve augmented reality. Use a webcam to identify the real environment. The video is output to PC monitors.
Hardware price in this mode of operation is quite cheap.
Therefore the popularity of augmented reality increases [3].
Another paper describes an RF-based approach to mobile augmented reality [4].
Because the rapid development of mobile devices, augmented reality has extended from personal computers to mobile devices. The highly interactive nature of augmented reality with its user has given rise to various augmented reality applications for mobile devices [5]. AR can be used in
handheld games, it have more novel play experience for people [6]. After integrating the virtual and real world, it can improve the convenience for people. AR can be used in classroom as teaching tool. A study analysis by the use of the AR book in classroom, the interaction can strengthen the study result, and it applies to all ages [7].
Fig. 1 The principle of the optical HMD and its appearance [1].
Fig. 2 The principle of the images HMD and its appearance [1].
B. Analyses and Discussions
The human-computer interaction toys can be divided into two types. The first type is physical toy communication with mobile device. The second type is mobile device combined with Augmented Reality. We analyze and discuss two types as below.
B.1. Interactive Toys and Mobile Device
Interactive toys and mobile device use sensor on mobile device connected to toys and interacting with each other to generate different experiences. The first type uses toys to control game. Apptivity action games consist of toys and an iPad app. Kids use the physical toys to play the app game.
The iPad recognizes the toy thanks to Mattel’s patented
“Active Touch” technology. The Apptivity action games will include several popular themes such as Angry Birds, Fruit Ninja, and Batman: The Dark Knight Rises. Toy recognition is achieved by adding different shapes of touch points to toy’s bottom. By touching the toy on iPad to recognize shapes and trigger the corresponding event. The second type uses mobile device for remote control. Mobile device connects with toy via Bluetooth, Infrared Module, Wireless, and so on. When the connection is successful, user can remotely control toys through mobile device.
Remote-control car and helicopter use this type. Table I shows a comparisons of various toys combined with mobile software.
B.2. Interactive Mobile Device and Augmented Reality The highly interactive nature of augmented reality with its user has given rise to various augmented reality
TABLE I
COMPARISONS OF VARIOUS TOYS COMBINED WITH MOBILE SOFTWARE.
Mobile Device Toy
Apptivity The iPad recognizes the toy thanks to Mattel’s patented
“Active Touch” technology.
Different shapes touch point to toys bottom.
RoboMe Microphone: voice command recognition, and remote video control.
Camera: Face detection and tracking
User will be able to change the face that appears on the screen and the voice to different accents.
Silverlit R/C
Car Bluetooth: Mobile device connects with toy via Bluetooth.
Bluetooth: Receive mobile device’s signal to control toy.
Silverlit helicopter
AR.Drone Bluetooth: Mobile device connects with toy via Bluetooth.
Bluetooth: Receive mobile device’s signal to control toy.
TABLE II
COMPARISONS OF VARIOUS AR SOFTWARE.
Advantage Disadvantage
Hoops AR
A basketball court for the marker, to enhance game play.
Need to prepare a marker to the game.
Layar Intuitive navigation screen.
Need to turn on the camera and Internet connection to use navigation, consume much electricity.
Word Lens
Simple and practical translation software can be used without Internet.
Four translation languages (English, Italian, French, and Spanish) are too few.
iButterfly
Butterfly shape combined with advertising, marketing effect by capturing various butterflies.
Single marketing functions, low user viscosity.
AR Drone The effect of the AR and
play against other players. Need to buy expensive AR Drone aircraft.
applications for mobile phones, ranging from mere interaction to marketing, games, navigation, and so on. As such, augmented reality has for years been one of the focuses in mobile phone application development. Use the camera on mobile device to recognize target and generate 3D models on the screen. With Hoops AR, user simply views the ticket using a mobile device, and then the basketball ticket turns into an interactive basketball game.
Control the speed of the shot by using user’s finger to shoot the basketball. Barbie Digital Mirror is an iPad app that lets kids try on makeup using augmented reality. User dips a make-up tool into pretend makeup and it appears on her face when she looks in the interactive mirror. The iButterfly app combines AR with Global Positioning System (GPS) information to generate a butterfly on the screen, and then user can catch butterfly which becomes a coupon. Popar Toys uses Augmented Reality technology to create an immersive reading experience. It is designed to change the way users interact and experience stories, adventures, and learning. Table II shows a comparisons of various AR software.
Augmented reality interactions have been classified into three categories. The first category uses computer and webcam to identify the specified tag. The disadvantage is lack of mobility in this method. The second category is a popular method in recent years. This method uses mobile device and camera to identify the specified tag, then generate 3D models on the screen. A user clicks on the touch screen to interact with digital content on the mobile device. The advantage is more mobility in this method. However the real interaction experiences are not enough. The third category is similar to the second category, but it provides more realistic experiences.
Through augmented reality detection effect, user directly uses fingers to interact with digital content in a real environment.
Table III shows a comparison of augmented reality interactions.
III. RESULTSANDDISCUSSIONS A. Structure and Process
In this study, the use of augmented reality features interactive games to construct a remote toy. Traffic signal theme features the design of the interactivity of the users and remote toy. The system’s hardware and software architecture is shown in Fig. 3. The mobile application can be divided into two platforms generally, Apple’s iOS and Google’s Android. Table IV shows a development of mobile device software.
B. Development Tools
The design tools of the game are categorized into the following four items:
1. 3D computer graphics software: Autodesk Maya 2. Augmented Reality: Unity 3D, Augmented Reality
(Vuforia™)
3. Arduino sofeware: Arduino 1.0.5
4. Eclipse: Eclipse 3.6.2 with Android Development Tools (ADT)
Table V shows a classification of the various augmented reality game design tools.
C. The Development Flow
C.1. Remote-Control Car Assembled
Remote-control car companies in the market do not release any Software Development Kit (SDK). The toy company only releases a control app to allow user use. In order to have much flexibility and programmability, we choose Arduino finally.
Arduino is a single-board microcontroller designed to make the process of using electronics in multidisciplinary projects more accessible. User can connect the board with many different devices, such as Light-Emitting Diode (LED), temperature sensor, speaker, servo motor, infrared sensor, Bluetooth chip, Ethernet, XBee, Radio Frequency Identification (RFID), Global Positioning System (GPS), and TABLE III
COMPARISON OF AUGMENTED REALITY INTERACTIONS. Illustration pictures Description
Augmented reality on an
early computer
Webcam identification of the main tag. Users also need to use other specified tag to interact with digital content.
Popular augmented
reality on the mobile device.
A user clicks on the touch screen to interact with digital content on the mobile device.
Virtual button of
AR
Through augmented reality detection effect, user directly uses fingers to interact with digital content in a real environment.
so on. Arduino is great development tool intended for everybody who wants to easily and quickly create his own application. For example it could be a blinking LED or a system to control home appliances via Wireless network.
The following are brief assembly steps:
1. Assemble Arduino Uno board and Arduino Shield board.
2. The servomotors connect with corresponding pins on Arduino Shield board.
3. Assemble car and Arduino borads.
4. Add battery power supply.
5. Connect Bluetooth module with corresponding pins on Arduino Shield board.
After assembling the car and then write an Arduino blink program. We write a control servo code in setup function and loop function. After finishing remote-control car assembly in Fig. 4, we start to implement connection function in mobile device and remote-control car.
C.2. Mobile Device and Remote-Control Car
There are many different ways to implement connection function, such as Bluetooth, Wi-Fi, infrared, RFID, and so on.
Because the Bluetooth has been widely utilized in mobile device, we choose it as our connection module.
Bluetooth connections comprise mobile device and Arduino car.
Implement connection function in mobile device:
1. Mobile device scans the Bluetooth device and connects with it.
2. Send control signals to Arduino car.
Implement connection function in Arduino car:
1. Add a function control to the Arduino car consisting of forward, backward, left turn, right turn, and stop.
2. Receive signals from mobile device and trigger the corresponding event.
C.3. Augmented Reality Interaction Design
Use Unity 3D and Qualcomm Vuforia SDK to develop the application. The game aims to construct traffic signals for education. We design some tags to be recognized. With Image Recognition AR, user holds camera over tags, and event happens in a virtual environment. The following are brief development steps:
1. Launch the Unity and load Vuforia kit.
2. Use the material dragged into the Unity project library.
3. Identify tag loading Unity.
4. Set up a virtual button on the scene.
5. Combine the connection function in Unity.
D. Game Mechanism and Interface
The game mechanism created from Augmented Reality integrates real and virtual features. The screen of game playing is shown in Fig. 6. There are five buttons consisting of “forward”, “left”, “right”, “back”, and “stop” on the screen.
User can control car via those buttons and the virtual objects shown on the screen. The use of a set of virtual buttons in the program allows the game to generate feedback on the car's behavior. We design a game situation when the user drives through a green light then the pass has been shown on the screen in Fig. 7. On the other hand, when the user drives through a red light, then the warning has been shown on the screen in Fig. 8. By this way, user will learn the traffic rules.
Fig. 3 Hardware and software architecture.
Fig. 4 Remote-control car.
TABLE IV
DEVELOPMENT OF MOBILE DEVICE SOFTWARE.
Advantage Disadvantage
iOS
Development of resource-rich unified development platform, a complete audit of APP trafficking mechanism.
Requires Macintosh computer and user becomes a paid member of the development.
Android Requiring a lower level of development equipment.
Too many different device specifications lead to maintenance difficulties.
Fig. 5 After remote-control car modification.
Fig. 6 The screen of game playing.
Fig. 7 The green traffic light is safe to pass.
Fig. 8 Passing through the red traffic light causes warning.
IV. CONCLUSIONS
Augmented Reality and remote toys as starting point, we use traffic signal to construct virtual object settings and create the appearance as well as various animation of virtual traffic light. Then use the mobile device to control the car. Finally, use augmented reality technology to allow car to interact directly with the virtual traffic light. As a platform to integrate
TABLE V
ANALYSIS OF HARDWARE AND SOFTWARE.
Name Diagram Description
Software
Autodesk Maya
Make a traffic signal model and
animation.
Vuforia Development of
AR main suite.
Unity
Integrate Vuforia suite, and output the APP on mobile
phone.
Eclipse (Android
SDK)
Use Bluetooth function to control
Arduino car
Hardware
Arduino Uno
Arduino is an open-source electronics prototyping platform based on
flexible, easy-to- use hardware and
software.
Arduino Shield
Shields are boards that can be plugged on top of the Arduino PCB extending its
capabilities.
Car
It consists of two servomotors and a
power device.
Bluetooth module
Receiving and transforming
signals via Bluetooth.
mobile phone, we complete the development of augmented reality car application. The difference with the general AR applications is that our proposed method combines the real toys with virtual objects.
Google Glass was announced in 2012. Our proposed interactive method is very suitable for Google Glass. Google Glass can directly catch the tag and generate virtual models immediately. However using mobile device’s camera to capture image is more uncomfortable. By this interaction, we can produce many different applications. In the future, we can combine with additional sensors to create a better interactive experience.
ACKNOWLEDGMENT
This research was supported by the National Science Council of Taiwan, R.O.C., under Grants NSC 98-2221-E- 002 -150 -MY3 and NSC 101-2221-E-002 -194, by Winstar Technology, Test Research, and Lite-on.
REFERENCES
[1] R. Azuma, “A Survey of Augmented Reality,” Presence:
Teleoperators and Virtual Environments, Vol. 6, No. 4, pp. 355- 385, 1997.
[2] P. Milgram and F. Kishino, “A Taxonomy of Mixed Reality Visual Displays,” IEICE Transactions on Information System, Vol. 77, pp. 1321-1329, 1994.
[3] J. P. Roland, R. L. Holloway, and H. Fuchs, “A Comparison of Optical and Video See-through Head-Mounted Displays,”
Proceedings of Society of Photographic Instrumentation Engineers: Telemanipulator and Telepresence Technologies, Vol. 2351, pp. 293-307, 1994.
[4] H. T. Regenbrecht and R. Specht, “A Mobile Passive Augmented Reality Device-mPARD,” Proceedings of International Symposium on Augmented Reality, Munich, pp.
81-84, 2000.
[5] D. Wagner and D. Schmalstieg, “First steps towards handheld augmented reality,” Seventh IEEE International Symposium on Wearable Computers, Proceedings., pp. 127–135, 2003.
[6] Y.N. Chang, R.K.C. Koh, and H. B.-L Duh, “Handheld Ar Games—a Triarchic Conceptual Design Framework,
“Proceedings of ISMAR, IEEE, pp. 29-36, 2011.
[7] M. Billinghurst, and A. Duenser, “Augmented reality in the classroom,” Computer, vol. 45, no.7, pp. 56-63, July 2012
