The Design of Cloud-based 4G/LTE for Mobile Augmented Reality
with Smart Mobile Devices
Bao-Shuh P. Lin, Wen-Hsiang Tsai, C.C. Wu, P.H. Hsu, J.Y. Huang, Tsai-Hwa Liu Dept. of Computer Science
Intelligent Information & Communications Research Center National Chiao Tung University
ABSTRACT
The system characteristics of 4G and beyond 4G broadband mobile system (BMS) are high data rate (throughput), low latency (delay), high mobility (speed), and high capacity. The current recognized 4G BMS needs to meet the requirements specified by IMT-Advanced of
ITU-T. Those BMSs include
3GPP-LTE/LTE-Advanced and IEEE 802.16e/m (WiMAX 1/WiMAX 2). In the meantime, the smart device (smartphone and tablet) with powerful CPU/GPU, HD digital camera, digital compass, GPS, and various sensors are becoming rapidly popular. In addition, the architecture and capability of cloud computing are getting adopted in various applications and services, a cloud-based 4G/LTE is one example of telecommunications services. With the combination of more deployments of cloud-based BMSs and increasing usages of smart mobile devices, there are many potential appealing applications and services with real-time and/or interactive features can be created. In this article, we explore the technology and applications of mobile augmented reality (MAR) on the cloud-based 4G
BMS (TD-LTE) and smart devices environment. The developed smart device-based MAR system
(SD-MAR) with the 4G/TD-LTE experimental network testbed is located at MIRC/BML in the campus of National Chiao Tung University. This testbed consists of several brandy dongles/tablets/smartphones (as UE), two NSN TD-LTE base stations (as eNodeB), one core network (as EPC), and cloud-based servers and data center. To study the technology and applications on SD-MAR system, we have integrated research teams/people specialized in the areas of cloud computing, smart device technology, 4G broadband mobile system, computer vision and image processing, gesture recognition, computer graphics and rendering,
and system integration. The applications discussed in the article include real-time accurate navigation/tourism for indoor and outdoor, collaborative urban design, and multiuser interactive motion learning system in the mobile environment.
1. Introduction
Cloud computing technology has been applied to various applications and services, a cloud-based broadband mobile system (BMS) for telecommunications services is a good example. 4G BMS based on 3GPP LTE/LTE-A is becoming the mainstream of global telecommunications service operators. Currently, the network providers in North America (US/Canada) and APEC (Japan/South Korea/Australia) are very aggressive in deploying 4G/LTE networks. In the meantime, the technology advancement of mobile smart devices (SD) such as smartphones and tablets are
becoming rapidly popular. The SD is
well-equipped with high resolution camera(s), GPS, digital compass, color display, GPU/powerful CPU, and many different kinds useful sensors. With the combination of more deployment of cloud-based 4G/LTE and increasing usages of smart mobile devices there are many potential compelling applications and services with real-time and/or interactive features in the mobile environment can be created.
Mobile augmented reality (Mobile AR or MAR) [2, 3] is defined as AR applications running in a mobile environment. Augmented reality is a live view of physical real-world whose elements are augmented (enhanced) by virtual computer-generated data and/or images. AR technology has been around for more 20 years but limited to a niche market such as military, medical, and heavy industry
2013 IEEE Seventh International Symposium on Service-Oriented System Engineering 2013 IEEE Seventh International Symposium on Service-Oriented System Engineering
applications. The advancement of smart mobile device (smartphone and tablet), AR applications move to mainstream set of application in the consumer market, particularly for Mobile AR applications on SD (SD-MAR) operating in the 4G/LTE mobile network.
In this article, we address the technology and applications of cloud-based 4G/LTE networks for mobile augmented reality (MAR) on the smart devices. The proposed prototype SD-based MAR systems (SD-MAR) over the experimental 4G/LTE network testbed is located at MIRC/BML (Microelectronics and Information Research Center/Broadband Mobile Lab.) in the campus of National Chiao Tung University. This testbed several brandy terminals (USB dongles, tablets, and smartphones as 4G UE), two NSN TD-LTE (LTE FDD) base stations (as 4G eNodeB), one core network (as 4G EPC) and a cloud computing-based platform, called NSC-C3AP, providing IaaS and PaaS services. NSC-C3AP is a cloud computing & communication application platform (C3AP) sponsored by the National Science Council (NSC) of Taiwan. Figure 1 describes the experimental 4G/LTE network testbed.
Figure 1. The experimental cloud-based 4G/LTE network testbed
Besides the Introduction Section, the rest of this article is divided into four sections. Section 2 addresses the system architecture of SD-MAR on cloud-based 4G/LTE network. The proposed applications of SD-MAR system is described in Section 3. The study of applications performance analysis of SD-MAR in term of quality of service (QoS) and quality of user esperience (QoE) is addressed in Section 4. Finally, the conclusion remarks is in Section 5.. 2. Architecture of SD-MAR on Cloud-based
4G/LTE Network
In the era of 3G mobile network (WCDMA, CDMA2000, and TD-SCDMA) or 3.5G mobile (HSUPA, HSDPA, and HSPA+) network [4] due
to the limited (low) data rate of network transmission and the large system delay (latency), the mobile augmented reality application can only be run in the mobile device following the download of the application from the backend server (e.g. Apple’s App Store). Figure 2 indicates the system configuration of SD-MAR in the 3G/3.5G mobile network. There are several drawbacks of executing MAR application program on the standalone SD including the limited computing power, storage volume, memory size and the big energy consumption. In the paper [1, 5] for mobile visual search in 3G networks, it points out that network latency remains the bottleneck for the system. For 3G connections, it also shows the average energy consumption in send image mode is almost 3 times as much as send feature mode. This is due to the slow transmission speed.
Figure 2. The standalone MAR application run on client SD in 3G/3.5G network
2.1. System Characteristics of 4G/LTE Network
Due to the advancement of wireless technology in the areas of MIMO, OFDM and other enhancement, the system characteristics of 4G and beyond 4G networks are in Table 1.:
Table 1 System characteristics of 4G/LTE network
System feature Performance/remarks
High data rate (throughput)
15 Mbps ~ 100 Mbps Low delay (latency) 20 ms ~ 40 ms
High mobility 300 km/h ~ 500km/h
High capacity more small cell/BS
deployment
The system performance improvement (from 3G/3.5G to 4G/LTE)) has made the interactivity capability and real-time applications possible. Figure 3 shows the client-server (or distributed) configuration as the application execution platform for 4G/LTE network with client SD and cloud server.
Figure 3. Client-server (distributed) configuration for 4G/LTE network
2.2 Layer Architecture for SD-MAR System Development
The system architecture for SD-MAR cloud-based 4G/LTE network is organized into three layers: application, middleware, and platform. The application layer consists of SD-MAR application program and MAR interface. The middleware layer covers SD-MAR API and functions of client, connectivity, and cloud-based server. The platform layer includes library/virtual machine, hardware abstract layer, OS (Android, Windows Mobile, and iOS) and hardware. Figure 4 illustrates this system layer architecture for SD-MAR.
Figure 4. System layer architecture for SD-MAR with
cloud-based 4G/LTE
In Figure 4 indicates the SD-MAR
applications can run over 4G/LTE network with SD supports operating system: Android,
Windows Mobile and Apple iOS. The
hardware for smart mobile device may include ARM, CPU/GPU, digital compass, GPS, digital camera, Wi-Fi, and several sensors
2.3 Configuration for SD-MAR System
Figure 5 shows the system configuration for SD-MAR with cloud-based 4G/LTE network as the application platform on NCTU campus.
Figure 5. The proposed overall system configuration of SD-MAR
3. Applications Proposed for SD-MAR on Cloud-based 4G/LTE Network
According to Visiongain’s analysis of [3], the most salient application segments to SD-MAR and papers [6, 7, 8, 10, 11, 14] are listed in the following:
ಷġ Location-based services, ಷġ Mobile gaming, ಷġ Education and reference, ಷġ Geo Social networking, ಷġ Multimedia and entertainment, ಷġ Travel/Navigation, and
ಷġ Lifestyle and healthcare With the available resources and expertises, we have decided to investigate the cloud-based SD-MAR applications including:
ಷġ Navigation/Tourism applications for indoor (e.g. store and mall) outdoor (e.g. campus and city),
ಷġ Collaborative urban design, and
ಷġ Multiuser interactive motion learning system in mobile environment
3.1 Navigation/Tour Guidance Applications on SD-MAR
A study on indoor navigation by
augmented reality using mobile devices [15] and outdoor tour guidance by car driving in park areas using augmented reality technique [16] had been prototyped by Prof. Tsai of NCTU and his graduate students. The prototype systems were implemented in a LAN and Wi-Fi configuration and without cloud computing environment. Figure 6 and Figure 7 show the architectures of their implementations. The disadvantages of those systems are in the following:
ಷġ Mobile device does all the processing of computer vision (CV) that is computing intensive and big energy consumption,
ಷġ System only works in the LAN/Wi-Fi environment that is no mobility and not available everywhere, and
ಷġ System is not always available and also not scalable
(a) (b)
Figure 6. Indoor AR-based navigation using computer vision techniques [15]
Figure 7. Outdoor AR-based tour guidance using computer vision techniques [16]
The proposed new cloud-based architectures of SD-MAR with 4G/LTE network for indoor navigation application and for outdoor tourism are shown in Figure 8 and Figure 9 respectively. In those new architectures, the computing intensive CV can be done in the cloud side. By using this new approach, the SD-MAR-based indoor navigation and outdoor tourism services can be available everywhere and anytime when the infrastructure of 4G/LTE network is ready.
Figure 8.Cloud-based architecture of MAR for indoor navigation application
Figure 9. Cloud-based architecture of MAR for outdoor tourism application
3.2 Collaborative Urban Design with SD-MAR
With advances in technologies related to smart devices and broadband mobile wireless network, the computer-supported simulations for urban design can move from a conventional model-based approach to simulative activities in actual urban spaces. The urban design application scenario take advantages of such technologies and therefore be treated as a testbed for the next generation SD-MAR system for collaborative urban design.
Followed the analyses of activities and design requirements related to collaborative urban design, we come out a design for s multiuser SD-MAR to support collaborative urban design. Then making of information contents includes 2D and 3D forms. Implement a simulative design system by integrating of the AR application and the position-tracking module. Figure 10 shows the concept of urban design with a mobile smart device while Figure 11 illustrates the multiuser collaborative urban design with cloud-based 4G/LTE by using SD-MAR.
Figure 10. The concept of urban design using a mobile smart device Local Network Computer A Computer B Video surveillance car Camera System A Camera System B Affixed on Affixed on Pad AP
Figure 11. Multiuser collaborative urban design with cloud-based using SD-MAR
3.3 Multiuser Interactive Motion Learning System with SD-MAR
Interactive motion learning system can be possible due to the high data rate and low latency of 4G/LTE and the high performance computing power of cloud-based server. Figure 12 shows an intelligent robot playing important roles in an interactive motion learning application. The system architecture of instructor-learner-through-robot as a multiuser interactive motion learning system is illustrated in Figure 13. The real prototype system in Figure 14 demonstrates the capability of robot in a cloud-based 4G/LTE for SD-MAR.
Figure 12. Robot roles in a cloud-based 4G/LTE with SD-MAR for an interactive motion learning application
Figure 13. System architecture of robot learning system in SD-MAR
Figure 14. Prototype system demo
4. System Performance Analysis of QoS/QoE for SD-MAR
The system performance analysis [9, 12, 13] for applications of cloud-based 4G/LTE for MAR run on SD includes search/retrieval accuracy, system latency, transmission delay, and end-to-end latency. Those performance measurements are more related to quality of service (QoS). From a real-time application point of view, the response time is essential to the SD-MAR user so is the interactive applications. The image or video quality for display is very important. Those two performance indices satisfying with user expectation or not are related to user experience, called quality of experience (QoE).
Following the application prototypes, we will further investigate the QoS for navigation/tourism applications and address the QoE for collaborative urban design application and interactive motion learning system.
5. Conclusion
This paper has addressed the importance of cloud computing technology that impacts the architecture of SD-MAR with 4G/LTE network, consequently creates a variety of new and compelling applications for the consumer segment. The designs of navigation/tourism applications for indoor and outdoor, collaborative urban design, and multiuser interactive motion learning system are discussed. Finally, the QoS/QoE of SD-MAR applications running on a cloud-based 4G/LTE will be further investigated
Acknowledgements
The authors would like to express their
sincere appreciations to NSC for the
sponsorships of 4G/LTE(TDD) experimental
network deployment project (100-2219-E-009-006-) and exploration of SD-MAR system with 4G/LTE network environment project (101-2221-E-009-043-), also many thanks to MOE (Ministry of Education) of Taiwan for the establishment of I2CRC (Intelligent Information and Communications Research Center) and MIRC of NCTU.
References
[1] Bernd Girod, Vijay Chandrasekhar, David M Chen, Ngai-Man Cheung, Radck Grzeszczuk, Yuriy Reznik, Gabriel Takacs, Sam S Tsai, and Ramakrishna Vedantham, “Mobile Visual Search,” in IEEE Signal Processing Magazine Special Issue on Mobile Media Search, 2011
[2] Tobias H Hollerer and Steven K Feiner, “Mobile Augmented Reality,” Chapter Nine in book Telegeoinformatics: Location-Based Computing and Services: H. Karimi and A. Hammad (eds.), Taylor & Francis Books Ltd, Jan. 2004
[3] “Mobile Augmented Reality 2011-2016,” Visiongain, 2010
[4] Papagiannakis, Singh, and Thalmann, “A survey of mobile and wireless technologies for augmented reality systems,” MIRA Lab, Uinv. Of Geneva, Switzerland and Naval Postgraduate School, CA, USA, p.30, 2008, also in Computer Animation and Visual Worlds, 19(1):3, 2008
[5] Junfeng He, Tai-Hsu Lin, Jinyuan Feng, and Shih-Fu Chang, “Mobile Product Search with Bag of Hash Bits,” ACM MM-11, 2011
[6] Feiner, MacIntyre, Hollerer, and Webster, “A Touring Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring Urban Environment,” Personal Technologies, 1(4), pp. 208-217, 1997. [7] Reitmayr and Schmalasteig, “Location
based Applications for Mobile Augmented Reality,” Vienna University of Technology, Austria, p. 10, 2002
[8] Korah, Wither, Tsai, and Azuma, “Mobile Augmented Reality at the Hollywood Walk of Fame,” Nokia Research Center, Hollywood, CA, p6, 2007.
[9] Azuma, Neely, and Daily, “Performance Analysis of an Outdoor Augmented Reality Tracking System that Relies Upon a Few Mobile Beacons,” Proceedings ISMAR, pp. 101-104, Oct. 2006.
[10] Chen, Tsai, Hsu, Singh, and Girod, “Mobile Augmented Reality for Books on a Shelf,” in IEEE Workshop on Visual Content Identification and Search, p. 6, July 2011.
[11] Chen, Tsai, Vedanham, Grzeszczuk, and Girod, “Streaming Mobile Augmented Reality on Mobile Phones,” Proceedings ISMAR, Oct. 2009.
[12] Lee, Zhang, Fang, Srinirasan, Iyer, and Newell, “Accelerating Mobile Augmented Reality on a Handheld Platform,” Proceedings IEEE, pp. 419-426, 2009. [13] Lee, Zhang, Fang, Srinirasan, Iyer, and
Newell, “Performance Characteristics and Optimization of MAR on a Handheld Platform,” IEEE International Symposium on Workload Characterization, Oct. 2009 . [14] Study Tour Pixel 2010 – University of
Twente, “The future directions of mobile augmented reality applications,” Univ. of Twente, in Toward the use of augmented reality indoors, 2011.
[15] M.Y. Hsieh and Wen-Hsiang Tsai, “A Study on Indoor Navigation by Augmented Reality and Down-Looking Omni-Vision Techniques Using Mobile Devices,” Technical Report, Institute of Multimedia Engineering, Department of Computer Science, NCTU, Hsinchu, Taiwan, July, 2012
[16] B.C. Chen and Wen-Hsiang Tsai, “Tour Guidance by Car Driving in Park Area Using Augmented Reality and Omni-Vision Techniques,” Technical Report, Institute of Multimedia Engineering, Department of Computer Science, NCTU, Hsinchu, Taiwan, July, 2012
[17] C.C. Wu, “DUIWAPA – Interactive Motion Learning Platform,” Technical Report, Department of Computer Science, NCTU, Hsinchu, Taiwan, July, 2012
