多重螢幕的文字分割顯示透過無線傳輸在手機裝置上的方法與實作

全文

(1) . . . . . . . . . . .

(2) . . . . . . . . . . . . . . . . Method and implementation of displaying text on multiple display devices on mobile phone. . %. &. !" '. (. #. $. !). *. +. .. ,. /. -. 0. '. 1. (. 2. 3. 4567.

(3)

(4). . . . . . . . . . . . . . Method and implementation of displaying text on multiple display devices on mobile phone. $. %. &. '. (. ! ). ". # *. +. ,. Student. Fu-Yuang Huang. Advisor. Dr. Hsi-Lu Chao. -./.0.1.2.3.. . . 4.,.5.. . A Thesis Submitted to College of Computer Science National Chiao Tung University in partial Fulfillment of the Requirements for the Degree of Master of Science In Computer Science Jan 2008. .. Hsinchu, Taiwan, Republic of China. /. 0. 1. 2. 3. 4567.

(5)

(6). . . . . . . . . . . . . .. 3. !. ". #. ... .. ........................$. %. &. '. (. ). *. , . .+. .. - / 0 1 2 3 . . 6. 7. 3 8. . . 6. ;. =. >. . 2O. P. a. T. b . . . Q. . . AN c. ?. @. R. d. e. AB. S. f. T. g. v. . w. Ax. . WLQ. A. ¨. ©. µ. ¶. . . P. Ð. . d. L. . Ð. . ª. ¸ . . . e. ·. rÂ Å Æ Â Å L Ï. . ¤ v Ç. Ñ ¤. V. y. . ¬. ¤. ¹ Lº. . F X. Q. . É . ®. » £. Ê. . . . h. ½. 4 , :.. i. . . [. . \. LM. ]. . ^. _. ¢. . L . . L. s. `.

(7). ]. £. . . ¤. ¥. J. ¾ ¿ À Á ¿ p Â Ã Ì L¢. . tM. ¦. K. Q ±. Ä . h ¥. ° i. L. u. >. . . . . 2. V. AP. . ². q q rQ P. N. `. L 2~. . . K. 17. l m n o p L.q q r s. ¾ ¿ À Á ¿ p Â Ë. t.. K. K. {. ¡. J. J. . A. k. J. k }. . I. Z. ¯ ¼. H. A. e Lk . G Y. A|. Q. d. È. F. WLj. {. . «. V. Lz g. E. WLF. i. 3 9. <. D. U. h. A L Lf 2~. C. 7. @. ³. § ´. G. Al m n o p .. Ín Á Î o o .

(8) Ò. Ó. « . Þ . . ß . ë ]. 5AÔ . ô Ò. úLû. õ. . Ó. ö. . þ. à «. . ÷. ]. . ¦. ×. . Ø. Ù. . ã. . . ý. ÍÖ. . L¡. 5

(9) ü. Õ. . . ó. . ä. å. . ø ³. . . ÷ . Õ. ³. Ù . æ Õ. . ù. Lá H. Lì Õ. ×. . ii. =. ². Ø. >. _. Û. 2â. Ù î. ï. Ü LÝ. . Ú. . ç î. è. . ?. Ù @. 6 ð. ñ. ¢ é. Ú. . . ò. ê ñ. . . t.. N. K. Íí ù. Ú ]. Ú L@. . . . . à. ¦ . . Y . K. ¤ t.

(10) Method and implementation of displaying text on multiple display devices on mobile phone studentFu-Yuang Huang. AdvisorsDr. Hsi-Lu Chao. Degree Program of Computer Science National Chiao Tung University ABSTRACT. In recent years, mobile phone have become a common communication device in our life, with more large bandwidth and more interesting network service, people can use their handset to access these service easily, such as HTTP, Email, Ftp event On-line game. Even from the low-end handset to high-end handset , these mobile phones’ screen, display area from 70*92,120*144, 240*240 or 640*480, their display area are limited to various font size and network application. When use such device to view a content with mounts of word/picture with fixed lines/width, if handset force to re-arrange the layout of the text/picture, it will result to user have many reading confusion with the original content. A real case, when we open a console to connect to Email or BBS, the console screen is fixed at width of 80 ASCII chars in one line, and total 25 lines. In this thesis, we propose a novel method of displaying text on multiple devices with socket, and implement the real work to prove the method can load a single content, display on different handsets, use the master device to browse and scroll up/down, view the full content on different handsets and display well simultaneously. In this thesis, we prove the proposed algorithm by implementation. The results show that the proposed method provides a new mechanism to display text on multiple handsets, and the future applications on mobile phone.. iii.

(11) ! 5. . . . G. 8. G. b. c B. y . _. . T. o". p . M. ..

(12) ¯. ` . J. K´ Õ¼. ¢. 5 T. + ú. ]. û. . . ~. Ö. ¡. e. . j. W. § ´. . . Ë. ". C. /. © b. µ . ¶. . Á. I. .

(13). h. Ì. i. . Í. Î. . . Ï. f. [. 0. P. \. %. . r. 2. E. 3. 4. F. T. st. ). . 1. ^ _. U. M. ". u. . `. . . . a. w . r st. .. . sv. V . opq e. . Q. ]. 5. . C D.

(14). . .. ¤. A . u. . . . z . . KKpQ . ¹. ¬ º. Ã. . . /. ) opq. . Â. .. ". YZ. .

(15). > %. . C . < 0. '. . > ·. ©. . £. «. -. 4. n. ! b. . + ,. X. m. '. ¢. ª. ·. T.

(16). /. W. e. . ¸. :. . ". . g. l. V. . 3. . T . . ·. J. . 0. k. . * 0. %. U. . 0. ) /. O. T. . /. ¡. ¨. À. e. p. >. . . B. N. j. . ( A. .. C. . . M. . @. K.. . 9. ¦. . Ø. J. Kã. . . . ?. >. . '. 4. i. . . . Ä. P. ». A. >. ® ¼. B. C. ¯ i 9. Å. ~. ©. Æ. ½. g. ¿. . . T g. {. . T. b. . Ç Ð. {. > Ñ. °. ± '. C. T . ®. ". » Ò. Ó. È Ô. . Q T. í. ¥. ~. >. J. h. . . 3. g. . . &. = L. f. . . <. F S. . ; . ¿. Ê. ×. ì. T. . ³. J S. . 4. .. . . . FJ. E. . o². Ë S. . . K.. É ~. . 3. {. . . K. }. J. ¾. ;. (1 e. %. KR. |. d. _. $. . J. {. . . C.

(17). . FJ. >. . #. . d. .

(18). I. z g . . :. . x. . 9. H ". . ". 6 7 .. . . ©. . î. R. ï. KT. ï ô . É 5. A. B. Ä å . ©. . Ù g. 1. ã. R. T. T. T. Ù. Ú. ä. J ª. :. ¢. . Ú. M. ð. 5 å. . ñ !. Fü. 1. T. . . õ ý. k. Æ. ö. þ. iv. oÛÜ. Ý. oæ. . . |. . ã. Þ. ß . © _ ã. KKã. © ©. _ . ß. ~ T. ª ò. T. ~ . ê. k %. ë. oà ç. . Ø. è . J. á. â. é ê. :. +ó ÷. ø. â ë. . ~ ùJ. L K.

(19) Contents . ................................................................................................................. i. ABSTRACT ................................................................................................................iii K ............................................................................................................... iv Contents ...................................................................................................................... v List of Figures ............................................................................................................ vi Chaper 1 Introduction ......................................................................................... 1 1.1 Traditional Console output.................................................................. 1 1.2 Display Limitation Of Mobile Handset Device................................... 3 1.3 Motivation............................................................................................ 4 Chaper 2 The Proposed Method.......................................................................... 6 2.1 Initial State........................................................................................... 6 2.2 Information change state..................................................................... 7 2.3 Display mode change state................................................................... 9 2.4 Final state........................................................................................... 11 2.5 Multiple display examples ................................................................. 11 Chaper 3 Development Environment................................................................ 29 3.1 Software Architecture........................................................................ 29 3.2 User Interface..................................................................................... 30 3.3 Enhancements and Applications ....................................................... 32 Chaper 4 Conclusion.......................................................................................... 33 Reference................................................................................................................... 34. v.

(20) List of Figures Figure 1.1 To access FTP service by using traditional terminal console .................................. 2 Figure 1.2 To access BBS service by using traditional terminal console ................................. 2 Figure 1.3 Wrap the text layout to meet the screen size for reading......................................... 3 Figure 1.4 Scale the picture to view in full screen mode ......................................................... 4 Figure 2.1 Flow chart of the initial state ................................................................................. 7 Figure 2.2 Flow chart of the information change state ............................................................ 8 Figure 2.3 Simple adjustment of the display area.................................................................... 9 Figure 2.4 Flow chart of the display mode change state........................................................ 10 Figure 2.5 Flow chart of the final state ................................................................................. 11 Figure 2.6 Two handsets with different display size.............................................................. 12 Figure 2.7 Open document on master device ........................................................................ 12 Figure 2.8 Master display the text in original display mode .................................................. 13 Figure 2.9 Master initial the network layer ........................................................................... 13 Figure 2.10 Slave initial the network layer ........................................................................... 14 Figure 2.11 Master connect to slave ..................................................................................... 14 Figure 2.12 Master send HELLO message to slave............................................................... 15 Figure 2.13 Slave response with HELLO message ............................................................... 15 Figure 2.14 Master query local display information.............................................................. 16 Figure 2.15 Slave query local display information................................................................ 16 Figure 2.16 Master send data size message to slave.............................................................. 17 Figure 2.17 Master send raw data of text to slave ................................................................. 17 Figure 2.18 Slave send local display information to master .................................................. 18 Figure 2.19 Master adjudge display area of each device ....................................................... 18 Figure 2.20 The result is Master: 30*15, and slave: 35*15 ................................................... 19 Figure 2.21 Master send display area message to slave......................................................... 19.

(21) Figure 2.22 Master send CHANGEMODE message to slave................................................ 20 Figure 2.23 Slave response with CHANGEOK message to master ....................................... 20 Figure 2.24 Master change to new display mode .................................................................. 21 Figure 2.24 Master display the partial text on the screen ...................................................... 21 Figure 2.25 Slave change to new display mode .................................................................... 22 Figure 2.26 Slave display partial text on the screen .............................................................. 22 Figure 2.27 User view and input with device........................................................................ 23 Figure 2.28 Master send CLOSE message to slave ............................................................... 23 Figure 2.29 Slave response with CLOSEOK message .......................................................... 24 Figure 2.30 Master close socket and network ....................................................................... 24 Figure 2.31 Master change to original display mode............................................................. 25 Figure 2.32 Master display in previous display mode ........................................................... 25 Figure 2.33 Slave close socket and network ......................................................................... 26 Figure 2.34 Slave change to original display mode............................................................... 26 Figure 2.35 Slave display in previous display mode ............................................................. 27 Figure 2.36 Another example of display text on multiple display devices ............................. 28 Figure 3.1 Software architecture of the implementation........................................................ 30 Figure 3.2 AP Main UI and get network information on WM6 QVGA handset..................... 30 Figure 3.3 Connection setting and load document on WM6 QVGA handset ......................... 31 Figure 3.4 AP Main UI and get network information on WM6 Landscape QVGA handset ... 31 Figure 3.5 Connection setting and load document on WM6 Landscape QVGA handset........ 31. vii.

(22) Chaper 1. Introduction. Nowadays, most mobile phone equip with connect to Internet with CSD/GPRS, event with 3G communication. People use mobile phones to connect to Internet to browse web pages, send/read Email/MMS, view photos and play on-line games. These applications bring a gorgeous reading experience to handset users. But for some application, such as traditional console to open a connection to access email, ftp, even BBS(Bulletin Board system), the tradition console is defined as a text output area of width about 80 ASCII characters width and column of 25 lines. But to recent various mobile phones, with different display screens and different font types, their screen may not meet the requirement of display such a content, to fit displaying issue, application usually need to load the content, or lunch the application in a wrap width/height display layout, but it usually make a big confusion and trouble to mobile phones users. Not only the console applications, even other viewer , like web browser, picture views, photo editor, when user try to use their handset to view large page content, they often get some confusion about the original content when they got the re-arrange layout.. 1.1 Traditional Console output Base on the definition of the terminal window size, not matter in Windows, Unix, Linux, any operating system, the windows size is 80 characters in width, and 25 lines in height. In the terminal windows, all command send and response the result to local user. User can input any text command in the terminal windows to access remote data, act as a client and acquire any service such get web page data, receive email, login BBS to read news, transfer files with FTP protocol, or play on-line game such as MOD.. 1.

(23) Figure 1.1 To access FTP service by using traditional terminal console. Figure 1.2 To access BBS service by using traditional terminal console. 2.

(24) 1.2 Display Limitation Of Mobile Handset Device Mobile phones from the basic function , just dial up and talk to other people, to the rich function about the play music, take snapshot by the attached camera, even equip with GPS to track their travel waypoint. No matter the low-end or hi-end device, their screens differ in width and height, their colors and pixels also are different to each others. By the way, the fonts display on the screen also differs, some handsets use only one fixed font style and size, but other handsets can change their font style to Italic, Bold or underline. But with these different display sizes, some display sizes , the columns and lines are usually less then the real layout of the web page, picture, or the terminal windows size, so application usually wrap the text, scale the picture , but the result are usually hard to read.. Figure 1.3 Wrap the text layout to meet the screen size for reading. 3.

(25) Figure 1.4 Scale the picture to view in full screen mode. 1.3 Motivation When mobile phones are quipped with advanced modules such as WIFI, Bluetooth, IrDA , people can use these modules to access Internet services, to connect to other mobile phones , and with the common open platform, I want to connect two mobile phones together, display single document on these two display screens, to view synchronously on the devices. With the new protocol, user have two or more mobile phones can read document with a large virtual display screen. With the handset wireless connection capabilities, I define the handshake of the display size of different model handset, and try to gather their different screen display as a large display. Try to open a document, and split the portion data on each device in the device group, make the view size and layout keep the original style the document present. Provide a handshake to connect two devices, a new protocol to negotiate handset display properties. With the split display mechanism, User can put two or even more device together as a big screen, use the physical wireless connection to act as server/client handsets, to operate. 4.

(26) viewing one documents with the device array. To split one document on multiple display devices, the idea come from the my work experience, when we development the email client to retrieve mail from the POP3 server, or send email with SMTP server, the terminal console size are much different in PC and on the handset, even the later development, I wish to use the console window to access BBS or other Internet services, but I feel hard to view the command text in my mobile handset, due to my small display size. So I try to construct the new idea and apply it to patent, and final I got the patent of US Patent No. 7242369, China Patent No. ZL 2005 1 0118170.0, Taiwan Patent No. I276991, but the real implementation is not realized due to the product schedule. Until now, most handsets may be implemented on base of Microsoft Windows Mobile OS, it provide a basic development with many APIs and user can build customized applications, with the new defined socket APIs, we can get an open environment to setup the connections, send/receive data, and my proposed protocol can implemented as a real module to run in the common handsets with the open API.. 5.

(27) Chaper 2. The Proposed Method. In this chapter, I propose a novel method of displaying text on multiple devices, use sockets to connect two mobile phones, load contents and send commands to each device, to browse content simultaneously. Display text on multiple display devices. The multiple display devices include a master device with a master display and at least one slave device with a slave display. The method includes establishing communications between the master device and the slave device, dividing the text into a first part and a second part based on a first display dimension of the master display and a second display dimension of the slave display, transmitting the second part to the slave device, and simultaneously displaying the first part on the master display and the second part on the slave display. For each individual line of text that is displayed on both the master display and the slave display, a fraction of text is displayed on the master display and a remaining fraction of text is displayed on the slave display. The protocol consist four states, the initial state, the information change state, display mode change state, and final state. All commands in each states, are human readable text, each is prefixed with MTD, and next field is the role, and the final field is the operation name. The prototype of the command is MTD_role_operation, the detail description will list in the following.. 2.1 Initial State In the initial state, master and slave device initial their network layer and the socket modules, then master connect to slave, after setup the connection with slave, master send the MTD_MASTER_HELLO message to slave, slave receive the message and make decision with the message, if slave choose to reject the operation , slave will send back MTD_SLAVE_REJECT message to master, and the whole protocol just exit now. Other wise,. 6.

(28) slave agree with the current operation and send back MTD_SLAVE_HELLO message to master.. Figure 2.1 Flow chart of the initial state. 2.2 Information change state In the state, master and slave need to exchange the display information of their screen display screen and the data of document, master and slave query their local display screen size, then master send text data to slave, master send MTD_MASTER_DATASIZE_###, ### is the size of the documents, and send the raw data of the document to slave, slave get the size of the document and receive the full data of the documents, and just send back the local display information to master with command MTD_SLAVE_INFO_C###L###, ### means the width of the columns and the height of lines, after master gather the information of slave’s device, master need to adjudge display area of each device, I just propose the simplest way to the adjustments, that is to combine the width of the two device , and choose the smaller height of the two devices, the reason just try to extend the virtual screen width as the sum of the two. 7.

(29) device, and for the layout correction of the text displaying, the smaller height is the suitable result to view the documents. After. master. make. the. adjustment. of. the. display. area,. master. send. MTD_AREA_M###*###S###*### to slave, in the command, consist of master display ration and slave display ratio information.. Figure 2.2 Flow chart of the information change state. 8.

(30) Figure 2.3 Simple adjustment of the display area. 2.3 Display mode change state After master and slave negotiate the final display ratio, master will send MTD_MASTER_CHANGEMODE command to slave, slave receive the command and ready to change to new display mode, just send back the MTD_SLAVE_CHANGE message to master. Master and slave will change to the new display mode in the current state, and user will see a new display mode on the two display device, and user can view or do any operation with the application.. 9.

(31) Figure 2.4 Flow chart of the display mode change state. 10.

(32) 2.4 Final state When user want to exit the multiple display mode, user request the operation , and master will send MTD_MASTER_CLOSE command to slave, slave just response MTD_SLAVE_CLOSEOK message to master, master and slave need to close their socket and network functions gracefully and change their display mode to previous display mode, and the protocol just complete.. Figure 2.5 Flow chart of the final state. 2.5 Multiple display examples In the case, if we use two handset, one screen display is 30 columns and 15 lines, and the other handset is 35 columns and 20 lines, after open a documents about 80 columns and 25 lines, the message flow and view result are listed as following. 11.

(33) Figure 2.6 Two handsets with different display size. Figure 2.7 Open document on master device. 12.

(34) Figure 2.8 Master display the text in original display mode. Figure 2.9 Master initial the network layer. 13.

(35) Figure 2.10 Slave initial the network layer. Figure 2.11 Master connect to slave. 14.

(36) Figure 2.12 Master send HELLO message to slave. Figure 2.13 Slave response with HELLO message. 15.

(37) Figure 2.14 Master query local display information. Figure 2.15 Slave query local display information. 16.

(38) Figure 2.16 Master send data size message to slave. Figure 2.17 Master send raw data of text to slave. 17.

(39) Figure 2.18 Slave send local display information to master. Figure 2.19 Master adjudge display area of each device. 18.

(40) Figure 2.20 The result is Master: 30*15, and slave: 35*15. Figure 2.21 Master send display area message to slave. 19.

(41) Figure 2.22 Master send CHANGEMODE message to slave. Figure 2.23 Slave response with CHANGEOK message to master. 20.

(42) Figure 2.24 Master change to new display mode. Figure 2.24 Master display the partial text on the screen. 21.

(43) Figure 2.25 Slave change to new display mode. Figure 2.26 Slave display partial text on the screen. 22.

(44) Figure 2.27 User view and input with device. Figure 2.28 Master send CLOSE message to slave. 23.

(45) Figure 2.29 Slave response with CLOSEOK message. Figure 2.30 Master close socket and network. 24.

(46) Figure 2.31 Master change to original display mode. Figure 2.32 Master display in previous display mode. 25.

(47) Figure 2.33 Slave close socket and network. Figure 2.34 Slave change to original display mode. 26.

(48) Figure 2.35 Slave display in previous display mode. 27.

(49) Figure 2.36 Another example of display text on multiple display devices. 28.

(50) Chaper 3. Development Environment. In this chapter, our proposed protocol is implemented on real mobile phone and simulator. We choose VC2005 and Windows Mobile 6 SDK to develop the application on the WM6 handset. With WM6 simulator and two real WM6 mobile phones, we implement our method on the real target device and verify the mechanism.. 3.1 Software Architecture The view of the software architecture, program first gather the display information, the information contains the local display size, and will negotiate in the later handshake. After gather information, program setup the wireless connection, it base on the windows socket API, the physical layer may be WIFI or Bluetooth, even the IR transmitter. To setup up network, also need to invoke BSD like socket API, such as listen(), bind(), accept() , send() and recv(). To catch command in a independent process, create a persistent thread, in the thread will send/receive command data, according to input command, master and slave handset will response to various action, like scroll page of text display, refresh screen. In the final step, when user requests to exit the protocol, send close command. When the thread receives close command, just finalize network functionality, and graceful exit the protocol.. 29.

(51) Figure 3.1 Software architecture of the implementation. 3.2 User Interface To acquire user operation, program design a simple user interface, for the use to setup network connection, Bluetooth configuration, and a working area to output debug message, and the main window to display text on master and slave device.. Figure 3.2 AP Main UI and get network information on WM6 QVGA handset 30.

(52) Figure 3.3 Connection setting and load document on WM6 QVGA handset. Figure 3.4 AP Main UI and get network information on WM6 Landscape QVGA handset. Figure 3.5 Connection setting and load document on WM6 Landscape QVGA handset 31.

(53) 3.3 Enhancements and Applications In our protocol and the implementation, just process the simplest content about pure text data. But lot of multimedia contents, such as graphic data (BMP, JPEG, GIF), vector file (Flash) or movie clips (AVI, MPEG, 3GP) files. With applying our proposed protocol, we can communicate two handsets, negotiate display capability, and with a suitable engine to decode such multimedia contents, we can split the presentation on two or more heterogeneous mobile devices. And develop more fantasy applications, for example, a group mobile phones grouped as a huge TV wall, displaying a interesting movie synchronously, or design a funny mobile KTV service, using two handsets, one act as the master device, the other act as the slave device. They both playback the same music, but just display the different lyric on each device, and the users can held such device to sing a romantic love song with together.. Figure 3.6 Multiple display application – mobile KTV service 32.

(54) Chaper 4. Conclusion. In this thesis, we propose a novel method of displaying content on multiple devices, with the protocol we calculate two different handset display capabilities, and calculate the text content properties to figure out a best display dispatching on mobile phones, setup the connection. between. heterogeneous. devices,. display. contents. and. operate. with. view/page-up/page-down, the content display action simultaneously on different devices. I also implement the server/client application on device to verify our mechanism work well on the real products. In this thesis, we only consider loading single text content and display that on two heterogeneous mobile phones, we just prove the concept can be implemented on nowadays mobile phones with limited wireless connection capability, the protocol and mechanism can be extend to multiple device array and accepted as a transport module, to view more complicated multimedia content with a collection of mobile phones, event the hot application such as on-line game or movie show.. 33.

(55) Reference [1] Wai Yip Lum, and Francis C.M. Lau, “User-Centric Adaptation of Structured Web Documents for Small Devices, ” Advanced Information Networking and Applications 19th International Conference, pp. 507- 512, 2005. [2] Vatsa, R. and Kumar, V, “Role of media transformation in multimedia messaging, ” Personal Wireless Communications, pp. 258-262, 2005. [3] Hongjiang Zhang, and Wei-Ying Ma, “Adaptive content delivery on mobile Internet across multiple form factors, ” Multimedia Modelling Conference, pp. 5-7, 2004. [4] Won-Ho Choi, Dong-Gi Im ,and Min-Soo Jung, “An advanced Java mobile browser using a box model to display the contents, ” Virtual Environments, Human-Computer Interfaces and Measurement Systems, pp. 31-35, 2004. [5] US Patent, US 7242369 B2: Method of displaying text on multiple display devices, 2007. [6] MSDN online help web page: http://msdn2.microsoft.com/en-us/default.aspx. 34.

(56)