User Interface for Visual - Impairment

全文

(1)Workshop on Computer System User Interface for Visual – Impairment Jiung-Yao Huang*, Kuo-Jui Chang and Ming-Chih Tung Department of Computer Science and Information Engineering, Tamkang University, Taipei, Taiwan * jhuang@mail.tku.edu.tw. Abstract This paper proposed a user interface for visual impairment to operate a computer using Braille input method. This system is focus on embedding the visual-impaired user enabling functions into the desktop operating environment for the visual-impairment. The proposed system can simultaneously display the context, which is inputted by the Braille input method, on the tactile Braille reading device, the computer screen with printed text and the audio device with voice. Moreover, the system can also detect the printed text, which was updated by other application program, on the screen and exports new content to blind system (tactile reading device and audio device) to remind the visual-impaired operator about this event. The major contribution of this paper is to enable the visual-impaired user to use their proficient Braille input method to operate the computer and the tactile reading device, simultaneously, without learning other input method. The performance of the proposed system has been evaluated by sightless users and has shown encouraging results.. Correspondence Address: Jiung-yao Huang Associate Professor , Department of Computer Science and Information Engineering, Tamkang University Telephone: 02-26215656 Ext 2747(Office) Fax: 02- 26209749 Mailing Address: Department of CS&IE, Tamkang University, 151 Ying-Chung Rd., Tamsui, Taipei, Taiwan 251 E-mail: jhuang@mail.tku.edu.tw.

(2) User Interface for Visual – Impairment Jiung-Yao Huang*, Kuo-Jui Chang and Ming-Chih Tung Department of Computer Science and Information Engineering, Tamkang University, Taipei, Taiwan * jhuang@mail.tku.edu.tw Abstract This paper proposed a user interface for. 1. Introduction. visual impairment to operate a computer. This paper is to study an integrated user. using Braille input method. This system is. interface. focus on embedding the visual-impaired. environment for the visual-impairment. A. user enabling functions into the desktop. modular and systematical approach to. operating. design such a user interface system is. environment. for. the. of. the. desktop. operating. elaborated in this paper.. visual-impairment. The proposed system can simultaneously display the context, which is inputted by the Braille input method, on the. Instead of designing an add-on application. tactile Braille reading device, the computer. program, this research is focus on proposing. screen with printed text and the audio. a method to modify the user interface of the. device with voice. Moreover, the system can. traditional Operating System to support the. also detect the printed text, which was. visual-impairment.. updated by other application program, on. designing a user interface as an add-on. the screen and exports new content to blind. program is that only applications that follow. system (tactile reading device and audio. the specification of this user interface can. device) to remind the visual-impaired. then accept input from this specific interface.. operator about this event. The major. To solve this issue, this paper attempts to. contribution of this paper is to enable the. integrate the proposed user interface into the. visual-impaired user to use their proficient. Operating System kernel so that the blind. Braille. can easily use most of applications.. input. method. to. operate. the. The. problem. of. computer and the tactile reading device, simultaneously, without learning other input. By modifying the user interface of the. method. The performance of the proposed. Operating System, this research attempts to. system has been evaluated by sightless users. break through the barrier of the information. and has shown encouraging results.. acquisition for the visual-impairment by facilitate them using the computer. Using. Keywords: User interface for the visual. the Braille input device, the visual-impaired. impairment,. user can input characters through the. The. supplemental. Braille. display instrument, The computer for the. keyboard.. blind, Barrier-free technology, Linux.. pronunciation and the supplemental Braille. 1. Furthermore,. the. audio.

(3) the. other applications update the screen buffer,. visual-impaired user to read the information. this system will inform the visual-impaired. from the computer.. user by displaying the newest content on the. display. instrument. enable. tactile reading device and the audio device. This research uses the RedHat Linux as the experimental environment to develop the. With the help of these capabilities, the. user interface for the visual-impaired user.. visual-impaired user can use the Braille. In order to achieve the goal of integrating. input method to edit the computer text and. the user interface for the bind into the. operate computer applications. This system. Operating System, this paper employs the. will automatically translate the user inputted. kernel mechanism to mount the proposed. Braille code into Chinese/English text and. user interface. Using this approach, the. output to application program. In addition,. proposed user interface can effective accept. because the system can display the content. the Braille input without creating overhead. on the location of the cursor on the tactile. to the Operating System.. reading device and the audio device, the visual-impaired user can move the cursor to edit text. That is, the visual-impaired user. 2. System Architecture. can know where the cursor moved in the. In this section, we present a schematic. context by touching the tactile reading. overview of the user interface for the visual. device and hearing the voice from the audio. impairment system proposed in this paper.. device. Furthermore, in order to support the. The block schematic of the overall system is. visual-impaired user to interact with the. shown in Figure 1. There are three major. computer application program, such as the. capabilities to help the visual-impaired. Internet chat, text game, etc., this system. people read/write the files from/to computer:. will automatic scan the screen buffer and. (1). The visual impairment can use the. display the newest content to the tactile. keyboard to input contents using the Braille. reading device and the audio device.. code, the system will display them on the tactile reading devices first. The system is. 3. The Braille and Chinese/English Input. then translates these Braille code to. Mechanism. Chinese/English context and display them. This section is to study a mechanism to help. on the computer screen and the audio device;. the visual-impaired people to use Braille. (2). When the visual-impaired user moves. input method to operate the computer and. the cursor on the screen, this system will. application. scan the screen buffer and display the text. difficult for the beginner to master the. information (a sentence or a line words) on. complex rules when translating Chinese text. the location of cursor on the tactile reading. into Braille and vice versa. Unlike English. device and the audio device; (3). When any. Braille, for example, a sentence is not. 2. programs.. However,. it. is.

(4) translated letter by letter, and there is not a. State 0: Waiting for a key to be pressed. one-to-one relationship between the printed. down;. text and the Braille code. [1] To facilitate. If any of the eight keys is pressed. the production of the Braille code and. down, change to State 1;. encourage the visual-impaired user to use. Else loop on State 0. State 1: Waiting for the event of a new. the computer, this paper proposed a new Braille-translation. system. to. key is pressed down or a. translates. pressed key is released;. regular Chinese text into Braille and vice versa by using the translation rules and the. If another new key is pressed. dictionary table.. down then go to State 2; If a pressed key is released then go to State3.. As shown in Figure 2, the block schematic of. the. Braille. and. State 2: Recording the pressed and. Chinese/English. released keys to the sets Φ and Ψ.(?). translation mechanism using the Braille. Go to State 1.. input method. The Braille Input module read the Braille code from the keyboard. The. State 3: Erase the fields of Φ that the. Braille/Chinese Translator then translates. presses down keys were released;. the Braille code into phonetic symbols and. If the set Φ is empty then go to. uses the combination of these symbols. State 4.. combination to find the corresponded. Else go to State 1. State 4: Output the set Ψ and clear it;. Chinese characters.. Go to State 0. The Braille code uses a group of eight dots. The State 4 output the set Ψ to the Braille. that are arranged in four by two matrix.. Output. [2][3] The Braille Input Module uses eight. Translator in Figure 2, simultaneously. The. keys, i.e. “a”, ”s”, ”d”, ”f”, ”j”, ”k”, ”l”,. Braille Output module then exports the. and ”;”, on the keyboard to match these. Braille to the tactile reading device which. eight dots. By this way, the visual-impaired. enables the visual-impaired user to confirm. user can use these 8 keys to input the Braille. his input. The Braille/Chinese Translator is. code. These eight keys input algorithm for. followed by translates Braille input data to. the Braille Input Module is shown by the. Chinese. following states:. application program.. module. character. and. and. Braille/Chinese. exports. it. to. Since the Traditional Chinese Braille (or. Let. Mandarin Braille) is a phonetic symbols. Φ = {W | W is the set of the pushing. representation, [4][5] the Braille/Chinese. down keys}. Translator uses the Traditional Chinese. Ψ = {S | S is the set of the set-free. phonetic input method to translate Braille. keys}. 3.

(5) codes into Chinese characters. As shown in. that match this pronunciation for the user.. Figure 2, there are three stages in the. Finally,. the. Braille/Chinese Translator : (1) it will. module. outputs. translate. Phonetic. phrases, which are stored in Chinese phrases. Symbols first, (2) these Phonetic Symbols is. hashing table, of each Chinese character to. then input to Chinese phonetic input method. the Braille Output Module for the user. The. module to find all of related Chinese. visual-impaired user can then read these. characters with their pronunciation, (3) the. phrases from the tactile reading device and. user can then uses the tactile reading device. select correct Chinese character according to. to select correct Chinese character.. these phrases. The Event Control Module. Braille. codes. into. Chinese. Words. corresponded. Selection Chinese. also output the Chinese character to the However, there are two problems must to be. Screen Buffer Control module and the Voice. solved when translating the Braille code into. Output. a Chinese character:. translator received an English Braille code,. 1) The system must be compatible with. it directly finds the corresponded English. many different Chinese-Braille input. character and exports it to the Event Control. methods.. Module.. module.. Furthermore,. if. the. 2) Because the Chinese Braille is phonetic symbols representation, there is more. 4. The cursor control. than one Chinese character having the. The Display Area Control module in Figure. same pronunciation. In another word, it. 2 enables the visual-impaired user to move. is. the cursor and scroll the display. When the. one-to-many. matching. between. visual-impaired user move the cursor on the. Braille and Chinese printed text.. screen, this Screen Buffer Control module This paper proposed an “antithesis table” to. will scan the Display Text Buffer and. combine with the legacy “Chinese phonetic. calculate the screen coordinate of the cursor.. input method” to solve these two problems.. The scanned result is forwarded to the. The antithesis table is a table contains all of. Braille/Chinese Translator module for it to. the legal combinations of Chinese phonetic. display the information on the display area. transcription and their corresponding Braille. of the tactile reading device through the. code as illustrated in Table 1. As shown in. Braille Output module. Furthermore, the. Figure. Braille/Chinese. Screen Buffer Control module will also. Translator received keys from the set Ψ, it. exports the sentence on the location of the. searches the antithesis table based on the. cursor to the audio device.. 2,. when. the. Braille code and finds the corresponded phonetic symbols. The phonetic input. In addition, this proposed system will. method module read the resulted phonetic. synchronize the textual display between the. symbols and finds all Chinese characters. tactile reading device and the Display Text. 4.

(6) Buffer. Since the size of each Chinese. device must be carefully controlled so that. Braille code is not a constant, it uses two or. its movement is based on a character or. three Braille cells to present a Chinese. punctuations. Without this synchronization,. character. On the other hand, English. the cursor movement on the tactile reading. characters, punctuation marks, and numerals. device and the screen will be out of order. only required one Braille cell to present. whenever the user moves the cursor. This. them. Furthermore, the movement on the. paper proposed a Braille access structure to. display area of the tactile reading device is. synchronize the character on the computer. based on a Braille cell whereas the cursor. screen text and the Braille code shown on. movement on the screen is based upon a. the tactile reading device. The Braille access. character or punctuations. Therefore, the. structure is showed as follow:. cursor’s moving step on the tactile reading { Braille_Length; Braille_Code; Position;. // The coordination on display area of tactile reading device. } Hence, the Braille Output module transfers. into the Braille access structure which is. all the characters which showing on the. stored in the Braille output buffer. The. tactile reading device in this structure and. Braille cursor on the display area of the. save them in the Braille output buffer. When. tactile reading device is then moved based. the system receives the cursor movement. on the saved values.. command from the keyboard, the Screen Buffer Control module scan the Display Text Buffer based upon the coordinate of the. 5. Spontaneous update of the tactile. cursor and exports these texts to the. reading device. Braille/Chinese Translator. The translator. In order to support the visual-impaired user. translates these texts to Braille codes and. to interact with the computer application. passes them to the Braille Output module.. program, this system will automatic scan the. The Braille Output module analyzes these. screen buffer and display the newest content. Braille codes to calculate the position and. to the tactile reading device. The legacy. the Braille_Length value of a character. Chinese system intercepts and recognizes. corresponded to each Braille code. The. the output characters of an application. position value is the coordinate of a. program. The results are then transform into. character on the display area of the tactile. Chinese fonts and display on the computer. reading device. The Braille_Length and. screen. Generally, these screen characters. position values of each character are saved. are stroke-based fonts and they are rendered. 5.

(7) as graphical mode pictures. It is difficult to. In addition, since the display area of the. recognize the characters from the graphical. tactile reading device is much smaller than. mode pictures on the screen. Although, we. the size of the computer screen, a further. can use the pattern recognition technique to. refinement is required when output the. recognize characters from a screen image,. result to the display area of the tactile. however, this technology is not always. reading device. During the execution, the. successfully recognizing every character on. Chinese system will synchronize the output. the screen. In addition, the recognition. on the screen buffer and the Display Text. process is difficult to be executed in. Buffer. When a text is updated on the screen,. real-time. Hence, this paper proposed. the Screen Buffer Control module will scan. another approach to resolve this problem.. the Display Text Buffer and export lines of text. to. the. Braille/Chinese. Translator. In order to successfully synchronize the. according to the location of the cursor and. screen display and the tactile reading device,. the size of the display area of the tactile. a new output function, called Display Text. reading device. Figure 4(c) illustrates the. Buffer, is plugged into the Chinese system.. case when the display area is composed of 3. Hence, after the legacy Chinese system. by 40 Braille cells. Since the length of. recognizes. an. Chinese Braille code is not fixed, these three. application, the results are transfer to. lines of Braille codes will not have the same. Display Text Buffer before they are. length when displayed on the Braille cells.. transforming. as. Hence, the visual impaired user can move. illustrated in Figure 3. Figure 3 shows the. the cursor to read the screen text by. operational schematic of exporting Chinese. touching the display area on the tactile. characters to the computer screen as well as. reading device to know the position of each. the tactile reading device and the audio. character on the screen. As the result, the. device. visual-impaired user can easily operate the. output. When. into. an. characters. Chinese. application. of. fonts. program. computer like the normal user.. receives a character, the Chinese system will calculate the position of that character and output it to the computer screen and Display Text Buffer simultaneously. When an image. 6. The Implementation. is encountered during recognition, the image. The. data is not stored in Display Text Buffer.. visual-impairment is implemented on Linux. Instead, it is replaced by a set of empty. environment with JMCE. The JMCE is a. symbols as depicted in Figure 4(a) and (b).. popular Chinese system on Linux. It. The Screen Buffer Control Module then can. intercepts keyboard input and the computer. scans Display Text Buffer and read the text. screen output of an application program. By. directly.. scanning. proposed. the. user-interface. keyboard. buffer,. for. the. JMCE. transform the input into Chinese fonts and. 6.

(8) device. The Event Control module will. render to the screen in graphic mode.. schedule the Chinese character output and This paper modified the input and output. cursor movement and transmit these events. rules of JMCE on Linux to implement the. to the Screen Buffer Control module and the. user interface for the visual impairment. The. Audio Output module.. input/output functional modules depicted in Figure 2 are embedded into respective. In addition, in order to exhibit the character. device drivers. As shown in Figure 5, the. on the display area of the tactile reading. Braille Input module and Translate Braille. device, our system modifies JMCE output. Code to Phonetic Symbols functional. procedure by adding the function of. module are embedded in keyboard driver.. exporting the text mode Chinese characters. User can use the hot-key to switch the input. to the Display Text Buffer. This insures that. method among English, Chinese, English. the Big5/ASCII text in the Display Text. Braille, and Chinese Braille. In English and. Buffer is synchronized with the graphic. Chinese Braille input modes, our system. mode text on the screen. The Screen Buffer. will intercept the “up”, “down”, “left” and. Control can then simply scan the Display. “right” keys to implement the Display Area. Text Buffer to export some text data. Control module and synchronize the cursor. (according to display size of tactile reading. movement between the computer screen and. device and cursor position) back to the. the display area on the tactile reading device.. improved JMCE. The improved JMCE. Furthermore, our system will also intercept. phonetic input method will output the. “a”, ”s”, ”d”, ”f”, ”j”, ”k”, ”l”, and ”;” keys. corresponding phonetic transcription from. to read the Braille code and translate the. the inputted Big5 code to the Braille Output. resulted code into a Chinese phonetic. module. Which will in turns transform this. symbol by searching the “antithesis table”.. phonetic symbol into Braille code.. This phonetic symbol is then sent to JMCE Chinese phonetic input method. JMCE will. Finally, an experiment was conducted on. output all of the detected Chinese characters. Pentium II-233 with 64M RAM to verify. with the same pronunciation to the Chinese. the efficiency of the implemented system. A. Word Selection module. For each Chinese. file that contains 170 lines of text and each. character, the Chinese Word Selection. line include a mixture of Chinese and. module is then searching the Chinese. English. phrases. its. randomly moved the cursor within this text. corresponded phrase and export this phrase. file in 150 times, the average time to export. to the Braille Output module. The visual. Braille code to the Braille output buffer. impaired user can then choose the correct. upon detecting the cursor position is 11ms.. character according to these phases shown. This timing is smaller enough for the visual. in the display area of the tactile reading. impaired user to fluently operate the. hashing. table. to. find. 7. characters. were. created.. By.

(9) computer. Reference [1] Jiang Minghu, Zhu Xiaoyan, Xia Yang, 7. Conclusion. Tan. Gang,. et.. al,. This paper proposed an architecture and. Mandarin. mechanism to design a user interface for the. Translation Based on Multi-knowledge”,. visual impaired user. The proposed user. Signal. Braille. Processing. Word. and. Proceedings, th. Braille 2000,. WCCC-ICSP. user to operate a common desktop computer. Conference on, Vol. 3, 2000, pp. 2070 –. using the Braille input method without. 2073.. learning new. input. [2] Jan Mennens, Luc van Tichelen, Guido. hardware. Furthermore, this system allows. Francois, and Jan J. Engelen, “Optical. the visual impaired user to read the screen. Recognition. text and, as a result, browse the Internet.. Standard Equipment”, IEEE Transactions. The proposed system was implemented and. On Rehabilitation Engineering, Vol. 2, No. 4,. is actually used in the Blind Information. December 1994.. Center, Tamkang University, since April,. [3]. 2002. It assists the visual impaired students. Technology for the Blind”, in Proc. Ninth. in operating the computer and helps them. IEEE Annu. Conf. Eng. Med. Biol., 1987,. study the courses of the computer science.. pp. 1791-1792.. method. or. J.. of. 5 ,. of. interface system enables the visual impaired. input. 2000,. “Segmentation. International. Braille Writing Using. Brabyn,. ”Electronic. Braille. [4] “Chinese Braille Code Self-Study Significantly, the architecture of the user. Manual”, National Central Library Taiwan. interface proposed in this paper can support. Branch, R.O.C. 1997.. multilingual environment by replacing the. [5] Teng Weimin, Li Weihong, China Braille,. Chinese system with other language system.. China Press, 1996.. In addition, this proposed architecture of the Braille input model can be applied to the operating system of the wearable computing since it allow the user to manipulates the system with limited keys.. 8.

(10) Figure 1 The block schematic of the user interface for the visual impairment. Figure 2 The system architecture of the user interface for the visual impairment 9.

(11) Table 1 The sample of the antithesis table Phonetic Symbols. Braille Code. Phonetic Symbol keys on keyboard. ㄐㄧ. K*. ru. ㄍㄜ. K!. ek. ·. ·. ·. ·. ·. ·. Figure 3 The operational procedure of exporting Chinese characters to the tactile reading device and the audio devices. 10.

(12) Figure 4 (a) The text showed on the computer screen (b) The text stored on the Display Text Buffer (c) The system will transform 3 lines of text from Chinese/English to Braille based on the cursor location and the display area of the tactile reading device. 11.

(13) Figure 5 The implemented on Linux. 12.

(14)