In a typical window-base interface system, users control application programs containing the HCI (Human Computer Interaction) control functions in speech or handheld device such as PDA or cellular phone directly and conveniently. The commonly used traditional method which allows developed window application programs with speech or wireless remote HCI control ability to follow the control commands for achieving the goal of operating application programs, will directly integrate the HCI control function and API procedures of speech or handheld device into the application programs by using low-order designing formula to package procedures into single application program through a direct and intense method.
Under such devising method, the designers must be equipped with certain knowledge about application system designing and programming in order to devise an application system with speech or remote HCI control functions. Even if the design is completed, it is relatively difficult to revise or add the interface functions without the primitive code, as shown in figure 1-1.
Figure 1-1 General developing method of speech control
Three major problems exist in such development of speech or remote HCI control procedures: first, system designer are required to possess abundant knowledge about system design for control interface and programming language in order to design an application with
speech or remote HCI control functions. Second, if we want to obtain an application program with speech or remote HCI control functions lacking previous interface control ability, we must obtain the primitive code for that application program due to the difficulty in designing without the code. Third, even if the code is attainable, we are still required to reanalyze the entire structure of application in order to write a suitable interface control program. These tasks will leave the designer with much difficulty and seemingly resulting in less flexibility and efficiency.
To overcome these issues, in this thesis we emphasize on the research of Software Engineer Methodology to develop a visual generic interface bridge (GIB) system and introducing this system into two parts: First, the “Integration of GIB and Speech HCI,” and secondly, “GIB-based Application Interface (GAI) generation,” in which a wireless handheld device is taken as an example. Under the GIB system, designers devise the application system with speech or wireless remote HCI control functions in an easier and more efficient manner with the only requirement for defining some parameters of square objects in a application environment where users do not need to write any program code.
In part one, most of voice control robots employ such development method in designing their voice control interface, such is in the case of AT&T’s Speech-Actuated Manipulator (SAM) [1], whereas the voice commands via telephone are comprehended and then performed with respective actions. Under such means of intense design, we discover that if there is any need for modification in speech control function, the primitive source code of application system must be acquired and comprehended before using the low level design method to write application control functions, while the entire program must be recompiled and executed.
Such design manner does not offer extreme efficiency, moreover it is hard to design, increase or revise the control function without the original source code of the application.
(a) IVOS system architecture
(b) IVOS system
Figure 1-2 OS integration approach
An OS integration approach allowing one speech recognizer to be intact with the domain of applications such as Vspeech 4.0 [2], Voxx 5.0 [3] and IVOS [4], utilizes the appearance of windows interface, designs some fixed pronunciation operation functions and establishes the name of relative operation instructions. Different instruction names are used to control the same operating function such as “open file” to <Alt>-F and “file open” to <Alt>-F, and then using the designed file name of speech command to achieve the goal of controlling the
application system with speech commands, as shown in figure 1-2.
However such development method of speech control system contains two problems:
first, the current approach leaves no flexibility for future modifications about control functions. The control functions were already designed steadily in the HCI system and unable to be increase or revise. If some functions we use to control application were not defined, we must get the primitive source code of the application and redesign it. This will increase difficulty and loading on the work. Second, the current approach may only control application functions with hot-key commands. Due to the system only defines the control functions of keyboard, therefore only keyboard actions can be controlled. Nonetheless there are still many applications with control functions of mouse actions, the current approach will not be able to fully handle theses applications.
Based on the reasons states above, we develop a visual generic interface bridge (GIB) system between speech HCI and window-base application programs. The GIB provides visual operating interface, under which designers draw recognizing square object at any corresponding position on the windows and name each square object. Subsequently, we can easily use speech command to control mouse and keyboard actions corresponding to the position of square object. By increasing the operation of application with more flexibility and expandability, we use macro command to define and combine the control commands. One macro command may be combined with several control commands; this will avoid noise effect between long commands and make the application control more flexible with grammar analysis technology.
Through this process we can make any application program which did not have speech HCI control ability previously, and to control easily with speech commands simply after defining simple square object and sets of macro commands without the need for writing any program code, as shown in figure 1-3.
Figure 1-3 Architecture of GIB control system
In part two, due to remote control methods have been discussed in recent years; in 2001, it was proposed that cellular phone to be used in controlling remote HCI running on the PC.
Cellular phone is a mobile device containing a small screen without showing the same GUI (Graphical User Interface) as on the PC screen. Therefore we must analyze the GUI and develop some function programs for the cellular phone based on its use in the remote control interaction with GUI on the PC. However along with progressing existing techniques and increasing functions of communication equipments for family use, operation and interaction styles between people and devices have increased in complexity. Most of the multimedia contents can be run and displayed on different kinds of platforms without containing remote control ability originally. Consequently people may believe that if they can simply use some simple instruments, such as cellular phone or PDA, to remotely control the multimedia application module running on the PC or digital TV, subsequently the control will become more vivid and interesting. However due to the variation in control instruments, display devices, and different kinds of methods, it is not easy to make a specified device with remote control ability. To achieve this, we must repeat the design process: (1) Write control command protocols and HTTP wireless protocols into the applications running on the PC; (2)
Write interface program and wireless control protocols into cellular phone, as shown in figure 1-4, for every application system in successive sequence. Despite this designing process is a difficult work in nature; we must write many complex procedures even for simply adding or modifying a new control functions. In addition, there is still one major problem remaining if we lack the original source code of application system, as it becomes impossible to make the specified device with remote control ability. Such repeated design procedures would increase developing application systems a financial burden, waste of time, inflexibility and inefficiency.
Figure 1-4 General developing method of handheld device control
As the problems reveal, Rajicon System [31] was using cellular phone to control the remote PC, as shown in figure 1-5. It works on a specific device, so, if we want to perform some complex operation functions, we must define a set of interactive rules. Rajicon System uses macro commands to formulate every operating function, therefore, the more operating functions there are, the more likely that more macro commands would be required, as described in table 1.
Figure 1-5 Rajicon System Architecture
Table 1-1 remote control functions with cellular phone keypad mapping
Key Zoom mode command result
1 View entire desktop ?a Hold down Alt key until next key pressed 2 Dec cursor height ?c Hold down Ctrl key until next key pressed
3 Refresh ?n Pressed Enter key
4 Dec cursor width ?m Press left mouse button
5 Execute macro ?^ Maximize the window at the cursor
Through pressing keypad of cellular phone to input command for controlling application running on the PC, these operation methods get complex and disorient, in addition to the difficulties in memorizing these control commands. On the other hand, this system is designed for a specific device when there are some control functions that need to be created or modified; such designed procedures should be restarted repeatedly.
In 2002, Jeffery Nichols and Brad A. Myers published a paper on “Generating remote control interfaces for complex appliances” in [32]. They proposed a method of personal universal control (PUC) with a content describing how to create a control interface with graphics or speech by downloading a functional specification explanation of equipment for family use, as shown in figure 1-6.
Figure 1-6 PUC System Architecture
Hence the PUC system analyzes specification document and uses decision tree algorithm to create a specification group tree, as shown in figure 1-7. Then, establish an appropriate control interface according to the structure of this group tree. However there are many design factors which need to be taken into account, including: (1) Download a functional specification explanation for a specific equipment before establishing an interface, however it is not an easy task to establish this specification description; (2) Design a control interface is weary because interface designed algorithm needs to take into account users’
requirement with the presentation of control device objects.
Figure 1-7 Sample group decision tree for a shelf stereo
In [36], because traditional remote control typically allows users to activate the functionality of a single device, consequently qualitative and quantitative results from a study of two promising approaches creating such a remote control are presented: end-user programming and machine learning. In end-user programming, users manually assign the buttons which they believe to be sufficient to accomplish their tasks of graphical “screens”.
They then work with a single, handheld remote control that can display those screens.
Machine learning also uses a single, handheld remote to display screens; however, the learning uses the recorded history of a user’s actual remote interactions, to infer appropriate groups of buttons for the performed tasks. Some questions arise as described in the following:
(1) the end-user programming is too complicated to carry on the design of remote control to the device with graphical screens. User must define the components required for operation on a screen which can control many device interfaces remotely concurrently, and all the control procedures require writing a large number of programs; (2) The ML can record the operation of users automatically and utilizes the performing algorithm to produce the operating component. Perhaps due to the difference in device, it cannot completely define each function of device component; (3) The methods of automatically producing remote control interface are required to follow to the characteristic of the device and the preference of user, but it also needs a large number of procedures to be written in order to design algorithm automatically.
In [38], a new widget and interaction technique known as a “Frisbee,” was described for interacting with areas of a large display that is difficult or impossible to be accessed directly.
It consists of a local “telescope” and a remote “target”. This design meets five design principles in: (1) minimizing physical travel, (2) supporting multiple concurrent users, (3) minimizing visual disruption while working, (4) maintaining visual persistence of space, and (5) application independence. However, the design requires writing many procedures for the specific and large-scale showing device. Moreover, because of the limitation of hardware
specification and the difficulty in obtaining relevant information, it is comparatively difficult and complicated to design the interface.
Figure 1-8 Architecture of proposed handheld device control system
Based on the issues mentioned above, our proposal for solving this issue will be to construct an interface generating system; with it the designer can easily develop remote control interface program into cellular phone without even having to write the program of cellular phones. We comply with the concepts of GIB to develop a bridge interface for remote signal control system, under which designer can easily transform from specified control object of Java application system running on PC, such as control buttons and labels, and generating remote control interface with objects, into cellular phone automatically. This will simplify the development process of creating a control interface and makes the control system development and modification more flexible and elastic. Through directly generating controlling objects and functions onto the cellular phone by interface generator, we may directly and easily control the Java application system running on the PC without having to write complex programs into the cellular phone, as shown in figure 1-8.
This thesis emphasizes on the research of Software Engineering Methodology in order to unfold the feasibility and the serviceability. We literally develop a GIB system by adopting a
proposed method which is used as the bridge between HCI and window applications, and through which the interaction of HCI and applications will become more vivid and friendly without any requirement of writing any program code from the user.