Multi-modal user interface system development: The case of emergency triage
Shen Yu Xiang Chen Ying stirrup Zhu original Ka Po- Lun
Health, National Yang-Ming University, Institute of Information and Decision Making
normansheng1211@yahoo.com.tw genego@mail2000.com.tw yuanchia@ebtnet.net polun@ym.edu.tw
Abstract
Figure 1
Logitech © Cordless Freedom Headset Model No. 8701
Figure 2 SONY PCV LX900
This study integrates voice and touch technology to the emergency triage, for example, through a combination of voice and touch technology, provide a suitable system of triage for help nurses identify patients more precise condition, and the need for care. The developed system consists of a large and clear touch
interface, hands and eyes free voice input, and use simple rules, some through the user input data to provide triage triage classification, sorting, partition recommended value, can be used as reference for nursing staff category, and thus to reduce manual work and increase the accuracy of patient medical
information and real time. More important is to take this opportunity to establish voice and touch technology used in the prototype structure triage. And
requested the 30 acute care colleagues to help test, so as to provide multi-modal user interface into care information system pilot test experience. The research and development environment for Microsoft Visual Basic 6.0, Speech SDK 5.1,
wireless headset microphone Logitech cordless freedom headsetand the large size touch screen SONY PCV LX900,the developed system consists of a large and clear
touch interface, and hands-free eye voice input, and use simple rules, through the user input some of the triage 资料 to provide triage classification, classification, partition of the recommended value, classification for nursing colleagues
reference. This system a totalof 26 images, voice, vocabulary 338. Assessment is that if the use of voice as much as touch mixed mode interface, voice interface than simply have a significantly higher fault tolerance. Comparison of voice and touch, while the right touch high, but the average speed of fast speech.
Comparison of multi-mode, single mode, and the old system acceptance, nursing staff favorite multi-mode interactive interface developed more than one mode of the emergency triage support system, and the system built in this study averaged 106seconds to complete a triage operation, at 63 db of noise environments up to 94% of the voice recognition rate.
Keyword
Multi-mode user interface(Multi-Modal Interaction),ED triage (Emergency Triage) Background
In order to improve health care quality and effectiveness of touch in recent years, speech recognition technology research quite common, and many university laboratories constantly developing better and more efficient use of voice recognition technology: such as the MIT Laboratory for Computer Science dialogue language system (Spoken Language System) [1] group has developed a series of voice systems, users can use the phone said the dialogue with the computer to obtain information; and with the Massachusetts Institute of Technology's Media Lab (Media Lab ) [2] is committed to R & D permeability calculated (pervasive computing) environment, an important voice for one of man-machine interface.
Stanford University's Media X Lab is developing a dialogue control system (Dialog Control System) [3],want to develop smooth (seamless) natural (transparent) interface session; Berkeley University of user interface research group (Group of User Interface Research, GUIR) [4] developed a software calledSUEDU key is to provide a
development environment for voice applications, through the integrated development environment, developers can easily develop voice applications.
There are many domestic and foreign research units are also continuing research into voice-related issues.
First, the development principles
In order for systems similar to the previously operated by the nurses of the old system, to the availability of "similar to the original work" and the "easy learning"
principle [5], this study of nurses who used the old 3270terminal inspection trauma systems analysis to the nurses to familiarize themselves with the system
architecture, re its user interface. And the graphical user interface design, information design of its new PDAsystem [6], as far as possible to familiarize themselves with the original user interaction model, but adjusted to fit the design of a large touch screen. Analysis of voice interface, please help nurses during the day classes, respectively, the midnight shift, evening shift on the triage record the actual situation, to analyze the actual time for the patients to triage the case of voice interaction, to find the best audio interface mode. System development method is used embryonic development (prototyping), and nursing staff continued to modify and discussed.
Second, development tools
The system uses a program development tool for the Microsoft of Microsoft Visual Basic 6.0, with its speech recognition development tools, Microsoft Speech SDK 5.1 and Simplified Chinese Language Pack [7]. Wireless headset microphone you use Logitech Logitech Cordless Freedom Headset, as shown in Figure 1, touch-screen desktop computers for the SONY PCV LX900 its carrying Intel Pentium III 1GHz, 128 RAM, and 17-inch touch screen[8 ], as shown in Figure 2.
Third, multi-mode interactive design
The touch-screen design according to user needs, design a large screen, big buttons, large font visual interface. All the boxes are the screen can touch the button, a pink top left corner of the first part of the basic data for the area, is registered by the emergency area by the incoming data, the blue box below the button input for the triage, no value has not yet entered the box to show the deep blue, after the value input box is displayed as light blue, if the touch of the box will flash after three seconds, the role of the box is rendered as yellow, and
making the use of in the box to the default color rendering, the orange part of the information for the computer-aided decision-making were triage classification, triage, partition and so on. Voice control box if the green is ongoing voice input, if the click will become orange is the voice to suspend the case, and then click will restore the status of voice input. Voice of the controlling party status for the volume below the article, would speak with the voice volume to change the size of
the length of the best radio results is in the range of green, yellow is too sound, too loud red.
Figure 3 Touch screen user interface design, is divided into the basic
information region (upper left red zone), triage information region (central blue region) and computer-assisted area (the right side of Orange District) Third District.
Figure 4 Interactive voice user interface design flow chart. Users input the words to say, computer literacy, the complex song to computer users to listen to confirm, if no end is waiting for the user that the next word, if the end of the stop identification.
4, system architecture design
Figure 5 System architecture diagram for the module square, round side for the data source, virtual party for the external system
The system uses object-oriented programming methods and principles of the structure formed the core object to triage patients object, the object store to take care of emergency patients present all information and information stored in accordance with a simple check, and then trigger event, such as blood pressure, diastolic blood pressure higher than 100 issued a warning on the event too.
Triage was an intelligent agents module, triage patients by monitoring the properties of objects and events, the decision for the proposed classification and
disposal, and can perform a number of built-in actions, such as the transmission of information related processes, or notify the relevant emergency doctor or nurse. Since this system uses multi-modal user interface, and therefore user interaction module is responsible for coordinating all information should be presented to the user to coordinate distribution, the best way to present to the user. Coordination and synchronization of voice, touch screen, and the future is combined with the PDA, PDA on the information presented to the user.
5, efficiency calibration
Efficiency is the availability of several indicators (easy to learn, easy to use, easy to remember, user satisfaction, efficiency, fault tolerance, with the original work of similar capacity) in a more challenging. In this system, the speech recognition rate, the system processing speed, noise resistance for the greatest impact on the overall efficiency of the system. The test prior to this study, these issues find viable option to be adjusted to an acceptable state before entering the testing phase, testing the results after adjustment according to test recommendations.
6 to assess the design
Assessment of this study is divided into two phases, the first pre-assessment for the development of the second stage after the completion of the assessment for the development. In the early stages of development, in order to receive early user responses and collect 开发 systems have the knowledge, and offer a
opportunity for users to participate in the design process [1], the early
development of this study is limited information Kuaisu Genju out a prototype system developed and test the system by nursing staff and offer suggestions.
When the system prototype developed by the method of stability and meet the requirements, then enter the second stage assessment. The second stage assessment prior to the implementation of the objectives will be measured by an official of the display shows those with education and training, after training the subjects of this study will have some degree of understanding of the system.
About the implementation of a formal assessment of 30 minutes per test, the test is divided into two stages: the first phase of training computers for the
establishment of voice files, time-consuming about ten minutes, the second stage of system testing. Test record completion time, environmental noise, the wrong situation and the number of items to calculate the error recognition rate,
recognition rate compared with the completion time to assess the relevance of its
fault tolerance. Please test after test to fill a similar technology acceptance model (TAM) questionnaire, and make recommendations on testing and test record in the situation observed in the sidelines.
Result
1, system functions and features
In order to achieve system availability, the system combines wireless headset microphone, large touch screen and other equipment for nursing colleagues to provide a friendly and intuitive human-computer interaction model, which contains a large and clear touch interface, hands and eyes free voice input, can use voice commands to control the computer, such as [deleted], [print] and so on.
The system and use simple rules, some through the user input data to provide triage triage classification, classification, area of the proposed value can be used as reference for nursing staff category.
2, object structure triage patients
Patient triage system core logic of object-oriented layer structure, a total of 23 sub-objects (objects), of which 6 to group objects (collections). Total structure of the whole thing a total of 60 attributes (properties), 47more options (methods), 18 incident has (events). All triage and timely information generated will be temporarily stored in this object, this object and intelligent features, in itself contains the logic of triage, when the value of objects found within the
appropriate conditions, it will trigger the event. Systems triage agents will be able to monitor the initiatives generated by this event object to decide how to respond. Regular monitoring can also be passive objects of a particular property, once they meet certain conditions can make the appropriate response actions.
Third, performance tuning results
Testing process, we found that the factors affecting the efficiency of speech recognition there are many factors which can control the main environmental noise, speech recognition engines, dictionary size. Good selection of
microphones can be ruled out environmental noise as possible, select the appropriate speech recognition engine can improve engine performance, in addition, also can reduce the size of lexicon methods to enhance identification performance.
The original design for the identification of all the words are put in the same dictionary, this method has the advantage of input there will be no class
restrictions, users can enter at any time to enter the vocabulary, do not need to enter the category into the category of terms after This can reduce the memory and recall, such as direct input "penicillin" "heart" "headache," without input
"allergic" "penicillin" "history" "heart," "complaints" and "headaches" and are very convenient .
However, with the growth of those thesaurus, testing recognition errors often occur when the response, for example, to enter the allergic response to the complaints of the project Shique project. Therefore, the system only uses second approach, a large lexicon is divided into several small cut by category thesaurus, and controlled within two (major categories - value), although there are still 249 words complained of, the total number of 61words %, but in order to avoid the need for users to remember items complained of sub-categories, to the
availability of "easy to remember", or not to molecular classification.
After classification, and further adjustments when the situation will be impossible for some of the vocabulary used to remove, the recognition of the success rate of 70% of pre-measured and marked increase to 95%.
The effect of this noise canceling microphone is extremely well, even a few decibels up to 77dB in the environment, can give uninterrupted test conducted for improvement if the part is to connect the computer to the digital connector, such as USB, can prevent the computer noise, so that the noise canceling better.
In addition, there is still a need to use the situation to overcome the difficulties, the original idea is to allow nurses to use that model to replace the eye's input by hand, but in fact the nurses are from time to time to speak , in the scene, nurses need to constantly communicate with the working partners, but also to the
patients condition and their families, or communication, or to answer the phone.
Therefore, we can not assume that this system can be "exclusively" user interface to speak, must have a good mechanism for interrupts. The current design is the use of software with hardware interrupt mode according to the said equipment (push to talk) button. In the busy times when the hand when using the software interrupt for the way, when the hands are not busy when using the "as is said,"
The effect will be better.
Fourth, the assessment results
The study subjects were mostly around 30 years of age (60%)of women (90%), the more qualifications for the emergency room more than a decade (43%) of nurses (60%), mostly college educated (53%) and specialist (43%). Average monthly participation in triage work are 1 to 5 times (80%). Most users will use a computer at least once a week (76%), of which one third of the participants for the high- volume users, the computer several times per day (33%), with Qi Cheng (73%) or more subjects in the test before the speech recognition has come into contact with the relevant information. Test subjects for more positive about this system, although now consider themselves to use unskilled (90%), but that can be qualified as long as little preparation (87%), sticking (63%) of users the current system efficiency has been satisfactory, while the majority (87%) of the user that the system (multi-modal user interface) than the PDA to the good.
Discuss
1, multi-modal interface usability of triage support system
Usability of the system, after the test questionnaire showed that 87% of users believe that they can easily test After the learn to use the system, of which 80%
thought that only a little preparation can, 6.7% of users completed the test later and think they are competent. Shows that the system in the "ease of use" and
"easy learning" has been recognized by a majority of users.
Comparison of the triage system and PDA system, nearly 87% of users believe that this system better than the PDA system, if requested by the test model were a pure voice, mixed voice and touch mode and the old 3270terminal system for points, ranking 1 as one-third, two points awarded second place, ranking third as one point. Hybrid touch pattern is the average score of2.93 points, or 1.6hours of pure voice, the old system of 3270was 1.46, showing that multi-mode input of the "user acceptance" was significantly higher than the original use of voice and the old system.
Nurses believe that if the average energy in 30 seconds to complete triage for a patient, then the system will be very useful. At present the system average for the patient to perform a comprehensive triage time was 106 seconds, the fastest completion time is 45 seconds, the slowest completion time is 268seconds, compared to the United States triage relevant professional associations of seven minutes [20]to be fast many.
In theory, the recognition rate should affect the overall processing speed, because every identification error of a project, you must re-enter once, also must spend an input time. Test results in this study, the average spend per project must enter a3.52 seconds, so every time the average recognition error of more than 3.52 must spend time to make up for the error. Suppose a triage average input of 30 items, the average recognition rate increases by 1 percentage point, you can save 1.17 seconds. Actual test results will be brought to statistics, by the pure voice patterns and visual patterns in the voice input data, if executed more than 97.5%recognition rate, the average time required to complete 86 seconds, if the recognition rate of 80% of the words The average completion time Zexu 178 seconds, from the statistical point of view really would affect the recognition rate of speed to complete the work.
Figure 6 Recognition rate and the average completion time diagramN = 60
The findings appear in the "fault tolerance", the user testing the system using voice and touch screen models with the best fault tolerance. Use of voice with the screen display, but the wrong one, the user can easily observe the current situation, and in addition to using voice, you can also use touch to rule out error.
If you just use plain speech input mode, an error the user, but confusing, if not very familiar with the system it is difficult to immediately remove barriers.
Overall, the system was "ease of use", "easy learning" and "user satisfaction"
get more positive comments, and "efficiency" and "tolerance" is still the focus of improvement.
However, for the subjects 30 seconds to complete the proposed time limit, there can be discussed between. If they could upgrade the system to other functions such as decision support, for users who have helped in other areas, whether the user will complete a longer time for the acceptance of more space. While and
make up 30 seconds to complete the goal, whether the associated benefits for patients, it can improve patient satisfaction, are also worth considering.
Second, multi-modal interface development technology discussion As the user's behavior is unpredictable, the system development on the light as far as possible in order to be able to all users of the system made possible action,
in the development of the code will likely be more complex and scattered. If the
point of view from the hierarchical, the closer the information layer, the structure of the code, the easier the closer the more close to the user interface layer, exceptions will be more and more code written thus will be more difficult to grasp the structure of error-prone. Multi-mode user interface to give users greater flexibility, on the other hand also represents such a large order to provide the flexibility to use in the development must be better ways to design and management of code.
During development, this study first user interface and data structure triage patients and determine the logical part of the triage to dismantle as much as possible, the data structure and triage patients to judge the logical design for the device, and users referred plane will no longer have any logical data structure and determine the code can be simplified to a part of the function. As part of the user interface less structured, to design the components is difficult, therefore the first to be modular. The current design, multi-mode interface mode for each interface module are designed, such as voice module and touch module, these modules are coordinated through an interactive module to communicate with the underlying components and, apart from the addition, coordination module through an interactive mode synchronous behavior of all interfaces, such as when to voice input on a project, it has also produced a touch input of the project should be in return. Will coordinate the functions of independent simultaneous dismantling them will simplify the code and then make the code easier to manage and reduce errors. If we can more clearly interactive mode, you can try to change the design for the device, the user interaction module is a fundamental design element, and all of the interactive mode, such as voice, touch and other Zeyi inheritance based approach to handle all components to modify the content, so will be able to enjoy the advantages of easy maintenance.
However, although this structure can cope with most of the situation, but there will still be there a number of possible exceptions, different users use different modes of operation are more likely to produce unexpected results, so if multiple users want to have to ask tests to detect problems and to ameliorate.
Conclusion
In this study, triage is an example of a combination of voice and touch develop as many models interactive information system, to examine the multi-modal
human-machine interface in the care of the issue of information availability. The
results showed a multi-modal user interface will enhance the system's fault tolerance, the user can interact through with a different use models to cater for different situations and conditions, the shortfall in order to achieve optimal efficiency. Overall, users of multi-modal user interface of acceptance than a single mode (voice only). The recommendations of the emergency triage mode of voice as the main input devices, with touch screen of the auxiliary. The
Institute developed the overall system efficiency and fault tolerance is still strengthening.
The system is designed for nurses simulation as an assessment of the situation is the test, because subjects were faced with the situation on paper, so the test results should be with the real situation on the difference between the test results.
In addition, the current design of the system can only process a case, but
discussions with the nurses, because they must triage, we often deal with multiple cases simultaneously, so the system also need to have this function next to meet the demand.
The future system will be developed in this research can have two continuity issues, a front-end user interface usability studies for the extension of an intelligent decision support for the back-end component design of the study.
Usefulness of the front, which can be tested in accordance with advice provided by users of the system continued to improve, but also can continue to try to join a different multi-modal human-computer interaction model combination, for
example, test subjects had at the time mentioned Handwriting recognition and fingerprint recognition and other technologies to explore the extension of the availability of multi-modal human-machine interface. Back-end components of the extension of triage of patients is quite large, in addition to the specifications of the components continue to develop more suitable for the more perfect, the present study only to join the simple rules of triage, the future can try to standardize the rules of the database integration (eg, Arden Syntax) Let the system more flexible, able to provide a better quality of decision support.
References
1. Spoken Language System. MIT Laboratory of Computer Science 2004 [cited 20040 Jun 30];
2004Available from: URL: http://www.sls.csail.mit.edu
2. Media Lab. MIT 2004 [cited 20040 Jun 30]; 2004Available from: URL:
http://www.media.mit.edu/
3. Media X. Stanford 2004 [cited 20040 Jun 30]; 2004Available from: URL:
http://mediax.stanford.edu
4. Group of User Interface Research. 2004 [cited 1906 30]; 2004Available from: URL:
http://guir.berkeley.edu/
5. Staggers N. Usability Concepts and Clinical Computing. In: Marion J. Ball KJH, editor.
Nursing Informatics-Where Caring and Technology Meet. Springer; 2000. P. 95-109.
6. Chang P, Tzeng YM, Sang YY. The Development of Wireless PDA Support Systems for Comprehensive and Intelligent Triage in Emergency Nursing. Journal of Nursing (in chinese) 2003 Aug; 50 (4) :29-39.
7. Speech SDK 5.1 for Windows applications. Speech SDK 5 1 for Windows applications 2002 June 24 [cited 20040 Jul 1]; Available from: URL:
http://www.microsoft.com/speech/download/sdk51
8. SONY PCV LX900. SONY 2004 [cited 2004 Jun 30]; 2004Available from:
URL: http://www.sony.com
Figure4.1 The basic initial screen
Figure 4.2Enter the
"source" project picture
Figure4.3 Enter "to hospital" project picture
Figure 4.4Type "escort"
project picture
Figure 4.5, "chief complaint -When"
Project Screen
Figure 4.6, "chief complaint -
Genitourinary" project picture
Figure 4.7, "chief complaint -features"
project picture
Figure 4.8, "chief
complaint - nerve" project picture
Figure 4.9, "chief complaint -the skin"
project picture
Figure 4.10, "chief complaint - the spirit of"
Project Screen
