應用動態重視度表現值分析法評估醫療資訊
系統之可用性
Usability Evaluation of Healthcare Information System
through Dynamic Importance-Performance Analysis
陳柏嘉
1Po-Chia Chen
蔡佩芳
2Pei-Fang Tsai
陳凱瀛
2Kai-Ying Chen
國立臺北科技大學工商管理研究所 國立臺北科技大學工業工程與管理系
1
Graduate Institute of Industrial Engineering and Management, National Taipei
University of Technology and
2Department of Industrial Engineering and
Management, National Taipei University of Technology
(Received December 3, 2013; Final Version July 24, 2014)
摘要：醫療資訊系統透過支援必要的醫院管理與醫療作業，協助提供高品質與高效率的病患照 護。本研究提出一個結合可用性要素與問卷調查為基礎的方法，評估醫療從業人員對於所使用 之醫療資訊系統的評價。其中包括動態重視度表現值分析法，分析使用者認知中對於可用性要 素的滿意程度與期望程度。根據調查而得的資料，將受訪者對於系統可用性要素的認知各別以 第三四分位數與第一四分位數區分為高重視度/高表現值與低重視度/低表現值。所提出方法實 際應用於案例醫院提供之醫療行政服務的出院流程中。結果顯示所提方法具備在不同醫療作業 環境中有效識別醫療資訊系統中需要改進的可用性要素，且能夠進一步提供醫療資訊系統設計 時重要的使用者需求資訊。 關鍵字：動態重視度表現值分析法、醫療資訊系統、可用性、系統評估 本文之通訊作者為陳柏嘉，e-mail: [email protected]。
The study has received financial support from the National Taipei University of Technology - Mackay Memorial Hospital Joint Research Program (NTUT-MMH-100-15). The authors would like to thank Director Fu-Man Lin and all the administrative staff in the participating hospital for collaborating on the project. They also appreciate the valuable comments and suggestions by anonymous reviewers for improving the quality of the paper.
Abstract: Healthcare information system (HIS) contributes to a high-quality, efficient patient care by
providing good supports to the administrative and medical tasks required. In this research, a questionnaire-based approach is proposed to evaluate the use of HIS in terms of usability factors. The perception of the satisfaction and expectation in these factors is then investigated using the generalized version of importance-performance analysis, or referred to as dynamic importance-performance analysis in this paper. Based on the survey data, the relative levels of “high importance/performance” and “low importance/performance” are dynamically distinguished by the 75th percentiles and 25th percentiles, respectively. The proposed methodology is demonstrated in a case study conducted for the discharge process in a hospital administration service. The analytical results presented in the case study provide insights in system design to further improve the usability of HIS under specific clinical environment.
Keywords: Dynamic Importance-performance Analysis, Healthcare Information System, Usability,
System Evaluation
1. Introduction
The important issue of hospital management is to effectively reduce hospitals’ operating cost and to improve the quality of medical service. The use of information system to assist in medical operation is regarded as one approach that can help achieving these two objectives (Reichertz, 2006). Even as the computer-based hospital information system has been prevailed, paper-based medical records are usually found coexisted in most medical institutions due to the ease of use and legal consideration (Haux, 2006).
In a study evaluating the extent use of healthcare information technology found a rather low reliance than expected in medical institutions at all levels. For integrate delivery networks (IDNs) and stand-one hospitals, only administrative functions such as electronic results review and claims and eligibility checking were frequently used. Other functions which could be beneficial to patients’ safety and satisfaction, such as computerized physician order entry, electronic health record, and patient-physician communication, were much less utilized (Poon et al., 2006). A nation-wide questionnaire survey on clinical information and communication technologies (ICTs) systems in Finland in spring 2010 showed that system usability was a common issue for physicians. Factors that hindered the efficiency of clinical ICT use and physician’s routine work were the lack of system features to support typical clinic tasks and inflexibility in ways to store or retrieve patient data (Viitanen et al., 2011).
Research by DeLone and McLean (1992) proposed a success model for information system (IS) which suggested the degree of system and information quality in creating IS had direct impact on system use and user satisfaction. When measuring IS success, systems use was a popular but rather complex measure due to the multidimensional interpretation. An updated IS success model included the positive causal relation in higher “user satisfaction” to increased “intention to use”, and hence increased “system use” (DeLone and McLean, 2003).
For healthcare information system (HIS), literature found that the leading factor affecting medical and nursing personnel’ intention to use a system was the perception of system usefulness (Bhattacherjee and Hikmet, 2007; Klein, 2007). This perceived usefulness of a system was also related with perceived system compatibility, or whether the system functions met operational needs (Bhattacherjee and Hikmet, 2007). Although users’ perceived ease-of-use did not directly affect their intention to use, it was found to have significant impact in perceived usefulness of a system (Klein, 2007). The relationship among various factors is summarized as in Figure 1.
To identify factors that relate to the use of HIS, three challenging issues have to be taken into consideration:
(1) The existing HIS often consists with heterogeneous systems.
Most HIS creating process has temporal, which multiple systems are developed at different periods on various techniques platform for specific need in certain departments or functional units (Yang et al., 2011). With the limitation of the existing HIS structure, most medical service requires the collaboration among or switches between departmental information systems. It leads to the assumption that HIS use evaluation should go beyond assessing single, departmental information system. Hence, the analysis of the medical processes goals and the associated tasks provided by HIS, or the goal-task mapping, should be conducted before usability evaluation.
(2) The measure of perceived usefulness on HIS is a relative scale.
Research by Klein (2007) found that individual factors such as computer self-efficacy and personal innovativeness in the domain of IT had positive effects on perceived ease-of-use, which then enhanced the perceived usefulness (Klein, 2007). According to the research of Adams et al. (1992), there is no absolute measure of ease-of-use or usefulness. Users’ perceptions of ease-of-use or usefulness toward IT vary with time and individual experience. The consistent effect of IT is not able to be expected (Adams et al., 1992). As a result, when answering the questions concerning system usability, the respondents’ scales are usually different as well. Therefore, an approach which can effectively evaluate users’expectation to the system is required.
(3) Potential changes in HIS use due to the dynamic nature of healthcare processes.
The challenges in providing integrated HIS along healthcare processes were the inconsistent characteristic between the organizational processes and the medical treatment processes (Lenz and Reichert, 2007). While organizational processes had often been stable with generic process patterns, the medical treatment processes had to constantly be adjusted with the latest medical knowledge and patient specific information (Rebuge and Ferreira, 2012). Nevertheless, any change in the medical treatment processes is affecting the administrative ones while information for hospital policies and physicians’ medical decisions are exchanged in both directions. Moreover, as the HIS adoption is redarded as organisational development, the new work process is recommended to be developed prio to the deployment (Janols et al., 2014).
A research of Nabovati et al. (2014) found the Laboratory and Radiology Information Systems suffer from usability problems (Nabovati et al., 2014). For an investigation of electronic patient record systems in Sweden, usability and the possibility of changing and improving the HIS is considered as one of the important factors relate to users’ adoption (Janols et al., 2014). According to the healthcare IT, a research of user participation compared the developers’ views with the views of the physicians who primarily used the case company’s products. In this case of Finland, both users and developers seem to be “willing” but “not able” to collaborate with each other. Limit return channel of communication of users’ feedback is considerd as the cause. Besides, the developers may usually collaborate with customer representatives who are not end users (Martikainen et al., 2014).
Previous reviews of research indicate a lack of system evaluation method to distinguish the important factors to improve “Perceived Ease of Use” and “Perceived Compatibility” of HIS. To bridge this gap, the research issues of this paper focused on the following three questions: (1) How to access the priority of factors relate to positive effect for system use from end users? (2) What are these factors in the case study? (3) Will the priority differentiate by scenarios?
The purpose of this study is to develop a methodology to evaluate HIS usability in the healthcare delivery. The proposed methodology includes the use of goal-task mapping technique, which is useful in healthcare environment given the characteristics of heterogeneous designs in existing HIS. While investigating the usefulness of the existing HIS as the usability performance, this method includes users’ expectation for the ideal HIS in the survey. The consequent importance-performance analysis (IPA) in usability provides priorities in further improving HIS to meet users’ expectation. A case study is then conducted to validate our methodology in hospitals located in both downtown and suburban area of Taipei city in Taiwan. The organization of this paper is as follows: Section 2 includes a brief literature review in usability evaluation and the evaluation methods. Section 3 depicts the details of the proposed methodology in usability evaluation for HIS. The case study at the hospitals in Taipei is presented in Section 4, as the concluding remarks and managerial insights are summarized in Section 5.
2. Information System Use and Usability
Usability originates from ease of use, and is developed from the structure composed of user, task, tool, and environment. In the concept where users’ acceptance of tool is taken into primary consideration, acceptability is the balance among utility (whether the function of a tool meets the needs), usability (whether users successfully use the tool to complete the task), and likeability (whether users find the tool suitable) in a possible alternative. Extending from this concept, research by Shackel (2001, republished in 2009) defined usability as: “the usability of a system or equipment is the capability in human functional terms to be used easily and effectively by the specified range of users, given specified training and user support, to fulfill the specified range of tasks, within the specified range of environmental scenarios”.
According to Nielsen’s definition of usability in 1993, usability is one of the important aspects affecting product acceptance. His analysis divided acceptability into two aspects, practical aspect and social aspect (Wilson, 2009). Among the factors affecting practical acceptance, usefulness is also affected by usability and utility. Under this structure, usability is defined as a term which includes five characteristics.
Learnability: ease to learn, which enables users to rapidly start to use the system to complete their work.
Efficiency: after users complete the learning of the system, the system can efficiently improve work productivity.
Memorability: users still can easily recall how to operate the system after not getting exposed to the system for a while.
Errors: the error rate of the system should be lower to enable users to make fewer errors during the operation of system. Even when they make errors, it should be easy to restore the errors. Moreover, significant error which cannot be restored should not take place.
Satisfaction: the system should be able to be used happily and subjectively liked by users.
Report of ISO 9241-11 (1998) defines usability as “Extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” In this definition, usability includes the following three characteristics.
Effectiveness: the extent that users accurately and thoroughly complete work progress.
Efficiency: the condition under which resources required to be invested to enable users to accurately and thoroughly complete their work.
Satisfaction: the extent of comfortableness and acceptance for users to use.
In the software design of information system, usability and user interface design tend to be taken into consideration together. In the model framework for software engineering quality, ISO 9126-1 (2001) defines usability as the one of the independent factors affecting software engineering quality. In addition, it is correlated with user interface and the interaction between users and software. The main reason why ISO 9241-210 (2010) regulates system usability of design is that “Usable systems can provide a number of benefits, including improved productivity, enhanced user well-being, avoidance of stress, increased accessibility and reduced risk of harm” (ISO 9241-210, 2010). The specific explanations of usability include the following four evaluation indices (ISO 9126-1, 2001). Understandability: the ability of a software product to enable users to understand whether such a
product is applicable and how to use it under specific tasks and usage condition. Learnability: the ability of a software product to enable users to learn to apply it. Operability: the ability of a software product to enable users to operate and control it. Attractiveness: the ability of a software product to attract users to use it.
Delice and Güngör (2009) applied analytic hierarchical process (AHP) to rate the severity of usability issues detected by the heuristic evaluation (HE). The usability problems are first detected by HE, the priority of each problem then be decided according to the pair-wise comparison process of AHP by the evaluators. With the enhancing focus on the acceptance of healthcare IT, Liljegren and Osvalder (2004) applied users’ questionnaires and cognitive engineering (including cognitive walkthrough and usability tests) for medical equipment selection. Edwards et al. (2008) applied
Heuristic walkthrough (HW), a hybrid inspection method designed to take advantage of the benefits of both heuristic evaluation (HE) and cognitive walkthrough (CW), to make the choice of commercial alternatives of electronic health record.
A better usability evaluation result is expected to ensure safety and to enable clinicians (users) to concentrate on their patients rather than the technology. Svanæs et al. (2010) performed a lab-based usability testing for mobile electronic patient record (EPR) systems in physical environments. To get valid results, the tests must be run as role-plays with multiple stakeholders as participants, including physicians, nurses, and patients (Svanæs et al., 2010). Lottridge et al. (2011) conducted a relatively large sample user testing of handheld devices to identify user subgroups based on the background variables (including Age, Time in clinic, Search engine use and Medical DB use). Different groups were found to have different needs and preferences for information resources and device form, which imply the opportunities for requirements customization (Lottridge et al., 2011). To evaluate the acceptance of EHR-integrated mobile patient questionnaires, System Usability Scale (SUS) was applied by Fritz et al. (2012). The implementation is proved to be cost-efficient and the user satisfaction toward the web-based application is high according to the result of SUS (Fritz et al., 2012). Usability evaluations methods can be divided into the following three categories according to the difference in evaluators and evaluation processes (Folmer and Bosch, 2004).
(1) Usability testing: To perform usability testing, representative users using the system or system prototype to perform typical tasks are required. Even though the final product has not been prepared, prototyping can be used to develop the model of final product and thus the characteristics of final product can be evaluated. Evaluators can evaluate how user interface assists users in completing tasks based on the testing result.
(2) Usability inspection: To perform usability inspection, the participation of usability professionals, software developers, users, and other professionals, is required. Such an inspection examines and judges whether each element included in user interface or system prototype meets the established usability principles.
(3) Usability inquiry: To perform usability inquiry, the participation of evaluators and users is required. Evaluators should talk to users, observe them using the system in real work or enable them to answer questions through language or words to find out their likes, dislikes, as well as needs and understanding of the system. The usability inquiry methods include usability questionnaire.
Even though the definition of usability puts more emphasis on users’ preference and satisfaction, it is necessary to define the environment, the objectives, and the user-specific attributes prior to the
discussion on system usability. Moreover, most of the usability evaluation methods are used before system development or during prototype testing. In terms of HIS, the variability of medical environment is higher than that of other industries, the system structure is more complicated (the cost for the re-design or establishment of system is huge), and the on-site operating personnel (who are also the system users) are most directly exposed to the change of operating environment. Based on the literatures, most attempts of usability evaluation in the healthcare setting are to select an existing IT alternative. Somehow, the users’ requirements are usually a formulation process before the realization of prototype. It’s necessary to design a more flexible approach to evaluate HIS and better enable users to elaborate their experiences, in order to understand the change in the future system usability need under a changing environment.
3. Research Method
Usability evaluation includes factors concerning system quality and information quality, and is closely linked with the users’ perception of system usefulness. The methodology of this study utilizes usability questionnaire to evaluate information systems for healthcare processes. Each usability factor in the questionnaire collects the users’ perceived usefulness and their expectation for the defined system functions. The gaps between expectation and perceived usefulness originate from: (1) the gap between users’ usability expectation and analysis of system need; (2) the gap between analysis of system need and system design; (3) the gap between system design and system usability performance. To a certain extent, user satisfaction can be identified by the difference between users’ expectation and the perception of HIS usefulness. When expectation is significantly higher than perceived usefulness, it is interpreted as users’ dissatisfaction with the system. The HIS evaluation approach proposed in this study includes the following 3 stages:
Stage 1. Goal-Task Mapping
The definition of usability includes two main elements: Context of use (including user, tasks, tools, and environment) and the goal of the process. Usability is the index that tools assist users in achieving specific goals under different context of use (Shackel, 2009; ISO 9241-11, 1998). To evaluate HIS usefulness, it is necessary to clearly link the operating objectives and the tasks involved since the healthcare processes are usually inter-departmental. The goal-task analysis help understand the existing information system and how these tasks are included in supporting the processes. To conduct the goal-task mapping, the detail steps in which part of HIS are utilized by users to achieve process goals are carefully recorded using flow charts with all routes under different conditions.
Stage 2. Usability Factors Selection
To define the system usefulness depends on the context of use for the system (ISO 9241-11, 1998), and the usability of HIS should not be merely regarded as a characteristic possessed by user interface. To evaluate usability for different processes, different factors might be adopted with the consideration of environment, objective, work, information system, and users. For example, some usability factors are more suitable for the overall system rather than within individual sub-systems, such as the communication and integration among system.
Stage 3. Dynamic Importance-Performance Analysis (DIPA)
The usability evaluation proposed in this research involves two dimensions: the perceived system usefulness (system performance) and the perceived importance of system aspects (system expectation/importance). Similar to the inconsistency between the designed quality and the perceived quality in any products or services, the performance-expectation gaps raise concerns for both the service providers and customers. Further, the user satisfaction and potential improvement in the system design can be identified in comparing the performance and expected importance.
The “Importance-Performance Analysis (IPA)” proposed by Martilla and James (1977) was an effective technique in evaluating marketing programs. The method used questionnaire survey on specific product or service to collect customers’ perceived importance and performance/satisfaction in different measuring attributes. The results were then illustrated on a two-dimensional graph as shown in Figure 2, and associated management meaning was given for each quadrant. This technique had been applied in the evaluation of service quality for information systems (Ainin and Hisham, 2008).
The definition provided by ISO 9241-210 puts emphasis on users’ experiences of system, including the emotions, beliefs, preferences, perception, and physical and psychological responses and behaviors before, during, and after using the system. Because individual users’ knowledge and familiarity with operating procedures and information system is different, their perspectives on information system are established based on their previous relevant experiences. Therefore, their responses are a relative scale, instead of an absolute scale.
To incorporate the subjective influence of individual factors such as personal innovativeness in the survey, this research proposed a generic framework in relative measures, which is referred to as Dynamic Importance-Performance Analysis (DIPA). This generalized, non-absolute DIPA dynamically defines the boundary using the concept of percentiles. As shown in Figure 3, four meaningful quadrants are defined with the 25th percentile, or the first quartile (Q1), and the 75th
Figure 2 Importance-Performance Grid with Attribute Ratings for Automobile Dealer’s Service Department (Martilla and James, 1977)
Figure 3 Dynamic Importance-Performance Analysis Graph
B. Keep Up the Good Work A. Concentrate Here
Fair Performance
C. Low Priority D. Possible Overkill
Slightly Important
Excellent Performance Extremely
the mean importance of an attribute is no greater than the 1st quartile of the overall importance, then this attribute is said to be “low importance”. On the other hand, if the mean importance of an attribute is no less than the 3rd quartile of the overall importance, then this attribute has “high importance”. Similar terminology is applied to the performance measure. The identification of four quadrants is redefined as follows.
Quadrant A (High Importance and Low Performance): Users are dissatisfied with these usability
attributes and expect highly for further improvement. Managers should give priority in this quadrant “concentrate here.”
Quadrant B (High Importance and High Performance): The users’ expectation is high and satisfied
at the same time, just “keep up the good work.”
Quadrant C (Low Importance and Low Performance): Even through users are dissatisfied, but the
expectation of these attributes is low. Effort to further improve attributes in this quadrant should have “low priority.”
Quadrant D (Low Importance and High Performance): Users are satisfied with these system
attributes but don’t expect them more. It might mean that more efforts than enough had been put in these attributes and “possible overkill.”
Other than these four quadrants, any attribute with mean performance or importance between Q1
and Q3 is assumed to have no significant difference to the grand average. It is suggested that for
attributes in this “insignificant” area, or the shaded area in Figure 4, would be considered only after all attributes in the previous four quadrants are fully exanimated.
4. A Case Study
The proposed evaluation method is applied to assess the usability of the existing HIS of the case medical institution, a teaching hospital with two branches in Taiwan. One is with a 1200-bed capacity located at the center Taipei City (or referred to as TP branch); another is with a 900 acute bed capacity located on Tamsui District (or TS branch). This study focuses on the administrative HIS related to the discharge process.
Based on the empirical observation and the interview with chief officers and senior staff of Medical Affairs Office, The conceptual mapping of three main operations with the related operations is depicted as in Figure 4. The detail discharge processes (goals) are as follows:
Figure 4 Mapping between the Discharge Process and Existing Operations
(1) Confirm discharge order – After physicians issue discharge orders, the nurses in the associated wards confirm the orders’ detail and pass the information to administrative staff. The goal of this process is for the hospitalists to get notified on the change in bed status.
(2) Prepare medical bills and related paperwork – The staff collects records and insurance policy for the medical treatments for patients, such as room type, meal plan, tests or examinations, and related medical costs. The goal is to generate the final bills after information is validated by patients. (3) Receive payment and provide discharge education – Once the patients make payments by visiting
cashiers, prescriptions and discharge education are then given by the administrative staff. The goal is to reduce the risk of re-admission.
One should notice that the functions of different tasks (admission management process: register the inpatient to the corresponding bed vs. discharge process: enable the bed) may be designed within the same information system (Patient Registration Sub-System) in the current system architecture. Different system requirements designed in the same user interface may lead to some usability problems relate to understandability or operability.
The subset of usability factors used in this study is based on the list in the research by Shafinah et
al. (2010) as in Table 1. All factors but attractiveness are adopted in the questionnaire. Further, three
factors (understandability, error prevention, and operability) are linked more directed to the sub-systems. The remaining four factors (effectiveness, efficiency, learnability, and accuracy) are for the quality of the overall information system.
A total of 58 questions was developed in the questionnaire, similar to the design in the research by Tsai et al. (2011), and pre-tested by senior hospital managers as listed in Table 2. Further, these questions are classified into three types in describing tasks of information system (Reichertz, 2006), including system description (or SD, for information retrieval), system management (or SM, for information logistics) and system control (or SC, for problem solving). This evaluation uses Likert scale with value ranging from 1 to 5, for 1 being strongly disagree and 5 being strongly agree.
The users who were familiar with the discharge processes are from the department of General Affairs/Admission Counter Affairs in both TP branch and TS branch. A total 23 out of 32 participants from TP branch and a total 16 out of 17 participants from TS branch completed the questionnaire. The scope of this case study focused on the empirical application of proposed method, although this sampling is not able to represent the whole country, the different workload and different users’ experience (with same HIS) in these two cases are valuable to examine the effect of different scenarios. From Table 3, in general, the users’ age in TP branch is elder (than in TS branch), resulting to more experience for hospital administration service, more experience in the case hospital and more experience for discharge process. The almost identical “Experience for hospital administration service” and “Experience in the case hospital.” demonstrate most users focus their career in the same hospital.
Table 1 Definition of Usability Factors (Shafinah et al., 2010)
Usability Factor Definition Efficiency System should be efficient to use (time).
Understandability The capability of the system to enable the user to understand whether the software is suitable and how it can be used for particular tasks and conditions of use.
Operability The capability of the system to enable the user to operate and control it. Error Prevention The systems should be low rate errors. User can do recovery if they make
mistakes.
Learnability The capability of the system to enable the user to learn its application. Accuracy The correctness of the output information.
Table 2 Questions for Usability Factors in the Questionnaire Design
Usability Factor Questions Type
Understandability (F1)
A1. Terminology used throughout the system is easy to understand SD A2. The system instructions are apparent SD A3. System command are easy to understand SD A4. Sequence of screens is clear SD Error Prevention
(F2)
A5. Seldom or never make mistakes SD A6. Error message is helpful SD A7. Easy to correct the mistakes SD Operability
(F3)
A8. Quick to find where to enter or edit data SD A9. Quick to enter data without using mouse clicks SD A10. Quick to browse the provided data SD A11. Data displayed is sufficient and without redundancy SD A12. Quick to find statistic data(e.g.#of patients to be discharged) SM A13. System contains complete functions SC Effectiveness
(F4)
B1. Helpful for the entire discharge process SC B2. Helpful in reducing phone calls to check discharge order SC B3. Helpful in distributing bed status to corresponds SC B4. Helpful for the related tasks SC B5. Helpful in verifying patients’ medical bills SM B6. Helpful in obtaining real-time bed status SC B7. Helpful in estimating when the bed becomes available SM B8. Helpful in predicting the length of stay(date) SC Efficiency
(F5)
B9. Response time of the system is acceptable SD B10. Waiting time for each query is within 5 seconds SD B11. Waiting time to complete each billing is within 1 minute SD B12. Beneficial to enter data quickly SD B13. Time to access required information is acceptable SD Learnability
(F6)
B14. Learning to use this module is easy SD B15. Easy to remember the names and use of commands SD B16. Easy to figure out different functions without reading user manual
SD
Accuracy (F7)
B17. The information of related tasks is always correct and reliable SM B18. Don’t have to check the medical bills manually SD B19. The result accuracy is high among subsystems SM
Table 3 Demographic Data of the Participants
TP branch TS branch Min Max Average Standard
Deviation Min Max Average
Standard Deviation Age. (Years Old) 28 63 48.00 9.29 28 53 42.64 7.46 Experience for hospital administration
service. (Years)
7 40 22.93 10.23 2 28 19.14 8.37
Experience in the case hospital. (Years)
7 38 22.98 9.88 1 28 19.00 8.67
Experience for discharge process. (Years)
0.5 38 15.07 12.60 1 27 13.21 8.40
Experience to use computer. (Years) 0.5 31 19.74 9.57 10 28 21.07 5.65 Frequency to use computer (duty
excluded). (Hours/Week)
0 60 8.33 13.03 0 60 11.71 16.49
The demographic data shows the participants were familiar with the discharge processes in average, and the users in TS branch seem to be more familiar with computer.
In TP branch, the Q1, average (or Q2), and Q3 of the system performance was 3.22, 3.44, and 3.74;
for system expectation, the Q1, Q2, and Q3 was 4.22, 4.28, and 4.39; respectively. In TS branch, the Q1,
Q2, and Q3 of the system performance was 3.31, 3.48, and 3.73 while those of the expectation was 4.25,
4.33, and 4.38; respectively. The results were consistent for both branches in the overall measure of system performance and importance.
The DIPA graph for TP branch and TS branch is demonstrated as in Figure 5. None of the usability factors was located in quadrant B or quadrant D. F7 (Accuracy) was the only factor in quadrant A and should have the highest priority to be improved based on the result from TP branch. But this factor was not conceived as critical for the users in TS branch. For quadrant C, F6 (Learnability) was the only factor identified in this quadrant in both branches. But it was worth noting that F5 (Efficiency) had similar perceived importance for both branches, this factor was least satisfied in TS branch but not in TP branch.
The detail DIPA analysis on each question for users in the TP branch and TS branch is illustrated as in Figure 6 and Figure 7; respectively. For TP branch, two usability questions in quadrant A were question number B7 (Helpful in predicting the length of stay (time)) and B19 (The result accuracy is high among subsystems). Especially for question B7, it identified the dire need for the hospitalists to know the approximate time it would take for patients to actually leave the hospital once the discharge
Figure 5 The DIPA graph of usability factors for hospital discharge service in two branches
Figure 7 The DIPA Graph of Usability Questions for Hospital Discharge Service in TS Branch
orders were confirmed by the nurses. For quadrant C, six out of eight questions were related to the overall system and question number B8 (Helpful in predicting the length of stay (date)) was the lowest in users’ satisfaction. However, the users didn’t expect this feature to be important compare to the estimated process time for the entire discharge process. The DIPA graph for TS branch is much concentrated near the means of both dimensions. Even for the questions which were least satisfied like for B10 and B11, the expectation was on the border line of being insignificant. The major complain for TS branch users was the question A5 and A13 for the billing subsystem. As for the overall system, they were most satisfied by the feature in question B5 (Helpful in integrating expenses list) located in quadrant B.
Next, we aggregate the questions by the task types of information system as in Table 2 and the corresponding DIPA graph is depicted as in Figure 8. It was found that the perceived relative importance of these three types, SM>SC>SD, were consistent in both branches. As for SM type of functions, the performance of the system was considered “low” for TP branch users but not for TS branch users. The discrepancy in the system satisfactory helped in explaining the reason of question B19 (The result accuracy is high among subsystems) was identified to have high priority to be improved for TP branch but not in TS branch. Also the narrower range in thresholds showed that the group perception of the information system was more consensual and conservative in TS branch.
Figure 8 The DIPA Graph of Task Types for Hospital Discharge Service in Both Branches
Since the Discharge Systems are the same, it seems surprising to have almost different system evaluation results between TP branch and TS branch. The similar phenomenon can be observed from the research by Lottridge et al. (2011). When it comes to the users’ requirements (or expectation), individual’s background plays an important role in the users’ needs and preferences.
5. Conclusion
Healthcare industrial has been grown tremendously in size and complexity as the medical science and technology have advanced at a rapid rate. Further, the care delivery process has been evolved from merely providing treatment to patients into an integrated effort from primary care, specialists, hospital managers, even insurance policy makers in the entire system. The use of healthcare information system (HIS) has becoming one of the most critical tools in hospital management to not only reduce hospitals’ operating cost effectively, but to improve the quality of medical service. This study proposes a systematic, three-stage approach to evaluate the use of system and to identify factors to further improve the usability of the existing HIS. Moreover, factors and features of HIS usability are prioritized using an extended version of the importance-performance analysis, referred to as dynamic importance-performance analysis (DIPA). This DIPA incorporates the nature of relative
scale in the survey by defining thresholds for four quadrants based on the case-specific percentile of the entire dataset.
In terms of practical application, this proposed approach is used to evaluate the HIS for the hospital discharge process in selected hospital in Taipei. The results showed that, even with the same information systems and discharge processes, users in two branches of the hospital had different perception in factors to have higher priority. It was critical for users in TP branch to improve the “Accuracy” of the current system but not for users in TS branch. It also implied that it is possible to improve the user satisfaction and encourage more reliance on computer-based information system by introducing functions like an estimated time for the discharge process. This DIPA approach avoids the potential misleading in users’ perception without considering the group dynamics in different environments or time frames in evaluating the use of the system. It provides valuable insights for system design to accommodate users’ most desirable features to improve the system use.
This research was limited to time and the project scope. Therefore, the participants didn’t include the healthcare professionals, and it is hard to pratically revise the HIS according to the analysis result. For further research, a complete cycle of proposed method could include a first round analysis, system revision (re-design), and a second round analysis to trace the change of users’ overall satisfaction and adoption toward the HIS. Furthermore, the information system structure and the system design method should be developed to improve the HIS re-design process and make the change of HIS more flexible to meet the end users’ need.
References
Adams, D. A., Nelson, R. R., and Todd, P. A., “Perceived Usefulness, Ease of Use, and Usage of Information Technology: A Replication,” MIS Quarterly, Vol. 16, No. 2, 1992, pp. 227-247.
Ainin, S. and Hisham, N. H., “Applying Importance-Performance Analysis to Information Systems: An Exploratory Case Study,” Journal of Information, Information Technology, and Organizations, Vol. 3, 2008, pp. 95-103.
Bhattacherjee, A. and Hikmet, N., “Physicians’ Resistance toward Healthcare Information Technology: a Theoretical Model and Empirical Test,” European Journal of Information Systems, Vol. 16, 2007, pp. 725-737.
Delice, E. K. and Güngör, Z., “The Usability Analysis with Heuristic Evaluation and Analytic Hierarchy Process,” International Journal of Industrial Ergonomics, Vol. 39, No. 6, 2009, pp. 934-939.
DeLone, W. H. and McLean, E. R., “Information Systems Success: The Quest for the Dependent Variable,” Information Systems Research, Vol. 3, No. 1, 1992, pp. 60-95.
DeLone, W. H. and McLean, E. R., “The DeLone and McLean Model of Information Systems Success: A Ten-Year Update,” Journal of Management Information Systems, Vol. 19, No. 4, 2003, pp. 9-30. Edwards, P. J., Moloney, K.P., Jacko, J. A., and Sainfort, F., “Evaluating Usability of a Commercial
Electronic Health Record: A Case Study,” International Journal of Human-Computer Studies, Vol. 66, No. 10, 2008, pp. 718-728.
Folmer, E. and Bosch, J., “Architecting for Usability: A Survey,” Journal of Systems and Software, Vol. 70, No. 1-2, 2004, pp. 61-78.
Fritz, F., Balhorn, S., Riek, M., Breil, B., and Dugas, M., “Qualitative and Quantitative Evaluation of EHR-integrated Mobile Patient Questionnaires Regarding Usability and Cost-efficiency,”
International Journal of Industrial Ergonomics, Vol. 81, No. 5, 2012, pp. 303-313.
Haux, R., “Health Information Systems – Past, Present, Future,” International Journal of Medical
Informatics, Vol. 75, 2006, pp. 268-281.
ISO 9241-11, International Standard: ISO 9241 Ergonomic Requirements for Office Work with Visual Display Terminals -- Part 11: Guidance on Usability, International Organization for Standardization, 1998.
ISO/IEC 9126-1, Software engineering -- Product quality -- Part 1: Quality model, International Organization for Standardization, 2001.
ISO 9241-210, International Standard: Ergonomics of Human–System Interaction -- Part 210: Human-Centred Design for Interactive Systems, International Organization for Standardization, 2010. Janols, R., Lind, T., Göransson, B., and Sandblad, B., “Evaluation of User Adoption during Three
Module Deployments of Region-wide Electronic Patient Record Systems,” International Journal of
Medical Informatics, 2014, Article in Press.
Klein, R., “An Empirical Examination of Patient-physician Portal Acceptance,” European Journal of
Information Systems, Vol. 16, 2007, pp. 751-760.
Lenz, R. and Reichert, M., “IT Support for Healthcare Processes - Premises, Challenges, Perspectives,” Data and Knowledge Engineering, Vol. 61, No. 1, 2007, pp. 39-58.
Liljegren, E. and Osvalder, A.-L., “Cognitive Engineering Methods as Usability Evaluation Tools for Medical Equipment,” International Journal of Industrial Ergonomics, Vol. 34, 2004, pp. 49-62. Lottridge, D., Chignell, M., and Straus, S. E., “Requirements Analysis for Customization using
Handheld Devices in Healthcare,” International Journal of Industrial Ergonomics, Vol. 41, 2011, pp. 208-218.
Martikainen, S., Korpela, M., and Tiihonen, T., “User Participation in Healthcare IT Development: A Developers’ Viewpoint in Finland,” International Journal of Medical Informatics, Vol. 83, No. 3, 2014, pp. 189-200.
Martilla, J. A. and James, J. C., “Importance-performance Analyses,” Journal of Marketing, Vol. 41, No. 1, 1977, pp. 77-79.
Nabovati, E., Vakili-Arki, H., Eslami S., and Khajouei, R., “Usability Evaluation of Laboratory and Radiology Information Systems Integrated into a Hospital Information System,” Journal of Medical
Systems, Vol. 38, No. 4, 2014, Article in Press.
Poon, E. G., Jha, A. K., Christino, M., Honour, M. M., Fernandopulle, R., Middleton, B., Newhouse, J., Leape L., Bates, D. W., Blumenthal, D., and Kaushal, R., “Assessing the Level of Healthcare Information Technology Adoption in the United States: A Snapshot,” BMC Medical Informatics and
Decision Making, Vol. 6, No. 1, 2006. doi:10.1186/1472-6947-6-1.
Rebuge, A. and Ferreira, D. R., “Business Process Analysis in Health Care Environments: A Methodology based on Process Mining,” Information Systems, Vol. 37, No. 2, 2012, pp. 99-116. Reichertz, P. L., “Hospital Information Systems - Past, Present, Future,” International Journal of
Medical Informatics, Vol. 75, 2006, pp. 282-299.
Shackel, B., “Usability - Context, Framework, Definition, Design and Evaluation,” Interacting with
Computers, Vol. 21, No. 5-6, 2009, pp. 339-346.
Shafinah, K., Selamat, M. H., Abdullah, R., Muhamad, A. N., and Noor, A. A., “System Evaluation for a Decision Support System,” Information Technology Journal, Vol. 9, No. 5, 2010, pp. 889-898. Svanæs, D., Alsos, O. A., and Dahl, Y., “Usability Testing of Mobile ICT for Clinical Settings:
Methodological and Practical Challenges,” International Journal of Medical Informatics, Vol. 79, 2010, pp. 24-34.
Tsai, P.-F., Chen, P.-C., Chiu, C.-Y., and Chen, K.-Y., “Information System Evaluation for Hospital Admission Management: A Case Study,” In 20th Annual Industrial Engineering Research
Conference, Reno, Nevada, May 21-25, 2011.
Viitanen, J., Hyppönen, H., Lääveri, T., Vänskä, J., Reponen, J., and Winblad, I., “National Questionnaire Study on Clinical ICT Systems Proofs: Physicians Suffer from Poor Usability,”
Wilson, C., User Experience Re-Mastered: Your Guide to Getting the Right Design, San Francisco: Morgan Kaufmann Publishers, 2009.
Yang, T. H., Sun, Y. S., and Lai, F., “A Scalable Healthcare Information System Based on a Service-oriented Architecture,” Journal of Medical Systems, Vol. 35, No. 3, 2011, pp. 391-407.
