Safeguarding clinical software – A managerial case study about project management and oversight

Safeguarding clinical software – A managerial case study about project

management and oversight

Thomas Wetter

University of Heidelberg, Medical Informatics

thomas.wetter@urz.uni-hd.de

Abstract

Clinical software has become such an important fac -tor in health care that government agencies have considered to regulate its release like the one for pharmaceuticals. Scientific organizations have doubted fea sibility of all encompassing regulation and rather sug -gested to establish Software Oversight Commi ttees (SOC’s). By using their local expertise they should “detect software problems, analyze their impact, and develop timely solutions”.

This article suggests that SOC’s can be more effec tive if they focus on both strengthes and weaknesses of their organ izations and when they extend their scope from technology to human resource and to vendor rela-tionships. It presents data gathered around a long last-ing failure to integrate critical software components that occurred in a community with 21 hospitals, one be -ing a historical leader in hospital information systems.

1. Introduction

Clinical software has become a factor in health care delivery that may be regarded as essential as medication. Therefore, initiatives to regulate clinical software in the same way as new drugs (i.e. requiring approval before going into nationwide use, requiring ongoing monitoring after introduction, etc) seem to make sense. On the other hand, Miller and Gardner [1] have argu ed that the attempts to replicate for software what has been practiced for pharmaceuticals for decades would cause major damage to the newly developing field of vendor purchased clinical software. Their major arguments include:

• Medical software often consists of a set of complexly interfaced applications, and that regulating any given individual application misses the essential point that only the integration of that application with the other components provides the required safe clinical data processing functionality .

• Attempting to regulate all possible application combinations, however, would create a number of combinations that is beyond imagination, and

is certainly beyond the number that one could reasonably expect a regulatory agency to handle.

Furthermore, even if the release process for software were twice as fast as the one for pharmaceuticals - although there is no reason to believe this possible -, it would still mean that software would be released when it is technically outdated. In contrast to the situation with pharmaceuticals, where a newly approved drug can be sold profitably for a decade, information technology products can have turn over times as short as 18 months. As a consequence, Miller and Gardner, working with a variety of professional and scientific organizations in the field 1_{, came up with the Software Oversight Committee}

(SOC) suggestions which can be summarized as follows:

• Regulate high-risk software such as that which controls cardiac pacemakers or insulin pumps. Such software is characterized by its stand-alone operation implying that it can be tested in isolation, and by its immediate impact on patient parameters with virtually no means for a human to intervene in case of malfunction.

• Don't regulate software with only administrative or non-patient specific data (i.e. software that is used in the context of patient care but does not directly influence it).

•

For the latter suggestion Miller and Gardner recommended using local expertise to form committees that “detect software problems, analyze their impact, and develop timely solutions”. These committees would be analogous to IRBs, but they would focus on issues of software integration including “health care informatics, clinical practice, data quality, biomedical ethics, patients' perspectives, and quality improvement" expertise and to “monitor the procedures for needs assessment,

1

American Medical Informatics Association, Computer-based Patient Record Institute, Medical Library Association, Association of Academic Health Science Libraries, American Health Information Management Association, American Nurses Association

specification, selection of vendors' products, testing, training, installation, followup, users competences, help -desk”.

In other words: the “why” and the “what” of the SOCs were clearly specified, whereas the “how” was purposefully left open: it was implicitly assumed that local knowledge and insight would lead to an SOC lineup that makes proper use of its institutions' strengths.

Subsequent to the Miller and Gardner article, research grants were acquired by several institutions to instantiate and evaluate the SOC concept. Now, seven years down the road, it seems worth considering how the concept has evolved, and whether it has delivered to its expectations. A PubMed search did not identify any journal publications on this topic since 1997, although Gardner and Tate [2] have start ed to teach the topic to a general Medical Informatics Audience. This article can, therefore, be regarded as a first systematic empirical assessment of the concept of SOCs after its inception. It is a detailed yet circumscribed investigation of a software integration issue in a major health-care provider, and considers a problem set typical of those SOC’s were designed to address. This article will present the case study by first reporting the few necessary details about the software, its users, vendors, developers and the health care provider environment etc, then will try to convey a more elaborate metaphor for the SOC lineup and how this drove the methods that were used to investigate the case at hand. Next, the methods and results are discussed. They are presented in the order that best makes the case, rather than trying to introduce all subjectivist observation methods first and then proceeding to objectivist data collection [3]. Then the actions that were implemented as a consequence of the investigation are described. Lastly, a wider outlook how the approach developed for this case study can be generalized to other healthcare organizations with different enterprise characters and cultures will be presented

.

2. Approach

2.1.

Scenario

Decision makers in the enterprise where the author conducted the study 2 _{decided to replace the home made}

blood gas data software that delivered results directly into the respiratory intensive care unit software in a

2

Intermountain Healthcare (IHC) is a nonprofit health care system, and is the largest health care facility in the Intermountain West. It provides hospital and other medical services in Utah and Idaho and is headquartered in Salt Lake City, Utah. IHC operates 21 hospitals in Utah and Idaho,, among them the Latter Day Saints (LDS) hospital, a major tertiary care hospital, and serves 50% of the state of Utah (55% of Utah inpatient services and 45% of Utah outpatient services) [1]

tertiary care hospital with a vendor based software product. Offers from two vendors were evaluated, and the one with the best adherence to standards and highest responsiveness was chosen. During the negotiations and before a contract was made, the vendor was purchased by another company who, without acknowledging it, backed away from delivering some of the formerly agreed upon functionality specifically required by the tertiary care center.

. To the other hospitals in the enterprise the vendor offered former versions of the software with reduced functionalities. While this was satisfactory for these hospitals, it did jeopardize the quality of care in the respiratory intensive care unit (RICU) in the enterpirse’s tertiare care center and did not succeed for years to upgrade. Workarounds were found in the RICU to assure patient safety, but at a higher manual effort. A common fully integrated vendor made blood gas data system remained an enterprise-wide goal. The enterprise was seeking a way to achieve that goa l, and was also seeking recommendations as to how to organize software oversight to prevent such problems in the future.

The author of this investigation came as a visiting scholar to the institution. Preliminary contacts suggested that he qualified as an unbiased third party who might touch upon issues that escaped the attention of all local protagonists. After a brief presentation to the institution’s SOC, he was given unanimous consent to investigate the nature of the problem, provide advice regarding solutions, and summarize and publish the findings of the investigation in the scientific literature.

2.2.

Hypotheses

While Miller and Gardner recommend drawing on the local strengths of an organization in order to lineup an SOC, this article suggests considering the other side of the coin as well.

Hyp.1. Along with its strengths, an enterprise also has its specific weaknesses. Risks, including software related risks, are related to weaknesses in organizational structure and behavior.

Hyp.2. Therefore, it might be as productive as the “strengths” metaphor to look to consider that an SOC has all the specific software related weaknesses of its associated enterprise in its focus.

In a hierarchical enterprise, for instance, an SOC could focus on the weakness that networked operation is not encouraged. As a consequence the SOC should identify those parts of the organization that are not immediately involved in but affected by a software project and get them co-opted in a project team.

Applying the metaphor of limiting the impact of weaknesses of an enterprise entails the following activities:

• finding out weaknesses of that kind of enterprise culture

• trying to find additional evidence that these weaknesses are indeed present locally

•

way that the impact of these weaknesses is limited.

Other useful data collected in this process can and should also be used to deliver specific advice concerning the project.

3. Methods

3.1.

Rationale

In addition to his role as scientific investigator, the author of this study also served as a consultant to the enterprise. As a result, the process of data collection had to be integrated into ongoing, day-to-day peer-to-peer cooperation. Therefore, formal data gathering was kept to a minimum.

3.2.

Methods

Partially standardized interviews were conducted which included elements that were intended to identify the enterprise culture and weaknesses entailed by that culture. The interviews explored the interviewees’ roles with the software integration situation in the past, his/her assessment of the most serious problems in the present, and his/her expectations about how and with whom the

problem should be solved in the future. One question specifically addressed the role that the SOC should play in this process. As one formal element the interviewees were asked to draw a chart of persons or, if they liked, groups in the enterprise that were related to the problem, and to describe in what way these groups or pers ons were related to each other. The interviewees were prompted to draw uni- or bidirectional arcs between the entities and name these arcs. They were encouraged to use arc labels they felt were most appropriate. Another quasi formal element was the question whom they regarded as the most appropriate to take the lead in the stalled software integration project. Finally, they were asked to indirectly classify their enterprise's culture. For this purpose, marker sentences were designed, each of which described the central element of one enterprise culture (cf. Table 1) written on cards and offered to the interviewees, prompting them to bring the cards into a rank order according to which best, second best etc characterized how they felt as employees of the enterprise.

3.3.

Interviews as data

Handwritten notes of the interviews were transcribed using commercial voice recognition software, submitted to the interviewees for changes, withdrawal of parts they regarded unappropriate for subsequent use and, finally, approval. The interviewees knew that in the case of sufficiently valuable results scientific publication was planned. They were also assured that nothing from our conversations would be used as data unless explicitly approved.

Table 1 Enterprise culture Marker sentence

Hierarchical Being part of a precisely defined hierarchy makes life easy.

Quality driven I contribute to a process of care that must the carried out in the best achievable quality.

Examples of marker sentences that characterize enterprise cultures

4. Analytical results

Interviews were planned with 15 individuals, and these occurred (with one exception) one person at a time. Interviewees were selected such that the largest variety of perspectives on the critical software situation that was seemed productive was obtained: Medical vs. information management, managerial vs. technical, local hospital vs. enterprise-wide; as well as human-resources, legal, and risk management.

All individuals selected for an interview agreed and were talked to within an interval of 7 weeks, and the interview transcriptions were approved in the subsequent weeks.

4.1.

Enterprise culture

The institution is very clearly quality driven. If an enterprise culture were judge equally as top ranking by 15 interviewed individuals, it would collect 15 point. As Figure 1 shows, the “quality driven” marker sentence, reaches 28; meaning that it was ranked top, second, or third out of seven cultures by all employees interviewed. Other enterprise cultures form a middle field which will not be addressed in detail here. However, as clearly as the institution is quality driven, it is clearly not hierarchical. 11 of 15 interviewees rank “hierarchical” as the least appropriate to describe the enterpris e culture as they experience it. “Empowerment” is ranked almost as low, which can be interpreted in support of the

conclusions drawn, but will subsequently not be pursued.

Figure 1

Prevailing enterprise model

0

15

30

45

60

75

90

105

value chain

quality driven

mission driven

technology driven

hierarchical

visionary

empowerment

rank sums

best: 15

worst: 105

4.2.

Enterprise culture – a second look

Looking at the details of the interviews and other standardized data reveals that “quality driven” has the more specific meaning of trying to achieve the best for one's own work context while not covering a wider com-mon good. There was no evidence in the interviews that this localized concept of “quality driven” resulted from an explicit agenda or implementation of an enterprise mis -sion statement. Rather, it reflected a lack of visible struc-ture, power, and managerial tools needed to work towards a wider common good. The quote in one of the interviews with a senior management medical officer “I have no power” is paramount evidence for an implied “good in my nutshell” set of behaviors. This can also be read from Figure 2 which displays the interviewees' re-commendations when questioned as to who should take the lead to resolve the blood gas data integration prob -lem. In the 14 interviews, 8 different persons and 3 func-tions within the institution were suggested for this role.

None of these recommendations was out of place. But the fact that this variety existed among people who have been in contact about the problem for up to four years suggested what approaches (other than technical) should the insitution start with to address the problem.

5. Managerial results

The above and some additional evidence was presented to different bodies within the institution, fol-

lowed by suggestions concerning the project team itself and the SOC lineup.

IS1 ISx IS2 IS7 IS5 IS3 IS6 Med1 Med x Mg1 Mg2 Mg3 Mg7 Med y Med4 Med3 SOC IS4 Mg4 Med2 Mg6 Mg5 IS1 ISx IS2 IS7 IS5 IS3 IS6 Med1 Med x Mg1 Mg2 Mg3 Mg7 Med y Med4 Med3 SOC SOC IS4 Mg4 Med2 Mg6 Mg5

Figure 2. Each shape represents a person

(numbers 1 .. 7) or organizational unit (letters x, y) that was suggested as a candidate to take the lead for the continuation of the blood gas data integration project. Med: Medical; IS: Information Services; Mg: Management. Arrow from a to b: a suggests b. Manager 7 suggested the SOC itself.

5.1.

Project team

Concrete staffing, role allocations and a lead for the project team has been suggested. Required roles to be filled are displayed and briefly commented in the following:

External role: One “face” to the vendor

• In analogy to the “One face to the customer” principle already cherished in service industries: Make sure that the vendor need not negotiate separately with all hospitals and cannot make use of their partially diverging interest. Internal roles: Make sure that the face represents you

• Define and maintain highest standard for tertiary care hospital

• Collect variety of solutions re quired by primary through tertiary hospitals

Interface roles: Make sure that vendor can and does deliver

• Make tough contracts (including penalties)

• Control execution

• Offer technical support; make the own asset of the skills of those who developed the former home made system available to the vendor.

5.2.

Software oversight committee

The recommendation was made that the SOC, in addition to its role of technically monitoring projects, also take on the role of makin g sure that projects are appropriately staffed. A major strength of this specific committee is that some of the committee members are academic teachers of medical informatics who have an excellent account of the specific qualifications of many of the employees of the institution because they know them as former students. Once they take on the role of monitoring the staffing of projects, they should be very effective.

The recommendation was also made that the SOC widen its scope from a local (the tertiary care hospital only) to a global (all 21 hospitals) perspective. This is intended to limit the impact of the weakness that local quality oriented people miss the global picture.

6. Discussion and outlook

6.1.

The hypotheses revisited

Both formal and informal data collected support the hypothesis that a certain enterprise culture – as positive as “quality driven” may appear initially - comes with weak points. The weak points may be covered or worked around for some time. Since work arounds need extra energy and effort, efficiency may decay unnoticed until a catastrophic failure hits unexpectedly. This has been

discussed for clinical software in [4]. Here the weak point was the lack of managerial tools to handle the project as an organizational and contractual rather than as a technical challenges.

While hypothesis 1 is supported through a small but comprehensive and representative data set, support for hypothesis 2 is weaker. The strongest support for hypothesis 2 comes from the fact that the SOC under study was granted the extension of its scope from “tertiary hospital only” to enterprise wide immediately after presentation of these results. Whether this had longer ranging effects that would statistically support hypothesis 2 could not be investigated because a new software generation was started soon after this study, which entaile d so many changes that data can not be compared.

6.2.

A broader view at safeguarding

clinical software

To generalize this approach, other case studies could be cond ucted and the instruments used to investigate the enterprise culture could be refined. With some investigations of that type completed, a culture of functional benchmarking could evolve: enterprises could be given advice about how to lineup their SOCs based on how other organizations with similar cultures organize their SOC’s. For software engineering in a more restrictive sense the term “experience factory” has already been coined [5]. From an artificial intelligence perspective this strategy can be called a case based approach. In addition to a case based approach, one could also draw on literature from the managerial sciences or organizational psychology to produce intelligent guesses about what weaknesses to expect given a certain enterprise culture. In either case, having the weaknesses of an institution in mind (and not just its strengths) is likely an important aspect to the process of defining the appropriate focus of the organization’s SOC. SOC have been explicitly given the mission to handle multi vendor situations safely [1]. This implies that observations reported in [4] should be taken into account when staffing an SOC and directing its attention. [4] analyzes factors that may lead to late failures of software that has been successfully launched. Among theses factors, vendor strategies, mergers, and bankruptcies are among the hardest to handle because they are outsid e the sphere where the SOC has a definite role or significant power. If taking seriously this extends the scope of SOC’s beyond their mission as of their inception which is still being taught in [2]: to analyze vendor products rather than taking a thorough look at the vendors as well. If neglected one may feel safe until an unattended and hence unexpected move of one or more vendors occurs.

6.3.

A broader view at human resource

issues

The case study also revealed human resource deficiencies which may transfer to other institutions. The hosting institution of this investigation was and still is world renowned for its technical and scientific achievements in both medicine and medical informatics. More than half of the individuals “be hind” Figure 2 publish in the best medical or medical informatics journals. However, none of them was an MBA, and the only person with a law qualification did not know whether a contract had ever been written with the defaulting vendor.

This may not be the only case. Health care providers who have been successful with home made systems for decades and eventually decide to purchase the new generation of information systems may be wrong in believing that skillful researchers and developers are also smart purchasers. This has also been neglected in the only major investigation of the past years about skills required in hospital IT service departments [6]. This survey bas ed assessment produces top 10 skills through two different polling methods. Both lists include soft skills such as relational skills or oral and written communication in prominent places but only one specifically purchasing related one (“Administrative and business functions”), occurs near the bottom of the lists. Therefore, skills for successfully purchasing are one issue.

Making efficient use of its asset of technical skills of its researcher and developers and of first of a kind solutions they may have and which the enterprise wants preserved in the to be purchased software is another challenge for an enterprise that converts from home -made to purchasing. The challenge is twofold. The enterprise needs to motivate the respective specialists to alienate what used to be their pride by tolerating that it becomes an undistinguishable part of a vendor software that they will no longer be able to call their own. And the enterprise needs to find policies and business agreements (in terms of license fees, free installations, premier customer status, ..) for having offered the skills of its employers to develop the software which is subsequently purchased and installed.

If the challenges are mastered, all parties can win. The health care enterprise wins because it no longer needs to hold avaibale major human workforce for maintenance and upgrading of software. It also wins because it gets a return on investment on its employees’ skills without having to start being a software vendor.

The vendor wins because he can incorporate best of breed technology without having the respective persons on its payroll.

The experts win – once they decide to like the idea – because eventually they can claim that their efforts bear fruit not only in one hospital but in all hospital wh ere the product is purchased from the vendor. They also win because they no longer need to waste time on maintaining their code but can pick up the next technical challenge once they have helped transfer the former one to the vendor.

Finally, the experts remain experts and can safeguard correct function while the software is installed and during its long term function.

7. References

[1] R.A. Miller, R.M. Gardner. „Recommendations for responsible monitoring and regulation of clinical software systems”. JAMIA (1997) 4(6), 442-57

[2] R.M. Gardner, K.E. Tate. „The Software Oversight Committee: Monitoring the Clinical Computing Environment to Reduce Errors and Improve Patient Safety”. Abstract. Proc. AMIA 2003 Annual Symposium, Washington DC Nov. 2003, Tutorial T20

[3] C.P. Friedman, J.C. Wyatt. Evaluation Methods in Biomedical Informatics (Health Informatics), Springer, New York, 1997

[4] Th. Wetter, “To decay is system: The challenges of keeping a health information system alive”. IJMI in press August 2006

[5] V.R. Basili, G. Caldiera, H.D. Rombach (eds) - Encyclopedia of Software Engineering, 1994,

ftp://ftp.cs.umd.edu/pub/sel/papers/fact.pdf (last visited May 17, 2006)

[6] S. Hoffmann, J.Ash, “A Survey of Academic and Industry Professionals Regarding the Preferred Skillset of Graduates of Medical Informatics Programs”. V. Patel et al. (eds) Proc. MedInfo 2001, London, Sept. 2001

Acknowledgements

Univ. of Heidelberg partially funded the author’s stay in Salt Lake City as a sabbatical leave.

Reed Gardner suggested the project and gave valuable advice throughout its duration.

All interviewees offered an openness to have the case investigated that is beyond imagination. The author was not refused a single interview request and never had the slightest impression that details were tried to be hidden. Withdrawals from the interview transcripts before release for this external study were absolutely minor. All personal concerns that may have been present were put behind a common attitude to use the opportunity for improving on ones quality of service.

Bob Skeate read a former version of the manuscript and provided valuable suggestions on wording an readibility.