• 沒有找到結果。

Building and Validating Autoverification System in Clinical Chemistry Laboratory

N/A
N/A
Protected

Academic year: 2021

Share "Building and Validating Autoverification System in Clinical Chemistry Laboratory"

Copied!
18
0
0

加載中.... (立即查看全文)

全文

(1)

Building and Validating Autoverification System in Clinical Chemistry Laboratory

Mu-Chin Shih,MT,1,2 Huey-Mei Chang,MS,6 Ni Tien,MS,1,2 Chiung-Tzu Hsiao,MS1,5 Ching-Tien Peng,MD,3,4

(1Department of Laboratory Medicine, 2Medical Laboratory Science and Biotechnology,

3Pediatrics, Children’s Hospital, China Medical University & Hospital, Taichung, Taiwan , 4

Department of Biotechnology, Asia University, 5Medical Laboratory Science and Biotechnology, Central Taiwan University of Science and Technology, Taichung, Taiwan ,6Department of Chemistry, Beckman Coulter Inc. Taiwan)

Corresponding author: Mu-Chin Shih

Deputy Director of Department of Laboratory Medicine, China Medical University Hospital, Taichung, 40447, Taiwan

Associate Professor in Department of Medical Laboratory Science and Biotechnology, China Medical University, Taichung, 40406, Taiwan

Address: No.2, Yude Rd., North Dist., Taichung City 40447, Taiwan (R.O.C.) E-mail : t8147@mail.cmuh.org.tw

TEL : 886-4-22052121 ext.1202 FAX : 886-4-22031029

Mobile : 0975682172

Brief title:

Autoverification System in Clinical Laboratory

Keywords :

(2)

Objective: In this study we give a detailed description of how to construct verification rules

and then evaluate the benefits brought to the laboratory.

Methods: All logic processes and verification rules are constructed in middleware with

reference to the CLSI Auto10-A guideline. 569,001 patient test results are collected to

establish the range of the limit check, delta check, and the consistence rule check.

Results: Daily results show the autoverification passing rate of all test results to be 92~95%.

About 80% of test reports can be auto released.

Conclusions: Individual differences in the verification of test results are eliminated, TAT is

shortened and FTE reduced, thus enabling medical technologists to devote more time and

(3)

Clinical laboratories must respond to challenges such as reducing manpower

requirements, increasing service quality, simplifying processes and decreasing the report

release TAT (Turn Around Time). In addition to the introduction of automated equipment and

the development of LIS (Laboratory Information System) technology, another way to raise

working efficiency is to build an autoverification system (1-3) by which test reports are

automatically verified against report check rules based on LIS or middleware. Middleware is

information software, installed between LIS and the instruments, which delivers information

such as the test orders from LIS to the instrument, and the test results of the instrument back

to LIS. In an autoverification system, the verification rules and the criteria of the test results

are built into the middleware; so, instead of the results requiring a manual check, they are

automatically verified by computer. These verifications include limit check rules, critical

values, comparison with former results (delta check) and consistency of related results

(consistence check) (4). After the check rules are set, each medical technologist performs the

test result verifications based on the same judgment platform. In this way, check rules for all

test results are standardized (5-7). Furthermore, the autoverification system can expedite the

report check. Central laboratories deal with an enormous number of tests each day and are

always under pressure to quickly report the test results. With an autoverification system, at

least 80% of the test reports can be autoverified without the need of manual intervention,

(4)

intercepted in middleware. In May 2008, LAS (Laboratory Automation System) was installed

in our laboratory. After two years of operation trials, LAS had taken the place of most of the

manual efforts. In 2010, the utilization of middleware in an autoverification system was

planned. In order to validate whether the verification rules could actually be implemented and

meet our requirements, a validation and management mechanism based on a CAP checklist

and the CLSI guideline (4, 8, and 9) was established. This manuscript gives a detailed

description of the entire validation and management process, and also an evaluation of the

benefits brought by the autoverification system to the laboratory.

Materials and Methods

COLLECTION OF PATIENT TEST RESULTS

To define the range of the limit check and the delta check for each test item, 569,001 test

results were collected in December 2008. These data were arranged by size, and the

distribution percentages of the limit check and the delta check were calculated. These values

were then used as the basis for establishing and adjusting the limit check and the delta check

of each test item. Also, in order to validate the practicality of the check rules in the

autoverification system: (1) 105,164 patient test results were collected for verification of the

check rules and their correctness in the autoverification system; (2) 830,233 test results,

(5)

percentage of all items; and (3) 25,526 test reports were collected for probing into the causes

of manual verification (MV) so as to obtain the true positivity rate and the false positivity rate

of the check rules in the autoverification system.

CONSTRUCTION OF INFORMATION TRANSFER SYSTEM

Both the computer algorithm and the technical data base of the autoverification system were

built in middleware (DM2; provided by Beckman Coulter Inc.). The construction of the

information transfer system is shown in Fig. 1. With DM2 as the core center, LIS sends the

test orders and patient information to three Beckman Coulter DxC800 biochemical analyzers,

two DxI800 immunoassay analyzers and an automated track system (PrepLink, from

Beckman Coulter Inc.). The test results from these five instruments are sent back to LIS after

verification in DM2; LIS then sends the test reports to HIS (Hospital Information System).

COMPUTER ALGORITHM OF AUTOVERIFICATION PROCESS

Based on the reference methods of the manual check which was regularly used in the past and

the CLSI Auto10-A guideline, the critical value check, limit check, delta check, and

consistence check were selected for the verification process (Fig. 2), in addition to patient

information and an instrument warning flag.

(6)

within the laboratory’s acceptable limit. Together with test results released from the

instruments, patient information and the sample condition were entered into DM2 for

verification. The order of validation was the critical value check, followed by the limit check,

delta check and, finally, the consistency check. If the results passed the critical value check

but failed the limit check, the delta check was applied, by which the current data were

compared with previous data to determine the differences. When the degree of difference fell

within the limit of delta check acceptability, a consistency check followed. Test results

passing the above check rules were regarded as AV (autoverified) reports, while results which

failed any of the above rules were intercepted as MV (manually verified) reports. There are

various reasons why checks fail and thus require MV results; hence, medical technologists

must make professional judgments based on their experience. When previous data were used

for delta check comparisons the stipulation was that they were to be no older than 7 days.

CONSTRUCTION OF VERIFICATION RULES

The purpose of using autoverification was to produce real time and more accurate test reports

with computer verification. Therefore, the infrastructure of the autoverification rules was

constructed so as to increase the precision and efficiency of the data checks (10). The

verification rules included a limit check, delta check and consistency check. 569,001 test

(7)

principles for constructing the verification rules were as follows:

Limit Check: Data that did not fall within the analytical measurement range (AMR) were

considered invalid. For a wide AMR, the acceptable range for the limit check was determined

using a distribution interval of patient data of between 2% and 98%. For instance, 2~98% of

the 14,239 test results of glucose concentration were within the range of 3.6~20.5 mmol/L,

while the AMR for the glucose test was 0.2~33.3 mmol/L; 3.6~20.5 mmol/L was thus

established as the limit check interval of glucose when the verification rule was applied. The

limit check intervals of all 76items were constructed using the same principle.

Delta Check: Opinions on the scope of acceptability of the delta check are not consistent.

False positive rates rise when the acceptable range is stringent, whereas a high number of

false negatives can be expected from the opposite (11, 12). Historical delta check data were

analyzed, and a relevant difference was assigned as the acceptable range according to the

distribution of the delta check of each test item and its clinical specificity in pathological

changes. The parameter of the delta check was set between 5% and 200%: for example, (a)

Sodium (Na+) differed from the previous results by 5%; (b) Chloride ( CL-) by 50%; (c)

Potassium ( K+) by 20%; (d) Vit.B12, total thyroxine (TT4), and total triiodothyronine (TT3)

by 50%; (e) there was no delta check for the C-reactive protein (CRP) test when the test

value was lower than 8 mg/L, and the difference was 100% when the CRP test value was

(8)

lower than 0.5 ug/L, and 50% for test results above 0.5 ug/L for Troponin I.

Critical Value Check: As it has been utilized in operation, the critical value bulletined in our

hospital was followed.

Consistence Check: As the medical procedures for acute diseases rapidly change and the

clinical test results fluctuate, it was difficult to perform a consistency check on each item, and

only portions of the test items were established based on practical and clinical diagnostic

criteria. For example, if the test result for thyroid stimulating hormone (TSH) was lower than

0.3 mIU/L and free thyroxine (FT4) was lower than 18.1 pmol/L, the report was intercepted

as an MV report. In addition, at least 30 consistence-check rules were used in the system.

VALIDATION METHODS

To validate whether or not the verification rules and their settings in DM2 were able to meet

our requirements and could actually be executed, electronic simulated data and special

sample validation methods were established as follows:

Electronic simulated data validation: In order to support the validity of the limit check,

delta check and consistency check, 25 entries of simulated data were created on simulation

software built in DM2. Of these,13 data entries fell outside the acceptable range in the limit

check, delta check or critical value check, and 12 data entries did not fulfill the consistency

(9)

validation procedure was shown to have a satisfactory performance and to fulfill the

necessary requirements.

Special sample validation: Special samples included abnormal proficiency test samples and

more than 20 patient samples. Most of the test results fell outside the acceptable range of the

limit check and the critical value check .These test results of the special samples were

selected to validate the autoverification system’s functionality and the reliability of the

reports.

Results

LABORATORY TRIAL RUN OF ACTUAL PATIENT TEST RESULTS

According to the CLSI Auto10-A guideline (4), an autoverification system must be validated

using actual patient results upon startup. A total of 105,164 test results from August 18 to

August 26, 2010 were collected and accessed in the autoverification system to verify the

results. The failing and passing rates of the delta checks and the limit checks were then

computed, and the results are shown in Table 1.

As shown in Table 1, 11~17% of all test results underwent a delta check each day, with

10~14% of the test results failing the delta check. Approximately 2.2% to 3.4% of all test

results failed the limit check rules each day and, of these, about 71~83% of the test results did

(10)

limit check, 5~11% also failed the delta check, whereas 11~20% passed the delta check. The

daily passing rate for autoverifications was 95% to 97%. Due to the uniqueness and

specificity of certain medical treatments, there can be numerous associated unpredictable

factors; however, the above day-to-day variation was considered acceptable.

AUTOVERIFICATION PASSING RATE

There were 76 test items involved in our autoverification system, with 42 biochemically (BIO)

related and 34 immunoassay (IA) related. For the purpose of testing the autoverification

passing rate, 830,233 test results which contained 139,650 requisition sheets were collected

from Feb. to May 2010. By category, the average passing rate was 96.1% for the BIO-related

test items, 93.9% for the IA test items, and the overall passing rate for the BIO and IA tests

was 95.6%. In terms of the test requisition sheets, the passing rate was 81.5% (Fig. 3); these

reports required no manual check and could be automatically verified (AV). Because each

requisition sheet usually included several test items, manual verification was necessary if one

item failed any of the verification rules. Therefore, the autoverification passing rate

calculated in terms of the test requisition sheets was far less than that done for individual test

items (81.5% vs. 95.6%).In addition, when compiling the statistics of the 830,233 test results

of the 76 test items, the results showed that 62 test items had a passing rate higher than 90% ,

(11)

were lower than 70% (Fig. 4).

CAUSE ANALYSIS OF MV TEST REPORTS

To understand the causes necessitating the requirement of manual intervention and, thus, MV

data, a total of 25,526 patient reports were collected, with 4,903 reports being classified as

MV. The causes of MV reports were analyzed, and the results are shown in Table 2. Of these

MV reports, 41 reports (0.8%) had remarkably abnormal data, which may have been due to

sample contamination or anticoagulant interference. Another 300 MV test reports (6.1%)

were marked with error flags from the analyzer: the data were questionable and all were

intercepted for further confirmation. Another 4562 MV test reports (93.1%) were classified as

acceptable MV data, since they were released without the need of any modification in the

original test results after consultation with a clinician. Hence, it could be estimated that the

true positivity rate of the intercepted reports was 6.9%.

Discussion

In the given validation of the check rules for the test reports, patient test results were used as

the main framework of the validation (see Table 1), as their representation of the actual

patient results spanned the ranges acceptable for most rules of validation. However, some of

(12)

autoverification system; as well, such samples were not easily identified in a short period of

time. For example: albumin>total protein, Creatine Kinase-MB>Creatine Kinase. In order to

validate these rules or cases, we used simulation software in the middleware to validate

whether the verification rules and process were well implemented. Our results demonstrated

that the rules and workflow designed were able to be correctly executed. Table 1 and Figure

3 show that the autoverification system designed in this study yielded an average passing rate

of 95% for all test results, and a higher AV rate, as compared to previously published data, of

73% (13). This could be because different laboratories have different acceptable ranges for

the limit check and the delta check for each test item.For example, the range of the delta

check was set between 5% and 200% in our laboratory, while others may adopt 20% to 30%

for their acceptable range (13). Furthermore, as shown in Fig. 4, two test items, ethanol and

human chorionic gonadotropin (hCG), were accepted via AV with a passing rate lower than

70% (27% and 68%, respectively).It was concluded that the legal cutoff of the alcohol

concentration for illegal drunk driving in Taiwan was 10.8 mmol/L, and requests for blood

alcohol tests always involved traffic issues. For this reason, the blood alcohol concentration

was usually higher than the legal ethanol concentration cutoff. As the reports might be used

for legal purposes, the removal of such test items and the application of manual checks to all

ethanol-related reports was considered. A low AV passing rate of hCG was correlated with the

(13)

women usually have an hCG level higher than AMR. Although the instrument might

automatically dilute the samples and mark it with a dilution flag, the processing of the result

still required manual intervention.

One of the most important functions in the autoverification system was to hold the

samples with errors reports. Table 2 shows that of the 25,526 patient reports, 41 were

classified as having remarkably abnormal data and requiring interception or rejection. Further

investigation revealed that the abnormalities in the data were due to serious hemolysis during

blood sampling (test items such as K, LD, AST, etc.), EDTA contamination (test items K and

CA failed the limit check and the delta check), and severe lipemia or insufficient samples.

Also, 6.9% of the MV data was rendered non-reportable due to an error flag marked by the

instrument. The remaining 4,562 (93.1%) reports were classified as acceptable MV data, and

the required manual intervention was identified to be caused by changes during medical

procedure.These test results were reported after a brief communication with a clinician.

These samples were re-tested, and the results were consistent with the previous results.When

compiling the statistics according to the panel tests in the test requisition sheets, the AV

passing rate was 81.5% at the current stage, which greatly alleviated the burden in report

verification.

The stability of the performance of the automated analyzers now in use is relatively better.

(14)

function of an autoverification system is not only to detect errors in the test results, but also,

by its application, to improve patient clinical care. For example, results in the delta check

falling outside of the acceptable range could imply a significant change in the patient’s

condition. In such cases, the laboratory would inform the physician so the patient can receive

appropriate treatment.

Quality control (QC) results should confirm if an item falls within the acceptable range

before the autoverification system starts; otherwise the verification procedure should cease.

At present, the QC system is not connected with the autoverification system of the

middleware; hence, manual intervention had to be adopted to control the mechanism which

determined whether the QC results were acceptable. For this reason, the QC system should

be integrated with the autoverification system; then, when QC failure occurs, the

autoverification of the given test item will automatically stop.

Most publications describe the autoverification system as a way to shorten TAT in

reporting test results and thus reduce the labor burden in the laboratory (13). However, as

different laboratories have different working configurations and workflow, it was estimated

that the use of an autoverification system would have an FTE of 3.5 full time employees per

year in our laboratory. The TAT of patient reporting was dramatically reduced because we

released the AV test reports immediately instead of the previous batch release. However, our

(15)

test result verifications done by different medical technologists. All the test results were

autoverified based on the same standard and treated in the same manner, thus ensuring the

quality of the reports. However, an autoverification system is also subject to limitations. Even

though the check rules seem flawless, errors can still occur. An example might be the

mislabeling of the patient sample. If the data have merely slight differences and are not

intercepted by the delta check, incorrect reports might be issued. On the other hand,

erroneous data might still be reported when the interferences in the sample, such as a partial

clot, have not been detected by the instrument. Technical errors of this kind, although few,

remain unavoidable. Another reason for using the system would be to allow medical

technologists to spend more time and effort focusing on the handling of MV test reports and,

thus, improve the quality of patient care.

Acknowledgments: The authors thank Dr. Jang Jih Lu for his encouragement and Beckman

(16)

References

1. Southwick K. Expert systems a feast for leaner laboratories. CAP Today;Jan 2002. http://www.cap.org/apps/docs/cap_today/feature_stories/systems_feature.html (accessed October 2005).

2. Pearlman ES, Bilello L, Stauffer J, Kamarinos A, Miele R, Wolfert MS. Implications of autoverification for the clinical laboratory. Clin Leadersh Manag Rev. 2002;16:237-239. 3. Felder R. Laboratory reporting for the future: linking autoverification to the electronic

medical record. American Association for Clinical Chemistry. AACC Continuing Education. http://www.aacc.org/access/record/default.stm (accessed October 2005).

4. Autoverification of Clinical Laboratory Test Results; Approved Guideline CLSI

AUTO10-A,2006.

5. Rao LV, Okorodudu AO. Integrated automation in the clinical laboratory. Lewandrowski K eds. Clinical chemistry: laboratory management and clinical correlations 2002:205-211 Lippincott New York.

6. Davis GM. Autoverification of the peripheral blood count. Lab Med 1994;25:528. 7. Davis GM. Autoverification of macroscopic urinalysis. Lab Med 1999;30:56-60. 8. College of American Pathologists Lab General Checklist AUTOVERIFICATION

GEN.43850~43893, 2009.

9. Duca DJ. Autoverification in a laboratory information system. Lab Med 2002;33:21-25. 10. Crolla LJ, Westgard JO. Evaluation of rule-based autoverification protocols. Clin

Leadersh Manag Rev. 2003;17:268-272.

11. Wheeler LA, Sheiner LB. A clinical evaluation of various delta check methods. Clin

Chem. 1981;27:5-9.

12. Ohara T, Itoh K. [Revisited univariate delta check method for hematologic laboratories (I)--Usefulness for detection of specimen mix-up in patients with hematologic disorders].

Rinsho Byori. 2002;50:1072-1075.

13. Torke N, Boral L, Nguyen T, Perri A, Chakrin A. Process improvement and operational efficiency through test result autoverification. Clin Chem. 2005;51:2406-2408.

(17)

Table 1: Data of the 105,164 test results generated from the trial run of the autoverification system.

Check rules Aug.18 Aug.19 Aug.20 Aug.23 Aug.24 Aug.25 Aug.26 Total

Number of test results 14712 16388 13726 17250 14135 13949 15004 105164

1. Test results which

undergo Delta check 1699,(12%) 2297,(14%) 1815,(13%) 2874,(17%) 1925,(14%) 1513,(11%) 2134,(14%) 14257,(14%) (1). Test results failing

Delta check (MV) 167,(10%) 256,(11%) 218,(12%) 323,(11%) 257,(13%) 206,(14%) 249,(12%) 1676,(12%) 2. Test results failing

Limit check 407,(2.8%) 376,(2.2%) 354,(2.6%) 458,(2.7%) 381,(2.7%) 471,(3.4%) 378,(2.5%) 2825,(2.7%) (1). Test results failing

Limit check but

without Delta check (MV)

325,(80%) 267,(71%) 282,(80%) 336,(73%) 292,(77%) 393,(83%) 274,(72%) 2169,(77%)

(2). Test results failing Limit check and Delta

check (MV)

19,(5%) 32,(9%) 17,(5%) 38,(8%) 33,(9%) 27,(6%) 40,(11%) 206,(7%)

(3). Test results failing Limit check but passing Delta check (AV)

63,(15%) 77,(20%) 55,(15%) 84,(19%) 56,(14%) 51,(11%) 64,(17%) 450,(16%)

Autoverification passing rate

% (by test) 95.4% 96.7% 95.1% 94.8% 94.7% 94.0% 95.2% 95.2%

Table 2: Cause analysis of MV test reports

Time interval Oct. 1 ~ Oct. 15 Total requisition sheet 25526

AV requisition sheet / (%) 20623 / (80.8%) MV requisition sheet / (%) 4903 / (19.2%) *Remarkably abnormal data / (%) 41 / (0.8%) Data with error flag / (%) 300 / (6.1%) †Acceptable MV data / (%) 4562 / (93.1%)

*Extremely abnormal results intercepted.

(18)

Legends for Figures

Fig. 1: DM2 as the core center of the autoverification system

Fig. 2: Algorithm design of autoverification (MV: manual verification; AV: auto verification)

Fig. 3: Autoverification passing rate of immunoassay (IA) items, biochemically related test

items (BIO), and all (IA+BIO) test items. About 81.5% of patient reports could be

auto-released without manual intervention.

數據

Table 1: Data of the 105,164 test results generated from the trial run of the autoverification  system

參考文獻

相關文件

The first row shows the eyespot with white inner ring, black middle ring, and yellow outer ring in Bicyclus anynana.. The second row provides the eyespot with black inner ring

You are given the wavelength and total energy of a light pulse and asked to find the number of photons it

好了既然 Z[x] 中的 ideal 不一定是 principle ideal 那麼我們就不能學 Proposition 7.2.11 的方法得到 Z[x] 中的 irreducible element 就是 prime element 了..

Wang, Solving pseudomonotone variational inequalities and pseudocon- vex optimization problems using the projection neural network, IEEE Transactions on Neural Networks 17

volume suppressed mass: (TeV) 2 /M P ∼ 10 −4 eV → mm range can be experimentally tested for any number of extra dimensions - Light U(1) gauge bosons: no derivative couplings. =>

For pedagogical purposes, let us start consideration from a simple one-dimensional (1D) system, where electrons are confined to a chain parallel to the x axis. As it is well known

The observed small neutrino masses strongly suggest the presence of super heavy Majorana neutrinos N. Out-of-thermal equilibrium processes may be easily realized around the

Define instead the imaginary.. potential, magnetic field, lattice…) Dirac-BdG Hamiltonian:. with small, and matrix