Building and Validating Autoverification System in Clinical Chemistry Laboratory
Mu-Chin Shih,MT,1,2 Huey-Mei Chang,MS,6 Ni Tien,MS,1,2 Chiung-Tzu Hsiao,MS1,5 Ching-Tien Peng,MD,3,4
(1Department of Laboratory Medicine, 2Medical Laboratory Science and Biotechnology,
3Pediatrics, Children’s Hospital, China Medical University & Hospital, Taichung, Taiwan , 4
Department of Biotechnology, Asia University, 5Medical Laboratory Science and Biotechnology, Central Taiwan University of Science and Technology, Taichung, Taiwan ,6Department of Chemistry, Beckman Coulter Inc. Taiwan)
Corresponding author: Mu-Chin Shih
Deputy Director of Department of Laboratory Medicine, China Medical University Hospital, Taichung, 40447, Taiwan
Associate Professor in Department of Medical Laboratory Science and Biotechnology, China Medical University, Taichung, 40406, Taiwan
Address: No.2, Yude Rd., North Dist., Taichung City 40447, Taiwan (R.O.C.) E-mail : t8147@mail.cmuh.org.tw
TEL : 886-4-22052121 ext.1202 FAX : 886-4-22031029
Mobile : 0975682172
Brief title:
Autoverification System in Clinical Laboratory
Keywords :
Objective: In this study we give a detailed description of how to construct verification rules
and then evaluate the benefits brought to the laboratory.
Methods: All logic processes and verification rules are constructed in middleware with
reference to the CLSI Auto10-A guideline. 569,001 patient test results are collected to
establish the range of the limit check, delta check, and the consistence rule check.
Results: Daily results show the autoverification passing rate of all test results to be 92~95%.
About 80% of test reports can be auto released.
Conclusions: Individual differences in the verification of test results are eliminated, TAT is
shortened and FTE reduced, thus enabling medical technologists to devote more time and
Clinical laboratories must respond to challenges such as reducing manpower
requirements, increasing service quality, simplifying processes and decreasing the report
release TAT (Turn Around Time). In addition to the introduction of automated equipment and
the development of LIS (Laboratory Information System) technology, another way to raise
working efficiency is to build an autoverification system (1-3) by which test reports are
automatically verified against report check rules based on LIS or middleware. Middleware is
information software, installed between LIS and the instruments, which delivers information
such as the test orders from LIS to the instrument, and the test results of the instrument back
to LIS. In an autoverification system, the verification rules and the criteria of the test results
are built into the middleware; so, instead of the results requiring a manual check, they are
automatically verified by computer. These verifications include limit check rules, critical
values, comparison with former results (delta check) and consistency of related results
(consistence check) (4). After the check rules are set, each medical technologist performs the
test result verifications based on the same judgment platform. In this way, check rules for all
test results are standardized (5-7). Furthermore, the autoverification system can expedite the
report check. Central laboratories deal with an enormous number of tests each day and are
always under pressure to quickly report the test results. With an autoverification system, at
least 80% of the test reports can be autoverified without the need of manual intervention,
intercepted in middleware. In May 2008, LAS (Laboratory Automation System) was installed
in our laboratory. After two years of operation trials, LAS had taken the place of most of the
manual efforts. In 2010, the utilization of middleware in an autoverification system was
planned. In order to validate whether the verification rules could actually be implemented and
meet our requirements, a validation and management mechanism based on a CAP checklist
and the CLSI guideline (4, 8, and 9) was established. This manuscript gives a detailed
description of the entire validation and management process, and also an evaluation of the
benefits brought by the autoverification system to the laboratory.
Materials and Methods
COLLECTION OF PATIENT TEST RESULTS
To define the range of the limit check and the delta check for each test item, 569,001 test
results were collected in December 2008. These data were arranged by size, and the
distribution percentages of the limit check and the delta check were calculated. These values
were then used as the basis for establishing and adjusting the limit check and the delta check
of each test item. Also, in order to validate the practicality of the check rules in the
autoverification system: (1) 105,164 patient test results were collected for verification of the
check rules and their correctness in the autoverification system; (2) 830,233 test results,
percentage of all items; and (3) 25,526 test reports were collected for probing into the causes
of manual verification (MV) so as to obtain the true positivity rate and the false positivity rate
of the check rules in the autoverification system.
CONSTRUCTION OF INFORMATION TRANSFER SYSTEM
Both the computer algorithm and the technical data base of the autoverification system were
built in middleware (DM2; provided by Beckman Coulter Inc.). The construction of the
information transfer system is shown in Fig. 1. With DM2 as the core center, LIS sends the
test orders and patient information to three Beckman Coulter DxC800 biochemical analyzers,
two DxI800 immunoassay analyzers and an automated track system (PrepLink, from
Beckman Coulter Inc.). The test results from these five instruments are sent back to LIS after
verification in DM2; LIS then sends the test reports to HIS (Hospital Information System).
COMPUTER ALGORITHM OF AUTOVERIFICATION PROCESS
Based on the reference methods of the manual check which was regularly used in the past and
the CLSI Auto10-A guideline, the critical value check, limit check, delta check, and
consistence check were selected for the verification process (Fig. 2), in addition to patient
information and an instrument warning flag.
within the laboratory’s acceptable limit. Together with test results released from the
instruments, patient information and the sample condition were entered into DM2 for
verification. The order of validation was the critical value check, followed by the limit check,
delta check and, finally, the consistency check. If the results passed the critical value check
but failed the limit check, the delta check was applied, by which the current data were
compared with previous data to determine the differences. When the degree of difference fell
within the limit of delta check acceptability, a consistency check followed. Test results
passing the above check rules were regarded as AV (autoverified) reports, while results which
failed any of the above rules were intercepted as MV (manually verified) reports. There are
various reasons why checks fail and thus require MV results; hence, medical technologists
must make professional judgments based on their experience. When previous data were used
for delta check comparisons the stipulation was that they were to be no older than 7 days.
CONSTRUCTION OF VERIFICATION RULES
The purpose of using autoverification was to produce real time and more accurate test reports
with computer verification. Therefore, the infrastructure of the autoverification rules was
constructed so as to increase the precision and efficiency of the data checks (10). The
verification rules included a limit check, delta check and consistency check. 569,001 test
principles for constructing the verification rules were as follows:
Limit Check: Data that did not fall within the analytical measurement range (AMR) were
considered invalid. For a wide AMR, the acceptable range for the limit check was determined
using a distribution interval of patient data of between 2% and 98%. For instance, 2~98% of
the 14,239 test results of glucose concentration were within the range of 3.6~20.5 mmol/L,
while the AMR for the glucose test was 0.2~33.3 mmol/L; 3.6~20.5 mmol/L was thus
established as the limit check interval of glucose when the verification rule was applied. The
limit check intervals of all 76items were constructed using the same principle.
Delta Check: Opinions on the scope of acceptability of the delta check are not consistent.
False positive rates rise when the acceptable range is stringent, whereas a high number of
false negatives can be expected from the opposite (11, 12). Historical delta check data were
analyzed, and a relevant difference was assigned as the acceptable range according to the
distribution of the delta check of each test item and its clinical specificity in pathological
changes. The parameter of the delta check was set between 5% and 200%: for example, (a)
Sodium (Na+) differed from the previous results by 5%; (b) Chloride ( CL-) by 50%; (c)
Potassium ( K+) by 20%; (d) Vit.B12, total thyroxine (TT4), and total triiodothyronine (TT3)
by 50%; (e) there was no delta check for the C-reactive protein (CRP) test when the test
value was lower than 8 mg/L, and the difference was 100% when the CRP test value was
lower than 0.5 ug/L, and 50% for test results above 0.5 ug/L for Troponin I.
Critical Value Check: As it has been utilized in operation, the critical value bulletined in our
hospital was followed.
Consistence Check: As the medical procedures for acute diseases rapidly change and the
clinical test results fluctuate, it was difficult to perform a consistency check on each item, and
only portions of the test items were established based on practical and clinical diagnostic
criteria. For example, if the test result for thyroid stimulating hormone (TSH) was lower than
0.3 mIU/L and free thyroxine (FT4) was lower than 18.1 pmol/L, the report was intercepted
as an MV report. In addition, at least 30 consistence-check rules were used in the system.
VALIDATION METHODS
To validate whether or not the verification rules and their settings in DM2 were able to meet
our requirements and could actually be executed, electronic simulated data and special
sample validation methods were established as follows:
Electronic simulated data validation: In order to support the validity of the limit check,
delta check and consistency check, 25 entries of simulated data were created on simulation
software built in DM2. Of these,13 data entries fell outside the acceptable range in the limit
check, delta check or critical value check, and 12 data entries did not fulfill the consistency
validation procedure was shown to have a satisfactory performance and to fulfill the
necessary requirements.
Special sample validation: Special samples included abnormal proficiency test samples and
more than 20 patient samples. Most of the test results fell outside the acceptable range of the
limit check and the critical value check .These test results of the special samples were
selected to validate the autoverification system’s functionality and the reliability of the
reports.
Results
LABORATORY TRIAL RUN OF ACTUAL PATIENT TEST RESULTS
According to the CLSI Auto10-A guideline (4), an autoverification system must be validated
using actual patient results upon startup. A total of 105,164 test results from August 18 to
August 26, 2010 were collected and accessed in the autoverification system to verify the
results. The failing and passing rates of the delta checks and the limit checks were then
computed, and the results are shown in Table 1.
As shown in Table 1, 11~17% of all test results underwent a delta check each day, with
10~14% of the test results failing the delta check. Approximately 2.2% to 3.4% of all test
results failed the limit check rules each day and, of these, about 71~83% of the test results did
limit check, 5~11% also failed the delta check, whereas 11~20% passed the delta check. The
daily passing rate for autoverifications was 95% to 97%. Due to the uniqueness and
specificity of certain medical treatments, there can be numerous associated unpredictable
factors; however, the above day-to-day variation was considered acceptable.
AUTOVERIFICATION PASSING RATE
There were 76 test items involved in our autoverification system, with 42 biochemically (BIO)
related and 34 immunoassay (IA) related. For the purpose of testing the autoverification
passing rate, 830,233 test results which contained 139,650 requisition sheets were collected
from Feb. to May 2010. By category, the average passing rate was 96.1% for the BIO-related
test items, 93.9% for the IA test items, and the overall passing rate for the BIO and IA tests
was 95.6%. In terms of the test requisition sheets, the passing rate was 81.5% (Fig. 3); these
reports required no manual check and could be automatically verified (AV). Because each
requisition sheet usually included several test items, manual verification was necessary if one
item failed any of the verification rules. Therefore, the autoverification passing rate
calculated in terms of the test requisition sheets was far less than that done for individual test
items (81.5% vs. 95.6%).In addition, when compiling the statistics of the 830,233 test results
of the 76 test items, the results showed that 62 test items had a passing rate higher than 90% ,
were lower than 70% (Fig. 4).
CAUSE ANALYSIS OF MV TEST REPORTS
To understand the causes necessitating the requirement of manual intervention and, thus, MV
data, a total of 25,526 patient reports were collected, with 4,903 reports being classified as
MV. The causes of MV reports were analyzed, and the results are shown in Table 2. Of these
MV reports, 41 reports (0.8%) had remarkably abnormal data, which may have been due to
sample contamination or anticoagulant interference. Another 300 MV test reports (6.1%)
were marked with error flags from the analyzer: the data were questionable and all were
intercepted for further confirmation. Another 4562 MV test reports (93.1%) were classified as
acceptable MV data, since they were released without the need of any modification in the
original test results after consultation with a clinician. Hence, it could be estimated that the
true positivity rate of the intercepted reports was 6.9%.
Discussion
In the given validation of the check rules for the test reports, patient test results were used as
the main framework of the validation (see Table 1), as their representation of the actual
patient results spanned the ranges acceptable for most rules of validation. However, some of
autoverification system; as well, such samples were not easily identified in a short period of
time. For example: albumin>total protein, Creatine Kinase-MB>Creatine Kinase. In order to
validate these rules or cases, we used simulation software in the middleware to validate
whether the verification rules and process were well implemented. Our results demonstrated
that the rules and workflow designed were able to be correctly executed. Table 1 and Figure
3 show that the autoverification system designed in this study yielded an average passing rate
of 95% for all test results, and a higher AV rate, as compared to previously published data, of
73% (13). This could be because different laboratories have different acceptable ranges for
the limit check and the delta check for each test item.For example, the range of the delta
check was set between 5% and 200% in our laboratory, while others may adopt 20% to 30%
for their acceptable range (13). Furthermore, as shown in Fig. 4, two test items, ethanol and
human chorionic gonadotropin (hCG), were accepted via AV with a passing rate lower than
70% (27% and 68%, respectively).It was concluded that the legal cutoff of the alcohol
concentration for illegal drunk driving in Taiwan was 10.8 mmol/L, and requests for blood
alcohol tests always involved traffic issues. For this reason, the blood alcohol concentration
was usually higher than the legal ethanol concentration cutoff. As the reports might be used
for legal purposes, the removal of such test items and the application of manual checks to all
ethanol-related reports was considered. A low AV passing rate of hCG was correlated with the
women usually have an hCG level higher than AMR. Although the instrument might
automatically dilute the samples and mark it with a dilution flag, the processing of the result
still required manual intervention.
One of the most important functions in the autoverification system was to hold the
samples with errors reports. Table 2 shows that of the 25,526 patient reports, 41 were
classified as having remarkably abnormal data and requiring interception or rejection. Further
investigation revealed that the abnormalities in the data were due to serious hemolysis during
blood sampling (test items such as K, LD, AST, etc.), EDTA contamination (test items K and
CA failed the limit check and the delta check), and severe lipemia or insufficient samples.
Also, 6.9% of the MV data was rendered non-reportable due to an error flag marked by the
instrument. The remaining 4,562 (93.1%) reports were classified as acceptable MV data, and
the required manual intervention was identified to be caused by changes during medical
procedure.These test results were reported after a brief communication with a clinician.
These samples were re-tested, and the results were consistent with the previous results.When
compiling the statistics according to the panel tests in the test requisition sheets, the AV
passing rate was 81.5% at the current stage, which greatly alleviated the burden in report
verification.
The stability of the performance of the automated analyzers now in use is relatively better.
function of an autoverification system is not only to detect errors in the test results, but also,
by its application, to improve patient clinical care. For example, results in the delta check
falling outside of the acceptable range could imply a significant change in the patient’s
condition. In such cases, the laboratory would inform the physician so the patient can receive
appropriate treatment.
Quality control (QC) results should confirm if an item falls within the acceptable range
before the autoverification system starts; otherwise the verification procedure should cease.
At present, the QC system is not connected with the autoverification system of the
middleware; hence, manual intervention had to be adopted to control the mechanism which
determined whether the QC results were acceptable. For this reason, the QC system should
be integrated with the autoverification system; then, when QC failure occurs, the
autoverification of the given test item will automatically stop.
Most publications describe the autoverification system as a way to shorten TAT in
reporting test results and thus reduce the labor burden in the laboratory (13). However, as
different laboratories have different working configurations and workflow, it was estimated
that the use of an autoverification system would have an FTE of 3.5 full time employees per
year in our laboratory. The TAT of patient reporting was dramatically reduced because we
released the AV test reports immediately instead of the previous batch release. However, our
test result verifications done by different medical technologists. All the test results were
autoverified based on the same standard and treated in the same manner, thus ensuring the
quality of the reports. However, an autoverification system is also subject to limitations. Even
though the check rules seem flawless, errors can still occur. An example might be the
mislabeling of the patient sample. If the data have merely slight differences and are not
intercepted by the delta check, incorrect reports might be issued. On the other hand,
erroneous data might still be reported when the interferences in the sample, such as a partial
clot, have not been detected by the instrument. Technical errors of this kind, although few,
remain unavoidable. Another reason for using the system would be to allow medical
technologists to spend more time and effort focusing on the handling of MV test reports and,
thus, improve the quality of patient care.
Acknowledgments: The authors thank Dr. Jang Jih Lu for his encouragement and Beckman
References
1. Southwick K. Expert systems a feast for leaner laboratories. CAP Today;Jan 2002. http://www.cap.org/apps/docs/cap_today/feature_stories/systems_feature.html (accessed October 2005).
2. Pearlman ES, Bilello L, Stauffer J, Kamarinos A, Miele R, Wolfert MS. Implications of autoverification for the clinical laboratory. Clin Leadersh Manag Rev. 2002;16:237-239. 3. Felder R. Laboratory reporting for the future: linking autoverification to the electronic
medical record. American Association for Clinical Chemistry. AACC Continuing Education. http://www.aacc.org/access/record/default.stm (accessed October 2005).
4. Autoverification of Clinical Laboratory Test Results; Approved Guideline CLSI
AUTO10-A,2006.
5. Rao LV, Okorodudu AO. Integrated automation in the clinical laboratory. Lewandrowski K eds. Clinical chemistry: laboratory management and clinical correlations 2002:205-211 Lippincott New York.
6. Davis GM. Autoverification of the peripheral blood count. Lab Med 1994;25:528. 7. Davis GM. Autoverification of macroscopic urinalysis. Lab Med 1999;30:56-60. 8. College of American Pathologists Lab General Checklist AUTOVERIFICATION
GEN.43850~43893, 2009.
9. Duca DJ. Autoverification in a laboratory information system. Lab Med 2002;33:21-25. 10. Crolla LJ, Westgard JO. Evaluation of rule-based autoverification protocols. Clin
Leadersh Manag Rev. 2003;17:268-272.
11. Wheeler LA, Sheiner LB. A clinical evaluation of various delta check methods. Clin
Chem. 1981;27:5-9.
12. Ohara T, Itoh K. [Revisited univariate delta check method for hematologic laboratories (I)--Usefulness for detection of specimen mix-up in patients with hematologic disorders].
Rinsho Byori. 2002;50:1072-1075.
13. Torke N, Boral L, Nguyen T, Perri A, Chakrin A. Process improvement and operational efficiency through test result autoverification. Clin Chem. 2005;51:2406-2408.
Table 1: Data of the 105,164 test results generated from the trial run of the autoverification system.
Check rules Aug.18 Aug.19 Aug.20 Aug.23 Aug.24 Aug.25 Aug.26 Total
Number of test results 14712 16388 13726 17250 14135 13949 15004 105164
1. Test results which
undergo Delta check 1699,(12%) 2297,(14%) 1815,(13%) 2874,(17%) 1925,(14%) 1513,(11%) 2134,(14%) 14257,(14%) (1). Test results failing
Delta check (MV) 167,(10%) 256,(11%) 218,(12%) 323,(11%) 257,(13%) 206,(14%) 249,(12%) 1676,(12%) 2. Test results failing
Limit check 407,(2.8%) 376,(2.2%) 354,(2.6%) 458,(2.7%) 381,(2.7%) 471,(3.4%) 378,(2.5%) 2825,(2.7%) (1). Test results failing
Limit check but
without Delta check (MV)
325,(80%) 267,(71%) 282,(80%) 336,(73%) 292,(77%) 393,(83%) 274,(72%) 2169,(77%)
(2). Test results failing Limit check and Delta
check (MV)
19,(5%) 32,(9%) 17,(5%) 38,(8%) 33,(9%) 27,(6%) 40,(11%) 206,(7%)
(3). Test results failing Limit check but passing Delta check (AV)
63,(15%) 77,(20%) 55,(15%) 84,(19%) 56,(14%) 51,(11%) 64,(17%) 450,(16%)
Autoverification passing rate
% (by test) 95.4% 96.7% 95.1% 94.8% 94.7% 94.0% 95.2% 95.2%
Table 2: Cause analysis of MV test reports
Time interval Oct. 1 ~ Oct. 15 Total requisition sheet 25526
AV requisition sheet / (%) 20623 / (80.8%) MV requisition sheet / (%) 4903 / (19.2%) *Remarkably abnormal data / (%) 41 / (0.8%) Data with error flag / (%) 300 / (6.1%) †Acceptable MV data / (%) 4562 / (93.1%)
*Extremely abnormal results intercepted.
Legends for Figures
Fig. 1: DM2 as the core center of the autoverification system
Fig. 2: Algorithm design of autoverification (MV: manual verification; AV: auto verification)
Fig. 3: Autoverification passing rate of immunoassay (IA) items, biochemically related test
items (BIO), and all (IA+BIO) test items. About 81.5% of patient reports could be
auto-released without manual intervention.