Evaluating diagnostic tests with imperfect standards.

New diagnostic tests frequently are evaluated against gold standards that are assumed to classify patients with unerring accuracy according to the presence or absence of disease. In practice, gold standards rarely are perfect predictors of disease and tend to misclassify a small number of patients. When an imperfect standard is used to evaluate a diagnostic test, many commonly used measures of test performance are distorted. It is not widely appreciated that these distortions occur in predictable directions and that they may be of considerable magnitude, even when the gold standard has a high degree of accuracy. The diagnostic powers of clinical tests will be more accurately reported if consideration is given to the types of biases that result from the use of imperfect standards. Several different approaches may be used to minimize these distortions when evaluating new tests.

Unnecessary microscopy in routine urinalysis.

The consequence of omitting urine sediment microscopy in specimens with normal physicochemical testing was assessed in a retrospective review of laboratory and clinical data in 1,000 patients. The pathologic states of clinical significance most likely to be overlooked were Trichomonas infection and occasional cases of significant bacteriuria. However, the medical benefit of performing urine microscopy in these two groups of patients was not clear. The authors cautiously recommend reserving microscopy for urine specimens that show physicochemical abnormalities.

Sensitivity, specificity, and reproducibility of four measures of laboratory turnaround time.

The authors studied the performance of four measures of laboratory turnaround time: the mean, median, 90th percentile, and proportion of tests reported within a predetermined cut-off interval (proportion of acceptable tests [PAT]). Measures were examined with the use of turnaround time data from 11,070 stat partial thromboplastin times, 16,761 urine cultures, and 28,055 stat electrolyte panels performed by a single laboratory. For laboratories with long turnaround times, the most important quality of a turnaround time measure is high reproducibility, so that improvement in reporting speed can be distinguished from random variation resulting from sampling. The mean was found to be the most reproducible of the four measures, followed by the median. The mean achieved acceptable precision with sample sizes of 100-500 tests. For laboratories with normally rapid turnaround times, the most important quality of a measure is high sensitivity and specificity for detecting whether turnaround time has dropped below standards. The PAT was found to be the best measure of turnaround time in this setting but required sample sizes of at least 500 tests to achieve acceptable accuracy. Laboratory turnaround time may be measured for different reasons. The method of measurement should be chosen with an eye toward its intended application.

Teaching pathology residents to evaluate the quality of laboratory testing.

In-house reviews of laboratory quality have become an important component of efforts to preserve the caliber and breadth of laboratory services in the face of contracting budgets. We describe a training rotation in quality assessment that provides pathology residents with the skills necessary to systematically investigate the quality of laboratory testing. The rotation has been of mutual benefit to our residents and our institution, serving as an introduction to a variety of analytic techniques and as a mechanism for documenting and improving the performance and efficiency of our hospital laboratory.

Ordering accuracy. A College of American Pathologists Q-Probes study of 577 institutions.

Five hundred seventy-seven institutions examined how accurately physicians' test orders on inpatients were transmitted to the laboratory. Written orders could be found on laboratory requisitions or the medical record for 97.5% of 224,431 completed tests (median institution = 99.3%). Participants indicated that entry of extra tests into a hospital computer was the most common reason for completing unordered tests. In a multivariate analysis, factors associated with completing unordered tests were the lack of a policy requiring nursing staff to recheck computer orders against the medical record, average census of 301-450 patients, College of American Pathologists accreditation, and the use of preprinted "checkoff" order forms. Overall, 97.1% of 225,457 test orders were completed by the laboratories (median institution = 98.1%). Factors associated with not completing ordered tests were the lack of a policy requiring staff to check computer orders, teaching hospital status, and urban hospital location. Several interventions commonly thought to improve communication of orders were not found to affect performance. These results indicate that many institutions have a problem accurately transmitting test orders to their clinical laboratories.

Employee competence and performance-based assessment: A college of American pathologists Q-probes study of larboratory personnel in 522 institutions.

OBJECTIVE: To survey employee competence assessment practices in departments of pathology and laboratory medicine and provide suggestions for improvement. DESIGN: A 3-part study consisting of a questionnaire about current competence assessment practices, an evaluation of compliance with stated competence assessment practices using personnel records of 30 employees, and a written appraisal of competence of 5 specimen-processing staff members per institution. SETTING: A total of 522 institutions participating in the College of American Pathologists 1996 Q-Probes program. MAIN OUTCOME MEASURES: Institutional competence assessment practices, compliance of each institution with their own practices, and determination of competence of specimen-processing personnel. RESULTS: Of the participating institutions, 89.8% had a written competence plan and 98.1% reported reviewing employee competence at least yearly. General competence was reviewed by direct observations (87.5%), review of test or quality control results (77.4%), review of instrument preventive maintenance (60.0%), written testing (52.2%), and/or other methods (20.8%). In 8.6% of institutions, employees who failed competence assessment were not allowed to continue their usual work. On review of records of 14 029 employees for adherence to the laboratory's general competence plan, adherence was 89.7% for direct observations, 85.8% for review of quality control and test results, 78.0% for review of instrument records, and 74.0% for written testing. Employee failure rate ranged from 0.9% to 6.4%, depending on the competence evaluated. Adherence to an institution's plan was 90.4% for new employees, 93.1% for computer skills, 95.8% for laboratory safety, and 92.1% for continuing education. When a written competence assessment was given to 2853 specimen-processing staff members, 90.0% responded satisfactorily. CONCLUSIONS: Opportunities for improvement in employee competence assessment are numerous, and we provide several specific suggestions.

Gynecologic cytology turnaround time. A College of American Pathologists Q-Probes Study of 371 laboratories.

OBJECTIVES: To determine the turnaround time for gynecologic cytology in a large sample of laboratories and to identify laboratory and specimen characteristics associated with better and worse performance. DESIGN AND SETTING: Prospective evaluation of gynecologic cytology turnaround times in 371 laboratories. MAIN OUTCOME MEASURE: Gynecologic cytology case turnaround time. RESULTS: Three hundred seventy-one laboratories submitted information regarding laboratory characteristics and processes, and turnaround times of 66 042 gynecologic cytology cases. Half of the participating laboratories had mean turnaround times of 6 calendar days or less and were able to complete 90% of their cases within 8 calendar days. Ten percent of participants had mean turnaround times greater than 13 days and required 19 or more days to report 90% of their cases. Longer turnaround times were associated with the use of reference laboratories for all or part of the evaluation; contacting the physician's office for additional information; using cytotechnology students, residents, or fellows in the evaluation; and providing service on the weekend. CONCLUSION: Practice patterns contribute to the long turnaround times for gynecologic cytology found in some laboratories and may be improved by local site-specific process analysis.

Six-year trends in productivity and utilization of 73 clinical laboratories: a College of American Pathologists Laboratory Management Index Program study.

OBJECTIVES: To describe longitudinal trends in the efficiency, labor productivity, and utilization of clinical laboratories in the United States. METHODS: Financial and activity data were prospectively collected from 73 clinical laboratories continuously enrolled in the College of American Pathologists Laboratory Management Index Program from 1994 through 1999. Each laboratory reported quarterly on its costs, labor inputs, and test activity using uniform data definitions. RESULTS: During the 6-year study period, there was a significant increase in laboratory labor productivity (2.1% more tests/full-time equivalent/y; P <.001). Productivity increases were offset by increasing labor expense (1.5%/full-time equivalent/y; P <.001), consumable expense (1.7%/on-site test/y; P =.005), and blood expense, which comprised more than 10% of laboratory expenses by 1999 (4.4% increase/y; P <.001). As a result, overall expense per test showed no significant change in non-inflation-adjusted dollars. Reference laboratory expense per test did not change significantly during the study period; the proportion of tests sent to reference laboratories grew slightly (0.06% increase/y; P <.001). Test volume of the median laboratory grew by 5442 tests per year (2.3% annual increase; P <.001), while the proportion of testing from inpatients declined by 1.7% per year (P <.001). Inpatient test utilization declined on a discharge basis (annual decline of 1.2 tests/inpatient discharge; P <.001) and on a per diem basis (annual decline of 0.08 tests/inpatient day; P =.002). Inpatient laboratory expense declined on a discharge basis (annual decline of $2.40 or 1.3% per discharge; P <.001), but did not change significantly per inpatient day. Most of the reduction in the expense per discharge occurred during 1994-1996. CONCLUSIONS: Between 1994 and 1999, clinical laboratories in the United States experienced significant changes in the cost of operations, utilization, and labor productivity. Laboratory administrators who compare local institutional performance with that of peers are advised to use current or forward-trended peer data. Quarter-to-quarter improvement in many measures of laboratory financial activity may not signal a superior operation, as performance of the whole industry appears to be improving.

Interinstitutional variation in glycohemoglobin monitoring and glycemic control of diabetic patients.

OBJECTIVES: To compare how frequently institutions monitor glycohemoglobin in diabetic patients, the level of glycemic control achieved and to identify institutional factors associated with higher rates of monitoring and lower glycohemoglobin levels. METHODS: A total of 212 institutions retrospectively abstracted laboratory and outpatient records of up to 30 diabetic patients who had initial glycohemoglobin monitoring performed in their laboratories. Data from a cohort of 5586 diabetic patients and 17 365 assays were analyzed. RESULTS: Overall, 31.3% of patients underwent glycohemoglobin monitoring at least quarterly, the frequency recommended by the American Diabetes Association (ADA) to stabilize patients at target hemoglobin A(1c) (HbA(1c)) levels. A total of 64.9% of patients were monitored at least semiannually, the ADA recommendation for patients with stable diabetes in glycemic control (final HbA(1c) level <7%). When we compared the top and bottom deciles of the 212 institutions, there was more than an eightfold difference in the proportion of patients monitored at least quarterly and more than a twofold difference in the proportion of patients monitored at least semiannually. Glycemic control was assessed by examining the value of the last glycohemoglobin determination on record after at least 8 months of management. For all 5586 diabetic patients, the median value of the last HbA(1c) assay was 7.4%. Comparing the top and bottom deciles, there was almost a fourfold difference among institutions in the proportion of diabetic patients in glycemic control. The use of reminders to order glycohemoglobin monitoring was associated with higher rates of semiannual monitoring (P <.05) and tighter glycemic control (P <.05). In addition, patients who were monitored more frequently experienced glycohemoglobin reductions of greater magnitude (P <.001). The presence of diabetes clinics and the use of rapid methods for testing glycohemoglobin were not associated with monitoring frequency or glycohemoglobin levels. CONCLUSIONS: There is wide interinstitutional variation in the frequency with which diabetic patients are monitored and the level of glycemic control achieved. The use of prompting systems to remind providers to order glycohemoglobin monitoring was associated with more frequent monitoring and superior glycemic control.

Semiquantitation of bacteria in sputum gram stains.

In many clinical laboratories, bacteria seen in Gram-stained sputum smears are reported semiquantitatively, using a three- or four-category scale consisting of ratings such as numerous, moderate, rare, and none seen. The consistency with which these categories are assigned was evaluated by repeatedly presenting coded smears to seven experienced microbiology technologists. Technologists rated the same smear twice, pairs of smears prepared from the same specimen, and smears prepared after specimen refrigeration. Agreement was assessed with the weighted kappa test. Semiquantitation of gram-negative rods, gram-positive diplococci, and gram-positive cocci in clusters all showed poor reproducibility (kappa = 0.32, 0.34, and 0.17, respectively). Twenty-four percent of paired ratings differed by two or more categories. Lack of reproducibility was due mainly to the inability of the technologists to render a consistent rating when viewing the same slide on separate occasions (P less than 0.001). Variation in the rating styles of different technologists, differences between smears prepared from the same specimen, and specimen refrigeration tended to further decrease the consistency of ratings, but the reductions were not statistically significant. The quantity of potentially pathogenic bacteria in sputum smears is not estimated consistently with standard microscopy procedures and should not be reported.

Randomized trial interpreting sputum quality in a clinical laboratory.

The role for laboratory interpretation of microbiologic results remains controversial, and many laboratories leave the interpretation of culture results entirely to physicians. We examined the effects of furnishing a laboratory interpretation of sputum quality on physician decision making. Quality of sputum was determined on Gram-stained smears by using a modification of the criteria of Bartlett (R. C. Bartlett, Medical Microbiology: Quality, Cost, and Clinical Relevance, p. 24-31, 1974). A total of 301 poor-quality specimens were randomized either to receive written interpretation of Gram stain results or to a control group for which Gram stain results were reported without interpretation. Physicians were more likely to follow up a poor-quality specimen with a second specimen if they had been furnished an interpretation of the results from the original Gram stain (22 versus 12%; P = 0.025). We conclude that laboratory-based interpretation of microbiologic results can improve physician decision making.

Alteration of physicians' orders by nonphysicians.

We prospectively evaluated the accuracy with which clerical and laboratory staff carried out physicians' written orders for thyroid function testing in 181 patients at two institutions. In 54% of the patients studied, support staff were found to have added or deleted one or more tests from the original written orders. When the ordering physicians were asked to identify the clinical indications for the tests they had ordered, 37% of their orders either lacked an appropriate test or included an unnecessary test request. In contrast, after clerical and laboratory staff had changed orders, only 25% of patients had inappropriate tests performed or necessary tests omitted. Clerical staff using computer-based ordering menus significantly improved the appropriateness of physicians' orders compared with clerks who lacked such guidance. Laboratory technologists who used informal, knowledge-based rules also tended to improve the appropriateness of physicians' orders. We conclude that a substantial proportion of physicians' orders for diagnostic tests may be modified during the test requisitioning process. The use of testing regimens by ward clerks and laboratory technologists may explain their ability to improve on physicians' orders.

Purulent pericarditis caused by Candida: report of three cases and identification of high-risk populations as an aid to early diagnosis.

Purulent pericarditis due to fungal organisms is rare and often unrecognized because of the subtle clinical clues and insidious onset. The records of 11 cases of purulent pericarditis were selected from records of 11,000 cases of pericarditis at Duke University Medical Center and reviewed, and experience with three cases of candida purulent pericarditis (CPP) was evaluated. One case occurred in a patient recovering from complicated cardiac surgery, one in a patient with hematologic malignancy, and one in an alcoholic patient requiring intubation for a severe respiratory infection. Each case is representative of a group at increased risk for the development of CPP. Given the poor prognosis for CPP, treatment should include both medical and surgical interventions. Although amphotericin B achieves good penetration into the inflamed pericardial space, the only survivors of CPP have received both amphotericin B and pericardiectomy. Careful attention to clinical indications of pericardial inflammation and systemic infection in the three groups of patients may lead to earlier recognition of CPP, implementation of appropriate therapy, and perhaps a higher rate of cure.