Recommendations for reporting measures of diagnostic accuracy.

This article serves as the third in a series that offers recommendations for optimal data reporting, specifically focusing on the statistical methods most frequently reported in Cytopathology articles. Measures of diagnostic accuracy were among the most commonly reported results in Cytopathology, with sensitivity and/or specificity reported in 32% of the reviewed articles. This article will provide a brief overview of common measures of diagnostic accuracy, including sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, overall diagnostic accuracy, receiver operating characteristic (ROC) curve, and area under the curve (AUC). Reporting recommendations for these measures will be reviewed, including the suggestion of providing a 2 × 2 contingency table when possible, or numerator and denominator values for calculations when all values needed for a contingency table are not known, and calculation of ROC and AUC if relevant. Additionally, paired measures should be reported, including sensitivity with specificity, positive with negative predictive values, and positive with negative likelihood ratios. Calculating 95% confidence intervals (CI) for the measures is recommended, with several methods to choose from, including the Wald interval, Wilson interval, Clopper-Pearson interval, Agresti-Coull interval, and Bayesian highest posterior density (HPD) interval. Since there are various methods for CI calculations, the author encourages the reader to consult with a trained statistician to identify the most appropriate method based on the data, which should be reported in the methods section of the resulting write-up.

Parametric or nonparametric statistical tests: Considerations when choosing the most appropriate option for your data.

This article serves as the second in a series that offers recommendations for optimal data reporting, specifically focusing on statistical methods most frequently reported by the Cytopathology audience. The inaugural article, Recommendations for reporting statistical results when comparing proportions, dealt with the most common category of reported statistical tests over 2.5 years of Cytopathology articles-comparing proportions. Comparing samples using t tests, Mann-Whitney U, analysis of variance, and Kruskal-Wallis tests was another common category of statistical test reported among this audience. An important distinction between these tests is based on whether the samples follow a normal distribution. Therefore, Parametric or nonparametric statistical tests: Choosing the most appropriate option for your data is the second topic in the series. While this article will review considerations when selecting parametric or nonparametric statistical tests, an extensive review of each method is beyond the scope of this summary. The author encourages the reader to consult with a trained statistician to map out a thorough analytical plan (including their recommendations for the appropriate statistical test[s] to use) prior to data collection.

Recommendations for reporting statistical results when comparing proportions.

This article serves as the first in a series that will offer recommendations for optimal data reporting, specifically focusing on statistical methods most frequently reported by the Cytopathology audience. Recommendations for reporting statistical results when comparing proportions is the first topic in the series since comparing proportions, particularly utilising the chi-square test, was the most commonly reported inferential statistical test category over 2.5 years of Cytopathology articles.1 While this article will provide a summary of common tests used to compare proportions, an extensive review of each method, including test assumptions and calculations, is beyond the scope of this summary. The author recommends consulting a trained statistician prior to data collection to discuss an analytical plan, including determining the appropriate statistical test(s) to use.

Opportunity to elevate the quality of data reporting in Cytopathology.

Accurate and thorough data analysis and reporting are essential to ensuring that the findings of research studies are interpreted appropriately. A detailed methods section in a manuscript contains enough information to permit a researcher with access to the study components, including the data, to replicate the findings. This descriptive study of 2.5 years of Cytopathology articles uncovers opportunities to elevate the quality of methods and data reporting. The findings include the following: approximately 4 out of 5 articles (79%) identified the calculations or statistical test performed, nearly 3 out of 4 articles (74%) that involved significance testing included the threshold for significance in the methods section, 7 out of 10 (70%) articles defined the abbreviations and/or symbols used, and roughly half (51%) of the articles documented the software used for analysis. Omission of any of these areas of reporting (calculations or statistics performed, significance level utilized, abbreviations and/or symbols applied, and software used) could lead to a misinterpretation in the reported study or the inability to replicate findings in similar research. This study identified that there is an opportunity to elevate the quality of data reporting in this journal.

Amended reports: development and validation of a taxonomy of defects.

Amended pathology reports produce rework, confusion, and distrust. To develop a reproducible amendment taxonomy we derived a classification from 141 amended reports, then validated it with 130 new cases before 4 observers independently reviewed 430 cases measuring agreement (k). Next, agreement in classifying 30 other amended reports in 7 institutions was measured. We further tracked amendment rates, defect categories, defect discoverers, and discovery mechanisms. In the 430-case validation set agreement was excellent (k = 0.8780 [range, 0.8416-0.9144]). Among the 7 institutions, agreement was good (k = 0.6235 [range, 0.3105-0.8975]). Amendment rates ranged from 2.6 to 4.8 per 1,000 reports. Misinterpretation fractions varied least (23%-29%). Misidentification fractions ranged more widely (20%-38%). Specimen defects were least frequent (4%-10%) and report defects most frequent (29%-48%). Misidentifications and report defects inversely correlated. Pathologists discovered most misinterpretations, and clinicians found most misidentifications. Conference review revealed 40% to 80% of misinterpretations. This taxonomy produced excellent reproducibility and good agreement across institutions.

Dissemination of Lean methods to improve Pap testing quality and patient safety.

OBJECTIVE: To determine if the implementation of Lean methods resulted in improved Pap test quality and diagnostic accuracy in 5 clinician practices. MATERIALS AND METHODS: We performed a 1-year case-control study that included 5,384 control (preintervention) and 5,442 case (postintervention) women who had a Pap test procured by 1 of 5 clinicians. Using Lean methods, the clinicians increased their focus of Pap test procurement by creating a "one-by-one" workflow and recorded process completion using a Lean checklist. We compared the case and control Pap test quality and accuracy measures using the proportion of Pap tests lacking a transformation zone component, proportion of unsatisfactory Pap tests, frequency of newly detected cervical intraepithelial neoplasia following a previous benign Pap test, and proportion of Pap tests with a diagnosis of atypical squamous cells of unknown significance. RESULTS: After the intervention, there was a statistically significant decrease in the mean proportion of Pap tests lacking a transformation zone component, p =.011. Two of 5 clinicians showed a statistically significant decrease in their unsatisfactory Pap test frequency, although the overall Pap test unsatisfactory frequency for the case group was not statistically significant lower, p =.087. The case group showed a 114% increase in newly detected cervical intraepithelial neoplasia following a previous benign Pap test, p =.004. There was no statistically significant difference for the proportion of Pap tests with a diagnosis of atypical squamous cells of unknown significance, p =.908. CONCLUSIONS: Disseminating Lean methods across a group of clinicians resulted in improved Pap test quality and diagnostic accuracy.

Frequency and outcome of cervical cancer prevention failures in the United States.

We measured the frequency and outcome of cervical cancer prevention failures that occurred in the Papanicolaou (Pap) and colposcopy testing phases involving 1,646,580 Pap tests in 4 American hospital systems between January 1, 1998, and December 31, 2004. We defined a screening failure as a 2-step or greater discordant Pap test result and follow-up biopsy diagnosis. A total of 5,278 failures were detected (0.321% of all Pap tests); 48% and 52% of failures occurred in the Pap test and colposcopy phases, respectively. Missed squamous cancers (1 in 187,786 Pap tests), glandular cancers (1 in 19,426 Pap tests), and high-grade lesions (1 in 6,870 Pap tests) constituted 4.1% of all failures. Unnecessary repeated tests or diagnostic delays occurred in 70.8% and 63.9% of failures involving high- and low-grade lesions, respectively. We conclude that cervical cancer prevention practices are remarkably successful in preventing squamous cancers, although a high frequency of failures results in low-impact negative outcomes.

Urine cytology discrepancies: frequency, causes, and outcomes.

Although urine cytology is used for the early detection and surveillance of urothelial carcinoma, there has been little study of the frequency, causes, and outcomes of cytologic-histologic discrepancies. We obtained histologic follow-up in 361 (6.2%) of 5,785 voided, 124 (19.5%) of 636 lower tract instrumented, and 23 (33%) of 69 upper tract urinary cytologic specimens from 1 institution during a 2-year timeframe to determine diagnostic discrepancy frequency and outcomes. Cytologic-histologic discrepancies were observed in 208 (40.9%) cases with histologic followup, and the cause of discrepancy was interpretation and sampling in 35.1% and 63.0%, respectively. Of all discrepancies, 101 (48.6%) resulted in minimal or mild harm, consisting mainly of repeated testing and/or diagnostic delays. Severe harm never was observed. We conclude that current screening and surveillance methods that incorporate urine cytology are accurate in diagnosing urothelial cancer. However, the current protocols result in potentially reducible errors that lead to unnecessary testing and diagnostic delays.

Effectiveness of Toyota process redesign in reducing thyroid gland fine-needle aspiration error.

Our objective was to determine whether the Toyota Production System process redesign resulted in diagnostic error reduction for patients who underwent cytologic evaluation of thyroid nodules. In this longitudinal, nonconcurrent cohort study, we compared the diagnostic error frequency of a thyroid aspiration service before and after implementation of error reduction initiatives consisting of adoption of a standardized diagnostic terminology scheme and an immediate interpretation service. A total of 2,424 patients underwent aspiration. Following terminology standardization, the false-negative rate decreased from 41.8% to 19.1% (P = .006), the specimen nondiagnostic rate increased from 5.8% to 19.8% (P < .001), and the sensitivity increased from 70.2% to 90.6% (P < .001). Cases with an immediate interpretation had a lower noninterpretable specimen rate than those without immediate interpretation (P < .001). Toyota process change led to significantly fewer diagnostic errors for patients who underwent thyroid fine-needle aspiration.

Errors in thyroid gland fine-needle aspiration.

Scant published data exist on redesigning pathology practice based on error data. In this first step of an Agency for Healthcare Research and Quality patient safety project, we measured the performance metrics of thyroid gland fine-needle aspiration, performed root cause analysis to determine the causes of error, and proposed error-reduction initiatives to address specific errors. Eleven cytologists signed out 1,543 thyroid gland aspirates in 2 years, and surgical pathology follow-up was obtained in 364 patients. Of the 364 patients, 91 (25.0%) had a false-negative diagnosis and 36 (9.9%) a false-positive diagnosis. Root cause analysis showed that major sources of error were pre-analytic (poor specimen quality) and analytic (interpretation of unsatisfactory specimens as nonneoplastic and lack of diagnostic category standardization). We currently are evaluating the effectiveness of error reduction initiatives that target pre-analytic and analytic portions of the diagnostic pathway.

Double slide viewing as a cytology quality improvement initiative.

Few studies have measured the effect of pre-sign out double viewing of cytology cases as a means to decrease error. Three Agency for Healthcare Research and Quality-funded project sites performed pre-sign out double viewing of 431 pulmonary cytology cases. Two-step or more differences in diagnosis were arbitrated as interpretive errors, and the effect of double viewing was measured by comparing the frequency of cytologic-histologic correlation-detected errors in the previous 2 years with the double-viewing period. The number of interpretive errors detected by double viewing for the 3 institutions was 2.7%, 0% and 1.9%, respectively. Double viewing did not lower the frequency of cytologic-histologic correlation false-negative errors. We conclude that double viewing detects errors in up to 1 of every 37 cases and that biases in the double-viewing process limit error detection.

Variability in cytologic-histologic correlation practices and implications for patient safety.

CONTEXT: The Clinical Laboratory Improvement Amendments of 1988 require that laboratories perform cytologic-histologic correlation, although the optimal methods and the value of performing correlation have not been determined. OBJECTIVE: To determine the similarities and differences in how laboratories perform cytologic-histologic correlation. DESIGN: One hundred sixty-two American laboratories were sent a letter requesting copies of the materials they used in the cytologic-histologic correlation process. The returned materials were classified into the categories of forms, logs, and tally sheets. A checklist (derived from the College of American Pathologists Laboratory Accreditation Cytopathology Checklist) was developed to classify the "minimum expected" (15) and "additional" data points that laboratories collected when they performed a correlation. PARTICIPANTS: American pathology laboratories. MAIN OUTCOME MEASURES: Measures were percentage of laboratories that recorded minimum expected and additional data points and the frequency with which specific minimum expected data points were recorded. RESULTS: The response frequency was 32.1%, and a total of 84 cytologic-histologic correlation materials were obtained. The only minimum expected variables recorded on forms or logs by more than 50% of laboratories were cytology case number, sign-out cytology diagnosis, surgical pathology case number, and sign-out surgical pathology diagnosis. Nine (17.3%) laboratories did not record data on forms, logs, or tally sheets. The mean number of minimum expected and additional variables recorded on forms was 6.5 and 8.7, respectively. CONCLUSIONS: Laboratories record data from the cytologic-histologic correlation process in a number of ways, indicating the lack of standardization of the data collection process.

Use of physician extenders in surgical pathology practice.

CONTEXT: Use of a variety of nonphysician personnel for surgical pathology gross examination is generally known to be increasing, although detailed information regarding nonphysician use is currently unavailable. OBJECTIVE: To measure and describe the use of nonphysician personnel for surgical pathology gross examination in order to gain a better understanding of the current surgical pathology workforce. DESIGN: A voluntary, mailed questionnaire containing items related to the use of multiple nonphysician personnel types in surgical pathology was distributed to (1) a cross-sectional sample (n = 968) of US pathologists and (2) a purposive sample of pathologist directors of surgical and/or anatomic pathology (n = 77) located at teaching institutions. Responses were analyzed using descriptive statistics, correlation analyses, the chi2 test, and 1-way analysis of variance. Staffing ratios were calculated for multiple nonphysician personnel types. RESULTS: The overall response rate was 22% (n = 225). Of the US sample, 56% of respondents reported using nonphysician laboratory personnel to perform gross examinations, compared with 91% of the directors' sample. The most frequently reported personnel type for both samples was pathologists' assistants, but multiple other personnel types were used as well. Significant associations existed between certain practice types and personnel types used, as well as differences in the scope of responsibilities between personnel types. Calculated staffing ratios were variable across personnel types and were highest for pathologists' assistants. CONCLUSIONS: The use of a variety of nonphysician laboratory personnel for surgical pathology gross examination is common, particularly in academic pathology practice. Further studies are needed to examine the impact of physician extenders on laboratory efficiency and quality of care.

Pathology resident attitudes and opinions about pathologists' assistants.

CONTEXT: Changes in health care economics and organization have resulted in increased use of nonphysician providers in most health care settings. Attitudinal acceptance of nonphysician providers is important in the current health care environment. OBJECTIVES: To obtain descriptive information regarding pathology resident attitudes and opinions about pathologists' assistants in anatomic pathology practice and to assess the implications of resident attitudes and opinions for pathology practice and training. DESIGN: A self-administered, mailed, voluntary, anonymous questionnaire was distributed to a cross-sectional sample of pathology residents in the United States (2531 pathology residents registered as resident members of one of the national pathology professional organizations). The questionnaire contained (1) items relating to resident demographics and program characteristics, (2) Likert-scale response items containing positive and negative statements about pathologists' assistants, (3) a multiple-choice item related to pathologists' assistants scope of practice, and (4) an open-ended item inviting additional comments. Both quantitative and qualitative analysis of responses was performed. RESULTS: The overall response rate was 19.4% (n = 490); 50% of the respondents were women, and 77% reported use of pathologists' assistants in their program. Most respondents were 25 to 35 years old and in postgraduate years 3 through 5 of their training, and most were located in the Midwestern United States. The majority of residents expressed overall positive attitudes and opinions about pathologists' assistants and felt that pathologists' assistants enhanced resident training by optimizing resident workload. A minority (10%-20%) of residents expressed negative attitudes or opinions about pathologists' assistants. Additionally, some residents reported a lack of knowledge about pathologists' assistants' training or roles. CONCLUSIONS: Increased resident education and open discussion concerning pathologists' assistants may be beneficial for optimizing resident attitudes about and training experiences with pathologists' assistants.