ABSTRACT
Background Despite variation in performance characteristics among radiologists, the pairing of radiologists for the double reading of screening mammograms is performed randomly. It is unknown how to optimize pairing to improve screening performance. Purpose To investigate whether radiologist performance characteristics can be used to determine the optimal set of pairs of radiologists to double read screening mammograms for improved accuracy. Materials and Methods This retrospective study was performed with reading outcomes from breast cancer screening programs in Sweden (2008-2015), England (2012-2014), and Norway (2004-2018). Cancer detection rates (CDRs) and abnormal interpretation rates (AIRs) were calculated, with AIR defined as either reader flagging an examination as abnormal. Individual readers were divided into performance categories based on their high and low CDR and AIR. The performance of individuals determined the classification of pairs. Random pair performance, for which any type of pair was equally represented, was compared with the performance of specific pairing strategies, which consisted of pairs of readers who were either opposite or similar in AIR and/or CDR. Results Based on a minimum number of examinations per reader and per pair, the final study sample consisted of 3 592 414 examinations (Sweden, n = 965 263; England, n = 837 048; Norway, n = 1 790 103). The overall AIRs and CDRs for all specific pairing strategies (Sweden AIR range, 45.5-56.9 per 1000 examinations and CDR range, 3.1-3.6 per 1000; England AIR range, 68.2-70.5 per 1000 and CDR range, 8.9-9.4 per 1000; Norway AIR range, 81.6-88.1 per 1000 and CDR range, 6.1-6.8 per 1000) were not significantly different from the random pairing strategy (Sweden AIR, 54.1 per 1000 examinations and CDR, 3.3 per 1000; England AIR, 69.3 per 1000 and CDR, 9.1 per 1000; Norway AIR, 84.1 per 1000 and CDR, 6.3 per 1000). Conclusion Pairing a set of readers based on different pairing strategies did not show a significant difference in screening performance when compared with random pairing. © RSNA, 2023.
Subject(s)
Mammography , Physical Examination , Humans , Retrospective Studies , England , RadiologistsABSTRACT
BACKGROUND: This protocol concerns the assessment of cost-effectiveness of hospital health information technology (HIT) in four hospitals. Two of these hospitals are acquiring ePrescribing systems incorporating extensive decision support, while the other two will implement systems incorporating more basic clinical algorithms. Implementation of an ePrescribing system will have diffuse effects over myriad clinical processes, so the protocol has to deal with a large amount of information collected at various 'levels' across the system. METHODS/DESIGN: The method we propose is use of Bayesian ideas as a philosophical guide.Assessment of cost-effectiveness requires a number of parameters in order to measure incremental cost utility or benefit - the effectiveness of the intervention in reducing frequency of preventable adverse events; utilities for these adverse events; costs of HIT systems; and cost consequences of adverse events averted. There is no single end-point that adequately and unproblematically captures the effectiveness of the intervention; we therefore plan to observe changes in error rates and adverse events in four error categories (death, permanent disability, moderate disability, minimal effect). For each category we will elicit and pool subjective probability densities from experts for reductions in adverse events, resulting from deployment of the intervention in a hospital with extensive decision support. The experts will have been briefed with quantitative and qualitative data from the study and external data sources prior to elicitation. Following this, there will be a process of deliberative dialogues so that experts can "re-calibrate" their subjective probability estimates. The consolidated densities assembled from the repeat elicitation exercise will then be used to populate a health economic model, along with salient utilities. The credible limits from these densities can define thresholds for sensitivity analyses. DISCUSSION: The protocol we present here was designed for evaluation of ePrescribing systems. However, the methodology we propose could be used whenever research cannot provide a direct and unbiased measure of comparative effectiveness.
Subject(s)
Electronic Prescribing/economics , Models, Economic , Technology Assessment, Biomedical , Algorithms , Bayes Theorem , Cost-Benefit Analysis , Humans , Medication Errors/prevention & control , Quality of Life , Quality-Adjusted Life YearsABSTRACT
PURPOSE: To develop a model that simulates radiologist assessments and use it to explore whether pairing readers based on their individual performance characteristics could optimize screening performance. METHODS: Logistic regression models were designed and used to model individual radiologist assessments. For model evaluation, model-predicted individual performance metrics and paired disagreement rates were compared against the observed data using Pearson correlation coefficients. The logistic regression models were subsequently used to simulate different screening programs with reader pairing based on individual true-positive rates (TPR) and/or false-positive rates (FPR). For this, retrospective results from breast cancer screening programs employing double reading in Sweden, England, and Norway were used. Outcomes of random pairing were compared against those composed of readers with similar and opposite TPRs/FPRs, with positive assessments defined by either reader flagging an examination as abnormal. RESULTS: The analysis data sets consisted of 936,621 (Sweden), 435,281 (England), and 1,820,053 (Norway) examinations. There was good agreement between the model-predicted and observed radiologists' TPR and FPR (r ≥ 0.969). Model-predicted negative-case disagreement rates showed high correlations (r ≥ 0.709), whereas positive-case disagreement rates had lower correlation levels due to sparse data (r ≥ 0.532). Pairing radiologists with similar FPR characteristics (Sweden: 4.50% [95% confidence interval: 4.46%-4.54%], England: 5.51% [5.47%-5.56%], Norway: 8.03% [7.99%-8.07%]) resulted in significantly lower FPR than with random pairing (Sweden: 4.74% [4.70%-4.78%], England: 5.76% [5.71%-5.80%], Norway: 8.30% [8.26%-8.34%]), reducing examinations sent to consensus/arbitration while the TPR did not change significantly. Other pairing strategies resulted in equal or worse performance than random pairing. CONCLUSIONS: Logistic regression models accurately predicted screening mammography assessments and helped explore different radiologist pairing strategies. Pairing readers with similar modeled FPR characteristics reduced the number of examinations unnecessarily sent to consensus/arbitration without significantly compromising the TPR. HIGHLIGHTS: A logistic-regression model can be derived that accurately predicts individual and paired reader performance during mammography screening reading.Pairing screening mammography radiologists with similar false-positive characteristics reduced false-positive rates with no significant loss in true positives and may reduce the number of examinations unnecessarily sent to consensus/arbitration.
ABSTRACT
Evidence-based clinical guidelines are essential to maximize patient benefit and to reduce clinical uncertainty and inconsistency in clinical practice. Gaps in the evidence base can be addressed by data acquired in routine practice. At present, there is no international consensus on management of women diagnosed with atypical lesions in breast screening programmes. Here, we describe how routine NHS breast screening data collected by the Sloane atypia project was used to inform a management pathway that maximizes early detection of cancer and minimizes over-investigation of lesions with uncertain malignant potential. A half-day consensus meeting with 11 clinical experts, 1 representative from Independent Cancer Patients' Voice, 6 representatives from NHS England (NHSE) including from Commissioning, and 2 researchers was held to facilitate discussions of findings from an analysis of the Sloane atypia project. Key considerations of the expert group in terms of the management of women with screen detected atypia were: (1) frequency and purpose of follow-up; (2) communication to patients; (3) generalizability of study results; and (4) workforce challenges. The group concurred that the new evidence does not support annual surveillance mammography for women with atypia, irrespective of type of lesion, or woman's age. Continued data collection is paramount to monitor and audit the change in recommendations.
Subject(s)
Breast Neoplasms , Clinical Decision-Making , Female , Humans , Consensus , Uncertainty , Breast/diagnostic imaging , Breast/pathology , Mammography/methods , Breast Neoplasms/diagnostic imaging , Breast Neoplasms/pathologyABSTRACT
BACKGROUND: To develop patient-reported outcome instruments, statistical techniques (e.g., principal components analysis; PCA) are used to examine correlations among items and identify interrelated item subsets (empirical factors). However, interpretation and labelling of empirical factors is a subjective process, lacking precision or conceptual basis. We report a novel and reproducible method for mapping between theoretical and empirical factor structures. We illustrate the method using the pilot Aberdeen Glaucoma Questionnaire (AGQ), a new measure of glaucoma-related disability developed using the International Classification of Functioning and Disability (ICF) as a theoretical framework and tested in a sample representing the spectrum of glaucoma severity. METHODS: We used the ICF to code AGQ item content before mailing the AGQ to a UK sample (N = 1349) selected to represent people with a risk factor for glaucoma and people with glaucoma across a range of severity. Reflecting uncertainty in the theoretical framework (items with multiple ICF codes), an exploratory PCA was conducted. The theoretical structure informed our interpretation of the empirical structure and guided the selection of theoretically-derived factor labels. We also explored the discrimination of the AGQ across glaucoma severity groups. RESULTS: 656 (49%) completed questionnaires were returned. The data yielded a 7-factor solution with a simple structure (using cut-off point of a loading of 0.5) that together accounted for 63% of variance in the scores. The mapping process resulted in allocation of the following theoretically-derived factor labels: 1) Seeing Functions: Participation; 2) Moving Around & Communication; 3) Emotional Functions; 4) Walking Around Obstacles; 5) Light; 6) Seeing Functions: Domestic & Social Life; 7) Mobility. Using the seven factor scores as independent variables in a discriminant function analysis, the AGQ scores resulted in correct glaucoma severity grading of 32.5% of participants (p < 0.001). CONCLUSIONS: This paper addresses a methodological gap in the application of classical test theory (CTT) techniques, such as PCA, in instrument development. Labels for empirically-derived factors are often selected intuitively whereas they can inform existing bodies of knowledge if selected on the basis of theoretical construct labels, which are more explicitly defined and which relate to each other in ways that are evidence based.
Subject(s)
Disability Evaluation , Glaucoma/diagnosis , Self Report , Surveys and Questionnaires , Aged , Cross-Sectional Studies , Empirical Research , Female , Health Status , Humans , Male , Middle Aged , Pilot Projects , Principal Component Analysis , Reproducibility of Results , Self Report/standards , Severity of Illness IndexABSTRACT
CONTEXT: Vitamin D or calcium supplementation may have effects on vascular disease and cancer. OBJECTIVE: Our objective was to investigate whether vitamin D or calcium supplementation affects mortality, vascular disease, and cancer in older people. DESIGN AND SETTING: The study included long-term follow-up of participants in a two by two factorial, randomized controlled trial from 21 orthopedic centers in the United Kingdom. PARTICIPANTS: Participants were 5292 people (85% women) aged at least 70 yr with previous low-trauma fracture. INTERVENTIONS: Participants were randomly allocated to daily vitamin D(3) (800 IU), calcium (1000 mg), both, or placebo for 24-62 months, with a follow-up of 3 yr after intervention. MAIN OUTCOME MEASURES: All-cause mortality, vascular disease mortality, cancer mortality, and cancer incidence were evaluated. RESULTS: In intention-to-treat analyses, mortality [hazard ratio (HR) = 0.93; 95% confidence interval (CI) = 0.85-1.02], vascular disease mortality (HR = 0.91; 95% CI = 0.79-1.05), cancer mortality (HR = 0.85; 95% CI = 0.68-1.06), and cancer incidence (HR = 1.07; 95% CI = 0.92-1.25) did not differ significantly between participants allocated vitamin D and those not. All-cause mortality (HR = 1.03; 95% CI = 0.94-1.13), vascular disease mortality (HR = 1.07; 95% CI = 0.92-1.24), cancer mortality (HR = 1.13; 95% CI = 0.91-1.40), and cancer incidence (HR = 1.06; 95% CI = 0.91-1.23) also did not differ significantly between participants allocated calcium and those not. In a post hoc statistical analysis adjusting for compliance, thus with fewer participants, trends for reduced mortality with vitamin D and increased mortality with calcium were accentuated, although all results remain nonsignificant. CONCLUSIONS: Daily vitamin D or calcium supplementation did not affect mortality, vascular disease, cancer mortality, or cancer incidence.
Subject(s)
Calcium/administration & dosage , Cholecalciferol/administration & dosage , Mortality , Neoplasms/epidemiology , Osteoporotic Fractures/drug therapy , Aged , Aged, 80 and over , Cause of Death , Dietary Supplements , Drug Combinations , Female , Follow-Up Studies , Humans , Male , Mortality/trends , Neoplasms/mortality , Osteoporotic Fractures/complications , Osteoporotic Fractures/epidemiology , Osteoporotic Fractures/mortality , Placebos , Time Factors , Vascular Diseases/epidemiology , Vascular Diseases/mortalityABSTRACT
OBJECTIVE: To determine whether inclusion of glutamine, selenium, or both in a standard isonitrogenous, isocaloric preparation of parenteral nutrition influenced new infections and mortality among critically ill patients. DESIGN: Randomised, double blinded, factorial, controlled trial. SETTING: Level 2 and 3 (or combined) critical care units in Scotland. All 22 units were invited, and 10 participated. PARTICIPANTS: 502 adults in intensive care units and high dependency units for ≥ 48 hours, with gastrointestinal failure and requiring parenteral nutrition. INTERVENTIONS: Parenteral glutamine (20.2 g/day) or selenium (500 µg/day), or both, for up to seven days. MAIN OUTCOME MEASURES: Primary outcomes were participants with new infections in the first 14 days and mortality. An intention to treat analysis and a prespecified analysis of patients who received ≥ 5 days of the trial intervention are presented. Secondary outcomes included critical care unit and acute hospital lengths of stay, days of antibiotic use, and modified SOFA (Sepsis-related Organ Failure Assessment) score. RESULTS: Selenium supplementation did not significantly affect patients developing a new infection (126/251 v 139/251, odds ratio 0.81 (95% CI 0.57 to 1.15)), except for those who had received ≥ 5 days of supplementation (odds ratio 0.53 (0.30 to 0.93)). There was no overall effect of glutamine on new infections (134/250 v 131/252, odds ratio 1.07 (0.75 to 1.53)), even if patients received ≥ 5 days of supplementation (odds ratio 0.99 (0.56 to 1.75)). Six month mortality was not significantly different for selenium (107/251 v 114/251, odds ratio 0.89 (0.62 to 1.29)) or glutamine (115/250 v 106/252, 1.18 (0.82 to 1.70)). Length of stay, days of antibiotic use, and modified SOFA score were not significantly affected by selenium or glutamine supplementation. CONCLUSIONS: The primary (intention to treat) analysis showed no effect on new infections or on mortality when parenteral nutrition was supplemented with glutamine or selenium. Patients who received parenteral nutrition supplemented with selenium for ≥ 5 days did show a reduction in new infections. This finding requires confirmation. Trial registration Current Controlled Trials ISRCTN87144826.
Subject(s)
Critical Illness/therapy , Cross Infection/prevention & control , Glutamine/administration & dosage , Parenteral Nutrition , Selenium/administration & dosage , Aged , Critical Care , Critical Illness/mortality , Cross Infection/mortality , Dietary Supplements , Double-Blind Method , Drug Combinations , Female , Humans , Male , Middle Aged , Quality of Life , Treatment OutcomeABSTRACT
OBJECTIVES: A treatment pathway model was developed to examine the costs and benefits of the current bowel cancer service in England and to evaluate potential alternatives in service provision. To use the pathway model, various parameters and probability distributions had to be specified. They could not all be determined from empirical evidence and, instead, expert opinion was elicited in the form of statistical quantities that gave the required information. The purpose of this study is to describe the procedures used to quantify expert opinion and note examples of good practice contained in the case study. METHODS: The required information was identified and preparatory discussion with four experts refined the questions they would be asked. In individual elicitation sessions they quantified their opinions, mainly in the form of point and interval estimates for specified variables. New methods have been developed for quantifying expert opinion and these were implemented in specialized software that uses interactive graphics. This software was used to elicit opinion about quantities related to measurable covariates. RESULTS: Assessments for thirty-four quantities were elicited and available checks supported their validity. Eight points of good practice in eliciting and using expert judgment were evident. Parameters and probability distributions needed for the pathway model were determined from the elicited assessments. Simulation results from the pathway model were used to inform policy on bowel cancer service provision. CONCLUSIONS: The study illustrates that quantifying and using expert judgment can be acceptable in real problems of practical importance. For full benefit to be gained from expert knowledge, elicitation must be conducted carefully and should be reported in detail.